I am learning how to use the threading and the multiprocessing modules in Python to run certain operations in parallel and speed up my code.
I am finding this hard (maybe because I don’t have any theoretical background about it) to understand what the difference is between a threading.Thread() object and a multiprocessing.Process() one.
Also, it is not entirely clear to me how to instantiate a queue of jobs and having only 4 (for example) of them running in parallel, while the other wait for resources to free before being executed.
I find the examples in the documentation clear, but not very exhaustive; as soon as I try to complicate things a bit, I receive a lot of weird errors (like a method that can’t be pickled, and so on).
So, when should I use the threading and multiprocessing modules?
Can you link me to some resources that explain the concepts behind these two modules and how to use them properly for complex tasks?
