I am learning how to use the threading and the multiprocessing modules in Python to run certain operations in parallel and speed up my code. I am finding this hard (maybe because I don’t have any theoretical background about it) to understand what the difference is between a threading.Thread() object and a multiprocessing.Process() one. Also, … Read more
When I run something like: from multiprocessing import Pool p = Pool(5) def f(x): return x*x p.map(f, [1,2,3]) it works fine. However, putting this as a function of a class: class calculate(object): def run(self): def f(x): return x*x p = Pool() return p.map(f, [1,2,3]) cl = calculate() print cl.run() Gives me the following error: Exception … Read more
Python 3.2 introduced Concurrent Futures, which appear to be some advanced combination of the older threading and multiprocessing modules. What are the advantages and disadvantages of using this for CPU bound tasks over the older multiprocessing module? This article suggests they’re much easier to work with – is that the case? 6 Answers 6
I am trying my very first formal python program using Threading and Multiprocessing on a windows machine. I am unable to launch the processes though, with python giving the following message. The thing is, I am not launching my threads in the main module. The threads are handled in a separate module inside a class. … Read more
I’m trying to use multiprocessing‘s Pool.map() function to divide out work simultaneously. When I use the following code, it works fine: import multiprocessing def f(x): return x*x def go(): pool = multiprocessing.Pool(processes=4) print pool.map(f, range(10)) if __name__== ‘__main__’ : go() However, when I use it in a more object-oriented approach, it doesn’t work. The error … Read more
I found that in Python 3.4 there are few different libraries for multiprocessing/threading: multiprocessing vs threading vs asyncio. But I don’t know which one to use or is the “recommended one”. Do they do the same thing, or are different? If so, which one is used for what? I want to write a program that … Read more
I’m trying to learn how to use Python’s multiprocessing package, but I don’t understand the difference between map_async and imap. I noticed that both map_async and imap are executed asynchronously. So when should I use one over the other? And how should I retrieve the result returned by map_async? Should I use something like this? … Read more
Right now I have a central module in a framework that spawns multiple processes using the Python 2.6 multiprocessing module. Because it uses multiprocessing, there is module-level multiprocessing-aware log, LOG = multiprocessing.get_logger(). Per the docs, this logger has process-shared locks so that you don’t garble things up in sys.stderr (or whatever filehandle) by having multiple … Read more
I am sorry that I can’t reproduce the error with a simpler example, and my code is too complicated to post. If I run the program in IPython shell instead of the regular Python, things work out well. I looked up some previous notes on this problem. They were all caused by using pool to … Read more
I have not seen clear examples with use-cases for Pool.apply, Pool.apply_async and Pool.map. I am mainly using Pool.map; what are the advantages of others? 3 Answers 3
