Search This Blog

Tuesday, March 25, 2014

Multiprocessing (in Python)


I needed to do some multi-threading in Python. As I needed effect quick I decided to use standard threading module. However, every time I had to use it I felt it was rather complicated beast. At threading module documentation page there is a link to multiprocessing module. I was tired with threading, on the one hand, but didn't have enough time to learn about greenlet or another competing project, on the other, so I decided to take a quick glance at multiprocessing module...
... And my life became much easier, sky bluer, grass, greener, oh and scripts faster ;-).
I don't do anything special with it, so just one simple code example, but this is very good tutorial you can find much more: http://pymotw.com/2/multiprocessing/communication.html.

Main part of nearly all my scripts looks the same:

import multiprocessing 


SIZE = 30
pool = multiprocessing.Pool(processes=SIZE)
pool_output = pool.map(get_values, servers)
pool.close() # no more tasks
pool.join()

Where servers is a list with servers I need get information from, and get_values is a function (sometimes with a different name).  Simple, isn't it?
Post a Comment