lzmp – multiprocessing for the lazy

lzmp is a simple wrapper around the multiprocessing module, allowing the lazy programmer to run batches of processes.

lzmp contains the class Pool which lets the user specify one or more callable objects (such as functions) along with lists of arguments to process. lzmp collects the return value of each call and returns the whole lot as a list, keeping the original submission order. For a single type of callable, the standalone function run() wraps the wrapper and allows one-line parallelization.

Installation

> pip install lzmp

Reference

lzmp.run(f, args, final=None, final_args=None, max_threads=None, shuffle=False, echo=False)

Convenience function to parallelise a single type of tasks.

Parameters

  • f – callable (typically a function) to execute

  • args – iterable of arguments (each one must be a sequence). wrap() can generate a proper iterable out of a single-item iterable.

  • final – optional callable (function) to apply to the list of results. See Pool.run() for details.

  • final_args – arguments to pass to final if specified.

  • max_threads – maximum number of process to run simultaneously.

  • shuffle – randomly shuffle tasks. Will not affect the order or returned items.

  • echo – return the arguments (but not the callable) along with the corresponding results, wrapped in two-item tuples (args, res). If arguments are all one-item tuples, they are unwrapped automatically. It is not possible to set both final and echo.

Returns

The list of return values of the passed iterable, or the return value of final, if specified.

lzmp.wrap(iterable)

Return a generator wherein each item yields by iterable is included as a single-item tuple.

class lzmp.Pool(max_threads=None)

Class to run tasks in parallel. Can be used to run different functions at once.

max_threads can be passed as an argument at object creation to limit the number of processes created simultaneously. Values below 1 and None are ignored.

add(f, args)

Add a type of tasks.

Parameters

  • f – callable.

  • args – iterable of arguments values (each as a list of tuples).

property max_threads

maximum number of simultaneous processes

run(final=None, final_args=None, shuffle=False, echo=False)

Run the requested tasks. If shuffle is true, the order of tasks is randomized. Return a list of return values of the called function (order is not altered by shuffling). If final is not None, call this function on this list and return the result. The callable passed as final will receive the combined return values of the callable passed to Pool.add() as first argument and, if specified, final_args as second argument. If echo is set, the arguments (but not the callable) are returned along with the corresponding results, wrapped in two-item tuples (args, res). If arguments are all one-item tuples, they are unwrapped automatically. It is not possible to set both final and echo.

History

  • 0.2.0: Add the echo argument.

  • 0.1.0: Initial release.