lzmp – multiprocessing for the lazy

lzmp is a simple wrapper around the multiprocessing module, allowing the lazy programmer to run batches of processes.

lzmp contains the class Pool which lets the user specify one or more callable objects (such as functions) along with lists of arguments to process. lzmp collects the return value of each call and returns the whole lot as a list, keeping the original submission order. For a single type of callable, the standalone function run() wraps the wrapper and allows one-line parallelization.

Installation

> pip install lzmp

Reference

lzmp.run(f, args, final=None, final_args=None, post=None, max_threads=None, shuffle=False, echo=False)

Convenience function to parallelise a single type of tasks.

Parameters:

  • f – callable (typically a function) to execute

  • args – iterable of arguments (each one must be a sequence). wrap() can generate a proper iterable out of a single-item iterable.

  • final – optional callable (function) to apply to the list of results. See Pool.run() for details.

  • post – optional callable (function) to apply to the return value of each job immediately as it finishes (in the main thread).

  • final_args – arguments to pass to final if specified.

  • max_threads – maximum number of process to run simultaneously.

  • shuffle – randomly shuffle tasks. Will not affect the order or returned items.

  • echo – return the arguments (but not the callable) along with the corresponding results, wrapped in two-item tuples (args, res). If arguments are all one-item tuples, they are unwrapped automatically. It is not possible to set both final and echo.

Returns:

The list of return values of the passed iterable, or the return value of final, if specified.

Changed in version 0.2: echo argument

lzmp.wrap(iterable, *extra)

Return a generator wherein each item yields by iterable is included as a single-item tuple. Constant extra arguments are appended to the tuple at each iteration round.

Changed in version 0.3: extra arguments

class lzmp.Pool(max_threads=None)

Class to run tasks in parallel. Can be used to run different functions at once.

max_threads can be passed as an argument at object creation to limit the number of processes created simultaneously. Values below 1 and None are ignored.

add(f, args)

Add a type of tasks.

Parameters:

  • f – callable.

  • args – iterable of arguments values (each as a list of tuples).

property max_threads

maximum number of simultaneous processes

run(final=None, final_args=None, post=None, shuffle=False, echo=False)

Run the requested tasks. If shuffle is true, the order of tasks is randomized. Return a list of return values of the called function (order is not altered by shuffling). If final is not None, call this function on this list and return the result. The callable passed as final will receive the combined return values of all jobs as first argument and, if specified, final_args as second argument. If post is set, this callable is called with the return value of each job in a tuple with the job index, on the main thread, after it finishes. If echo is set, the arguments (but not the callable) are returned along with the corresponding results, wrapped in two-item tuples (args, res). If arguments are all one-item tuples, they are unwrapped automatically. It is not possible to set both final and echo.

Changed in version 0.2: echo argument

Changed in version 0.3: close automatically terminated processes upon completion

Changed in version 0.4: added post argument

History

  • 0.4.0: Add post argument.

  • 0.3.2: Fix another bug in the section checking errors.

  • 0.3.1: Fix a bug in the section checking errors (bug present in 0.3.0).

  • 0.3.0: Close forked processes while running to avoid OS saturation and

    add the extra argument to wrap().

  • 0.2.0: Add the echo argument.

  • 0.1.0: Initial release.