Python Multiprocessing Pool Timeout

I want to use multiprocessing.Pool, but multiprocessing.Pool can't abort a task after a timeout. I found solution and some modify it. from multiprocessing import util, Pool, Timeou

Solution 1:

There is no implicit risk in stopping a running job, the OS will take care of correctly terminating the process.

If your job is writing on files, you might end up with lots of truncated files on your disk.

Some small issue might also occur if you write on DBs or if you are connected with some remote process.

Nevertheless, Python standard Pool does not support worker termination on task timeout. Terminating processes abruptly might lead to weird behaviour within your application.

Pebble processing Pool does support timing-out tasks.

from pebble import process, TimeoutError

with process.Pool() as pool:
    task = pool.schedule(function, args=[1,2], timeout=5)

    try:
        result = task.get()
    except TimeoutError:
        print"Task: %s took more than 5 seconds to complete" % task