Multiprocessing: Pool And Pickle Error -- Pickling Error: Can't Pickle : Attribute Lookup __builtin__.instancemethod Failed

I have two files: x.py class BF(object) def __init__(): . . def add(self,z): . . y.py from y import BF def FUNC((a,b,bf)) . . bf.add(x) . . ret

Solution 1:

I'm going to basically use what you have above, but turn it into working code. There is no problem serializing, if you use dill. I'm using a fork of multiprocessing called pathos.multiprocessing, which uses dill instead of pickle.

>>> def FUNC((a,b,bf)):
...   z = a+b
...   bf.add(z)
...   return bf
... 
>>> class BF(object):
...   def add(self, z):
...     self.z += z
...   def __init__(self):
...     self.z = 0
... 
>>> from pathos.multiprocessing import ProcessingPool as Pool
>>> pool = Pool()
>>> 
>>> f = BF()
>>> f.add(1)
>>> f.z
1
>>> 
>>> FUNC((0,1,f))
<__main__.BF object at 0x10d387f50>
>>> 
>>> FUNC((0,1,f)).z
2
>>> 
>>> sl = [BF() for i in range(10)]
>>> results = pool.map(FUNC, zip(range(len(sl)), range(len(sl)), sl))
>>> [bf.z for bf in results]
[0, 2, 4, 6, 8, 10, 12, 14, 16, 18]

This works, because pathos uses dill, which can serialize almost anything in python.

>>> import dill as pickle
>>> pickle.loads(pickle.dumps(bf.add))
<bound method BF.add of <__main__.BF object at 0x10d383950>>
>>> pickle.loads(pickle.dumps(BF.add))
<unbound method BF.add>

Get pathos and dill at: https://github.com/uqfoundation