进程池中传递实例方法问题

1、多个进程,参数给实例方法,逐次运行时没有问题的,代码如下:

import multiprocessing

class MyClass(object):
    def calc(self,num):
        print 'the number is ',num

if __name__ == '__main__':
    mc = MyClass()
    ths = [multiprocessing.Process(target = mc.calc,args = (i,)) for i in range(3)]
    for th in ths:
        th.start()

运行结果:

the number is  0
the number is  1
the number is  2

2、放到进程池中一起并发

import multiprocessing

class MyClass(object):
    def calc(self,num):
        print 'the number is ',num

if __name__ == '__main__':
    mc = MyClass()
    pool = multiprocessing.Pool(processes = 3)
    pool.apply_async(mc.calc,(3,))
    pool.apply_async(mc.calc,(4,))
    pool.apply_async(mc.calc,(5,))
    pool.close()
    pool.join()
    print 'end!'

运行结果:

end!

只打印了最终结果,传递进去的实例方法压根没有运行!!!

3、解决方法如下:

import multiprocessing
import copy_reg
import types

def _reduce_method(m):
    if m.im_self is None:
        return getattr, (m.im_class, m.im_func.func_name)
    else:
        return getattr, (m.im_self, m.im_func.func_name)
copy_reg.pickle(types.MethodType, _reduce_method)

class MyClass(object):
    def calc(self,num):
        print 'the number is ',num

if __name__ == '__main__':
    mc = MyClass()
    pool = multiprocessing.Pool(processes = 3)
    pool.apply_async(mc.calc,(3,))
    pool.apply_async(mc.calc,(4,))
    pool.apply_async(mc.calc,(5,))
    pool.close()
    pool.join()
    print 'end!'

运行结果:

the number is  4
the number is  3
the number is  5
end!

当在子进程的进程池中使用实例方法,程序实际调用multiprocessing.Pipe进行序列化数据。python2.7中,multiprocessing.Pipe使用C语言实现的,会立即调用

pickle_dumps,而不是调用ForkingPickler,所以传递进去的实例方法无法工作。

然而,如果使用copy_reg注册实例方法,如上不可能将变为可能。

python3.0之后的版本,可正常序列化实例方法,就不需要使用copy_reg了。 

详细说明还可参考:

http://stackoverflow.com/questions/27318290/why-can-i-pass-an-instance-method-to-multiprocessing-process-but-not-a-multipro 

  

  

  

  

  

原文地址:https://www.cnblogs.com/hester/p/5141512.html