Multiprocessing and PyOctree error

[DavideBassano]

Hello,

My code uses PyOctree package and works fine.

I'm now trying to run my code in parallel using Python Multiprocessing.
I get this error and I have no idea about hot to solve it:

File "stringsource", line 2, in pyoctree.pyoctree.pyoctree.PyOctree.__reduce_cython__
TypeError: no default __reduce__ due to non-trivial __cinit__

How could I solve this error?

Thanks,
Cheers,

Davide

PS: I can provide more code if needed

[mhogg]

Hi Davide,
I assume this error relates to the PyOctree class? My guess is to add a __repr__ function to this class, similar to what I have in the PyOctnode class.
I will test this theory and get back to you. The question may then be what should this class return.
Cheers,
Michael

[mhogg]

Sorry, should be __reduce__ function, not __repr__. Reduce is required to pickle a class, which is what multiprocessing involves.

[DavideBassano]

Hi Michael,
thanks for your reply.

So you are suggesting to add a reduce function to the PyOctree class. I can try.
What are the input and the output of this function?
Do I have to call this function somewhere else in the code?

Thanks,
Cheers,

Davide

[mhogg]

Hi Davide,

I don't know much about pickling/unpickling of extension modules such as PyOctree, but it appears there are several ways of making the PyOctree classes picklable. A __reduce__ method is just one of these. See here. The __setstate__ and __getstate__ methods are also a higher level option.

I found quite a few links on this topic, some of which include:

1. Cython classes are non-pickable pele-python/pele#34
2. https://stackoverflow.com/questions/36301322/pickle-cython-class-with-c-pointers
3. https://snorfalorpagus.net/blog/2016/04/16/pickling-cython-classes/

You don't need to call these methods directly. You just need to provide them and they will get called automatically during pickling / unpickling. The __getstate__ methods store the current data in picklable python objects (numpy arrays, lists etc), which are then used to recreate the PyOctree classes when unpickled via __setstate__. All that should be needed to recreate the class are the initial inputs i.e. pointCoords and connectivity. That is:

tree = ot.PyOctree(vertexCoords3D,polyConnectivity)

Therefore, my guess is that the __reduce__ method for the PyOctree class might be something like:

cdef __reduce__(self):
    return (PyOctree,(self.vertexCoords3D,self.polyConnectivity))

However, vertexCoords3D and polyConnectivity are not stored in the class by default. So we will need to either modify the __cinit__ method to store them (which is undesirable, as it will use more memory).

Alternative we could add __setstate__ and __getstate__ methods. I think this would essentially involve converting and storing the values from the self.thisptr.vertexCoords3D and self.thisptr.polyConnectivity within PyOctree class as a numpy array during pickling i.e. using __getstate__. When unpickling, i.e. using __setstate__, these would be returned to recreate the PyOctree object from scratch.

Cheers,
Michael