Distributed.jl - possibility to use other Serialization libraries?

[schlichtanders]

Hi,
since recently Pluto can run Distributed, however it does not work out of the box because the standard serialization is known not to work in Pluto, but Distributed relies on it (I couldn't find a way to change this)

The PlutoSerialization is maintained in this repo https://github.com/MichaelHatherly/PlutoSerialization.jl

It would be great to be able to use this Serialization instead of the default Serialization when using Distributed inside a Pluto notebook.

[jpsamaroo]

ClusterSerializer could have a hook for customizing name binding lookups. I'm not sure what such a design would look like in the face of multiple libraries wanting to hook this functionality. Probably a hook could be defined per-Module, so that this hook will only be called on modules that Pluto creates.

[schlichtanders]

Looking into which reasoning was taken for the global random number generator it seems that the preferred route is to make the ClusterSerializer pluggable. JuliaLang/julia#23199 (comment)

These are the two places where the ClusterSerializer is constructed

• Distributed.jl/src/cluster.jl

  Line 122 in fdf56f4

  w.w_serializer = ClusterSerializer(w.w_stream)

• Distributed.jl/src/process_messages.jl

  Line 159 in fdf56f4

  serializer = ClusterSerializer(r_stream)

So I guess we would need:

• an AbstractClusterSerializer type for clarity
• and make the above two places (and respective wrappers around it) somehow being able to pass an AbstractClusterSerializer object (similar to how rand and other methods pass a RNG object)

[schlichtanders]

@MichaelHatherly you build PlutoSerialization.jl, probably you have some valuable insights what could be potential solutions/problems

[MichaelHatherly]

Thanks for pointing this out @schlichtanders. I've got no extra insights at the moment since I've not had a chance yet to look into how Malt changes things now that Pluto is using it instead. If I get a chance I'll update this thread.

[jpsamaroo]

Malt shouldn't be relevant here since this issue is relating to running code under Distributed directly (IIRC Pluto only runs its own internal code under Malt)

[schlichtanders]

I would really love to see Distributed be fully supported inside Pluto and would like to try out things myself. (e.g. creating the pluggable ClusterSerializer interface thought about above)

@jpsamaroo do you know whether and how Distributed.jl could already be replace by a fork or the like?