Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

recorders cannot be serialised with dill #34

Open
VolodyaCO opened this issue Feb 15, 2023 · 1 comment
Open

recorders cannot be serialised with dill #34

VolodyaCO opened this issue Feb 15, 2023 · 1 comment
Assignees
@AthenaCaesura
Labels
bug Something isn't working

Comments

@VolodyaCO
Copy link
Contributor

VolodyaCO commented Feb 15, 2023 •
edited
Loading

Describe the bug

Can't serialise optimisers with dill because of the recorder

How to Reproduce

>>> opt = ScipyOptimizer("Powell", options={"maxiter":1})
>>> dill.dumps(opt)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/vladimir/micromamba/envs/zapata/lib/python3.8/site-packages/dill/_dill.py", line 263, in dumps
    dump(obj, file, protocol, byref, fmode, recurse, **kwds)#, strictio)
  File "/Users/vladimir/micromamba/envs/zapata/lib/python3.8/site-packages/dill/_dill.py", line 235, in dump
    Pickler(file, protocol, **_kwds).dump(obj)
  File "/Users/vladimir/micromamba/envs/zapata/lib/python3.8/site-packages/dill/_dill.py", line 394, in dump
    StockPickler.dump(self, obj)
  File "/Users/vladimir/micromamba/envs/zapata/lib/python3.8/pickle.py", line 487, in dump
    self.save(obj)
  File "/Users/vladimir/micromamba/envs/zapata/lib/python3.8/site-packages/dill/_dill.py", line 388, in save
    StockPickler.save(self, obj, save_persistent_id)
  File "/Users/vladimir/micromamba/envs/zapata/lib/python3.8/pickle.py", line 603, in save
    self.save_reduce(obj=obj, *rv)
  File "/Users/vladimir/micromamba/envs/zapata/lib/python3.8/pickle.py", line 717, in save_reduce
    save(state)
  File "/Users/vladimir/micromamba/envs/zapata/lib/python3.8/site-packages/dill/_dill.py", line 388, in save
    StockPickler.save(self, obj, save_persistent_id)
  File "/Users/vladimir/micromamba/envs/zapata/lib/python3.8/pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
  File "/Users/vladimir/micromamba/envs/zapata/lib/python3.8/site-packages/dill/_dill.py", line 1186, in save_module_dict
    StockPickler.save_dict(pickler, obj)
  File "/Users/vladimir/micromamba/envs/zapata/lib/python3.8/pickle.py", line 971, in save_dict
    self._batch_setitems(obj.items())
  File "/Users/vladimir/micromamba/envs/zapata/lib/python3.8/pickle.py", line 997, in _batch_setitems
    save(v)
  File "/Users/vladimir/micromamba/envs/zapata/lib/python3.8/site-packages/dill/_dill.py", line 388, in save
    StockPickler.save(self, obj, save_persistent_id)
  File "/Users/vladimir/micromamba/envs/zapata/lib/python3.8/pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
  File "/Users/vladimir/micromamba/envs/zapata/lib/python3.8/site-packages/dill/_dill.py", line 1824, in save_function
    _save_with_postproc(pickler, (_create_function, (
  File "/Users/vladimir/micromamba/envs/zapata/lib/python3.8/site-packages/dill/_dill.py", line 1084, in _save_with_postproc
    pickler._batch_setitems(iter(source.items()))
  File "/Users/vladimir/micromamba/envs/zapata/lib/python3.8/pickle.py", line 997, in _batch_setitems
    save(v)
  File "/Users/vladimir/micromamba/envs/zapata/lib/python3.8/site-packages/dill/_dill.py", line 388, in save
    StockPickler.save(self, obj, save_persistent_id)
  File "/Users/vladimir/micromamba/envs/zapata/lib/python3.8/pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
  File "/Users/vladimir/micromamba/envs/zapata/lib/python3.8/site-packages/dill/_dill.py", line 1698, in save_type
    _save_with_postproc(pickler, (_create_type, (
  File "/Users/vladimir/micromamba/envs/zapata/lib/python3.8/site-packages/dill/_dill.py", line 1084, in _save_with_postproc
    pickler._batch_setitems(iter(source.items()))
  File "/Users/vladimir/micromamba/envs/zapata/lib/python3.8/pickle.py", line 997, in _batch_setitems
    save(v)
  File "/Users/vladimir/micromamba/envs/zapata/lib/python3.8/site-packages/dill/_dill.py", line 388, in save
    StockPickler.save(self, obj, save_persistent_id)
  File "/Users/vladimir/micromamba/envs/zapata/lib/python3.8/pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
  File "/Users/vladimir/micromamba/envs/zapata/lib/python3.8/site-packages/dill/_dill.py", line 1698, in save_type
    _save_with_postproc(pickler, (_create_type, (
  File "/Users/vladimir/micromamba/envs/zapata/lib/python3.8/site-packages/dill/_dill.py", line 1084, in _save_with_postproc
    pickler._batch_setitems(iter(source.items()))
  File "/Users/vladimir/micromamba/envs/zapata/lib/python3.8/pickle.py", line 997, in _batch_setitems
    save(v)
  File "/Users/vladimir/micromamba/envs/zapata/lib/python3.8/site-packages/dill/_dill.py", line 388, in save
    StockPickler.save(self, obj, save_persistent_id)
  File "/Users/vladimir/micromamba/envs/zapata/lib/python3.8/pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
  File "/Users/vladimir/micromamba/envs/zapata/lib/python3.8/site-packages/dill/_dill.py", line 1824, in save_function
    _save_with_postproc(pickler, (_create_function, (
  File "/Users/vladimir/micromamba/envs/zapata/lib/python3.8/site-packages/dill/_dill.py", line 1084, in _save_with_postproc
    pickler._batch_setitems(iter(source.items()))
  File "/Users/vladimir/micromamba/envs/zapata/lib/python3.8/pickle.py", line 997, in _batch_setitems
    save(v)
  File "/Users/vladimir/micromamba/envs/zapata/lib/python3.8/site-packages/dill/_dill.py", line 388, in save
    StockPickler.save(self, obj, save_persistent_id)
  File "/Users/vladimir/micromamba/envs/zapata/lib/python3.8/pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
  File "/Users/vladimir/micromamba/envs/zapata/lib/python3.8/site-packages/dill/_dill.py", line 1824, in save_function
    _save_with_postproc(pickler, (_create_function, (
  File "/Users/vladimir/micromamba/envs/zapata/lib/python3.8/site-packages/dill/_dill.py", line 1084, in _save_with_postproc
    pickler._batch_setitems(iter(source.items()))
  File "/Users/vladimir/micromamba/envs/zapata/lib/python3.8/pickle.py", line 997, in _batch_setitems
    save(v)
  File "/Users/vladimir/micromamba/envs/zapata/lib/python3.8/site-packages/dill/_dill.py", line 388, in save
    StockPickler.save(self, obj, save_persistent_id)
  File "/Users/vladimir/micromamba/envs/zapata/lib/python3.8/pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
  File "/Users/vladimir/micromamba/envs/zapata/lib/python3.8/site-packages/dill/_dill.py", line 1698, in save_type
    _save_with_postproc(pickler, (_create_type, (
  File "/Users/vladimir/micromamba/envs/zapata/lib/python3.8/site-packages/dill/_dill.py", line 1070, in _save_with_postproc
    pickler.save_reduce(*reduction, obj=obj)
  File "/Users/vladimir/micromamba/envs/zapata/lib/python3.8/pickle.py", line 692, in save_reduce
    save(args)
  File "/Users/vladimir/micromamba/envs/zapata/lib/python3.8/site-packages/dill/_dill.py", line 388, in save
    StockPickler.save(self, obj, save_persistent_id)
  File "/Users/vladimir/micromamba/envs/zapata/lib/python3.8/pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
  File "/Users/vladimir/micromamba/envs/zapata/lib/python3.8/pickle.py", line 901, in save_tuple
    save(element)
  File "/Users/vladimir/micromamba/envs/zapata/lib/python3.8/site-packages/dill/_dill.py", line 388, in save
    StockPickler.save(self, obj, save_persistent_id)
  File "/Users/vladimir/micromamba/envs/zapata/lib/python3.8/pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
  File "/Users/vladimir/micromamba/envs/zapata/lib/python3.8/site-packages/dill/_dill.py", line 1186, in save_module_dict
    StockPickler.save_dict(pickler, obj)
  File "/Users/vladimir/micromamba/envs/zapata/lib/python3.8/pickle.py", line 971, in save_dict
    self._batch_setitems(obj.items())
  File "/Users/vladimir/micromamba/envs/zapata/lib/python3.8/pickle.py", line 997, in _batch_setitems
    save(v)
  File "/Users/vladimir/micromamba/envs/zapata/lib/python3.8/site-packages/dill/_dill.py", line 388, in save
    StockPickler.save(self, obj, save_persistent_id)
  File "/Users/vladimir/micromamba/envs/zapata/lib/python3.8/pickle.py", line 578, in save
    rv = reduce(self.proto)
TypeError: cannot pickle '_abc_data' object
>>> opt.recorder = None
>>> dill.dumps(opt)
b'\x80\x04\x95\xb1\x00\x00\x00\x00\x00\x00\x00\x8c(orquestra.opt.optimizers.scipy_optimizer\x94\x8c\x0eScipyOptimizer\x94\x93\x94)\x81\x94}\x94(\x8c\x08recorder\x94N\x8c\x06method\x94\x8c\x06Powell\x94\x8c\x07options\x94}\x94\x8c\x07maxiter\x94K\x01s\x8c\x0bconstraints\x94]\x94\x8c\x06bounds\x94N\x8c\x15store_best_parameters\x94\x88ub.'
>>>

Expected behavior

It really seems as though the recorder cannot be serialised with dill:

>>> opt = ScipyOptimizer("Powell", options={"maxiter":1})
>>> dill.dumps(opt.recorder)

fails

Actual behavior

See above

Environment (please complete the following information):**

  • OS: OSX
  • Python version 3.8

Additional context

See https://github.com/zapatacomputing/orquestra-qml-core/issues/430
@VolodyaCO VolodyaCO added the bug Something isn't working label Feb 15, 2023
@VolodyaCO
Copy link
Contributor Author

A possible solution is to define __getstate__ and __setstate__ for recorders: https://stackoverflow.com/questions/1939058/simple-example-of-use-of-setstate-and-getstate

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@AthenaCaesura AthenaCaesura

Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@VolodyaCO @AthenaCaesura