Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

TCPTransport: multiple "dictionary changed size during iteration" issues #68

Open
rayrapetyan opened this issue Mar 27, 2022 · 0 comments

Comments

@rayrapetyan
Copy link

opensearch-benchmark 0.0.2
thespian 3.10.1

While running opensearch-benchmark (based on thespian) I'm sporadically getting following errors in logs:

2022-03-27 00:35:36.463023 p374 ERR xmit UNcaught exception dictionary changed size during iteration; aborting intent.
Traceback (most recent call last):
File "/opt/sm/python/lib/python3.9/site-packages/thespian/system/transport/TCPTransport.py", line 801, in _nextTransmitStep
return getattr(self, 'next_XMIT%s' % intent.stage,
File "/opt/sm/python/lib/python3.9/site-packages/thespian/system/transport/TCPTransport.py", line 830, in _next_XMIT_1
if any(T for T in self._transmitIntents.values()
File "/opt/sm/python/lib/python3.9/site-packages/thespian/system/transport/TCPTransport.py", line 830, in
if any(T for T in self._transmitIntents.values()
RuntimeError: dictionary changed size during iteration

2022-03-26 23:28:41.846563 p844 ERR Actor osbenchmark.worker_coordinator.worker_coordinator.Worker @ ActorAddr-(T|:19163) transport run exception: Traceback (most recent call last):
File "/opt/sm/python/lib/python3.9/site-packages/thespian/system/actorManager.py", line 87, in run
r = self.transport.run(self.handleMessages)
File "/opt/sm/python/lib/python3.9/site-packages/thespian/system/transport/wakeupTransportBase.py", line 74, in run
rval = self._run_subtransport(incomingHandler, max_runtime)
File "/opt/sm/python/lib/python3.9/site-packages/thespian/system/transport/wakeupTransportBase.py", line 80, in _run_subtransport
rval = self._runWithExpiry(incomingHandler)
File "/opt/sm/python/lib/python3.9/site-packages/thespian/system/transport/TCPTransport.py", line 1098, in _runWithExpiry
partition(TCPTransport._waitForSendable,
File "/opt/sm/python/lib/python3.9/site-packages/thespian/system/utilis.py", line 207, in partition
return foldl(appendLeftOrRight, inp_iterable, (output_type(), output_type()))
RuntimeError: dictionary changed size during iteration

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant