InvalidStateError while estimating sparsity

[JeffreyBoucher]

Hello all!

I've been trying to run kilosort 3 on a concatenated Neuropixels 2 dataset. Lately I've been running into an issue with create_sorting_analyzer, while it is estimating sparsity. Basically, the code is able to run about 70-80% (not any specific number) of the way through, then I get an exception "InvalidStateError". I guess this means that some aspect of my data doesn't work well with the sparsity-estimating algorithm, but I have no guess what that would be.

Here is an example of the exception:

Exception in thread Thread-2:
Traceback (most recent call last):
File "/home/sjjgjbo/.conda/envs/neurovis_try2/lib/python3.9/threading.py", line 980, in _bootstrap_inner
self.run()
File "/home/sjjgjbo/.conda/envs/neurovis_try2/lib/python3.9/concurrent/futures/process.py", line 323, in run
self.terminate_broken(cause)
File "/home/sjjgjbo/.conda/envs/neurovis_try2/lib/python3.9/concurrent/futures/process.py", line 458, in terminate_broken
work_item.future.set_exception(bpe)
File "/home/sjjgjbo/.conda/envs/neurovis_try2/lib/python3.9/concurrent/futures/_base.py", line 549, in set_exception
raise InvalidStateError('{}: {!r}'.format(self._state, self))
concurrent.futures._base.InvalidStateError: CANCELLED: <Future at 0x2aaf7896ea00 state=cancelled>

Additionally, here are the inputs into create_sorting_analyzer:

we = si.create_sorting_analyzer(recording=rec, sorting=sorting, folder=outDir / 'sortings_folder',
format="binary_folder",
sparse=True
)

Please help me if you can, I would be very greatful, it's been confounding. Let me know if I can offer any additional information that would help!

Thanks,

Jeff Boucher

[zm711]

I don't think any of us are using 3.9 at this point. But this is good to know. Our test suite is on 3.9 and doesn't have a problem. I'm not sure. I think we need to ping @alejoe91 and @samuelgarcia to take a look at this.

What n_jobs are you using? Could you try n_jobs=1 to confirm it is a multiprocessing issue.?

[JeffreyBoucher]

Hello!

Thanks for the response! I am using n_jobs -1, which should be 10 cores on the cluster I'm using. I'll try with n_jobs = 1 and get back to you!

[zm711]

Thanks let us know with the n_jobs=1. Sometimes there can be issues with how a server shares resources so we need to trouble shoot 3 things:

1. python 3.9 issue
2. multiprocessing issue
3. spikeinterface + server issue

[JeffreyBoucher]

Hello!

Setting n_jobs = 1 indeed let me get through the sparsity estimation without error! Naturally this takes much longer to do, though.

Since you have implied I might be able to solve my problem by updating from python 3.9 I'll maybe give that a shot next. It's been a while since I chose my version, but I think having 3.9 isn't critical at this stage of the pipeline.

Thanks for your help! I'll let you know if changing versions doesn't solve it for me; let me know if you need any more information from me.

Jeffrey Boucher

[zm711]

Yeah it would be great if you could test python 3.10 or 3.11. There have been some improvements in multiprocessing at the python level. If updating python works it tells us that 3.9 might not be as well supported as we thought for our multiprocessing. If 3.10/3.11/3.12 doesn't work then it might be a problem in our multiprocessing itself.