kilosort3 errors - CUDA "illegal memory access encountered" / "maximum variable size allowed on the device is exceeded"

[GuthrieDyce]

Hello world,

Summary:
I have been trying to sort a recording using kilosort 3 and I keep running into two errors in particular:
One of the errors I have been having returns the message "maximum variable size allowed on the device is exceeded".
If I close everything that is open on the computer except for the instance of matlab running kilosort then I get a different error during CUDA execution: "an illegal memory access was encountered".

Specifications/Notes:
I am using a Neuropixel 1 probe phase3b1 with staggered recording sites.
The working directory and output directories are the same (the input file is within this directory).
I am using the default specifications for this version of kilosort.
I am also using MATLAB 2019a.
The input file is 5.8 GB in size.
The hard drive used for the input, working and output folders has 3.22 TB of free space on it.
I am using Windows 10, 64 bit, 3.7 GHz, with 64 GB of RAM.
I am using an NVIDIA GeForce RTX 2060 graphics card.
It is also worth noting that I have used this computer to run kilosort on other recordings without this problem.

Attachments:
I have attached screenshots of the GUI, preferences, drift map, estimated drift traces, the kilosort folder (showing, as I understand it that I am in fact using kilosort 3), and the command line associated with the errors.

Questions:
Is the size of the input file too large?
Are the kilosort specifications wrong?
Is using the same directory for the input file as the working and output directory causing this issue?
Is there too much jitter in the recording resulting in duplication of units? Is this overwhelming the graphics card?
Can someone help me identify (& perhaps even fix) the problem?

Thanks so much!

Guthrie

[GuthrieDyce]

Additional notes
Both of the errors mentioned occur during the "template matching on binary file" stage.
The original drift map and estimated drift traces are associated with the first error ("maximum variable size...")
The drift map and estimated drift traces attached to this comment are associated with the second error ("illegal memory access...")
I hope this helps someone help me.

[achristensen56]

I also have the illegal memory access encountered issue. I was wondering if it had something to do with cuda versions and incompatibility with old kilosort, but I have the same issue on a fresh install and successful compile on a new computer! Curious if you've found the issue.

[hannahay]

Hi,
I also have the same issue with illegal memory access. I am using kilosort 2.5. Did you solve the issue?

[GuthrieDyce]

Hi Hannah, I am uncertain exactly why, but I have not been having this issue lately. The best I can do is make a few wild guesses I'm afraid:

…

-Make sure the input, working and output directories for kilosort are the same.

-Close other programs.

-Consider which version of MATLAB you are using (from memory 2019 works better than 2021).

-Turn your computer off and on again. I wish I could help more. Good luck. Guthrie

________________________________ From: hannahay ***@***.***> Sent: Tuesday, November 23, 2021 5:31:48 AM To: cortex-lab/KiloSort ***@***.***> Cc: Guthrie Dyce ***@***.***>; Author ***@***.***> Subject: Re: [cortex-lab/KiloSort] kilosort3 errors - CUDA "illegal memory access encountered" / "maximum variable size allowed on the device is exceeded" (#244) Hi, I also have the same issue with illegal memory access. I am using kilosort 2.5. Did you solve the issue? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub<https://aus01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fcortex-lab%2FKiloSort%2Fissues%2F244%23issuecomment-975806843&data=04%7C01%7Cguthrie.dyce%40anu.edu.au%7Cceb2e814db384d2121a708d9ade659cd%7Ce37d725cab5c46249ae5f0533e486437%7C0%7C0%7C637732027131694897%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=T0rAaIFbI1UAY1d%2BNlk%2FWkLQ8ns2aiPF%2BLt%2BqjXgzyc%3D&reserved=0>, or unsubscribe<https://aus01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FASAOQTKLJWQEUIHUIX45NDDUNKEBJANCNFSM5DDR2OBA&data=04%7C01%7Cguthrie.dyce%40anu.edu.au%7Cceb2e814db384d2121a708d9ade659cd%7Ce37d725cab5c46249ae5f0533e486437%7C0%7C0%7C637732027131704861%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=WhExP0Ymu7NaCxBMzEYVEAHVMgkbElcb0Nwhtur8r3E%3D&reserved=0>. Triage notifications on the go with GitHub Mobile for iOS<https://aus01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fapps.apple.com%2Fapp%2Fapple-store%2Fid1477376905%3Fct%3Dnotification-email%26mt%3D8%26pt%3D524675&data=04%7C01%7Cguthrie.dyce%40anu.edu.au%7Cceb2e814db384d2121a708d9ade659cd%7Ce37d725cab5c46249ae5f0533e486437%7C0%7C0%7C637732027131704861%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=LfaHankIaLYQ6xE3oAMTEBLoDZ5Zr3azuj9DPyUb5%2Fk%3D&reserved=0> or Android<https://aus01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fplay.google.com%2Fstore%2Fapps%2Fdetails%3Fid%3Dcom.github.android%26referrer%3Dutm_campaign%253Dnotification-email%2526utm_medium%253Demail%2526utm_source%253Dgithub&data=04%7C01%7Cguthrie.dyce%40anu.edu.au%7Cceb2e814db384d2121a708d9ade659cd%7Ce37d725cab5c46249ae5f0533e486437%7C0%7C0%7C637732027131714811%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=BQX8kffbqdZvq4ElIsD%2BVOIO0QVzCSjtGRxKHbYtECk%3D&reserved=0>.