Memory error for large dataset (5 million spikes, 120 channels)

[shabnamkadir]

(globalphy)skadir@martinotti:~/globalphy/global_superclustering/global_code$ python parallel_global_script.py
INFO     klustakwik: KlustaKwik2 version 0.2.2
Running script:  parallel_global_script.py
(4667599, 120)
(4667599, 360)
Writing /home/skadir/globalphy/nicktest/nick_global_4.6676e+06.fet.1 file
Writing /home/skadir/globalphy/nicktest/nick_global_4.6676e+06.fmask.1 file
Making full adjacency graph
{0: {47, 0, 14, 15}, 1: {1, 13, 46, 14}, 2: {2, 12, 45, 13}, 4: {10, 43, 11, 4}, 5: {9, 42, 10, 5}, 6: {8, 41, 9, 6}, 7: {8, 7, 39}, 8: {7, 8, 6, 39}, 9: {9, 5, 6, 38}, 10: {10, 4, 5, 37}, 11: {11, 4, 36, 12, 44}, 12: {2, 35, 11, 44, 12}, 13: {1, 2, 34, 13}, 14: {0, 1, 33, 14}, 15: {0, 32, 15, 31}, 16: {31, 16, 30, 63}, 17: {17, 29, 30, 62}, 18: {18, 28, 29, 61}, 19: {27, 28, 60, 52, 19}, 21: {26, 27, 53, 21}, 22: {25, 26, 54, 22}, 23: {24, 25, 23, 55}, 24: {56, 23, 24, 71}, 25: {57, 25, 22, 23}, 26: {58, 26, 21, 22}, 27: {59, 19, 21, 52, 27}, 28: {18, 51, 19, 28}, 29: {17, 50, 18, 29}, 30: {16, 49, 17, 30}, 31: {16, 32, 15, 31, 63}, 32: {63, 31, 47, 32, 15}, 33: {14, 33, 46, 47}, 34: {34, 45, 13, 46}, 35: {35, 44, 12, 45}, 36: {43, 11, 44, 36}, 37: {42, 10, 43, 37}, 38: {41, 9, 42, 38}, 39: {8, 41, 39, 7}, 41: {39, 41, 38, 6}, 42: {42, 5, 38, 37}, 43: {43, 4, 37, 36}, 44: {35, 36, 11, 12, 44}, 45: {2, 35, 34, 45}, 46: {1, 34, 33, 46}, 47: {0, 33, 32, 47}, 49: {63, 49, 62, 30}, 50: {50, 29, 62, 61}, 51: {51, 28, 61, 60}, 52: {59, 60, 19, 27, 52}, 53: {58, 59, 21, 53}, 54: {57, 58, 22, 54}, 55: {56, 57, 55, 23}, 56: {24, 71, 55, 56, 103}, 57: {25, 57, 54, 55}, 58: {26, 58, 53, 54}, 59: {27, 52, 53, 59}, 60: {51, 19, 52, 60}, 61: {50, 18, 51, 61}, 62: {49, 17, 50, 62}, 63: {16, 49, 32, 63, 31}, 64: {96, 80, 64, 79}, 65: {97, 65, 78, 79}, 66: {98, 66, 77, 78}, 67: {99, 68, 77, 108, 67}, 68: {75, 100, 67, 108, 68}, 69: {74, 75, 101, 69}, 70: {72, 105, 73, 70}, 71: {24, 72, 56, 71, 103}, 72: {71, 72, 70, 103}, 73: {73, 74, 70, 102}, 74: {73, 106, 74, 69}, 75: {107, 68, 69, 75}, 77: {66, 67, 109, 77}, 78: {65, 66, 110, 78}, 79: {64, 65, 79, 111}, 80: {64, 96, 95, 80, 127}, 81: {113, 81, 94, 95}, 82: {114, 82, 93, 94}, 83: {115, 92, 93, 83}, 85: {90, 91, 117, 85}, 86: {89, 90, 118, 86}, 87: {88, 89, 87, 119}, 88: {120, 88, 87}, 89: {121, 89, 86, 87}, 90: {122, 90, 85, 86}, 91: {123, 92, 85, 116, 91}, 92: {83, 124, 91, 116, 92}, 93: {82, 83, 125, 93}, 94: {81, 82, 126, 94}, 95: {80, 81, 95, 127}, 96: {64, 80, 111, 96, 127}, 97: {65, 97, 110, 111}, 98: {66, 98, 109, 110}, 99: {67, 108, 109, 99}, 100: {107, 68, 108, 100}, 101: {106, 107, 69, 101}, 102: {105, 73, 106, 102}, 103: {72, 105, 56, 103, 71}, 105: {103, 105, 102, 70}, 106: {74, 106, 101, 102}, 107: {75, 100, 101, 107}, 108: {99, 100, 68, 67, 108}, 109: {98, 99, 77, 109}, 110: {97, 98, 78, 110}, 111: {96, 97, 111, 79}, 113: {81, 113, 126, 127}, 114: {82, 114, 125, 126}, 115: {83, 124, 125, 115}, 116: {123, 124, 92, 91, 116}, 117: {122, 123, 85, 117}, 118: {121, 122, 86, 118}, 119: {120, 121, 119, 87}, 120: {88, 120, 119}, 121: {89, 121, 118, 119}, 122: {90, 122, 117, 118}, 123: {91, 116, 117, 123}, 124: {115, 92, 116, 124}, 125: {114, 115, 93, 125}, 126: {113, 114, 94, 126}, 127: {96, 113, 80, 127, 95}}
Making globalcl_dict
Traceback (most recent call last):
  File "parallel_global_script.py", line 84, in <module>
    pg.make_subset_fetmask(fetmask_dict, fetty, triplemasky, channel_order_dict, full_adjacency, globalcl_dict)
  File "/home/skadir/globalphy/global_superclustering/global_code/parallel_global.py", line 159, in make_subset_fetmask
    masky_group = triplemasky[:,pcs_inds]
MemoryError

[shabnamkadir]

It may be better to exploit the sparse structure present in KK2.