Why Mamba2 runs out of GPU memory when it faces to extreme long inputs?

[YuBicheng]

When I used Mamba2 for tasks related to ultra long sequence inputs, I found that it would result in GPU memory depletion errors. SSD should theoretically be able to handle inputs of infinite length. Every time an error is reported in the mamba_stplit_comv1d_scan_combined() function

[YuBicheng]

Note: The problem occurred during the testing phase, and it can be run after the input is truncated

[Hprairie]

Mamba2 still uses extra memory to store hidden states in between chunks. If your chunk size is too small, then you could be creating a bunch of extra tensors that take up memory. It should also be noted that increasing the chunk_size too large could slow down the kernels which work better for shorter lengths.