local/cuda/lib64 -lcudart -o wkv6_v1.so Loading extension module wkv6_v1... Aborted (core dumped)

[45ghost]

您好我想问一下关于Aborted (core dumped)的问题，我一开始还能训练 后来就突然报错这个问题。 我调整了T_MAX参数。也重新配置了环境。都没有解决问题，请问您遇到相同的问题吗？这是怎么引起的

[45ghost]

/usr/local/miniconda3/envs/RWKV/lib/python3.10/site-packages/mmcv/init.py:20: UserWarning: On January 1, 2023, MMCV will release v2.0.0, in which it will remove components related to the training process and add a data transformation module. In addition, it will rename the package names mmcv to mmcv-lite and mmcv-full to mmcv. See https://github.com/open-mmlab/mmcv/blob/master/docs/en/compatibility.md for more details.
warnings.warn(
Using /root/.cache/torch_extensions/py310_cu121 as PyTorch extensions root...
Detected CUDA files, patching ldflags
Emitting ninja build file /root/.cache/torch_extensions/py310_cu121/wkv/build.ninja...
/usr/local/miniconda3/envs/RWKV/lib/python3.10/site-packages/torch/utils/cpp_extension.py:1967: UserWarning: TORCH_CUDA_ARCH_LIST is not set, all archs for visible cards are included for compilation.
If this is not desired, please set os.environ['TORCH_CUDA_ARCH_LIST'].
warnings.warn(
Building extension module wkv...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
[1/3] /usr/local/cuda/bin/nvcc --generate-dependencies-with-compile --dependency-output wkv_cuda.cuda.o.d -DTORCH_EXTENSION_NAME=wkv -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE="gcc" -DPYBIND11_STDLIB="libstdcpp" -DPYBIND11_BUILD_ABI="cxxabi1011" -isystem /usr/local/miniconda3/envs/RWKV/lib/python3.10/site-packages/torch/include -isystem /usr/local/miniconda3/envs/RWKV/lib/python3.10/site-packages/torch/include/torch/csrc/api/include -isystem /usr/local/miniconda3/envs/RWKV/lib/python3.10/site-packages/torch/include/TH -isystem /usr/local/miniconda3/envs/RWKV/lib/python3.10/site-packages/torch/include/THC -isystem /usr/local/cuda/include -isystem /usr/local/miniconda3/envs/RWKV/include/python3.10 -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS -D__CUDA_NO_HALF_CONVERSIONS_ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_89,code=compute_89 -gencode=arch=compute_89,code=sm_89 --compiler-options '-fPIC' -res-usage --maxrregcount 60 --use_fast_math -O3 -Xptxas -O3 -DTmax=256 -std=c++17 -c /hy-tmp/2024HMLIC/RWKV/classification/mmcls_custom/models/backbones/cuda/wkv_cuda.cu -o wkv_cuda.cuda.o
ptxas info : 0 bytes gmem
ptxas info : Compiling entry function 'Z15kernel_backwardIfEviiiPKT_S2_S2_S2_S2_PS0_S3_S3_S3' for 'sm_89'
ptxas info : Function properties for Z15kernel_backwardIfEviiiPKT_S2_S2_S2_S2_PS0_S3_S3_S3
96 bytes stack frame, 0 bytes spill stores, 0 bytes spill loads
ptxas info : Used 60 registers, 10240 bytes smem, 440 bytes cmem[0], 16 bytes cmem[2]
ptxas info : Compiling entry function 'Z14kernel_forwardIfEviiiPKT_S2_S2_S2_PS0' for 'sm_89'
ptxas info : Function properties for Z14kernel_forwardIfEviiiPKT_S2_S2_S2_PS0
0 bytes stack frame, 0 bytes spill stores, 0 bytes spill loads
ptxas info : Used 60 registers, 6144 bytes smem, 408 bytes cmem[0], 16 bytes cmem[2]
[2/3] c++ -MMD -MF wkv_op.o.d -DTORCH_EXTENSION_NAME=wkv -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE="gcc" -DPYBIND11_STDLIB="libstdcpp" -DPYBIND11_BUILD_ABI="cxxabi1011" -isystem /usr/local/miniconda3/envs/RWKV/lib/python3.10/site-packages/torch/include -isystem /usr/local/miniconda3/envs/RWKV/lib/python3.10/site-packages/torch/include/torch/csrc/api/include -isystem /usr/local/miniconda3/envs/RWKV/lib/python3.10/site-packages/torch/include/TH -isystem /usr/local/miniconda3/envs/RWKV/lib/python3.10/site-packages/torch/include/THC -isystem /usr/local/cuda/include -isystem /usr/local/miniconda3/envs/RWKV/include/python3.10 -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++17 -c /hy-tmp/2024HMLIC/RWKV/classification/mmcls_custom/models/backbones/cuda/wkv_op.cpp -o wkv_op.o
[3/3] c++ wkv_op.o wkv_cuda.cuda.o -shared -L/usr/local/miniconda3/envs/RWKV/lib/python3.10/site-packages/torch/lib -lc10 -lc10_cuda -ltorch_cpu -ltorch_cuda -ltorch -ltorch_python -L/usr/local/cuda/lib64 -lcudart -o wkv.so
Loading extension module wkv...
Using /root/.cache/torch_extensions/py310_cu121 as PyTorch extensions root...
Creating extension directory /root/.cache/torch_extensions/py310_cu121/wkv6...
Detected CUDA files, patching ldflags
Emitting ninja build file /root/.cache/torch_extensions/py310_cu121/wkv6/build.ninja...
/usr/local/miniconda3/envs/RWKV/lib/python3.10/site-packages/torch/utils/cpp_extension.py:1967: UserWarning: TORCH_CUDA_ARCH_LIST is not set, all archs for visible cards are included for compilation.
If this is not desired, please set os.environ['TORCH_CUDA_ARCH_LIST'].
warnings.warn(
Building extension module wkv6...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
[1/3] c++ -MMD -MF wkv6_op.o.d -DTORCH_EXTENSION_NAME=wkv6 -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE="gcc" -DPYBIND11_STDLIB="libstdcpp" -DPYBIND11_BUILD_ABI="cxxabi1011" -isystem /usr/local/miniconda3/envs/RWKV/lib/python3.10/site-packages/torch/include -isystem /usr/local/miniconda3/envs/RWKV/lib/python3.10/site-packages/torch/include/torch/csrc/api/include -isystem /usr/local/miniconda3/envs/RWKV/lib/python3.10/site-packages/torch/include/TH -isystem /usr/local/miniconda3/envs/RWKV/lib/python3.10/site-packages/torch/include/THC -isystem /usr/local/cuda/include -isystem /usr/local/miniconda3/envs/RWKV/include/python3.10 -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++17 -c /hy-tmp/2024HMLIC/RWKV/classification/mmcls_custom/models/backbones/cuda_v6/wkv6_op.cpp -o wkv6_op.o
[2/3] /usr/local/cuda/bin/nvcc --generate-dependencies-with-compile --dependency-output wkv6_cuda.cuda.o.d -DTORCH_EXTENSION_NAME=wkv6 -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE="gcc" -DPYBIND11_STDLIB="libstdcpp" -DPYBIND11_BUILD_ABI="cxxabi1011" -isystem /usr/local/miniconda3/envs/RWKV/lib/python3.10/site-packages/torch/include -isystem /usr/local/miniconda3/envs/RWKV/lib/python3.10/site-packages/torch/include/torch/csrc/api/include -isystem /usr/local/miniconda3/envs/RWKV/lib/python3.10/site-packages/torch/include/TH -isystem /usr/local/miniconda3/envs/RWKV/lib/python3.10/site-packages/torch/include/THC -isystem /usr/local/cuda/include -isystem /usr/local/miniconda3/envs/RWKV/include/python3.10 -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS -D__CUDA_NO_HALF_CONVERSIONS -D__CUDA_NO_BFLOAT16_CONVERSIONS -D__CUDA_NO_HALF2_OPERATORS --expt-relaxed-constexpr -gencode=arch=compute_89,code=compute_89 -gencode=arch=compute_89,code=sm_89 --compiler-options '-fPIC' -res-usage --use_fast_math -O3 -Xptxas -O3 --extra-device-vectorization -D_N=64 -D_T_=256 -std=c++17 -c /hy-tmp/2024HMLIC/RWKV/classification/mmcls_custom/models/backbones/cuda_v6/wkv6_cuda.cu -o wkv6_cuda.cuda.o
ptxas info : 1 bytes gmem
ptxas info : Compiling entry function 'Z19kernel_backward_333IfEviiiiPKT_S2_S2_PKfS2_S2_PS0' for 'sm_89'
ptxas info : Function properties for Z19kernel_backward_333IfEviiiiPKT_S2_S2_PKfS2_S2_PS0
1016 bytes stack frame, 0 bytes spill stores, 0 bytes spill loads
ptxas info : Used 96 registers, 512 bytes smem, 424 bytes cmem[0]
ptxas info : Compiling entry function 'Z19kernel_backward_222IfEviiiiPKT_S2_S2_PKfS2_S2_PS0' for 'sm_89'
ptxas info : Function properties for Z19kernel_backward_222IfEviiiiPKT_S2_S2_PKfS2_S2_PS0
1016 bytes stack frame, 0 bytes spill stores, 0 bytes spill loads
ptxas info : Used 96 registers, 512 bytes smem, 424 bytes cmem[0]
ptxas info : Compiling entry function 'Z19kernel_backward_111IfEviiiiPKT_S2_S2_PKfS2_S2_PS0_S5_S5_S5' for 'sm_89'
ptxas info : Function properties for Z19kernel_backward_111IfEviiiiPKT_S2_S2_PKfS2_S2_PS0_S5_S5_S5
0 bytes stack frame, 0 bytes spill stores, 0 bytes spill loads
ptxas info : Used 244 registers, 1536 bytes smem, 448 bytes cmem[0]
ptxas info : Compiling entry function 'Z14kernel_forwardIfEviiiiPKT_S2_S2_PKfS2_PS0' for 'sm_89'
ptxas info : Function properties for Z14kernel_forwardIfEviiiiPKT_S2_S2_PKfS2_PS0
0 bytes stack frame, 0 bytes spill stores, 0 bytes spill loads
ptxas info : Used 100 registers, 1024 bytes smem, 416 bytes cmem[0]
[3/3] c++ wkv6_op.o wkv6_cuda.cuda.o -shared -L/usr/local/miniconda3/envs/RWKV/lib/python3.10/site-packages/torch/lib -lc10 -lc10_cuda -ltorch_cpu -ltorch_cuda -ltorch -ltorch_python -L/usr/local/cuda/lib64 -lcudart -o wkv6.so
Loading extension module wkv6...
Using /root/.cache/torch_extensions/py310_cu121 as PyTorch extensions root...
The input conditions for extension module wkv6 have changed. Bumping to version 1 and re-building as wkv6_v1...
Detected CUDA files, patching ldflags
Emitting ninja build file /root/.cache/torch_extensions/py310_cu121/wkv6/build.ninja...
/usr/local/miniconda3/envs/RWKV/lib/python3.10/site-packages/torch/utils/cpp_extension.py:1967: UserWarning: TORCH_CUDA_ARCH_LIST is not set, all archs for visible cards are included for compilation.
If this is not desired, please set os.environ['TORCH_CUDA_ARCH_LIST'].
warnings.warn(
Building extension module wkv6_v1...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
[1/3] c++ -MMD -MF wkv6_op.o.d -DTORCH_EXTENSION_NAME=wkv6_v1 -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE="gcc" -DPYBIND11_STDLIB="libstdcpp" -DPYBIND11_BUILD_ABI="cxxabi1011" -isystem /usr/local/miniconda3/envs/RWKV/lib/python3.10/site-packages/torch/include -isystem /usr/local/miniconda3/envs/RWKV/lib/python3.10/site-packages/torch/include/torch/csrc/api/include -isystem /usr/local/miniconda3/envs/RWKV/lib/python3.10/site-packages/torch/include/TH -isystem /usr/local/miniconda3/envs/RWKV/lib/python3.10/site-packages/torch/include/THC -isystem /usr/local/cuda/include -isystem /usr/local/miniconda3/envs/RWKV/include/python3.10 -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++17 -c /hy-tmp/2024HMLIC/RWKV/classification/mmcls_custom/models/backbones/cuda_v6/wkv6_op.cpp -o wkv6_op.o
[2/3] /usr/local/cuda/bin/nvcc --generate-dependencies-with-compile --dependency-output wkv6_cuda.cuda.o.d -DTORCH_EXTENSION_NAME=wkv6_v1 -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE="gcc" -DPYBIND11_STDLIB="libstdcpp" -DPYBIND11_BUILD_ABI="cxxabi1011" -isystem /usr/local/miniconda3/envs/RWKV/lib/python3.10/site-packages/torch/include -isystem /usr/local/miniconda3/envs/RWKV/lib/python3.10/site-packages/torch/include/torch/csrc/api/include -isystem /usr/local/miniconda3/envs/RWKV/lib/python3.10/site-packages/torch/include/TH -isystem /usr/local/miniconda3/envs/RWKV/lib/python3.10/site-packages/torch/include/THC -isystem /usr/local/cuda/include -isystem /usr/local/miniconda3/envs/RWKV/include/python3.10 -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS -D__CUDA_NO_HALF_CONVERSIONS -D__CUDA_NO_BFLOAT16_CONVERSIONS -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_89,code=compute_89 -gencode=arch=compute_89,code=sm_89 --compiler-options '-fPIC' -res-usage --use_fast_math -O3 -Xptxas -O3 --extra-device-vectorization -D_N_=64 -D_T_=1024 -std=c++17 -c /hy-tmp/2024HMLIC/RWKV/classification/mmcls_custom/models/backbones/cuda_v6/wkv6_cuda.cu -o wkv6_cuda.cuda.o
ptxas info : 1 bytes gmem
ptxas info : Compiling entry function 'Z19kernel_backward_333IfEviiiiPKT_S2_S2_PKfS2_S2_PS0' for 'sm_89'
ptxas info : Function properties for Z19kernel_backward_333IfEviiiiPKT_S2_S2_PKfS2_S2_PS0
4088 bytes stack frame, 0 bytes spill stores, 0 bytes spill loads
ptxas info : Used 96 registers, 512 bytes smem, 424 bytes cmem[0]
ptxas info : Compiling entry function 'Z19kernel_backward_222IfEviiiiPKT_S2_S2_PKfS2_S2_PS0' for 'sm_89'
ptxas info : Function properties for Z19kernel_backward_222IfEviiiiPKT_S2_S2_PKfS2_S2_PS0
4088 bytes stack frame, 0 bytes spill stores, 0 bytes spill loads
ptxas info : Used 96 registers, 512 bytes smem, 424 bytes cmem[0]
ptxas info : Compiling entry function 'Z19kernel_backward_111IfEviiiiPKT_S2_S2_PKfS2_S2_PS0_S5_S5_S5' for 'sm_89'
ptxas info : Function properties for Z19kernel_backward_111IfEviiiiPKT_S2_S2_PKfS2_S2_PS0_S5_S5_S5
0 bytes stack frame, 0 bytes spill stores, 0 bytes spill loads
ptxas info : Used 244 registers, 1536 bytes smem, 448 bytes cmem[0]
ptxas info : Compiling entry function 'Z14kernel_forwardIfEviiiiPKT_S2_S2_PKfS2_PS0' for 'sm_89'
ptxas info : Function properties for Z14kernel_forwardIfEviiiiPKT_S2_S2_PKfS2_PS0
0 bytes stack frame, 0 bytes spill stores, 0 bytes spill loads
ptxas info : Used 100 registers, 1024 bytes smem, 416 bytes cmem[0]
[3/3] c++ wkv6_op.o wkv6_cuda.cuda.o -shared -L/usr/local/miniconda3/envs/RWKV/lib/python3.10/site-packages/torch/lib -lc10 -lc10_cuda -ltorch_cpu -ltorch_cuda -ltorch -ltorch_python -L/usr/local/cuda/lib64 -lcudart -o wkv6_v1.so
Loading extension module wkv6_v1...
Aborted (core dumped)