You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Below is my log of running the code. The code stops at the last line so long time, there's no error. I don't know why:
...
Initializing [clip] visual encoder from ~/.cache/clip/ViT-L-14.pt [cuda]...
Visual encoder initialized.
Initializing language decoder from /home/project/Mllm/LAMM/model_zoo/vicuna_ckpt/vicuna-13b-delta-v0 ...
Visual encoder initialized.
Initializing language decoder from /home/project/Mllm/LAMM/model_zoo/vicuna_ckpt/vicuna-13b-delta-v0 ...
Visual encoder initialized.
Initializing language decoder from /home/project/Mllm/LAMM/model_zoo/vicuna_ckpt/vicuna-13b-delta-v0 ...
Visual encoder initialized.
Initializing language decoder from /home/project/Mllm/LAMM/model_zoo/vicuna_ckpt/vicuna-13b-delta-v0 ...
Loading checkpoint shards: 100%|██████████| 3/3 [00:17<00:00, 5.86s/it]
Loading checkpoint shards: 100%|██████████| 3/3 [00:19<00:00, 6.53s/it]
Loading checkpoint shards: 100%|██████████| 3/3 [00:20<00:00, 6.96s/it]
Loading checkpoint shards: 100%|██████████| 3/3 [00:20<00:00, 6.71s/it]
trainable params: 52428800 || all params: 13068303360 || trainable%: 0.4011905643426998
You are using the legacy behaviour of the <class 'transformers.models.llama.tokenization_llama.LlamaTokenizer'>. This means that tokens that come after special tokens will not be properly handled. We recommend you to read the related pull request available at huggingface/transformers#24565
Language decoder initialized.
LLaMa projection layer initialized.
DeepSpeedAgent
trainable params: 52428800 || all params: 13068303360 || trainable%: 0.4011905643426998
You are using the legacy behaviour of the <class 'transformers.models.llama.tokenization_llama.LlamaTokenizer'>. This means that tokens that come after special tokens will not be properly handled. We recommend you to read the related pull request available at huggingface/transformers#24565
Language decoder initialized.
LLaMa projection layer initialized.
DeepSpeedAgent
trainable params: 52428800 || all params: 13068303360 || trainable%: 0.4011905643426998
You are using the legacy behaviour of the <class 'transformers.models.llama.tokenization_llama.LlamaTokenizer'>. This means that tokens that come after special tokens will not be properly handled. We recommend you to read the related pull request available at huggingface/transformers#24565
Language decoder initialized.
LLaMa projection layer initialized.
DeepSpeedAgent
trainable params: 52428800 || all params: 13068303360 || trainable%: 0.4011905643426998
You are using the legacy behaviour of the <class 'transformers.models.llama.tokenization_llama.LlamaTokenizer'>. This means that tokens that come after special tokens will not be properly handled. We recommend you to read the related pull request available at huggingface/transformers#24565
Language decoder initialized.
LLaMa projection layer initialized.
DeepSpeedAgent
[2024-01-24 16:43:41,259] [INFO] [logging.py:96:log_dist] [Rank 0] DeepSpeed info: version=0.9.3, git-hash=4e80e29, git-branch=HEAD
[2024-01-24 16:43:41,259] [INFO] [comm.py:619:init_distributed] Distributed backend already initialized
[2024-01-24 16:44:17,242] [INFO] [logging.py:96:log_dist] [Rank 0] DeepSpeed Flops Profiler Enabled: False
Using /home/.cache/torch_extensions/py310_cu118 as PyTorch extensions root...
Using /home/.cache/torch_extensions/py310_cu118 as PyTorch extensions root...
Using /home/.cache/torch_extensions/py310_cu118 as PyTorch extensions root...
Using /home/.cache/torch_extensions/py310_cu118 as PyTorch extensions root...
Detected CUDA files, patching ldflags
Emitting ninja build file /home/.cache/torch_extensions/py310_cu118/cpu_adam/build.ninja...
Building extension module cpu_adam...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
[1/3] /usr/bin/nvcc -DTORCH_EXTENSION_NAME=cpu_adam -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE="gcc" -DPYBIND11_STDLIB="libstdcpp" -DPYBIND11_BUILD_ABI="cxxabi1011" -I/data2/miniconda3/envs/lamm3/lib/python3.10/site-packages/deepspeed/ops/csrc/includes -I/usr/include -isystem /data2/miniconda3/envs/lamm3/lib/python3.10/site-packages/torch/include -isystem /data2/miniconda3/envs/lamm3/lib/python3.10/site-packages/torch/include/torch/csrc/api/include -isystem /data2/miniconda3/envs/lamm3/lib/python3.10/site-packages/torch/include/TH -isystem /data2/miniconda3/envs/lamm3/lib/python3.10/site-packages/torch/include/THC -isystem /data2/miniconda3/envs/lamm3/include/python3.10 -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS -D__CUDA_NO_HALF_CONVERSIONS_ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_90,code=compute_90 -gencode=arch=compute_90,code=sm_90 --compiler-options '-fPIC' -O3 --use_fast_math -std=c++17 -U__CUDA_NO_HALF_OPERATORS__ -U__CUDA_NO_HALF_CONVERSIONS__ -U__CUDA_NO_HALF2_OPERATORS__ -gencode=arch=compute_90,code=sm_90 -gencode=arch=compute_90,code=compute_90 -DBF16_AVAILABLE -c /data2/miniconda3/envs/lamm3/lib/python3.10/site-packages/deepspeed/ops/csrc/common/custom_cuda_kernel.cu -o custom_cuda_kernel.cuda.o
[2/3] c++ -MMD -MF cpu_adam.o.d -DTORCH_EXTENSION_NAME=cpu_adam -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE="gcc" -DPYBIND11_STDLIB="libstdcpp" -DPYBIND11_BUILD_ABI="cxxabi1011" -I/data2/miniconda3/envs/lamm3/lib/python3.10/site-packages/deepspeed/ops/csrc/includes -I/usr/include -isystem /data2/miniconda3/envs/lamm3/lib/python3.10/site-packages/torch/include -isystem /data2/miniconda3/envs/lamm3/lib/python3.10/site-packages/torch/include/torch/csrc/api/include -isystem /data2/miniconda3/envs/lamm3/lib/python3.10/site-packages/torch/include/TH -isystem /data2/miniconda3/envs/lamm3/lib/python3.10/site-packages/torch/include/THC -isystem /data2/miniconda3/envs/lamm3/include/python3.10 -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++17 -O3 -std=c++17 -g -Wno-reorder -L/usr/lib64 -lcudart -lcublas -g -march=native -fopenmp -D__AVX512 -D__ENABLE_CUDA_ -DBF16_AVAILABLE -c /data2/miniconda3/envs/lamm3/lib/python3.10/site-packages/deepspeed/ops/csrc/adam/cpu_adam.cpp -o cpu_adam.o
[3/3] c++ cpu_adam.o custom_cuda_kernel.cuda.o -shared -lcurand -L/data2/miniconda3/envs/lamm3/lib/python3.10/site-packages/torch/lib -lc10 -lc10_cuda -ltorch_cpu -ltorch_cuda -ltorch -ltorch_python -L/usr/lib64 -lcudart -o cpu_adam.so
Loading extension module cpu_adam...
Time to load cpu_adam op: 23.828491926193237 seconds
Loading extension module cpu_adam...
Time to load cpu_adam op: 23.78138303756714 seconds
Loading extension module cpu_adam...
Time to load cpu_adam op: 23.877806186676025 seconds
Loading extension module cpu_adam...
Time to load cpu_adam op: 23.85369896888733 seconds
The text was updated successfully, but these errors were encountered:
Below is my log of running the code. The code stops at the last line so long time, there's no error. I don't know why:
...
Initializing [clip] visual encoder from ~/.cache/clip/ViT-L-14.pt [cuda]...
Visual encoder initialized.
Initializing language decoder from /home/project/Mllm/LAMM/model_zoo/vicuna_ckpt/vicuna-13b-delta-v0 ...
Visual encoder initialized.
Initializing language decoder from /home/project/Mllm/LAMM/model_zoo/vicuna_ckpt/vicuna-13b-delta-v0 ...
Visual encoder initialized.
Initializing language decoder from /home/project/Mllm/LAMM/model_zoo/vicuna_ckpt/vicuna-13b-delta-v0 ...
Visual encoder initialized.
Initializing language decoder from /home/project/Mllm/LAMM/model_zoo/vicuna_ckpt/vicuna-13b-delta-v0 ...
Loading checkpoint shards: 100%|██████████| 3/3 [00:17<00:00, 5.86s/it]
Loading checkpoint shards: 100%|██████████| 3/3 [00:19<00:00, 6.53s/it]
Loading checkpoint shards: 100%|██████████| 3/3 [00:20<00:00, 6.96s/it]
Loading checkpoint shards: 100%|██████████| 3/3 [00:20<00:00, 6.71s/it]
trainable params: 52428800 || all params: 13068303360 || trainable%: 0.4011905643426998
You are using the legacy behaviour of the <class 'transformers.models.llama.tokenization_llama.LlamaTokenizer'>. This means that tokens that come after special tokens will not be properly handled. We recommend you to read the related pull request available at huggingface/transformers#24565
Language decoder initialized.
LLaMa projection layer initialized.
DeepSpeedAgent
trainable params: 52428800 || all params: 13068303360 || trainable%: 0.4011905643426998
You are using the legacy behaviour of the <class 'transformers.models.llama.tokenization_llama.LlamaTokenizer'>. This means that tokens that come after special tokens will not be properly handled. We recommend you to read the related pull request available at huggingface/transformers#24565
Language decoder initialized.
LLaMa projection layer initialized.
DeepSpeedAgent
trainable params: 52428800 || all params: 13068303360 || trainable%: 0.4011905643426998
You are using the legacy behaviour of the <class 'transformers.models.llama.tokenization_llama.LlamaTokenizer'>. This means that tokens that come after special tokens will not be properly handled. We recommend you to read the related pull request available at huggingface/transformers#24565
Language decoder initialized.
LLaMa projection layer initialized.
DeepSpeedAgent
trainable params: 52428800 || all params: 13068303360 || trainable%: 0.4011905643426998
You are using the legacy behaviour of the <class 'transformers.models.llama.tokenization_llama.LlamaTokenizer'>. This means that tokens that come after special tokens will not be properly handled. We recommend you to read the related pull request available at huggingface/transformers#24565
Language decoder initialized.
LLaMa projection layer initialized.
DeepSpeedAgent
[2024-01-24 16:43:41,259] [INFO] [logging.py:96:log_dist] [Rank 0] DeepSpeed info: version=0.9.3, git-hash=4e80e29, git-branch=HEAD
[2024-01-24 16:43:41,259] [INFO] [comm.py:619:init_distributed] Distributed backend already initialized
[2024-01-24 16:44:17,242] [INFO] [logging.py:96:log_dist] [Rank 0] DeepSpeed Flops Profiler Enabled: False
Using /home/.cache/torch_extensions/py310_cu118 as PyTorch extensions root...
Using /home/.cache/torch_extensions/py310_cu118 as PyTorch extensions root...
Using /home/.cache/torch_extensions/py310_cu118 as PyTorch extensions root...
Using /home/.cache/torch_extensions/py310_cu118 as PyTorch extensions root...
Detected CUDA files, patching ldflags
Emitting ninja build file /home/.cache/torch_extensions/py310_cu118/cpu_adam/build.ninja...
Building extension module cpu_adam...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
[1/3] /usr/bin/nvcc -DTORCH_EXTENSION_NAME=cpu_adam -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE="gcc" -DPYBIND11_STDLIB="libstdcpp" -DPYBIND11_BUILD_ABI="cxxabi1011" -I/data2/miniconda3/envs/lamm3/lib/python3.10/site-packages/deepspeed/ops/csrc/includes -I/usr/include -isystem /data2/miniconda3/envs/lamm3/lib/python3.10/site-packages/torch/include -isystem /data2/miniconda3/envs/lamm3/lib/python3.10/site-packages/torch/include/torch/csrc/api/include -isystem /data2/miniconda3/envs/lamm3/lib/python3.10/site-packages/torch/include/TH -isystem /data2/miniconda3/envs/lamm3/lib/python3.10/site-packages/torch/include/THC -isystem /data2/miniconda3/envs/lamm3/include/python3.10 -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS -D__CUDA_NO_HALF_CONVERSIONS_ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_90,code=compute_90 -gencode=arch=compute_90,code=sm_90 --compiler-options '-fPIC' -O3 --use_fast_math -std=c++17 -U__CUDA_NO_HALF_OPERATORS__ -U__CUDA_NO_HALF_CONVERSIONS__ -U__CUDA_NO_HALF2_OPERATORS__ -gencode=arch=compute_90,code=sm_90 -gencode=arch=compute_90,code=compute_90 -DBF16_AVAILABLE -c /data2/miniconda3/envs/lamm3/lib/python3.10/site-packages/deepspeed/ops/csrc/common/custom_cuda_kernel.cu -o custom_cuda_kernel.cuda.o
[2/3] c++ -MMD -MF cpu_adam.o.d -DTORCH_EXTENSION_NAME=cpu_adam -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE="gcc" -DPYBIND11_STDLIB="libstdcpp" -DPYBIND11_BUILD_ABI="cxxabi1011" -I/data2/miniconda3/envs/lamm3/lib/python3.10/site-packages/deepspeed/ops/csrc/includes -I/usr/include -isystem /data2/miniconda3/envs/lamm3/lib/python3.10/site-packages/torch/include -isystem /data2/miniconda3/envs/lamm3/lib/python3.10/site-packages/torch/include/torch/csrc/api/include -isystem /data2/miniconda3/envs/lamm3/lib/python3.10/site-packages/torch/include/TH -isystem /data2/miniconda3/envs/lamm3/lib/python3.10/site-packages/torch/include/THC -isystem /data2/miniconda3/envs/lamm3/include/python3.10 -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++17 -O3 -std=c++17 -g -Wno-reorder -L/usr/lib64 -lcudart -lcublas -g -march=native -fopenmp -D__AVX512 -D__ENABLE_CUDA_ -DBF16_AVAILABLE -c /data2/miniconda3/envs/lamm3/lib/python3.10/site-packages/deepspeed/ops/csrc/adam/cpu_adam.cpp -o cpu_adam.o
[3/3] c++ cpu_adam.o custom_cuda_kernel.cuda.o -shared -lcurand -L/data2/miniconda3/envs/lamm3/lib/python3.10/site-packages/torch/lib -lc10 -lc10_cuda -ltorch_cpu -ltorch_cuda -ltorch -ltorch_python -L/usr/lib64 -lcudart -o cpu_adam.so
Loading extension module cpu_adam...
Time to load cpu_adam op: 23.828491926193237 seconds
Loading extension module cpu_adam...
Time to load cpu_adam op: 23.78138303756714 seconds
Loading extension module cpu_adam...
Time to load cpu_adam op: 23.877806186676025 seconds
Loading extension module cpu_adam...
Time to load cpu_adam op: 23.85369896888733 seconds
The text was updated successfully, but these errors were encountered: