xformers installation for RTX 4090 on linux

[cyber-meow]

I finally get this work but it took me so long time so I decide it to share it here in case others also need it.
Better solutions may exist and I would like to learn about it.

The instruction in the repository is only for windows and I cannot get any existing linux wheel work, so I compile xformers on my own in the following way

1. Get cuda 11.8
Older version does not support RTX 4090. Cuda 12.0 throws version mismatch error.

2. Modify torch/utils/cpp_extension.py
Add ('Lovelace', '8.9+PTX') in named_arches and change supported_arches to
supported_arches = ['3.5', '3.7', '5.0', '5.2', '5.3', '6.0', '6.1', '6.2', '7.0', '7.2', '7.5', '8.0', '8.6', '8.9']

3. Compile xformers 0.0.15

git clone https://github.com/facebookresearch/xformers.git
cd xformers
git checkout d647bb5
git submodule update --init --recursive
pip install -r requirements.txt
pip install ninja     # I guess this is optional
pip install -e .

That's it. For Lora training I can get 150 images x 10 epochs done in 1.5 minutes at batch size 10 (it uses around 19G of vram and the speed is ~1.5it/s).

Edit
The cudnn trick works for training as well. I didn't test it on kohya trainer but it accelerates significantly my training with Everydream2. I just point LD_LIBRARY_PATH to the folder of new cudnn files and delete the corresponding ones from the original environment.

---

Other solutions that I have tried:

1. Install via pip install xformers==0.0.16rc425 as in AUTOMATIC1111/stable-diffusion-webui: it runs but loss goes to nan
2. Compile xformers without checkout to previous commit: this installs xformers 0.0.16 and again we get nan loss
3. pip install -U -I --no-deps https://github.com/C43H66N12O12S2/stable-diffusion-webui/releases/download/linux/xformers-0.0.14.dev0-cp310-cp310-linux_x86_64.whl: this seems to be the linux equivalence of the suggested wheel, but I get
   RuntimeError: No such operator xformers::efficient_attention_forward_cutlass - did you forget to build xformers with python setup.py develop?
4. Compile with other version of cuda: no it doesn't work as explained above
5. conda install -c xformers/label/dev xformers: error xFormers wasn't built with CUDA support

[runner22k]

i faced this error too xFormers wasn't built with CUDA support

[St33lMouse]

Followed your instructions. I have a 3090, just skipped step 2. Seems to have worked. I ran the instructions from within the sd-scripts folder.

I was able to follow the windows instructions well enough to install kohya, got your xformers to install, then there's this:

cp .\bitsandbytes_windows*.dll .\venv\Lib\site-packages\bitsandbytes
cp .\bitsandbytes_windows\cextension.py .\venv\Lib\site-packages\bitsandbytes\cextension.py
cp .\bitsandbytes_windows\main.py .\venv\Lib\site-packages\bitsandbytes\cuda_setup\main.py

accelerate config

hope that last bit isn't terribly important...

[cyber-meow]

I didn't run the three commands for bitsandbytes and I think it is only for windows.

[runner22k]

@cyber-meow How did you install kohya on Linux? Did you face any other problems, other than xformers?

[cyber-meow]

No the only problem that I encountered was about xformers. The remaining things work smoothly.
(That is, just run the following commands as explained in the readme

pip install torch==1.12.1+cu116 torchvision==0.13.1+cu116 --extra-index-url https://download.pytorch.org/whl/cu116
pip install --upgrade -r requirements.txt

[runner22k]

Thanks @cyber-meow I should shift my process back to linux. But few good things are happening on windows based projects. I need to install Both Linux and windows now. :p

Did you check this new video from Kohya_ss GUI's LR-Free branch? He is using new method to train LoRA by automatically calculating and adjusting Learning rate and learning steps. I don't know how to use it on linux. But one of the comments says that it speeds up training.

[shwangtangjun]

Thanks! I have tried so many other methods, and yours is the only one that works.

BTW, I recommend you to use pip install ninja although it is optional.
ninja can significantly accelerate the process of installing xformers(which takes hours...). See facebookresearch/xformers#481

[runner22k]

I can confirm that, ninja took less than an hour to compile xformers on Ubuntu 20.04.