Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Update docs #765

Open
wants to merge 15 commits into
base: main
Choose a base branch
from
Open

Update docs #765

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
cea32ba
update sp docs
HIT-cwh Jun 12, 2024
941f4c6
add triton kernel docs
HIT-cwh Jun 12, 2024
b8eb184
fix typos
HIT-cwh Jun 12, 2024
313483d
add train moe
HIT-cwh Jun 12, 2024
72c5623
fix docs
HIT-cwh Jun 12, 2024
1102746
fix docs
HIT-cwh Jun 12, 2024
426a081
add warning
HIT-cwh Jun 12, 2024
418e5e3
Update docs/zh_cn/acceleration/triton_kernels.rst
HIT-cwh Jun 12, 2024
57bff5e
Update docs/zh_cn/acceleration/triton_kernels.rst
HIT-cwh Jun 12, 2024
2342f97
Update docs/zh_cn/acceleration/triton_kernels.rst
HIT-cwh Jun 12, 2024
36b6256
Update docs/zh_cn/acceleration/train_moe.rst
HIT-cwh Jun 12, 2024
ca180c2
Update docs/zh_cn/acceleration/train_moe.rst
HIT-cwh Jun 12, 2024
8b2fc09
Update docs/zh_cn/acceleration/train_extreme_long_sequence.rst
HIT-cwh Jun 12, 2024
7918537
Update docs/zh_cn/acceleration/train_moe.rst
HIT-cwh Jun 12, 2024
29bb5b2
fix docs
HIT-cwh Jun 13, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions benchmark/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
# 速度基准

请参考 [速度基准文档](https://github.com/InternLM/xtuner/tree/main/docs/zh_cn/acceleration/benchmark.rst)
0 ...b/llama2_70b_full_alpaca_enzh_128k_sp8.py → ...b/llama2_70b_full_alpaca_enzh_128k_sp8.py
View file
Open in desktop
File renamed without changes.
0 .../llama2_70b_full_alpaca_enzh_256k_sp16.py → .../llama2_70b_full_alpaca_enzh_256k_sp16.py
View file
Open in desktop
File renamed without changes.
0 ...0b/llama2_70b_full_alpaca_enzh_32k_sp4.py → ...0b/llama2_70b_full_alpaca_enzh_32k_sp4.py
View file
Open in desktop
File renamed without changes.
0 ...70b/llama2_70b_full_alpaca_enzh_8k_sp1.py → ...70b/llama2_70b_full_alpaca_enzh_8k_sp1.py
View file
Open in desktop
File renamed without changes.
0 ...7b/llama2_7b_full_alpaca_enzh_128k_sp8.py → ...7b/llama2_7b_full_alpaca_enzh_128k_sp8.py
View file
Open in desktop
File renamed without changes.
0 ..._7b/llama2_7b_full_alpaca_enzh_1M_sp16.py → ..._7b/llama2_7b_full_alpaca_enzh_1M_sp16.py
View file
Open in desktop
File renamed without changes.
0 ...7b/llama2_7b_full_alpaca_enzh_256k_sp8.py → ...7b/llama2_7b_full_alpaca_enzh_256k_sp8.py
View file
Open in desktop
File renamed without changes.
0 ..._7b/llama2_7b_full_alpaca_enzh_32k_sp1.py → ..._7b/llama2_7b_full_alpaca_enzh_32k_sp1.py
View file
Open in desktop
File renamed without changes.
0 ...2_7b/llama2_7b_full_alpaca_enzh_8k_sp1.py → ...2_7b/llama2_7b_full_alpaca_enzh_8k_sp1.py
View file
Open in desktop
File renamed without changes.
0 .../yi_34b_200k_full_alpaca_enzh_128k_sp8.py → .../yi_34b_200k_full_alpaca_enzh_128k_sp8.py
View file
Open in desktop
File renamed without changes.
0 .../yi_34b_200k_full_alpaca_enzh_256k_sp8.py → .../yi_34b_200k_full_alpaca_enzh_256k_sp8.py
View file
Open in desktop
File renamed without changes.
0 ...b/yi_34b_200k_full_alpaca_enzh_32k_sp2.py → ...b/yi_34b_200k_full_alpaca_enzh_32k_sp2.py
View file
Open in desktop
File renamed without changes.
0 ...4b/yi_34b_200k_full_alpaca_enzh_8k_sp1.py → ...4b/yi_34b_200k_full_alpaca_enzh_8k_sp1.py
View file
Open in desktop
File renamed without changes.
28 changes: 14 additions & 14 deletions docs/zh_cn/acceleration/benchmark.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -59,31 +59,31 @@
- 8k
- 3028.3
- 185.3
- `llama2_70b_full_alpaca_enzh_8k_sp1.py <https://github.com/InternLM/xtuner/tree/main/xtuner/configs/llama_speed_benchmark/llama2_7b/llama2_7b_full_alpaca_enzh_8k_sp1.py>`_
- `llama2_70b_full_alpaca_enzh_8k_sp1.py <https://github.com/InternLM/xtuner/tree/main/benchmark/llama2_7b/llama2_7b_full_alpaca_enzh_8k_sp1.py>`_
* - Llama2-7B
- 8
- 32k
- 2234.2
- 193.0
- `llama2_7b_full_alpaca_enzh_32k_sp1.py <https://github.com/InternLM/xtuner/tree/main/xtuner/configs/llama_speed_benchmark/llama2_7b/llama2_7b_full_alpaca_enzh_32k_sp1.py>`_
- `llama2_7b_full_alpaca_enzh_32k_sp1.py <https://github.com/InternLM/xtuner/tree/main/benchmark/llama2_7b/llama2_7b_full_alpaca_enzh_32k_sp1.py>`_
* - Llama2-7B
- 8
- 128k
- 948.6
- 180.3
- `llama2_7b_full_alpaca_enzh_128k_sp8.py <https://github.com/InternLM/xtuner/tree/main/xtuner/configs/llama_speed_benchmark/llama2_7b/llama2_7b_full_alpaca_enzh_128k_sp8.py>`_
- `llama2_7b_full_alpaca_enzh_128k_sp8.py <https://github.com/InternLM/xtuner/tree/main/benchmark/llama2_7b/llama2_7b_full_alpaca_enzh_128k_sp8.py>`_
* - Llama2-7B
- 8
- 256k
- 540.1
- 176.9
- `llama2_7b_full_alpaca_enzh_256k_sp8.py <https://github.com/InternLM/xtuner/tree/main/xtuner/configs/llama_speed_benchmark/llama2_7b/llama2_7b_full_alpaca_enzh_256k_sp8.py>`_
- `llama2_7b_full_alpaca_enzh_256k_sp8.py <https://github.com/InternLM/xtuner/tree/main/benchmark/llama2_7b/llama2_7b_full_alpaca_enzh_256k_sp8.py>`_
* - Llama2-7B
- 8
- 1M
- 133.6
- 153.9
- `llama2_7b_full_alpaca_enzh_1M_sp16.py <https://github.com/InternLM/xtuner/tree/main/xtuner/configs/llama_speed_benchmark/llama2_7b/llama2_7b_full_alpaca_enzh_1M_sp16.py>`_
- `llama2_7b_full_alpaca_enzh_1M_sp16.py <https://github.com/InternLM/xtuner/tree/main/benchmark/llama2_7b/llama2_7b_full_alpaca_enzh_1M_sp16.py>`_

.. list-table::
:widths: 30 15 20 20 20 50
Expand All @@ -100,25 +100,25 @@
- 8k
- 485.1
- 165.6
- `yi_34b_200k_full_alpaca_enzh_8k_sp1.py <https://github.com/InternLM/xtuner/tree/main/xtuner/configs/llama_speed_benchmark/yi_34b/yi_34b_200k_full_alpaca_enzh_8k_sp1.py>`_
- `yi_34b_200k_full_alpaca_enzh_8k_sp1.py <https://github.com/InternLM/xtuner/tree/main/benchmark/yi_34b/yi_34b_200k_full_alpaca_enzh_8k_sp1.py>`_
* - Yi-34B-200K
- 32
- 32k
- 491.5
- 209.1
- `yi_34b_200k_full_alpaca_enzh_32k_sp2.py <https://github.com/InternLM/xtuner/tree/main/xtuner/configs/llama_speed_benchmark/yi_34b/yi_34b_200k_full_alpaca_enzh_32k_sp2.py>`_
- `yi_34b_200k_full_alpaca_enzh_32k_sp2.py <https://github.com/InternLM/xtuner/tree/main/benchmark/yi_34b/yi_34b_200k_full_alpaca_enzh_32k_sp2.py>`_
* - Yi-34B-200K
- 32
- 128k
- 251.1
- 191.8
- `yi_34b_200k_full_alpaca_enzh_128k_sp8.py <https://github.com/InternLM/xtuner/tree/main/xtuner/configs/llama_speed_benchmark/yi_34b/yi_34b_200k_full_alpaca_enzh_128k_sp8.py>`_
- `yi_34b_200k_full_alpaca_enzh_128k_sp8.py <https://github.com/InternLM/xtuner/tree/main/benchmark/yi_34b/yi_34b_200k_full_alpaca_enzh_128k_sp8.py>`_
* - Yi-34B-200K
- 32
- 256k
- 119.7
- 145.3
- `yi_34b_200k_full_alpaca_enzh_256k_sp8.py <https://github.com/InternLM/xtuner/tree/main/xtuner/configs/llama_speed_benchmark/yi_34b/yi_34b_200k_full_alpaca_enzh_256k_sp8.py>`_
- `yi_34b_200k_full_alpaca_enzh_256k_sp8.py <https://github.com/InternLM/xtuner/tree/main/benchmark/yi_34b/yi_34b_200k_full_alpaca_enzh_256k_sp8.py>`_

.. list-table::
:widths: 30 15 20 20 20 50
Expand All @@ -135,31 +135,31 @@
- 8k
- 216.8
- 144.7
- `llama2_70b_full_alpaca_enzh_8k_sp1.py <https://github.com/InternLM/xtuner/tree/main/xtuner/configs/llama_speed_benchmark/llama2_70b/llama2_70b_full_alpaca_enzh_8k_sp1.py>`_
- `llama2_70b_full_alpaca_enzh_8k_sp1.py <https://github.com/InternLM/xtuner/tree/main/benchmark/llama2_70b/llama2_70b_full_alpaca_enzh_8k_sp1.py>`_
* - Llama2-70B
- 32
- 32k
- 300.9
- 239.6
- `llama2_70b_full_alpaca_enzh_32k_sp4.py <https://github.com/InternLM/xtuner/tree/main/xtuner/configs/llama_speed_benchmark/llama2_70b/llama2_70b_full_alpaca_enzh_32k_sp4.py>`_
- `llama2_70b_full_alpaca_enzh_32k_sp4.py <https://github.com/InternLM/xtuner/tree/main/benchmark/llama2_70b/llama2_70b_full_alpaca_enzh_32k_sp4.py>`_
* - Llama2-70B
- 32
- 128k
- 144.7
- 189.7
- `llama2_70b_full_alpaca_enzh_128k_sp8.py <https://github.com/InternLM/xtuner/tree/main/xtuner/configs/llama_speed_benchmark/llama2_70b/llama2_70b_full_alpaca_enzh_128k_sp8.py>`_
- `llama2_70b_full_alpaca_enzh_128k_sp8.py <https://github.com/InternLM/xtuner/tree/main/benchmark/llama2_70b/llama2_70b_full_alpaca_enzh_128k_sp8.py>`_
* - Llama2-70B
- 32
- 256k
- 63.8
- 127.6
- `llama2_70b_full_alpaca_enzh_256k_sp16.py <https://github.com/InternLM/xtuner/tree/main/xtuner/configs/llama_speed_benchmark/llama2_70b/llama2_70b_full_alpaca_enzh_256k_sp16.py>`_
- `llama2_70b_full_alpaca_enzh_256k_sp16.py <https://github.com/InternLM/xtuner/tree/main/benchmark/llama2_70b/llama2_70b_full_alpaca_enzh_256k_sp16.py>`_
* - Llama2-70B
- 64
- 1M
- 21.8
- 133.5
- `llama2_70b_full_alpaca_enzh_1M_sp64.py <https://github.com/InternLM/xtuner/tree/main/xtuner/configs/llama_speed_benchmark/llama2_70b/llama2_70b_full_alpaca_enzh_1M_sp64.py>`_
- `llama2_70b_full_alpaca_enzh_1M_sp64.py <https://github.com/InternLM/xtuner/tree/main/benchmark/llama2_70b/llama2_70b_full_alpaca_enzh_1M_sp64.py>`_

.. note::
所有实验都会将 Alpaca 数据集拼接为最大长度。由于 Alpaca 数据集所含
Expand Down
2 changes: 1 addition & 1 deletion docs/zh_cn/acceleration/deepspeed.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -100,4 +100,4 @@ XTuner 内置了五种 DeepSpeed ZeRO 配置：
DeepSpeed Config 中的 ``gradient_clipping`` 会被 XTuner config 中的 ``optim_wrapper.clip_grad.max_norm`` 设置覆盖

.. warning::
XTuner 会根据所使用的 GPU 架构自动选择 ``fp16`` 或 ``bf16`` 训练，不受
XTuner 会根据所使用的 GPU 架构自动选择 ``fp16`` 或 ``bf16`` 训练，不受 DeepSpeed 配置文件的控制
Loading
Loading