Skip to content

Commit

Permalink
Merge branch 'main' into phi-3-checkpoint
Browse files Browse the repository at this point in the history
  • Loading branch information
@rasbt
rasbt authored May 3, 2024
2 parents 29760ab + f334378 commit 6c4cd25
Show file tree
Hide file tree
Showing 60 changed files with 824 additions and 203 deletions.
74 changes: 39 additions & 35 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -33,43 +33,45 @@ Uses the latest state-of-the-art techniques:
<img src="https://pl-bolts-doc-images.s3.us-east-2.amazonaws.com/GithubLitGPTDAG2.png" alt="LitGPT steps" width="auto"/>
&nbsp;

# Finetune, pretrain and deploy LLMs Lightning fast ⚡⚡
# Finetune, pretrain and deploy LLMs Lightning fast ⚡⚡
LitGPT is a command-line tool designed to easily [finetune](#finetune-an-llm), [pretrain](#pretrain-an-llm), [evaluate](#use-an-llm), and [deploy](#deploy-an-llm) [20+ LLMs](#choose-from-20-llms) **on your own data**. It features highly-optimized [training recipes](#training-recipes) for the world's most powerful open-source large language models (LLMs).

We reimplemented all model architectures and training recipes from scratch for 4 reasons:
We reimplemented all model architectures and training recipes from scratch for 4 reasons:

1. Remove all abstraction layers and have single file implementations.
2. Guarantee Apache 2.0 compliance to enable enterprise use without limits.
3. Optimized each model's architectural detail to maximize performance, reduce costs, and speed up training.
4. Highly-optimized [recipe configs](#training-recipes) we have tested at enterprise scale.
1. Remove all abstraction layers and have single file implementations.
2. Guarantee Apache 2.0 compliance to enable enterprise use without limits.
3. Optimized each model's architectural detail to maximize performance, reduce costs, and speed up training.
4. Highly-optimized [recipe configs](#training-recipes) we have tested at enterprise scale.

---

&nbsp;

# Choose from 20+ LLMs
LitGPT has 🤯 **custom, from-scratch implementations** of [20+ LLMs](tutorials/download_model_weights.md) without layers of abstraction:
LitGPT has 🤯 **custom, from-scratch implementations** of [20+ LLMs](tutorials/download_model_weights.md) without layers of abstraction:

| Model | Model size | Author | Reference |
|----|----|----|----|
| Llama 3 | 8B, 70B | Meta AI | [Meta AI 2024](https://github.com/meta-llama/llama3) |
| Llama 2 | 7B, 13B, 70B | Meta AI | [Touvron et al. 2023](https://arxiv.org/abs/2307.09288) |
| Llama 3 | 8B, 70B | Meta AI | [Meta AI 2024](https://github.com/meta-llama/llama3) |
| Llama 2 | 7B, 13B, 70B | Meta AI | [Touvron et al. 2023](https://arxiv.org/abs/2307.09288) |
| Code Llama | 7B, 13B, 34B, 70B | Meta AI | [Rozière et al. 2023](https://arxiv.org/abs/2308.12950) |
| Mistral | 7B | Mistral AI | [Mistral website](https://mistral.ai/) |
| Mixtral MoE | 8x7B | Mistral AI | [Mistral AI 2023](https://mistral.ai/news/mixtral-of-experts/) |
| Mistral | 7B | Mistral AI | [Mistral AI 2023](https://mistral.ai/news/announcing-mistral-7b/) |
| CodeGemma | 7B | Google | [Google Team, Google Deepmind](https://ai.google.dev/gemma/docs/codegemma) |
| ... | ... | ... | ... |

<details>
<summary>See full list of 20+ LLMs</summary>

&nbsp;
&nbsp;

#### All models

| Model | Model size | Author | Reference |
|----|----|----|----|
| CodeGemma | 7B | Google | [Google Team, Google Deepmind](https://ai.google.dev/gemma/docs/codegemma) |
| Code Llama | 7B, 13B, 34B, 70B | Meta AI | [Rozière et al. 2023](https://arxiv.org/abs/2308.12950) |
| Danube2 | 1.8B | H2O.ai | [H2O.ai](https://h2o.ai/platform/danube-1-8b/) |
| Dolly | 3B, 7B, 12B | Databricks | [Conover et al. 2023](https://www.databricks.com/blog/2023/04/12/dolly-first-open-commercially-viable-instruction-tuned-llm) |
| Falcon | 7B, 40B, 180B | TII UAE | [TII 2023](https://falconllm.tii.ae) |
| FreeWilly2 (Stable Beluga 2) | 70B | Stability AI | [Stability AI 2023](https://stability.ai/blog/stable-beluga-large-instruction-fine-tuned-models) |
Expand All @@ -78,7 +80,8 @@ LitGPT has 🤯 **custom, from-scratch implementations** of [20+ LLMs](tutorials
| Llama 2 | 7B, 13B, 70B | Meta AI | [Touvron et al. 2023](https://arxiv.org/abs/2307.09288) |
| Llama 3 | 8B, 70B | Meta AI | [Meta AI 2024](https://github.com/meta-llama/llama3) |
| LongChat | 7B, 13B | LMSYS | [LongChat Team 2023](https://lmsys.org/blog/2023-06-29-longchat/) |
| Mistral | 7B | Mistral AI | [Mistral website](https://mistral.ai/) |
| Mixtral MoE | 8x7B | Mistral AI | [Mistral AI 2023](https://mistral.ai/news/mixtral-of-experts/) |
| Mistral | 7B | Mistral AI | [Mistral AI 2023](https://mistral.ai/news/announcing-mistral-7b/) |
| Nous-Hermes | 7B, 13B, 70B | NousResearch | [Org page](https://huggingface.co/NousResearch) |
| OpenLLaMA | 3B, 7B, 13B | OpenLM Research | [Geng & Liu 2023](https://github.com/openlm-research/open_llama) |
| Phi | 1.3B, 2.7B | Microsoft Research | [Li et al. 2023](https://arxiv.org/abs/2309.05463) |
Expand Down Expand Up @@ -121,41 +124,42 @@ pip install -e '.[all]'

&nbsp;
# Quick start
After installing LitGPT, select the model and action you want to take on that model (finetune, pretrain, evaluate, deploy, etc...):
After installing LitGPT, select the model and action you want to take on that model (finetune, pretrain, evaluate, deploy, etc...):

```bash
# ligpt [action] [model]
litgpt download meta-llama/Meta-Llama-3-8B-Instruct
litgpt chat meta-llama/Meta-Llama-3-8B-Instruct
litgpt finetune meta-llama/Meta-Llama-3-8B-Instruct
litgpt pretrain meta-llama/Meta-Llama-3-8B-Instruct
litgpt serve meta-llama/Meta-Llama-3-8B-Instruct
litgpt finetune meta-llama/Meta-Llama-3-8B-Instruct
litgpt pretrain meta-llama/Meta-Llama-3-8B-Instruct
litgpt serve meta-llama/Meta-Llama-3-8B-Instruct
```

&nbsp;

### Use an LLM for inference
Use LLMs for inference to test its chatting capabilities, run evaluations, or extract embeddings, etc...
Here's an example showing how to use the Mistral 7B LLM.
Use LLMs for inference to test its chatting capabilities, run evaluations, or extract embeddings, etc...
Here's an example showing how to use the Phi-2 LLM.

<a target="_blank" href="https://lightning.ai/lightning-ai/studios/litgpt-chat">
<img src="https://pl-bolts-doc-images.s3.us-east-2.amazonaws.com/app-2/studio-badge.svg" alt="Open In Studio"/>
</a>

&nbsp;
&nbsp;

```bash
# 1) Download a pretrained model
litgpt download --repo_id mistralai/Mistral-7B-Instruct-v0.2
litgpt download --repo_id microsoft/phi-2

# 2) Chat with the model
litgpt chat \
--checkpoint_dir checkpoints/mistralai/Mistral-7B-Instruct-v0.2
--checkpoint_dir checkpoints/microsoft/phi-2

>> Prompt: What do Llamas eat?
```

For more information, refer to the [download](tutorials/download_model_weights.md) and [inference](tutorials/inference.md) tutorials.
The download of certain models requires an additional access token. You can read more about this in the [download](tutorials/download_model_weights.md#specific-models-and-access-tokens) documentation.
For more information on the different inference options, refer to the [inference](tutorials/inference.md) tutorial.

&nbsp;

Expand All @@ -166,7 +170,7 @@ For more information, refer to the [download](tutorials/download_model_weights.m
<img src="https://pl-bolts-doc-images.s3.us-east-2.amazonaws.com/app-2/studio-badge.svg" alt="Open In Studio"/>
</a>

&nbsp;
&nbsp;

```bash
# 1) Download a pretrained model
Expand All @@ -189,14 +193,14 @@ litgpt chat \

&nbsp;

### Pretrain an LLM
### Pretrain an LLM
Train an LLM from scratch on your own data via pretraining:

<a target="_blank" href="https://lightning.ai/lightning-ai/studios/litgpt-pretrain">
<img src="https://pl-bolts-doc-images.s3.us-east-2.amazonaws.com/app-2/studio-badge.svg"; alt="Open In Studio"/>
</a>

&nbsp;
&nbsp;

```bash
mkdir -p custom_texts
Expand Down Expand Up @@ -224,15 +228,15 @@ litgpt chat \

&nbsp;

### Continue pretraining an LLM
This is another way of finetuning that specializes an already pretrained model by training on custom data:
### Continue pretraining an LLM
This is another way of finetuning that specializes an already pretrained model by training on custom data:


<a target="_blank" href="https://lightning.ai/lightning-ai/studios/litgpt-continue-pretraining">
<img src="https://pl-bolts-doc-images.s3.us-east-2.amazonaws.com/app-2/studio-badge.svg"; alt="Open In Studio"/>
</a>

&nbsp;
&nbsp;

```bash
mkdir -p custom_texts
Expand Down Expand Up @@ -260,7 +264,7 @@ litgpt chat \
&nbsp;

### Deploy an LLM
Once you're ready to deploy a finetuned LLM, run this command:
Once you're ready to deploy a finetuned LLM, run this command:

<a target="_blank" href="https://lightning.ai/lightning-ai/studios/litgpt-serve">
<img src="https://pl-bolts-doc-images.s3.us-east-2.amazonaws.com/app-2/studio-badge.svg" alt="Open In Studio"/>
Expand All @@ -269,20 +273,20 @@ Once you're ready to deploy a finetuned LLM, run this command:
&nbsp;

```bash
# locate the checkpoint to your finetuned or pretrained model and call the `serve` command:
# locate the checkpoint to your finetuned or pretrained model and call the `serve` command:
litgpt serve --checkpoint_dir path/to/your/checkpoint/microsoft/phi-2

# Alternative: if you haven't finetuned, download any checkpoint to deploy it:
# Alternative: if you haven't finetuned, download any checkpoint to deploy it:
litgpt download --repo_id microsoft/phi-2
litgpt serve --checkpoint_dir checkpoints/microsoft/phi-2
```

Test the server in a separate terminal and integrate the model API into your AI product:
Test the server in a separate terminal and integrate the model API into your AI product:
```python
# 3) Use the server (in a separate session)
import requests, json
response = requests.post(
"http://127.0.0.1:8000/predict",
"http://127.0.0.1:8000/predict",
json={"prompt": "Fix typos in the following sentence: Exampel input"}
)
print(response.json()["output"])
Expand Down Expand Up @@ -491,7 +495,7 @@ litgpt finetune \

&nbsp;

# Community
# Community

## Get involved!

Expand Down
Loading

0 comments on commit 6c4cd25

Please sign in to comment.