Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Typo] Fix missing links in the bitnet integration's docs #136

Merged
merged 5 commits into from
Aug 9, 2024
Merged

[Typo] Fix missing links in the bitnet integration's docs #136

Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
7aee8f4
fix install with absolute path
LeiWang1999 Aug 9, 2024
5cb5349
efficient inference with torch compile
LeiWang1999 Aug 9, 2024
e73c563
update vllm ckpt tutorial for bitnet
LeiWang1999 Aug 9, 2024
cd116f9
ReadME Fix.
LeiWang1999 Aug 9, 2024
a118f60
Merge branch 'main' of https://github.com/microsoft/BitBLAS into bitnet
LeiWang1999 Aug 9, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 4 additions & 3 deletions integration/BitNet/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,12 +2,13 @@
license: mit
---

## Latest News

- 08/09/2024 ✨: We provide a more efficient implementation for bitnet with vLLM, which should use special model checkpoints, to make the ckpt, please reach [].

This is a BitBLAS Implementation for the reproduced 1.58bit model from [1bitLLM/bitnet_b1_58-3B](https://huggingface.co/1bitLLM/bitnet_b1_58-3B). We replaced the original simulated Int8x3bit Quantized Inference Kernel with BitBLAS INT8xINT2 Kernel. We also evaluated the model's correctness and performance through `eval_correctness.py` and `benchmark_inference_latency.py`.

## Latest News

- 08/09/2024 ✨: We provide a more efficient implementation for bitnet with vLLM, which should use special model checkpoints, to make the ckpt and study how to deploy, please checkout [Make Checkpoints for vLLM](#make-checkpoints-for-vllm).

## Make Checkpoints for vLLM

We provide two scripts to make the checkpoints for vLLM. The first script is `generate_bitnet_model_native_format.sh`, which is used to make a checkpoint with fp16 uncompressed metaadta, the main difference with the original checkpoint is the `quant_config.json`, which allow vLLM to load the model and execute with a quant extension.
Expand Down
Loading