Skip to content

Latest commit

 

History

History
28 lines (22 loc) · 2.78 KB

File metadata and controls

28 lines (22 loc) · 2.78 KB

Quantization in RayLLM

Quantization is a technique to reduce the computational and memory costs of running inference by representing the weights and/or activations with low-precision data types like 4-bit integer (int4) instead of the usual 16-bit floating point (float16). Quantization allows users to deploy models with cheaper hardware requirements with potentially lower inference costs.

RayLLM supports AWQ and SqueezeLLM weight-only quantization by integrating with vLLM. Quantization can be enabled in RayLLM by specifying the quantization method in engine_kwargs and using a quantized model for model_id and hf_model_id. See the configs in this directory for quantization examples. Note that the AWQ and SqueezeLLM quantization methods in vLLM have not been fully optimized and can be slower than FP16 models for larger batch sizes.

See the following tables for benchmarks conducted on Llama2 models using the llmperf evaluation framework with vLLM 0.2.2. The quantized models were benchmarked for end-to-end (E2E) latency, time to first token (TTFT), iter-token latency (ITL), and generation throughput using default llmperf parameters.

Llama2 7B on 1 A100 80G

Quantization Method Mean E2E (ms) Mean TTFT (ms) Mean ITL (ms/token) Mean Throughput (tok/s)
Baseline (W16A16) 3212 362 18.81 53.44
AWQ (W4A16) 4148 994 21.76 47.09
SqueezeLLM (W4A16) 42372 13857 109.77 9.13

Llama2 13B on 1 A100 80G

Quantization Method Mean E2E (ms) Mean TTFT (ms) Mean ITL (ms/token) Mean Throughput (tok/s)
Baseline (W16A16) 4371 644 31.06 32.25
AWQ (W4A16) 5626 1695 41.35 24.48
SqueezeLLM (W4A16) 64293 21676 628.71 5.6

Llama2 70B on 4 A100 80G (SqueezeLLM Llama2 70B not available on huggingface)

Quantization Method Mean E2E (ms) Mean TTFT (ms) Mean ITL (ms/token) Mean Throughput (tok/s)
Baseline (W16A16) 8048 1073 58.1 17.26
AWQ (W4A16) 9902 2174 69.64 14.4