document for huggingface(vllm) servingruntime for multi-node #402
Conversation
✅ Deploy Preview for elastic-nobel-0aef7a ready!
To edit notification comments on pull requests, go to your Netlify site configuration. |
Signed-off-by: jooho lee <[email protected]>
Signed-off-by: jooho lee <[email protected]>
Signed-off-by: jooho lee <[email protected]>
| timeoutSeconds: 5 | ||
| initialDelaySeconds: 10 | ||
| .. | ||
| ~~~ |
There was a problem hiding this comment.
maybe add a note on how and where to do it?
If using custom runtime or isvc.
There was a problem hiding this comment.
this needs to be set in servingruntime. like huggingfaceserver-multinode
I add this simply to the doc.
|
|
||
| ### Key Validations | ||
|
|
||
| - `TENSOR_PARALLEL_SIZE` and `PIPELINE_PARALLEL_SIZE` cannot be set via environment variables. They must be configured through `workerSpec.tensorParallelSize` and `workerSpec.pipelineParallelSize`. |
There was a problem hiding this comment.
| - `TENSOR_PARALLEL_SIZE` and `PIPELINE_PARALLEL_SIZE` cannot be set via environment variables. They must be configured through `workerSpec.tensorParallelSize` and `workerSpec.pipelineParallelSize`. | |
| - `TENSOR_PARALLEL_SIZE` and `PIPELINE_PARALLEL_SIZE` cannot be set via environment variables. They must be configured through `workerSpec.tensorParallelSize` and `workerSpec.pipelineParallelSize` respectively. |
| ### Key Validations | ||
|
|
||
| - `TENSOR_PARALLEL_SIZE` and `PIPELINE_PARALLEL_SIZE` cannot be set via environment variables. They must be configured through `workerSpec.tensorParallelSize` and `workerSpec.pipelineParallelSize`. | ||
| - In a ServingRuntime designed for multi-node, both `workerSpec.tensorParallelSize` and `workerSpec.pipelineParallelSize` must be set. |
There was a problem hiding this comment.
there is no default expected default values for pipelineParallelSize (as it should be > 2)
There was a problem hiding this comment.
line number 16 explains the minimum value per field
| - In a ServingRuntime designed for multi-node, both `workerSpec.tensorParallelSize` and `workerSpec.pipelineParallelSize` must be set. | ||
| - The minimum value for `workerSpec.tensorParallelSize` is **1**, and the minimum value for `workerSpec.pipelineParallelSize` is **2**. | ||
| - Currently, four GPU types are allowed: `nvidia.com/gpu` (*default*), `intel.com/gpu`, `amd.com/gpu`, and `habana.ai/gaudi`. | ||
| - You can specify the GPU type via InferenceService, but if it differs from what is set in the ServingRuntime, both GPU types will be assigned to the resource. Then it can cause issues. |
There was a problem hiding this comment.
| - You can specify the GPU type via InferenceService, but if it differs from what is set in the ServingRuntime, both GPU types will be assigned to the resource. Then it can cause issues. | |
| - You can specify the GPU type via InferenceService, but if it differs from what is set in the ServingRuntime, both GPU types will be assigned to the resource. Then it can cause issues. |
There was a problem hiding this comment.
make sure it is nested from the previous bullet regarding GPUs.
There was a problem hiding this comment.
it was implemented but after discussion we decided to add a validation check for it. User have to set this.
| - The minimum value for `workerSpec.tensorParallelSize` is **1**, and the minimum value for `workerSpec.pipelineParallelSize` is **2**. | ||
| - Currently, four GPU types are allowed: `nvidia.com/gpu` (*default*), `intel.com/gpu`, `amd.com/gpu`, and `habana.ai/gaudi`. | ||
| - You can specify the GPU type via InferenceService, but if it differs from what is set in the ServingRuntime, both GPU types will be assigned to the resource. Then it can cause issues. | ||
| - The Autoscaler must be configured as `external`. |
There was a problem hiding this comment.
if no other autoscaler is supported, why not default to it independently of what the user defines?
There was a problem hiding this comment.
I don't think there is a way to set default autoscaler in ServingRuntime.
| - Currently, four GPU types are allowed: `nvidia.com/gpu` (*default*), `intel.com/gpu`, `amd.com/gpu`, and `habana.ai/gaudi`. | ||
| - You can specify the GPU type via InferenceService, but if it differs from what is set in the ServingRuntime, both GPU types will be assigned to the resource. Then it can cause issues. | ||
| - The Autoscaler must be configured as `external`. | ||
| - The only supported storage protocol for StorageURI is `PVC`. |
There was a problem hiding this comment.
isn't modelCar already supported by KServe?
There was a problem hiding this comment.
The first phase only supports PVC. modelcar might be included in the next phase.
|
|
||
|
|
||
| 2. **`workerSpec.pipelineParallelSize`** | ||
| This setting determines how many nodes are involved in the deployment. This variable represents the total number of nodes, including both the head and worker nodes. |
There was a problem hiding this comment.
I think it's worth mentioning that all nodes must have GPU available.
There was a problem hiding this comment.
Not all nodes need to have GPUs, as affinity can be used to select GPU-enabled nodes. Additionally, NFD (Node Feature Discovery) can add GPU labels to nodes, allowing the openshift to choose nodes with GPUs automatically.
|
|
||
| ### 2. Download the Model to the PVC | ||
|
|
||
| To download the model, export your Hugging Face token (`HF_TEST_TOKEN`) as an environment variable. Replace `%token%` with your actual token. |
There was a problem hiding this comment.
It has no special meaning.
|
|
||
| ### 2. Download the Model to the PVC | ||
|
|
||
| To download the model, export your Hugging Face token (`HF_TEST_TOKEN`) as an environment variable. Replace `%token%` with your actual token. |
There was a problem hiding this comment.
is the %string% the pattern for place holders?
I've seen on other places {{string}}
There was a problem hiding this comment.
correct, it is for placeholders.
The pattern is what I usually use but not something special. The most important thing is that the user understands what the string means. I think it is understandable for users
| !!! success "Expected Output" | ||
| ```{ .bash .no-copy } | ||
| NAME URL READY PREV LATEST PREVROLLEDOUTREVISION LATESTREADYREVISION AGE | ||
| huggingface-llama3 http://huggingface-llama3.default.example.com 12m |
There was a problem hiding this comment.
It does not matter to me. And also, actually 405b model take 12 mins only for loading the model.
| huggingface-llama3 http://huggingface-llama3.default.example.com 12m | ||
| ``` | ||
|
|
||
|
|
Signed-off-by: jooho lee <[email protected]>
| @@ -0,0 +1,330 @@ | |||
| # Multi-node/Multi-GPU Inference with Hugging Face LLM Serving Runtime | |||
There was a problem hiding this comment.
| # Multi-node/Multi-GPU Inference with Hugging Face LLM Serving Runtime | |
| # Multi-node/Multi-GPU Inference with Hugging Face vLLM Serving Runtime |
| @@ -0,0 +1,330 @@ | |||
| # Multi-node/Multi-GPU Inference with Hugging Face LLM Serving Runtime | |||
|
|
|||
| This guide provides step-by-step instructions on setting up multi-node and multi-GPU inference using Hugging Face's LLM Serving Runtime. Before proceeding, please ensure you meet the following prerequisites and understand the limitations of this setup. | |||
There was a problem hiding this comment.
| This guide provides step-by-step instructions on setting up multi-node and multi-GPU inference using Hugging Face's LLM Serving Runtime. Before proceeding, please ensure you meet the following prerequisites and understand the limitations of this setup. | |
| This guide provides step-by-step instructions on setting up multi-node and multi-GPU inference using Hugging Face's vLLM Serving Runtime. Before proceeding, please ensure you meet the following prerequisites and understand the limitations of this setup. |
| - `TENSOR_PARALLEL_SIZE` and `PIPELINE_PARALLEL_SIZE` cannot be set via environment variables. They must be configured through `workerSpec.tensorParallelSize` and `workerSpec.pipelineParallelSize` respectively. | ||
| - In a ServingRuntime designed for multi-node, both `workerSpec.tensorParallelSize` and `workerSpec.pipelineParallelSize` must be set. | ||
| - The minimum value for `workerSpec.tensorParallelSize` is **1**, and the minimum value for `workerSpec.pipelineParallelSize` is **2**. | ||
| - Currently, four GPU types are allowed: `nvidia.com/gpu` (*default*), `intel.com/gpu`, `amd.com/gpu`, and `habana.ai/gaudi`. |
There was a problem hiding this comment.
Should we add reference on how to add additional types? e.g. adding them here https://github.com/kserve/kserve/blob/21b103e1ce7166a444d0145412cee19bd6574309/pkg/constants/constants.go#L218-L223
There was a problem hiding this comment.
Oh I see. There are instructions later
| "nvidia.com/gpu" | ||
| "amd.com/gpu" | ||
| "intel.com/gpu" | ||
| "habana.ai/gaudi" |
There was a problem hiding this comment.
Already mentioned these earlier
| | NVIDIA-SMI 550.90.07 Driver Version: 550.90.07 CUDA Version: 12.4 | | ||
| |-----------------------------------------+------------------------+----------------------+ | ||
| | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | | ||
| | Fan Temp Perf Pwr:Usage/Cap | # Specifying workerSpec indicates that multi-node functionality will be used Memory-Usage | GPU-Util Compute M. | |
There was a problem hiding this comment.
There's some unexpected copy paste here
|
|
||
| - Multi-node functionality is only supported in **RawDeployment** mode. | ||
| - **Auto-scaling is not available** for multi-node setups. | ||
| - A **Persistent Volume Claim (PVC)** is required for multi-node configurations, and it must support the **ReadWriteMany (RWM)** access mode. |
There was a problem hiding this comment.
| - A **Persistent Volume Claim (PVC)** is required for multi-node configurations, and it must support the **ReadWriteMany (RWM)** access mode. | |
| - A **Persistent Volume Claim (PVC)** is required for multi-node configurations, and it must support the **ReadWriteMany (RWX)** access mode. |
?
"Fixes #issue-number" or "Add description of the problem this PR solves"
Proposed Changes
This PR add a new documentation for setting up multi-node/multi-GPU inference using the Hugging Face LLM Serving Runtime. It includes detailed instructions on prerequisites, key configurations, model inference, and sample requests for OpenAI completions and chat endpoints. This documentation aims to enhance user understanding and streamline the deployment process, ensuring a smooth experience for developers looking to leverage Hugging Face's capabilities in a Kubernetes environment
This documentation is valid only after kserve/kserve#3972 is merged.