This repository has been archived by the owner on May 28, 2024. It is now read-only.
Add more details about prompt format in the docs #126
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Alan Guo <[email protected]>
alanwguo
commented
Jan 24, 2024
models/README.md
Outdated
| @@ -69,6 +69,24 @@ RayLLM supports continuous batching, meaning incoming requests are processed as | |||
| You can follow the TensorRT-LLM example to generate the model.(https://github.com/NVIDIA/TensorRT-LLM/tree/v0.6.1/examples/llama). After generating the model, you can upload the model artifact to S3 and use the `s3_mirror_config` to load the model from S3. You can also place the model artifacts in a local directory and use the `model_local_path` to load the model from the local directory. See the [llama example](continuous_batching/trtllm-meta-llama--Llama-2-7b-chat-hf.yaml) for more details. | |||
|
|
|||
|
|
|||
| #### Prompt Format | |||
| A prompt format is used to convert a chat completions API input into a prompt to feed into the LLM engine. The format is a dictionary where the key refers to one of the chat actors and the value is a string template for which to convert the text of the actor into a string to add to the overall prompt. The template is used to generate a portion of the prompt and each portion is assembled together to form the final prompt. | |||
There was a problem hiding this comment.
I wasn't sure if this prompt_format is only used in the ChatCompletions API.
I assume it is
Signed-off-by: Alan Guo <[email protected]>
tchordia
approved these changes
Jan 25, 2024
|
There's an existing example in the docs below. I'll add a reference to it. |
Signed-off-by: Alan Guo <[email protected]>
alanwguo
pushed a commit
that referenced
this pull request
Jan 25, 2024
Signed-off-by: Max Pumperla <[email protected]>
alanwguo
added a commit
that referenced
this pull request
Jan 25, 2024
Trying to make it easier for users to self-service add custom models to use with ray-llm. --------- Signed-off-by: Alan Guo <[email protected]>
alanwguo
added a commit
that referenced
this pull request
Jan 26, 2024
Trying to make it easier for users to self-service add custom models to use with ray-llm. Cherry-pick of #126 --------- Signed-off-by: Alan Guo <[email protected]>
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Trying to make it easier for users to self-service add custom models to use with ray-llm.