Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Multiple gpus on different nodes #427

Closed
Tracked by #418
bjwswang opened this issue Dec 22, 2023 · 4 comments
Closed
Tracked by #418

Multiple gpus on different nodes #427

bjwswang opened this issue Dec 22, 2023 · 4 comments
Assignees
@nkwangleiGIT
Labels
k8s-operator LLM priority-medium
Milestone
v0.2.0

Comments

@bjwswang
Copy link
Collaborator

bjwswang commented Dec 22, 2023 •
edited
Loading

I found a discussion in vllm community about deploy vllm model across multiple nodes in kubernetes
vllm-project/vllm#1363

image
This was referenced Dec 22, 2023
Closed
Merged
@bjwswang
Copy link
Collaborator Author

We may try

  • kuberay -> manger gpu cluster
  • vllm -> run ray application(llm models)
  • rdam -> IB(infiniBand)

@bjwswang bjwswang self-assigned this Dec 22, 2023
@bjwswang bjwswang added this to the v0.2.0 milestone Dec 22, 2023
@bjwswang bjwswang assigned Lanture1064 and bjwswang and unassigned bjwswang Dec 25, 2023
@nkwangleiGIT
Copy link
Contributor

Use ray as distributed runtime to support GPU resource pool across nodes, we can run big LLM models using multiple GPUs now.

@nkwangleiGIT nkwangleiGIT mentioned this issue Dec 30, 2023
Closed
@nkwangleiGIT
Copy link
Contributor

Add docs here: http://kubeagi.k8s.com.cn/docs/category/distributed-inference

nkwangleiGIT added a commit to nkwangleiGIT/arcadia that referenced this issue Jan 5, 2024
@nkwangleiGIT
kubeagi#427 support to run models using ray cluster
7e9a5fb
nkwangleiGIT added a commit to nkwangleiGIT/arcadia that referenced this issue Jan 5, 2024
@nkwangleiGIT
feat: kubeagi#427 support to run models using ray cluster
b83b158
nkwangleiGIT added a commit to nkwangleiGIT/arcadia that referenced this issue Jan 5, 2024
@nkwangleiGIT
feat: kubeagi#427 support to run models using ray cluster
8d83804
nkwangleiGIT added a commit that referenced this issue Jan 5, 2024
@nkwangleiGIT
Merge pull request #500 from nkwangleiGIT/main
cf9e6d4 
feat: #427 support to run models using ray cluster
@nkwangleiGIT
Copy link
Contributor

Fixed by #500

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@nkwangleiGIT nkwangleiGIT

Labels
k8s-operator LLM priority-medium
Projects
None yet
Milestone
v0.2.0
Development

No branches or pull requests
3 participants
@nkwangleiGIT @bjwswang @Lanture1064