Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature] LISA multi GPU support #899

Open
wants to merge 2 commits into
base: main
Choose a base branch
from

Conversation

wheresmyhair
Copy link
Collaborator

@wheresmyhair wheresmyhair commented Sep 25, 2024

Description

LISA now supports multi GPU.
Key points:

  1. When initialize models, ds will wrap all model params with optimizer, which uses huge amount of gpu mem. However, according to LISA's logic, we do not need to wrap all params with optim, since only activated layers will be updated. So we hack the initialization such that trainer only wrap one layer at the beginning (to avoid CUDA oom error).
  2. Currently cannot save training args when using LISATrainer, as it will raise error when pickling. Model weights can be saved correctly.

@wheresmyhair wheresmyhair mentioned this pull request Sep 25, 2024
30 tasks
Copy link
Contributor

@research4pan research4pan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

src/lmflow/pipeline/finetuner.py

  • [Style] line 423-424, 428, 486: remove print or change them to logger.info.
  • [Question] It is highly recommended to add unittest for this script.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants