New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Various fixes for TP #260

Merged

michaelbenayoun merged 3 commits into main from fix_trainer_for_tp

Oct 23, 2023

Member

michaelbenayoun commented Oct 13, 2023 •

edited

Loading

Evaluation can now happens in a data parallel setting when it makes sense.
Loss is averaged properly under TP setting

HuggingFaceDocBuilderDev commented Oct 13, 2023

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint.

michaelbenayoun changed the title ~~Fix loss log when using TP~~ Various fix for TP

michaelbenayoun changed the title ~~Various fix for TP~~ Various fixes for TP

michaelbenayoun added 3 commits

October 20, 2023 11:58


          Fix loss log when using TP

8fe5667


          Make evaluation work with DP / TP

522d295


          Final changes

7c835b0

michaelbenayoun force-pushed the fix_trainer_for_tp branch from 7be5db6 to 7c835b0 Compare

October 20, 2023 09:59

michaelbenayoun merged commit 4b37209 into main

14 of 16 checks passed

michaelbenayoun deleted the fix_trainer_for_tp branch

October 23, 2023 09:00

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet