Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

FP8 AllGather Support in Fairscale #1185

Open
wants to merge 21 commits into
base: ngoyal_changes_for_pp_fp8_jiecaoyu_debug
Choose a base branch
from
Open

FP8 AllGather Support in Fairscale #1185

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
73ce4b4
added option for no PG validation for faster init (#1161)
ngoyal2707 Jan 24, 2024
33457b3
Mirros Jiecao's change.
levendlee Mar 13, 2024
70f5ff5
Debug non-determinism issues.
levendlee Mar 27, 2024
16c682d
Moves amax update logic into params downcasting function.
levendlee Mar 28, 2024
24a769f
Cleans up code.
levendlee Mar 29, 2024
3e2e77f
Fix `main_grad` attribute checking.
levendlee Apr 2, 2024
1be7aa0
Fix no pp hanging error.
levendlee Apr 8, 2024
57eb557
Clean up shard offset calculation logic.
levendlee Apr 10, 2024
e9e8f8e
Unify compute dtype setting.
levendlee Apr 16, 2024
21f8e05
Have FP16 and FP8 sharded in the same way.
levendlee Apr 17, 2024
8ec7c1d
added option for no PG validation for faster init (#1161)
ngoyal2707 Jan 24, 2024
f27ab17
Mirros Jiecao's change.
levendlee Mar 13, 2024
fa9cf77
Debug non-determinism issues.
levendlee Mar 27, 2024
6fa19e0
Moves amax update logic into params downcasting function.
levendlee Mar 28, 2024
80ffd54
Cleans up code.
levendlee Mar 29, 2024
afb2ca1
Fix `main_grad` attribute checking.
levendlee Apr 2, 2024
25b2322
Fix no pp hanging error.
levendlee Apr 8, 2024
0d1502b
Clean up shard offset calculation logic.
levendlee Apr 10, 2024
5edb109
Unify compute dtype setting.
levendlee Apr 16, 2024
2df199f
Have FP16 and FP8 sharded in the same way.
levendlee Apr 17, 2024
da36e31
Merge branch 'shikaili_fp8_allgather_no_pp_fix' of github.com:faceboo…
levendlee May 20, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Loading
Loading