-
Notifications
You must be signed in to change notification settings - Fork 525
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Enable gauntlet training #501
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for putting this PR together, left a few comments:
- the new yaml files have unexplained changes (batch size, fsdp config), and also have several hardcoded internal OCI bucket names and cluster names. Please remove the hardcoded values prior to pushing PRs.
- can we avoid silently adding in an InMemoryLogger, purely for the purposes of passing around data? Seems like there's a deeper design issue if we have to resort to this approach.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Might be worth using state to get the metrics instead of inmemorylogger
I don't think |
…undry into enable_gauntlet_training
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
see remaining comments around backwards compatibility and a couple code changes that I think were a bad merge.
This PR enables us to run the gauntlet during training. It also enables you to set a subset of batches to run eval over in order to speed up ICL during training.
It also changes eval to no longer use the InMemoryLogger and instead pull the metrics from the State.
Eval test run:
all-eval-F2oe1D
Training test run:
test-1b-C5QG5k
test-1b-c7sqCh
https://wandb.ai/mosaic-ml/gauntlet/runs/rrbn0qbb?workspace=user-jemdohmann
Eval results still good: