Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Output eval logging batch #961

Merged
merged 50 commits into from
Mar 28, 2024

Conversation

maxisawesome
Copy link
Contributor

@maxisawesome maxisawesome commented Feb 8, 2024

PR with adding output eval logging callback for eval.py. Works with recent changes to composer

Required in your yaml:

integrations:
  - integration_type: wandb
    entity: mosaic-ml
    project: <ur_project>

parameters:
  callbacks:
    eval_output_logging: {}

  loggers:
    wandb: {}

test run here: `test-batch-logging-izC0mH
wandb logged results here: https://wandb.ai/mosaic-ml/test-eval-logging/runs/km61skch/workspace

@maxisawesome maxisawesome marked this pull request as ready for review March 9, 2024 00:57
Copy link
Contributor

@bmosaicml bmosaicml left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great work Max! this was a many months effort to get a design acceptable to everyone and it involved some very complex torch metrics related engineering. Kudos!

@maxisawesome
Copy link
Contributor Author

Might need to wait for composer release (from composer.callbacks import EvalOutputLogging) @dakinggg

@nik-mosaic
Copy link
Contributor

@maxisawesome now that Composer 0.21 is out, is this still blocked?

@maxisawesome
Copy link
Contributor Author

yes @nik-mosaic I'm running some tests and then will merge

@maxisawesome maxisawesome changed the title [WIP] Output eval logging batch Output eval logging batch Mar 28, 2024
@maxisawesome maxisawesome merged commit 7a8a156 into mosaicml:main Mar 28, 2024
9 checks passed
@maxisawesome maxisawesome deleted the output_eval_logging_batch branch March 28, 2024 21:38
KuuCi pushed a commit that referenced this pull request Apr 18, 2024
* Skip flaky lion8b test (#598)

* relax atol and add retries to reduce flakiness in lion8b timing test

* add eval output logging

* add back tasks

* foo

* add rlhf prompts

* add rlhf prompts

* add rlhf prompts

* add rlhf prompts

* add rlhf prompts

* fix prompt

* fix prompt

* modify mcli

* test

* test

* fix

* fix merge

* wip

* merge

* reset files, wip commit

* rm small changes

* reduce changes

* reduce changes

* .

* wip

* rm batch keys

* revert init device

* linting

* add import

* fix import

* add eval_output_logging to registry

* readd import

* pyright + linting

---------

Co-authored-by: dblalock <[email protected]>
Co-authored-by: Jeremy Dohmann <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants