[WIP] Error logging callback #2533
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…r into error_logging_callback
Add pytorch nightly and CUDA 12.1 support for composer docker images What issue(s) does this change relate to? Related to https://mosaicml.atlassian.net/browse/GRT-2305 Tests docker image: mosaicml/ci-staging:72744756-794c-4390-94db-72c212dd5e00 (cuda 12.1, pytorch 2.1.0) mcli connect temp-test-ZAVxMh Python 3.10.12 (main, Jun 7 2023, 12:45:35) [GCC 9.4.0] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import torch >>> print(torch.version) <module 'torch.version' from '/usr/lib/python3/dist-packages/torch/version.py'> >>> print(torch.__version__) 2.1.0.dev20230623+cu121 >>> print(torch.version.cuda) 12.1 Integration Test @mvpatel2000 has validated that this trains on initial mpt-2 experiments and speeds up training by +7-8% from 0.25 MFU to 0.27 MFU
* fix autoresume with slashed directory * Revert "fix autoresume with slashed directory" This reverts commit 3dfb5f5. revert * fix * fix precommit * Update in_context_learning_evaluation.py * Update in_context_learning_evaluation.py * Update in_context_learning_evaluation.py * add tests
Signed-off-by: Prithvi Kannan <[email protected]> Co-authored-by: Evan Racah <[email protected]> Co-authored-by: eracah <[email protected]>
Upstreams and generalizes the callback that logs generations to wandb from foundry to composer.
…2476) Upgrade torch docker nightly version to 08-23-23 so that we get nccl version 0.18.3 which was merged on 08-18-23.
Test pytorch 2.1.0 docker images on ci/cd mosaicml#2469
Will be removed in v0.18.
* Update RTD build config with build.os * Remove python.version --------- Co-authored-by: Bandish Shah <[email protected]>
…r into error_logging_callback
This was referenced Feb 8, 2024
|
Successfully merged here |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What does this PR do?
This PR adds a callback that logs ICL outputs during eval. It modifies the custom metrics to keep track of incorrect model outputs. Each metric is responsible for specifying the table schema for logging cached responses as well as specifying how to format the cached responses using the tokenizer.
The EvalOutputLogging callback is then responsible for logging the cached results in table format after each evaluation.
Design doc
What issue(s) does this change relate to?
Before submitting
pre-commiton your change? (see thepre-commitsection of prerequisites)