min/max flagging added to system_metrics_monitor with only non-redundant, necessary gpu metrics logged #3373
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
mvpatel2000
reviewed
Jun 6, 2024
mvpatel2000
reviewed
Jun 7, 2024
j316chuck
approved these changes
Jun 10, 2024
mvpatel2000
added a commit
to mvpatel2000/composer
that referenced
this pull request
Jul 21, 2024
…ant, necessary gpu metrics logged (mosaicml#3373) * implemented min_max flag * fixed string parsing * refactoring compute_system_metrics for all_reduce * keep track of rank within dict * added compute_min_max * added flag for both min_max and all_logging * corrected min_max call with model_device * removing total bytes (always going ot be constant) * handled no gpu case in min_max flag * removed unnecessary imports, patched unit tests * fixed assert statement for with gpu case, world size 1 * case min_rank and max_rank as int to guarantee them working as indices * fixed indent issue from fixing font * made docs more concise and readable * fixing unexpected unindent * fixing unit test device * modifying device to equal model_device.type * reverting to device=model_device * setting device in unit test = 'gpu' * setting device = 'cuda' in unit testing * reverting to next(state.model.parameters()).device * removed torch as a dependecy for unit_testing * cleaned up UI to be consistent + removed calling next to obtain device --------- Co-authored-by: Mihir Patel <[email protected]> Co-authored-by: Charles Tang <[email protected]>
mvpatel2000
added a commit
that referenced
this pull request
Jul 21, 2024
…ant, necessary gpu metrics logged (#3373) * implemented min_max flag * fixed string parsing * refactoring compute_system_metrics for all_reduce * keep track of rank within dict * added compute_min_max * added flag for both min_max and all_logging * corrected min_max call with model_device * removing total bytes (always going ot be constant) * handled no gpu case in min_max flag * removed unnecessary imports, patched unit tests * fixed assert statement for with gpu case, world size 1 * case min_rank and max_rank as int to guarantee them working as indices * fixed indent issue from fixing font * made docs more concise and readable * fixing unexpected unindent * fixing unit test device * modifying device to equal model_device.type * reverting to device=model_device * setting device in unit test = 'gpu' * setting device = 'cuda' in unit testing * reverting to next(state.model.parameters()).device * removed torch as a dependecy for unit_testing * cleaned up UI to be consistent + removed calling next to obtain device --------- Co-authored-by: Mihir Patel <[email protected]> Co-authored-by: Charles Tang <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Added logging for gpu power and removed all gpu metrics covered in other callbacks (used, free, total memory).
The following GPU metrics pertaining to Straggler Detection are logged:
Added clearer documentation for the SystemsMetricMonitor class and removed the user inputted boolean parameter for whether the model uses GPU's or not (gpu_available).
Added a boolean flag for users to select whether to log the min/max values for the GPU metrics listed above and their corresponding their ranks, or log all values for all ranks (log_all_data, default set to false).
Before submitting
pre-commiton your change? (see thepre-commitsection of prerequisites)