Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add debug log before training loop starts #3005

Merged
merged 1 commit into from
Feb 13, 2024

Conversation

mvpatel2000
Copy link
Contributor

What does this PR do?

Streaming often takes a while to start first batch. This changes last seen debug message from user to starting training loop. This avoids confusion with users thinking spinning dataloaders is taking a really long time.

Copy link
Contributor

@bigning bigning left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

simple change, lgtm.

@dblalock
Copy link
Contributor

dblalock commented Feb 13, 2024

Bonus points: can we make batch 0 print for any console_log_interval? So that with an interval of 10 we print after batches {0, 10, 20}, etc instead of {9, 19, 29}, etc? I think this would address 80% of the motivation for the more granular logging I mentioned in the JIRA issue. (May or may not make sense to do in this PR.)

@mvpatel2000
Copy link
Contributor Author

Bonus points: can we make batch 0 print for any console_log_interval? So that with an interval of 10 we print after batches {0, 10, 20}, etc instead of {9, 19, 29}, etc? I think this would address 80% of the motivation for the more granular logging I mentioned in the JIRA issue. (May or may not make sense to do in this PR.)

I'll follow up offline. We already do this, maybe you were testing a resumption?

@mvpatel2000 mvpatel2000 merged commit 953ee88 into mosaicml:dev Feb 13, 2024
14 checks passed
@mvpatel2000 mvpatel2000 deleted the mvpatel2000/add-log branch February 13, 2024 18:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants