Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Incorrect Imagenet evals with pytorch_eval_num_workers > 0 #732

Open
priyakasimbeg opened this issue Mar 27, 2024 · 2 comments
Open

Incorrect Imagenet evals with pytorch_eval_num_workers > 0 #732

priyakasimbeg opened this issue Mar 27, 2024 · 2 comments
Labels
🐛 Bug Something isn't working 🔥 PyTorch Issue that mainly deals with the PyTorch version of the code

Comments

@priyakasimbeg
Copy link
Contributor

AlgoPerf submitter team reports that they are no longer able to reproduce the NAdam baseline results in PyTorch using the current repo in PyTorch on the ImageNet workloads (both ResNet and ViT).
See the plot below in terms of differences in the training/validation loss and accuracy between the given NAdam Jax results and the current run's results on ImageNet ViT.

They did not see a change in OGBG and FastMRI.

The list of commits that we merged were from 389fe3f823a5016289b55b48aa8061a37b18b401 to 79ccc5e860d7928cf896ffe12ec686c72fd840d4.

image

Steps to Reproduce

Running submission runner with eval_num_workers=4 (recently changed default to help speed up evals).

Source or Possible Fix

Setting the eval_num_workers to 0 resolves the discrepancy in evals. We are still investigating why.

@priyakasimbeg priyakasimbeg changed the title Incorrect Imagenet evals for PyTorch data loader num workers > 0 Incorrect Imagenet evals with pytorch_eval_num_workers > 0 Mar 27, 2024
@priyakasimbeg
Copy link
Contributor Author

priyakasimbeg commented Mar 27, 2024

Changed default number of workers for PyTorch data loaders to 0.
Important update: for speech workloads the pytorch_eval_num_workers flag to submission_runner.py has to be set to >0, to prevent data loader crash in jax code.

@runame runame added 🔥 PyTorch Issue that mainly deals with the PyTorch version of the code 🐛 Bug Something isn't working labels Mar 28, 2024
@runame
Copy link
Contributor

runame commented Apr 3, 2024

I tried reproducing the issue by running the target setting run on the current dev branch with pytorch_eval_num_workers=4, but I don't see the drop in eval metrics compared to an older reference run (this one).

If someone can share the exact command and commit they used to produce the run in the plot I will try to run this instead.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
🐛 Bug Something isn't working 🔥 PyTorch Issue that mainly deals with the PyTorch version of the code
Projects
None yet
Development

No branches or pull requests

2 participants