Script to automatically split off eval set #1525
Conversation
| ) | ||
| if _is_empty_or_nonexistent(dirpath=dataset_name): | ||
| log.error("Failed to safely load the dataset from HF Hub.") | ||
| raise InvalidFileExtensionError( |
There was a problem hiding this comment.
Moved HF safe download logic into a separate function so I could reuse it. Code is basically unchanged.
Unrelated to meat of the PR, but per Daniel request, tried refactoring further to get rid of try-except block. I don't think it can be done without significant added complexity. You want to barrier/sync before throwing this InvalidFileExtensionError; however, this must be nested within this download logic so that only rank0 encounters it. So you can't both separate out this logic and have graceful exit without try-except. :'(
| import numpy as np | ||
| from typing import Optional | ||
|
|
||
| import composer.utils as utils |
There was a problem hiding this comment.
import is formatted this way due to mocking difficulties - https://bhfsteve.blogspot.com/2012/06/patching-tip-using-mocks-in-python-unit.html
What changes are proposed in this pull request?
See go/sweeps-eval for more context. Corresponding MAPI changes are contained in https://github.com/databricks-mosaic/mcloud/pull/4562.
How is this tested?
Added unit tests.
The predecessor to the unit tests was this bash script I used to manually inspect the file outputs. Tests are the same as unit tests, but it invokes the
scripts/...file instead of thecommand_utils/...one. Confirmed file outputs looked OK._test.txt
Everything passes:
