Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Handle serialization/loading of ConfigHolder with missing values #25

Closed
rusheb opened this issue Feb 20, 2023 · 4 comments
Closed

Handle serialization/loading of ConfigHolder with missing values #25

rusheb opened this issue Feb 20, 2023 · 4 comments

Comments

@rusheb
Copy link
Collaborator

rusheb commented Feb 20, 2023

If ConfigHolder has missing values it will fail to serialize, as demonstrated in test_serialize_and_load_missing_values.

This might be an issue if we don't have all the configs, e.g. at the dataset creation stage.

However maybe at that point we wouldn't serialize the entire config. I'm not sure what the best behaviour should be

@valedan
Copy link
Contributor

valedan commented Mar 22, 2023

This isn't causing any problems currently because during dataset creation we don't use ConfigHolder, just the dataset config.

I'm not really sure what our long-term plans are with configs and ConfigHolder specifically, so I can't say if this is something we should work on or not. @mivanit do you have any input here?

@mivanit
Copy link
Member

mivanit commented Mar 23, 2023

If we aren't currently using ConfigHolder with missing values anywhere, I think its best to have it throw an exception if we try to serialize one with missing values.

@valedan
Copy link
Contributor

valedan commented Mar 23, 2023

Sounds reasonable. In that case the only TODO for this ticket would be to delete the test that @rusheb linked.

@mivanit
Copy link
Member

mivanit commented Aug 6, 2023

I haven't had any issues with this for a while, despite several changes to the ConfigHolder (i.e. maze_tokenizer attr in #191 ). Key point is to make sure you allow for missing values in the loading_fn

@mivanit mivanit closed this as completed Aug 6, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants