Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

tuner.get_best_models() issues warning Detecting that an object or model or tf.train.Checkpoint is being deleted with unrestored values #982

Open
giulatona opened this issue Nov 29, 2023 · 1 comment
Labels
bug Something isn't working

Comments

@giulatona
Copy link

Describe the bug
Calling tuner.get_best_models() at the end of hp tuning issues several warnings due to missing checkpoints.

WARNING:tensorflow:Detecting that an object or model or tf.train.Checkpoint is being deleted with unrestored values. See the following logs for the specific values in question. To silence these warnings, use status.expect_partial(). See https://www.tensorflow.org/api_docs/python/tf/train/Checkpoint#restorefor details about the status object returned by the restore function.
WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).layer_with_weights-0.kernel
WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).layer_with_weights-0.bias
WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).layer_with_weights-1.kernel
WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).layer_with_weights-1.bias
WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).optimizer._variables.1
WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).optimizer._variables.2
WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).optimizer._variables.3
WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).optimizer._variables.4
WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).optimizer._variables.5
WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).optimizer._variables.6
WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).optimizer._variables.7
WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).optimizer._variables.8

To Reproduce
The official example in the documentation shows this problem.
It can be reproduced in the related Colab Notebook.

Expected behavior
No warnings should be issued.

Would you like to help us fix it?
I do not know the codebase enough.

@giulatona giulatona added the bug Something isn't working label Nov 29, 2023
@grmelacz
Copy link

grmelacz commented May 9, 2024

I might not be your case, however I have fixed this error by removing my custom-defined metric (in my case f1score) from the model used in build_model(). It seems it was not saved properly/not implemented properly for this specific (tuner) case.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants