-
Notifications
You must be signed in to change notification settings - Fork 1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
HELM evaluation framework tries to generate with temperature=0
which leads to inf
tensors
#401
Comments
Do you know why is that? It's very strange. Do you have any links? |
I don't have any link, unfortunately! I tried printing the input request where the temperature was set to 0. |
Weird assumption: this framework relies on this additive in the denominator part and thus wants to use the lowest possible temperature so during sampling the logits with the highest probability will be outputted. |
thanks for finding this @carmocca! It seems that neurips client doesn't have this condition to the raw request - https://github.com/stanford-crfm/helm/pull/1693/files#diff-a9de0c80d20516c09d5131d26c59dfdab923a2c7cfd2e09b8814e9bc4bd7521eR47 |
stanford-crfm/helm#1693 will close this issue |
HELM evaluation framework tries to generate with
temperature=0
which leads toinf
tensors error.I don't have much idea how LLM will react to adding
1e-9
to the temperature value but this would avoid the error here.The text was updated successfully, but these errors were encountered: