You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
This repository has been archived by the owner on Sep 12, 2023. It is now read-only.
Whenever the test is hashed, we should make sure to avoid the fail message saying things like
"Error: the value is not 8"
because of course that leaks the answer. Should perhaps replace with
"Error: the value is not [hidden answer!]" or something like that.
One twist: for things like type checking, we may want to avoid doing the above. e.g. if someone has the answer
a = np.float64(3)
and the test returns "Fail, the type is not float", that is useful information to help someone answer the question such that the tests pass... then again, maybe testing comparable types is a separate issue...
The text was updated successfully, but these errors were encountered:
int:
- test: "type({{snippet}})"
fail: "type is not int"
- test: "{{snippet}}"
fail: "value is not correct"
AutoTest will leak the type check for hashed tests (which I find OK) but won't leak the value (also good). So maybe this isn't an issue yet...let's leave this open for now. At least it's something to include in documentation at some point.
Sign up for freeto subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Whenever the test is hashed, we should make sure to avoid the fail message saying things like
"Error: the value is not 8"
because of course that leaks the answer. Should perhaps replace with
"Error: the value is not [hidden answer!]" or something like that.
One twist: for things like type checking, we may want to avoid doing the above. e.g. if someone has the answer
and the test returns "Fail, the type is not
float
", that is useful information to help someone answer the question such that the tests pass... then again, maybe testing comparable types is a separate issue...The text was updated successfully, but these errors were encountered: