Skip to content
This repository has been archived by the owner on Sep 12, 2023. It is now read-only.

Fail message shouldn't contain answer for hashed tests #18

Open
trevorcampbell opened this issue Jun 10, 2022 · 1 comment
Open

Fail message shouldn't contain answer for hashed tests #18

trevorcampbell opened this issue Jun 10, 2022 · 1 comment
Labels
bug Something isn't working

Comments

@trevorcampbell
Copy link
Contributor

Whenever the test is hashed, we should make sure to avoid the fail message saying things like

"Error: the value is not 8"

because of course that leaks the answer. Should perhaps replace with

"Error: the value is not [hidden answer!]" or something like that.

One twist: for things like type checking, we may want to avoid doing the above. e.g. if someone has the answer

a = np.float64(3)

and the test returns "Fail, the type is not float", that is useful information to help someone answer the question such that the tests pass... then again, maybe testing comparable types is a separate issue...

@trevorcampbell trevorcampbell added the bug Something isn't working label Jun 10, 2022
@trevorcampbell
Copy link
Contributor Author

more thinking here -- take a look at the example

    int:
        - test: "type({{snippet}})"
          fail: "type is not int"

        - test: "{{snippet}}"
          fail: "value is not correct"

AutoTest will leak the type check for hashed tests (which I find OK) but won't leak the value (also good). So maybe this isn't an issue yet...let's leave this open for now. At least it's something to include in documentation at some point.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant