Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Property-based unit tests #662

Open
lsf37 opened this issue Dec 8, 2019 · 5 comments
Open

Property-based unit tests #662

lsf37 opened this issue Dec 8, 2019 · 5 comments
Assignees
Labels
task Process tasks testing Adding tests or test infrastructure.

Comments

@lsf37
Copy link
Member

lsf37 commented Dec 8, 2019

The current unit testing in JFlex is a bit thin, mostly because the cost/benefit ratio of writing those single test cases is fairly low.

We should use property-based unit tests, at least for the different set implementations in JFlex, as well as partitions and other mathematical objects that have nicely statable properties. These would automate the test object construction, and let us specify the contracts for those objects in a way that gains some actual assurance about implementation correctness.

juni-quickcheck (https://github.com/pholser/junit-quickcheck) looks like it would be usable for the job.

@lsf37 lsf37 added task Process tasks testing Adding tests or test infrastructure. labels Dec 8, 2019
@lsf37 lsf37 self-assigned this Dec 8, 2019
@lsf37
Copy link
Member Author

lsf37 commented Dec 8, 2019

I'll have a look at how much work it would be to set up. @regisd, @sarowe do you have any opinions on this?

lsf37 added a commit that referenced this issue Dec 8, 2019
includes generator for IntCharSet; addresses #662
lsf37 added a commit that referenced this issue Dec 8, 2019
includes generator for IntCharSet; addresses #662
@regisd
Copy link
Member

regisd commented Dec 8, 2019

I discovered this kind of testing when I learned scala, and liked it, but never used it in real life.

I'm a bit uncertain about the randomness of things ; as in can this make tests flaky?

regisd pushed a commit that referenced this issue Dec 8, 2019
* quickcheck/property tests for IntCharSet

includes generator for IntCharSet; addresses #662

* add junit-quickcheck bazel dependencies
* add junit-quickcheck pom dependencies

* IntCharSet: add debug assertions on add/sub

Sort-of obvious, but would be nasty if violated. Discovered in quickcheck property testing.
regisd pushed a commit that referenced this issue Dec 8, 2019
commit 6835dda
Author:     Gerwin Klein <[email protected]>
AuthorDate: Mon Dec 9 00:12:40 2019 +1030
Commit:     Régis Décamps <[email protected]>
CommitDate: Sun Dec 8 14:42:40 2019 +0100

    Quickcheck (#663)

    * quickcheck/property tests for IntCharSet

    includes generator for IntCharSet; addresses #662

    * add junit-quickcheck bazel dependencies
    * add junit-quickcheck pom dependencies

    * IntCharSet: add debug assertions on add/sub

    Sort-of obvious, but would be nasty if violated. Discovered in quickcheck property testing.

Updated from target/jflex-parent-1.8.0-SNAPSHOT-sources.jar
@lsf37
Copy link
Member Author

lsf37 commented Dec 9, 2019

In theory the randomness could lead to a test that does not always fail, in practice that happens rarely, although it does happen. How often it happens depends on the number of tests it runs for each checked property (default is 100), and how well the input space is covered in that number of tests. So far, if something was wrong, it failed pretty much always, but it's not guaranteed.

When a test fails, it shows the inputs and random seeds for which it failed, so it can be reproduced. It's usually a good idea to then turn that failing instance into a unit test for reliably checking regressions. E.g. for IntCharSet, you'd take the parameters it shows, construct them manually in the old IntCharSetTest unit test, but call the IntCharSetProperties method that failed with it, i.e. you don't have to duplicate the actual test code. This will unit test will now always fail until the problem is fixed.

@lsf37
Copy link
Member Author

lsf37 commented Dec 11, 2019

It seems to be working fine, and it's at least for me way easier to find good tests to write, so I'll continue working on this a bit and add tests for state sets and char classes.

@lsf37
Copy link
Member Author

lsf37 commented Dec 16, 2019

#687 bring this to reasonably comprehensive tests for IntCharSet, StateSet, and CharClasses. I'll see if it make sense to do something on RegExp next, and if that is successful, on NFA and DFA.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
task Process tasks testing Adding tests or test infrastructure.
Projects
None yet
Development

No branches or pull requests

3 participants