Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Grow size faster with a small number of tests #474

Open
wants to merge 2 commits into
base: master
Choose a base branch
from

Conversation

ChickenProp
Copy link
Contributor

@ChickenProp ChickenProp commented Jan 30, 2023

(Oops, intended to make this PR targeting my fork. Now that it's here might as well keep it.)

As described in #472. We now grow test size in fixed increments from 0 to 99, or as close as we can without going over, in however many tests we run. If we run over n * 100 tests, then we go from 0 to 99 n times, and then do fixed increments for the remainder. Additionally, if we discard a bunch of times in a row we start to grow the size.

I've removed recheck because "specify the starting size" doesn't really make sense with this, and that function is no longer suggested by failure reports. If you want to keep it for backwards compatibility we can figure something out.

It's not useful with `recheckAt`, and makes fast growth more awkward to
implement. This might break compatibility more than upstream wants, but
at this point I've mostly given up on getting things merged there. If
they do want fast growth we can figure out something.
@ChickenProp
Copy link
Contributor Author

So the exact implementation here is incompatible with #489, because it means the exact sequence of tests and discards is now relevant. Currently it affects neither the seed nor the size that we run the next test at, but with this PR it can affect the size. If we've skipped 10 failing tests and 10 discards, we'll be running one size if the sequence was (T×10, D×10) and a different size if it was anything else.

A few possible approaches:

  1. Save the exact sequence of tests and discards in the skip. This would make the compressed string longer and harder to read, but maybe that's fine. Might be the most robust to future changes, allowing the exact sequence to affect both seed and size.
  2. Save the final size in the skip (maybe testCount[/discardCount]@size[:path]). Still relies on the exact sequence not changing seed, but allows it to change size.
  3. Save the number of recent discards in the skip (maybe testCount[/discardCount[+recentDiscardCount]][:path]). Makes assumptions about what features of the sequence can change the size.
  4. Pass calculateSize the number of total discards, and stop tracking the number of recent discards. Instead of loosening the "exact sequence doesn't affect size" assumption, we simply make that assumption true again. This would make size monotonic (except for resetting to 0 every 100 tests), which the current implementation isn't.

All of these seem basically okay, except I think (3) doesn't really have anything to recommend it over (2). If this feature is wanted, I can implement any of them; my recommendation would be (2) or maybe (4). If it's not, then I'll probably implement (4) on my fork next time I merge master, just because it's easiest.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant