Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
ci: Daily CI to soak flaky tests (#5382)
### Description Setting up a daily scheduled build to run the LSP tests 5 times on each platform, in an effort to proactively uncover flaky test failures that show up at the worst possible time as you're trying to get your wonderful PR you've been working on for weeks merged or even worse that break the nightly build for the third day in a row forcing the team to drop everything to click Retry Failed Jobs two or three times before ANY PRs can be merged not that I'm frustrated or anything. :) The PR CI was previously running the LSP tests twice every time for the same reason. Given this daily job will soak them more deeply, and that the `osx` unit test job on PRs in particular has been taking almost 45 minutes and becoming the limiting factor, I've opted to just run them once on PRs now. Note that unlike the original double run, it was simpler to use matrices just as we do for integration tests, only in this case just iterating the same test run in parallel 5 times. It's possible running twice on the same runner might trigger more failures, but it seems unlikely since the second run happens in a fresh process and no state leaks between the runs AFAICT. ### How has this been tested? Dry-run on my fork: https://github.com/robin-aws/dafny/actions/runs/8914692810 I've run it a few times without any failures yet, but we'll have to see what happens after a week or so of scheduled runs. <small>By submitting this pull request, I confirm that my contribution is made under the terms of the [MIT license](https://github.com/dafny-lang/dafny/blob/master/LICENSE.txt).</small>
- Loading branch information