Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Speed up UI tests #342

Merged
merged 17 commits into from
Sep 26, 2024
Merged

Speed up UI tests #342

merged 17 commits into from
Sep 26, 2024

Conversation

LukeMathWalker
Copy link
Owner

@LukeMathWalker LukeMathWalker commented Sep 22, 2024

We currently have ~140 different test projects, with the fastest platform taking almost 10 minutes to run the whole UI test suite. That's not sustainable.
This PR significantly restructures pavex_test_runner, the custom test harness used to write UI tests for Pavex.

The main obstacle is cargo's locking system—each test needs to compile some crates (the generated app, sometimes an integration test) as well as generate some JSON documentation via rustdoc. All these activities compete for the lock.

This PR rearchitects pavex_test_runner to minimise lock contention and maximise core usage.
In particular:

  • All JSON docs are generated upfront, in batch, ensuring that each test hits the SQLite cache rather than invoking cargo directly, thus avoid contention;
  • All UI tests now live in the same (separate) workspace, under ui_tests. They share a single target folder and their features are unified via a dedicated workspace hack crate. This ensures that we get full cache re-use across tests.
  • All compilation jobs (e.g. generated apps or integration tests) are batched to maximise core utilisation

The speed-up has been substantial. In CI, we now run UI tests on Linux in ~7m compared to the ~11m we were seeing on main. The difference is even more extreme on Windows: from 16m to 9m. We also scored further wins there, by carving out some particularly slow tests, saving 5 minutes on executing unit tests.

Some leftover work to be done in followup PRs:

  • Fix all warnings in UI test code
  • Ensure that all test_config.toml files are only using valid keys
  • Enforce programmatically the crate naming structure ({prefix}_{hash}) to avoid divergence over time
  • Add utilities to generate the scaffolding for a new test

@LukeMathWalker
Copy link
Owner Author

/ok-to-test sha=7ea7aa7

@LukeMathWalker
Copy link
Owner Author

/ok-to-test sha=e248d9d

@LukeMathWalker
Copy link
Owner Author

/ok-to-test sha=e617916

@LukeMathWalker
Copy link
Owner Author

/ok-to-test sha=a14b195

@LukeMathWalker
Copy link
Owner Author

/ok-to-test sha=07a0248

@LukeMathWalker
Copy link
Owner Author

/ok-to-test sha=0bef47b

@LukeMathWalker LukeMathWalker marked this pull request as ready for review September 26, 2024 13:28
@LukeMathWalker LukeMathWalker merged commit c6f10c5 into main Sep 26, 2024
22 checks passed
@LukeMathWalker LukeMathWalker deleted the faster-tests branch September 26, 2024 13:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant