smoke tests: parallelization

[midleman]

Intent

The goal is to optimize our E2E smoke tests, which currently run serially, by parallelizing them to reduce overall execution time.

Approach

Initially, all tests were run from a single file (main.ts), which handled shared dependencies but prevented parallel execution. To address this, I split the tests into multiple "main" files, which improved execution times but made it challenging to balance workloads effectively. Ultimately, the best solution was to refactor each test to run independently, allowing for true parallelization.

Because mocha-junit-reporter doesn’t support parallel runs, I switched to mocha-xunit-reporter, which natively handles parallel execution. After the tests complete, I convert the results to the junit format to ensure compatibility with TestRail.

Impact on Workflow

1. ❌ Previously, when adding new test specs, we needed to manually import and include them in the main.ts file. We no longer need to do this! 🎉

2. ❌ We also no longer need to wrap all tests in setup().

3. ✅ The only thing we need to do is add setupEnvAndHooks(); (previously installAllHandlers) in the parent describe block of the test

How to Run in Parallel

Use the --parallel flag and specify --jobs:

yarn smoketest-no-compile --tracing --parallel --jobs 2

GHA Workflow Updates

I performed an analysis on run time for the full and pr test suites with varying concurrent jobs. Summary of my findings:

Full Test Suite (avg time)

• 1 machine: ~25 minutes
• 2 machines: ~11 minutes
• 3 machines: ~9.5 minutes

PR Test Suite (avg time)

• 1 machine: ~6 minutes
• 2 machines: ~3 minutes
• 3 machines: ~2.5 minutes

To promote stability and prevent constrained resources, I set GHA workflows to use 2 jobs. This is still a major speed improvement.

QA Validation

• Confirmed that --tracing works as expected and all expected logs exist.
• Verified that results (pass, fail, skip) are correctly imported into TestRail. See example.
• Skipping data-explorer-100x100.test.ts - I think it's just a performance issue and we can do a quick follow to look into the test.

[midleman]

this test has been flakey. and on retry it can't create a new project bc project name already exists - so i appended a random # at end to avoid that conflict.