More options to limit / control concurrency

[hbgl]

I am running tests that perform relatively heavy weight tasks like rendering pages in headless chrome. My tests also need to complete within a certain time window or they are considered failed. With the standard ava execution strategy, test files are run in parallel and all asynchronous tests within each test file are executed concurrently. So when ava is started it's like a massive stampede of tests that hits the machine. This causes tests to run much slower than they otherwise would.

I have a proposal for two new features that would help a lot in this situation:

1. Serial test files
2. Limit concurrent async tasks per test file

1) Serial test files

Ava has the feature to mark individual test cases as serial, which causes them to be run serially before all other test cases. This new proposed feature would work the same way but for test files instead, causing them to be run serially before all other test files.

My current ugly workaround is to run multiple instances of ava:

npx ava serial_file1.spec.ts && \
npx ava serial_file2.spec.ts && \
npx ava all.spec.ts other.spec.ts files.spec.ts

2) Limit concurrent async tasks per test file

Ava executes all asynchronous test cases within a single test file concurrently. So if I have 100 test cases in a file and each test case opens a tab in headless chrome, then I have 100 tabs open which puts a lot of strain on the system. The new proposed feature would limit the number of test cases that are executed concurrently per test file.

My current workaround is to use a custom semaphore implementation to throttle test cases:

import test from 'ava';
import { Semaphore } from './somewhere';

const semaphore = new Semaphore({ maxConcurrency: 4 });

test.beforeEach(async () => {
  await semaphore.enter();
});

test.afterEach.always(() => {
  semaphore.exit();
});

// This runs 100 test cases but only at most 4 at a time.
for (let i = 0; i< 100; i++) {
  test(`test #${i}`, async t => {
    await doWhatNeedsToBeDone();
    t.pass();
  });
}

It's not pretty but workable.

---

I would like to know from you if you think these features would be worth adding to ava.

[novemberborn]

The recommended approach is to limit concurrency (e.g. -c 1 on the CLI). Within tests, while you need to declare them all synchronously, you could use a semaphore to schedule execution, and you could do that through a macro (e.g. test('test 1', max(5), async t => {}) where max(5) returns a function that receives the async t () callback and can take care of scheduling.

[tommy-mitchell]

Using @shopify/semaphore with the environmentVariables option is a pretty lightweight solution to this:

// package.json

"ava": {
	"environmentVariables": {
		"concurrency": "10"
	}
}

// test.ts

import anyTest, { type TestFn } from "ava";
import { Semaphore, type Permit } from "@shopify/semaphore";

const test = anyTest as TestFn<{
	permit: Permit;
}>;

const semaphore = new Semaphore(Number(process.env.concurrency) || 5);

test.beforeEach("setup concurrency", async t => {
	t.context.permit = await semaphore.acquire();
});

test.afterEach.always(async t => {
	await t.context.permit.release();
});

// ...

[novemberborn]

Using https://github.com/avajs/cooperate#semaphores you can do this across worker threads.

[tommy-mitchell]

Should we create a recipe for using these?

[novemberborn]

Sounds good!

[titanism]

@tommy-mitchell did you create a recipe for this somewhere? not seeing it on this GH or in any PR's

[tommy-mitchell]

@titanism no I never did 😅 are you having any issues with the setup I showed?