Make TestsRunner.test.ts and TestsRunner.test.ts independent.

[dblock]

Description

If you run npm run test the two test suites will conflict as they both create/delete the same indices (books). Split the fixtures, called one books and the other shoes.

Use beforeAll to avoid global setup.

Renamed tools/tests/tester/overall_result.test.ts to helpers.test.ts to match the file being tested.

Removes the need for --runInBand, runs faster and with defaults.

The only downside is duplication of the YAML tests.

Issues Resolved

Part of #313.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

Changes Analysis

Commit SHA: 350f4c0
Comparing To SHA: 6711e8f

API Changes

Summary

NO CHANGES

Report

The full API changes report is available at: https://github.com/opensearch-project/opensearch-api-specification/actions/runs/9391821068/artifacts/1572914731

API Coverage

	Before	After	Δ
Covered (%)	476 (46.62 %)	476 (46.62 %)	0 (0 %)
Uncovered (%)	545 (53.38 %)	545 (53.38 %)	0 (0 %)
Unknown	24	24	0

[nhtruong]

Did you create books and shoes folders to circumvent the need for --runInBand where each set creates and deletes a different index? While that will solve this particular set of integ tests, as we add more integ tests in the future, the tests can interfere with each other unpredictably and result in flaky tests. This adds extra complexity for test writers as we will have to be sure that all resources created in all tests are mutually exclusive.

[dblock]

Did you create books and shoes folders to circumvent the need for --runInBand where each set creates and deletes a different index? While that will solve this particular set of integ tests, as we add more integ tests in the future, the tests can interfere with each other unpredictably and result in flaky tests. This adds extra complexity for test writers as we will have to be sure that all resources created in all tests are mutually exclusive.

Yes. I come from the point of view that tests should be able to run in parallel and not be interfering with each-other. But I understand why integration tests may be different. Would you prefer a solution where we make these tests "integration" and run them with --runInBand?

[nhtruong]

Yes. I think running integration tests with --runInBand is the way to go. We can divide the tools/tests folder into:

• tests/integ/tester
• tests/unit/tester
• tests/unit/merger
• tests/unit/linter

and have 2 npm run commands

• test:tools:unit --> jest tools/tests/unit
• test:tools:integ --> jest tools/tests/integ --runInBand

[dblock]

Closing in favor of #320.