Test Framework #299

nhtruong · 2024-05-21T23:19:52Z

This PR adds a custom test framework for OpenSearch Open API Spec. Each YAML file in the ./tests folder (and its sub-folders) is a test story with 3 main components:

prologues: An array of requests made to the OpenSearch cluster to prepare it for the test (e.g. create and index and a few docs to test the search operations)
chapters: An array of requests, run sequentially, that actually test the spec:
- Test if the operation exists in the spec
- Test the provided parameters against their schemas in the spec
- Test the provided request body against its schema in the spec
- Test if the response code matches the expected response code
- Test the response payload against its schema in the spec
epilogues: An array of requests co clean up the test environment (e.g. delete all indices and documents created during the test)

A JSON Schema for the test stories is also provided so that the IDE can help developers with autocomplete and validation when drafting the stories.

Make sure you have an OpenSearch cluster running at http://localhost:9200 before running the following command to initiate the test suite

npm run test:spec

Bellow is the result of running npm run test:spec -- --ignore_passed N
Check start.ts to see the options that can be passed to npm run test:spec command.

Remaining work:

Grab server URL from environment variable and add authentication.
Add a github workflow
Update repo docs
Add tests fro the framework (heavy lift)
Test should fail when a field is found in the response but not defined in the spec

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

github-actions · 2024-05-21T23:21:25Z

API specs implemented for 313/649 (48%) APIs.
Commit 39bb904.

dblock

This looks good overall! Needs READMEs, tests, and all the TODOs.

Left small usage/naming comments.

Try to get something that runs under CI first in.

json_schemas/story.schema.yaml

tests/index_lifecycle.yaml

tools/src/tester/start.ts

json_schemas/story.schema.yaml

tools/src/tester/helpers.ts

tools/src/tester/ChapterEvaluator.ts

tools/src/tester/StoryEvaluator.ts

tools/src/tester/ResultsDisplayer.ts

tools/src/tester/TestsRunner.ts

dblock

Tested this out.

tools/src/tester/ChapterReader.ts

tools/src/tester/start.ts

tests/index_lifecycle.yaml

github-actions · 2024-05-27T19:25:22Z

Changes Analysis

Commit SHA: 078326f
Comparing To SHA: c58908d

API Changes

Summary


├─┬Paths
│ └─┬/{index}
│   └─┬HEAD
│     └─┬Responses
│       └──[➕] codes (23760:7)
└─┬Components
  └──[➕] responses (23760:7)

Document Element	Total Changes	Breaking Changes
paths	1	0
components	1	0

Total Changes: 2
Additions: 2

Report

The full API changes report is available at: https://github.com/opensearch-project/opensearch-api-specification/actions/runs/9291588290/artifacts/1549682775

API Coverage

	Before	After	Δ
Covered (%)	476 (46.62 %)	476 (46.62 %)	0 (0 %)
Uncovered (%)	545 (53.38 %)	545 (53.38 %)	0 (0 %)
Unknown	24	24	0

json_schemas/test_story.schema.yaml

Signed-off-by: Theo Truong <[email protected]>

nhtruong · 2024-05-29T18:57:52Z

This is ready for the view and merged so people can start adding tests. I'll handle the remaining work in followup PRs:

Update repo docs
Add tests for the framework (heavy lift)
Additional Feature: Test should fail when a field is found in the response but not defined in the spec

dblock

This LGTM! @Xtansia any objections to merging as is?

Xtansia

There's still some parts I think could be improved, but not against merging as is to iterate on it

Xtansia · 2024-05-29T21:43:43Z

tools/src/tester/start.ts

+  .addOption(new Option('--spec, --spec_path <path>', 'path to the root folder of the multi-file spec').default('./spec'))
+  .addOption(new Option('--tests, --tests_path <path>', 'path to the root folder of the tests').default('./tests'))
+  .addOption(new Option('--tab_size <size>', 'tab size for displayed results').default('4'))


CLI args are generally kebab-case. Additionally I'd say it's more commonly referred to as a tab width than a tab size.

THanks for the callout. Will update them in the next PR

#301 with tests

Xtansia · 2024-05-29T21:45:51Z

tools/src/tester/helpers.ts

+export function overall_result (evaluations: Evaluation[]): Result {
+  if (evaluations.some(e => e.result === Result.ERROR)) return Result.ERROR
+  if (evaluations.some(e => e.result === Result.FAILED)) return Result.FAILED
+  if (evaluations.some(e => e.result === Result.SKIPPED)) return Result.SKIPPED


Is this correct? If I have 24 results, 23 passed but 1 skipped, then the overall result is skipped?

This is for evaluating components results. A component is skipped when a previous component results in an error (say if a chapter could not be executed for validation, all subsequent chapters will be skipped). However that would result in ERROR for overall. So for now line 6 is actually redundant.

Hypothetically when we allow the user to skip some component evaluation (say skip evaluating the response body of a new endpoint still being developed), I think we should still mark the parent component as skipped. The print out will show all subcomponents so you can tell which subcomponent causes the SKIPPED status for the parent component. So:

If you see ERROR as the result for a component, at least one of its subcomponent has ERROR

otherwise, If you see FAILED as the result for a component, at least one of its subcomponent has FAILED

otherwise, If you see SKIPPED as the result for a component, at least one of its subcomponent has SKIPPED

else, all components have been vested and passed the test.

nhtruong force-pushed the test_framework branch from c22f6e6 to 9d0b528 Compare May 21, 2024 23:22

dblock reviewed May 22, 2024

View reviewed changes

Xtansia reviewed May 23, 2024

View reviewed changes

dblock reviewed May 23, 2024

View reviewed changes

nhtruong force-pushed the test_framework branch from c566451 to 84756df Compare May 27, 2024 19:23

nhtruong force-pushed the test_framework branch from a60b73d to 0e472c4 Compare May 29, 2024 15:28

dblock reviewed May 29, 2024

View reviewed changes

json_schemas/test_story.schema.yaml Outdated Show resolved Hide resolved

Spec Test Framework

078326f

Signed-off-by: Theo Truong <[email protected]>

nhtruong force-pushed the test_framework branch from c22e52a to 078326f Compare May 29, 2024 18:54

nhtruong marked this pull request as ready for review May 29, 2024 18:57

nhtruong requested review from harshavamsi, sachetalva, VachaShah, Tokesh and aabeshov as code owners May 29, 2024 18:57

dblock approved these changes May 29, 2024

View reviewed changes

Xtansia reviewed May 29, 2024

View reviewed changes

Xtansia merged commit 69247b0 into opensearch-project:main May 29, 2024
7 checks passed

dblock mentioned this pull request May 30, 2024

Added testing to the generated OpenAPI. #286

Closed

nhtruong deleted the test_framework branch May 30, 2024 18:40

This was referenced May 30, 2024

Renaming parameters and adding command line tests. #301

Merged

Add text embedding processor #304

Merged

[PROPOSAL] Add integration tests #277

Closed

Adds API spec for security plugin #271

Merged

DarshitChanpura mentioned this pull request Jun 5, 2024

Add tests for the new API spec opensearch-project/security#4404

Closed

nhtruong mentioned this pull request Jun 26, 2024

Added StoryValidator #362

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Test Framework #299

Test Framework #299

nhtruong commented May 21, 2024 •

edited

Loading

github-actions bot commented May 21, 2024 •

edited

Loading

dblock left a comment

dblock left a comment

github-actions bot commented May 27, 2024 •

edited

Loading

nhtruong commented May 29, 2024 •

edited

Loading

dblock left a comment

Xtansia left a comment

Xtansia May 29, 2024

nhtruong May 30, 2024

dblock May 30, 2024

Xtansia May 29, 2024

nhtruong May 30, 2024

Test Framework #299

Test Framework #299

Conversation

nhtruong commented May 21, 2024 • edited Loading

github-actions bot commented May 21, 2024 • edited Loading

dblock left a comment

Choose a reason for hiding this comment

dblock left a comment

Choose a reason for hiding this comment

github-actions bot commented May 27, 2024 • edited Loading

Changes Analysis

API Changes

Summary

Report

API Coverage

nhtruong commented May 29, 2024 • edited Loading

dblock left a comment

Choose a reason for hiding this comment

Xtansia left a comment

Choose a reason for hiding this comment

Xtansia May 29, 2024

Choose a reason for hiding this comment

nhtruong May 30, 2024

Choose a reason for hiding this comment

dblock May 30, 2024

Choose a reason for hiding this comment

Xtansia May 29, 2024

Choose a reason for hiding this comment

nhtruong May 30, 2024

Choose a reason for hiding this comment

nhtruong commented May 21, 2024 •

edited

Loading

github-actions bot commented May 21, 2024 •

edited

Loading

github-actions bot commented May 27, 2024 •

edited

Loading

nhtruong commented May 29, 2024 •

edited

Loading