Test Plan

[nibix]

The purpose of this issue is to define the areas that shall be considered in the integration tests and to track the progress.

Note: This will certainly require some adjustments during project progress.

General rules

In order to achieve a test suite that finishes in a reasonable time (less than 10 minutes), we will need to keep the number of different cluster configurations relatively low.

This will result in a couple of gaps in test coverage. We need to fill these with unit tests. This, however, will be done as a second step.

Scope

These tests shall only cover pure OpenSearch topics. Functionality related to OpenSearch Dashboards or other parts of the ecosystem is not covered.

Not in scope (TBD):

• OIDC
• SAML
• Multi-Tenancy

Test categories

OpenSearch Operations with do_not_fail_on_forbidden disabled

This category verifies that the OpenSearch operations remain available and functional with the activated security plugin.
The tests shall be authentication agnostic. By default these tests will be performed with users authenticated by basic
authentication against the internal user database.

Thus, this category should only require one cluster configuration.

Optional, subject for discussion: The test framework however will also provide the option to have the users authenticated via: Basic Auth with LDAP, JWT, Proxy Auth

The tests will only test operations via the REST API. We won't test via transport actions here, as the REST API bases on transport actions; thus it can be assumed that these are still available.

DNFOF is disabled for these tests. It will be tested in a separate category.

These tests are also used to test basic audit logging. For operations, which trigger logging events, the presence of log events is verified.

Tested Operations

• Search
  • simple
  • with index pattern
  • with aliases
  • _all
  • scrolling
  • aggregation
  • matrix stats aggregation
• Get
• MGet
• MSearch
• Field Caps
• Index
• Update
• Delete
• Bulk
• Reindex
• Alias creation
• Alias deletion
• Snapshot/Restore
• Templates

Tested properties

Each test case must verify:

• The status code, resp. return value of the operation
• The effectiveness of the operation. This means that further operations might be necessary to check whether the operation was effective.
• If the operation triggers audit logging events, the presence of these events is verified.

In order to verify correct authorization, at least one negative test for missing privileges will be performed for each operation.

OpenSearch Operations with do_not_fail_on_forbidden enabled

This category covers the our good old special configuration option whose name could be also the name of a German late 90s indie pop band.

It requires a cluster configuration where the Do Not Fail On Forbidden option is activated.

• DNFOF for empty results

Tested operations

DNFOF only affects a specific set of OpenSearch operations, which include search, field caps, etc.
Only these shall be tested in this environment.

• Search
  • simple
  • with index pattern
  • with aliases
  • _all
  • scrolling
  • aggregation
  • matrix stats aggregation
• MGet
• MSearch
• Field Caps
• Point in time operations
• IndicesAliasesRequest which deletes an index
• CreateIndexRequest which creates an alias

Tested properties

Each test case must verify:

• The status code, resp. return value of the operation
• The effectiveness of the operation. This means that further operations might be necessary to check whether the operation was effective.
• If the operation triggers audit logging events, the presence of these events is verified.

Security Configuration

This category validates dynamic updates on the security configuration.

Tested operations

• Correct authorization after config update
• Automatic initialization of default configuration
• Environment variable expansion
• Update of configuration via admin cert and security index access
• Update of configuration via admin tool
• Update of configuration via REST API

These tests are also used to test basic audit logging. For operations, which trigger logging events, the presence of log events is verified.

Authc Process

The authc process is the logic that is run for every request in order to extract and verify authentication information from it and to retrieve authorization information (i.e., roles).

Basic Auth

Internal user database

• Successful authentication
  • Username is correctly retrieved
  • Roles are correctly retrieved
  • Roles are correctly retrieved with recursive role search
  • Attributes are correctly retrieved
• 401 response when user does not exist
• 401 response when user does exist but password is wrong
• User impersonation

LDAP

• Successful authentication
  • Username is correctly retrieved
  • Roles are correctly retrieved
  • Roles are correctly retrieved with recursive role search
  • Attributes are correctly retrieved
• 401 response when user does not exist in LDAP
• 401 response when user does exist but password is wrong
• User impersonation with LDAP

JWT

• Successful authentication with signed JWT
  • Username is correctly retrieved
  • Roles are correctly retrieved
• 401 response when JWT with wrong signature is encountered
• 401 response when invalid JWT is encountered
• Unsuccessful authentication when expired JWT is encountered

Proxy

• Successful authentication
  • Username is correctly retrieved
  • Roles are correctly retrieved
• 401 response when headers do not exist

Client Certs

• Successful authentication
  • Username is correctly retrieved
• 401 response when cert is invalid

Kerberos

• Successful authentication
  • Username is correctly retrieved
  • Roles are correctly retrieved
  • Attributes are correctly retrieved
• 401 response when Authorization header is invalid

Anonymous

• Successful authentication
• Username is correctly retrieved
• Roles are correctly retrieved

Misc

• Blocked users cannot login
• Login failure rate limiting is effective

DLS/FLS

In this test category, we will test effectiveness for DLS/FLS/field masking for these operations:

• Search
  • simple
  • with index pattern
  • with aliases
  • with filtered aliases
  • _all
  • scrolling
  • aggregation
  • matrix stats aggregation
• Get
• MGet
• MSearch
• Field Caps

The tests will only run with users authenticated via basic auth on the internal user database.

CCS

CCS (Cross Cluster Search) is only applicable for search operations. We need to test the authorization is properly performed both on the remote cluster and on the local cluster.

• Index based authorization for remote indices
• DLS/FLS authorization for remote indicies

Additionally, two more dimensions need to be considered:

• Search requests which combine local indices and remote indices.
• ccs_minimize_roundtrips: false. By default, ccs_minimize_roundtrips is true. We also need to perform the tests with ccs_minimize_roundtrips: false. See https://opendistro.github.io/for-elasticsearch-docs/docs/elasticsearch/rest-api-reference/ for docs.

TLS

All other test categories also operate on TLS encrypted operation. Thus, basic operation is already quite well covered.
This category covers primarily negative cases and special configurations.

Negative cases include:

• Plaintext connections are rejected and logged to audit log

Special configurations include:

• ssl_only
• Custom cipher configs

Misc

• SecurityHealthAction: /_opendistro/_security