test.py is a simple regression testing harness shipped along with
scylla.git, which runs C++ unit and CQL tests.
This is a manual for test.py.
To run test.py, Python 3.7 or higher is required.
./install-dependencies.sh should install all the required Python
modules. If install-dependencies.sh does not support your distribution,
please manually install all Python modules it lists with pip.
In order to invoke test.py, you need to build first. ./test.py will
run all existing tests in all configured build modes:
$ ./test.py
If you want to specify a specific build mode:
$ ./test.py --mode=dev
If you want to run only a specific test:
$ ./test.py suitename/testname
Build artefacts, such as test output and harness output is stored
in ./testlog. Scylla data files are stored in /tmp.
On start, test.py invokes ninja to find out configured build modes. Then
it searches all subdirectories of ./test/ for suite.yaml files: each
directory containing suite.yaml is a test suite, in which test.py then looks
for tests. All files ending with _test.cc or _test.cql are considered
tests.
A suite must contain tests of the same type, as configured in suite.yaml.
The list of found tests is matched with the optional command line test name
filter. A match is registered if filter substring exists anywhere in test
full name. For example:
$ ./test.py cql
runs cql/lwt_test, cql/lwt_batch_test, as well as
boost/cql_query_test.
The ./testlog directory is created if it doesn't exist, otherwise it is
cleared from the previous run artefacts.
Matched tests are run concurrently, with concurrency factor set to the
number of available CPU cores. test.py continues until all tests are run
even if any one of them fails.
To run CQL tests, test.py uses an auxiliary program, test/tools/cql_repl.cc.
This program reads CQL input from stdin, evaluates it using Scylla CQL
database front-end, and prints output in JSON format to stdout. A default
keyspace is created automatically.
test.py invokes cql_repl providing the test file for its standard
input and redirecting its output to a temporary file in testlog directory.
After cql_repl finishes, test.py compares the output stored in the
temporary file with a pre-recorded output stored in
test/suitename/testname_test.result.
The test is considered failed if executing any CQL statement produced an
error (e.g. because the server crashed during execution) or server output
does not match one recorded in testname.result, or there is no
testname.result. The latter is possible when it's the first invocation of
the test ever.
In the event of output mismatch file test/suitename/testname_test.reject
is created, and first lines of the diff between the two files are output.
To update .result file with new output, simply overwrite it with the
reject file:
mv test/suitename/testname.re*
Note Since the result file is effectively part of the test, developers
must thorougly examine the diff and understand the reasons for every
change before overwriting .result files.
The same unit test can be run in different seastar configurations, i.e. with
different command line arguments. The custom arguments can be set in
custom_args key of the suite.yaml file.
If a test fails, its log can be found in testlog/${mode}/testname.log.
By default, all unit tests are built stripped. To build non-stripped tests,
./configure with --tests-debuginfo list-of-tests.
test.py adds some command line arguments to unit tests. The exact way in
which the test is invoked is recorded in testlog/test.py.log.
To debug CQL tests, one can invoke gdb cql_repl, which will start
cql_repl in interactive mode, and then supply faulty CQL statements
on the terminal.
If any of the tests fails, test.py returns a non-zero exit status.
JUNIT and XUNIT execution status XML files can be found in
testlog/${mode}/xml/ directory.
For command line help and available options, please see also:
$ ./test.py --help