How to: tests

Testing the code

This guide's aim is to provide instructions and clarifications regarding making and using tests for DKB. Therefore, it is presumed that general questions, such as "What is testing?" or "Why do we need to make tests?", are already answered. See Further reading if some of them are not.

For now, DKB uses standard unittest module for Python testing.

It should be noted that working with DKB tests means adhering to rules of two sets:

1. Unittest module's guidelines and requirements.
2. DKB's own decisions regarding tests.

This guide is describing the second set of rules which is more strict than the first one. For example, unittest states that files with tests can be discovered as long as their names follow a pattern which is 'test*.py' by default (but can be changed), while it was decided that DKB's test files' names should start with "test_" and must correlate with code files' names. The first set of rules can be found in unittest's documentation.

Running tests

Each module should be tested in a similar way – running python -m unittest discover command from the module directory. Possibility of doing so relies on meeting some requirements regarding names and other things – following this guide's rules should be sufficient to make discover work properly.

Directory structure

Each stage should have a directory called tests. Test files should be placed in this directory, and their names should consist of two parts: name of the code file they are testing, and indication of what they are testing. For example, if stage consists of one script stage123.py, file with tests for its functions can be called test_stage123_functions.py, while file with tests for specific function func1() can be called test_stage123_func1.py. Apart from the test files, the directory should also have an __init__.py file (empty one will do), because the tests will not be detected otherwise.

In the future it is also planned to make stage-level tests – tests which ensure the correct implementation of the whole stage. These tests should be provided with sample input and output data, and should check the stage's ability to correctly obtain the latter by processing the former. How exactly these tests and data should be named and handled is yet to be decided – this guide will be updated accordingly.

pyDKB is a special case. Planned layout of pyDKB directories is as follows:

Utils/Dataflow/
        |-- pyDKB/
               |-- src/
               |   |-- pyDKB/
               |   |-- tests/
               |
               |-- setup.py

Writing tests

Apart from imports and other standard things, a test file should include two constructions: test case and test suite setup.

Test case

A test case must be a subclass of unittest.TestCase and should contain test methods and optional utility methods setUp() and tearDown(). It was decided to put each test case into a separate file - therefore, each case should be simply called Case, since the filename will be used to identify the case.

A test method's name must start with "test" (to be found with discover) and be as informative as possible (it won't be called by hand too often, but will be displayed when something goes wrong). A test method's goal is to ensure that either something is correct:

def test_sum(self):
    s = my_func_sum(2, 2)
    self.assertEqual(s, 4)

or an exception is raised:

def test_divide_by_zero(self):
    with self.assertRaises(ZeroDivisionError):
        a = 1 / 0

See unittest's documentation for a basic list of assert functions as well as several additional ones.

Sometimes several tests are performing identical actions before/after doing something. Consider the following code:

def test1(self):
    data = {… very complex combination of variables …}
    r = some_function(data)
    self.assertEqual(r, desirable_result)
def test2(self):
    data = {… same data as above …}
    r = some_other_function(data)
    self.assertEqual(r, other_desirable_result)

Such structure means that the data definition must be written twice (or copied) and both instances must be updated when a change occurs. This can be simplified by setUp() method which is executed before every test. In a similar way, tearDown() method runs after every test. Using these methods, the code above can be rewritten:

def setUp(self):
    self.data = {… very complex combination of variables …}
def tearDown(self):
    self.data = None
def test1(self):
    r = some_function(self.data)
    self.assertEqual(r, desirable_result)
def test2(self):
    r = some_other_function(self.data)
    self.assertEqual(r, other_desirable_result)

Note the "cleanup" of used self.data in tearDown().

Test suite setup

Test suite is used to make tests discover-able. It can be more or less copypasted from existing tests:

def load_tests(loader, tests, pattern):
    suite = unittest.TestSuite()
    suite.addTest(loader.loadTestsFromTestCase(Case))
    return suite

Adding tests in a cycle

Sometimes it is necessary to make a large amount of similar tests in such way that setUp() and tearDown() will not be of much use. Let's take a look at the following simple function:

def sum_function(a, b):
    return a + b

If we, for some reason, need to test this function on a large set of data, writing the tests by hand will be ineffective:

class Case(unittest.TestCase):
    def test_one_plus_one_equals_two(self):
        self.assertEqual(sum_function(1, 1), 2)
    def test_two_plus_two_equals_four(self):
        self.assertEqual(sum_function(2, 2), 4)
...

Apart from the effort spent on writing or copy-pasting pretty much the same thing, another problem is the fact that if we change the set of data, we also have to remember about these tests and change them as well.

It is possible to define class methods in a cycle in Python, which can be used to simplify our task:

class Case(unittest.TestCase): pass


test_data = [
        (1, 1, 2),
        (2, 2, 4),
        (3, 5, 8),
    ]


def add_sum_test(a, b, s):
    def f(self):
        self.assertEqual(sum_function(a, b), s)
    setattr(Case, 'test_%d_plus_%d_equals_%d' % (a, b, s), f)


for (a, b, s) in test_data:
    add_sum_test(a, b, s)

...

The result is:

test_1_plus_1_equals_2 (tests.test_code.Case) ... ok
test_2_plus_2_equals_4 (tests.test_code.Case) ... ok
test_3_plus_5_equals_8 (tests.test_code.Case) ... ok

----------------------------------------------------------------------
Ran 3 tests in 0.004s

OK

Stage 016 in sample_tests branch contains another example of using this mechanism.

Further reading

https://docs.python-guide.org/writing/tests

https://docs.python.org/2/library/unittest.html

Examples

Code in sample_tests branch – note that it wasn't updated to fit this guide to the letter yet (for example, test structure does not reflect the code structure). However, it can still be used as a working demonstration of DKB testing.