Testing Standards

Code Testing

All models and tools within the IDAES code base are expected to have accompanying tests, which check for both coding errors and final results.

Types of Tests

IDAES test are divided into three categories, which are used for organizing tests into different levels of rigor and complexity (and thus execution time). Lower level tests are expected to run frequently, and thus need to keep execution time to a minimum to avoid delays, whilst higher level tests are run less frequently and can thus take longer to complete. The three categories used by IDAES are:

• unit: Unit tests are used to test basic functionality and execution of individual parts of the code. Within IDAES, these tests are generally used to test model construction, but not model solutions. These tests should be fast to run (less than 2 seconds), and should not require the use of a solver. Unit tests should ideally cover all lines of code within a model with the exception of those requiring a solver to complete (initialization and final solves).
• component: Component tests are used to test model solutions for single example cases in order to check that a model can be solved. These test obviously require the use of a solver, but should still be relatively quick to execute (ideally less than 10 seconds).
• integration: The final level of tests are integration tests, which are used for longer duration verification and validation tests. These tests are used to confirm model accuracy and robustness over a wide range of conditions, and as such can take longer to execute. integration tests are also used to execute all examples in the IDAES Examples to ensure that any changes to the core codebase do not break the examples.

As a general rule, any tool or module should have a set of unit and component tests that exercise and solve all possible options/combinations for a single (generally simple) test case to confirm that the code works as expected and can be solved. Each model or tool should also have a more extensive suite of integration tests which test and verify the accuracy and robustness of the model/tool over as wide a range of conditions as possible.

More detailed guidelines and standards for different types of models and tools can be found in the sidebar to the right.

How to Write Good Tests

Writing tests is something of an art form, and it takes a while to learn what should be tested and how best to achieve this. Below are some suggestions compiled from the experience of the IDAES developers.

Writing good tests first starts with writing good code. A single large method or function is not conducive to good testing, as it is impossible to break it down and see what is happening at each step – all you can tell is whether the entire method/function gave the expected result or not (and even then you cannot be sure if that was just coincidence). Writing your code in small, testable methods or functions will greatly increase your ability to test and verify your code.

Things to Include

• Tests should always contain an assert statement (or equivalent). It is easy to write a test that executes the code, but unless you add checks for specific behaviors all the test will tell you is if there are any Exceptions during execution.
• All tests should include a pytest.mark to indicate the type of test.
• Tests should be written that execute all branches in conditional statements, and should check to make sure the correct branch was taken.
• Testing for failures is often as important as testing for success.
• When creating a fix for a known bug, always add a test that confirms the fix to ensure it does not get broken again in future.
• Any Exceptions raised by the code should be tested. You can use pytest.raises(ExceptionType, matches=str) to check that the correct type of Exception was raised and that the message matches the expected string.
• Similarly, logger messages output to the screen can be captures and tested using capsys and caplog: https://docs.pytest.org/en/6.2.x/capture.html
• When testing model solution, always begin by checking that the solver returned an optimal solution.

results = solver.solve(model)
assert results.solver.termination_condition == TerminationCondition.optimal
assert results.solver.status == SolverStatus.ok

• When testing model solutions, check for as many key variables as possible. Include intermediate variables as well to help narrow down any failures.
• Also keep in mind solver tolerances when testing results. The default solver tolerance is 1e-6, so you should you should try to test to a slightly looser tolerance (we often use 1e-5). Be aware that IPOPTs tolerance is both absolute and relative.

Things to Avoid

• Single, monolithic tests give less information than multiple smaller tests. A test stops at the first failure, so any subsequent failures will not be revealed until after the first problem has been resolved. Similarly, smaller tests make it easier to track down the exact cause of the failure, as there are fewer changes to keep track of.