diff --git a/content/code/R/example.R b/content/code/R/example.R new file mode 100644 index 0000000..940e2bf --- /dev/null +++ b/content/code/R/example.R @@ -0,0 +1,15 @@ +if (!require(testthat)) install.packages(testthat) +add <- function(a, b) { + return(a + b) +} + +test_that("Adding integers works", { + res <- add(2, 3) + + # Test that the result has the correct value + expect_identical(res, 5) + + # Test that the result is numeric + expect_true(is.numeric(res)) +}) + diff --git a/content/code/julia/example.jl b/content/code/julia/example.jl new file mode 100644 index 0000000..7d96f65 --- /dev/null +++ b/content/code/julia/example.jl @@ -0,0 +1,8 @@ +function myadd(a,b) + return a+b +end + +using Test +@testset "myadd" begin + @test myadd(2,3) == 5 +end \ No newline at end of file diff --git a/content/code/python/example.py b/content/code/python/example.py new file mode 100644 index 0000000..92ca3e4 --- /dev/null +++ b/content/code/python/example.py @@ -0,0 +1,6 @@ +def add(a, b): + return a + b + +def test_add(): + assert add(2, 3) == 5 + assert add('space', 'ship') == 'spaceship' \ No newline at end of file diff --git a/content/guide.md b/content/guide.md index 7e7a478..6292551 100644 --- a/content/guide.md +++ b/content/guide.md @@ -3,7 +3,7 @@ ## Detailed schedule - 9:00-9:15 [Motivation](https://coderefinery.github.io/testing/motivation/) -- 9:15-9:40 [Testing locally](https://coderefinery.github.io/testing/pytest/) +- 9:15-9:40 [Testing locally](https://coderefinery.github.io/testing/locally/) - explain the exercise: 5 min - **20 min exercise** - 9:40-10:00 [Automated testing](https://coderefinery.github.io/testing/continuous-integration/) diff --git a/content/index.rst b/content/index.rst index de66f07..573e993 100644 --- a/content/index.rst +++ b/content/index.rst @@ -39,7 +39,7 @@ see how automated testing works and practice designing and writing tests. :delim: ; 15 min ; :doc:`motivation` - 25 min ; :doc:`pytest` + 25 min ; :doc:`locally` 30 min ; :doc:`continuous-integration` 30 min ; :doc:`test-design` 5 min ; :doc:`conclusions` @@ -50,7 +50,7 @@ see how automated testing works and practice designing and writing tests. :caption: The lesson motivation - pytest + locally continuous-integration test-design conclusions diff --git a/content/locally.md b/content/locally.md new file mode 100644 index 0000000..04cb0f5 --- /dev/null +++ b/content/locally.md @@ -0,0 +1,299 @@ +# Testing locally + +```{questions} +- How hard is it to set up a test suite for a first unit test? +``` + + +## Exercise + + +In this exercise we will make a simple function and use +one of the language specific test frameworks to test it. + +* This is easy to use by almost any project and doesn't rely on any + other servers or services. +* The downside is that you have to remember to run it yourself. + +```````{exercise} Local-1: Create a minimal example (15 min) +In this exercise, we will create a minimal example using +the [pytest](http://doc.pytest.org), run the test, and show what +happens when a test breaks. + + +1. Create a new directory and change into it: + ```console + $ mkdir local-testing-example + $ cd local-testing-example + ``` + +2. Create an example file and paste the following code into it +`````{tabs} + ````{group-tab} Python + Create `example.py` with content + + ```{literalinclude} code/python/example.py + :language: python + ``` + This code contains one genuine function and a test function. + `pytest` finds any functions beginning with `test_` and treats them + as tests. + ```` + + ````{group-tab} R + Create `example.R` with content + ```{literalinclude} code/R/example.R + :language: R + ``` + A test with `testthat` is created by calling + `test_that()` with a test name and code as arguments. + ```` + + ````{group-tab} Julia + Create `example.jl` with content + ```{literalinclude} code/julia/example.jl + :language: Julia + ``` + The package `Test.jl` handles all testing. + A test(set) is added with `@testset` + and a test itself with `@test`. + ```` + +````` + +3. Run the test +`````{tabs} + ````{group-tab} Python + ```console + $ pytest -v example.py + + ============================================================ test session starts ================================= + platform linux -- Python 3.7.2, pytest-4.3.1, py-1.8.0, pluggy-0.9.0 -- /home/user/pytest-example/venv/bin/python3 + cachedir: .pytest_cache + rootdir: /home/user/pytest-example, inifile: + collected 1 item + + example.py::test_add PASSED + + ========================================================= 1 passed in 0.01 seconds =============================== + ``` + Yay! The test passed! + + Hint for participants trying this inside Spyder or IPython: try `!pytest -v example.py`. + ```` + ````{group-tab} R + ```console + $ Rscript example.R + + Loading required package: testthat + Test passed 🎉 + ``` + Yay! The test passed! + + Note that the emoji is random and might be different for you. + ```` + ````{group-tab} Julia + ```console + $ julia example.jl + + Test Summary: | Pass Total Time + myadd | 1 1 0.0s + ``` + Yay! The test passed! + ```` +````` +4. Let us break the test! + +Introduce a code change which breaks the code and check +whether out test detects the change: +`````{tabs} + ````{group-tab} Python + ```console + $ pytest -v example.py + + ============================================================ test session starts ================================= + platform linux -- Python 3.7.2, pytest-4.3.1, py-1.8.0, pluggy-0.9.0 -- /home/user/pytest-example/venv/bin/python3 + cachedir: .pytest_cache + rootdir: /home/user/pytest-example, inifile: + collected 1 item + + example.py::test_add FAILED + + ================================================================= FAILURES ======================================= + _________________________________________________________________ test_add _______________________________________ + + def test_add(): + > assert add(2, 3) == 5 + E assert -1 == 5 + E --1 + E +5 + + example.py:6: AssertionError + ========================================================= 1 failed in 0.05 seconds ============== + ``` + Notice how pytest is smart and includes context: lines that failed, + values of the relevant variables. + ```` + ````{group-tab} R + ```console + $ Rscript example.R + + ── Failure: Adding integers works ────────────────────────────── + `res` not identical to 5. + 1/1 mismatches + [1] -1 - 5 == -6 + + Error: Test failed + Execution halted + ``` + `testthat` tells us exactly which test failed and how + but does not include more context. + ```` + ````{group-tab} Julia + ```console + $ julia example.jl + + myadd: Test Failed at /home/user/local-testing-example/example.jl:7 + Expression: myadd(2, 3) == 5 + Evaluated: -1 == 5 + + Stacktrace: + [1] macro expansion + @ ~/opt/julia-1.10.0/share/julia/stdlib/v1.10/Test/src/Test.jl:672 [inlined] + [2] macro expansion + @ ~/local-testing-example/example.jl:7 [inlined] + [3] macro expansion + @ ~/opt/julia-1.10.0/share/julia/stdlib/v1.10/Test/src/Test.jl:1577 [inlined] + [4] top-level scope + @ ~/local-testing-example/example.jl:7 + Test Summary: | Fail Total Time + myadd | 1 1 0.6s + ERROR: LoadError: Some tests did not pass: 0 passed, 1 failed, 0 errored, 0 broken. + in expression starting at /home/user/local-testing-example/example.jl:6 + ``` + Notice how `Test.jl` is smart and includes context: + Lines that failed, evaluated and expected results. + ```` +````` +``````` + +```````{challenge} (optional) Local-2: Create a test that considers numerical tolerance (10 min) +Let's see an example where the test has to be more clever in order to +avoid false negative. + +In the above exercise we have compared integers. In this optional exercise we +want to learn how to compare floating point numbers since they are more tricky +(see also ["What Every Programmer Should Know About Floating-Point Arithmetic"](https://floating-point-gui.de/)). + +The following test will fail and this might be surprising. Try it out: +`````{tabs} + ````{group-tab} Python + ```python + def add(a, b): + return a + b + + def test_add(): + assert add(0.1, 0.2) == 0.3 + ``` + ```` + ````{group-tab} R + ```R + add <- function(a, b){ + return a + b + } + + test_that("Adding floats works", { + expect_identical(add(0.1, 0.2),0.3) + }) + ``` + ```` + ````{group-tab} Julia + ```Julia + function myadd(a,b) + return a + b + end + + using Test + @testset "myadd" begin + @test myadd(0.1, 0.2) == 0.3 + end + ``` + ```` +````` + +Your goal: find a more robust way to test this addition. +``````` + +```````{solution} Solution: Local-2 + +`````{tabs} + ````{group-tab} Python + One solution is to use + [pytest.approx](https://docs.pytest.org/en/4.6.x/reference.html#pytest-approx): + ```python + from pytest import approx + + def add(a, b): + return a + b + + def test_add(): + assert add(0.1, 0.2) == approx(0.3) + ``` + + But maybe you didn't know about + [pytest.approx](https://docs.pytest.org/en/4.6.x/reference.html#pytest-approx): + and did this instead: + ```python + def test_add(): + result = add(0.1, 0.2) + assert abs(result - 0.3) < 1.0e-7 + ``` + This is OK but the `1.0e-7` can be a bit arbitrary. + ```` + ````{group-tab} R + One solution is to use + [expect_equal](https://testthat.r-lib.org/reference/equality-expectations.html) which allows for roundoff errors: + ```R + test_that("Adding floats works with equal", { + res <- add(0.1, 0.2) + expect_equal(res,0.3) + expect_true(is.numeric(res)) + }) + ``` + + But maybe you didn't know about it and used the 'less than' comparison of [expect_lt](https://testthat.r-lib.org/reference/comparison-expectations.html) instead: + ```R + test_that("Adding floats works with lt", { + res <- add(0.1, 0.2) + expect_lt(abs(res-0.3),1.0e-7) + expect_true(is.numeric(res)) + }) + ``` + This is OK but the `1.0e-7` can be a bit arbitrary. + ```` + ````{group-tab} Julia + One solution is to use `\approx`: + ```Julia + @testset "Add floats with approx" begin + @test myadd(0.1,0.2) ≈ 0.3 + #Variant with specifying a tolerance + @test myadd(0.1,0.2) ≈ 0.3 atol=1.0e-7 + end + ``` + + But maybe you didn't know about `\approx` + and did this instead: + ```python + @test abs(myadd(0.1,0.2)-0.3) < 1.0e-7 + ``` + This is OK but the `1.0e-7` can be a bit arbitrary. + ```` +````` +```````` + +--- + +```{keypoints} +- Each test framework has its way of collecting and running all test functions, e.g. functions beginning with `test_` for `pytest`. +- Python, Julia and C/C++ have better tooling for automated tests than Fortran and you can use those also for Fortran projects (via `iso_c_binding`). +``` diff --git a/content/pytest.md b/content/pytest.md deleted file mode 100644 index 9272744..0000000 --- a/content/pytest.md +++ /dev/null @@ -1,146 +0,0 @@ -# Testing locally - -```{questions} -- How hard is it to implement a test suite using pytest? -``` - - -## Exercise [pytest](http://doc.pytest.org) - -In this exercise we will make a simple Python function and use -[pytest](http://doc.pytest.org) to test it. - -* This is easy to use by almost any Python project and doesn't rely on any - other servers or services. Similar things exist in other languages. -* The downside is that you have to remember to run it yourself. - -We will try to -keep things simple so that those who do not use Python can follow. - -````{challenge} Local-1: Create a minimal Pytest example (15 min) -In this exercise, we will create a minimal example using -the [pytest](http://doc.pytest.org), run the test, and show what -happens when a test breaks. - - -1. Create a new directory and change into it: - ```console - $ mkdir pytest-example - $ cd pytest-example - ``` - -2. Then create a file called `example.py` and copy-paste the following code into it: - ```python - def add(a, b): - return a + b - - - def test_add(): - assert add(2, 3) == 5 - assert add('space', 'ship') == 'spaceship' - ``` - This code contains one genuine function and a test function. - `pytest` finds any functions beginning with `test_` and treats them - as tests. - -3. Let us try to test it with pytest: - ```console - $ pytest -v example.py - - ============================================================ test session starts ================================= - platform linux -- Python 3.7.2, pytest-4.3.1, py-1.8.0, pluggy-0.9.0 -- /home/user/pytest-example/venv/bin/python3 - cachedir: .pytest_cache - rootdir: /home/user/pytest-example, inifile: - collected 1 item - - example.py::test_add PASSED - - ========================================================= 1 passed in 0.01 seconds =============================== - ``` - Yay! The test passed! - - Hint for participants trying this inside Spyder or IPython: try `!pytest -v example.py`. - -4. Let us break the test! - Introduce a code change which breaks the code and check - whether pytest detects the change: - ```console - $ pytest -v example.py - - ============================================================ test session starts ================================= - platform linux -- Python 3.7.2, pytest-4.3.1, py-1.8.0, pluggy-0.9.0 -- /home/user/pytest-example/venv/bin/python3 - cachedir: .pytest_cache - rootdir: /home/user/pytest-example, inifile: - collected 1 item - - example.py::test_add FAILED - - ================================================================= FAILURES ======================================= - _________________________________________________________________ test_add _______________________________________ - - def test_add(): - > assert add(2, 3) == 5 - E assert -1 == 5 - E --1 - E +5 - - example.py:6: AssertionError - ========================================================= 1 failed in 0.05 seconds ============== - ``` - Notice how pytest is smart and includes context: lines that failed, - values of the relevant variables. - -```` - -````{challenge} (optional) Local-2: Create a test that considers numerical tolerance (10 min) -Let's see an example where the test has to be more clever in order to -avoid false negative. - -In the above exercise we have compared integers. In this optional exercise we -want to learn how to compare floating point numbers since they are more tricky -(see also ["What Every Programmer Should Know About Floating-Point Arithmetic"](https://floating-point-gui.de/)). - -The following test will fail and this might be surprising. Try it out: - ```python - def add(a, b): - return a + b - - - def test_add(): - assert add(0.1, 0.2) == 0.3 - ``` - -Your goal: find a more robust way to test this addition. -```` - -````{solution} Solution: Local-2 -One solution is to use -[pytest.approx](https://docs.pytest.org/en/4.6.x/reference.html#pytest-approx): -```python -from pytest import approx - -def add(a, b): - return a + b - -def test_add(): - assert add(0.1, 0.2) == approx(0.3) -``` - -But maybe you didn't know about -[pytest.approx](https://docs.pytest.org/en/4.6.x/reference.html#pytest-approx): -and did this instead: -```python -def test_add(): - result = add(0.1, 0.2) - assert abs(result - 0.3) < 1.0e-7 -``` -This is OK but the `1.0e-7` can be a bit arbitrary. -```` - ---- - -```{keypoints} -- pytest collects and runs all test functions starting with - `test_`. If run on a directory, it collects all files matching `test_*.py` -- Python and C/C++ have better tooling for automated tests than Fortran and you can use those also for Fortran projects (via `iso_c_binding`). -```