Skip to content

Commit

Permalink
Merge pull request betterscientificsoftware#2141 from bartlettroscoe/…
Browse files Browse the repository at this point in the history 
…howto-improve-testing-2024

Fixes for How to Improve Testing for CSE Software article
  • Loading branch information
@bartlettroscoe
bartlettroscoe authored Aug 23, 2024
2 parents b0ef302 + d45b911 commit 20af21b
Showing 1 changed file with 66 additions and 41 deletions.
107 changes: 66 additions & 41 deletions Articles/Blog/HowtoImproveTestinginyourCSESoftwareProject.md
View file
Original file line number Diff line number Diff line change
@@ -1,74 +1,85 @@
# How to Improve Testing for CSE Software
#### Contributed by [Ulrike Yang](https://github.com/ulrikeyang)

#### Contributed by [Roscoe A. Bartlett](https://github.com/bartlettroscoe), [Barry Smith](https://github.com/BarrySmith), [Ulrike Meier Yang](https://github.com/ulrikeyang), [Glenn Hammond](https://github.com/ghammond86), [Xiaoye Li](https://github.com/xiaoyeli), and [James Willenbring](https://github.com/jwillenbring)

#### Publication Date: August 6, 2019

<!-- deck text start -->
Software requires regular extensive testing to ensure correctly functioning code. This article provides basic information on CSE software testing.
Software requires regular extensive testing to ensure correctly functioning code.
This article provides a straightforward process to add testing to an existing software project that has no testing (or insufficient testing).
<!-- deck text end -->

### Overview
Adding tests of sufficient coverage and quality improves confidence in software and makes it easier to change and extend. Tests should be added to existing code before the code is changed. Tests should be added to new code before (or while) it is being written. These tests then become the foundation of a regression test suite that helps effectively drive future
development and improves long-term sustainability.
Adding tests of sufficient coverage and quality improves confidence in software and makes it easier to change and extend.
Tests should be added to existing uncovered code before the code is changed.
Tests should be added to new code before (or while) it is being written.
These tests then become the foundation of a regression test suite that helps effectively drive future development while maintaining behavior and improves long-term sustainability.

### Target Audience
CSE software project leaders and developers who are facing significant refactoring efforts because of hardware architecture changes or increased demands for multiphysics and multiscale coupling, and who want to increase the quality and speed of
development and reduce development and maintenance costs.
Computational Science and Engineering (CSE) software project leaders and developers who are facing significant refactoring efforts because of hardware architecture changes or increased demands for multi-physics and multi-scale coupling, and who want to increase the quality and speed of development and reduce development and maintenance costs.

### Purpose
Show how to add quality testing to a project in order to support efficient modification of existing code or addition of new code. Show how to add tests to support (1) **adding a new feature**, (2) **fixing a bug**,(3) **improving the design and implementation**,or (4) **optimizing resource usage**
Show how to add quality testing to a project in order to support efficient modification of existing code or addition of new code.
Show how to add tests to support (1) **adding a new feature**, (2) **fixing a bug**, (3) **improving the design and implementation**, or (4) **optimizing resource usage**<sup>[1]</sup>

### Prerequisites

First read the document *[What Are Software Testing Practices?](./UnderstandingSoftwareTestingPractices.md)* and browse through *[Definition and Categorization of Tests for CSE Software](./DefinitionsCategorizationsOfTests.md)*.

### Steps

1. Set up **automated builds of the code** with high warning levels and eliminate all warnings.
2. **Select test harness frameworks**
a. **Select a system-level test harness** for system-executable tests that report results appropriately (e.g., CTest/CDash, Jenkins).
b. **Select a unit test harness** to effectively define and run finer-grained integration and unit tests (e.g., Google Test, pFUnit).
c. **Customize or streamline** system-level and/or unit test frameworks for use in your particular project.
1. **Select a system-level test harness** for system-executable tests that report results appropriately (e.g., CTest/CDash).
1. **Select a unit test harness** to effectively define and run finer-grained integration and unit tests (e.g., Google Test, pFUnit).
1. **Customize or streamline** system-level and/or unit test frameworks for use in your particular project.
3. **Add system-level tests** to protect major user functionality.
a. Select inputs for several important problem classes and run code to produce outputs.
b. Set up no-change or verification tests with a system-level test harness in order to pin down important behavior.
1. Select inputs for several important problem classes and run code to produce outputs.
1. Set up no-change or verification tests with a system-level test harness in order to pin down important behavior.
4. **Add integration and unit tests** (as needed for adding/changing code)
a. **Incorporate test** [1, 2] **for code to be changed**
* **Identify change points** for target change or new code
* **Find test points** where code behavior can be sensed.
* **Break dependencies** in order to get the targeted code into the unit test harness.
* **Cover targeted code** to be changed with sufficient (characterization) tests.
b. **Add new features or fix bugs with tests** [1, 2, 3, 4]
* **Add new tests** that define desired behavior (feature or bug).
* Run new tests and **verify they fail**.
* Add the minimal code to **get new tests to pass**.
* **Refactor** the covered code to clean up and remove duplication.
* **Review** all changes to existing code, new code and new tests.
5. Select **code coverage** (e.g., gcov/lcov) and **memory usage error detection** (e.g., valgrind) analysis tools.
1. **Incorporate tests for uncovered code to be changed** using the ***Legacy Software Change Algorithm***<sup>[1]</sup>
* **Identify change points** for target change or new code.
* **Find test points** where code behavior can be sensed.
* **Break dependencies** in order to get the targeted code into the unit test harness.
* **Cover targeted code** to be changed with sufficient (characterization) tests.
2. **Add new features or fix bugs**<sup>[1],[2],[3]</sup>
* **Add new tests** that define desired behavior (feature or bug).
* Run new tests and **verify they fail**.
* Add the minimal code to **get new tests to pass**.
* **Refactor** the covered code to clean up and remove duplication.
* **Review** all changes to existing code, new code and new tests.
5. Select **code coverage** (e.g., gcov/lcov) and **memory usage error detection** (e.g., valgrind, clang memory/address/leak sanitizers) analysis tools.
6. Define a set of **regression test suites**.
a. Define a faster-running **pre-push regression test suite** (e.g., single build with faster running tests) and **run it before every push**.
b. Define a more comprehensive nightly **regression test suite** (e.g., builds and all tests on several platforms and compilers, code coverage, and memory usage error detection) and **run every night**.
7. Have a policy of **100% passing pre-push regression tests** and work hard to maintain that.
8. Work to **fix all failing nighty regression tests** on a reasonable schedule.
1. Define a faster-running **pre-merge regression test suite** (e.g., single build with faster running tests) and **run it before every merge to the mainline branch**.
1. Define a more comprehensive nightly **regression test suite** (e.g., builds and all tests on several platforms and compilers, code coverage, and memory usage error detection) and **run every night**.
7. Have a policy of **100% passing pre-merge regression tests** (run using a CI testing system like GitHub Actions, GitLab CI, or Jenkins) and work hard to maintain that.
8. Work to **fix all failing nightly regression tests** on a reasonable schedule.


### FAQs:
**Q: Why do you need both a system-level and a unit test harness?**
A: A unit test harness aggregates hundreds of unit and integration tests into single executables. A system-level test harness runs these aggregate integration and unit test executables along with the other system-level acceptance and verification tests and alerts developers of any failures.

**Q: Why not just add all of the tests for an existing code and get it over with?**
A: Taking weeks or months (or years) to add sufficient tests for an entire existing code (that lacks sufficient testing) is not usually economical or necessary. Tests need to be added to code only when it is changed (or when adding new code). In that way tests can be added while regular development work is being done.
**Q: Why do you need both a system-level and a unit test harness?**<br>
**A**: A unit test harness aggregates hundreds of unit and integration tests into single executables.
A system-level test harness runs these aggregate integration and unit test executables along with the other system-level acceptance and verification tests and alerts developers of any failures.

**Q: Why not just add all of the tests for an existing code and get it over with?**<br>
**A**: Taking weeks or months (or years) to add sufficient tests for an entire existing code (that lacks sufficient testing) is not usually economical or necessary.
Tests need to be added to code only when it is changed (or when adding new code).
In that way tests can be added while regular development work is being done.

**Q: Why demand 100% passing pre-push regression tests?**
A: This avoids expensive debugging and other investigations needed to determine whether your changes are breaking failing tests or not (hard). If all tests pass, then your changes could be breaking them (easy).
**Q: Why demand 100% passing pre-merge regression tests?**<br>
**A**: This avoids expensive debugging and other investigations needed to determine whether your changes are breaking failing tests or not (hard).
If all tests pass, then your changes could be breaking them (easy).

### References:
<!---
ToDos:
1. [Working Effectively with Legacy Code. Prentice Hall](https://www.oreilly.com/library/view/working-effectively-with/0131177052/), Michael Feathers, 2005.
2. [Legacy Software Change Algorithm](https://trilinos.org/trac/trilinos/wiki/TribitsLegacySoftwareChangeAlgorithm).
3. [Test Driven Development](https://www.oreilly.com/library/view/test-driven-development/0321146530/), Kent Beck, 2003.
4. [Code Complete (Second Edition)](https://www.microsoftpressstore.com/store/code-complete-9780735619678), Steve McConnell, 2004.
* Improve the deck text ...
--->

This document was prepared with input from Roscoe A. Bartlett, Glenn Hammond, Xiaoye Li, Barry Smith,
and James M. Willenbring.
<!--- References --->

<!---
Publish: yes
Expand All @@ -77,3 +88,17 @@ Topics: Testing
Track: how to
--->

[1-sfer-ezikiw]: https://bssw.io/items/working-effectively-with-legacy-code "Working Effectively with Legacy Code {Michael Feathers. Prentice Hall, 2005}"
[2-sfer-ezikiw]: https://www.oreilly.com/library/view/test-driven-development/0321146530/ "Test Driven Development {Kent Beck, Addison-Wesley Professional, 2003, ISBN: 0321146530}"
[3-sfer-ezikiw]: https://bssw.io/items/code-complete-a-practical-handbook-of-software-construction "Code Complete: Second Edition}"

<!-- DO NOT EDIT BELOW HERE. THIS IS ALL AUTO-GENERATED (sfer-ezikiw) -->
[1]: #sfer-ezikiw-1 "Working Effectively with Legacy Code"
[2]: #sfer-ezikiw-2 "Test Driven Development"
[3]: #sfer-ezikiw-3 "Code Complete: Second Edition}"
<!-- (sfer-ezikiw begin) -->
### References
<!-- (sfer-ezikiw end) -->
* <a name="sfer-ezikiw-1"></a><sup>1</sup>[Working Effectively with Legacy Code<br>Michael Feathers. Prentice Hall, 2005](https://bssw.io/items/working-effectively-with-legacy-code)
* <a name="sfer-ezikiw-2"></a><sup>2</sup>[Test Driven Development<br>Kent Beck, Addison-Wesley Professional, 2003, ISBN: 0321146530](https://www.oreilly.com/library/view/test-driven-development/0321146530/)
* <a name="sfer-ezikiw-3"></a><sup>3</sup>[Code Complete: Second Edition}](https://bssw.io/items/code-complete-a-practical-handbook-of-software-construction)

0 comments on commit 20af21b

Please sign in to comment.