Skip to content

Commit

Permalink
Docs: add introductions to ci/cd and new modules (deepmodeling#3610)
Browse files Browse the repository at this point in the history
* Add instructions on adding a component

* Add docs on ci/cd pipelines
  • Loading branch information
caic99 authored Feb 18, 2024
1 parent 21f5c60 commit dd227ae
Show file tree
Hide file tree
Showing 3 changed files with 109 additions and 32 deletions.
84 changes: 65 additions & 19 deletions docs/CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -38,8 +38,8 @@ The source code of ABACUS is based on several modules. Under the ABACUS root dir

For those who are interested in the source code, the following figure shows the structure of the source code.

```
|-- module_base A basic module including
```text
|-- module_base A basic module including
| | (1) Mathematical library interface functions: BLAS, LAPACK, Scalapack;
| | (2) Custom data classes: matrix, vector definitions and related functions;
| | (3) Parallelization functions: MPI, OpenMP;
Expand All @@ -59,7 +59,7 @@ For those who are interested in the source code, the following figure shows the
| |-- potentials The module for calculating the potentials, including Hartree, exchange-correlation, local pseudopotential, etc.
|-- module_esolver The module defining task-specific driver of corresponding workflow for evaluating energies, forces, etc., including lj, dp, ks, sdft, ofdft, etc.
| | TDDFT, Orbital-free DFT, etc.
|-- module_hamilt_general The module for defining general Hamiltonian that can be used both in PW and LCAO calculations.
|-- module_hamilt_general The module for defining general Hamiltonian that can be used both in PW and LCAO calculations.
| |-- module_ewald The module for calculating the Ewald summation.
| |-- module_surchem The module for calculating the surface charge correction.
| |-- module_vdw The module for calculating the van der Waals correction.
Expand All @@ -77,7 +77,7 @@ For those who are interested in the source code, the following figure shows the
| |-- hamilt_pwdft The module for defining the Hamiltonian in PW-DFT calculations.
| | |-- operator_pw The module for defining the operators in PW-DFT calculations.
| `-- hamilt_stodft The module for defining the Hamiltonian in STODFT calculations.
|-- module_hsolver The module for solving the Hamiltonian with different diagonalization methods, including CG, Davidson in PW
|-- module_hsolver The module for solving the Hamiltonian with different diagonalization methods, including CG, Davidson in PW
| | calculations, and scalapack and genelpa in LCAO calculations.
|-- module_io The module for reading of INPUT files and output properties including band structure, density of states, charge density, etc.
|-- module_md The module for performing molecular dynamics.
Expand Down Expand Up @@ -201,42 +201,55 @@ To add a unit test:

## Running unit tests

1. Compiling ABACUS with unit tests.
1. Compiling ABACUS with unit tests.

In order to run unit tests, ABACUS needs to be configured with `-D BUILD_TESTING=ON` flag. For example:

```bash
cmake -B build -DBUILD_TESTING=ON
```
then build ABACUS and unit testing with
```

then build ABACUS and unit testing with

```bash
cmake --build build -j${number of processors}
```

It is import to run the folloing command before running unit tests:

```bash
cmake --install build
```
to install mandatory supporting input files for unit tests.

to install mandatory supporting input files for unit tests.
If you modified the unit tests to add new tests or learn how to write unit tests, it is convenient to run

```bash
cmake --build build -j${number of processors} --target ${unit test name}
```

to build a specific unit test. And please remember to run `cmake --install build` after building the unit test if the unit test requires supporting input files.

2. Running unit tests

The test cases are located in `build/source/${module_name}/test` directory. Note that there are other directory names for unit tests, for example, `test_parallel` for running parallel unit tests, `test_pw` for running unit tests only used in plane wave basis calculation.
The test cases are located in `build/source/${module_name}/test` directory. Note that there are other directory names for unit tests, for example, `test_parallel` for running parallel unit tests, `test_pw` for running unit tests only used in plane wave basis calculation.

You can run a single test in the specific directory. For example, run
```

```bash
./cell_unitcell_test
```
under the directory of `build/source/module_cell/test` to run the test `cell_unitcell_test`.

under the directory of `build/source/module_cell/test` to run the test `cell_unitcell_test`.
However, it is more convenient to run unit tests with `ctest` command under the `build` directory. You can check all unit tests by

```bash
ctest -N
```
The results will be shown as
```

The results will be shown as

```text
Test project /root/abacus/build
Test #1: integrated_test
Test #2: Container_UTs
Expand All @@ -245,23 +258,30 @@ To add a unit test:
Test #5: base_timer
...
```

Note that the first one is integrated test, which is not a unit test. It is the test
suite for testing the whole ABACUS package. The examples are located in the `tests/integrate` directory.

To run a subset of tests, run the following command
To run a subset of tests, run the following command

```bash
ctest -R <test-match-pattern> -V
```
For example, `ctest -R cell` will perform tests with name matched by `cell`.
You can also run a single test with
```

For example, `ctest -R cell` will perform tests with name matched by `cell`.
You can also run a single test with

```bash
ctest -R <test-name>
```
For example, `ctest -R cell_unitcell_test_readpp` will perform test `cell_unitcell_test_readpp`.

For example, `ctest -R cell_unitcell_test_readpp` will perform test `cell_unitcell_test_readpp`.
To run all the unit tests, together with the integrated test, run

```bash
cmake --build build --target test ARGS="-V --timeout 21600"
```

in the `abacus-develop` directory.

## Debugging the codes
Expand Down Expand Up @@ -290,6 +310,32 @@ Run ABACUS as usual, and it will automatically detect the buffer overflow proble

[Valgrind](https://valgrind.org/) is another option for performing dynamic analysis.

## Adding a new building component

ABACUS uses CMake as its default building system. To add a new building component:

1. Add an `OPTION` to toggle the component to the `CMakeLists.txt` file under root directory. For example:

```cmake
OPTION(ENABLE_NEW_COMPONENT "Enable new component" OFF)
```

2. Add the new component. For example:

```cmake
IF (ENABLE_NEW_COMPONENT)
add_subdirectory(module_my_new_feature) # if the feature is implemented in a subdirectory
find_package(NewComponent REQUIRED) # if third-party libs are required
target_link_libraries(${ABACUS_BIN_NAME} PRIVATE NewComponent) # if the component is linked
include_directories(${NewComponent_INCLUDE_DIRS}) # if the component is included
endif()
```

3. Add the required third-party libraries to Dockerfiles.
4. After the changes above are merged, submit another PR to build and test the new component in the CI pipeline.
- For integration test and unit test: add `-DENABLE_NEW_COMPONENT=ON` to the building step at `.github/workflows/test.yml`.
- For building test: add `-DENABLE_NEW_COMPONENT=ON` as a new configuration at `.github/workflows/build_test_cmake.yml`.

## Generating code coverage report

This feature requires using GCC compiler. We use `gcov` and `lcov` to generate code coverage report.
Expand All @@ -306,7 +352,7 @@ This feature requires using GCC compiler. We use `gcov` and `lcov` to generate c
cmake --build build --target test ARGS="-V --timeout 21600"
```

If configuration fails unfortunately, you can find [required files](https://github.com/baixiaokuang/CMake-codecov/tree/master/cmake) (including three *.cmake and llvm-cov-wrapper), and copy these four files into `/abacus-develop/cmake`. Alternatively, you can define the path with option `-D CMAKE_CURRENT_SOURCE_DIR`.
If configuration fails unfortunately, you can find [required files](https://github.com/baixiaokuang/CMake-codecov/tree/master/cmake) (including three *.cmake and llvm-cov-wrapper), and copy these four files into `/abacus-develop/cmake`. Alternatively, you can define the path with option `-D CMAKE_CURRENT_SOURCE_DIR`.

3. Generate HTML report.

Expand Down
30 changes: 30 additions & 0 deletions docs/community/cicd.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
# Lists of continuous integration (CI) pipelines

The directory `./github/workflows` contains the continuous integration (CI) pipelines for the project. The pipelines are written in YAML format and are executed by GitHub Actions. Check the [Actions page](https://github.com/deepmodeling/abacus-develop/actions) of the repo for the status of the pipelines.

> The tests without mentioning are not executed.
## On Pull Request (PR)

The following CI pipelines are triggered on pull request (PR) creation or update (the user pushes a new commit to the incoming branch):

- Integration test and unit tests (`test.yml`): This pipeline builds ABACUS with all available features, runs integration tests, and runs unit tests.

- Building tests with CMake (`build_test_cmake.yml`): This pipeline builds ABACUS with each feature separately, ensuring: (i) there are no conflicts between features, (ii) the compilation is successful whether the feature is enabled, and (iii) it works well on multiple platforms, i.e. with GNU+OpenBLAS toolchain and Intel+MKL toolchain.

- [Rerender the docs site](https://readthedocs.org/projects/abacus-rtd/builds/): This pipeline rerenders the documentation site on Read the Docs. It is automatically triggered when the documentation is updated.

- Testing GPU features (`cuda.yml`): This pipeline builds ABACUS with GPU support and runs several tests on the GPU. **Currently disabled for the lack of GPU resource.**

After the PR merges into the main branch, the following pipelines are triggered:

- Building Docker images(`devcontainer.yml`): This pipeline builds the Docker images with latest codes and executables. The images are tagged as `abacus-gnu:latest`, `abacus-intel:latest`, and `abacus-cuda:latest`, and then pushed to the GitHub Container Registry (ghcr.io/deepmodeling) and AliCloud Container mirror(registry.dp.tech/deepmodeling). For example: `docker pull ghcr.io/deepmodeling/abacus-intel:latest`.

## On Routine

- Dynamic analysis (`dynamic_analysis.yml`): This pipeline runs integration tests with [AddressSanitizer](https://github.com/google/sanitizers/wiki/AddressSanitizer) to detect memory errors. The pipeline is scheduled to run **every Sunday**. The results are published on [GitHub Pages](https://deepmodeling.github.io/abacus-develop/).

## On Release

- Coverage test (`coverage.yml`): This pipeline builds ABACUS with all available features, runs integration tests, and runs unit tests. It also measures the code coverage of the tests. The results are published at [codecov.io](https://app.codecov.io/gh/deepmodeling/abacus-develop).
- Building tagged Docker images (`image.yml`): The built image is tagged in the pattern of `abacus:3.5.0`, and pushed to the GitHub Container Registry (ghcr.io/deepmodeling) and AliCloud Container mirror(registry.dp.tech/deepmodeling). Use `abacus:latest` to fetch the latest image. For example: `docker pull ghcr.io/deepmodeling/abacus:latest`.
27 changes: 14 additions & 13 deletions docs/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -7,19 +7,19 @@
ABACUS Documentation
=================================================

ABACUS (Atomic-orbital Based Ab-initio Computation at UStc) is
an open-source computer code package based on density functional
theory (DFT). The package utilizes both plane wave and numerical
atomic basis sets with the usage of norm-conserving pseudopotentials
to describe the interactions between nuclear ions and valence electrons.
ABACUS supports LDA, GGA, meta-GGA, and hybrid functionals. Apart from
single-point calculations, the package allows geometry optimizations
and ab-initio molecular dynamics with various ensembles. The package
also provides a variety of advanced functionalities for simulating materials,
including the DFT+U, VdW corrections, and implicit solvation model, etc.
In addition, ABACUS strives to provide a general infrastructure to facilitate
the developments and applications of novel machine-learning-assisted DFT methods
(DeePKS, DP-GEN, DeepH, etc.) in molecular and material simulations.
ABACUS (Atomic-orbital Based Ab-initio Computation at UStc) is
an open-source computer code package based on density functional
theory (DFT). The package utilizes both plane wave and numerical
atomic basis sets with the usage of norm-conserving pseudopotentials
to describe the interactions between nuclear ions and valence electrons.
ABACUS supports LDA, GGA, meta-GGA, and hybrid functionals. Apart from
single-point calculations, the package allows geometry optimizations
and ab-initio molecular dynamics with various ensembles. The package
also provides a variety of advanced functionalities for simulating materials,
including the DFT+U, VdW corrections, and implicit solvation model, etc.
In addition, ABACUS strives to provide a general infrastructure to facilitate
the developments and applications of novel machine-learning-assisted DFT methods
(DeePKS, DP-GEN, DeepH, etc.) in molecular and material simulations.

.. toctree::
:maxdepth: 2
Expand Down Expand Up @@ -62,6 +62,7 @@ the developments and applications of novel machine-learning-assisted DFT methods

community/contribution_guide
CONTRIBUTING
community/cicd.md

.. toctree::
:glob:
Expand Down

0 comments on commit dd227ae

Please sign in to comment.