Skip to content

Commit

Permalink
Revisions
Browse files Browse the repository at this point in the history
  • Loading branch information
tennlee committed Jun 6, 2024
1 parent e1a63a3 commit 3d0eac5
Showing 1 changed file with 10 additions and 8 deletions.
18 changes: 10 additions & 8 deletions docs/paper.md
Original file line number Diff line number Diff line change
Expand Up @@ -94,16 +94,18 @@ In order to meet the needs of researchers, `scores` provides the following key b

**Compatability**

- Provides its own implementations where relevant, avoiding extensive dependencies and allowing greater API consistency.
- Easy to integrate and use in a wide variety of environments. It has been tested and used on workstations, servers and in high performance computing (supercomputing) environments.
- Highly modular - provides its own implementations, avoids extensive dependencies and allows greater API consistency.
- Easy to integrate and use in a wide variety of environments. It has been used on workstations, servers and in high performance computing (supercomputing) environments.
- Maintains 100% automated test coverage.
- Uses Dask [@Dask:2016] for scaling and performance.
- Expanding support for `pandas` [@pandas:2024; @McKinney:2010].


## Metrics, Statistical Techniques and Data Processing Tools Included in `scores`

At the time of writing, `scores` includes over 50 metrics, statistical techniques and data processing tools. For an up to date list, please see the `scores` documentation.

The ongoing development roadmap includes additional metrics, tools, and statistical tests.
The ongoing development roadmap includes the addition of more metrics, tools, and statistical tests.

Table: A **Curated Selection** of the Metrics, Tools and Statistical Tests Currently Included in `scores`

Expand All @@ -121,23 +123,23 @@ Table: A **Curated Selection** of the Metrics, Tools and Statistical Tests Curre

## Use in Academic Work

In 2015, the Australian Bureau of Meteorology began developing a new verification system called Jive. For a description of Jive see @loveday2024jive. The Jive verification metrics have been used to support several publications [@Griffiths:2017; @Foley:2020; @Taggart:2022d; @Taggart:2022b; @Taggart:2022c]. `scores` has arisen from the Jive verification system and was created to modularise the Jive verification functions and make them available as an open source package.
In 2015, the Australian Bureau of Meteorology began developing a new verification system called Jive, which became operational in 2022. For a description of Jive see @loveday2024jive. The Jive verification metrics have been used to support several publications [@Griffiths:2017; @Foley:2020; @Taggart:2022d; @Taggart:2022b; @Taggart:2022c]. `scores` has arisen from the Jive verification system and was created to modularise the Jive verification functions and make them available as an open source package.

`scores` has been used to explore user-focused approaches to evaluating probabilistic and categorical forecasts [@Loveday2024ts].

## Related Software Packages

There are multiple open source verification packages in a range of languages. Below is a comparison of `scores` to other open source Python verification packages. None of these include all of the functions and key benefits of `scores`(and vice versa). None of these include specific support for emerging metrics.

There are multiple open source verification packages in a range of languages. Below is a comparison of `scores` to other open source Python verification packages. None of these include all of the metrics implemented in `scores` (and vice versa). None of these include specific support for emerging metrics.
`xskillscore` [@xskillscore] provides many but not all of the same functions as `scores` and does not have direct support for pandas. The Jupyter Notebook tutorials in `scores` cover a wider array of metrics.

`climpred` [@Brady:2021] utilises `xskillscore` combined with data handling functionality, and is focused on ensemble forecasts for climate and weather. `climpred` makes some design choices related to data structure (specifically associated with climate modelling) which may not generalise effectively to broader use cases. Releasing `scores` separately allows the differing design philosophies to be considered by the community.

`METplus` [@Brown:2021] is widely used by weather and climate model developers. `METplus` includes a database and a visualisation system, with Python and shell script wrappers to utilise the complex `MET` package. Verification scores in `MET` are implemented in C++ rather than Python. `METplus` is used as a system rather than providing a modular Python API.
`METplus` [@Brown:2021] is a verification system used by weather and climate model developers. `METplus` includes a database and a visualisation system, with Python and shell script wrappers to utilise the `MET` package for the calculation of scores. `MET` is implemented in C++ rather than Python. `METplus` is used as a system rather than providing a modular Python API.

`Verif` [@nipen2023verif] is a command line tool for generating verification plots whereas `scores` provides a Python API for generating numerical scores.

`Pysteps` [@gmd-12-4185-2019; @Imhoff:2023] is a package for short-term ensemble prediction systems, and includes a significant verification submodule with many useful verification scores. `PySteps` does not provide a modular (standalone) verification API.
`Pysteps` [@gmd-12-4185-2019; @Imhoff:2023] is a package for short-term ensemble prediction systems, and includes a significant verification submodule with many useful verification scores. `PySteps` does not provide a standalone verification API.

`PyForecastTools` [@Morley:2020] is a Python package for model and forecast validation which supports `dmarray` rather than `xarray` data structures and does not include Jupyter Notebook tutorials.

Expand Down

0 comments on commit 3d0eac5

Please sign in to comment.