From ac968905b96b531346a7088973cdaa9ebb73ab05 Mon Sep 17 00:00:00 2001 From: Tennessee Leeuwenburg Date: Fri, 7 Jun 2024 11:20:57 +1000 Subject: [PATCH] Response to feedback --- docs/paper.md | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/docs/paper.md b/docs/paper.md index 0d479ea1..61fd529a 100644 --- a/docs/paper.md +++ b/docs/paper.md @@ -71,11 +71,11 @@ The software repository can be found at [https://github.com/nci/scores/](https:/ # Statement of Need -The research purpose of this software is (a) to mathematically verify and validate scientific research and (b) to foster research into new scores and metrics. +The purpose of this software is (a) to mathematically verify and validate models and predictions and (b) to foster research into new scores and metrics. ## Key Benefits of `scores` -In order to meet the needs of researchers, `scores` provides the following key benefits. +In order to meet the needs of researchers and other users, `scores` provides the following key benefits. **Data Handling** @@ -107,7 +107,7 @@ At the time of writing, `scores` includes over 50 metrics, statistical technique The ongoing development roadmap includes the addition of more metrics, tools, and statistical tests. -Table: A **Curated Selection** of the Metrics, Tools and Statistical Tests Currently Included in `scores` +Table: A **Curated Selection** of the metrics, tools and statistical tests currently included in `scores` | | **Description** |**A Selection of the Functions Included in `scores`**| |--------------|-----------------|-----------------------------------------------------| @@ -115,7 +115,7 @@ Table: A **Curated Selection** of the Metrics, Tools and Statistical Tests Curre | | **Probability** |Scores for evaluating forecasts that are expressed as predictive distributions, ensembles, and probabilities of binary events. |Brier Score [@BRIER_1950], Continuous Ranked Probability Score (CRPS) for Cumulative Distribution Functions (CDFs) (including threshold-weighting, see @Gneiting:2011), CRPS for ensembles [@Gneiting_2007; @Ferro_2013], Receiver Operating Characteristic (ROC), Isotonic Regression (reliability diagrams) [@dimitriadis2021stable]. | -| **Categorical** |Scores for evaluating forecasts based on categories. |Probability of Detection (POD), Probability of False Detection (POFD), False Alarm Ratio (FAR), Success Ratio, Accuracy, Peirce's Skill Score [@Peirce:1884], Critical Success Index (CSI), Gilbert Skill Score [@gilbert:1884], Heidke Skill Score, Odds Ratio, Odds Ratio Skill Score, F1 Score, FIxed Risk Multicategorical (FIRM) Score [@Taggart:2022a]. +| **Categorical** |Scores for evaluating forecasts of categories. |Probability of Detection (POD), Probability of False Detection (POFD), False Alarm Ratio (FAR), Success Ratio, Accuracy, Peirce's Skill Score [@Peirce:1884], Critical Success Index (CSI), Gilbert Skill Score [@gilbert:1884], Heidke Skill Score, Odds Ratio, Odds Ratio Skill Score, F1 Score, FIxed Risk Multicategorical (FIRM) Score [@Taggart:2022a]. | | **Statistical Tests** |Tools to conduct statistical tests and generate confidence intervals. | Diebold-Mariano [@Diebold:1995] with both the @Harvey:1997 and @Hering:2011 modifications. | @@ -133,19 +133,19 @@ There are multiple open source verification packages in a range of languages. Be `xskillscore` [@xskillscore] provides many but not all of the same functions as `scores` and does not have direct support for pandas. The Jupyter Notebook tutorials in `scores` cover a wider array of metrics. -`climpred` [@Brady:2021] utilises `xskillscore` combined with data handling functionality, and is focused on ensemble forecasts for climate and weather. `climpred` makes some design choices related to data structure (specifically associated with climate modelling) which may not generalise effectively to broader use cases. Releasing `scores` separately allows the differing design philosophies to be considered by the community. +`climpred` [@Brady:2021] uses `xskillscore` combined with data handling functionality, and is focused on ensemble forecasts for climate and weather. `climpred` makes some design choices related to data structure (specifically associated with climate modelling) which may not generalise effectively to broader use cases. Releasing `scores` separately allows the differing design philosophies to be considered by the community. -`METplus` [@Brown:2021] is a verification system used by weather and climate model developers. `METplus` includes a database and a visualisation system, with Python and shell script wrappers to utilise the `MET` package for the calculation of scores. `MET` is implemented in C++ rather than Python. `METplus` is used as a system rather than providing a modular Python API. +`METplus` [@Brown:2021] is a substantial verification system used by weather and climate model developers. `METplus` includes a database and a visualisation system, with Python and shell script wrappers to utilise the `MET` package for the calculation of scores. `MET` is implemented in C++ rather than Python. `METplus` is used as a system rather than providing a modular Python API. `Verif` [@nipen2023verif] is a command line tool for generating verification plots whereas `scores` provides a Python API for generating numerical scores. `Pysteps` [@gmd-12-4185-2019; @Imhoff:2023] is a package for short-term ensemble prediction systems, and includes a significant verification submodule with many useful verification scores. `PySteps` does not provide a standalone verification API. -`PyForecastTools` [@Morley:2020] is a Python package for model and forecast validation which supports `dmarray` rather than `xarray` data structures and does not include Jupyter Notebook tutorials. +`PyForecastTools` [@Morley:2020] is a Python package for model and forecast verification which supports `dmarray` rather than `xarray` data structures and does not include Jupyter Notebook tutorials. # Acknowledgements -We acknowledge and are grateful for the support of the Australian Bureau of Meteorology in supporting scientific research and the academic process. +We would like to thank Jason West and Robert Johnson from the Bureau of Meteorology for their feedback on this manuscript. We would like to thank and acknowledge the National Computational Infrastructure (nci.org.au) for hosting the `scores` repository within their GitHub organisation.