Merge pull request #188 from jmccreight/feat_run_param_sep

Feat run param sep
EC-USGS · Jun 30, 2023 · 256dfbf · 256dfbf
2 parents b86e4bb + 32d905e
commit 256dfbf
Show file tree

Hide file tree

Showing 134 changed files with 5,525 additions and 6,181 deletions.
diff --git a/.github/RELEASE.md b/.github/RELEASE.md
@@ -1,16 +1,26 @@
+#  -----------------------------------------------------------------------------
 # Release guide
 
-This document describes release procedures, conventions, and utilities for `pywatershed`.
+This document describes release procedures, conventions, and utilities for
+`pywatershed`.
 
 ## Conventions
 
-- Releases follow the [git flow](https://nvie.com/posts/a-successful-git-branching-model/).
+- Releases follow the
+  [git flow](https://nvie.com/posts/a-successful-git-branching-model/).
 - Release numbers follow [semantic version](https://semver.org/) conventions.
 - Minor and major releases branch from `develop`. Patches branch from `main`.
 
 ## Releasing `pywatershed`
 
-The release procedure is mostly automated. The workflow is defined in `.github/workflows/release.yaml` and triggers when a release or patch branch is pushed to this repo.
+The release procedure is mostly automated. The workflow is defined in
+`.github/workflows/release.yaml` and triggers when a release or patch branch is
+pushed to this repo.
+
+Prior to release:
+1. Run asv tests without asv, perform an asv regression test: 
+1. On develop update the `what's new.rst` to include  the date of the release
+
 
 To release a new version:
 
@@ -43,7 +53,7 @@ To release a new version:
     - Update `version.txt` and `pywatershed/version.py` to match the just-released version, with a '+' appended to the version number in `version.txt` to indicate preliminary/development status.
     - Draft a PR against `develop` with the updated version files and the updates previously merged to `main`.
 
- 5. Merge the PR to `develop`. As above, it is important to *merge* the PR, not squash, to preserve history and keep `develop` and `main` from diverging. 
+ 5. Merge the PR to `develop`. As above, it is important to *merge* the PR, not squash, to preserve history and keep `develop` and `main` from diverging.
 
  ## Utility scripts
 

diff --git a/.github/workflows/ci.yaml b/.github/workflows/ci.yaml
@@ -23,11 +23,6 @@ jobs:
       - name: Checkout repo
         uses: actions/checkout@v3
 
-      - name: Set environment variables
-        run: |
-          echo "PYWS_FORTRAN=true" >> $GITHUB_ENV
-          echo 'SETUPTOOLS_ENABLE_FEATURES="legacy-editable"' >> $GITHUB_ENV
-
       - name: Setup Python
         uses: actions/setup-python@v4
         with:

diff --git a/DEVELOPER.md b/DEVELOPER.md
@@ -1,11 +1,12 @@
 # Developing `pywatershed`
 
-This document describes how to set up a development environment, install dependencies, compile C/Fortran code, generate example data, and run the tests and example notebooks.
+This document describes how to set up a development environment, install
+dependencies, compile C/Fortran code, generate example data, and run the tests
+and example notebooks.
 
 <!-- START doctoc generated TOC please keep comment here to allow auto update -->
 <!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE -->
 
-
 - [Requirements](#requirements)
   - [Git](#git)
   - [Compilers](#compilers)
@@ -26,41 +27,48 @@ This document describes how to set up a development environment, install depende
 
 ### Git
 
-[Git](https://git-scm.com/) is required to contribute code back to the repository. GitHub's [Guide to Installing Git](https://help.github.com/articles/set-up-git) is a good source of information.
+[Git](https://git-scm.com/) is required to contribute code back to the
+repository. GitHub's
+[Guide to Installing Git](https://help.github.com/articles/set-up-git)
+is a good source of information.
 
 ### Compilers
 
-C and Fortran compilers are required. We are currently using gnu (gcc, gfortran)
-11 and 12 as well as intel (icc, ifort) 2021 on Windows, Linux, and MacOS
-(including Apple Silicon). Both of these are freely obtainable but the installation
-process varies widely. We are looking for a conda-based approach to obtaining
-compilers, but currently do not have a solution. Compilers are needed for
-two applications:
+C and Fortran compilers are required. We are currently using gnu (gcc,
+gfortran) 11 and 12 as well as intel (icc, ifort) 2021 on Windows, Linux, and
+MacOS (including Apple Silicon). Both of these are freely obtainable but the
+installation process varies widely. We are looking for a conda-based approach
+to obtaining compilers, but currently do not have a solution. Compilers are
+needed for two applications:
 
-  1. Compiling and running C/Fortran PRMS code to generate testing/verification data
-  2. Compiling (installing) and running fortran backends/kernels for some hydrological
-     process representations in pywatershed
+  1. Compiling and running C/Fortran PRMS code to generate testing/verification
+  data 2. Compiling (installing) and running fortran backends/kernels for some
+  hydrological process representations in pywatershed
 
-On Apple Silicon, the PRMS source code is only currently known to compile with intel while
-the fortran kernels in pywatershed only compile with gnu.
+On Apple Silicon, the PRMS source code is only currently known to compile with
+intel while the fortran kernels in pywatershed only compile with gnu.
 
 ### Python
 
 Python >= 3.9 is required.
 
 #### Creating a virtual environment
 
-A virtual Python environment is recommended, and can be created with Python's builtin `venv` or a conda distribution like Anaconda or Miniconda.
+A virtual Python environment is recommended, and can be created with Python's
+builtin `venv` or a conda distribution like Anaconda or Miniconda.
 
 ##### Conda
 
-To create a conda environment with all core, testing and optional dependencies, run from the project root:
+To create a conda environment with all core, testing and optional dependencies,
+run from the project root:
 
 ```
 conda env create -f environment.yml
 ```
 
-The conda `environment.yml` contains all dependencies needed to develop, test, and run example notebooks. More detailed Conda environment installation instructions can be found in `examples/00_python_virtual_env.ipynb`.
+The conda `environment.yml` contains all dependencies needed to develop, test,
+and run example notebooks. More detailed Conda environment installation
+instructions can be found in `examples/00_python_virtual_env.ipynb`.
 
 ##### Pip
 
@@ -70,28 +78,34 @@ To install all dependencies with `pip`:
 pip install ".[all]"
 ```
 
-Several  dependency groups are defined in `pyproject.toml` and can be selected instead of `all` for a more lightweight environment:
+Several  dependency groups are defined in `pyproject.toml` and can be selected
+instead of `all` for a more lightweight environment:
 
 - `lint`
 - `test`
 - `optional`
 
 #### Installing `pywatershed` in development mode
 
-Once the python environment and dependencies are established and activated (e.g. `conda activate pywatershed` or, assuming a virtual environment in `venv` in the root, `source venv/bin/activate`), `pywatershed` can be installed in [editable development mode](https://setuptools.pypa.io/en/latest/userguide/development_mode.html) with:
+Once the python environment and dependencies are established and activated
+(e.g. `conda activate pywatershed` or, assuming a virtual environment in `venv`
+in the root, `source venv/bin/activate`), `pywatershed` can be installed in
+[editable development
+mode](https://setuptools.pypa.io/en/latest/userguide/development_mode.html)
+with:
 
-```
-pip install -e .
-```
+``` pip install -e .  ```
 
 #### F2PY
 
-The numpy extension F2PY is used to provide fortran compiled kernels of core calculations to boost
-performance. F2PY is documented [within numpy](https://numpy.org/doc/stable/f2py/index.html). This
-repository is configured NOT to compile on install by default. Currently, we have not established
-this compilation procedure for Windows. On Linux and MacOS, compilation of fortran kernels on package
-installation is achieved by setting several environent variables before installing the `pywatershed` module.
-For instance, from the project root:
+The numpy extension F2PY is used to provide fortran compiled kernels of core
+calculations to boost performance. F2PY is documented [within
+numpy](https://numpy.org/doc/stable/f2py/index.html). This repository is
+configured NOT to compile on install by default. Currently, we have not
+established this compilation procedure for Windows. On Linux and MacOS,
+compilation of fortran kernels on package installation is achieved by setting
+several environent variables before installing the `pywatershed` module.  For
+instance, from the project root:
 
 ```
 export SETUPTOOLS_ENABLE_FEATURES="legacy-editable"
@@ -101,12 +115,15 @@ export PYWS_FORTRAN=true
 pip install -e .
 ```
 
+Note that an editable (`-e` above) is required to compile the fotran code.
+
 ## Testing
 
-Once the dependencies are available, we want to verify the software by running its test suite. The
-following testing procedures are also covered in the notebook `examples/01_automated_testing.ipynb`.
-To run the tests, we first need to generate the test data. This consists of running PRMS
-and then converting the output to netcdf. From `test_data/scripts`:
+Once the dependencies are available, we want to verify the software by running
+its test suite. The following testing procedures are also covered in the
+notebook `examples/01_automated_testing.ipynb`.  To run the tests, we first
+need to generate the test data. This consists of running PRMS and then
+converting the output to netcdf. From `test_data/scripts`:
 
 ```
 pytest -v -n=4 test_run_domains.py
@@ -115,24 +132,35 @@ pytest -v -n=8 test_nc_domains.py
 
 Finally, the tests can be run from the `autotest` directory:
 
-```
-pytest -v -n=8
-```
+``` pytest -v -n=8 ```
 
-All tests should pass, XPASS, or XFAIL. XFAIL is an expected failure. Substitute `-n auto` to automatically use all available cores on your machine.
+All tests should pass, XPASS, or XFAIL. XFAIL is an expected
+failure. Substitute `-n auto` to automatically use all available cores on your
+machine.
 
 ## Branching model
 
-This project uses the [git flow](https://nvie.com/posts/a-successful-git-branching-model/): development occurs on the `develop` branch, while `main` is reserved for the state of the latest release. Development PRs are typically squashed to `develop`, to avoid merge commits. At release time, release branches are merged to `main`, and then `main` is merged back into `develop`.
+This project uses the [git
+flow](https://nvie.com/posts/a-successful-git-branching-model/): development
+occurs on the `develop` branch, while `main` is reserved for the state of the
+latest release. Development PRs are typically squashed to `develop`, to avoid
+merge commits. At release time, release branches are merged to `main`, and then
+`main` is merged back into `develop`.
+
+## Documentation
+[Google-style docstrings](https://google.github.io/styleguide/pyguide.html#38-comments-and-docstrings)
+are used for documenting source code.
 
 ## Miscellaneous
 
 ### Locating the root
 
-Python scripts often need to reference files elsewhere in the project. To allow scripts to be run from anywhere in the project hierarchy, scripts should locate the project root relative to themselves, then use paths relative to the root for file access, rather than using relative paths (e.g., `../some/path`).
+Python scripts often need to reference files elsewhere in the project. To allow
+scripts to be run from anywhere in the project hierarchy, scripts should locate
+the project root relative to themselves, then use paths relative to the root
+for file access, rather than using relative paths (e.g., `../some/path`).
 
-For a script in a subdirectory of the root, for instance, the conventional approach would be:
+For a script in a subdirectory of the root, for instance, the conventional
+approach would be:
 
-```Python
-project_root_path = Path(__file__).parent.parent
-```
+```Python project_root_path = Path(__file__).parent.parent ```
diff --git a/asv_benchmarks/benchmarks/__init__.py b/asv_benchmarks/benchmarks/__init__.py
@@ -4,7 +4,13 @@
 
 print("sys.version: ", sys.version)
 
+# TODO remove backwards compatiability with pynhm once
+#      reported slows downs are sorted output
+# TODO remove backwards compatability with <0.2.0 once
+#      it is released.
+
 try:
+    # backwards compatability
     import pynhm as pws
 
     _is_pws = False
@@ -13,7 +19,6 @@
 
     _is_pws = True
 
-# this is in asv dev
 if not "constants" in pws.__dict__.keys():
     del pws
     import pywatershed as pws
@@ -22,7 +27,12 @@
 
 # The package is installed without data. The test data are relative to the repo
 # used to do the install. So use the asv env var to find that.
-asv_conf_dir = pl.Path(os.environ["ASV_CONF_DIR"]).resolve()
+try:
+    asv_conf_dir = pl.Path(os.environ["ASV_CONF_DIR"]).resolve()
+except KeyError:
+    asv_conf_dir = pl.Path(".")
+
+
 assert asv_conf_dir.exists()
 pws_root = asv_conf_dir / "../pywatershed"
 test_data_dir = asv_conf_dir / "../test_data"

diff --git a/asv_benchmarks/benchmarks/import.py → asv_benchmarks/benchmarks/imports.py b/asv_benchmarks/benchmarks/import.py → asv_benchmarks/benchmarks/imports.py
diff --git a/asv_benchmarks/benchmarks/prms.py b/asv_benchmarks/benchmarks/prms.py
@@ -2,14 +2,17 @@
 import shutil
 from typing import Union, Literal
 
-
 from . import _is_pws, parameterized, test_data_dir
 
 if _is_pws:
     import pywatershed as pws
 else:
     import pynhm as pws
 
+# TODO remove backwards compatiability with pynhm once
+#      reported slows downs are sorted output
+# TODO remove backwards compatability with <0.2.0 once
+#      it is released.
 
 domains = ["hru_1", "drb_2yr", "ucb_2yr"]
 outputs = [None, "separate", "together"]
@@ -73,13 +76,25 @@ def setup(self, *args):
 
         self.control_file = test_data_dir / f"{self.domain}/control.test"
         self.parameter_file = test_data_dir / f"{self.domain}/myparam.param"
-        print(f"model_setup_run tag: {self.tag}")
+
+        # backwards compatability pre pywatershed
         if _is_pws:
-            params = pws.parameters.PrmsParameters.load(self.parameter_file)
+            self.params = pws.parameters.PrmsParameters.load(
+                self.parameter_file
+            )
         else:
-            params = pws.PrmsParameters.load(self.parameter_file)
+            self.params = pws.PrmsParameters.load(self.parameter_file)
+
+        # backwards compatability
+        try:
+            self.control = pws.Control.load(
+                self.control_file, params=self.params
+            )
+            self.ge_v0_2_0 = False
+        except:
+            self.control = pws.Control.load(self.control_file)
+            self.ge_v0_2_0 = True
 
-        self.control = pws.Control.load(self.control_file, params=params)
         self.control.edit_n_time_steps(n_time_steps)
 
         # setup input_dir with symlinked prms inputs and outputs
@@ -104,13 +119,25 @@ def model_setup_run(
         processes: tuple = None,
         write_output: Union[bool, Literal["separate", "together"]] = None,
     ):
-        model = pws.Model(
-            *self.processes,
-            control=self.control,
-            input_dir=self.tag_input_dir,
-            budget_type="warn",
-            calc_method="numba",
-        )
+        if self.ge_v0_2_0:
+            model = pws.Model(
+                self.processes,
+                control=self.control,
+                discretization_dict=None,
+                parameters=self.params,
+                input_dir=self.tag_input_dir,
+                budget_type="warn",
+                calc_method="numba",
+            )
+        else:
+            model = pws.Model(
+                *self.processes,
+                control=self.control,
+                input_dir=self.tag_input_dir,
+                budget_type="warn",
+                calc_method="numba",
+            )
+
         if write_output is not None:
             model.initialize_netcdf(
                 self.tag_dir, separate_files=(write_output == "separate")
@@ -131,6 +158,13 @@ def time_prms_run(
         processes: tuple,
         output: Union[None, Literal["separate", "together"]],
     ):
+        print(
+            "\nPRMSModels args: \n",
+            f"domain: {domain}\n",
+            f"processes:{processes}\n",
+            f"output: {output}\n",
+        )
+
         _ = self.model_setup_run(
             domain=domain, processes=processes, write_output=output
         )
diff --git a/asv_benchmarks/run_benchmarks_wo_asv.py b/asv_benchmarks/run_benchmarks_wo_asv.py
@@ -0,0 +1,23 @@
+from itertools import product
+from benchmarks.prms import domains, outputs, model_tests, PRMSModels
+
+# Just running the PRMSModels benchmarks
+
+all_tests = list(
+    product(
+        *[
+            domains,
+            model_tests.values(),
+            outputs,
+        ]
+    )
+)
+
+
+all_tests = all_tests
+
+for args in all_tests:
+    mm = PRMSModels()
+    mm.setup(*args)
+    mm.time_prms_run(*args)
+    mm.teardown(*args)