Merge branch 'master' into ncc/cookbook-docs

README.md

@@ -11,43 +11,71 @@
 
 # cosima-cookbook
 
-The COSIMA Cookbook is a framework for analysing output from ocean-sea ice models. The focus is on the ACCESS-OM2 suite of models being developed and run by members of [COSIMA: Consortium for Ocean-Sea Ice Modelling in Australia](http://cosima.org.au). But this framework is suited to analysing any MOM5/MOM6 output, as well as output from other models.
+The COSIMA Cookbook is a framework for analysing output from ocean-sea ice models. The focus is on
+the ACCESS-OM2 suite of models being developed and run by members of [COSIMA: Consortium for Ocean-Sea
+Ice Modelling in Australia](http://cosima.org.au). But this framework is suited to analysing any MOM5/MOM6
+output, as well as output from other models.
 
 The cookbook is structured as follows:
  * This repository includes boiler-plate code and scripts that underpin the cookbook.
- * The [`cosima-recipes`](https://github.com/COSIMA/cosima-recipes) repository includes example notebooks on which you can base your analyses.
- * The [`cosima-recipes` template](https://github.com/COSIMA/cosima-recipes/blob/master/Tutorials/Template_For_Notebooks.ipynb) provides you with a template if you want to contribute your own scripts to the analysis.
+ * The [`cosima-recipes`](https://github.com/COSIMA/cosima-recipes) repository includes example
+   notebooks on which you can base your analyses.
+ * The [`cosima-recipes` template](https://github.com/COSIMA/cosima-recipes/blob/master/Tutorials/Template_For_Notebooks.ipynb)
+   provides you with a template if you want to contribute your own scripts to the analysis.
 
 
 ## Getting Started
 
-The easiest way to use the COSIMA Cookbook is through NCI's HPC systems (either via the Australian Research Environment (ARE), Open OnDemand (OOD), or on Gadi). The cookbook is preinstalled in the latest `conda/analysis3` environment.
+The easiest way to use the COSIMA Cookbook is through NCI's HPC systems (either via the Australian
+Research Environment (ARE), Open OnDemand (OOD), or on Gadi). The cookbook is preinstalled in the
+latest `conda/analysis3` environment.
 
 Once you have an NCI account you can connect via
 
 
 - the Australian Research Environment (ARE) at https://are.nci.org.au
 
-Alternatively, you might prefer to download the `gadi_jupyter` scripts hosted in the CLEx CMS Github Repository [coecms/nci_scripts](https://github.com/coecms/nci_scripts). Running the script will open a Jupyter notebook in your local browser window.
+Alternatively, you might prefer to download the `gadi_jupyter` scripts hosted in the CLEx CMS Github
+Repository [coecms/nci_scripts](https://github.com/coecms/nci_scripts). Running the script will open
+a Jupyter notebook in your local browser window.
 
-**Note**: Access to COSIMA ocean-sea ice model output requires you are a member of NCI projects `hh5` and `ik11` and, potentially also of `cj50`, and `jk72`.
+**Note**: Access to COSIMA ocean-sea ice model output requires you are a member of NCI projects `hh5`
+and `ik11`, and potentially also `cj50` and `jk72`.
 
 
 ## Using the Cookbook
 
 The COSIMA Cookbook relies on several components:
 
-1. There needs to be a database of simulations -- on the NCI system, model output that is stored in the COSIMA space on the `/g/data/ik11/` directory.
-
-2. Once you have access to data, the best place to start is the [`cosima-recipes`](https://github.com/COSIMA/cosima-recipes) repository which includes a series of jupyter notebooks containing examples that guide you through to use the cookbook to load model output and then proceed doing simple (or elaborate) computations. The best starting point of exploring the [`cosima-recipes`](https://github.com/COSIMA/cosima-recipes) is the [Tutorials](https://cosima-recipes.readthedocs.io/en/latest/tutorials/index.html) and the [Documented Examples](https://cosima-recipes.readthedocs.io/en/latest/documented_examples/index.html). A collection of useful examples leveraging the `cosima-cookbook` that were used in the [ACCESS-OM2 model announcement paper](https://doi.org/10.5194/gmd-13-401-2020) can be found at [COSIMA/ACCESS-OM2-1-025-010deg-report](https://github.com/COSIMA/ACCESS-OM2-1-025-010deg-report/tree/master/figures) repository.
+1. There needs to be a database of simulations -- on the NCI system, model output that is stored in the COSIMA
+   space on the `/g/data/ik11/` directory.
 
+2. Once you have access to data, the best place to start is the [`cosima-recipes`](https://github.com/COSIMA/cosima-recipes)
+   repository which includes a series of jupyter notebooks containing examples that guide you through to use the cookbook to
+   load model output and then proceed doing simple (or elaborate) computations. The best starting point of exploring the
+   [`cosima-recipes`](https://github.com/COSIMA/cosima-recipes) is the [Tutorials](https://cosima-recipes.readthedocs.io/en/latest/tutorials/index.html)
+   and the [Documented Examples](https://cosima-recipes.readthedocs.io/en/latest/documented_examples/index.html). A collection
+   of useful examples leveraging the `cosima-cookbook` that were used in the [ACCESS-OM2 model announcement paper](https://doi.org/10.5194/gmd-13-401-2020)
+   can be found at [COSIMA/ACCESS-OM2-1-025-010deg-report](https://github.com/COSIMA/ACCESS-OM2-1-025-010deg-report/tree/master/figures)
+   repository.
 
 ## Contributing to the Cookbook
 
 If you like the cookbook, you may like to interact more closely with us:
- * Contributions of new notebooks or analysis scripts are always welcome. Please check out the [`cosima-recipes`](https://github.com/COSIMA/cosima-recipes) repository.
+ * Contributions of new notebooks or analysis scripts are always welcome. Please check out the
+   [`cosima-recipes`](https://github.com/COSIMA/cosima-recipes) repository.
  * If you find a problem, or have a suggestion for improvement, please log an issue.
  * All code submitted as part of the `cosima-cookbook` itself must be formatted with [black](https://github.com/psf/black)
 
+## Conditions of use for ACCESS-OM2 data
+
+We request that users of ACCESS-OM2 model [code](https://github.com/COSIMA/access-om2) or output data:
+1. consider citing Kiss et al. (2020) ([http://doi.org/10.5194/gmd-13-401-2020](http://doi.org/10.5194/gmd-13-401-2020))
+2. include an acknowledgement such as the following:
+
+   > The authors thank the Consortium for Ocean-Sea Ice Modelling in Australia (COSIMA; [http://www.cosima.org.au](http://www.cosima.org.au)) for making the ACCESS-OM2 suite of models available at [https://github.com/COSIMA/access-om2](https://github.com/COSIMA/access-om2).*
+
+3. let us know of any publications which use these models or data so we can add them to
+   [our list](https://scholar.google.com/citations?hl=en&user=inVqu_4AAAAJ).
 
 [![Documentation Status](https://readthedocs.org/projects/cosima-cookbook/badge/?version=latest)](https://cosima-cookbook.readthedocs.org/en/latest)

conda/meta.yaml

@@ -26,7 +26,7 @@ requirements:
         - netcdf4
         - joblib
         - tqdm
-        - sqlalchemy
+        - sqlalchemy<2.0
         - ipywidgets
         - cftime>1.2.1
         - lxml

cosima_cookbook/database.py

@@ -844,7 +844,6 @@ def build_index(
 
     indexed = 0
     for directory in [Path(d) for d in directories]:
-
         expt = find_experiment(session, directory)
         if expt is None:
             expt = NCExperiment(

cosima_cookbook/diagnostics/overturning.py

@@ -225,7 +225,6 @@ def op(p):
 
 @memory.cache
 def zonal_mean(expt, variable, n=10, resolution=1):
-
     zonal_var = get_nc_variable(
         expt,
         "ocean.nc",

cosima_cookbook/diagnostics/simple.py

@@ -21,7 +21,6 @@ def annual_scalar(expt, variables):
     annual_average = darray.resample(time="A").mean("time")
 
     for v in annual_average.data_vars:
-
         avar = annual_average.variables[v]
         dvar = darray.variables[v]
         avar.attrs["long_name"] = dvar.attrs["long_name"] + " (annual average)"

cosima_cookbook/explore.py

@@ -25,7 +25,6 @@ def return_value_or_empty(value):
 
 
 class DatabaseExtension:
-
     # DEPRECATED
 
     def __init__(self, session, experiments=None):
@@ -147,7 +146,6 @@ def _update_selector(self, variables):
         for vals in variables.sort_values(["name"])[
             ["name", "long_name", "units"]
         ].values:
-
             var, name, units = map(str, vals)
 
             if firstvar is None:
@@ -329,7 +327,6 @@ class VariableSelectorInfo(VariableSelector):
     def __init__(
         self, parent, variables, daterange, frequency, cellmethods, rows=10, **kwargs
     ):
-
         # The cellmethods widget needs access to the session and experiment
         self.session = parent.session
         self.experiment = parent.experiment_name
@@ -586,7 +583,6 @@ class DatabaseExplorer(VBox):
     variables = None
 
     def __init__(self, session=None, de=None):
-
         if session is None:
             session = database.create_session()
         self.session = session
@@ -611,7 +607,6 @@ def __init__(self, session=None, de=None):
         self._set_handlers()
 
     def _make_widgets(self):
-
         style = "<style>.header p{ line-height: 1.4; margin-bottom: 10px }</style>"
 
         # Gui header
@@ -850,15 +845,13 @@ def data(self):
 
 
 class ExperimentExplorer(VBox):
-
     session = None
     _loaded_data = None
     experiment_name = None
     variables = None
     experiments = None
 
     def __init__(self, session=None, experiment=None):
-
         if session is None:
             session = database.create_session()
         self.session = session

cosima_cookbook/netcdf_index.py

@@ -201,7 +201,6 @@ def build_index(use_bag=False, careful=False, expt_dir_list=None):
         )
 
         def index_variables(ncfile):
-
             matched = find_output.match(ncfile)
             if matched is None:
                 return []

cosima_cookbook/plots/lineplots.py

@@ -69,7 +69,6 @@ def annual_scalar(expts=[], variables=[]):
 
     # plotting each variable in a separate plot
     for variable in variables:
-
         plt.figure(figsize=(12, 6))
 
         for result in results:

cosima_cookbook/plots/overturning.py

@@ -7,7 +7,6 @@
 
 
 def psi_avg(expts, n=10, clev=np.arange(-20, 20, 2)):
-
     if not isinstance(expts, list):
         expts = [expts]
 
@@ -66,7 +65,6 @@ def psi_avg(expts, n=10, clev=np.arange(-20, 20, 2)):
 
 
 def zonal_mean(expts, variable, n=10, resolution=1):
-
     if not isinstance(expts, list):
         expts = [expts]
 

cosima_cookbook/querying.py

@@ -160,7 +160,6 @@ def get_variables(
     ]
 
     if experiment:
-
         # Create aliases so as to able to join to the NCAttribute table
         # twice, for the name and value
         ncas1 = aliased(NCAttributeString)
@@ -520,7 +519,7 @@ def _ncfiles_for_variable(
             warnings.warn(
                 f"Your query returns variables from files with different {attr}: {unique_attributes}. "
                 "This could lead to unexpected behaviour! Disambiguate by passing "
-                f"attrs={{'{attr}'=''}} to getvar, specifying the desired attribute value.",
+                f"attrs={{'{attr}':''}} to getvar, specifying the desired attribute value.",
                 QueryWarning,
             )
 

docs/source/getting_started.rst

@@ -55,6 +55,6 @@ in the COSIMA recipes repository which demonstrates the available tools.
 Tutorials and examples
 ======================
 
-COSIMA recipes provides `tutorials <https://cosima-recipes.readthedocs.io/en/latest/tutorials/index.html>`_
-and `documented examples <https://cosima-recipes.readthedocs.io/en/latest/documented_examples/index.html>`_ 
+COSIMA recipes provides `tutorials <https://cosima-recipes.readthedocs.io/en/latest/tutorials.html>`_
+and `documented examples <https://cosima-recipes.readthedocs.io/en/latest/documented_examples.html>`_ 
 which can be used to learn how to use the Cookbook and for ideas and inspiration for your own analysis.

setup.py

@@ -18,7 +18,7 @@
         'bokeh',
         'netcdf4',
         'tqdm',
-        'sqlalchemy',
+        'sqlalchemy<2.0',
         'cftime',
         'f90nml',
         'joblib',

test/test_dates.py

@@ -71,7 +71,6 @@ def teardown_module(module):
 
 
 def test_format_parse_datetime():
-
     dates = [
         cftime.num2date(t, units="days since 01-01-01", calendar="noleap")
         for t in times
@@ -94,7 +93,6 @@ def test_format_parse_datetime():
 
 
 def test_rebase_times():
-
     # Should be a 10 year offset between original times and rebased times
     assert not np.any(
         (times + 365 * 10)
@@ -113,7 +111,6 @@ def test_rebase_times():
 
 
 def test_rebase_variable():
-
     timesvar = xr.DataArray(
         times, attrs={"units": "days since 1980-01-01", "calendar": "noleap"}
     )
@@ -154,7 +151,6 @@ def test_rebase_variable():
 
 
 def test_matching_time_units():
-
     testfile = "test/data/ocean_sealevel.nc"
 
     ds = xr.open_dataset(testfile, decode_times=False)
@@ -213,7 +209,6 @@ def test_matching_time_units():
 
 
 def test_chunking():
-
     # An offset is required as the target units are ahead of the data in time
     target_units = "days since 2000-01-01"
 

test/test_explore.py

@@ -68,7 +68,6 @@ def session(tmp_path_factory):
 
 
 def test_database_explorer(session):
-
     dbx = cc.explore.DatabaseExplorer(session=session)
 
     assert dbx.session is session
@@ -129,7 +128,6 @@ def test_database_explorer(session):
 
 
 def test_experiment_explorer(session):
-
     ee1 = cc.explore.ExperimentExplorer(session=session)
 
     # Experiment selector
@@ -169,7 +167,6 @@ def test_experiment_explorer(session):
 
 
 def test_get_data(session):
-
     ee = cc.explore.ExperimentExplorer(session=session)
 
     assert ee.data is None
@@ -190,7 +187,6 @@ def test_get_data(session):
 
 
 def test_model_property(session):
-
     # Grab all variables and ensure the SQL classification matches the python version
     # May be some holes, as not ensured all cases covered
     for expt in cc.querying.get_experiments(session, all=True).experiment:

test/test_indexing.py

@@ -21,6 +21,14 @@ def rm_tree(pth):
     pth.rmdir()
 
 
+def assert_dictionaries_same(expected, actual):
+    for key in expected.keys():
+        if key not in actual or expected[key] != actual[key]:
+            return False
+
+    return True
+
+
 @pytest.fixture
 def unreadable_dir(tmp_path):
     expt_path = tmp_path / "expt_dir"
@@ -36,7 +44,6 @@ def unreadable_dir(tmp_path):
 
 
 def test_find_files():
-
     files = database.find_files("test/data/indexing/")
     assert len(files) == 17
 
@@ -351,11 +358,14 @@ def test_index_attributes(session_db):
     database.build_index("test/data/querying", session)
 
     inspector = inspect(session.get_bind())
-    assert inspector.get_indexes("ncattributes")[0] == {
-        "name": "ix_ncattributes_ncvar_id",
-        "column_names": ["ncvar_id"],
-        "unique": 0,
-    }
+    assert assert_dictionaries_same(
+        {
+            "name": "ix_ncattributes_ncvar_id",
+            "column_names": ["ncvar_id"],
+            "unique": 0,
+        },
+        inspector.get_indexes("ncattributes")[0],
+    )
 
     ncfile = "output000/ocean.nc"
 
@@ -388,16 +398,6 @@ def test_index_attributes(session_db):
         assert attr in v.attrs and v.attrs[attr] == attr_val
 
 
-def test_distributed(client, session_db):
-    session, db = session_db
-    database.build_index("test/data/indexing/broken_file", session, client)
-
-    assert db.exists()
-    q = session.query(database.NCExperiment)
-    r = q.all()
-    assert len(r) == 1
-
-
 def test_prune_broken(session_db):
     session, db = session_db
     database.build_index("test/data/indexing/broken_file", session)

test/test_metadata.py

@@ -305,7 +305,6 @@ def test_get_experiments_with_keywords(session_db):
 
 
 def test_getvar_with_metadata(session_db):
-
     session, db = session_db
     database.build_index("test/data/indexing/metadata", session)