Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Increase test coverage #99

Merged
merged 25 commits into from
Jul 5, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
25 commits
Select commit Hold shift + click to select a range
4b4734e
add test_get_jsonschema
dougiesquire Jun 27, 2023
80c0fe0
add test_utils
dougiesquire Jun 27, 2023
e694fa4
add test_cli
dougiesquire Jun 27, 2023
8997714
add test of _check_args
dougiesquire Jun 27, 2023
867f1f1
delete unused match_substring function
dougiesquire Jun 27, 2023
2e556d9
add test data
dougiesquire Jun 28, 2023
3f74664
add test_parse_access_ncfile
dougiesquire Jun 28, 2023
ea989e2
python version issue
dougiesquire Jun 29, 2023
4b0df02
allow files with no time dim
dougiesquire Jun 29, 2023
67b0d4c
pin fsspec below 2023.6.0
dougiesquire Jun 30, 2023
ecbb538
add Builder tests
dougiesquire Jun 30, 2023
946b8dd
use Path for windows?
dougiesquire Jun 30, 2023
ffc59cb
wait, why am I testing on windows and macos anyway?
dougiesquire Jun 30, 2023
b156392
pin pydantic<2.0
dougiesquire Jul 3, 2023
0e592a2
require ensemble argument on cm2 and esm1.5 Builders
dougiesquire Jul 3, 2023
7432b84
omit data module from code coverage
dougiesquire Jul 3, 2023
d5c9c59
omit data module from code coverage
dougiesquire Jul 3, 2023
02b2b07
add tests for translator funcs
dougiesquire Jul 4, 2023
4655cfe
add Translator tests
dougiesquire Jul 4, 2023
f9f7914
add Translator tests
dougiesquire Jul 4, 2023
e66f1fe
add DefaultTranslator test
dougiesquire Jul 5, 2023
236e4c2
add EriTranslator test
dougiesquire Jul 5, 2023
c83b528
add CatalogManager tests
dougiesquire Jul 5, 2023
66430ed
test manager errors
dougiesquire Jul 5, 2023
2edd089
add build test from configs
dougiesquire Jul 5, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
3 changes: 1 addition & 2 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -11,11 +11,10 @@ jobs:
- uses: pre-commit/[email protected]

test:
runs-on: ${{ matrix.os }}
runs-on: ubuntu-latest
strategy:
fail-fast: true
matrix:
os: ["ubuntu-latest", "windows-latest", "macos-latest"]
python-version: ["3.9", "3.10", "3.11"]

steps:
Expand Down
2 changes: 2 additions & 0 deletions ci/environment-3.10.yml
Original file line number Diff line number Diff line change
Expand Up @@ -7,11 +7,13 @@ dependencies:
- python==3.10
- cftime
- ecgtools>=2022.10.07
- fsspec<=2023.5.0 # see https://github.com/ncar-xdev/ecgtools/issues/160
- intake-dataframe-catalog>=0.1.1
- intake-esm>=2023.4.20
- jsonschema
- pooch
- pre-commit
- pydantic<2.0 # see https://github.com/intake/intake-esm/issues/617, https://github.com/ncar-xdev/ecgtools/issues /161
- pytest
- xarray
- pip
Expand Down
2 changes: 2 additions & 0 deletions ci/environment-3.11.yml
Original file line number Diff line number Diff line change
Expand Up @@ -7,11 +7,13 @@ dependencies:
- python==3.11
- cftime
- ecgtools>=2022.10.07
- fsspec<=2023.5.0 # see https://github.com/ncar-xdev/ecgtools/issues/160
- intake-dataframe-catalog>=0.1.1
- intake-esm>=2023.4.20
- jsonschema
- pooch
- pre-commit
- pydantic<2.0 # see https://github.com/intake/intake-esm/issues/617, https://github.com/ncar-xdev/ecgtools/issues/161
- pytest
- xarray
- pip
Expand Down
4 changes: 3 additions & 1 deletion ci/environment-3.9.yml
Original file line number Diff line number Diff line change
Expand Up @@ -4,14 +4,16 @@ channels:
- accessnri
- defaults
dependencies:
- python==3.9
- python>=3.9.2 # see https://github.com/dask/distributed/issues/7956
- cftime
- ecgtools>=2022.10.07
- fsspec<=2023.5.0 # see https://github.com/ncar-xdev/ecgtools/issues/160
- intake-dataframe-catalog>=0.1.1
- intake-esm>=2023.4.20
- jsonschema
- pooch
- pre-commit
- pydantic<2.0 # see https://github.com/intake/intake-esm/issues/617, https://github.com/ncar-xdev/ecgtools/issues /161
- pytest
- xarray
- pip
Expand Down
26 changes: 14 additions & 12 deletions config/access-cm2.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -4,46 +4,48 @@ translator: DefaultTranslator

sources:

- path:
- metadata_yaml: /g/data/p73/archive/non-CMIP/ACCESS-CM2/bx944/metadata.yaml
path:
- /g/data/p73/archive/non-CMIP/ACCESS-CM2/bx944
- /g/data/p73/archive/non-CMIP/ACCESS-CM2/bx944a
- /g/data/p73/archive/non-CMIP/ACCESS-CM2/bx944b
- /g/data/p73/archive/non-CMIP/ACCESS-CM2/bx944c
- /g/data/p73/archive/non-CMIP/ACCESS-CM2/bx944d
metadata_yaml: /g/data/p73/archive/non-CMIP/ACCESS-CM2/bx944/metadata.yaml
ensemble: true

- path:
- metadata_yaml: /g/data/p73/archive/non-CMIP/ACCESS-CM2/by647/metadata.yaml
path:
- /g/data/p73/archive/non-CMIP/ACCESS-CM2/by647
- /g/data/p73/archive/non-CMIP/ACCESS-CM2/by647a
- /g/data/p73/archive/non-CMIP/ACCESS-CM2/by647b
- /g/data/p73/archive/non-CMIP/ACCESS-CM2/by647c
- /g/data/p73/archive/non-CMIP/ACCESS-CM2/by647d
metadata_yaml: /g/data/p73/archive/non-CMIP/ACCESS-CM2/by647/metadata.yaml
ensemble: true

- path:
- metadata_yaml: /g/data/p73/archive/non-CMIP/ACCESS-CM2/by473/metadata.yaml
path:
- /g/data/p73/archive/non-CMIP/ACCESS-CM2/by473
- /g/data/p73/archive/non-CMIP/ACCESS-CM2/by473a
- /g/data/p73/archive/non-CMIP/ACCESS-CM2/by473b
- /g/data/p73/archive/non-CMIP/ACCESS-CM2/by473c
- /g/data/p73/archive/non-CMIP/ACCESS-CM2/by473d
metadata_yaml: /g/data/p73/archive/non-CMIP/ACCESS-CM2/by473/metadata.yaml
ensemble: true

- path:
- metadata_yaml: /g/data/p73/archive/non-CMIP/ACCESS-CM2/by578/metadata.yaml
path:
- /g/data/p73/archive/non-CMIP/ACCESS-CM2/by578
- /g/data/p73/archive/non-CMIP/ACCESS-CM2/by578a
- /g/data/p73/archive/non-CMIP/ACCESS-CM2/by578b
- /g/data/p73/archive/non-CMIP/ACCESS-CM2/by578c
- /g/data/p73/archive/non-CMIP/ACCESS-CM2/by578d
metadata_yaml: /g/data/p73/archive/non-CMIP/ACCESS-CM2/by578/metadata.yaml
ensemble: true

- path:
- metadata_yaml: /g/data/p73/archive/non-CMIP/ACCESS-CM2/bz687/metadata.yaml
path:
- /g/data/p73/archive/non-CMIP/ACCESS-CM2/bz687
metadata_yaml: /g/data/p73/archive/non-CMIP/ACCESS-CM2/bz687/metadata.yaml
ensemble: false

- path:
- metadata_yaml: /g/data/p73/archive/non-CMIP/ACCESS-CM2/cj877/metadata.yaml
path:
- /g/data/p73/archive/non-CMIP/ACCESS-CM2/cj877
metadata_yaml: /g/data/p73/archive/non-CMIP/ACCESS-CM2/cj877/metadata.yaml
ensemble: false
51 changes: 30 additions & 21 deletions config/access-esm1-5.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -4,43 +4,52 @@ translator: DefaultTranslator

sources:

- path:
- metadata_yaml: /g/data/p73/archive/non-CMIP/ACCESS-ESM1-5/PI-GWL-B2035/metadata.yaml
path:
- /g/data/p73/archive/non-CMIP/ACCESS-ESM1-5/PI-GWL-B2035
metadata_yaml: /g/data/p73/archive/non-CMIP/ACCESS-ESM1-5/PI-GWL-B2035/metadata.yaml
ensemble: false

- path:
- metadata_yaml: /g/data/p73/archive/non-CMIP/ACCESS-ESM1-5/PI-GWL-B2040/metadata.yaml
path:
- /g/data/p73/archive/non-CMIP/ACCESS-ESM1-5/PI-GWL-B2040
metadata_yaml: /g/data/p73/archive/non-CMIP/ACCESS-ESM1-5/PI-GWL-B2040/metadata.yaml
ensemble: false

- path:
- metadata_yaml: /g/data/p73/archive/non-CMIP/ACCESS-ESM1-5/PI-GWL-B2045/metadata.yaml
path:
- /g/data/p73/archive/non-CMIP/ACCESS-ESM1-5/PI-GWL-B2045
metadata_yaml: /g/data/p73/archive/non-CMIP/ACCESS-ESM1-5/PI-GWL-B2045/metadata.yaml
ensemble: false

- path:
- metadata_yaml: /g/data/p73/archive/non-CMIP/ACCESS-ESM1-5/PI-GWL-B2050/metadata.yaml
path:
- /g/data/p73/archive/non-CMIP/ACCESS-ESM1-5/PI-GWL-B2050
metadata_yaml: /g/data/p73/archive/non-CMIP/ACCESS-ESM1-5/PI-GWL-B2050/metadata.yaml
ensemble: false

- path:
- metadata_yaml: /g/data/p73/archive/non-CMIP/ACCESS-ESM1-5/PI-GWL-B2055/metadata.yaml
path:
- /g/data/p73/archive/non-CMIP/ACCESS-ESM1-5/PI-GWL-B2055
metadata_yaml: /g/data/p73/archive/non-CMIP/ACCESS-ESM1-5/PI-GWL-B2055/metadata.yaml
ensemble: false

- path:
- metadata_yaml: /g/data/p73/archive/non-CMIP/ACCESS-ESM1-5/PI-GWL-B2060/metadata.yaml
path:
- /g/data/p73/archive/non-CMIP/ACCESS-ESM1-5/PI-GWL-B2060
metadata_yaml: /g/data/p73/archive/non-CMIP/ACCESS-ESM1-5/PI-GWL-B2060/metadata.yaml
ensemble: false

- path:
- metadata_yaml: /g/data/p73/archive/non-CMIP/ACCESS-ESM1-5/HI-C-05-r1/metadata.yaml
path:
- /g/data/p73/archive/non-CMIP/ACCESS-ESM1-5/HI-C-05-r1
metadata_yaml: /g/data/p73/archive/non-CMIP/ACCESS-ESM1-5/HI-C-05-r1/metadata.yaml
ensemble: false

- path:
- metadata_yaml: /g/data/p73/archive/non-CMIP/ACCESS-ESM1-5/HI-CN-05/metadata.yaml
path:
- /g/data/p73/archive/non-CMIP/ACCESS-ESM1-5/HI-CN-05
metadata_yaml: /g/data/p73/archive/non-CMIP/ACCESS-ESM1-5/HI-CN-05/metadata.yaml
ensemble: false

- path:
- metadata_yaml: /g/data/p73/archive/non-CMIP/ACCESS-ESM1-5/HI-nl-C-05-r1/metadata.yaml
path:
- /g/data/p73/archive/non-CMIP/ACCESS-ESM1-5/HI-nl-C-05-r1
metadata_yaml: /g/data/p73/archive/non-CMIP/ACCESS-ESM1-5/HI-nl-C-05-r1/metadata.yaml
ensemble: false

- path:
- metadata_yaml: /g/data/p73/archive/non-CMIP/ACCESS-ESM1-5/HI-noluc-CN-05/metadata.yaml
path:
- /g/data/p73/archive/non-CMIP/ACCESS-ESM1-5/HI-noluc-CN-05
metadata_yaml: /g/data/p73/archive/non-CMIP/ACCESS-ESM1-5/HI-noluc-CN-05/metadata.yaml
ensemble: false
92 changes: 43 additions & 49 deletions config/access-om2.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -4,93 +4,87 @@ translator: DefaultTranslator

sources:

- path:
- metadata_yaml: /g/data/tm70/intake/metadata/1deg_jra55_ryf9091_gadi/metadata.yaml
path:
- /g/data/ik11/outputs/access-om2/1deg_jra55_ryf9091_gadi
metadata_yaml: /g/data/tm70/intake/metadata/1deg_jra55_ryf9091_gadi/metadata.yaml


- path:

- metadata_yaml: /g/data/tm70/intake/metadata/1deg_jra55_iaf_omip2_cycle1/metadata.yaml
path:
- /g/data/ik11/outputs/access-om2/1deg_jra55_iaf_omip2_cycle1
metadata_yaml: /g/data/tm70/intake/metadata/1deg_jra55_iaf_omip2_cycle1/metadata.yaml

- path:
- metadata_yaml: /g/data/tm70/intake/metadata/1deg_jra55_iaf_omip2_cycle2/metadata.yaml
path:
- /g/data/ik11/outputs/access-om2/1deg_jra55_iaf_omip2_cycle2
metadata_yaml: /g/data/tm70/intake/metadata/1deg_jra55_iaf_omip2_cycle2/metadata.yaml

- path:
- metadata_yaml: /g/data/tm70/intake/metadata/1deg_jra55_iaf_omip2_cycle3/metadata.yaml
path:
- /g/data/ik11/outputs/access-om2/1deg_jra55_iaf_omip2_cycle3
metadata_yaml: /g/data/tm70/intake/metadata/1deg_jra55_iaf_omip2_cycle3/metadata.yaml

- path:
- metadata_yaml: /g/data/tm70/intake/metadata/1deg_jra55_iaf_omip2_cycle4/metadata.yaml
path:
- /g/data/ik11/outputs/access-om2/1deg_jra55_iaf_omip2_cycle4
metadata_yaml: /g/data/tm70/intake/metadata/1deg_jra55_iaf_omip2_cycle4/metadata.yaml

- path:
- metadata_yaml: /g/data/tm70/intake/metadata/1deg_jra55_iaf_omip2_cycle5/metadata.yaml
path:
- /g/data/ik11/outputs/access-om2/1deg_jra55_iaf_omip2_cycle5
metadata_yaml: /g/data/tm70/intake/metadata/1deg_jra55_iaf_omip2_cycle5/metadata.yaml

- path:
- metadata_yaml: /g/data/tm70/intake/metadata/1deg_jra55_iaf_omip2_cycle6/metadata.yaml
path:
- /g/data/ik11/outputs/access-om2/1deg_jra55_iaf_omip2_cycle6
metadata_yaml: /g/data/tm70/intake/metadata/1deg_jra55_iaf_omip2_cycle6/metadata.yaml


- path:
- metadata_yaml: /g/data/tm70/intake/metadata/025deg_jra55_ryf9091_gadi/metadata.yaml
path:
- /g/data/ik11/outputs/access-om2-025/025deg_jra55_ryf9091_gadi
metadata_yaml: /g/data/tm70/intake/metadata/025deg_jra55_ryf9091_gadi/metadata.yaml


- path:
- metadata_yaml: /g/data/tm70/intake/metadata/025deg_jra55_iaf_omip2_cycle1/metadata.yaml
path:
- /g/data/ik11/outputs/access-om2-025/025deg_jra55_iaf_omip2_cycle1
metadata_yaml: /g/data/tm70/intake/metadata/025deg_jra55_iaf_omip2_cycle1/metadata.yaml

- path:
- metadata_yaml: /g/data/tm70/intake/metadata/025deg_jra55_iaf_omip2_cycle2/metadata.yaml
path:
- /g/data/ik11/outputs/access-om2-025/025deg_jra55_iaf_omip2_cycle2
metadata_yaml: /g/data/tm70/intake/metadata/025deg_jra55_iaf_omip2_cycle2/metadata.yaml

- path:
- metadata_yaml: /g/data/tm70/intake/metadata/025deg_jra55_iaf_omip2_cycle3/metadata.yaml
path:
- /g/data/ik11/outputs/access-om2-025/025deg_jra55_iaf_omip2_cycle3
metadata_yaml: /g/data/tm70/intake/metadata/025deg_jra55_iaf_omip2_cycle3/metadata.yaml

- path:
- metadata_yaml: /g/data/tm70/intake/metadata/025deg_jra55_iaf_omip2_cycle4/metadata.yaml
path:
- /g/data/ik11/outputs/access-om2-025/025deg_jra55_iaf_omip2_cycle4
metadata_yaml: /g/data/tm70/intake/metadata/025deg_jra55_iaf_omip2_cycle4/metadata.yaml

- path:
- metadata_yaml: /g/data/tm70/intake/metadata/025deg_jra55_iaf_omip2_cycle5/metadata.yaml
path:
- /g/data/ik11/outputs/access-om2-025/025deg_jra55_iaf_omip2_cycle5
metadata_yaml: /g/data/tm70/intake/metadata/025deg_jra55_iaf_omip2_cycle5/metadata.yaml

- path:
- metadata_yaml: /g/data/tm70/intake/metadata/025deg_jra55_iaf_omip2_cycle6/metadata.yaml
path:
- /g/data/ik11/outputs/access-om2-025/025deg_jra55_iaf_omip2_cycle6
metadata_yaml: /g/data/tm70/intake/metadata/025deg_jra55_iaf_omip2_cycle6/metadata.yaml


- path:
- metadata_yaml: /g/data/tm70/intake/metadata/01deg_jra55v13_ryf9091/metadata.yaml
path:
- /g/data/ik11/outputs/access-om2-01/01deg_jra55v13_ryf9091
metadata_yaml: /g/data/tm70/intake/metadata/01deg_jra55v13_ryf9091/metadata.yaml

- path:
- metadata_yaml: /g/data/tm70/intake/metadata/01deg_jra55v140_iaf/metadata.yaml
path:
- /g/data/cj50/access-om2/raw-output/access-om2-01/01deg_jra55v140_iaf
metadata_yaml: /g/data/tm70/intake/metadata/01deg_jra55v140_iaf/metadata.yaml

- path:
- metadata_yaml: /g/data/tm70/intake/metadata/01deg_jra55v140_iaf_cycle2/metadata.yaml
path:
- /g/data/cj50/access-om2/raw-output/access-om2-01/01deg_jra55v140_iaf_cycle2
metadata_yaml: /g/data/tm70/intake/metadata/01deg_jra55v140_iaf_cycle2/metadata.yaml

- path:
- metadata_yaml: /g/data/tm70/intake/metadata/01deg_jra55v140_iaf_cycle3/metadata.yaml
path:
- /g/data/cj50/access-om2/raw-output/access-om2-01/01deg_jra55v140_iaf_cycle3
metadata_yaml: /g/data/tm70/intake/metadata/01deg_jra55v140_iaf_cycle3/metadata.yaml

- path:
- metadata_yaml: /g/data/tm70/intake/metadata/01deg_jra55v140_iaf_cycle4/metadata.yaml
path:
- /g/data/cj50/access-om2/raw-output/access-om2-01/01deg_jra55v140_iaf_cycle4
metadata_yaml: /g/data/tm70/intake/metadata/01deg_jra55v140_iaf_cycle4/metadata.yaml

- path:
- metadata_yaml: /g/data/tm70/intake/metadata/01deg_jra55v140_iaf_cycle4_jra55v150_extension/metadata.yaml
path:
- /g/data/ik11/outputs/access-om2-01/01deg_jra55v140_iaf_cycle4_jra55v150_extension
metadata_yaml: /g/data/tm70/intake/metadata/01deg_jra55v140_iaf_cycle4_jra55v150_extension/metadata.yaml


- path:
- metadata_yaml: /g/data/tm70/intake/metadata/01deg_jra55v150_iaf_cycle1/metadata.yaml
path:
- /g/data/ik11/outputs/access-om2-01/01deg_jra55v150_iaf_cycle1
metadata_yaml: /g/data/tm70/intake/metadata/01deg_jra55v150_iaf_cycle1/metadata.yaml

8 changes: 4 additions & 4 deletions config/cmip5.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -4,10 +4,10 @@ translator: Cmip5Translator

sources:

- path:
- metadata_yaml: /g/data/tm70/intake/metadata/cmip5_al33/metadata.yaml
path:
- /g/data/dk92/catalog/v2/esm/cmip5-al33/catalog.json
metadata_yaml: /g/data/tm70/intake/metadata/cmip5_al33/metadata.yaml

- path:
- metadata_yaml: /g/data/tm70/intake/metadata/cmip5_rr3/metadata.yaml
path:
- /g/data/dk92/catalog/v2/esm/cmip5-rr3/catalog.json
metadata_yaml: /g/data/tm70/intake/metadata/cmip5_rr3/metadata.yaml
8 changes: 4 additions & 4 deletions config/cmip6.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -4,10 +4,10 @@ translator: Cmip6Translator

sources:

- path:
- metadata_yaml: /g/data/tm70/intake/metadata/cmip6_fs38/metadata.yaml
path:
- /g/data/dk92/catalog/v2/esm/cmip6-fs38/catalog.json
metadata_yaml: /g/data/tm70/intake/metadata/cmip6_fs38/metadata.yaml

- path:
- metadata_yaml: /g/data/tm70/intake/metadata/cmip6_oi10/metadata.yaml
path:
- /g/data/dk92/catalog/v2/esm/cmip6-oi10/catalog.json
metadata_yaml: /g/data/tm70/intake/metadata/cmip6_oi10/metadata.yaml
4 changes: 2 additions & 2 deletions config/erai.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,6 @@ translator: EraiTranslator

sources:

- path:
- metadata_yaml: /g/data/tm70/intake/metadata/erai/metadata.yaml
path:
- /g/data/hh5/public/apps/nci-intake-catalogue/erai/catalogue.json
metadata_yaml: /g/data/tm70/intake/metadata/erai/metadata.yaml
1 change: 1 addition & 0 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -58,6 +58,7 @@ addopts = "--cov=./src --cov-report=xml"
[tool.coverage.run]
omit = [
"src/access_nri_intake/_version.py",
"src/access_nri_intake/data/__init__.py",
]

[tool.ruff]
Expand Down
2 changes: 1 addition & 1 deletion src/access_nri_intake/catalog/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@
NAME_COLUMN = "name"
TRANSLATOR_GROUPBY_COLUMNS = ["model", "realm", "frequency"]

SCHEMA_URL = "https://raw.githubusercontent.com/ACCESS-NRI/schema/4e3d10e563d7c1c9f66e9ab92a2926cdec3d6893/experiment_asset.json"
SCHEMA_URL = "https://raw.githubusercontent.com/ACCESS-NRI/schema/d4da77a0e627775c11ba394c0a3f72a2c654971c/experiment_asset.json"
SCHEMA_HASH = "b18cf5bdd06a6f5bcdc71dfc80f7336c63eb49f6d6f75c2cd3371e59eee5488b"

EXP_JSONSCHEMA, CATALOG_JSONSCHEMA = get_jsonschema(
Expand Down
Loading
Loading