Avoid using python 3.12 for now

[bonnland]

Fixes #12

We are seeing an error with python 3.12.

The error comes from the intake-esm library.

This should resolve eventually as python 3.12 matures. See this post: pypa/pip#11501

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Cell In[11], line 2
      1 # Have the catalog interpret the "na-cordex-models" column as a list of values, as opposed to single values.
----> 2 col = intake.open_esm_datastore(catalog_url, read_csv_kwargs={"converters": {"na-cordex-models": ast.literal_eval}},)
      3 col

File /srv/conda/envs/notebook/lib/python3.12/site-packages/intake/__init__.py:66, in __getattr__(attr)
     64 if attr[:5] == "open_":
     65     if attr[5:] in registry.drivers.enabled_plugins():
---> 66         driver = registry[attr[5:]]  # "open_..."
     67         return driver
     68     else:

File /srv/conda/envs/notebook/lib/python3.12/site-packages/intake/source/__init__.py:32, in DriverRegistry.__getitem__(self, item)
     30 it = self.drivers.enabled_plugins()[item]
     31 if isinstance(it, entrypoints.EntryPoint):
---> 32     return it.load()
     33 elif isinstance(it, str):
     34     return import_name(it)

File /srv/conda/envs/notebook/lib/python3.12/site-packages/entrypoints.py:79, in EntryPoint.load(self)
     76 def load(self):
     77     """Load the object to which this entry point refers.
     78     """
---> 79     mod = import_module(self.module_name)
     80     obj = mod
     81     if self.object_name:

File /srv/conda/envs/notebook/lib/python3.12/importlib/__init__.py:90, in import_module(name, package)
     88             break
     89         level += 1
---> 90 return _bootstrap._gcd_import(name[level:], package, level)

File <frozen importlib._bootstrap>:1381, in _gcd_import(name, package, level)

File <frozen importlib._bootstrap>:1354, in _find_and_load(name, import_)

File <frozen importlib._bootstrap>:1304, in _find_and_load_unlocked(name, import_)

File <frozen importlib._bootstrap>:488, in _call_with_frames_removed(f, *args, **kwds)

File <frozen importlib._bootstrap>:1381, in _gcd_import(name, package, level)

File <frozen importlib._bootstrap>:1354, in _find_and_load(name, import_)

File <frozen importlib._bootstrap>:1325, in _find_and_load_unlocked(name, import_)

File <frozen importlib._bootstrap>:929, in _load_unlocked(spec)

File <frozen importlib._bootstrap_external>:994, in exec_module(self, module)

File <frozen importlib._bootstrap>:488, in _call_with_frames_removed(f, *args, **kwds)

File /srv/conda/envs/notebook/lib/python3.12/site-packages/intake_esm/__init__.py:6
      4 # Import intake first to avoid circular imports during discovery.
      5 import intake
----> 6 from pkg_resources import DistributionNotFound, get_distribution
      8 from . import tutorial
      9 from .core import esm_datastore

File /srv/conda/envs/notebook/lib/python3.12/site-packages/pkg_resources/__init__.py:2191
   2187     dist_groups = map(find_distributions, resolved_paths)
   2188     return next(dist_groups, ())
-> 2191 register_finder(pkgutil.ImpImporter, find_on_path)
   2193 if hasattr(importlib_machinery, 'FileFinder'):
   2194     register_finder(importlib_machinery.FileFinder, find_on_path)

AttributeError: module 'pkgutil' has no attribute 'ImpImporter'

[github-actions]

👋 Thanks for opening this PR! The Cookbook will be automatically built with GitHub Actions. To see the status of your deployment, click below.
🔍 Git commit SHA: 2bc9414
✅ Deployment Preview URL: https://ProjectPythia.github.io/na-cordex-viz-cookbook/_preview/13

[bonnland]

This notebook relies on Dask parallelism to be responsive. With four workers, it should complete in under ten minutes.

The GitHub build timeout value of 5 minutes is probably a bit too low for this notebook.

[bonnland]

I remember talking with @brian-rose about running this notebook with a larger timeout value, and performing the nightly build action only once a week.

The nightly build workflow is still set up to run on Sundays at midnight. I don't know how to increase the timeout value for this notebook, however.

[bonnland]

With lower data requirements, the notebook ran in just over 6 minutes. I don't know how the 5 minute timeout limit was circumvented this time, but...yay?

[brian-rose]

With lower data requirements, the notebook ran in just over 6 minutes. I don't know how the 5 minute timeout limit was circumvented this time, but...yay?

This cookbook is currently set to a one-hour timeout, see this line in the config file:

na-cordex-viz-cookbook/_config.yml

Line 13 in 2b07b14

timeout: 3600 # this cookbook needs longer than default run time

[bonnland]

Thanks for checking! I guess this timeout error from yesterday was a fluke then?

https://github.com/ProjectPythia/na-cordex-viz-cookbook/actions/runs/7280739398/job/19839811580

[brian-rose]

Maybe! The notebook execution part of the builds are running on the Pythia binder, not on GitHub actions. Sometimes the lengthiest part of a Binder run is building the image, which needs to happen if there are any changes the environment file (which is the case in this PR).

We're also aiming to have better support for dask workers on demand for bigger builds through this Binder, but we're not there yet.

[brian-rose]

It's possible that there are still some hidden timeout conditions being passed through binderbot (the command line tool we are using the route the notebook execution to the Binder). Fixing up that software is also a 2024 priority. I don't currently have a good explanation for why you got a 5-minute timeout in one case but not others.

[bonnland]

Okay, I understand. We have another failure data point now. I removed an unneeded "import pprint" in the notebook and we got another Binder build error. I'm pretty sure the notebook is working now, despite the error.

[brian-rose]

Hmm different error this time. I will see what the old "unplug and plug back in" cycle accomplishes.

[brian-rose]

Unrelated failure. Now Binder is looking for the repo under my user name:

2023-12-21 18:06:10 [info     ] Binder: Get https://binder.projectpythia.org/build/gh/brian-rose/na-cordex-viz-cookbook/fix-environment action=binder-start phase=get-launch-url
2023-12-21 18:06:11 [info     ] Binder: Build Failed Could not resolve ref for gh:brian-rose/na-cordex-viz-cookbook/fix-environment. Double check your URL and that your repo is public. GitHub recently changed default branches from "master" to "main".

This is a bug in our workflows. It shouldn't matter who clicked the "reopen" button.

I'll try to document this elsewhere.

@bonnland you might try your own "close and reopen" and see what happens.

[brian-rose]

Unrelated failure. Now Binder is looking for the repo under my user name:

2023-12-21 18:06:10 [info     ] Binder: Get https://binder.projectpythia.org/build/gh/brian-rose/na-cordex-viz-cookbook/fix-environment action=binder-start phase=get-launch-url
2023-12-21 18:06:11 [info     ] Binder: Build Failed Could not resolve ref for gh:brian-rose/na-cordex-viz-cookbook/fix-environment. Double check your URL and that your repo is public. GitHub recently changed default branches from "master" to "main".

This is a bug in our workflows. It shouldn't matter who clicked the "reopen" button.

I'll try to document this elsewhere.

@bonnland you might try your own "close and reopen" and see what happens.

Actually this failure is for the same reason as ProjectPythia/cookbook-actions#79

[brian-rose]

The failure now looks like

2023-12-21 18:18:05 [info     ] Binder: Build Failed Failed to create temporary user for ktyle/binder-dev-bonnland-2dna-2dcordex-2dviz-2dcookbook-66d65c:8a8dc18034a5c8ade71d7841ece57f446d518f0d

I got the same failure trying to run an interactive Binder session with a different repo. There's something wrong with the Binder currently, unrelated (as far as I can tell) to this PR.

I will check to @ktyle to see if we can find out what's up with the Binder.

[brian-rose]

It seems the Binder failure may be related to current maintenance on Jetstream2 where the Binder is running. @bonnland sorry for the multiple hiccups, but you should try closing and reopening again later.

[bonnland]

It looks like the checks passed. Should I merge to main, or let someone else when the timing is better?

[brian-rose]

Looks like our Binder is back in business! You should go ahead and merge if you're happy with the changes.

[bonnland]

It looks like the merge to main workflow is failing because python 3.12 is being used somewhere. The notebook environment specifies python <= 3.11 to avoid python 3.12. But it must still be used somewhere in the build actions.

[brian-rose]

Ugh. I think this is another bug in our workflows. Looks like the Binder is being pointed to the wrong branch and picking up the wrong environment file.

I'll try manually triggering a new publish workflow.

[brian-rose]

Ok, so the manual publish run seemed to work fine: https://github.com/ProjectPythia/na-cordex-viz-cookbook/actions/runs/7294855106

You should be all set. I think there are still a number of bugs in our workflow logic for handling changes in the environment files. Things are simpler for PRs that don't modify environment.yml. Thanks for your patience as we continue to iron this out.