Decide how execution and cacheing fit into this

[choldgraf]

Currently, this CLI just does parsing and building with Sphinx. It doesn't handle any execution or cacheing. This means that currently, empty notebooks will be parsed as empty notebooks, and markdown files with computation in them will not be executed.

A few ways that we could do this:

• Use a command like jupyter-book run or that would point to a folder, and run its contents in-line
• Something similar that would also cache the contents in a jupyter-cache (at the root of the book folder?)
• Improvements to MyST-NB so that it knows how to look for a cache in inserting outputs as it parses the documents

I'm curious if @chrisjsewell thinks that jupyter-cache is ready to be used for these two use-cases above? If so, could you give some pointers for how you imagine this could be used? If I have a moment I can try to start making progress towards this as Chris finishes up the parsing improvements etc.

[chrisjsewell]

I'm curious if @chrisjsewell thinks that jupyter-cache is ready to be used for these two use-cases above?

Yeh I think so. From our discussion in executablebooks/jupyter-cache#16, it seems that people are OK with the general API now, just focussing on 'touch ups' to the CLI.

I'd imagine something like automatically adding all notebooks (+ .md/.myst) files from the _toc.yml (or a separate _exec.yml?) to the staging area (i.e. using the same API calls as 'jcache stage add-many'), then using the API calls from jcache execute in jupyter-book run, then MyST-NB would have the improvements to query the cache.

[choldgraf]

@chrisjsewell that sounds about right to me - one thing I wasn't able to figure out from playing around with jupyter-cache was how to access the cache. E.g., given a notebook path (relative to cache), how do I tell the cache "give me back the mimebundle for cell X in the notebook"?

[chrisjsewell]

• get_cache_bundle returns a notebook bundle (notebook + extra data), given a Primary Key in the database (mapped to a hash)
• match_cache_notebook/match_cache_file take a notebook, hashes it, and check if the hash is in the database. If so it returns a PK that can be used by the method above.
• merge_match_into_notebook / merge_match_into_notebook encapsulates both of the above, but also merges the cached notebook into the source notebook, returning a notebook then that contains both up-to-date text and code cells. This would be the best to use with MyST-NB
• There are no methods to return specific cells at present, since you have to load the whole notebook anyway; so its more efficient to use the above method, then get mime-bundles for all cells.

[choldgraf]

done via changes in myst-nb!