Should Jupyter Book simply pass-through configuration to Sphinx?

[choldgraf]

Right now, Jupyter Book has a YAML configuration that is a slight variation on Sphinx configuration. It is translated into a Sphinx conf.py-style config. This lets us do things like bundle a bunch of config values in a key. e.g, all the execution variables in space here:

https://github.com/executablebooks/jupyter-book/blob/master/jupyter_book/default_config.yml#L19

The downside to this is that it becomes complex to maintain - we are responsible for doing the minor mappings from Jupyter Book config to Sphinx config. It also becomes a bit awkward when you're depending on downstream extensions, as we'll have to document our own config docs for any "preferred" extensions we want to support.

Another approach we could take is to move away from "Jupyter Book-specific config" and instead to document how to directly add Sphinx config like so:

sphinx:
  config:
    sphinx_key_1: sphinx_val_2

In this case, we wouldn't actually be modifying any of the config values, we'd simply pass them through to Sphinx. I think this means we'd be at the wills of each extension author, but perhaps with the benefit of maintenance + docs simplicity? Maybe this would be better done for the "secondary extensions" that we depend on (especially the ones we don't control) but have custom config for the core MyST stuff like execute:?

Would be curious to hear what others think...

[rossbar]

I very much like this idea! I'm a huge fan of the executablebooks tooling, but have often used the "lower-level" tools (e.g. the myst-nb sphinx extension) for projects rather than jupyter-book. The main reason is that I am generally looking for a sphinx-based workflow, from which jupyter-book makes two significant departures: 1) the configuration and 2) the toc. I find myself having to look up the docs of these jupyterbook-specific layers whenever I'm starting a new project, which is the most significant hurdle in getting started on a new project.

Oftentimes I find that 90% of what I'm looking for (namely myst syntax support, text-based .md notebooks, and execution caching) is encapsulated in myst-nb, so I end up stopping there and using the myst-nb extension with sphinx rather than going the extra step and using jupyter-book (examples: numpy-tutorials, numpy-survey-results). On the other hand, I often find myself on the verge of opening an issue requesting that a feature that is available in jupyterbook be "moved down" one level so that it is available in myst-nb instead!

I think the ideal situation for my workflow would be if there were an option where jupyter-book could be used as a sphinx extension rather than a standalone project. I realize this is probably a larger-scope change than the original prompt, but moving more towards a sphinx-style configuration would definitely be an improvement for users w/ a sphinx-based workflow.

[choldgraf]

Thanks for this perspective!

Yeah I totally know what you mean about lower / higher in the stack. Though in a sense, this is the point of using Sphinx under the hood - precisely so that if you want full control, you've got that option.

Could you clarify what's a thing that you "wish existed in MyST-NB" that was instead in Jupyter Book?

I think for example it's important to let people configure via a YAML file instead of conf.py. If you want your tool to be language agnostic, you can't require a specific programming language to configure it (IMO).

I guess another approach would be to continue allowing these patterns, but to also set up the Jupyter Book infrastructure so that if someone structured their book like a Sphinx site (with in-page toctrees, as well as conf.py) then Sphinx could still build it. Not sure how to do that though 🤔

[rossbar]

Though in a sense, this is the point of using Sphinx under the hood - precisely so that if you want full control, you've got that option.

Agreed - and this was a very good choice IMO. Like I said, almost all of the functionality that I'd want is available via the myst-nb extension.

Could you clarify what's a thing that you "wish existed in MyST-NB" that was instead in Jupyter Book?

The main feature that comes to mind is the ability to open .md files directly in the browser with jupyter notebook. It seems like this only works if jupyter-book is installed in the environment? I might be wrong about this though.

I think for example it's important to let people configure via a YAML file instead of conf.py. If you want your tool to be language agnostic, you can't require a specific programming language to configure it (IMO).

I agree - I would not advocate for entirely removing/replacing the existing configuration style

I guess another approach would be to continue allowing these patterns, but to also set up the Jupyter Book infrastructure so that if someone structured their book like a Sphinx site (with in-page toctrees, as well as conf.py) then Sphinx could still build it. Not sure how to do that though

This is closer to what I was imagining as well, though I too am not sure how it could be accomplished/whether it would be worth the effort in the end.

[ericholscher]

I'm not 100% sure how the sphinx book config works currently, but it would be really great to have it closer to Sphinx. Given that Sphinx's config is just python, it would be really neat to be able to just write a normal sphinx config, add a jupyterbook extension, a jupyterbook_config value or something, and have it be able to "build" the same way.

Presumably you also want a cleaner way of handling this without a conf.py as well, to make the UX easier, but it would be really neat to have this integration style supported, to plug more into the existing Sphinx ecosystem. That would make both "jupyterbook-first" and "sphinx-first" integrations possible, perhaps with the advanced use-cases (eg. arbitrary existing sphinx extensions) only possible from the sphinx-style config.

Again, I don't fully understand how jupyterbook works, but that was just an idea I had for a possible way of handling this.

[ericholscher]

Another thought I just had, it would be cool if the YAML <-> conf.py extension was available for all users, not just jupyterbook users. It seems like that would actually be pretty easy to ship as an extension to Sphinx, which could then allow users who want to change the default config to manage it in YAML instead of Python. It could even raise a warning if the user had an conf.py in the spot where it would generate it on build.

[chrisjsewell]

it would be cool if the YAML <-> conf.py extension was available for all users

But how would you load this extension in the first place, i.e. how would you call sphinx-build such that it knows not to look for a conf.py?
Sounds like this would be better as an actual upstream PR to sphinx no?

[choldgraf]

Yeah I had the exact same thought @ericholscher , but I didn't know how to implement it in a way that would work with sphinx, per @chrisjsewell 's comments :-/

The only thing I imagined was you could call sphinx-build with a manually-passed configuration that'd simply load one extension like yaml_config, and that this would then load all of the other configuration. But that seemed quite hacky

[chrisjsewell]

I guess there really isn't much difference between the functionality available in myst-nb vs. jupyter-book.

Exactly; that was the goal of our modular approach 😄 once you get to jupyter-book you are mainly just tying all the modules together

[choldgraf]

I just opened up #291 to discuss creating a separate Sphinx extension for YAML config. The suggestion there I think feels a bit clunky still, so if anyone has suggestions for how to improve this, I'd love to hear it!