sphinx-build not scalable

[avivajpeyi]

sphinx-build taking several hours for ~2k notebooks
jupyter-book/jupyter-book#1571

[avivajpeyi]

Maybe we can try switching to sphinx-build: sphinx will allow more customisation in the building stage.

[avivajpeyi]

Ah! The issue must lie in the doctree pickles!

-d path
Since Sphinx has to read and parse all source files before it can write an output file, the parsed source files are cached as “doctree pickles”. Normally, these files are put in a directory called .doctrees under the build directory; with this option you can select a different cache directory (the doctrees can be shared between all builders).

https://www.sphinx-doc.org/en/master/man/sphinx-build.html

See _build/.doctrees

.
├── citation.doctree
├── content
│   ├── example_loader.doctree
│   ├── toi_fits.doctree
│   └── toi_notebooks
│       ├── toi_553.doctree
│       └── toi_560.doctree
├── environment.pickle
├── glue_cache.json
├── index.doctree
└── jupyter_execute
    ├── content
    │   └── toi_notebooks
    │       ├── toi_553.doctree
    │       └── toi_560.doctree

It'd be great if we could generate each of these independently...

[avivajpeyi]

Somewhat related:
https://stackoverflow.com/questions/51986357/how-can-i-make-sphinx-build-build-faster

[avivajpeyi]

Update: using -j 4 might have made it slightly faster? Still taking 4+ hours... hmm

[avivajpeyi]

Git issue about sphinx-build taking a long time: executablebooks/sphinx-book-theme#561

[avivajpeyi]

Ran it with 64GB ram, 6hrs

[avivajpeyi]

seems like it works with the slurm file

#SBATCH --job-name=webpage
#SBATCH --output=webpage.log
#
#SBATCH --ntasks=1
#SBATCH --time=10:00:00
#SBATCH --mem=64GB
#SBATCH --cpus-per-task=1

module load git/2.18.0 gcc/9.2.0 openmpi/4.0.2 python/3.8.5
source /fred/oz200/avajpeyi/envs/tess/bin/activate

sphinx-build -b html -j auto outdir outdir/_build

[avivajpeyi]

sphinx-build really isnt scalable -- it eats up more and more resources as we add in additional TOIs.

We should consider ditching this, generate individual pages, maybe serve them with django + use the sphinx-book theme CSS.

As i havent messed with Django since my undergrad days, this may take a some effort, but is doable.

[dfm]

We decided to use nbconvert to manually and statically convert each notebook instead of something heavy like Django