Merge pull request #58 from databio/bedboss

Bedboss

README.md

@@ -1,29 +1,68 @@
-# Bedbase
+# BEDbase documentation
 
-## Introduction
+This repository hosts the documentation for BEDbase and related tools. Please visit <https://docs.bedbase.org> to view this rendered documentation.
 
-![Test BEDBASE stack](https://github.com/databio/bedbase/workflows/Test%20BEDBASE%20stack/badge.svg)
+Documentation is written using [mkdocs](https://www.mkdocs.org/) and themed with [material for mkdocs](https://squidfunk.github.io/mkdocs-material/). The repository has:
 
-A project to aggregate, analyze, and serve genomic regions better (*aka* BED files).
+- `mkdocs.yml`: the primary configuration, as well as the structure of the documentation. 
+- `/docs`: the markdown files. Each tool gets its own folder.
 
-## Services
+## How to write/update documentation
 
-- API, production: https://api.bedbase.org/
-- API, dev: https://api-dev.bedbase.org/
-- Front-end, production: https://bedbase.org (alias https://bedhost-ui.pages.dev/)
-- Front-end, dev: https://dev.bedbase.org
-- Object store, production: https://data2.bedbase.org/ - base URL for cloudflare/backblaze
+Each tool gets a `nav` section in `mkdocs.yml`, which maps to its own section/tab in the rendered documentation. So to add a new page, change titles, or change structure, edit `mkdocs.yml`. To edit the documentation itself, edit the `.md` documentation files in the subfolders under `/docs`.
 
-## Tutorial
+### Prereqs
 
-There's a tutorial for bedbase in the [docs_jupyter](/docs_jupyter) folder (probably outdated).
+```
+pip install mkdocs-material
+```
 
-## Components
 
-- [bedboss](https://github.com/databio/bedboss): Main BEDbase processing pipeline, combining bedqc, bedmaker, bedstat, and bedbuncher
-- [bbconf](http://github.com/databio/bbconf): BEDbase configuration
-- [bedhost](http://github.com/databio/bedhost): FastAPI application with API for accessing data
-- [bedhost-ui](http://github.com/databio/bedhost-ui): Front-end user interface built with React
-- [bedbase.org repository](https://github.com/databio/bedbase.org): Repository for deploying the bedhost container to AWS.
-- [all_geo_beds](all_geo_beds): A subfolder, is the scripts to download all bed files on GEO using geofetch and build a backend to host the metadata using bedstat
-- [geniml](https://github.com/databio/geniml): Machine learning for genomic intervals
+### Building locally for development
+
+I recommend previewing your changes locally before deploying. You can get a hot-reload server going by cloning this repository, and then just running:
+
+```
+mkdocs serve
+```
+
+You can also use `mkdocs build` to build a portable local version of the docs.
+
+
+### Publishing updates
+
+The documentation is published automatically upon commits to `master` using a GitHub Action, which runs `mkdocs gh-deploy`. This builds the docs, and pushes them to the `gh-pages` branch. This branch is then published with GitHub Pages. There's no need to do this locally, just let the action deploy the updates for you automatically.
+
+## FAQ
+
+
+### Updating automatic documentation
+
+In the past, I had a plugin that would auto-document 2 things: 1. Python docs using lucidoc, and 2. Jupyter notebooks. This plugin was neat, but it caused me a lot of maintenance issues as well. So now, I've made it much simpler; now it's no longer a plugin, just a simple Python script. Update all the auto-generated docs (stored in `docs/autodoc_build`) by running the update script manually:
+
+```console
+python update_python_autodocs.py
+```
+
+#### Configuring lucidoc rendering
+
+Auto-generated Python documentation with `lucidoc` rendering is configured in the `lucidoc`  sections of `mkdocs.yml`.
+
+```yaml
+lucidoc:
+  peppy: path/to/output.md
+```
+
+#### Configuring jupyter rendering
+
+Configure jupyter notebeeoks in the `jupyter` section, where you specify a list of `in` (for `.ipynb` files) and `out` (for `.md` files) locations.
+
+```yaml
+jupyter:
+  - in: path/to/notebook_folder1
+    out: path/to/rendered_folder1
+  - in: path/to/notebook_folder2
+    out: path/to/rendered_folder2
+``` 
+
+There, you can specify which folders contain notebooks, and to where they should be rendered as markdown.

autodoc.py

@@ -0,0 +1,102 @@
+# This script will auto-generate documentation for Python code, CLI usage, and Jupyter notebooks
+# It is intended to be run as a pre-build step in a MkDocs project
+# It will read the mkdocs.yml file for configuration
+# It will use the lucidoc package to auto-generate documentation for Python code
+# It will use the subprocess package to run CLI commands and capture the output
+# It will use the nbconvert package to convert Jupyter notebooks to markdown
+
+import lucidoc
+import yaml
+import subprocess
+import glob
+import nbconvert
+import os
+from pathlib import Path
+
+import argparse
+
+parser = argparse.ArgumentParser(description="Description of your program")
+parser.add_argument(
+    "--x-usage",
+    help="Exclude usage",
+    required=False,
+    default=False,
+    action="store_true",
+)
+parser.add_argument(
+    "--x-lucidoc",
+    help="Exclude lucidoc",
+    required=False,
+    default=False,
+    action="store_true",
+)
+parser.add_argument(
+    "--x-jupyter",
+    help="Exclude jupyter",
+    required=False,
+    default=False,
+    action="store_true",
+)
+
+args = vars(parser.parse_args())
+
+print(args)
+
+# Read the mkdocs config
+with open("mkdocs.yml") as stream:
+    cfg = yaml.safe_load(stream)
+
+
+if "autodoc" not in cfg:
+    print("No autodoc configuration found in mkdocs.yml")
+    exit(1)
+else:
+    cfg = cfg["autodoc"]
+
+# Process auto-documented Python code
+if args["x_lucidoc"] is False and "lucidoc" in cfg:
+    for bundle in cfg["lucidoc"]:
+        print(f"Documenting lucidoc '{bundle['pkg']}' at {bundle['outfile']}")
+        lucidoc.run_lucidoc(parse_style="rst", **bundle)
+else:
+    print("Skipping lucidoc")
+
+
+usage_tpl = """
+\n`{cmd}`
+\n
+```console
+{usage}
+```
+"""
+
+# Process CLI usage
+if args["x_usage"] is False and "cli_usage" in cfg:
+    for item in cfg["cli_usage"]:
+        result = ""
+        with open(item["template"], "r") as file:
+            result = file.read()
+        for cmd in item["commands"]:
+            print(f"Documenting command '{cmd}' to '{item['outfile']}'")
+            usage = subprocess.check_output(cmd, shell=True).decode("utf-8")
+            content = usage_tpl.format(cmd=cmd, usage=usage)
+            result += content
+        with open(item["outfile"], "w") as file:
+            file.write(result)
+else:
+    print("Skipping usage documentation")
+
+# # Render Juptyer notebooks to markdown
+if args["x_jupyter"] is False and "jupyter" in cfg:
+    for item in cfg["jupyter"]:
+        files = glob.glob(f"docs/{item['in']}/*.ipynb")
+        for nb in files:
+            bn, _ = os.path.splitext(os.path.basename(nb))
+            out = f"docs/{item['out']}/{bn}.md"
+            print(f"Converting '{nb}' to '{out}'")
+            md_result = nbconvert.exporters.export(nbconvert.HTMLExporter(), nb)[0]
+            Path(os.path.dirname(out)).mkdir(parents=True, exist_ok=True)
+            with open(out, "w") as stream:
+                stream.write(md_result)
+else:
+    print("Skipping jupyter notebooks")

docs/README.md

@@ -0,0 +1,12 @@
+---
+hide:
+  - navigation
+  - toc
+  - navigation.footer
+---
+
+# Welcome to <https://docs.bedbase.org>
+
+This site hosts developer and user documentation for components of BEDbase and related tools, notably including [geniml](geniml/README.md), our package for machine learning on genomic intervals. Use the tab navigation above to find the project of interest.
+
+You can access the main BEDbase interface at <https://bedbase.org>.