♻️ REFACTOR: Move to markdown-it + mdformat renderer (#18)

Conversion now procedes via conversion to markdown-it tokens, followed by conversion to Markdown text with mdformat.

.github/workflows/tests.yml

@@ -25,7 +25,7 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        python-version: [3.6, 3.7, 3.8, 3.9]
+        python-version: [3.7, 3.8, 3.9]
         os: [ubuntu-latest, windows-latest]
 
     steps:
@@ -48,6 +48,7 @@ jobs:
         pytest --cov=rst_to_myst --cov-report=xml --cov-report=term-missing
 
     - name: Upload to Codecov
+      if: matrix.os == 'ubuntu-latest'
       uses: codecov/codecov-action@v1
       with:
         name: pytests

.pre-commit-config.yaml

@@ -1,6 +1,6 @@
 repos:
 - repo: https://github.com/pre-commit/pre-commit-hooks
-  rev: v3.4.0
+  rev: v4.0.1
   hooks:
   - id: end-of-file-fixer
   - id: mixed-line-ending
@@ -12,19 +12,19 @@ repos:
   - id: check-yaml
   - id: check-toml
 - repo: https://github.com/pre-commit/pygrep-hooks
-  rev: v1.7.0
+  rev: v1.9.0
   hooks:
   - id: python-check-blanket-noqa
 - repo: https://github.com/timothycrosley/isort
-  rev: 5.6.4
+  rev: 5.8.0
   hooks:
   - id: isort
 - repo: https://github.com/psf/black
-  rev: 20.8b1
+  rev: 21.6b0
   hooks:
   - id: black
 - repo: https://gitlab.com/pycqa/flake8
-  rev: 3.8.4
+  rev: 3.9.2
   hooks:
   - id: flake8
     additional_dependencies:

.readthedocs.yml

@@ -0,0 +1,13 @@
+version: 2
+
+python:
+  version: 3
+  install:
+      - method: pip
+        path: .
+        extra_requirements:
+          - docs
+
+sphinx:
+  builder: html
+  fail_on_warning: true

README.md

@@ -1,4 +1,4 @@
-# rst-to-myst [UNDER-DEVELOPMENT]
+# rst-to-myst
 
 [![Build Status][ci-badge]][ci-link]
 [![codecov.io][cov-badge]][cov-link]
@@ -21,179 +21,34 @@ or with sphinx:
 pip install rst-to-myst[sphinx]
 ```
 
-## Basic Usage
-
-### Command-Line Interface (CLI)
-
-For all commands see:
-
-```bash
-rst2myst --help
-```
-
-Parse *via* stdin:
-
-```console
-$ echo ":role:`content`" | rst2myst parse
-{role}`content`
-```
-
-Parse *via* file:
-
-```console
-$ rst2myst parse -f path/to/file.rst
-...
-```
-
-Warnings are written to `stderr` and converted text to `stdout`.
-
-List available directives/roles:
-
-```console
-$ rst2myst directives list
-acks admonition ...
-
-$ rst2myst roles list
-abbr abbreviation ...
-```
-
-Show details of a specific directive/role:
-
-```console
-$ rst2myst directives show admonition
-class: docutils.parsers.rst.directives.admonitions.Admonition
-description: ''
-has_content: true
-name: admonition
-optional_arguments: 0
-options:
-  class: class_option
-  name: unchanged
-required_arguments: 1
-
-$ rst2myst roles show abbreviation
-description: |-
-  Generic interpreted text role, where the interpreted text is simply
-  wrapped with the provided node class.
-module: docutils.parsers.rst.roles
-name: abbreviation
-```
-
-### Python Interface (API)
-
-```python
-from rst_to_myst import convert
-
-text, stderr_stream = convert("""
-Some RST
-========
-
-To **convert**
-""")
-```
-
-## Advanced Usage
-
-You can select a language to translate directive/role names:
+To then run a basic conversion of a whole project:
 
 ```console
-$ rst2myst parse -l fr -f path/to/file.rst
-...
+$ rst2myst convert docs/**/*.rst
 ```
 
-You can select whether sphinx directives/roles are loaded:
+For greater control, you can pass configuration with CLI options, or via a YAML configuration file:
 
 ```console
-$ rst2myst parse --no-sphinx -f path/to/file.rst
-...
+$ rst2myst convert --config config.yaml docs/**/*.rst
 ```
 
-You can load directives/roles from extensions:
-
-```console
-$ rst2myst parse -e sphinx.ext.autodoc -e sphinx_panels -f path/to/file.rst
-...
+`config.yaml`:
+
+```yaml
+language: en
+sphinx: true
+extensions:
+- sphinx_panels
+default_domain: py
+consecutive_numbering: true
+colon_fences: true
+dollar_math: true
+conversions:
+    sphinx_panels.dropdpwn.DropdownDirective: parse_all
 ```
 
-Directives are converted according to [rst_to_myst/data/directives.yml](rst_to_myst/data/directives.yml), which can also be updated with an external YAML file, using the `-c/--conversions` option.
-This is a mapping of directive import paths to a conversion type:
-
-- "eval_rst" (the default): no conversion, wrap in MyST eval_rst directive
-  ````
-  ```{eval_rst}
-  .. name:: argument `link`_
-     :option: value
-
-     content `link`_
-  ```
-  ````
-- "direct": convert directly to MyST directive, keeping original argument/content
-  ````
-  ```{name} argument `link`_
-  :option: value
-
-  content `link`_
-  ```
-  ````
-- "argument_only":  convert to MyST directive and convert the argument to Markdown
-  ````
-  ```{name} argument [link](link)
-  :option: value
-
-  content `link`_
-  ```
-  ````
-- "content_only":  convert to MyST directive and convert the content to Markdown
-  ````
-  ```{name} argument `link`_
-  :option: value
-
-  content [link](link)
-  ```
-  ````
-- "argument_content":  convert to MyST directive and convert the content to Markdown
-  ````
-  ```{name} argument [link](link)
-  :option: value
-
-  content [link](link)
-  ```
-  ````
-
-If a conversion type is prepended by "_colon", use `:::` delimiters instad of ```` ``` ````,
-e.g. "argument_content_colon"
-
-````
-:::{name} argument [link](link)
-:option: value
-
-content [link](link)
-:::
-````
-
-## Conversion Notes
-
-The conversion is designed to be fault tolerant,
-i.e. it will not check if referenced targets, roles, directives, etc exist nor fail if they do not.
-
-The only syntax where some checks are required is matching anonymous references and auto-number/symbol footnotes with their definitions; these definitions must be available.
-
-- enumerated lists with roman numerals or alphabetic prefixes will be converted to numbers
-- only one kind of footnote (i.e. no symbol prefixes)
-- citation are turned into footnotes, with label prepended by `cite_prefix`
-- inline targets are not convertible (and so ignored)
-- If tables are not compatible with Markdown (single header row, no merged cells, etc), then they will be wrapped in an `eval_rst`
-- Markdown blockquotes do not have an attribution syntax, so it is converted instead to `<p class="attribution">—text</p>` (the standard HTML render)
-
-## TODO
-
-The conversion covers almost all syntaxes (see <https://docutils.sourceforge.io/docs/user/rst/quickref.htm>) except:
-
-- line blocks
-- field lists (except at top of document, which are converted to front matter)
-- option lists
-
-Also custom functions for directive parsing would be desirable.
+See the documentation for more information.
 
 ## Development
 
@@ -230,6 +85,15 @@ or trigger the GitHub Action job, by creating a release with a tag equal to the
 
 Note, this requires generating an API key on PyPi and adding it to the repository `Settings/Secrets`, under the name `PYPI_KEY`.
 
+## TODO
+
+The conversion covers almost all syntaxes (see <https://docutils.sourceforge.io/docs/user/rst/quickref.htm>) except:
+
+- line blocks
+- option lists
+
+Also custom functions for directive parsing would be desirable.
+
 [ci-badge]: https://github.com/executablebooks/rst-to-myst/workflows/CI/badge.svg?branch=main
 [ci-link]: https://github.com/executablebooks/rst-to-myst/actions?query=workflow%3ACI+branch%3Amain+event%3Apush
 [cov-badge]: https://codecov.io/gh/executablebooks/rst-to-myst/branch/main/graph/badge.svg

docs/source/api.rst

@@ -0,0 +1,29 @@
+Python API
+==========
+
+Text to docutils AST
+--------------------
+
+.. autofunction:: rst_to_myst.parser.to_docutils_ast
+
+docutils AST to Markdown-It Tokens
+-----------------------------------
+
+.. autoclass:: rst_to_myst.markdownit.RenderOutput
+    :members:
+
+.. autoclass:: rst_to_myst.markdownit.MarkdownItRenderer
+    :members:
+
+Markdown-It Tokens to Text
+--------------------------
+
+.. autofunction:: rst_to_myst.mdformat_render.from_tokens
+
+Full Conversion
+---------------
+
+.. autoclass:: rst_to_myst.mdformat_render.ConvertedOutput
+    :members:
+
+.. autofunction:: rst_to_myst.mdformat_render.rst_to_myst

docs/source/cli.rst

@@ -0,0 +1,6 @@
+CLI Commands
+============
+
+.. click:: rst_to_myst.cli:main
+    :prog: rst2myst
+    :nested: full

docs/source/conf.py

@@ -0,0 +1,46 @@
+"""Configuration for Sphinx documentation build.
+
+It is recommended to use tox to run the build (see tox.ini):
+`tox -e docs-clean` and `tox -e docs-update`,
+or directly: `sphinx-build -n -W --keep-going docs/source docs/_build`
+"""
+from rst_to_myst import __version__
+
+project = "RST-to-MyST"
+copyright = "2021, Executable Book Project"  # noqa: A001
+author = "Executable Book Project"
+version = __version__
+
+extensions = [
+    # read Markdown files
+    "myst_parser",
+    "sphinx_panels",
+    # document CLI
+    "sphinx_click",
+    # document API
+    "sphinx.ext.autodoc",
+    "sphinx.ext.intersphinx",
+    "sphinx.ext.viewcode",
+]
+
+html_theme = "sphinx_book_theme"
+html_title = f"RST-to-MyST: v{__version__}"
+html_theme_options = {
+    "home_page_in_toc": True,
+    "github_url": "https://github.com/executablebooks/rst-to-myst",
+    "repository_url": "https://github.com/executablebooks/rst-to-myst",
+    "use_issues_button": True,
+    "use_repository_button": True,
+    "repository_branch": "main",
+    "path_to_docs": "docs",
+}
+
+intersphinx_mapping = {
+    "python": ("https://docs.python.org/3.8", None),
+    "sphinx": ("https://www.sphinx-doc.org/en/master", None),
+    "markdown_it": ("https://markdown-it-py.readthedocs.io/en/latest", None),
+}
+
+nitpick_ignore = [
+    ("py:class", name) for name in ["IO", "_io.StringIO", "docutils.nodes.document"]
+]