Skip to content

Commit

Permalink
Update Sphinx and documentation (#3710)
Browse files Browse the repository at this point in the history
Co-authored-by: Jeroen Dekkers <[email protected]>
Co-authored-by: Jan Klopper <[email protected]>
  • Loading branch information
3 people authored Oct 24, 2024
1 parent 5f4bfe1 commit 39b92e7
Show file tree
Hide file tree
Showing 96 changed files with 535 additions and 1,778 deletions.
7 changes: 7 additions & 0 deletions .github/workflows/build_docs_on_pr.yml
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,14 @@ name: Build docs artifact for PR
on:
pull_request:
paths:
# We generate documentation for boefje, bytes, keiko, mula and octopoes
# from code so the workflow should also depend on it.
- "boefje/**"
- "bytes/**"
- "docs/**"
- "keiko/**"
- "mula/**"
- "octopoes/**"
- "requirements.txt"
- ".github/workflows/build_docs_on_pr.yml"

Expand Down
9 changes: 8 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -437,7 +437,14 @@ nl-kat-*
/_version.py

# Automatically generated markdown files for the environment variables
/docs/source/technical_design/environment_settings/*.md
docs/source/installation-and-deployment/environment-settings/boefjes.md
docs/source/installation-and-deployment/environment-settings/bytes.md
docs/source/installation-and-deployment/environment-settings/keiko.md
docs/source/installation-and-deployment/environment-settings/mula.md
docs/source/installation-and-deployment/environment-settings/octopoes.md

docs/source/_static/d3.min.js
docs/source/_static/mermaid.min.js

# rpki cache
/boefjes/boefjes/plugins/kat_rpki/rpki.json
Expand Down
3 changes: 2 additions & 1 deletion .pre-commit-config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -188,5 +188,6 @@ repos:
\.json$ |
\.min\.js$ |
^rocky/assets/css/themes/soft/fonts |
^rocky/assets/vendors
^rocky/assets/vendors |
^docs/source/_static
)
13 changes: 10 additions & 3 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -12,12 +12,14 @@ UNAME := $(shell uname)
export DOCKER_BUILDKIT=1
export COMPOSE_DOCKER_CLI_BUILD=1

# We can't really return an error here, so if settings-doc fails we delete the
# file which will result in sphinx-build returning an error later on
define build-settings-doc
echo "# $(4)" > docs/source/installation_and_deployment/environment_settings/$(3).md
echo "# $(4)" > docs/source/installation-and-deployment/environment-settings/$(3).md
DOCS=True PYTHONPATH=./$(1) settings-doc generate \
-f markdown -m $(2) \
--templates docs/settings-doc-templates \
>> docs/source/installation_and_deployment/environment_settings/$(3).md
>> docs/source/installation-and-deployment/environment-settings/$(3).md || exit 1
endef


Expand Down Expand Up @@ -114,7 +116,12 @@ docs:
$(call build-settings-doc,bytes,bytes.config,bytes,Bytes)
$(call build-settings-doc,mula/scheduler,config.settings,mula,Mula)

PYTHONPATH=$(PYTHONPATH):boefjes/:bytes/:keiko/:mula/:octopoes/ sphinx-build -b html docs/source docs/_build
curl -sL -o - https://registry.npmjs.org/d3/-/d3-7.9.0.tgz | tar -Oxzf - package/dist/d3.min.js > docs/source/_static/d3.min.js
curl -sL -o - https://registry.npmjs.org/mermaid/-/mermaid-11.3.0.tgz | tar -Oxzf - package/dist/mermaid.min.js > docs/source/_static/mermaid.min.js
echo "f2094bbf6141b359722c4fe454eb6c4b0f0e42cc10cc7af921fc158fceb86539 docs/source/_static/d3.min.js" | sha256sum --quiet --check || exit 1
echo "0d2b6f2361e7e0ce466a6ed458e03daa5584b42ef6926c3beb62eb64670ca261 docs/source/_static/mermaid.min.js" | sha256sum --quiet --check || exit 1

PYTHONPATH=$(PYTHONPATH):boefjes/:bytes/:keiko/:mula/:octopoes/ sphinx-build -b html --fail-on-warning docs/source docs/_build


poetry-dependencies:
Expand Down
1,580 changes: 0 additions & 1,580 deletions docs/source/_static/mermaid-9.4.3.min.js

This file was deleted.

12 changes: 10 additions & 2 deletions docs/source/conf.py
Original file line number Diff line number Diff line change
Expand Up @@ -59,5 +59,13 @@
html_static_path = ["_static"]
html_css_files = ["openkat.css"]

mermaid_version = "" # Do not fetch from the CDN
html_js_files = ["mermaid-9.4.3.min.js"]
mermaid_use_local = "mermaid.min.js"
mermaid_include_elk = ""
d3_use_local = "d3.min.js"

autosectionlabel_prefix_document = True

suppress_warnings = [
f"autosectionlabel.installation-and-deployment/environment-settings/{document}"
for document in ("boefjes", "bytes", "keiko", "mula", "octopoes")
]
Original file line number Diff line number Diff line change
Expand Up @@ -265,7 +265,7 @@ having been implemented in these PRs:
- https://github.com/minvws/nl-kat-coordination/pull/2709
- https://github.com/minvws/nl-kat-coordination/pull/2832

#### Summary of decisions
### Summary of decisions

We decided not to focus on the following:

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
This module has several entry points discussed below, but let us first consider the prerequisites and scope.
If you already have a running setup and want to learn where each bit of functionality goes, read the following page:

[Developing Openkat Plugins](../introduction/makeyourown.rst)
[Developing Openkat Plugins](../introduction/make-your-own.rst)

## Prerequisites

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -77,7 +77,7 @@ After having done all that. The user should be able to create their own Greeting

Let's go to the reports tab, and change our filters again so we can see our Greetings OOI. Check one of their boxes and press the _Continue with selection_ button. Now in the grid of available report types you can make, you should see 2 options. The Findings Report and the Greetings Report. Let's check them both and press once more on the _Continue with selection_ button. Now you will see a report that includes both the Findings Report and the Greetings report inside a single web page. In the top right, you can press the _Export_ button which will make a pdf of your report. Including information about every finding of your project.

If you want to get more information inside your report, make sure to check our [technical report documentation](https://docs.openkat.nl/developer_documentation/rocky.html#testing) about reports.
If you want to get more information inside your report, make sure to check our documentation about {ref}`developer-documentation/reports:Creating reports`.

## Conclusion

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -24,9 +24,9 @@ Glossary
:caption: Contents
:maxdepth: 1

creating_a_boefje
testing_the_boefje
creating_a_new_model
creating_a_normalizer
creating_a_bit
creating_a_report
creating-boefje
testing-boefje
creating-model
creating-normalizer
creating-bit
creating-report
Original file line number Diff line number Diff line change
Expand Up @@ -14,8 +14,8 @@ Contains documentation for developers and contributors.
normalisers-runner
bytes
octopoes
octopoes_models
octopoes_research
octopoes-models
octopoes-research
keiko
reports
development_tutorial/index
development-tutorial/index
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,7 @@ template is started. After the report is generated, the PDF is available at `/re
## Logging

Keiko logging can be configured through by supplying a `logging.json`, relative to the current working directory of the
process. See [logging.json](logging.json) for an example.
process. See `keiko/logging.json` for an example.

## Building a new template

Expand Down
File renamed without changes.
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ _Status: Draft_

The current workflow is as follows:

```mermaid
```{mermaid}
graph LR;
Mula--Tasks-->Boefjes;
Boefjes--Information-->Bytes;
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -413,7 +413,7 @@ python3 octopoes/tools/xtdb-cli.py -n "aa" history --with-docs "BIT_METRIC" |jq

If you want to query the XTDB database directly, you can use the XTDB-tool. This is explained on the following page: https://docs.openkat.nl/developer_documentation/octopoes.html#xtdb-cli-tool.

## OOI
## OOI Objects

OOI objects are instances of relatively simple classes, which inherit from `OOI`.

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ class YourNameReport(Report):
```

5. Open `reports/report_types/helpers.py` and add your new class to the `REPORTS` constant list.
6. Implement a method within `report.py` to gather the required data for report generation. See the [Collecting data](#collecting-data) section for more information.
6. Implement a method within `report.py` to gather the required data for report generation. See the {ref}`developer-documentation/reports:Collecting data` section for more information.
7. Design the HTML structure for your report within `report.html`. The generated data from `report.py` can be used with Django Template in this file. For example by referring to the returned value like `{{ data }}`.
8. Save your changes and refresh the page to see the changes you made immediately.

Expand All @@ -43,7 +43,7 @@ The `generate_data` method only works on single OOIs and should not be used in n
Use all existing reports as examples to gather data for your report.

- In the file `rocky/reports/report_types/definitions.py` you can find some methods that may be useful.
- For querying data from Octopoes, consult `octopoes/octopoes/connector/octopoes.py` which contains various useful methods. Additional information on how to write queries can be found [here](https://docs.openkat.nl/developer_documentation/octopoes.html#querying).
- For querying data from Octopoes, consult `octopoes/octopoes/connector/octopoes.py` which contains various useful methods. Additional information on how to write queries can be found {ref}`here <developer-documentation/octopoes:Querying>`.

## Writing report unit tests

Expand Down Expand Up @@ -91,4 +91,4 @@ def test_my_new_report_multiple_results(mock_octopoes_api_connector, valid_time,

### Executing unit tests

Information about how to execute unit tests can be found [here](https://docs.openkat.nl/developer_documentation/rocky.html#testing).
Information about how to execute unit tests can be found {ref}`here <rocky-testing>`.
Original file line number Diff line number Diff line change
Expand Up @@ -98,6 +98,8 @@ $ make build-rocky-frontend

## Development

(rocky-testing)=

### Testing

To run all unit tests, run:
Expand Down
8 changes: 4 additions & 4 deletions docs/source/guidelines/contributions.rst
Original file line number Diff line number Diff line change
Expand Up @@ -18,15 +18,15 @@ Contribute to Codebase
======================


See :ref:`Development` for our code style, coding conventions, and overall workflow.
See :doc:`/guidelines/development` for our code style, coding conventions, and overall workflow.

- Fork the right repository in GitHub
- Create a new branch from either ``main`` or a release tag. Note that ``main`` changes rapidly, and as such may not be a suitable basis for your work.
- This branch should be in the following format:
- ``[feature|enhancement|bug|hotfix]/random-cat-popup-on-screen``
- Commit and push the code
- Make sure the code is linted, formatted and has correct typing. Use ``pre-commit`` locally for this, see :ref:`Pre-commit`.
- All commits must be signed, see :ref:`Signed commits`.
- Make sure the code is linted, formatted and has correct typing. Use ``pre-commit`` locally for this, see :ref:`guidelines/development:Pre-commit`.
- All commits must be signed, see :ref:`guidelines/development:Signed commits`.
- Submit Pull Request
- Make sure your code is tested and the PR has a good title and description
- Use the PR template
Expand Down Expand Up @@ -98,7 +98,7 @@ The new language should be automatically picked up by both Weblate and Django.

Contributor Social Contract
===========================
All contributors (including, but not limited to, developers and issue reporters) promise to do their best to adhere to the guidelines in :ref:`Project Guidelines`.
All contributors (including, but not limited to, developers and issue reporters) promise to do their best to adhere to the guidelines in :doc:`/guidelines/index`.
Everyone is encouraged to politely and constructively point out guidelines violations to others.
Actively enforcing these guidelines makes that the entire project benefits in quality control.

Expand Down
4 changes: 2 additions & 2 deletions docs/source/guidelines/development.rst
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ Developers are encouraged to write their code as strictly compliant as possible.
Tools
=====

To make development and validation easier, we adopted :ref:`pre-commit` hooks to automate most of this.
To make development and validation easier, we adopted :ref:`guidelines/development:pre-commit` hooks to automate most of this.
This will help identify broken/bad code, improve consistency and save time during code reviews.
Some tools and hooks have been adopted for both local development as well as in our CI/CD pipeline as GitHub actions.

Expand Down Expand Up @@ -113,7 +113,7 @@ It has a simple syntax and many plugins available that should improve our test c
Development Environment
=======================

See :ref:`Installation and deployment` for the overall installation instructions.
See :doc:`/installation-and-deployment/index` for the overall installation instructions.
In a development context, we strongly recommend to use the Docker setup to test and make changes in the codebase (and not production packages).

When it comes to development there is no specific IDE that must be used, although many of us would choose PyCharm as the preferred IDE.
Expand Down
4 changes: 2 additions & 2 deletions docs/source/guidelines/management.rst
Original file line number Diff line number Diff line change
Expand Up @@ -26,15 +26,15 @@ When a cycle has been completed, we hold a quick retrospective to evaluate what
Bugs and Feature Requests
=========================

For effective bug reporting and feature requests there are :ref:`GitHub Templates`.
For effective bug reporting and feature requests there are :doc:`/templates/index`.
These should be used and submitted in the coordination repository's `issues board <https://github.com/minvws/nl-kat-coordination/issues>`_.
Please make sure to link them to the central project board as an incoming feature/refinement.


Pull Requests
=============

Each unit of work shall be submitted as a pull request using a :ref:`Pull Request Template` and reviewed by at least one developer.
Each unit of work shall be submitted as a pull request using a :doc:`/templates/pull_request_template_author` and reviewed by at least one developer.
The checklist should be completed by a functional and a code reviewer.

In-depth content discussions
Expand Down
8 changes: 4 additions & 4 deletions docs/source/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -10,9 +10,9 @@ Welcome to the OpenKAT documentation!
introduction/index
manual/index
modules/index
installation_and_deployment/index
developer_documentation/index
ux_design/index
release_notes/index
installation-and-deployment/index
developer-documentation/index
ux-design/index
release-notes/index
guidelines/index
templates/index
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ OpenKAT has Debian packages available. In the near future we will have an apt
repository that will allow you to keep your installation up-to-date using apt.
An installation of KAT can be done on a single machine or spread out on several
machines for a high availability setup. This guide will take you through the
steps for installing it on a single machine. There are also :ref:`scripts<Scripts>`
steps for installing it on a single machine. There are also :doc:`/installation-and-deployment/scripts`
available if you don't want to do this by hand.

Supported distributions
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -80,4 +80,4 @@ Check in the user interface if the users have permission to perform scans and ar

The current usermodel also needs a superuser that is part of an organization. Normally this is set automagically. With several organizations in your instance the superuser might end up alone. This must be corrected through the Django interface, in which the superuser can be added to the organization.

You can reach the Django admin interface through ``/admin`` on the rocky instance. While you are there, do check the :ref:`Production: Hardening OpenKAT` page if you have not already done so.
You can reach the Django admin interface through ``/admin`` on the rocky instance. While you are there, do check the :doc:`/installation-and-deployment/hardening` page if you have not already done so.
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
Installation and deployment
Installation and Deployment
###########################

Contains documentation for developers and contributors.
Expand All @@ -9,15 +9,15 @@ Contains documentation for developers and contributors.

install
containers
debianinstall
debian-install
scripts
hardening
localinstall
windowsinstall
local-install
windows-install
gitpod
nginx
debuggingtroubleshooting
debugging-troubleshooting
latex
environment_settings/index
externalauthentication
environment-settings/index
external-authentication
cveapi
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ Scripts
There are some scripts in the `scripts/installation
<https://github.com/minvws/nl-kat-coordination/tree/main/scripts/installation>`__ directory
that can be used to install and update OpenKAT on Debian and will do all the
steps described on the :ref:`Debian installation<Production: Debian packages>` page.
steps described on the :doc:`/installation-and-deployment/debian-install` page.

Installation
------------
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -106,4 +106,4 @@ Now you can start OpenKat.
Troubleshooting
================

If you encounter any problems, please check `OpenKAT Troubleshooting <https://docs.openkat.nl/installation_and_deployment/index.html>`_.
If you encounter any problems, please check :doc:`/installation-and-deployment/index` for more information.
File renamed without changes.
4 changes: 2 additions & 2 deletions docs/source/introduction/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -8,5 +8,5 @@ Contains introduction into OpenKAT
:caption: Contents

intro
howdoesitwork
makeyourown
how-does-it-work
make-your-own
6 changes: 3 additions & 3 deletions docs/source/introduction/intro.rst
Original file line number Diff line number Diff line change
Expand Up @@ -37,14 +37,14 @@ The nicest playground for OpenKAT is a situation where many systems are active.
Where do I start with OpenKAT?
==============================

The documentation explains how the system works and what the main principles are. This gives an impression, but after reading the documentation, trying it yourself is the best way to find out how OpenKAT works. There are `several options to install OpenKAT <https://docs.openkat.nl/installation_and_deployment/index.html>`_.
The documentation explains how the system works and what the main principles are. This gives an impression, but after reading the documentation, trying it yourself is the best way to find out how OpenKAT works. There are :doc:`/installation-and-deployment/index`

The easiest way to get to know the system is a local installation. If you don't have a debian or ubuntu machine (yet), try the Gitpod test environment. `The installation chapter <https://docs.openkat.nl/installation_and_deployment/index.html>`_ has a comprehensive roadmap for creating a local installation. In addition to the documentation, read `the readme from the general repository <https://github.com/minvws/nl-kat-coordination>`_.
The easiest way to get to know the system is a local installation. If you don't have a debian or ubuntu machine (yet), try the Gitpod test environment. :doc:`/installation-and-deployment/install` has a comprehensive roadmap for creating a local installation. In addition to the documentation, read `the readme from the general repository <https://github.com/minvws/nl-kat-coordination>`_.

Where is the software located?
==============================

OpenKAT consists of separate modules that each perform a specific task. All modules are located in the `NL-KAT-Coordination <https://github.com/minvws/nl-kat-coordination>`_ repository. The :ref:`Modules of OpenKAT<modules>` section of the documentation goes into detail on each of these modules.
OpenKAT consists of separate modules that each perform a specific task. All modules are located in the `NL-KAT-Coordination <https://github.com/minvws/nl-kat-coordination>`_ repository. The :doc:`/modules/modules` section of the documentation goes into detail on each of these modules.

Responsible disclosure
======================
Expand Down
File renamed without changes.
File renamed without changes
2 changes: 1 addition & 1 deletion docs/source/manual/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,6 @@ An overview of all KAT functionality, from a user perspective.
:maxdepth: 4
:caption: Contents

usermanual
user-manual
reports
normalizers
Loading

0 comments on commit 39b92e7

Please sign in to comment.