Merge pull request #107 from epics-containers/ibek-j20-changes

changes to add new support functions as per #106

.github/workflows/code.yml

@@ -37,7 +37,7 @@ jobs:
         # Make one version be non-editable to test both paths of version code
         include:
           - os: "ubuntu-latest"
-            python: "3.8"
+            python: "3.9"
             install: ".[dev]"
 
     runs-on: ${{ matrix.os }}

README.rst

@@ -1,41 +1,113 @@
-ibek
-===========================
-
-|code_ci| |docs_ci| |coverage| |pypi_version| |license|
-
-IOC Builder for EPICS and Kubernetes:
-
-- In an EPICS support module describe what entities an IOC using it can create,
-  what arguments they take, and what database and st.cmd snippets it should
-  generate in a ``builder.yaml`` file
-- Build support modules together in a container image and use ``ibek`` in the
-  image to create a JSON schema of what an IOC using that image can contain
-- Write an ``ioc.yaml`` file against that schema listing instances of the
-  entities with arguments
-- Use ``ibek`` to generate a startup script and database that runs
-  up the IOC contained in the image with them
-
-============== ==============================================================
-PyPI           ``pip install ibek``
-Source code    https://github.com/epics-containers/ibek
-Documentation  https://epics-containers.github.io/ibek
-Releases       https://github.com/epics-containers/ibek/releases
-============== ==============================================================
-
-TODO
-----
-
-This project is approaching completion. The following items are still to do:
-
-    from ibek import __version__
-
-    print(f"Hello ibek {__version__}")
-
-    - DONE: Add ability to define embedded objects e.g. AsynIp and AsynSerial would
-      both be defined by embedding AsynPort (so similar to how original builder.py
-      works)
-
-    $ python -m ibek --version
+ibek: IOC Builder for EPICS and Kubernetes
+==========================================
+
+|code_ci| |docs_ci| |coverage| |pypi_version| |black| |license|
+
+
+This tool supports building container images for Generic EPICS IOCs and
+generating IOC Instances from those Generic IOCs at container runtime. It is
+targetted at running IOCs in Kubernetes, but the images it
+creates can execute in any container runtime, such as docker or podman.
+
+  ============== ==============================================================
+  PyPI           ``pip install ibek``
+  Source code    https://github.com/epics-containers/ibek
+  Documentation  https://epics-containers.github.io/ibek
+  Releases       https://github.com/epics-containers/ibek/releases
+  ============== ==============================================================
+
+
+The documentation is still under construction. For the moment the best
+pages to read are the following:
+
+- `Modules, Definitions and Entities <https://epics-containers.github.io/ibek/main/developer/explanations/entities.html//>`_
+  A description of the basic concepts of ibek.
+
+- `naming conventions in support YAML <https://epics-containers.github.io/ibek/main/user/reference/naming.html>`_
+  Conventions for the YAML that ibek consumes.
+
+- `epics-containers documentation <https://epics-containers.github.io>`_
+  Documentation and tutorials for the epics-containers suite of tools.
+
+The command line interface for ibek is as follows. The most up to date version
+of this can be found by running ``ibek --help``.
+
+.. code-block::
+
+  Usage: ibek [OPTIONS] COMMAND [ARGS]...
+
+  IOC Builder for EPICS and Kubernetes
+  Provides support for building generic EPICS IOC container images and for running IOC instances
+  in a Kubernetes cluster.
+
+  ╭─ Options ─────────────────────────────────────────────────────────────────────────────────────╮
+  │ --version                     Print the version of ibek and exit                              │
+  │ --install-completion          Install completion for the current shell.                       │
+  │ --show-completion             Show completion for the current shell, to copy it or customize  │
+  │                               the installation.                                               │
+  │ --help                        Show this message and exit.                                     │
+  ╰───────────────────────────────────────────────────────────────────────────────────────────────╯
+  ╭─ Commands ────────────────────────────────────────────────────────────────────────────────────╮
+  │ support    Commands for building support modules during container build                       │
+  │ ioc        Commands for building generic IOCs during container build                          │
+  │ startup    Commands for building IOC instance startup files at container runtime              │
+  ╰───────────────────────────────────────────────────────────────────────────────────────────────╯
+
+  Usage: ibek support [OPTIONS] COMMAND [ARGS]...
+
+  Commands for building support modules during container build
+
+  ╭─ Options ─────────────────────────────────────────────────────────────────────────────────────╮
+  │ --help          Show this message and exit.                                                   │
+  ╰───────────────────────────────────────────────────────────────────────────────────────────────╯
+  ╭─ Commands ────────────────────────────────────────────────────────────────────────────────────╮
+  │ apt-install         Install debian packages into the container. If they have an http:// or    │
+  │                     https:// prefix then they will be downloaded and installed from file.     │
+  │ git-clone           clone a support module from a remote repository                           │
+  │ register            prepare the configure RELEASE files to build a support module inside an   │
+  │                     epics-containers build                                                    │
+  │ add-libs            declare the libraries for this support module for inclusion in IOC        │
+  │                     Makefile                                                                  │
+  │ add-dbds            declare the dbd files for this support module for inclusion in IOC        │
+  │                     Makefile                                                                  │
+  │ add-release-macro   add or replace a macro the global RELEASE file                            │
+  │ add-config-macro    add or replace a macro in CONFIG_SITE.linux-x86_64.Common file            │
+  │ add-to-config-site  add some text to a support module's CONFIG_SITE file                      │
+  │ compile             compile a support module after preparation with `ibek support register`   │
+  │                     etc.                                                                      │
+  │ generate-links      generate symlinks to the bob, pvi and support YAML for a compiled IOC     │
+  │ generate-schema     Produce JSON global schema for all <support_module>.ibek.support.yaml     │
+  │                     files                                                                     │
+  ╰───────────────────────────────────────────────────────────────────────────────────────────────╯
+
+  Usage: ibek ioc [OPTIONS] COMMAND [ARGS]...
+
+  Commands for building generic IOCs during container build
+
+  ╭─ Options ─────────────────────────────────────────────────────────────────────────────────────╮
+  │ --help          Show this message and exit.                                                   │
+  ╰───────────────────────────────────────────────────────────────────────────────────────────────╯
+  ╭─ Commands ────────────────────────────────────────────────────────────────────────────────────╮
+  │ generate-makefile  get the dbd and lib files from all support modules and generate            │
+  │                    iocApp/src/Makefile from iocApp/src/Makefile.jinja                         │
+  │ compile            Compile a generic IOC after support modules are registered and compiled    │
+  │ build              EXPERIMENTAL: Attempt to interpret the Dockerfile and run it's commands    │
+  │                    inside the devcontainer. For internal, incremental builds of the           │
+  │                    Dockerfile.                                                                │
+  │ generate-schema    Create a json schema from a number of support_module.ibek.support.yaml     │
+  │                    files                                                                      │
+  ╰───────────────────────────────────────────────────────────────────────────────────────────────╯
+
+  Usage: ibek startup [OPTIONS] COMMAND [ARGS]...
+
+  Commands for building IOC instance startup files at container runtime
+
+  ╭─ Options ─────────────────────────────────────────────────────────────────────────────────────╮
+  │ --help          Show this message and exit.                                                   │
+  ╰───────────────────────────────────────────────────────────────────────────────────────────────╯
+  ╭─ Commands ────────────────────────────────────────────────────────────────────────────────────╮
+  │ generate          Build a startup script for an IOC instance                                  │
+  ╰───────────────────────────────────────────────────────────────────────────────────────────────╯
 
 .. |code_ci| image:: https://github.com/epics-containers/ibek/actions/workflows/code.yml/badge.svg?branch=change_linter_to_ruff
     :target: https://github.com/epics-containers/ibek/actions/workflows/code.yml
@@ -53,6 +125,9 @@ This project is approaching completion. The following items are still to do:
     :target: https://pypi.org/project/ibek
     :alt: Latest PyPI version
 
+.. |black| image:: https://img.shields.io/badge/code%20style-black-000000.svg
+    :target: https://github.com/psf/black
+
 .. |license| image:: https://img.shields.io/badge/License-Apache%202.0-blue.svg
     :target: https://opensource.org/licenses/Apache-2.0
     :alt: Apache License
@@ -61,4 +136,4 @@ This project is approaching completion. The following items are still to do:
     Anything below this line is used when viewing README.rst and will be replaced
     when included in index.rst
 
-See https://epics-containers.github.io/ibek for more detailed documentation.
+See https://epics-containers.github.io/ibek for current documentation.

pyproject.toml

@@ -1,5 +1,5 @@
 [build-system]
-requires = ["setuptools>=64", "setuptools_scm[toml]>=6.2", "wheel"]
+requires = ["setuptools>=64", "setuptools_scm[toml]==7.1.0", "wheel"]
 build-backend = "setuptools.build_meta"
 
 [project]
@@ -18,6 +18,8 @@ dependencies = [
     "typer",
     "ruamel.yaml",
     "jinja2",
+    "GitPython",
+    "rich",
 ] # Add project dependencies here, e.g. ["click", "numpy"]
 dynamic = ["version"]
 license.file = "LICENSE"

src/ibek/__main__.py

@@ -1,20 +1,34 @@
-import json
-from pathlib import Path
-from typing import List, Optional
+from typing import Optional
 
 import typer
 from ruamel.yaml import YAML
 
+from ibek.ioc_cmds.commands import ioc_cli
+from ibek.startup_cmds.commands import startup_cli
+from ibek.support_cmds.commands import support_cli
+
 from ._version import __version__
-from .gen_scripts import (
-    create_boot_script,
-    create_db_script,
-    ioc_create_model,
-    ioc_deserialize,
+from .globals import NaturalOrderGroup
+
+cli = typer.Typer(cls=NaturalOrderGroup)
+
+
+cli.add_typer(
+    support_cli,
+    name="support",
+    help="Commands for building support modules during container build",
+)
+cli.add_typer(
+    ioc_cli,
+    name="ioc",
+    help="Commands for building generic IOCs during container build",
+)
+cli.add_typer(
+    startup_cli,
+    name="startup",
+    help="Commands for building IOC instance startup files at container runtime",
 )
-from .support import Support
 
-cli = typer.Typer()
 yaml = YAML()
 
 
@@ -34,68 +48,12 @@ def main(
         help="Print the version of ibek and exit",
     )
 ):
-    """Do 3 things..."""
-
-
-@cli.command()
-def ibek_schema(
-    output: Path = typer.Argument(..., help="The filename to write the schema to")
-):
-    """Produce JSON global schema for all <support_module>.ibek.support.yaml files"""
-    output.write_text(Support.get_schema())
-
-
-@cli.command()
-def ioc_schema(
-    definitions: List[Path] = typer.Argument(
-        ..., help="The filepath to a support module definition file"
-    ),
-    output: Path = typer.Argument(..., help="The filename to write the schema to"),
-):
-    """
-    Create a json schema from a <support_module>.ibek.support.yaml file
-    """
-
-    ioc_model = ioc_create_model(definitions)
-    schema = json.dumps(ioc_model.model_json_schema(), indent=2)
-    output.write_text(schema)
+    """IOC Builder for EPICS and Kubernetes
 
-
-@cli.command()
-def build_startup(
-    instance: Path = typer.Argument(
-        ..., help="The filepath to the ioc instance entity file"
-    ),
-    definitions: List[Path] = typer.Argument(
-        ..., help="The filepath to a support module definition file"
-    ),
-    out: Path = typer.Option(
-        default="config/st.cmd",
-        help="Path to output startup script",
-    ),
-    db_out: Path = typer.Option(
-        default="config/db.subst",
-        help="Path to output database expansion shell script",
-    ),
-):
-    """
-    Build a startup script for an IOC instance
+    Provides support for building generic EPICS IOC container images and for
+    running IOC instances in a Kubernetes cluster.
     """
 
-    ioc_instance = ioc_deserialize(instance, definitions)
-
-    script_txt = create_boot_script(ioc_instance)
-
-    out.parent.mkdir(parents=True, exist_ok=True)
-
-    with out.open("w") as stream:
-        stream.write(script_txt)
-
-    db_txt = create_db_script(ioc_instance)
-
-    with db_out.open("w") as stream:
-        stream.write(db_txt)
-
 
 # test with:
 #     pipenv run python -m ibek

src/ibek/globals.py

@@ -2,13 +2,41 @@
 A few global definitions
 """
 
+import os
+from pathlib import Path
 from typing import Dict
 
 from jinja2 import Template
 from pydantic import BaseModel, ConfigDict
+from typer.core import TyperGroup
 
 from .utils import UTILS
 
+# note requirement for environment variable EPICS_BASE
+EPICS_BASE = Path(str(os.getenv("EPICS_BASE")))
+EPICS_ROOT = Path(str(os.getenv("EPICS_ROOT")))
+
+# all support modules will reside under this directory
+SUPPORT = Path(str(os.getenv("SUPPORT")))
+# the global RELEASE file which lists all support modules
+RELEASE = SUPPORT / "configure/RELEASE"
+# a bash script to export the macros defined in RELEASE as environment vars
+RELEASE_SH = SUPPORT / "configure/RELEASE.shell"
+# global MODULES file used to determine order of build
+MODULES = SUPPORT / "configure/MODULES"
+# the root IOC folder
+IOC_FOLDER = Path(str(os.getenv("IOC")))
+# Folder containing Makefile.jinja
+MAKE_FOLDER = IOC_FOLDER / "iocApp/src"
+# Folder containing ibek support scripts
+# WARNING: this will only work if PROJECT NAME has been set in devcontainer.json
+PROJECT_NAME = os.getenv("PROJECT_NAME", "no-project-name")
+PROJECT_ROOT_FOLDER = Path("/workspaces") / PROJECT_NAME
+
+IOC_DBDS = SUPPORT / "configure/dbd_list"
+IOC_LIBS = SUPPORT / "configure/lib_list"
+RUNTIME_DEBS = SUPPORT / "configure/runtime_debs"
+
 
 class BaseSettings(BaseModel):
     """A Base class for setting consistent Pydantic model configuration"""
@@ -18,6 +46,11 @@ class BaseSettings(BaseModel):
     )
 
 
+class NaturalOrderGroup(TyperGroup):
+    def list_commands(self, ctx):
+        return self.commands.keys()
+
+
 def render_with_utils(context: Dict, template_text: str) -> str:
     """
     Render a Jinja template with the global __utils__ object in the context