Flowgraph constructions (#773)

* basic flowgraph implementation

* add visualization test

* refactor

* pre-branch

* new backend

* rebase master with pypi packaging

* preliminary run graph algo

* implementing graph running

* passing basic run test

* more cycle reduction tests

* better capture of errors

* add serialization test

* using property decorators

* add step control and custom driver

* small linting fix

* graph runner should not raise exception

* add mermaid support

* add test for mermaid

* implement and test auto-insertion of aux actions

* update poetry.lock

* mermaid generator automatically writes to file

* implement hook test and direct imports

* change runner to iteratively run nodes via BFS

* add mermaid support to docs

* simplify module imports

* rename test

* start documentation

* fix wording

* bump packages

* bump lock file, add typing marker

* fix type-checking for networkx

* fix CI pipeline

* migrate to current step control names

* reorder sections

* make docs consistent

* convert back to legacy names for plugging into backend

.github/workflows/pr.yml

@@ -60,4 +60,5 @@ jobs:
         id: type-checks
         run: |
           touch .venv/lib/python${{ matrix.python-version }}/site-packages/ruamel/py.typed
+          touch .venv/lib/python${{ matrix.python-version }}/site-packages/networkx/py.typed
           poetry run mypy --namespace-packages -p hammer

doc/Hammer-Use/Flowgraphs.rst

@@ -0,0 +1,157 @@
+.. _flowgraphs:
+
+Flowgraphs
+==========
+
+Hammer has **experimental** support for flowgraph constructions, similar to tools like `mflowgen <https://github.com/mflowgen/mflowgen>`_.
+Their intention is to simplify the way flows are constructed and ran in Hammer.
+They can be imported via the ``hammer.flowgraph`` module.
+
+Construction
+------------
+
+Flowgraphs are nothing more than a collection of ``Node`` instances linked together via a ``Graph`` instance.
+Each ``Node`` "pulls" from a directory to feed in inputs and "pushes" output files to another directory to be used by other nodes.
+``Node`` instances are roughly equivalent to a single call to the ``hammer-vlsi`` CLI, so they take in similar attributes:
+
+* The action being called
+* The tool used to perform the action
+* The pull and push directories
+* Any *required* input/output files
+* Any *optional* input/output files
+* A driver to run the node with; this enables backwards compatibility with :ref:`hooks <hooks>`.
+* Options to specify steps within an action; this enables backwards compatibility with :ref:`flow control <flow-control>`.
+
+    * ``start_before_step``
+    * ``start_after_step``
+    * ``stop_before_step``
+    * ``stop_after_step``
+    * ``only_step``
+
+A minimal example of a ``Node``:
+
+.. code-block:: python
+    
+    from hammer.flowgraph import Node
+
+    test = Node(
+        action="foo",
+        tool="nop",
+        pull_dir="foo_dir",
+        push_dir="/dev/null",
+        required_inputs=["foo.yml"],
+        required_outputs=["foo-out.json"],
+    )
+
+Each ``Node`` has the ability to be "privileged", meaning that a flow *must* start with this node.
+This only occurs when a flow is being controlled using any of the steps.
+
+Running a Flowgraph
+-------------------
+
+``Node`` instances are linked together using an `adjacency list <https://en.wikipedia.org/wiki/Adjacency_list>`_.
+This list can be used to instantiate a ``Graph``:
+
+.. code-block:: python
+
+    from hammer.flowgraph import Graph, Node
+
+    root = Node(
+        "foo", "nop", "foo_dir", "",
+        ["foo.yml"],
+        ["foo-out.json"],
+    )
+    child1 = Node(
+        "bar", "nop", "foo_dir", "bar_dir",
+        ["foo-out.json"],
+        ["bar-out.json"],
+    )
+    graph = Graph({root: [child1]})
+
+Using the Hammer CLI tool, separate actions are manually linked via an *auxiliary* action, such as ``syn-to-par``.
+By using a flowgraph, ``Graph`` instances by default *automatically* insert auxiliary actions.
+This means that actions no longer need to be specified in a flow; the necessary nodes are inserted by the flowgraph tool.
+This feature can be disabled by setting ``auto_auxiliary=False``.
+
+A ``Graph`` can be run by calling the ``run`` method and passing in a starting node.
+When running a flow, each ``Node`` keeps an internal status based on the status of the action being run:
+
+* ``NOT_RUN``: The action has yet to be run.
+* ``RUNNING``: The action is currently running.
+* ``COMPLETE``: The action has finished.
+* ``INCOMPLETE``: The action ran into an error while being run.
+* ``INVALID``: The action's outputs have been invalidated (e.g. inputs or attributes have changed).
+
+The interactions between the statuses are described in the diagram:
+
+.. mermaid::
+
+    stateDiagram-v2
+        [*] --> NOT_RUN
+        NOT_RUN --> RUNNING
+        RUNNING --> INCOMPLETE
+        RUNNING --> COMPLETE
+        INCOMPLETE --> NOT_RUN
+        COMPLETE --> INVALID
+        INVALID --> NOT_RUN
+
+Regardless of whether a flow completes with or without errors, the graph at the time of completion or error is returned, allowing for a graph to be "resumed" once any errors have been fixed.
+
+Visualization
+-------------
+
+A flowgraph can be visualized in Markdown files via the `Mermaid <https://mermaid.js.org/>`_ tool.
+Calling a ``Graph`` instance's ``to_mermaid`` method outputs a file named ``graph-viz.md``.
+The file can be viewed in a site like `Mermaid's live editor <https://mermaid.live/>`_ or using Github's native support.
+
+The flowgraph below would appear like this:
+
+.. code-block:: python
+
+    from hammer.flowgraph import Graph, Node
+
+    syn = Node(
+        "syn", "nop",
+        os.path.join(td, "syn_dir"), os.path.join(td, "s2p_dir"),
+        ["syn-in.yml"],
+        ["syn-out.json"],
+    )
+    s2p = Node(
+        "syn-to-par", "nop",
+        os.path.join(td, "s2p_dir"), os.path.join(td, "par_dir"),
+        ["syn-out.json"],
+        ["s2p-out.json"],
+    )
+    par = Node(
+        "par", "nop",
+        os.path.join(td, "par_dir"), os.path.join(td, "out_dir"),
+        ["s2p-out.json"],
+        ["par-out.json"],
+    )
+    g = Graph({
+        syn: [s2p],
+        s2p: [par],
+        par: []
+    })
+
+
+Here are the contents of ``graph-viz.md`` after calling ``g.to_mermaid()``:
+
+.. code-block:: markdown
+
+    ```mermaid
+    
+    stateDiagram-v2
+        syn --> syn_to_par
+        syn_to_par --> par
+    ```
+
+Which would render like this:
+
+.. mermaid::
+
+    stateDiagram-v2
+        syn --> syn_to_par
+        syn_to_par --> par
+
+Note that the separators have been changed to comply with Mermaid syntax.

doc/Hammer-Use/index.rst

@@ -11,5 +11,6 @@ This documentation will walk through more advanced features of the Hammer infras
    Hammer-APIs
    Flow-Control
    Hooks
+   Flowgraphs
    Buildfile
    Hierarchical

doc/conf.py

@@ -17,12 +17,12 @@
 
 # -- Project information -----------------------------------------------------
 
-project = 'Hammer'
-copyright = '2023, Berkeley Architecture Research'
-author = 'Berkeley Architecture Research'
+project = "Hammer"
+copyright = "2023, Berkeley Architecture Research"
+author = "Berkeley Architecture Research"
 
 # The full version, including alpha/beta/rc tags
-release = '1.0.0'
+release = "1.0.0"
 
 
 # -- General configuration ---------------------------------------------------
@@ -33,30 +33,31 @@
 extensions = [
     "myst_parser",
     "sphinx-jsonschema",
-    "sphinx_rtd_size"
+    "sphinx_rtd_size",
+    "sphinxcontrib.mermaid",
 ]
 
 # Add any paths that contain templates here, relative to this directory.
-templates_path = ['_templates']
+templates_path = ["_templates"]
 
 # List of patterns, relative to source directory, that match files and
 # directories to ignore when looking for source files.
 # This pattern also affects html_static_path and html_extra_path.
-exclude_patterns = ['_build', 'Thumbs.db', '.DS_Store']
+exclude_patterns = ["_build", "Thumbs.db", ".DS_Store"]
 
-source_suffix = '.rst'
+source_suffix = ".rst"
 
-master_doc = 'index'
+master_doc = "index"
 
 # -- Options for HTML output -------------------------------------------------
 
 # The theme to use for HTML and HTML Help pages.  See the documentation for
 # a list of builtin themes.
 #
-html_theme = 'sphinx_rtd_theme'
+html_theme = "sphinx_rtd_theme"
 
 html_theme_options = {
-    'collapse_navigation': False,
+    "collapse_navigation": False,
 }
 
 sphinx_rtd_size_width = "1200px"

hammer/config/config_src.py

@@ -805,6 +805,7 @@ def get_config(self) -> dict:
             self.__config_cache_dirty = False
         return self.__config_cache
 
+    @property
     def get_config_types(self) -> dict:
         """
         Get the types for the configuration of a database.
@@ -906,9 +907,9 @@ def get_setting_type(self, key: str, nullvalue: Any = None) -> Any:
         :param nullvalue: Value to return for nulls.
         :return: Data type of key.
         """
-        if key not in self.get_config_types():
+        if key not in self.get_config_types:
             raise KeyError(f"Key type {key} is missing")
-        value = self.get_config_types()[key]
+        value = self.get_config_types[key]
         return nullvalue if value is None else value
 
     def set_setting_type(self, key: str, value: Any) -> None:
@@ -928,7 +929,7 @@ def has_setting_type(self, key: str) -> bool:
         :param key: Desired key.
         :return: True if the given setting exists.
         """
-        return key in self.get_config_types()
+        return key in self.get_config_types
 
     def check_setting(self, key: str, cfg: Optional[dict] = None) -> bool:
         """
@@ -940,12 +941,12 @@ def check_setting(self, key: str, cfg: Optional[dict] = None) -> bool:
 
         if cfg is None:
             cfg = self.get_config()
-        if key not in self.get_config_types():
+        if key not in self.get_config_types:
             #TODO: compile this at the beginning instead of emitting every instance
             #self.logger.warning(f"Key {key} is not associated with a type")
             return True
         try:
-            exp_value_type = parse_setting_type(self.get_config_types()[key])
+            exp_value_type = parse_setting_type(self.get_config_types[key])
         except ValueError as ve:
             raise ValueError(f'Key {key} has an invalid outer type: perhaps you have "List" instead of "list" or "Dict" instead of "dict"?') from ve
 

hammer/flowgraph/__init__.py

@@ -0,0 +1,5 @@
+#  Hammer logging code.
+#
+#  See LICENSE for licence details.
+
+from .flowgraph import *