refactor(#317): Better design

- Better use of modules instead of classes in stage 4.

.github/actions/sierra-setup/action.yml

@@ -40,8 +40,7 @@ runs:
       if: runner.os == 'Linux'
       shell: bash
       run: |
-        sudo apt update
-        sudo apt-get install -y texlive-fonts-recommended texlive-latex-extra
+        sudo apt update && sudo apt-get install -y texlive-fonts-recommended texlive-latex-extra
 
     ############################################################################
     # OSX setup

docs/src/tutorials/project/hooks.rst

@@ -85,19 +85,18 @@ files. You can put those definitions in a single location and then add them to
 the ``.yaml`` graph definitions that are unique to the :term:`Project` as
 follows:
 
-#. Create ``pipeline/stage4/yaml_config_loader.py``.
+#. Create ``pipeline/stage4/graphs/loader.py``.
 
-#. Extend the
-   :class:`sierra.core.pipeline.stage4.yaml_config_loader.YAMLConfigLoader` class:
+#. Extend/override the
+   :func:`sierra.core.pipeline.stage4.graphs.loader.load_config()` function:
 
    .. code-block:: python
 
-      import sierra.core.pipeline.stage4.yaml_config_loader as ycl
+      import sierra.core.pipeline.stage4 import loader
       from sierra.core import types
 
-      class YAMLConfigLoader(ycl.YAMLConfigLoader):
-          def __call__(self, cmdopts: types.Cmdopts) -> tp.Dict[str, types.YAMLDict]:
-              ...
+      def load_config(cmdopts: types.Cmdopts) -> tp.Dict[str, types.YAMLDict]:
+          ...
 
 Intra-Experiment Graph Generation
 ---------------------------------
@@ -109,7 +108,7 @@ reason. To do so:
 #. Create ``pipeline/stage4/graphs/intra/generate.py``.
 
 #. Override the
-   :ref:`sierra.core.pipeline.stage4.graphs.intra.generate.generate()`
+   :func:`sierra.core.pipeline.stage4.graphs.intra.generate.generate()`
    function:
 
    .. code-block:: python

docs/src/usage/rendering.rst

@@ -62,10 +62,11 @@ that:
   needed to get ARGoS to "render" its simulation into an offscreen buffer which
   we can output to a file.
 
-During stage 4, ``--platform-vc`` causes frames captured during stage 2 to be
-stitched together into a unique video file using :program:`ffmpeg` (precise
-command configurable via ``--render-cmd-opts``), and output to
-``<batch_root>/videos/<exp>``.
+During stage 2, ARGoS outputs captured frames to ``frames/`` in each output
+directory (see :ref:`config.kRendering`).  During stage 4, ``--platform-vc``
+causes frames captured during stage 2 to be stitched together into a unique
+video file using :program:`ffmpeg` (precise command configurable via
+``--render-cmd-opts``), and output to ``<batch_root>/videos/<exp>``.
 
 .. _ln-sierra-usage-rendering-project:
 
@@ -86,8 +87,11 @@ To use, do the following:
    residing each subdirectory under the ``main.run_metrics_leaf`` directory (no
    recursive nesting is allowed) in each run are treated as snapshots of 2D or
    3D data over time, and will be averaged together across runs and then turn
-   into image files suitable for video rendering in stage 4. The following
-   restrictions apply:
+   into image files suitable for video rendering in stage 4. If you have
+   subdirectories which do `NOT` contain CSVs for imagizing, then passing this
+   option will cause an error.
+
+   The following restrictions apply:
 
    - A common stem with a unique numeric ID is required for each CSV must
      be present for each CSV.
@@ -109,9 +113,11 @@ To use, do the following:
 
 #. Pass ``--project-vc`` during stage 4 after running imagizing via
    ``--project-imagizing`` during stage 3, either on the same invocation or a
-   previous one. SIERRA will take the imagized CSV files previously created
-   and generate a set of a videos in ``<batch_root>/videos/<exp>`` for each
-   experiment in the batch which was run.
+   previous one. SIERRA will take the imagized CSV files previously created and
+   generate a set of a videos in ``<batch_root>/videos/<exp>/<subdir>`` for each
+   experiment in the batch which was run, where ``<subdir>`` is the name of a
+   subdirectory in ``main.run_metrics_leaf`` which contained the CSVs to
+   imagize.
 
    .. IMPORTANT::
 
@@ -133,10 +139,10 @@ affect some aspect of behavior over time.
 
 To use, do the following:
 
-#. Pass ``--bc-rendering`` during stage 4 when at least inter-experiment heatmap
-   is generated. SIERRA will take the generated PNG files previously created and
-   generate a set of a videos in ``<batch_root>/videos/<heatmap name>`` for each
-   heatmap.
+#. Pass ``--bc-rendering`` during stage 4 when inter-experiment heatmaps are
+   generated. SIERRA will take the generated PNG files previously created in
+   ``<batch_root>/graphs/collated`` and generate a set of a videos in
+   ``<batch_root>/videos/<heatmap name>`` for each heatmap.
 
    .. IMPORTANT::
 

docs/src/usage/run_time_tree.rst

@@ -152,7 +152,8 @@ it runs:
           together in stage 4 to generated videos. Each experiment will get its
           own directory under here, with unique sub-directories for each
           different type of :term:`Experimental Run` data captured for
-          imagizing. See :ref:`ln-sierra-usage-rendering-project` for more details.
+          imagizing. See :ref:`ln-sierra-usage-rendering-project` for more
+          details.
 
         - ``videos`` - Root directory for holding rendered videos generated
           during stage 4 from either captured simulator frames for imagized

sierra/core/cmdline.py

@@ -888,7 +888,7 @@ def init_stage4(self) -> None:
         rendering.add_argument("--render-cmd-opts",
                                help="""
 
-                               Specify the: program: `ffmpeg` options to appear
+                               Specify the :program:`ffmpeg` options to appear
                                between the specification of the input image
                                files and the specification of the output
                                file. The default is suitable for use with ARGoS
@@ -1129,11 +1129,11 @@ def __call__(self, args: argparse.Namespace) -> None:
         self._check_bc(args)
         self._check_pipeline(args)
 
-        assert args.sierra_root is not None,\
+        assert args.sierra_root is not None, \
             '--sierra-root is required for all stages'
 
     def _check_bc(self, args: argparse.Namespace) -> None:
-        assert len(args.batch_criteria) <= 2,\
+        assert len(args.batch_criteria) <= 2, \
             "Too many batch criteria passed"
 
         if len(args.batch_criteria) == 2:
@@ -1145,29 +1145,29 @@ def _check_bc(self, args: argparse.Namespace) -> None:
 
     def _check_pipeline(self, args: argparse.Namespace) -> None:
         if any(stage in args.pipeline for stage in [1]) in args.pipeline:
-            assert args.n_runs is not None,\
+            assert args.n_runs is not None, \
                 '--n-runs is required for running stage 1'
-            assert args.template_input_file is not None,\
+            assert args.template_input_file is not None, \
                 '--template-input-file is required for running stage 1'
             assert args.scenario is not None, \
                 '--scenario is required for running stage 1'
 
-        assert all(stage in [1, 2, 3, 4, 5] for stage in args.pipeline),\
+        assert all(stage in [1, 2, 3, 4, 5] for stage in args.pipeline), \
             'Only 1-5 are valid pipeline stages'
 
         if any(stage in args.pipeline for stage in [1, 2, 3, 4]):
-            assert len(args.batch_criteria) > 0,\
+            assert len(args.batch_criteria) > 0, \
                 '--batch-criteria is required for running stages 1-4'
-            assert args.controller is not None,\
+            assert args.controller is not None, \
                 '--controller is required for running stages 1-4'
 
         if 5 in args.pipeline:
             assert args.bc_univar or args.bc_bivar, \
                 '--bc-univar or --bc-bivar is required for stage 5'
-            assert args.scenario_comparison or args.controller_comparison,\
+            assert args.scenario_comparison or args.controller_comparison, \
                 '--scenario-comparison or --controller-comparison required for stage 5'
             if args.scenario_comparison:
-                assert args.controller is not None,\
+                assert args.controller is not None, \
                     '--controller is required for --scenario-comparison'
 
 

sierra/core/pipeline/stage4/graphs/intra/generate.py

@@ -49,7 +49,6 @@ def generate(main_config: types.YAMLDict,
 
         criteria:  The :term:`Batch Criteria` used for the batch
                    experiment.
-
     """
     exp_to_gen = utils.exp_range_calc(cmdopts,
                                       cmdopts['batch_output_root'],

sierra/core/pipeline/stage4/graphs/loader.py

@@ -0,0 +1,92 @@
+# Copyright 2021 John Harwell, All rights reserved.
+#
+#  SPDX-License-Identifier: MIT
+
+# Core packages
+import typing as tp
+import logging
+import pathlib
+
+# 3rd party packages
+import yaml
+
+# Project packages
+from sierra.core import types, utils
+
+_logger = logging.getLogger(__name__)
+
+
+def load_config(cmdopts: types.Cmdopts) -> tp.Dict[str, types.YAMLDict]:
+    """Load YAML configuration for :term:`Project` graphs to be generated.
+
+       Load YAML configuratoin for graphs.
+
+       This includes:
+
+       - intra-experiment linegraphs
+
+       - inter-experiment linegraphs
+
+       - intra-experiment heatmaps
+
+       - inter-experiment heatmaps (bivariate batch criteria only)
+
+       Returns:
+
+           Dictionary of loaded configuration with keys for ``intra_LN,
+           inter_LN, intra_HM, inter_HM``.
+
+      This function can be extended/overriden using a :term:`Project` hook. See
+      :ref:`ln-sierra-tutorials-project-hooks` for details.
+
+    """
+    inter_LN_config = {}
+    intra_LN_config = {}
+    intra_HM_config = {}
+    inter_HM_config = {}
+
+    root = pathlib.Path(cmdopts['project_config_root'])
+    project_inter_LN = root / 'inter-graphs-line.yaml'
+    project_intra_LN = root / 'intra-graphs-line.yaml'
+    project_intra_HM = root / 'intra-graphs-hm.yaml'
+    project_inter_HM = root / 'inter-graphs-hm.yaml'
+
+    if utils.path_exists(project_intra_LN):
+        _logger.info("Intra-experiment linegraph config for project '%s' from %s",
+                     cmdopts['project'],
+                     project_intra_LN)
+        with utils.utf8open(project_intra_LN) as f:
+            intra_LN_config = yaml.load(f, yaml.FullLoader)
+
+    if utils.path_exists(project_inter_LN):
+        _logger.info("Inter-experiment linegraph config for project '%s' from %s",
+                     cmdopts['project'],
+                     project_inter_LN)
+        with utils.utf8open(project_inter_LN) as f:
+            inter_LN_config = yaml.load(f, yaml.FullLoader)
+
+    if utils.path_exists(project_intra_HM):
+        _logger.info("Intra-experiment heatmap config for project '%s' from %s",
+                     cmdopts['project'],
+                     project_intra_HM)
+        with utils.utf8open(project_intra_HM) as f:
+            intra_HM_config = yaml.load(f, yaml.FullLoader)
+
+    if utils.path_exists(project_inter_HM):
+        _logger.info("Inter-experiment heatmap config for project '%s' from %s",
+                     cmdopts['project'],
+                     project_inter_HM)
+        with utils.utf8open(project_inter_HM) as f:
+            inter_HM_config = yaml.load(f, yaml.FullLoader)
+
+    return {
+        'intra_LN': intra_LN_config,
+        'intra_HM': intra_HM_config,
+        'inter_LN': inter_LN_config,
+        'inter_HM': inter_HM_config
+    }
+
+
+__api__ = [
+    'load_config'
+]

sierra/core/pipeline/stage4/pipeline_stage4.py

@@ -22,7 +22,7 @@
 from sierra.core.pipeline.stage4.model_runner import InterExpModelRunner
 import sierra.core.variables.batch_criteria as bc
 
-from sierra.core.pipeline.stage4 import rendering
+from sierra.core.pipeline.stage4 import render
 import sierra.core.plugin_manager as pm
 from sierra.core import types, config, utils
 
@@ -102,8 +102,8 @@ def __init__(self,
 
         # Load YAML config
         loader = pm.module_load_tiered(project=self.cmdopts['project'],
-                                       path='pipeline.stage4.yaml_config_loader')
-        graphs_config = loader.YAMLConfigLoader()(self.cmdopts)
+                                       path='pipeline.stage4.graphs.loader')
+        graphs_config = loader.load_config(self.cmdopts)
         self.intra_LN_config = graphs_config['intra_LN']
         self.intra_HM_config = graphs_config['intra_HM']
         self.inter_HM_config = graphs_config['inter_HM']
@@ -142,14 +142,14 @@ def run(self, criteria: bc.IConcreteBatchCriteria) -> None:
 
         Video generation: The following is run:
 
-        #. :class:`~sierra.core.pipeline.stage4.rendering.PlatformFramesRenderer`,
+        #. :func:`~sierra.core.pipeline.stage4.render.from_platform()`,
            if ``--platform-vc`` was passed
 
-        #. :class:`~sierra.core.pipeline.stage4.rendering.ProjectFramesRenderer`,
+        #. :func:`~sierra.core.pipeline.stage4.render.from_project_imagized()`,
            if ``--project-imagizing`` was passed previously to generate frames,
            and ``--project-rendering`` is passed.
 
-        #. :class:`~sierra.core.pipeline.stage4.rendering.BivarHeatmapRenderer`,
+        #. :func:`~sierra.core.pipeline.stage4.render.from_bivar_heatmaps()`,
            if the batch criteria was bivariate and ``--HM-rendering`` was
            passed.
 
@@ -295,22 +295,19 @@ def _run_rendering(self, criteria: bc.IConcreteBatchCriteria) -> None:
         start = time.time()
 
         if self.cmdopts['platform_vc']:
-            rendering.PlatformFramesRenderer(self.main_config,
-                                             self.cmdopts)(criteria)
+            render.from_platform(self.main_config, self.cmdopts, criteria)
         else:
             self.logger.debug(("--platform-vc not passed--skipping rendering "
                                "frames captured by the platform"))
 
         if self.cmdopts['project_rendering']:
-            rendering.ProjectFramesRenderer(self.main_config,
-                                            self.cmdopts)(criteria)
+            render.from_project_imagized(self.main_config, self.cmdopts, criteria)
         else:
             self.logger.debug(("--project-rendering not passed--skipping "
                                "rendering frames captured by the project"))
 
         if criteria.is_bivar() and self.cmdopts['bc_rendering']:
-            rendering.BivarHeatmapRenderer(self.main_config,
-                                           self.cmdopts)(criteria)
+            render.from_bivar_heatmaps(self.main_config, self.cmdopts, criteria)
         else:
             self.logger.debug(("--bc-rendering not passed or univariate batch "
                                "criteria--skipping rendering generated graphs"))