deploy: 3a94c6e

DominicOram-patch-1/.buildinfo

@@ -0,0 +1,4 @@
+# Sphinx build info version 1
+# This file hashes the configuration used when building these files. When it is not found, a full rebuild will be done.
+config: 14f5acec0bd3e7f8317bd97f60f49ae7
+tags: 645f666f9bcd5a90fca523b33c5a78b7

DominicOram-patch-1/_sources/developer/explanations/architecture.rst.txt

@@ -0,0 +1,46 @@
+Architecture
+============
+
+Blueapi performs a number of tasks:
+
+* Managing the Bluesky RunEngine_, giving it instructions and handling its errors. Traditionally this job has been done by a human with an IPython_ terminal, so it requires automating.
+* Maintaining a registry of plans and devices. In the aforementioned IPython_ case, these would have just been global variables.
+* Communicating with the outside world, accepting instructions to run plans, providing updates on plan progress etc.
+
+These responsibilities are kept separate in the codebase to ensure a clean, maintainable architecture.
+
+Key Components
+--------------
+
+.. figure:: ../../images/blueapi-architecture.png
+    :width: 600px
+    :align: center
+
+    main components
+
+
+The ``BlueskyContext`` Object
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Holds registries of plans and devices as well as a number of helper methods for 
+registering en-masse from a normal Python module. 
+
+
+The Worker Object
+^^^^^^^^^^^^^^^^^
+
+Wraps the Bluesky ``RunEngine`` and accepts requests to run plans. The requests include the name 
+of the plan and a dictionary of parameters to pass. The worker validates the parameters against
+the known expectations of the plan, passes it to the ``RunEngine`` and handles any errors.
+
+
+The Service Object
+^^^^^^^^^^^^^^^^^^
+
+Handles communications and the API layer. This object holds a reference to the worker 
+can interrogate it/give it instructions in response to messages it recieves from the message
+bus. It can also forward the various events generated by the worker to topics on the bus.
+
+
+.. _RunEngine: https://nsls-ii.github.io/bluesky/run_engine_api.html
+.. _IPython: https://ipython.org/

DominicOram-patch-1/_sources/developer/explanations/decisions.rst.txt

@@ -0,0 +1,17 @@
+.. This Source Code Form is subject to the terms of the Mozilla Public
+.. License, v. 2.0. If a copy of the MPL was not distributed with this
+.. file, You can obtain one at http://mozilla.org/MPL/2.0/.
+
+Architectural Decision Records
+==============================
+
+We record major architectural decisions in Architecture Decision Records (ADRs),
+as `described by Michael Nygard
+<http://thinkrelevance.com/blog/2011/11/15/documenting-architecture-decisions>`_.
+Below is the list of our current ADRs.
+
+.. toctree::
+    :maxdepth: 1
+    :glob:
+
+    decisions/*

DominicOram-patch-1/_sources/developer/explanations/decisions/0001-record-architecture-decisions.rst.txt

@@ -0,0 +1,26 @@
+1. Record architecture decisions
+================================
+
+Date: 2022-02-18
+
+Status
+------
+
+Accepted
+
+Context
+-------
+
+We need to record the architectural decisions made on this project.
+
+Decision
+--------
+
+We will use Architecture Decision Records, as `described by Michael Nygard
+<http://thinkrelevance.com/blog/2011/11/15/documenting-architecture-decisions>`_.
+
+Consequences
+------------
+
+See Michael Nygard's article, linked above. To create new ADRs we will copy and
+paste from existing ones.

DominicOram-patch-1/_sources/developer/explanations/decisions/0002-no-queues.rst.txt

@@ -0,0 +1,27 @@
+2. No Queues
+============
+
+Date: 2023-05-22
+
+Status
+------
+
+Accepted
+
+Context
+-------
+
+In asking whether this service should hold and execute a queue of tasks.
+
+Decision
+--------
+
+We will not hold any queues. The worker can execute one task at a time and will return
+an error if asked to execute one task while another is running. Queueing should be the
+responsibility of a different service.
+
+Consequences
+------------
+
+The API must be kept queue-free, although transactions are permitted where the server
+caches requests.

DominicOram-patch-1/_sources/developer/explanations/decisions/0003-api-case.rst.txt

@@ -0,0 +1,26 @@
+2. API Model Case
+=================
+
+Date: 2023-05-23
+
+Status
+------
+
+Accepted
+
+Context
+-------
+
+Considering whether keys in JSON blobs from the API should be in snake_case or camelCase.
+This includes plan parameters which may be user-defined.
+
+Decision
+--------
+
+The priority is not to confuse users, so we will not alias any field names defined in Python.
+
+Consequences
+------------
+
+Most code will be written with pep8 enforcers which means most field names will be snake_case.
+Some user defined ones may differ.

DominicOram-patch-1/_sources/developer/explanations/events.rst.txt

@@ -0,0 +1,67 @@
+Events Emitted by the Worker
+============================
+
+Blueapi adds new events on top of the `bluesky event model`_.
+
+Reasons
+-------
+
+Since the ``RunEngine`` is traditionally used by a human in front of an IPython terminal, it
+sometimes assumes intuitive behaviour. The worker replaces the human and so must fill in the
+gaps.
+
+The base engine programatically emits data events conforming to the `bluesky event model`_. These
+are meant to be handled by other subscribing code (e.g. databroker) and are decoupled from concerns such as whether
+a plan has started, finished, paused, errored etc. See the example below:
+
+.. figure:: ../../images/bluesky-events.png
+    :width: 600px
+    :align: center
+
+    sequence of event emission compared to plan start/finish, in a complicated case
+
+Note the gap between the start of the plan and the issue of the first `run start document`_, and the similar gap
+for the stop document vs end of the plan, thsse are typically used for setup and cleanup. 
+Also note that a plan can produce an arbitrary number of runs. This decoupling is fine in an IPython terminal
+because a human user can see when a plan has started, can see when it's finished and can see which runs are
+associated with which plans.
+
+New Events
+----------
+
+For the case of automation, we introduce a new set of events outside of the event model, specifically
+pertaining to the running of the plan and state of the ``RunEngine``. At a mimimum, an event is emitted
+every time the engine:
+
+* Starts a new plan
+* Completes a plan
+* Fails/errors
+
+In the latter case, information about the error is also included.
+
+
+Correlation ID
+--------------
+
+When controlling plans programatically, it can be useful to verify that event model documents really are related to 
+the plan you just asked the worker to run. The worker will therefore bundle a correlation ID into the headers of
+messages containing documents.
+
+.. seealso:: `Microsoft Playbook on Correlation IDs`_
+
+ActiveMQ will give this header a different name depending on the protocol you use.
+
+.. list-table:: Correlation ID Headers
+   :widths: 25 25
+   :header-rows: 1
+
+   * - Protocol
+     - Header name
+   * - JMS
+     - jms_correlationID
+   * - STOMP
+     - correlation-id
+
+.. _`bluesky event model`: https://blueskyproject.io/event-model/main/index.html
+.. _`run start document`: https://blueskyproject.io/event-model/main/user/explanations/data-model.html#run-start-document
+.. _`Microsoft Playbook on Correlation IDs`: https://microsoft.github.io/code-with-engineering-playbook/observability/correlation-id/

DominicOram-patch-1/_sources/developer/explanations/lifecycle.rst.txt

@@ -0,0 +1,135 @@
+Lifecycle of a Plan
+===================
+
+The following demonstrates exactly what the code does with a plan through its lifecycle 
+of being written, loaded and run. Take the following plan.
+
+.. code:: python
+
+    from typing import List, Union, Mapping, Any, Optional
+
+    from blueapi.core import inject, MsgGenerator
+    from bluesky.protocols import Readable
+
+    def count(
+        detectors: List[Readable] = [inject("det")],  # default valid for Blueapi only
+        num: int = 1,
+        delay: Optional[Union[float, List[float]]] = None,
+        metadata: Optional[Mapping[str, Any]] = None,
+    ) -> MsgGenerator:
+        """
+        Take `n` readings from a collection of detectors
+
+        Args:
+            detectors (List[Readable]): Readable devices to read: when being run in Blueapi
+                                        defaults to fetching a device named "det" from its
+                                        context, else will require to be overriden.
+            num (int, optional): Number of readings to take. Defaults to 1.
+            delay (Optional[Union[float, List[float]]], optional): Delay between readings.
+                                                                Defaults to None.
+            metadata (Optional[Mapping[str, Any]], optional): Key-value metadata to include
+                                                            in exported data.
+                                                            Defaults to None.
+
+        Returns:
+            MsgGenerator: _description_
+
+        Yields:
+            Iterator[MsgGenerator]: _description_
+        """
+
+        yield from bp.count(detectors, num, delay=delay, md=metadata)
+
+
+Loading and Registration
+------------------------
+
+Blueapi will load this plan into its context if configured to load either this module or a module that 
+imports it. The ``BlueskyContext`` will go through all global variables in the module and register them
+if it detects that they are plans.
+
+At the point of registration it will inspect the plan's parameters and their type hints, from which it
+will build a pydantic_ model of the parameters to validate against. In other words, it will build something
+like this:
+
+
+.. code:: python
+
+    from pydantic import BaseModel
+
+    class CountParameters(BaseModel):
+        detectors: List[Readable] = ["det"]
+        num: int = 1
+        delay: Optional[Union[float, List[float]]] = None
+        metadata: Optional[Mapping[str, Any]] = None
+
+        class Config:
+            arbitrary_types_allowed = True
+            validate_all = True
+
+.. note:: 
+
+    This is for illustrative purposes only, this code is not actually generated, but an object
+    resembling this class is constructed in memory.
+    The default arguments will be validated by the context to inject the "det" device when the
+    plan is run. The existence of the "det" default device is not checked until this time.
+
+The model is also stored in the context.
+
+
+Startup
+-------
+
+On startup, the context is passed to the worker, which is passed to the service.
+The worker also holds a reference to the ``RunEngine`` that can run the plan.
+
+
+Request
+-------
+
+A user can send a request to run the plan to the service, which includes values for the parameters.
+It takes the form of JSON and may look something like this:
+
+.. code:: json
+
+    {
+        "name": "count",
+        "params": {
+            "detectors": [
+            "andor",
+            "pilatus"
+            ],
+            "num": 3,
+            "delay": 0.1
+        }
+    }
+
+The ``Service`` receives the request and passes it to the worker, which holds it in an internal queue
+and executes it as soon as it can. 
+
+
+Validation
+----------
+
+The pydantic model from earlier, as well as the plan function itself, is loaded out of the registry
+The parameter values in the request are validated against the model, this includes looking up devices
+with names ``andor`` and ``pilatus`` or, if detectors was not passed ``det``.
+
+
+.. seealso:: `./type_validators`
+
+Execution
+---------
+
+The validated parameter values are then passed to the plan function, which is passed to the RunEngine.
+The plan is executed. While it is running, the ``Worker`` will publish
+
+* Changes to the state of the ``RunEngine``
+* Changes to any device statuses running within a plan (e.g. when a motor changes position)
+* Event model documents emitted by the ``RunEngine``
+* When a plan starts, finishes or fails.
+
+If an error occurs during any of the stages from "Request" onwards it is sent back to the user
+over the message bus.
+
+.. _pydantic: https://docs.pydantic.dev/