NTIA/ITS SCOS Actions Plugin

This repository contains common actions and interfaces to be re-used by SCOS Sensor plugins. See the SCOS Sensor documentation for more information about SCOS Sensor, especially the Architecture and the Actions and Hardware Support sections which explain how SCOS Actions is used in the SCOS plugin architecture.

Table of Contents

• Overview of Repo Structure
• Running in SCOS Sensor
• Development
• License
• Contact

Overview of Repo Structure

• scos_actions/actions: This includes base Action classes and the following common action classes:
  • acquire_single_freq_fft: performs FFTs and calculates mean, median, min, max, and sample statistics at a single center frequency.
  • acquire_single_freq_tdomain_iq: acquires IQ data at a single center frequency.
  • acquire_stepped_freq_tdomain_iq: acquires IQ data at multiple center frequencies.
  • calibrate_y_factor: performs calibration using the Y-Factor method.
  • monitor_sigan: ensures a signal analyzer is available and is able to maintain a connection to the computer.
  • sync_gps: gets GPS location and syncs the host to GPS time
• scos_actions/calibration: This includes an interface for sensor calibration data
• scos_actions/configs/actions: This folder contains the YAML files with the parameters used to initialize the actions described above.
• scos_actions/discover: This includes the code to read YAML files and make actions available to SCOS Sensor.
• scos_actions/hardware: This includes the signal analyzer and GPS interfaces used by actions and the mock signal analyzer. The signal analyzer interface represents functionality common to all signal analyzers. Specific implementations of the signal analyzer interface for particular signal analyzers are provided in separate repositories like scos-usrp.
• scos_actions/metadata: This includes the SigMFBuilder class and related metadata structures used to generate SigMF-compliant metadata.
• scos_actions/signal_processing: This contains various common signal processing routines which are used in actions.
• scos_actions/status: This provides a class to register objects with the SCOS Sensor status endpoint.

Running in SCOS Sensor

Refer to the SCOS Sensor documentation for detailed instructions. To run SCOS Actions in SCOS Sensor with a mock signal analyzer, set MOCK_SIGAN and MOCK_SIGAN_RANDOM equal to 1 in docker-compose.yml before starting SCOS Sensor:

services:
  ...
  api:
    ...
    environment:
      ...
      - MOCK_SIGAN=1
      - MOCK_SIGAN_RANDOM=1

The following parameterized actions are offered for testing using a mock signal analyzer; their parameters are defined in scos_actions/configs/actions.

• test_multi_frequency_iq_action
• test_multi_frequency_y_factor_action
• test_single_frequency_iq_action
• test_single_frequency_m4s_action
• test_single_frequency_y_factor_action

Development

This repository is intended to be used by all SCOS Sensor plugins. Therefore, only universal actions that apply to most RF measurement systems should be added to SCOS Actions. Custom actions for specific hardware should be added to plugins in repositories supporting that specific hardware. New functionality should only be added to the signal analyzer interface defined in this repository if the new functionality can be supported by most signal analyzers.

Requirements and Configuration

Set up a development environment using a tool like Conda or venv, with python>=3.8. Then, from the cloned directory, install the development dependencies by running:

pip install .[dev]

This will install the project itself, along with development dependencies for pre-commit hooks, building distributions, and running tests. Set up pre-commit, which runs auto-formatting and code-checking automatically when you make a commit, by running:

pre-commit install

The pre-commit tool will auto-format Python code using Black and isort. Other pre-commit hooks are also enabled, and can be found in .pre-commit-config.yaml.

Building New Releases

This project uses Hatchling as a backend. Hatchling makes versioning and building new releases easy. The package version can be updated easily by using any of the following commands.

hatchling version major   # 1.0.0 -> 2.0.0
hatchling version minor   # 1.0.0 -> 1.1.0
hatchling version micro   # 1.0.0 -> 1.0.1
hatchling version "X.X.X" # 1.0.0 -> X.X.X

To build a new release (both wheel and sdist/tarball), run:

hatchling build

Running Tests

Ideally, you should add a test to cover any new feature that you add. If you've done that, then running the included test suite is the easiest way to check that everything is working. In any case, all tests should be run after making any local modifications to ensure that you haven't caused a regression.

The scos_actions package is tested using the pytest framework. Additionally, tox is used to run all available tests in a virtual environment against all supported versions of Python. Running pytest directly is faster but running tox is a more thorough test.

The following commands can be used to run tests. Note, for tox to run with all Python versions listed in the tox configuration (in tox.ini), all those versions must be installed on your system. Any missing versions will be skipped.

pytest          # faster, but less thorough
pytest --cov    # check where test coverage lacks
tox             # tests code in clean virtual environments, with multiple versions of Python
tox --recreate  # forces recreation of tox virtual environments

Adding Actions

To expose a new action to the API, check out the available action classes. An action is a parameterized implementation of an action class. If an existing class covers your needs, you can simply create YAML configs and use the init method in scos_actions.discover to make these actions available.

from scos_actions.discover import init
from scos_usrp.hardware import gps, sigan

actions = {
  "monitor_usrp": MonitorSignalAnalyzer(sigan),
  "sync_gps": SyncGps(gps),
}

yaml_actions, yaml_test_actions = init(yaml_dir=ACTION_DEFINITIONS_DIR)

actions.update(yaml_actions)

If no existing action class meets your needs, see Writing Custom Actions.

Creating a YAML config file for an action

Actions can be manually initialized in discover/__init__.py, but an easier method for non-developers and configuration-management software is to place a YAML file in the configs/actions directory which contains the action class name and parameter definitions.

The file name can be anything. File extensions must be .yml.

The action initialization logic parses all YAML files in this directory and registers the requested actions in the API.

Let's look at an example.

Example

Let's say we want to make an instance of the SingleFrequencyFftAcquisition.

First, create a new YAML file in the scos_actions/configs/actions directory. In this example we're going to create an acquisition for the LTE 700 C band downlink, so we'll call it acquire_700c_dl.yml.

Next, we want to find the appropriate string key for the SingleFrequencyFftAcquisition class. Look in actions/__init__.py at the action_classes dictionary. There, we see:

action_classes = {
    ...
    "single_frequency_fft": SingleFrequencyFftAcquisition,
    ...
}

That key tells the action loader which class to create an instance of. Put it as the first non-comment line, followed by a colon:

# File: acquire_700c_dl.yml

single_frequency_fft:

The next step is to see what parameters that class takes and specify the values. Open up actions/acquire_single_freq_fft.py and look at the documentation for the class to see what parameters are available and what units to use, etc.

class SingleFrequencyFftAcquisition(MeasurementAction):
    """Perform M4S detection over requested number of single-frequency FFTs.

    The action will set any matching attributes found in the signal
    analyzer object. The following parameters are required by the action:

        name: name of the action
        frequency: center frequency in Hz
        fft_size: number of points in FFT (some 2^n)
        nffts: number of consecutive FFTs to pass to detector

    For the parameters required by the signal analyzer, see the
    documentation from the Python package for the signal analyzer being
    used.

    :param parameters: The dictionary of parameters needed for the
        action and the signal analyzer.
    :param sigan: Instance of SignalAnalyzerInterface.
    """

Then look at the docstring for the signal analyzer class being used. This example will use the MockSignalAnalyzer. That file contains the following:

class MockSignalAnalyzer(SignalAnalyzerInterface):
    """
    MockSignalAnalyzer is mock signal analyzer object for testing.

    The following parameters are required for measurements:
    sample_rate: requested sample rate in samples/second
    frequency: center frequency in Hz
    gain: requested gain in dB
    """

Lastly, simply modify the YAML file to define any required parameters from the action and signal analyzer. Note that the sigan parameter is a special parameter that will get passed in separately when the action is initialized from the YAML. Therefore, it does not need to be defined in the YAML file.

# File: acquire_700c_dl.yml

single_frequency_fft:
  name: acquire_700c_dl
  frequency: 751e6
  gain: 40
  sample_rate: 15.36e6
  fft_size: 1024
  nffts: 300

You're done.

Writing Custom Actions

"Actions" are one of the main concepts used by SCOS Sensor. At a high level, they are the things that the sensor owner wants the sensor to be able to do. At a lower level, they are simply Python classes with a special method __call__. Actions use Django Signals to provide data and results to SCOS Sensor.

Start by looking at the Action base class. It includes some logic to parse a description and summary out of the action class's docstring, and a __call__ method that must be overridden. Actions are only instantiated with parameters. The signal analyzer implementation will be passed to the action at execution time through the call method's Sensor object.

A new custom action can inherit from the existing action classes to reuse and build upon existing functionality. A MeasurementAction base class, which inherits from the Action class, is also useful for building new actions. For example, SingleFrequencyTimeDomainIqAcquisition inherits from MeasurementAction, while SteppedFrequencyTimeDomainIqAcquisition inherits from SingleFrequencyTimeDomainIqAcquisition.

Depending on the type of action, a signal should be sent upon action completion. This enables SCOS Sensor to do something with the results of the action. This could range from storing measurement data to recycling a Docker container or to fixing an unhealthy connection to the signal analyzer. You can see the available signals in scos_actions/signals.py. The following signals are currently offered for actions:

• measurement_action_completed - signal expects task_id, data, and metadata
• location_action_completed - signal expects latitude and longitude
• trigger_api_restart - triggers a restart of the API docker container (where SCOS Sensor runs)

New signals can be added. However, corresponding signal handlers must be added to SCOS Sensor to receive the signals and process the results.

Adding custom action to SCOS Actions

A custom action meant to be re-used by other plugins can live in SCOS Actions. It can be instantiated using a YAML file, or directly in the actions dictionary in the discover/__init__.py module.

Adding system or hardware specific custom action

In the repository that provides the plugin to support the hardware being used, add the action to the actions dictionary in the discover/__init__.py file. Optionally, initialize the action using a YAML file by importing the YAML initialization code from SCOS Actions. For an example of this, see the Adding Actions subsection above.

Supporting a Different Signal Analyzer

scos_usrp adds support for the Ettus B2xx line of signal analyzers to SCOS Sensor. Follow these instructions to add support for another signal analyzer with a Python API.

• Create a new repository called scos-[signal analyzer name].
• Create a new virtual environment and activate it: python -m venv ./venv && source venv/bin/activate. Upgrade pip: python -m pip install --upgrade pip.
• In the new repository, add this repository as a dependency and create a class that inherits from the SignalAnalyzerInterface abstract class. Add properties or class variables for the parameters needed to configure the signal analyzer.
• Create YAML files with the parameters needed to run the actions imported from scos_actions using the new signal analyzer. Put them in the new repository in configs/actions. This should contain the parameters needed by the action as well as the signal analyzer settings based on which properties or class variables were implemented in the signal analyzer class in the previous step. The measurement actions in SCOS Actions are configured to check if any YAML parameters are available as attributes in the signal analyzer object, and to set them to the given YAML value if available. For example, if the new signal analyzer class has a bandwidth property, simply add a bandwidth parameter to the YAML file. Alternatively, you can create custom actions that are unique to the hardware. See Adding Actions subsection above.
• In the new repository, add a discover/__init__.py file. This should contain a dictionary called actions with keys of action names and values of action instances. If the repository also includes new action implementations, it should also expose a dictionary named action_classes with keys of actions names and values of action classes. You can use the init() and/or the load_from_yaml() methods provided in this repository to look for YAML files and initialize actions. You can use the existing action classes defined in this repository or create custom actions.

If your signal analyzer doesn't have a Python API, you'll need a Python wrapper that calls out to your signal analyzer's available API and reads the samples back into Python. Libraries such as SWIG can automatically generate Python wrappers for programs written in C/C++.

The next step in supporting a different signal analyzer is to create a class that inherits from the GPSInterface abstract class if the signal analyzer includes GPS capabilities. Then add the sync_gps and monitor_sigan actions to your actions dictionary, passing the gps object to the SyncGps constructor, and the signal analyzer object to the MonitorSignalAnalyzer constructor. See the example in the Adding Actions subsection above.

The final step would be to add a pyproject.toml to allow for installation of the new repository as a Python package. You can use the pyproject.toml in this repository as a reference. You can find more information about Python packaging here. Then add the new repository as a dependency to SCOS Sensor's requirements.txt using the following format: <package_name> @ git+<link_to_github_repo>@<branch_name>. If specific drivers are required for your signal analyzer, you can attempt to link to them within the package or create a docker image with the necessary files. You can host the docker image as a GitHub package. Then, when running SCOS Sensor, set the environment variable BASE_IMAGE=<image tag>.

License

See LICENSE.

Contact

For technical questions about SCOS Actions, contact Justin Haze, [email protected]