ReportDefinitions

Aggregations

Newts persists data as raw samples, and so must perform aggregations inline. When so queried, raw samples are processed through pipelines that are defined on a per report basis.

Fig. 1: Sample processing pipeline.

The 4 phases of sample processing are:

Normalization

The first phase in the processing pipeline coherces samples into the specified interval, aligned with the top of the hour.

Aggregation

Once normalized, the samples are aggregated on a per datasource basis according to the specified aggregate function. Samples that occur at an interval outside of the specified heartbeat result in a NaN value.

Calculations

Arbitrary calculations can be applied to aggregations to create new, additional result values. Calculations are applied in the order they are defined.

Export

Finally, the full list of output values (aggregations and/or calculations) are filtered down to a sequence of supplied exports.

Using the Java API

As a use-case, let's assume that we collect samples of inbound and outbound traffic (as bytes) on 5 minute intervals, and store them using metric names of ifInOctets and ifOutOctets respectively. We want to create aggregates of both metrics using AVERAGE, and calculate an additional value for total throughput (the sum of in and out). Finally, we want all values scaled to Kbytes.

Report aggregations are encapsulated by the org.opennms.newts.api.query.ResultDescriptor class, the constructor of which accepts an argument for the expected sample interval (in seconds).

import static org.opennms.newts.api.query.StandardAggregationFunctions.*;
import org.opennms.newts.api.query.*;
...

ResultDescriptor report = new ResultDescriptor(300);

Note: ResultDescriptor is a fluent interface, allowing method invocations to be chained to create more readable report definitions.

Datasources

Each datasource represents a persisted metric, and specifies a heartbeat value, the aggregation function to be applied, and an optional alternate name. The heartbeat value determines the maximum allowable time between samples before considering that interval unknown (NaN).

Every ResultDescriptor must have at least one datasource defined.

import static org.opennms.newts.api.query.StandardAggregationFunctions.*;
import org.opennms.newts.api.query.*;
...

ResultDescriptor report = new ResultDescriptor(300)
    .datasource("in",  "ifInOctets",  Duration.seconds(600), AVERAGE)
    .datasource("out", "ifOutOctets", Duration.seconds(600), AVERAGE);

The code above creates two datasources. The first, with a label of in, will produce averages of the ifInOctets metric, using a heartbeat of 600 seconds. The second, labeled out, likewise uses the AVERAGE aggregate function and heartbeat of 600 seconds for the metric ifOutOctets.

Calculations

Calculations are applied to one or more datasources, or previously performed calculations.

There are two ways to perform calculations, either using an implementation of org.opennms.newts.api.query.Calculation and the calculate method, or with a string expression, and the expression method.

import static org.opennms.newts.api.query.StandardAggregationFunctions.*;
import org.opennms.newts.api.query.*;
...

CalculationFunction scaleToKbytes = new CalculationFunction() {
    public double apply(double ds) {
        return ds / 8 / 1024;
    }
}

ResultDescriptor report = new ResultDescriptor(300)
    .datasource("in",  "ifInOctets",  Duration.seconds(600), AVERAGE)
    .datasource("out", "ifOutOctets", Duration.seconds(600), AVERAGE)
    .calculate("inKbytes",  scaleToKbytes, "in")
    .calculate("outKbytes", scaleToKbytes, "out")
    .expression("sumKbytes", "inKbytes + outKbytes");

Here, in and out are scaled to kilobytes using the scaleToKbytes function, resulting in two new values, inKbytes and outKbytes respectively. Then, using an expression this time, inKbytes and outKbytes are added together to create sumKbytes.

Export

By default, a ResultDescriptor specifies no output; Each label must be exported to the results explicitly.

import static org.opennms.newts.api.query.StandardAggregationFunctions.*;
import org.opennms.newts.api.query.*;
...

CalculationFunction scaleToKbytes = new CalculationFunction() {
    public double apply(double ds) {
        return ds / 8 / 1024;
    }
}

ResultDescriptor report = new ResultDescriptor(300)
    .datasource("in",  "ifInOctets",  Duration.seconds(600), AVERAGE)
    .datasource("out", "ifOutOctets", Duration.seconds(600), AVERAGE)
    .calculate("inKbytes",  scaleToKbytes, "in")
    .calculate("outKbytes", scaleToKbytes, "out")
    .expression("sumKbytes", "inKbytes + outKbytes")
    .export("inKbytes", "outKbytes", "sumKbytes");

Finally, inKbytes, outKbytes, and sumKbytes are exported to the final result set (in and out are ommited).

Configuring the REST service

Using the example from above, we'll assume that we collect samples of inbound and outbound traffic (as bytes) on 5 minute intervals, and store them using metric names of ifInOctets and ifOutOctets respectively. We want to create aggregates of both metrics using AVERAGE, and calculate an additional value for total throughput (the sum of in and out). Finally, we want all values scaled to Kbytes.

reports:

  ...

  traffic:
    interval: 300s
    datasources:
      - label: in
        source: ifInOctets
        function: AVERAGE
        heartbeat: 600s
      - label: out
        source: ifOutOctets
        function: AVERAGE
        heartbeat: 600s
    expressions:
      - label: inKbytes
        expression: in / 8 / 1024
      - label: outKbytes
        expression: out / 8 / 1024
      - label: sumKbytes
        expression: inKbytes + outKbytes
    exports:
      - sumKbytes
      - inKbytes
      - outKbytes

The snippet above defines a report names traffic (accessible as http://host.name/measurements/traffic), with an expected sample interval of 300 seconds. It contains two datasources, one labeled in that corresponds to the metric ifInOctets, and one labeled out that corresponds to ifOutOctets. Both datasources will be aggregated as averages, and have a heartbeat of 600 seconds.

In the expressions section, inKbytes is created to convert in to kilobytes, likewise for outKbytes. An expression labeled sumKbytes will hold the sum of inKbytes and outKbytes.

Finally, inKbytes, outKbytes, and sumKbytes are exported to the final result set (in and out are ommited).

[1]	http://en.wikipedia.org/wiki/Fluent_interface#Java