Skip to content

Commit

Permalink
Add a documentation example of how to gradually set up transformers
Browse files Browse the repository at this point in the history
  • Loading branch information
@KristianLyng
KristianLyng committed Jun 19, 2024
1 parent 5be6a30 commit 0b68707
Show file tree
Hide file tree
Showing 6 changed files with 384 additions and 0 deletions.
5 changes: 5 additions & 0 deletions docs/examples/README.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -42,3 +42,8 @@ client-certificates
A complex example for client certificate validation, including a script to
generate test certificates.

working-with-transformers
-------------------------

This is a demonstration of how to gradually work with relatively raw,
untransformed input data and an approach to flatten it sensibly.
30 changes: 30 additions & 0 deletions docs/examples/working-with-transformers/README.rst
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
Developing transformer configuration
====================================

When dealing with input data, the relatively easy part is getting the data,
but you often have to transform it significantly before it is suited for
storage. An example of this is juniper telemetry data, but the same
fundamental method applies to any semi-complex data, e.g.: if you have an
application that outputs statistics as a nested data structure.

The general approach to this is the same:

1. Set up a receiver for the appropriate data. This depends on the source.
Use a handler that simply prints to standard out to verify that your
receiver work. E.g.: the "print" sender. See the basics examples for
several examples of this.
2. Once you see data on standard output, switch form the "print" sender to
a file sender, use the "skogul" encoder. This will let you write json
data to disk, which is easier to work with.
3. Once you have a sufficiently large data set, you can stop skogul for
now, and start reading from disk instead.

This directory contains an example payload (``payload.json``) from Juniper
telemetry which is fairly nested.

The ``step1.json`` just reads that file and prints it. Which is the first
step. Then we start applying various transformers.

In ``step2.json`` we demonstrate basic splitting per interface and how to
extract ``if_name`` from the data fields to metadata. In ``step3.json``, we
further split per lane.
253 changes: 253 additions & 0 deletions docs/examples/working-with-transformers/payload
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,253 @@
{
"metrics": [
{
"timestamp": "2024-06-18T12:49:13.618+02:00",
"metadata": {
"componentId": 3,
"sensorName": "linecard_optics:/junos/system/linecard/optics/:/junos/system/linecard/optics/:PFE",
"subComponentId": 0,
"systemId": "router10-re0:10.1.2.3"
},
"data": {
"Optics_diag": [
{
"if_name": "et-3/1/2",
"optics_diag_stats": {
"laser_bias_current_high_alarm_threshold": 74.999,
"laser_bias_current_high_warning_threshold": 64.999,
"laser_bias_current_low_alarm_threshold": 4.999,
"laser_bias_current_low_warning_threshold": 7.9990000000000006,
"laser_output_power_high_alarm_threshold_dbm": 4.499871253967285,
"laser_output_power_high_warning_threshold_dbm": 2.4998061656951904,
"laser_output_power_low_alarm_threshold_dbm": -8.49860954284668,
"laser_output_power_low_warning_threshold_dbm": -6.50139856338501,
"laser_rx_power_high_alarm_threshold_dbm": 4.499871253967285,
"laser_rx_power_high_warning_threshold_dbm": 3.4999570846557617,
"laser_rx_power_low_alarm_threshold_dbm": -17.5205135345459,
"laser_rx_power_low_warning_threshold_dbm": -14.510088920593262,
"module_temp": 30.808000564575195,
"module_temp_high_alarm": false,
"module_temp_high_alarm_threshold": 75.9990005493164,
"module_temp_high_warning": false,
"module_temp_high_warning_threshold": 72.9990005493164,
"module_temp_low_alarm": false,
"module_temp_low_alarm_threshold": -5.999000072479248,
"module_temp_low_warning": false,
"module_temp_low_warning_threshold": -2.999000072479248,
"optics_lane_diag_stats": [
{
"lane_laser_bias_current": 22.525000000000002,
"lane_laser_bias_current_high_alarm": false,
"lane_laser_bias_current_high_warning": false,
"lane_laser_bias_current_low_alarm": false,
"lane_laser_bias_current_low_warning": false,
"lane_laser_output_power_dbm": 0.6039529,
"lane_laser_output_power_high_alarm": false,
"lane_laser_output_power_high_warning": false,
"lane_laser_output_power_low_alarm": false,
"lane_laser_output_power_low_warning": false,
"lane_laser_receiver_power_dbm": -1.2441856,
"lane_laser_receiver_power_high_alarm": false,
"lane_laser_receiver_power_high_warning": false,
"lane_laser_receiver_power_low_alarm": false,
"lane_laser_receiver_power_low_warning": false,
"lane_laser_temperature": 30.808000564575195,
"lane_number": 0,
"lane_rx_loss_of_signal_alarm": false,
"lane_tx_laser_disabled_alarm": false,
"lane_tx_loss_of_signal_alarm": false
},
{
"lane_laser_bias_current": 22.506,
"lane_laser_bias_current_high_alarm": false,
"lane_laser_bias_current_high_warning": false,
"lane_laser_bias_current_low_alarm": false,
"lane_laser_bias_current_low_warning": false,
"lane_laser_output_power_dbm": -0.50366044,
"lane_laser_output_power_high_alarm": false,
"lane_laser_output_power_high_warning": false,
"lane_laser_output_power_low_alarm": false,
"lane_laser_output_power_low_warning": false,
"lane_laser_receiver_power_dbm": -1.1611217,
"lane_laser_receiver_power_high_alarm": false,
"lane_laser_receiver_power_high_warning": false,
"lane_laser_receiver_power_low_alarm": false,
"lane_laser_receiver_power_low_warning": false,
"lane_laser_temperature": 30.808000564575195,
"lane_number": 1,
"lane_rx_loss_of_signal_alarm": false,
"lane_tx_laser_disabled_alarm": false,
"lane_tx_loss_of_signal_alarm": false
},
{
"lane_laser_bias_current": 22.525000000000002,
"lane_laser_bias_current_high_alarm": false,
"lane_laser_bias_current_high_warning": false,
"lane_laser_bias_current_low_alarm": false,
"lane_laser_bias_current_low_warning": false,
"lane_laser_output_power_dbm": 0.22881651,
"lane_laser_output_power_high_alarm": false,
"lane_laser_output_power_high_warning": false,
"lane_laser_output_power_low_alarm": false,
"lane_laser_output_power_low_warning": false,
"lane_laser_receiver_power_dbm": -2.0922298,
"lane_laser_receiver_power_high_alarm": false,
"lane_laser_receiver_power_high_warning": false,
"lane_laser_receiver_power_low_alarm": false,
"lane_laser_receiver_power_low_warning": false,
"lane_laser_temperature": 30.808000564575195,
"lane_number": 2,
"lane_rx_loss_of_signal_alarm": false,
"lane_tx_laser_disabled_alarm": false,
"lane_tx_loss_of_signal_alarm": false
},
{
"lane_laser_bias_current": 22.492,
"lane_laser_bias_current_high_alarm": false,
"lane_laser_bias_current_high_warning": false,
"lane_laser_bias_current_low_alarm": false,
"lane_laser_bias_current_low_warning": false,
"lane_laser_output_power_dbm": -0.835467,
"lane_laser_output_power_high_alarm": false,
"lane_laser_output_power_high_warning": false,
"lane_laser_output_power_low_alarm": false,
"lane_laser_output_power_low_warning": false,
"lane_laser_receiver_power_dbm": -0.9371948,
"lane_laser_receiver_power_high_alarm": false,
"lane_laser_receiver_power_high_warning": false,
"lane_laser_receiver_power_low_alarm": false,
"lane_laser_receiver_power_low_warning": false,
"lane_laser_temperature": 30.808000564575195,
"lane_number": 3,
"lane_rx_loss_of_signal_alarm": false,
"lane_tx_laser_disabled_alarm": false,
"lane_tx_loss_of_signal_alarm": false
}
],
"optics_type": 32
},
"snmp_if_index": 1257
},
{
"if_name": "et-3/1/5",
"optics_diag_stats": {
"laser_bias_current_high_alarm_threshold": 109.999,
"laser_bias_current_high_warning_threshold": 99.999,
"laser_bias_current_low_alarm_threshold": 19.999,
"laser_bias_current_low_warning_threshold": 29.999000000000002,
"laser_output_power_high_alarm_threshold_dbm": 5.499958515167236,
"laser_output_power_high_warning_threshold_dbm": 4.500023365020752,
"laser_output_power_low_alarm_threshold_dbm": -6.300424575805664,
"laser_output_power_low_warning_threshold_dbm": -5.300308704376221,
"laser_rx_power_high_alarm_threshold_dbm": 5.499958515167236,
"laser_rx_power_high_warning_threshold_dbm": 4.499871253967285,
"laser_rx_power_low_alarm_threshold_dbm": -13.605135917663574,
"laser_rx_power_low_warning_threshold_dbm": -10.599868774414062,
"module_temp": 23.429000854492188,
"module_temp_high_alarm": false,
"module_temp_high_alarm_threshold": 78.9990005493164,
"module_temp_high_warning": false,
"module_temp_high_warning_threshold": 74.9990005493164,
"module_temp_low_alarm": false,
"module_temp_low_alarm_threshold": -3.999000072479248,
"module_temp_low_warning": false,
"module_temp_low_warning_threshold": 0,
"optics_lane_diag_stats": [
{
"lane_laser_bias_current": 63.796,
"lane_laser_bias_current_high_alarm": false,
"lane_laser_bias_current_high_warning": false,
"lane_laser_bias_current_low_alarm": false,
"lane_laser_bias_current_low_warning": false,
"lane_laser_output_power_dbm": 1.727484,
"lane_laser_output_power_high_alarm": false,
"lane_laser_output_power_high_warning": false,
"lane_laser_output_power_low_alarm": false,
"lane_laser_output_power_low_warning": false,
"lane_laser_receiver_power_dbm": -11.307683,
"lane_laser_receiver_power_high_alarm": false,
"lane_laser_receiver_power_high_warning": false,
"lane_laser_receiver_power_low_alarm": false,
"lane_laser_receiver_power_low_warning": true,
"lane_laser_temperature": 23.429000854492188,
"lane_number": 0,
"lane_rx_loss_of_signal_alarm": false,
"lane_tx_laser_disabled_alarm": false,
"lane_tx_loss_of_signal_alarm": false
},
{
"lane_laser_bias_current": 61.349000000000004,
"lane_laser_bias_current_high_alarm": false,
"lane_laser_bias_current_high_warning": false,
"lane_laser_bias_current_low_alarm": false,
"lane_laser_bias_current_low_warning": false,
"lane_laser_output_power_dbm": 1.4497843,
"lane_laser_output_power_high_alarm": false,
"lane_laser_output_power_high_warning": false,
"lane_laser_output_power_low_alarm": false,
"lane_laser_output_power_low_warning": false,
"lane_laser_receiver_power_dbm": -11.331223,
"lane_laser_receiver_power_high_alarm": false,
"lane_laser_receiver_power_high_warning": false,
"lane_laser_receiver_power_low_alarm": false,
"lane_laser_receiver_power_low_warning": true,
"lane_laser_temperature": 23.429000854492188,
"lane_number": 1,
"lane_rx_loss_of_signal_alarm": false,
"lane_tx_laser_disabled_alarm": false,
"lane_tx_loss_of_signal_alarm": false
},
{
"lane_laser_bias_current": 89.849,
"lane_laser_bias_current_high_alarm": false,
"lane_laser_bias_current_high_warning": false,
"lane_laser_bias_current_low_alarm": false,
"lane_laser_bias_current_low_warning": false,
"lane_laser_output_power_dbm": 0.947853,
"lane_laser_output_power_high_alarm": false,
"lane_laser_output_power_high_warning": false,
"lane_laser_output_power_low_alarm": false,
"lane_laser_output_power_low_warning": false,
"lane_laser_receiver_power_dbm": -11.337186,
"lane_laser_receiver_power_high_alarm": false,
"lane_laser_receiver_power_high_warning": false,
"lane_laser_receiver_power_low_alarm": false,
"lane_laser_receiver_power_low_warning": true,
"lane_laser_temperature": 23.429000854492188,
"lane_number": 2,
"lane_rx_loss_of_signal_alarm": false,
"lane_tx_laser_disabled_alarm": false,
"lane_tx_loss_of_signal_alarm": false
},
{
"lane_laser_bias_current": 60.088,
"lane_laser_bias_current_high_alarm": false,
"lane_laser_bias_current_high_warning": false,
"lane_laser_bias_current_low_alarm": false,
"lane_laser_bias_current_low_warning": false,
"lane_laser_output_power_dbm": 1.0974734,
"lane_laser_output_power_high_alarm": false,
"lane_laser_output_power_high_warning": false,
"lane_laser_output_power_low_alarm": false,
"lane_laser_output_power_low_warning": false,
"lane_laser_receiver_power_dbm": -11.337186,
"lane_laser_receiver_power_high_alarm": false,
"lane_laser_receiver_power_high_warning": false,
"lane_laser_receiver_power_low_alarm": false,
"lane_laser_receiver_power_low_warning": true,
"lane_laser_temperature": 23.429000854492188,
"lane_number": 3,
"lane_rx_loss_of_signal_alarm": false,
"lane_tx_laser_disabled_alarm": false,
"lane_tx_loss_of_signal_alarm": false
}
],
"optics_type": 32
},
"snmp_if_index": 1258
}
]
}
}
]
}
16 changes: 16 additions & 0 deletions docs/examples/working-with-transformers/step1.json
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
{
"receivers": {
"file": {
"type": "wholefile",
"file": "payload",
"handler": "wip"
}
},
"handlers": {
"wip": {
"parser": "skogul",
"transformers": [],
"sender": "print"
}
}
}
29 changes: 29 additions & 0 deletions docs/examples/working-with-transformers/step2.json
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
{
"receivers": {
"file": {
"type": "wholefile",
"file": "payload",
"handler": "wip"
}
},
"handlers": {
"wip": {
"parser": "skogul",
"transformers": ["optics_diag","extract_names"],
"sender": "print"
}
},
"transformers": {
"optics_diag": {
"type": "split",
"field": ["Optics_diag"]
},
"extract_names": {
"type": "metadata",
"copyFromData": [
{"source": "if_name", "delete": true},
{"source": "snmp_if_index", "delete": true}
]
}
}
}
51 changes: 51 additions & 0 deletions docs/examples/working-with-transformers/step3.json
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,51 @@
{
"receivers": {
"file": {
"type": "wholefile",
"file": "payload",
"handler": "wip"
}
},
"handlers": {
"wip": {
"parser": "skogul",
"transformers": ["optics_diag","extract_names","flatten","extract_names2","split2","extract_names3"],
"sender": "print"
}
},
"transformers": {
"optics_diag": {
"type": "split",
"field": ["Optics_diag"]
},
"extract_names": {
"type": "metadata",
"copyFromData": [
{"source": "if_name", "delete": true},
{"source": "snmp_if_index", "delete": true}
]
},
"flatten": {
"type": "data",
"flatten": [
["optics_diag_stats"]
]
},
"extract_names2": {
"type": "metadata",
"copyFromData": [
{"source": "optics_diag_stats__optics_type", "destination": "optics_type", "delete": true}
]
},
"split2": {
"type": "split",
"field": ["optics_diag_stats__optics_lane_diag_stats"]
},
"extract_names3": {
"type": "metadata",
"copyFromData": [
{"source": "lane_number", "delete": true}
]
}
}
}

0 comments on commit 0b68707

Please sign in to comment.