From a7600f3b2dbaab00de0984b0a8008fef92ff24eb Mon Sep 17 00:00:00 2001 From: Johan Stokking Date: Mon, 29 Aug 2022 17:26:24 +0200 Subject: [PATCH] Document payload normalization (#939) * doc: Clarify filtering with field-mask * doc: Fix base URL configuration key * doc: Add normalized uplink payload to webhook templates * doc: Clarify Device Repository docs * doc: Clarify JavaScript payload formatters * doc: Document normalized payload * doc: Add clarifications --- .../integrations/payload-formatters/_index.md | 12 +- .../payload-formatters/device-repo.md | 4 +- .../payload-formatters/javascript/_index.md | 22 +- .../javascript/downlink-decoder/_index.md | 82 ------ .../javascript/downlink-encoder/_index.md | 104 ------- .../javascript/downlink/_index.md | 125 +++++++++ .../downlink-encoder.png | Bin .../javascript/uplink-decoder/_index.md | 112 -------- .../javascript/uplink/_index.md | 256 ++++++++++++++++++ .../uplink-decoder.png | Bin .../webhooks/webhook-templates/format.md | 82 +++++- .../webhook-templates/instantiation.md | 6 +- 12 files changed, 473 insertions(+), 332 deletions(-) delete mode 100644 doc/content/integrations/payload-formatters/javascript/downlink-decoder/_index.md delete mode 100644 doc/content/integrations/payload-formatters/javascript/downlink-encoder/_index.md create mode 100644 doc/content/integrations/payload-formatters/javascript/downlink/_index.md rename doc/content/integrations/payload-formatters/javascript/{downlink-encoder => downlink}/downlink-encoder.png (100%) delete mode 100644 doc/content/integrations/payload-formatters/javascript/uplink-decoder/_index.md create mode 100644 doc/content/integrations/payload-formatters/javascript/uplink/_index.md rename doc/content/integrations/payload-formatters/javascript/{uplink-decoder => uplink}/uplink-decoder.png (100%) diff --git a/doc/content/integrations/payload-formatters/_index.md b/doc/content/integrations/payload-formatters/_index.md index 39fbabb894..a494e0d8d7 100644 --- a/doc/content/integrations/payload-formatters/_index.md +++ b/doc/content/integrations/payload-formatters/_index.md @@ -9,7 +9,7 @@ This section explains how to set up payload formatters for a specific end device -In addition to the written instructions linked below, a video with instructions for creating Javascript and device repository payload formatters is available on [The Things Network youtube channel](https://youtu.be/4tii7MiD-yM). +In addition to the written instructions linked below, a video with instructions for creating JavaScript and device repository payload formatters is available on [The Things Network YouTube channel](https://youtu.be/4tii7MiD-yM).
Show video {{< youtube "4tii7MiD-yM" >}} @@ -17,11 +17,11 @@ In addition to the written instructions linked below, a video with instructions ## Types of Payload Formatters -{{% tts %}} supports three types of payload formatters: Javascript, CayenneLPP, and Repository payload formatters. +{{% tts %}} supports three types of payload formatters: JavaScript, CayenneLPP, and Repository payload formatters. -### Javascript +### JavaScript -{{% tts %}} allows you to write your own custom payload formatters in Javascript. To find out how to write a custom Javascript payload formatter, see the [Javascript Payload Formatters]({{< ref "/integrations/payload-formatters/javascript" >}}) section. +{{% tts %}} allows you to write your own custom payload formatters in JavaScript. To find out how to write a custom JavaScript payload formatter, see the [JavaScript Payload Formatters]({{< ref "/integrations/payload-formatters/javascript" >}}) section. ### CayeneLPP @@ -29,9 +29,9 @@ In addition to the written instructions linked below, a video with instructions ### Repository -Device manufacturers may submit payload formatters designed to work with their devices. These are publicly available in {{% tts %}} [Device Repository](https://github.com/TheThingsNetwork/lorawan-devices/tree/master). +Device manufacturers may submit payload formatters designed to work with their devices. These are publicly available in {{% tts %}} [Device Repository](https://github.com/TheThingsNetwork/lorawan-devices). -Device repository payload formatters work right out of the box - no customization needed! +Device repository payload formatters work right out of the box – no customization needed! ## Application and Device Specific Payload Formatters diff --git a/doc/content/integrations/payload-formatters/device-repo.md b/doc/content/integrations/payload-formatters/device-repo.md index bfe23b02bd..400dfc56ce 100644 --- a/doc/content/integrations/payload-formatters/device-repo.md +++ b/doc/content/integrations/payload-formatters/device-repo.md @@ -3,6 +3,6 @@ title: "Device Repository" description: "" --- -Device manufacturers may submit payload formatters designed to work with their devices. The source code for these is publicly available in the [lorawan-devices Github Repository](https://github.com/TheThingsNetwork/lorawan-devices/tree/master) and the devices are on display in The Things Industries [Device Repository](https://www.thethingsnetwork.org/device-repository/). +Device manufacturers may submit payload formatters designed to work with their devices. The data and code are publicly available in the [Github Repository](https://github.com/TheThingsNetwork/lorawan-devices). You can find all devices in the Device Repository on [this page](https://www.thethingsnetwork.org/device-repository/). -If you're a device manufacturer and you'd like to add your device to the open source device repo, find out how [here](https://github.com/TheThingsNetwork/lorawan-devices/blob/master/README.md). Adding your device makes it easy for users to use it with {{% tts %}}, and is a good way to publicly document information and code related to your device. +If you're a device manufacturer and you would like to add your device to the repository, find out how [here](https://github.com/TheThingsNetwork/lorawan-devices/blob/master/README.md). Adding your device makes it easy for users to use it with {{% tts %}}, and is a good way to publicly document information and code related to your device. diff --git a/doc/content/integrations/payload-formatters/javascript/_index.md b/doc/content/integrations/payload-formatters/javascript/_index.md index 30ab5a9ac7..38c277353c 100644 --- a/doc/content/integrations/payload-formatters/javascript/_index.md +++ b/doc/content/integrations/payload-formatters/javascript/_index.md @@ -1,28 +1,28 @@ --- -title: "Javascript" +title: "JavaScript" description: "" alias: "/integrations/payload-formatters/javascript" --- -Javascript payload formatters allow you to write your own functions to encode or decode messages. Javascript functions are executed using an [JavaScript ECMAScript 5.1](https://www.ecma-international.org/ecma-262/5.1/) engine. +JavaScript payload formatters allow you to write functions to encode or decode messages. JavaScript functions are executed using an [JavaScript ECMAScript 5.1](https://www.ecma-international.org/ecma-262/5.1/) runtime. Tips: + - The payload formatters should be simple and lightweight. - Use arithmetic operations and bit shifts to convert binary data to fields. - Avoid using non-trivial logic or polyfills. -{{< note >}} Payload formatters use ECMAScript 5 (2009), which has some distinct differences compared to newer, commonly used ECMAScript revisions. See [here](https://www.javatpoint.com/es5-vs-es6) for a quick comparison. Notably, `let`, `const`, and arrow functions are not supported by ES5. {{}} - -{{< note >}} For security, the runtime does not support modules, `require` syntax, or any input/output other than defined below. {{}} +{{< note >}} +Payload formatters use ECMAScript 5.1 (2009), which has some distinct differences compared to newer, commonly used ECMAScript revisions. See [here](https://www.javatpoint.com/es5-vs-es6) for a quick comparison. -{{< warning >}} The maximum size of a user-defined Javascript payload formatter is 40KB (40960 characters), unless the source of the payload formatter is [Device Repository]({{< ref "/integrations/payload-formatters/device-repo" >}}). {{}} +For security, the runtime does not support modules (`require` syntax) or any network or file system access. +{{}} -There are three different types of {{% tts %}} JavaScript payload formatters: +{{< warning >}} The maximum size of a JavaScript payload formatter is 40KB (40,960 bytes). Payload formatters loaded from the [Device Repository]({{< ref "/integrations/payload-formatters/device-repo" >}}) can be larger. {{}} -- [Uplink Decoder]({{< ref "/integrations/payload-formatters/javascript/uplink-decoder" >}}) -- [Downlink Encoder]({{< ref "/integrations/payload-formatters/javascript/downlink-encoder" >}}) -- [Downlink Decoder]({{< ref "/integrations/payload-formatters/javascript/downlink-decoder" >}}) +Learn more about payload formatters and examples: -Read the documentation below to further learn about these formatters and find associated examples. \ No newline at end of file +- [Uplink]({{< relref "uplink" >}}) +- [Downlink]({{< relref "downlink" >}}) diff --git a/doc/content/integrations/payload-formatters/javascript/downlink-decoder/_index.md b/doc/content/integrations/payload-formatters/javascript/downlink-decoder/_index.md deleted file mode 100644 index be8dc9d340..0000000000 --- a/doc/content/integrations/payload-formatters/javascript/downlink-decoder/_index.md +++ /dev/null @@ -1,82 +0,0 @@ ---- -title: "Downlink Decoder" -description: "" ---- - -The `decodeDownlink()` function is called to decode a downlink message to JSON format, if it was scheduled with payload as bytes, or if it was scheduled with human-readable JSON payload but also was encoded with the [`encodeDownlink()` function]({{< ref "/integrations/payload-formatters/javascript/downlink-encoder" >}}). - - - -Downlink messages sent upstream as part of events or downlink queue operations are therefore decoded, just like uplink messages (see [Uplink Decoder]({{< ref "/integrations/payload-formatters/javascript/uplink-decoder" >}})). - -The function takes an input object and returns an output object: - -```js -function decodeDownlink(input) { - return { - data: { ... } // JSON object - } -} -``` - -The input object has the following structure: - -```js -{ - "bytes": [1, 2, 3], // FRMPayload as byte array as returned by encodeDownlink() - "fPort": 1 // LoRaWAN FPort as returned by encodeDownlink() -} -``` - -The output object has the following structure: - -```js -{ - "data": { ... } // JSON object -} -``` - -`decodeDownlink()` must be symmetric with the `encodeDownlink()` and should therefore not return any errors. - -## Decode Downlink Example: The Things Node - -Here is an example of a function that decodes the output of `encodeDownlink()` (see above): - -```js -function decodeDownlink(input) { - switch (input.fPort) { - case 4: - return { - data: { - color: ["red", "green", "blue"][input.bytes[0]] - } - } - default: - throw Error("unknown FPort"); - } -} -``` - -The output of a previous call to `encodeDownlink()`: - -```json -{ - "fPort": 4, - "bytes": [2] -} -``` - -Yields the following data in the downlink message: - -```json -{ - "downlink_message": { - "f_port": 4, - "f_cnt": 7825, - "frm_payload": "Ag==", - "decoded_payload": { - "color": "blue" - } - } -} -``` diff --git a/doc/content/integrations/payload-formatters/javascript/downlink-encoder/_index.md b/doc/content/integrations/payload-formatters/javascript/downlink-encoder/_index.md deleted file mode 100644 index 7ae4d5654d..0000000000 --- a/doc/content/integrations/payload-formatters/javascript/downlink-encoder/_index.md +++ /dev/null @@ -1,104 +0,0 @@ ---- -title: "Downlink Encoder" -description: "" -weight: 2 ---- - -The `encodeDownlink()` function is called when a downlink message, with a payload in JSON format, is scheduled to be sent to the end device. The `encodeDownlink()` function encodes the JSON object of the downlink message to binary payload that gets transmitted to the end device. - - - -The function takes an input object and returns an output object: - -```js -function encodeDownlink(input) { - return { - bytes: [1, 2, 3], - fPort: 1 - } -} -``` - -The input object has the following structure: - -```json -{ - "field": "value" -} -``` - -The output object has the following structure: - -```json -{ - "bytes": [1, 2, 3], - "fPort": 1, - "warnings": ["warning 1", "warning 2"], - "errors": ["error 1", "error 2"] -} -``` - -The function's input and output shown above are incorporated in the [`as.down.data.receive`]({{< ref "/reference/api/events#event:as.down.data.receive" >}}) event, where the `data` object contains: - -```json -{ - "f_port": 1, - "frm_payload": "AQID", - "decoded_payload": { - "field": "value" - } -} -``` - -One can clearly see that the JSON payload (input) is contained in the `decoded_payload` field, while the output of the `encodeDownlink()` function is translated from hexadecimal to Base64 and as such contained in the `frm_payload` field. When the downlink reaches end device, it gets translated vice versa, from Base64 to hexadecimal, e.g. for example shown above, the received payload on the end device side is `01 02 03`. - -If an error is present in `errors`, the payload is invalid and the message will be dropped. Any warnings in `warnings` are informative. - -{{< note >}} Keep in mind that you can test your downlink encoder, as well as schedule downlinks from {{% tts %}} Console. {{}} - -## Encode Downlink Example: The Things Node - -Here is an example of an encoder function that uses the `color` field to send a byte on a specific FPort: - -```js -function encodeDownlink(input) { - var colors = ["red", "green", "blue"]; - return { - bytes: [colors.indexOf(input.data.color)], - fPort: 4, - }; -} -``` - -If the downlink with the following JSON payload is sent by the application on any FPort: - -```json -{ - "color": "green" -} -``` - -The output of the `encodeDownlink()` function will have the following structure: - -```json -{ - "bytes": [1], - "fPort": 4 -} -``` - -{{< figure src="downlink-encoder.png" alt="Testing a downlink encoder" >}} - -The `data` object of the `as.down.data.receive` event will therefore contain: - -```json -{ - "f_port": 4, - "frm_payload": "AQ==", - "decoded_payload": { - "color": "green" - } -} -``` - -From the output above, you can see that the JSON payload of the scheduled downlink is contained in the `decoded_payload` field. The `encodeDownlink()` function takes this value as input and outputs the index of the color `green` in the `colors` array. The index of `green` in `colors` is `01` in bytes, so that value is Base64-encoded to `AQ==` and stored in the `frm_payload` field. When the downlink reaches the end device, the payload gets translated from Base64 back to hexadecimal, so you will see the payload in bytes (`01`). Note also that, in this example, the message is always sent on the FPort 4 because this is explicitly defined in the downlink encoder function. diff --git a/doc/content/integrations/payload-formatters/javascript/downlink/_index.md b/doc/content/integrations/payload-formatters/javascript/downlink/_index.md new file mode 100644 index 0000000000..f0737470a6 --- /dev/null +++ b/doc/content/integrations/payload-formatters/javascript/downlink/_index.md @@ -0,0 +1,125 @@ +--- +title: "Downlink" +description: "" +weight: 2 +aliases: + - "/integrations/payload-formatters/javascript/downlink-encoder" + - "/integrations/payload-formatters/javascript/downlink-decoder" +--- + +The `encodeDownlink()` function is called when a downlink message with decoded payload is scheduled to be sent to the end device. The `encodeDownlink()` function encodes the JSON object of the downlink message to binary payload for transmission to the end device. + +The `decodeDownlink()` function does the opposite of `encodeDownlink()`: it decodes the binary payload back to decoded payload. + + + +The functions take an input object and return an output object: + +```js +function encodeDownlink(input) { + // input has the following structure: + // { + // field: "value" + // } + return { + bytes: [1, 2, 3], // FRMPayload (byte array) + fPort: 1, + warnings: ["warning 1", "warning 2"], // optional + errors: ["error 1", "error 2"] // optional (if set, the encoding failed) + } +} + +function decodeDownlink(input) { + // input has the following structure: + // { + // bytes: [1, 2, 3], // FRMPayload (byte array) + // fPort: 1 + // } + return { + data: { + field: "value" + }, + warnings: ["warning 1", "warning 2"], // optional + errors: ["error 1", "error 2"] // optional (if set, the decoding failed) + } +} +``` + +Encoded and decoded downlink payload are incorporated in the [`as.down.data.receive`]({{< ref "/reference/api/events#event:as.down.data.receive" >}}) event, where the `data` object contains: + +```json +{ + "f_port": 1, + "frm_payload": "AQID", + "decoded_payload": { + "field": "value" + } +} +``` + +The encoded payload is in `frm_payload` (Base64 encoded), while the decoded payload is in `decoded_payload`. + +If an error is present in `errors`, the payload is invalid and the message will be dropped. Any warnings in `warnings` are informative. + +{{< note >}} You can test your downlink encoder and decoder as well as schedule downlinks from {{% tts %}} Console. {{}} + +## Downlink Example: The Things Node + +Here is an example of an encoder function that uses the `color` field to send a byte on a specific FPort: + +```js +var colors = ["red", "green", "blue"]; + +function encodeDownlink(input) { + return { + bytes: [colors.indexOf(input.data.color)], + fPort: 4, + }; +} + +function decodeDownlink(input) { + switch (input.fPort) { + case 4: + return { + data: { + color: colors[input.bytes[0]] + } + } + default: + return { + errors: ["unknown FPort"] + } + } +} +``` + +If the downlink with the following JSON payload is sent by the application: + +```json +{ + "color": "green" +} +``` + +The output of the `encodeDownlink()` function will have the following structure: + +```json +{ + "bytes": [1], + "fPort": 4 +} +``` + +{{< figure src="downlink-encoder.png" alt="Testing a downlink encoder" >}} + +The `data` object of the `as.down.data.receive` event will therefore contain: + +```json +{ + "f_port": 4, + "frm_payload": "AQ==", + "decoded_payload": { + "color": "green" + } +} +``` diff --git a/doc/content/integrations/payload-formatters/javascript/downlink-encoder/downlink-encoder.png b/doc/content/integrations/payload-formatters/javascript/downlink/downlink-encoder.png similarity index 100% rename from doc/content/integrations/payload-formatters/javascript/downlink-encoder/downlink-encoder.png rename to doc/content/integrations/payload-formatters/javascript/downlink/downlink-encoder.png diff --git a/doc/content/integrations/payload-formatters/javascript/uplink-decoder/_index.md b/doc/content/integrations/payload-formatters/javascript/uplink-decoder/_index.md deleted file mode 100644 index 160637b25a..0000000000 --- a/doc/content/integrations/payload-formatters/javascript/uplink-decoder/_index.md +++ /dev/null @@ -1,112 +0,0 @@ ---- -title: "Uplink Decoder" -description: "" -weight: 1 ---- - -The Javascript `decodeUplink()` function is called when a data uplink message is received from a device. This function decodes the binary payload received from the end device to a human-readable JSON object that gets send upstream to the application. - - - -The function takes an input object and returns an output object: - -```js -function decodeUplink(input) { - return { - data: { - bytes: input.bytes - } - }; -} -``` - -The input object has the following structure: - -```json -{ - "bytes": [1, 2, 3], - "fPort": 1 -} -``` - -The output object has the following structure: - -```json -{ - "data": { - "bytes": [1, 2, 3] - }, - "warnings": ["warning 1", "warning 2"], - "errors": ["error 1", "error 2"] -} -``` - -The function's output shown above is incorporated in the [`as.up.data.forward`]({{< ref "/reference/api/events#event:as.up.data.forward" >}}) event, where the `data.uplink_message` object contains: - -```json -{ - "f_port": 1, - "frm_payload": "AQID", - "decoded_payload": { - "bytes": [1, 2, 3] - } -} -``` - -Uplink payload is constructed as an array of bytes on end device side. Before sending, the payload gets encoded with Base64 scheme. The object shown above contains this Base64-encoded value in the `frm_payload` field, because that is the format in which the payload is received on the server side. The result of the `decodeUplink()` function is contained in the `decoded_payload` field. - -If an error is present in `errors`, the payload is invalid and the message will be dropped. Any warnings in `warnings` are informative. - -{{< note >}} Keep in mind that you can test your uplink decoder, as well as simulate uplinks from {{% tts %}} Console. {{}} - -## Decode Uplink Example: The Things Node - -Here is an example `decodeUplink()` function from The Things Node: - -```js -function decodeUplink(input) { - var data = {}; - var events = { - 1: "setup", - 2: "interval", - 3: "motion", - 4: "button" - }; - data.event = events[input.fPort]; - data.battery = (input.bytes[0] << 8) + input.bytes[1]; - data.light = (input.bytes[2] << 8) + input.bytes[3]; - data.temperature = (((input.bytes[4] & 0x80 ? input.bytes[4] - 0x100 : input.bytes[4]) << 8) + input.bytes[5]) / 100; - var warnings = []; - if (data.temperature < -10) { - warnings.push("it's cold"); - } - return { - data: data, - warnings: warnings - }; -} -``` - -Let's assume that the end device performed event selection and some measurements (battery, light, temperature levels) and needs to send associated decimal values to {{% tts %}}: `12, 178, 4, 128, 247, 174`. Decimal values are first converted to bytes on the end device side, so we consider the following hexadecimal payload: `0C B2 04 80 F7 AE`. We can also assume that the message was sent on FPort 4. The payload is Base64-encoded before uplink transmission. - -The `data.uplink_message` object of the `as.up.data.forward` event will contain: - -```json -{ - "uplink_message": { - "f_port": 4, - "frm_payload": "DLIEgPeu", - "decoded_payload": { - "battery": 3250, - "event": "setup", - "light": 1152, - "temperature": -21.3 - }, - "decoded_payload_warnings": ["it's cold"] - } -} -``` - -From the object shown above, one can see that the Base64-encoded uplink payload is contained in the `frm_payload` field, while the result of the `decodeUplink()` function is contained in the `decoded_payload` field. You can also observe how warnings are mapped to the `decoded_payload_warnings` object if certain conditions are met. - -{{< figure src="uplink-decoder.png" alt="Testing an uplink decoder" >}} diff --git a/doc/content/integrations/payload-formatters/javascript/uplink/_index.md b/doc/content/integrations/payload-formatters/javascript/uplink/_index.md new file mode 100644 index 0000000000..6ced33cdb3 --- /dev/null +++ b/doc/content/integrations/payload-formatters/javascript/uplink/_index.md @@ -0,0 +1,256 @@ +--- +title: "Uplink" +description: "" +weight: 1 +aliases: + - "/integrations/payload-formatters/javascript/uplink-decoder" +--- + +The JavaScript `decodeUplink()` function is called when a data uplink message is received from a device. This function decodes the binary payload received from the end device to a human-readable JSON object that gets send upstream to the application. If present, the `normalizeUplink()` function is called to normalize the decoded payload in a common data structure. + + + +The decoder function takes an input object and returns an output object: + +```js +function decodeUplink(input) { + // input has the following structure: + // { + // "bytes": [1, 2, 3], // FRMPayload (byte array) + // "fPort": 1 + // } + return { + data: { + bytes: input.bytes + }, + warnings: ["warning 1", "warning 2"], // optional + errors: ["error 1", "error 2"] // optional (if set, the decoding failed) + }; +} +``` + +The decoder function's output shown above is incorporated in the [`as.up.data.forward`]({{< ref "/reference/api/events#event:as.up.data.forward" >}}) event, where the `data.uplink_message` object contains: + +```json +{ + "f_port": 1, + "frm_payload": "AQID", + "decoded_payload": { + "bytes": [1, 2, 3] + } +} +``` + +Uplink payload is transmitted as binary payload by the end device. {{% tts %}} shows this binary payload in the `frm_payload` field (Base64 encoded). The result of the `decodeUplink()` function is contained in the `decoded_payload` field. + +If an error is present in `errors`, the payload is invalid. If the decoder or the normalizer return an error, applications still receive the message but only the binary payload in `frm_payload`. Any warnings in `warnings` are informative. + +{{< note >}} You can test your uplink decoder, as well as simulate uplinks from {{% tts %}} Console. {{}} + +## Normalized Payload {{< new-in-version "3.21.2" >}} + +The uplink decoder can return any data structure. This provides full flexibility: a decoder can return any measurement in any unit as well as any additional information. + +However, developers building applications that combine data from various device types may find it difficult to work with different data structures. + +This is what the normalization step is for: the decoded payload gets normalized to a data structure that is common for all device types. Normalized payload is complementary to decoded payload: some measurements or some information may not fit in the normalized payload data structure. Therefore, some application may use normalized payload while others use decoded payload carrying device specific information. + +The normalizer function takes the output from the decoder function and returns an output object: + +```js +function normalizeUplink(input) { + // input has the following structure: + // { + // "data": { + // "temperature": 50, // Fahrenheit + // "windSpeed": 17.5 // knots + // } + // } + return { + data: { + air: { + temperature: (input.data.temperature - 32) * 5/9 // Fahrenheit to Celsius + }, + wind: { + speed: input.data.windSpeed * 0.5144 // knots to m/s + } + }, + warnings: ["warning 1", "warning 2"], // optional + errors: ["error 1", "error 2"] // optional (if set, the normalization failed) + } +} +``` + +The normalizer function's output shown above is incorporated in the [`as.up.data.forward`]({{< ref "/reference/api/events#event:as.up.data.forward" >}}) event, where the `data.uplink_message` object contains: + +```json +{ + "decoded_payload": { + "temperature": 50, + "windSpeed": 17.5 + }, + "normalized_payload": [ + { + "air": { + "temperature": 10 + }, + "wind": { + "speed": 9.002 + } + } + ] +} +``` + +The Application Server also publishes an event for each returned normalized payload, in the [`as.up.normalized.forward`]({{< ref "/reference/api/events#event:as.up.normalized.forward" >}}) event, where the `data.uplink_normalized` object contains: + +```json +{ + "normalized_payload": { + "air": { + "temperature": 10 + }, + "wind": { + "speed": 9.002 + } + } +} +``` + +{{< note >}} The uplink message data contains an array of normalized payload measurements (`normalized_payload` is an array) while the normalized uplink data contains a single normalized payload measurement. See [below](#multiple-measurements) why and how this is useful. {{< /note >}} + +### Normalized Payload Schema + +The normalized payload schema is continuously being extended to support more fields. You can find the latest JSON Schema definition in the [LoRaWAN Device Repository](https://github.com/TheThingsNetwork/lorawan-devices/blob/master/lib/payload.json). + +```js +{ + "time": "2022-08-29T13:50:15Z", // Timestamp (RFC 3339) + "air": { + "pressure": 1032, // Atmospheric pressure (hPa): [900..1100] + "relativeHumidity": 43.9, // Relative humidity (%): [0..100] + "temperature": 21.5 // Temperature (Celsius): [-273.15..) + }, + "wind": { + "direction": 321, // Direction (degrees): [0..360) + "speed": 5.2 // Speed (m/s): [0..) + } +} +``` + +### Advanced: Return Multiple Measurements {#multiple-measurements} + +Multiple measurements can be encoded by the end device in a single LoRaWAN uplink message. This is a good practice as it is more battery efficient. Therefore, the uplink decoder may find multiple measurements from the same sensor. The normalizer function can therefore return an array of measurements, as an alternative to a single measurement: + +```js +function normalizeUplink(input) { + // input has the following structure: + // { + // "data": { + // "readings": [ + // { "temperature": 51.4 }, + // { "temperature": 50.1 }, + // { "temperature": 56.8 } + // ] + // } + // } + var data = []; + for (var i = 0; i < input.data.readings.length; i++) { + data.push({ + air: { + temperature: (input.data.readings[i].temperature - 32) * 5/9 + } + }); + } + return { + data: data // this is now an array: [ { air: { temperature: ... } }, ... ] + } +} +``` + +When the normalizer function returns an array of measurements, Application Server publishes a normalized uplink event for each measurement, while there is still one uplink message event. + +## Uplink Example: The Things Node + +Here is an example `decodeUplink()` function from The Things Node: + +```js +function decodeUplink(input) { + var data = {}; + var events = { + 1: "setup", + 2: "interval", + 3: "motion", + 4: "button" + }; + data.event = events[input.fPort]; + data.battery = (input.bytes[0] << 8) + input.bytes[1]; + data.light = (input.bytes[2] << 8) + input.bytes[3]; + data.temperature = (((input.bytes[4] & 0x80 ? input.bytes[4] - 0x100 : input.bytes[4]) << 8) + input.bytes[5]) / 100; + var warnings = []; + if (data.temperature < -10) { + warnings.push("it's cold"); + } + return { + data: data, + warnings: warnings + }; +} + +function normalizeUplink(input) { + return { + data: { + air: { + temperature: input.data.temperature + } + } + } +} +``` + +For example, the end device observed an event and performed some measurements. The end device transmitted the following binary payload: `0C B2 04 80 F7 AE` (hex encoded) on FPort 4. The Base64 equivalent of that binary payload is `DLIEgPeu` and the JavaScript byte array is `[ 12, 178, 4, 128, 247, 174 ]`. This is all the same, just a different representation. + +The uplink decoder gets the JavaScript byte array of the binary payload and FPort as input. {{% tts %}} sets `frm_payload` to the Base64 representation of the binary payload, and `decoded_payload` to the output `data` of the uplink decoder. If there are warnings, they are set in `decoded_payload_warnings`. + +The `data.uplink_message` object of the `as.up.data.forward` event will contain: + +```json +{ + "uplink_message": { + "f_port": 4, + "frm_payload": "DLIEgPeu", + "decoded_payload": { + "battery": 3250, + "event": "setup", + "light": 1152, + "temperature": -21.3 + }, + "decoded_payload_warnings": ["it's cold"], + "normalized_payload": [ + { + "air": { + "temperature": -21.3 + } + } + ] + } +} +``` + +Since there is `normalized_payload`, there will also be a `as.up.normalized.forward` event published: + +```json +{ + "uplink_normalized": { + "f_port": 4, + "frm_payload": "DLIEgPeu", + "normalized_payload": { + "air": { + "temperature": -21.3 + } + } + } +} +``` + +{{< figure src="uplink-decoder.png" alt="Testing an uplink decoder" >}} diff --git a/doc/content/integrations/payload-formatters/javascript/uplink-decoder/uplink-decoder.png b/doc/content/integrations/payload-formatters/javascript/uplink/uplink-decoder.png similarity index 100% rename from doc/content/integrations/payload-formatters/javascript/uplink-decoder/uplink-decoder.png rename to doc/content/integrations/payload-formatters/javascript/uplink/uplink-decoder.png diff --git a/doc/content/integrations/webhooks/webhook-templates/format.md b/doc/content/integrations/webhooks/webhook-templates/format.md index d6fca124a9..3d146ad6af 100644 --- a/doc/content/integrations/webhooks/webhook-templates/format.md +++ b/doc/content/integrations/webhooks/webhook-templates/format.md @@ -40,24 +40,29 @@ The endpoint of the webhook can be configured using the following fields: - `format`: The format which the endpoint expects. Currently `json` and `protobuf` are supported. - `headers`: A mapping between the names of the headers and their values. The values can contain template fields. - `create-downlink-api-key`: Controls if an API Key specific to the service should be created on instantiation. -- `baseurl`: The base URL of the endpoint. Can contain template fields. +- `base-url`: The base URL of the endpoint. The message paths are provided in the `paths` object which can contain the following message types: -- `uplink-message`: The path to which uplink messages will be sent. Can contain template fields. -- `join-accept`: The path to which join accept messages will be sent. Can contain template fields. -- `downlink-ack`: The path to which downlink acknowledgements will be sent. Can contain template fields. -- `downlink-nack`: The path to which downlink not-acknowledged messages will be sent. Can contain template fields. -- `downlink-sent`: The path to which downlink sent will be sent. Can contain template fields. -- `downlink-failed`: The path to which downlink failures will be sent. Can contain template fields. -- `downlink-queued`: The path to which downlink queued status will be sent. Can contain template fields. -- `downlink-queue-invalidated`: The path to which downlink queue invalidated event will be sent. Can contain template fields. This is only used when the upstream platform carries out LoRaWAN `FRMPayload` encryption. -- `location-solved`: The path to which the location of the device will be sent when resolved. Can contain template fields. -- `service-data`: The path to which the data from integrations is sent. Data from services such as LoRa Cloud DAS and GLS will be sent here, for example. Can contain template fields. +- `uplink-message`: The path to which uplink messages will be sent. +- `uplink-normalized`: The path to which normalized uplink payloads will be sent. +- `join-accept`: The path to which join accept messages will be sent. +- `downlink-ack`: The path to which downlink acknowledgements will be sent. +- `downlink-nack`: The path to which downlink not-acknowledged messages will be sent. +- `downlink-sent`: The path to which downlink sent will be sent. +- `downlink-failed`: The path to which downlink failures will be sent. +- `downlink-queued`: The path to which downlink queued status will be sent. +- `downlink-queue-invalidated`: The path to which downlink queue invalidated event will be sent. This is only used when the upstream platform carries out LoRaWAN `FRMPayload` encryption. +- `location-solved`: The path to which the location of the device will be sent when resolved. +- `service-data`: The path to which the data from integrations is sent. Data from services such as LoRa Cloud DAS and GLS will be sent here, for example. -{{< note >}} Not all of the messages types must be handled by the service. By omitting the field in the `paths` object, the message type will be disabled in the final webhook and the related messages will not be passed to the endpoint. {{}} +{{< note >}} +The URL (`base-url`) and all paths can contain template fields. -## Field Mask {{< new-in-version "3.21.1" >}} +Not all of the messages types must be handled by the service. By omitting the field in the `paths` object, the message type will be disabled in the instantiated webhook configuration. +{{}} + +## Field Mask {{< new-in-version "3.21.1" >}} {#field-mask} Field mask can be configured by providing field paths as a list named `field-mask` in the body of the webhook template and contain the following paths: @@ -187,6 +192,8 @@ up.uplink_message.confirmed up.uplink_message.consumed_airtime up.uplink_message.decoded_payload up.uplink_message.decoded_payload_warnings +up.uplink_message.normalized_payload +up.uplink_message.normalized_payload_warnings up.uplink_message.f_cnt up.uplink_message.f_port up.uplink_message.frm_payload @@ -232,4 +239,53 @@ up.uplink_message.version_ids.model_id up.uplink_message.version_ids.serial_number up.uplink_message.version_ids.vendor_id up.uplink_message.version_ids.vendor_profile_id +up.uplink_normalized +up.uplink_normalized.confirmed +up.uplink_normalized.consumed_airtime +up.uplink_normalized.normalized_payload +up.uplink_normalized.normalized_payload_warnings +up.uplink_normalized.f_cnt +up.uplink_normalized.f_port +up.uplink_normalized.frm_payload +up.uplink_normalized.locations +up.uplink_normalized.network_ids +up.uplink_normalized.network_ids.cluster_address +up.uplink_normalized.network_ids.cluster_id +up.uplink_normalized.network_ids.net_id +up.uplink_normalized.network_ids.tenant_address +up.uplink_normalized.network_ids.tenant_id +up.uplink_normalized.received_at +up.uplink_normalized.rx_metadata +up.uplink_normalized.session_key_id +up.uplink_normalized.settings +up.uplink_normalized.settings.coding_rate +up.uplink_normalized.settings.concentrator_timestamp +up.uplink_normalized.settings.data_rate +up.uplink_normalized.settings.data_rate.modulation +up.uplink_normalized.settings.data_rate.modulation.fsk +up.uplink_normalized.settings.data_rate.modulation.fsk.bit_rate +up.uplink_normalized.settings.data_rate.modulation.lora +up.uplink_normalized.settings.data_rate.modulation.lora.bandwidth +up.uplink_normalized.settings.data_rate.modulation.lora.spreading_factor +up.uplink_normalized.settings.data_rate.modulation.lrfhss +up.uplink_normalized.settings.data_rate.modulation.lrfhss.coding_rate +up.uplink_normalized.settings.data_rate.modulation.lrfhss.modulation_type +up.uplink_normalized.settings.data_rate.modulation.lrfhss.operating_channel_width +up.uplink_normalized.settings.downlink +up.uplink_normalized.settings.downlink.antenna_index +up.uplink_normalized.settings.downlink.invert_polarization +up.uplink_normalized.settings.downlink.tx_power +up.uplink_normalized.settings.enable_crc +up.uplink_normalized.settings.frequency +up.uplink_normalized.settings.time +up.uplink_normalized.settings.timestamp +up.uplink_normalized.version_ids +up.uplink_normalized.version_ids.band_id +up.uplink_normalized.version_ids.brand_id +up.uplink_normalized.version_ids.firmware_version +up.uplink_normalized.version_ids.hardware_version +up.uplink_normalized.version_ids.model_id +up.uplink_normalized.version_ids.serial_number +up.uplink_normalized.version_ids.vendor_id +up.uplink_normalized.version_ids.vendor_profile_id ``` diff --git a/doc/content/integrations/webhooks/webhook-templates/instantiation.md b/doc/content/integrations/webhooks/webhook-templates/instantiation.md index 13af9588b3..09ef8382c5 100644 --- a/doc/content/integrations/webhooks/webhook-templates/instantiation.md +++ b/doc/content/integrations/webhooks/webhook-templates/instantiation.md @@ -89,7 +89,7 @@ If the user has filled in the value of `username` with `user1` and the value of ## Instantiation of Field Mask {{< new-in-version "3.21.1" >}} -The fields that are sent in the webhook uplink message can be filtered. Field paths not specified in `field-mask` will not be present in the uplink message. Field paths are provided as a list, for example as: +The request message data fields that are sent via webhooks can be filtered. If specified in `field-mask`, only these fields will be present in the request message to save bandwidth. Field paths are provided as a list, for example: ```yaml field-mask: @@ -98,4 +98,6 @@ field-mask: - up.service_data ``` -When there are no paths in the field mask or there is no `field-mask` in the template all the fields are sent in the uplink message. +When there are no paths in the field mask or there is no `field-mask` in the template, all fields will be present in the request message. + +For the full list of support fields, see [Field Mask]({{< relref "format#field-mask" >}}).