Merge branch 'master' into fix/update_subscriptions_device

CHANGES_NEXT_RELEASE

@@ -1,3 +1,10 @@
 - Fix store device subscriptions updates
+- Remove: extractVariables from jexl plugin (no needed anymore since the removal of bidireational plugin)
+- Fix: ensure service and subservice in context of error handlers using req headers
+- Fix: remove attribute of measures with name `id` or `type` to avoid collisions (#1485)
+- Fix: ensure entity id and type are string (#1476)
+- Fix: update ctxt allow nested expressions (#1493)
+- Fix: change log level contextAvailable expression exception (from WARN to INFO)
+- Fix: null values arithmetics in JEXL expressions (#1440)
 - Fix: remove mongo `DeprecationWarning: current Server Discovery and Monitoring engine is deprecated` by setting `useUnifiedTopology = true`
-- Upgrade mongodb dev dep from 4.17.0 to 4.17.1
+- Upgrade mongodb dev dep from 4.17.0 to 4.17.1

doc/api.md

@@ -27,6 +27,7 @@
     -   [Measurement transformation](#measurement-transformation)
         -   [Measurement transformation definition](#measurement-transformation-definition)
         -   [Measurement transformation execution](#measurement-transformation-execution)
+        -   [Measurement transformation order](#measurement-transformation-order)
         -   [Multientity measurement transformation support (`object_id`)](#multientity-measurement-transformation-support-object_id)
     -   [Timestamp Compression](#timestamp-compression)
     -   [Timestamp Processing](#timestamp-processing)
@@ -152,7 +153,7 @@ parameters defined at device level in database, the parameters are inherit from
 ## Entity attributes
 
 In the config group/device model there are four list of attributes with different purpose to configure how the
-information coming from the device is mapped to the Context Broker attributes:
+information coming from the device (measures) is mapped to the Context Broker attributes:
 
 -   **`attributes`**: Are measures that are pushed from the device to the IoT agent. This measure changes will be sent
     to the Context Broker as updateContext requests over the device entity. NGSI queries to the context broker will be
@@ -179,7 +180,9 @@ information coming from the device is mapped to the Context Broker attributes:
 All of them have the same syntax, a list of objects with the following attributes:
 
 -   **object_id** (optional): name of the attribute as coming from the device.
--   **name** (mandatory): ID of the attribute in the target entity in the Context Broker.
+-   **name** (mandatory): ID of the attribute in the target entity in the Context Broker. Note that `id` and `type`
+    are not valid attribute names at Context Broker. Thus, although a measure named `id` or `type` will not break the IoT Agent, they 
+    are silently ignored and never progress toward Context Broker entities.
 -   **type** (mandatory): name of the type of the attribute in the target entity.
 -   **metadata** (optional): additional static metadata for the attribute in the target entity. (e.g. `unitCode`)
 
@@ -208,6 +211,10 @@ Additionally for commands (which are attributes of type `command`) the following
     particular IOTAs documentation for allowed values of this field in each case.
 -   **contentType**: `content-type` header used when send command by HTTP transport (ignored in other kinds of
     transports)
+
+Note that, when information coming from devices, this means measures, are not defined neither in the group, nor in the 
+device, the IoT agent will store that information into the destination entity using the same attribute name than the 
+measure name, unless `explicitAttrs` is defined. Measures `id` or `type` names are invalid, and will be ignored. 
 
 ## Multientity support
 
@@ -753,6 +760,89 @@ following to CB:
 
 [Interactive expression `spaces | trim`][5]
 
+### Measurement transformation order
+
+The IoTA executes the transformaion looping over the `attributes` provision field. Every time a new expression is 
+evaluated, the JEXL context is updated with the expression result. The order defined in the `attributes` array is 
+taken for expression evaluation. This should be considered when using **nested expressions**, that uses values 
+calculated in other attributes. 
+
+For example, let's consider the following provision for a device which send a measure named `level`:
+
+```json
+"attributes": [
+    {
+        "name": "correctedLevel",
+        "type": "Number",
+        "expression": "level * 0.897"
+    },
+    {
+        "name": "normalizedLevel",
+        "type": "Number",
+        "expression": "correctedLevel / 100"
+    }
+]
+```
+
+The expression for `correctedLevel` is evaluated first (using `level` measure as input). Next, the `normalizedLevel` is evaluated (using `correctedLevel` calculated attribute, just calculated before).
+
+Note that if we reserve the order, this way:
+
+```json
+"attributes": [
+    {
+        "name": "normalizedLevel",
+        "type": "Number",
+        "expression": "correctedLevel / 100"
+    },
+    {
+        "name": "correctedLevel",
+        "type": "Number",
+        "expression": "level * 0.897"
+    },    
+]
+```
+
+It is not going to work. The first expression expects a `correctedLevel` which is neither a measure (remember the only measure sent by the device is named `level`) nor a previously calculated attribute. Thus, `correctedLevel` will end with a `null` value.
+
+In conclusion: **the order of attributes in the `attributes` arrays at provising time matters with regards to nested expression evaluation**.
+
+Let's consider the following example. It is an anti-pattern but it's quite illustrative on how ordering works:
+
+```json
+"attributes": [
+    {
+        "name": "A",
+        "type": "Number",
+        "expression": "B"
+    },
+    {
+        "name": "B",
+        "type": "Number",
+        "expression": "A"
+    }
+]
+```
+
+When receiving a measure with the following values:
+
+```json
+{
+    "A": 10,
+    "B": 20
+}
+```
+
+Then, as they are executed sequentially, the first attribute expression to be evaluated will be `A`, taking the 
+value of the attribute `B`, in this case, `20`. After that, the second attribute expression to be evaluated is 
+the one holded by `B`. In this case, that attribute would take the value of `A`. In that case, since the JEXL
+context was updated with the lastest execution, `B` the value will be `20`, being update at Context Broker entity:
+
+```json
+    "A": {"value": 20, "type": "Number"},
+    "B": {"value": 20, "type": "Number"}
+```
+
 ### Multientity measurement transformation support (`object_id`)
 
 To allow support for measurement transformation in combination with multi entity feature, where the same attribute is

lib/plugins/expressionPlugin.js

@@ -57,13 +57,8 @@ function contextAvailable(expression, context, typeInformation) {
     return jexlParser.contextAvailable(expression, context);
 }
 
-function extractVariables(expression) {
-    return jexlParser.extractVariables(expression);
-}
-
 exports.parse = parse;
 exports.setJEXLTransforms = setJEXLTransforms;
 exports.applyExpression = applyExpression;
 exports.extractContext = extractContext;
 exports.contextAvailable = contextAvailable;
-exports.extractVariables = extractVariables;

lib/plugins/jexlParser.js

@@ -64,31 +64,6 @@ function parse(expression, context, callback) {
     }
 }
 
-function extractVariables(expression) {
-    const inst = new Lexer(grammar);
-    let variables = [];
-
-    try {
-        const tokens = inst.tokenize(expression);
-
-        // Keep only root attributes, removing the dot and sub-attributes. For example, if we have
-        // a.0.b, a.1.b and a.2.b, we will only keep a
-        // Additionaly, it will remove the function calls, since they are also detected as identifiers
-        variables = tokens.filter(function (token, index, array) {
-            return (
-                (token.type === ' ' && array[index - 1].type !== 'dot') ||
-                (token.type === 'identifier' && array[index + 1] && array[index + 1].type !== 'openParen')
-            );
-        });
-
-        // Return only array of values
-        return variables.map((a) => a.value);
-    } catch (e) {
-        logger.warn(logContext, 'Wrong expression found "[%j]" error: "[%s]", it will be ignored', expression, e);
-        return false;
-    }
-}
-
 function extractContext(attributeList) {
     const context = {};
     let value;
@@ -125,12 +100,24 @@ function extractContext(attributeList) {
 
 function applyExpression(expression, context, typeInformation) {
     logContext = fillService(logContext, typeInformation);
+    // Delete null values from context. Related:
+    // https://github.com/telefonicaid/iotagent-node-lib/issues/1440
+    // https://github.com/TomFrost/Jexl/issues/133
+    deleteNulls(context);
     const result = parse(expression, context);
     logger.debug(logContext, 'applyExpression "[%j]" over "[%j]" result "[%j]" ', expression, context, result);
     const expressionResult = result !== undefined ? result : expression;
     return expressionResult;
 }
 
+function deleteNulls(object) {
+    for (let key in object) {
+        if (object[key] === null) {
+            delete object[key];
+        }
+    }
+}
+
 function isTransform(identifier) {
     return jexl.getTransform(identifier) !== (null || undefined);
 }
@@ -141,7 +128,7 @@ function contextAvailable(expression, context) {
         jexl.evalSync(expression, context);
         return true;
     } catch (e) {
-        logger.warn(logContext, 'Wrong expression found "[%j]" over "[%j]", it will be ignored', expression, context);
+        logger.info(logContext, 'Wrong expression found "[%j]" over "[%j]", it will be ignored', expression, context);
         return false;
     }
 }
@@ -189,7 +176,6 @@ function setTransforms(configMap) {
     logger.info(logContext, message);
 }
 
-exports.extractVariables = extractVariables;
 exports.extractContext = extractContext;
 exports.contextAvailable = contextAvailable;
 exports.applyExpression = applyExpression;

lib/services/common/genericMiddleware.js

@@ -24,8 +24,9 @@
 const logger = require('logops');
 const revalidator = require('revalidator');
 const errors = require('../../errors');
+const fillService = require('./domain').fillService;
 let iotaInformation;
-const context = {
+let context = {
     op: 'IoTAgentNGSI.GenericMiddlewares'
 };
 
@@ -39,7 +40,10 @@ const context = {
 /* eslint-disable-next-line  no-unused-vars */
 function handleError(error, req, res, next) {
     let code = 500;
-
+    context = fillService(context, {
+        service: req.headers['fiware-service'],
+        subservice: req.headers['fiware-servicepath']
+    });
     logger.debug(context, 'Error [%s] handling request: %s', error.name, error.message);
 
     if (error.code && String(error.code).match(/^[2345]\d\d$/)) {
@@ -56,6 +60,10 @@ function handleError(error, req, res, next) {
  *  Express middleware for tracing the complete request arriving to the IoTA in debug mode.
  */
 function traceRequest(req, res, next) {
+    context = fillService(context, {
+        service: req.headers['fiware-service'],
+        subservice: req.headers['fiware-servicepath']
+    });
     logger.debug(context, 'Request for path [%s] query [%j] from [%s]', req.path, req.query, req.get('host'));
 
     if (req.is('json') || req.is('application/ld+json')) {
@@ -129,6 +137,10 @@ function validateJson(template) {
             if (errorList.valid) {
                 next();
             } else {
+                context = fillService(context, {
+                    service: req.headers['fiware-service'],
+                    subservice: req.headers['fiware-servicepath']
+                });
                 logger.debug(context, 'Errors found validating request: %j', errorList);
                 next(new errors.BadRequest('Errors found validating request.'));
             }

lib/services/ngsi/entities-NGSI-v2.js

@@ -340,18 +340,29 @@ function sendUpdateValueNgsi2(entityName, attributes, typeInformation, token, ca
         attributes,
         typeInformation
     );
+    // if any attribute name of a measure is 'id' or 'type' should be removed
+    var attributesWithoutIdType = [];
+    attributes.forEach(function (attribute) {
+        if (attribute.name !== 'id' && attribute.name !== 'type') {
+            attributesWithoutIdType.push(attribute);
+        }
+    });
+    attributes = attributesWithoutIdType;
+
     const payload = {
         entities: [
             {
-                id: entityName
+                // CB entity id should be always a String
+                id: String(entityName)
             }
         ]
     };
 
     let url = '/v2/op/update';
 
     if (typeInformation && typeInformation.type) {
-        payload.entities[0].type = typeInformation.type;
+        // CB entity type should be always a String
+        payload.entities[0].type = String(typeInformation.type);
     }
 
     payload.actionType = 'append';
@@ -676,6 +687,8 @@ function sendUpdateValueNgsi2(entityName, attributes, typeInformation, token, ca
         try {
             logger.debug(context, 'sendUpdateValueNgsi2 entityNameExp %j ', typeInformation.entityNameExp);
             entityName = expressionPlugin.applyExpression(typeInformation.entityNameExp, ctxt, typeInformation);
+            // CB entity id should be always a String
+            entityName = String(entityName);
             payload.entities[0].id = entityName;
             ctxt['entity_name'] = entityName;
         } catch (e) {
@@ -725,7 +738,9 @@ function sendUpdateValueNgsi2(entityName, attributes, typeInformation, token, ca
                         attr,
                         newAttr
                     );
-                    delete payload.entities[0][attr.object_id];
+                    if (!['id', 'type'].includes(attr.object_id)) {
+                        delete payload.entities[0][attr.object_id];
+                    }
                     attr = undefined; // stop processing attr
                     newAttr = undefined;
                 } else {
@@ -787,16 +802,16 @@ function sendUpdateValueNgsi2(entityName, attributes, typeInformation, token, ca
                     }
                 } catch (e) {
                     logger.error(context, 'sendUpdateValueNgsi2 apply expression exception=%j', e);
-                    if (attr && attr.name) {
+                    if (attr && attr.name && ctxt[attr.name] !== undefined) {
                         res = ctxt[attr.name];
                     }
                 }
                 // jexl expression plugin
                 newAttr.value = res;
 
                 logger.debug(context, 'sendUpdateValueNgsi2 apply expression result=%j newAttr=%j', res, newAttr);
-                // update current context with value
-                if (attr && attr.name && ctxt[attr.name] !== undefined) {
+                // update current context with value after apply expression
+                if (attr && attr.name) {
                     ctxt[attr.name] = newAttr.value;
                 }
             }
@@ -830,10 +845,10 @@ function sendUpdateValueNgsi2(entityName, attributes, typeInformation, token, ca
                         payload
                     );
                 }
-
+                // CB entity id and type should be always a String
                 let newEntity = {
-                    id: newEntityName ? newEntityName : payload.entities[0].id,
-                    type: attr.entity_type ? attr.entity_type : payload.entities[0].type
+                    id: newEntityName ? String(newEntityName) : String(payload.entities[0].id),
+                    type: attr.entity_type ? String(attr.entity_type) : String(payload.entities[0].type)
                 };
                 // Check if there is already a newEntity created
                 const alreadyEntity = payload.entities.find((entity) => {