Unify ContextElement class into Entity class

CHANGES_NEXT_RELEASE

@@ -9,3 +9,5 @@
 - Fix: Missing or empty metadata values were not allowed in NGSIv2 create/update operations (#3121)
 - Fix: default types for entities and attributes in NGSIv2 was wrongly using "none" in some cases
 - Fix: With NGSIv2 replace operations the geolocalization field is inconsistent in DB (#1142) (#3167)
+- Fix: duplicated attribute/metadata keys in JSON response in some cases for dateCreated/dateModified/actionType
+- Fix: proper management of user attributes/metadata which name matches the one of a builtin (the user defined shadows the builtin for rendering but not for filtering)

doc/manuals/user/ngsiv2_implementation_notes.md

@@ -7,8 +7,7 @@
 * [Limit to attributes for entity location](#limit-to-attributes-for-entity-location)
 * [Legacy attribute format in notifications](#legacy-attribute-format-in-notifications)
 * [Datetime support](#datetime-support)
-* [`dateModified` and `dateCreated` attributes](#datemodified-and-datecreated-attributes)
-* [`dateModified` and `dateCreated` metadata](#datemodified-and-datecreated-metadata)
+* [User attributes or metadata matching builtin name](#user-attributes-or-metadata-matching-builtin-name)
 * [Subscription payload validations](#subscription-payload-validations)
 * [`actionType` metadata](#actiontype-metadata)
 * [`noAttrDetail` option](#noattrdetail-option)
@@ -138,126 +137,34 @@ The string "ISO8601" as type for attributes and metadata is also supported. The
 
 [Top](#top)
 
-## `dateModified` and `dateCreated` attributes
-
-The section "Attribute names restrictions" establishes that builtin attribute names cannot be used as user
-attribute names. Thus, `dateCreated` and `dateModified` (which are builtin attributes) cannot be used as user
-attribute names.
-
-However, due to legacy reasons, Orion doesn't reject attempts of using these attribute with 400 Bad Request errors. **You are
-strongly encouraged to not use dateCreate/dateModified as attribute names** but, if you at the end need to do so, take into
-account the following:
-
-You can use dateModified/dateCreated attributes at creation/update time, e.g.:
-
-```
-POST /v2/entities
-
-{
-  "id": "FutureEntity",
-  "type": "T",
-  ...
-  "dateModified": {
-      "type": "DateTime".
-      "value": "2050-01-01T00:00:00Z"
-  }
-}
-```
-
-Retrieval of such attributes using GET API operations will return the value you created/updated. You don't need to use
-`?attrs=dateModified,...` in this case, e.g.:
-
-```
-GET /v2/entities/FutureEntity/attrs
-```
-
-```
-{
-  ...
-  "dateModified": {
-      "type": "DateTime".
-      "value": "2050-01-01T00:00:00Z"
-  }
-}
-```
-
-However, when included in notifications, the value is not the one provided by the user (*), but the real
-one corresponding to entity modification/creation stored internally by Orion, e.g. (assuming the last update
-for that entity was sent at 11:16:05 at May 28th, 2018):
-
-```
-POST /notify
-
-{
-    "data": [
-        {
-            "dateModified": {
-                "metadata": {},
-                "type": "DateTime",
-                "value": "2018-05-28T11:16:05Z"
-            },
-            "id": "FutureEntity",
-            "type": "T"
-        }
-    ],
-    "subscriptionId": "5b082dc2c17960f8773dd74d"
-}
-```
-
-(*) Exception: initial notifications use the value provided by the user. However, this is probably a side
-effect of [this issue](https://github.com/telefonicaid/fiware-orion/issues/3182).
-
-In addition, filters will use the real modification/creation date as stored internally by Orion. For instance,
-the following filter will not retrieve any result (even when user provided 2050-01-01T00:00:00Z as dateModified, which
-would match the filter condition):
-
-```
-GET /v2/entities?q=dateModified>2030-01-01T00:00:00Z
-```
-
-This behaviour is assessed by the functional test `entity_dates_overriden_by_user.test`
-and `entity_dates_overriden_by_user_subs.test` cases included in
-[`cases/0876_entity_dates directory`](https://github.com/telefonicaid/fiware-orion/tree/master/test/functionalTest/cases/0876_entity_dates).
-Please have a look, if you need a more detailed description.
-
-[Top](#top)
-
-## `dateModified` and `dateCreated` metadata
-
-The section "Metadata name restrictions" establishes that builtin metadata names cannot be used as user metadta
-names. Thus, `dateCreated` and `dateModified` (which are builtin metadata) cannot be used as user metadata names.
-
-However, due to legacy reasons, Orion doesn't reject attempts of using these metadata with 400 Bad Request errors. **You are
-strongly encouraged to not use dateCreate/dateModified as metadata names** but, if you at the end need to do so, take into 
-account the following:
-
-You can use dateModified/dateCreated metadata at creation/update time, e.g.:
-
-```
-POST /v2/entities
-
-{
-  "id": "FutureEntity",
-  "type": "T",
-  ...
-  "futureAttr": {
-      "value": 42,
-      "type": "Number",
-      "dateModified": {
-          "type": "DateTime".
-           "value": "2050-01-01T00:00:00Z"
-      }
-  }
-}
-```
-
-However, they are ignored. Orion will keep attribute creation/modification internally and always using them in GET responses,
-notifications and filters.
-
-This behaviour is assessed by the functional test `attrs_dates_overriden_by_user.test`
-and `attrs_dates_overriden_by_user_subs.test` cases included in
-[`cases/0876_attribute_dates directory`](https://github.com/telefonicaid/fiware-orion/tree/master/test/functionalTest/cases/0876_attribute_dates).
-Please have a look, if you need a more detailed description.
+## User attributes or metadata matching builtin name
+
+(The content of this section applies to all builtins except `dateExpires` attribute. Check the document
+[on transient entities](transient_entities.md) for specific information about `dateExpires`).
+
+First of all: **you are strongly encouraged to not use attributes or metadata with the same name as an
+NGSIv2 builtin**. In fact, the NGSIv2 specification forbids that (check "Attribute names restrictions" and
+"Metadata names restrictions" sections in the specification).
+
+However, if you are forced to have such attributes or metadata (maybe due to legacy reasons) take into
+account the following considerations:
+
+* You can create/update attributes and/or metadata which name is the same of a NGSIv2 builtin.
+  Orion will let you do so.
+* User defined attributes and/or metadata are shown without need to explicit declare it in the GET request
+  or subscription. For instance, if you created a `dateModified` attribute with value
+  "2050-01-01" in entity E1, then `GET /v2/entities/E1` will retrieve it. You don't need to use
+  `?attrs=dateModified`.
+* When rendered (in response to GET operations or in notifications) the user defined attribute/metadata
+  will take preference over the builtin even when declared explicitly. For instance, if you created
+  a `dateModified` attribute with value "2050-01-01" in entity E1 and you request
+  `GET /v2/entities?attrs=dateModified` you will get "2050-01-01".
+* However, filtering (i.e. `q` or `mq`) is based on the value of the builtin. For instance, if you created
+  a `dateModified` attribute with value "2050-01-01" in entity E1 and you request
+  `GET /v2/entities?q=dateModified>2049-12-31` you will get no entity. It happens that "2050-01-01" is
+  greater than "2049-12-31" but the date you modified the entity (some date in 2018 or 2019 maybe) will
+  not be greater than "2049-12-31". Note this is somehow inconsistent (i.e. user defined takes preference
+  in rendering but not in filtering) and may change in the future.
 
 [Top](#top)
 

doc/manuals/user/transient_entities.md

@@ -130,7 +130,7 @@ interpreted as an expiration date (i.e. the entity will not be deleted after the
 
 However, even in the case the attribute would keep its previous value without any special semantic, note
 that `dataExpires` becomes a builtin attribute, so it is not shown except if explicitly requested with
-`attrs` URI parameter (in GET based queries) or `"attrs"` field (in `POST /v2/op/query`).
+`attrs` URI parameter (in GET based queries) or `"attrs"` field (in `POST /v2/op/query` and subscriptions).
 
 Once `dateExpires` attribute get updated for first time, it will start to mean an expiration date on the given
 entity, with the behaviour described in previous section. Please, **take this into account in the case

scripts/accumulator-server.py

@@ -207,6 +207,16 @@ def bad_response():
     r.data = '{"name":"ENTITY_NOT_FOUND","message":"The entity with the requested id [qa_name_01] was not found."}'
     return r
 
+# From https://stackoverflow.com/questions/14902299/json-loads-allows-duplicate-keys-in-a-dictionary-overwriting-the-first-value
+def dict_raise_on_duplicates(ordered_pairs):
+    """Reject duplicate keys."""
+    d = {}
+    for k, v in ordered_pairs:
+        if k in d:
+           raise ValueError("duplicate key: %r" % (k,))
+        else:
+           d[k] = v
+    return d
 
 def record_request(request):
     """
@@ -262,9 +272,12 @@ def record_request(request):
     if ((request.data is not None) and (len(request.data) != 0)):
         s += '\n'
         if pretty == True:
-            raw = json.loads(request.data)
-            s += json.dumps(raw, indent=4, sort_keys=True)
-            s += '\n'
+            try:
+                raw = json.loads(request.data, object_pairs_hook=dict_raise_on_duplicates)
+                s += json.dumps(raw, indent=4, sort_keys=True)
+                s += '\n'
+            except ValueError as e:
+                s += str(e)
         else:
             s += request.data
 

src/lib/apiTypesV2/Attribute.cpp

@@ -132,11 +132,11 @@ void Attribute::fill(QueryContextResponse* qcrsP, std::string attrName)
 
     ContextElementResponse* cerP = qcrsP->contextElementResponseVector[0];
 
-    for (std::size_t i = 0; i < cerP->contextElement.contextAttributeVector.size(); ++i)
+    for (std::size_t i = 0; i < cerP->entity.attributeVector.size(); ++i)
     {
-      if (cerP->contextElement.contextAttributeVector[i]->name == attrName)
+      if (cerP->entity.attributeVector[i]->name == attrName)
       {
-        pcontextAttribute = cerP->contextElement.contextAttributeVector[i];
+        pcontextAttribute = cerP->entity.attributeVector[i];
         break;
       }
     }

src/lib/apiTypesV2/Entities.cpp

@@ -62,11 +62,13 @@ Entities::~Entities()
 */
 std::string Entities::render
 (
-  std::map<std::string, bool>&         uriParamOptions,
-  std::map<std::string, std::string>&  uriParam
+  RenderFormat                     renderFormat,
+  const std::vector<std::string>&  attrsFilter,
+  bool                             blacklist,
+  const std::vector<std::string>&  metadataFilter
 )
 {
-  return vec.render(uriParamOptions, uriParam);
+  return vec.render(renderFormat, attrsFilter, blacklist, metadataFilter);
 }
 
 
@@ -131,7 +133,7 @@ void Entities::fill(QueryContextResponse* qcrsP)
 
   for (unsigned int ix = 0; ix < qcrsP->contextElementResponseVector.size(); ++ix)
   {
-    ContextElement* ceP = &qcrsP->contextElementResponseVector[ix]->contextElement;
+    Entity* eP = &qcrsP->contextElementResponseVector[ix]->entity;
     StatusCode* scP = &qcrsP->contextElementResponseVector[ix]->statusCode;
 
     if (scP->code == SccReceiverInternalError)
@@ -146,16 +148,16 @@ void Entities::fill(QueryContextResponse* qcrsP)
     }
     else
     {
-      Entity*         eP  = new Entity();
+      Entity*         newP  = new Entity();
 
-      eP->id        = ceP->entityId.id;
-      eP->type      = ceP->entityId.type;
-      eP->isPattern = ceP->entityId.isPattern;
-      eP->creDate   = ceP->entityId.creDate;
-      eP->modDate   = ceP->entityId.modDate;
+      newP->id        = eP->id;
+      newP->type      = eP->type;
+      newP->isPattern = eP->isPattern;
+      newP->creDate   = eP->creDate;
+      newP->modDate   = eP->modDate;
 
-      eP->attributeVector.fill(&ceP->contextAttributeVector);
-      vec.push_back(eP);
+      newP->attributeVector.fill(eP->attributeVector);
+      vec.push_back(newP);
     }
   }
 }

src/lib/apiTypesV2/Entities.h

@@ -55,8 +55,10 @@ class Entities
   Entities();
   ~Entities();
 
-  std::string  render(std::map<std::string, bool>&         uriParamOptions,
-                      std::map<std::string, std::string>&  uriParam);
+  std::string  render(RenderFormat                     renderFormat,
+                      const std::vector<std::string>&  attrsFilter,
+                      bool                             blacklist,
+                      const std::vector<std::string>&  metadataFilter);
 
   std::string  check(RequestType requestType);
   void         release(void);