Flattening of data in log data model example mappings

[jmm]

Hello,

The log data model example mappings include things like Elastic Common Schema error.id => attributes["error.id"]. That's illustrating flattening the ECS structure into top-level properties of Attributes (the casing "attributes" here appears to be a mistake), right? I'd be interested in understanding why it was decided to do it that way. That seems like a good thing to comment on in the specification.

[SergeyKanzhelev]

First, status of the document is still experimental: https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/logs/data-model.md#elastic-common-schema So if you have a suggestion what needs to be fixed, please create issue and/or PR to fix things.

As for flattening, do you suggest the error be an object with id as one of the fields?

[jmm]

@SergeyKanzhelev thanks for your reply.

As for flattening, do you suggest the error be an object with id as one of the fields?

That seems to me like the most natural way to map it since Attributes supports that structure and the data already has that structure in the source model. But if people think otherwise, I'd like to understand the reasoning.

Also, one of the stated requirements of the log data model is that:

Translating log data from an arbitrary log format to this Data Model and back should ideally result in identical data.

If the original ECS document is...

{
  "error": {
    "id": "..."
  }
}

...and you flatten it into Attributes, then to round trip it you have to get back to that anyway.

---

If I were to create an issue at this point it would repeat what I posted here (which is primarily a couple of questions). I could go either way on it being an issue or discussion -- I'm happy to file it as an issue if you think that's a good / better fit based on this project's preferences for issues.

What's the guidance on when an issue or discussion is more appropriate (or Slack or Gitter)? It's not super clear to me from the contributing guidelines in this repo or the OpenTelemetry general contributing guidelines it links to. But that does say:

For general questions and troubleshooting, use the standard lines of communication.

And that document says:

We use GitHub discussions for most communications.

[tigrannajaryan]

Elastic mapping was proposed by Elastic folks, they can answer this question best. @cyrille-leclerc can you help?

[cyrille-leclerc]

Thanks @jmm and @tigrannajaryan.

Regarding the case typo on the experimental mapping Elastic ECS and OpenTelemetry Unified Model, I have created #1641 to fix the problem.

Regarding the question of flattening the data structured, Elastic has presented to the OpenTelemetry working group:

• The benefits we saw in having the top level element being semantic elements (e.g. "service", "cloud", "host"...) rather than wrapping them in "technical" top level elements "Attributes", "Resource", and "Body" with a delineation that may not be intuitive (what should be an "Attribute", what should be in the "Body"?)
• The benefits of supporting nested data structures acknowledging that Elastic, for the sake of human editing, supports both flattened and hierarchical data structures.
• Our understanding that OpenTelemetry already successfully uses flattened fields for traces and metrics so we see the value in being consistent also using a flattened model for logs

We felt the thought process of the working group was fruitful and we are eager to experience OpenTelemetry logs, @urso is working at it implementing an OpenTelemetry Collector exporter for Elasticsearch (see open-telemetry/opentelemetry-collector-contrib#3101). We hope to be able to share with the community our lessons learned soon.

Regarding your point on the double mapping of logs from ECS to OpenTelemetry to ECS, you are right that we would lose the information of whether the original message was using flattened fields or hierarchical fields but I this loss would not change the semantic of the message and I think the same challenge exists internally in Elasticsearch.

Does it make sense?

[jmm]

Thanks @tigrannajaryan and @cyrille-leclerc.

The benefits we saw in having the top level element being semantic elements [...]

Thanks for that context.

with a delineation that may not be intuitive (what should be an "Attribute", what should be in the "Body"?

Without commenting on the broader issue of semantic vs. technical top level elements, I have to agree on this point. I previously opened #1613 about that very thing.

---

Our understanding that OpenTelemetry already successfully uses flattened fields for traces and metrics so we see the value in being consistent also using a flattened model for logs

I actually didn't realize that the OpenTelemetry well-known fields are all flat with dot delimited names. I see it now in the examples. Thank you for mentioning that.

So in that respect it completely makes sense that mapping ECS to the unified model has a flat structure for things like error.id.

I wonder if there's a good way to call that out more explicitly, since in some other contexts (ECS for example) dot delimited names indicate nesting structure rather than literal strings.

---

The benefits of supporting nested data structures acknowledging that Elastic, for the sake of human editing, supports both flattened and hierarchical data structures.

Regarding your point on the double mapping of logs from ECS to OpenTelemetry to ECS, you are right that we would lose the information of whether the original message was using flattened fields or hierarchical fields but I this loss would not change the semantic of the message and I think the same challenge exists internally in Elasticsearch.

To make sure I understand you correctly, my understanding is that both of the following documents would have equivalent internal treatment and semantic meaning by the time they've completed ingestion into Elasticsearch, and that's what you're referencing here:

{
  "error": {
    "id": "..."
  }
}

{
  "error.id": "..."
}

However, the document in the _source field would differ, so they're not equivalent for all / external purposes. My understanding is also that they're not equivalent during the ingestion process. As you said:

the same challenge exists internally in Elasticsearch.

But, the ECS documentation seems to indicate stricter requirements for the input than Elasticsearch in general:

Guidelines for field names
[...]
Nest fields inside a field set with dots
The document structure should be nested JSON objects. If you use Beats or Logstash, the nesting of JSON objects is done for you automatically. If you’re ingesting to Elasticsearch using the API, your fields must be nested objects, not strings containing dots.

Am I understanding that correctly to say that the flat version of the document above would not be valid ECS? It's in a document of guidelines and best practices and in one case says "should", but then also says "must".

It's also rather confusing because it then links to "Why does ECS use a dot notation instead of an underline notation?", which seems to indicate that strings containing dots are interchangeable with nesting, e.g.:

Ingesting user.firstname: Nicolas and user.lastname: Ruflin is identical to ingesting the following JSON:

"user": {
  "firstname": "Nicolas",
  "lastname": "Ruflin"
}

Even if it's not a problem with regard to ECS, I think it's worth noting that if you round trip from the nested version and back using the flattened version, some fidelity is lost.