Initial Entity and Resource proposal.

[jsuereth]

This is a proposal to address Resource and Entity data model interactions, including a path forward to address immediate friction and issues in the current resource specification.

The proposal includes all links and context needed to justify it, but duplicating a snapshot here:

Motivation

This proposal attempts to focus on the following problems within OpenTelemetry to unblock multiple working groups:

• Allowing mutating attributes to participate in Resource (OTEP 208).
• Allow Resource to handle entities whose lifetimes don't match the SDK's lifetime (OTEP 208).
• Provide support for async resource lookup (spec#952).
• Fix current Resource merge rules in the specification, which most implementations violate (oteps#208, spec#3382, spec#3710).
• Allow semantic convention resource modeling to progress (spec#605, spec#559, etc).

[tigrannajaryan]

Thanks for the draft @jsuereth !

This is an important next step for entities. I left some comments, but they don't prevent us from making progress.

[austinlparker]

A question for @jsuereth (and perhaps this has been addressed in WG meetings, if so, feel free to point me to those discussions) but my read of this proposal is that there is some future state where resource attributes are all contained inside entities, rather than flat, thus changing the 'logical' top-level resource attributes into complex attributes rather than flat ones. This is not a pattern that existing tools really use; Is the intention that these attributes should be flattened on ingest in order to allow traditional group-by/analysis functions and the entity definition ingested as some sort of hash (or other platform-appropriate representation)?

[bboreham]

Drive-by comment: I expected the top-level description to reference https://github.com/open-telemetry/oteps/blob/main/text/entities/0256-entities-data-model.md

[jack-berg]

Is there any prototype implementation of this? It sounds reasonable to me but is pretty abstract.

[jsuereth]

A question for @jsuereth (and perhaps this has been addressed in WG meetings, if so, feel free to point me to those discussions) but my read of this proposal is that there is some future state where resource attributes are all contained inside entities, rather than flat, thus changing the 'logical' top-level resource attributes into complex attributes rather than flat ones. This is not a pattern that existing tools really use; Is the intention that these attributes should be flattened on ingest in order to allow traditional group-by/analysis functions and the entity definition ingested as some sort of hash (or other platform-appropriate representation)?

The goal of this is that tooling can continue to engage with the flattened resource attribtue model we provide today. optionally tooling can engage with the new bundled entity definitions (and OTEL itself, e.g. the collector, would do so).

This gives flexibility for users to engage or not engage with entities, depending on their need, and opens up further exploration of entities.

[tigrannajaryan]

Is there any prototype implementation of this? It sounds reasonable to me but is pretty abstract.

@jack-berg There are early prototypes in Go and in Collector (core and contrib). The prototypes somewhat deviate from this proposal (e.g. the prototype only allows one associated EntityRef in the Resource, it uses key/value pairs, not key arrays, etc). Despite some deviation I think the prototypes demonstrate the idea (and we can update the prototypes to align with this proposal).

[ArthurSens]

Sorry for the delay reviewing this! It's a great read and I think that a clear separation of identifying/descriptive attributes is a big step forward for better compatibility with Prometheus.

I'd like to propose something different though :)

[carlosalberto]

Overall LGTM, although I'd like to know about details going forward, to set expectations on how the SDKs will be updated:

• Will Resource stay immutable (with the addition of the new entities field), and will be internally updated by the SDK with the latest instance tracked? If not, will attributes stay as read-only at least, in a best-effort attempt?
• The merge algorithm mentions schema_url as a comparison point between entities of the same type, and then one of the examples mentions having the same schema_url with different schema version. Is this something to be tuned/clarified later on, regarding on how to merge values in such case?

[aknuds1]

LGTM, thanks (see suggested change)!

[ArthurSens]

Happy to see this idea moving forward. With clear distinction of identifying attributes I believe we can design a better UX for Prometheus as a OTel metrics backend :)

[jsuereth]

Thanks @carlosalberto !

Will Resource stay immutable (with the addition of the new entities field), and will be internally updated by the SDK with the latest instance tracked? If not, will attributes stay as read-only at least, in a best-effort attempt?

I think a follow on OTEP will define the needs and design for "mutable" resource. This OTEP calls out that users want descriptive/mutable attributes on Resource. I expect the ResourceProvider concept to be expanded on to provide that (cc @tedsuo who I think is working on that proposal, or updating the existing one).

For now - this proposal has resource remain read-only and defines a merge algorithm for a ResourceProvider, but with identifying "descriptive" (or changing) attributes as a first step. This does not solve the needs of browser until the follow on OTEP is complete and approved.

The merge algorithm mentions schema_url as a comparison point between entities of the same type, and then one of the examples mentions having the same schema_url with different schema version. Is this something to be tuned/clarified later on, regarding on how to merge values in such case?

We clarify schema url versions in the specificaiton already - https://github.com/open-telemetry/opentelemetry-specification/tree/main/specification/schemas#schema-url

So we already have a way to understand version and whether or not they match. We do not have any notion of "compatibility" encoded in version but we can check for equality and order them (to know which is latest).

[annabokhan]

Nothing blocking, follow ups noted

[jsuereth]

Thanks @annabokhan !

[svrnm]

LGTM! Just added a comment for things we might want to address in the future as well

[tigrannajaryan]

We have a broad consensus, all comments are resolved. Merging. Thanks @jsuereth