How does the Datadog Exporter attach metrics to Hosts?

[bertysentry]

Say we have the below pipeline:

• Custom Prometheus exporter, exposing custom metrics on /metrics
• OpenTelemetry Collector Prometheus Receiver consuming these metrics
• OpenTelemetry Collector Datadog Exporter pushing these metrics to Datadog

Problem: All metrics are attached to the same Host (in Datadog's terminology), which is simply the host where our OpenTelemetry Collector is running.

Expected: The collected metrics have a hostname label, and the corresponding Host entries should be listed in Datadog (in Infrastructure, and Host map, etc.), and the corresponding metrics should be associated to it.

The Datadog Exporter code shows that it has some logic to translate various hostname labels and convert them as Hosts. However, I tried adding various labels to my Prometheus metrics (host, hostname, host.name, datadog.host.name, datadog_host_name, etc.), and none of them triggered the creation of the corresponding Hosts.

So, what labels must be added to the metrics so that the Datadog Exporter creates the corresponding Hosts?

Thank you!

[bertysentry]

@mx-psi, @ericmustin and @KSerrania any idea? Thank you! 😊

[ericmustin]

@bertysentry I'm no longer employed with datadog and defer to @mx-psi and @KSerrania here. Off the top of my head, i would say that you may need to ensure you're operating on resource attributes, but not too sure how prometheus labels get massaged to OTLP attributes or resource attributes so can't say with much confidence whether that's already the case or whether you need to set those details somewhere else on your metrics.

[bertysentry]

Thank you @ericmustin! Yeah I think you're getting close to the cause of my problem. Don't worry about it though if you no longer work at Datadog, I just got in touch with other persons who may be able to help. Again, thank you! 😊

[mx-psi]

Thanks @ericmustin for the ping :) We will speak in private about this with @bertysentry's team (and I will post something in here for everyone to see at some point, so that people stumbling upon this can get some answers too!)

[bertysentry]

Maybe this is the solution: #6232 (Add support for metrics in groupbyattrsprocessor)
;-)

[bertysentry]

As @ericmustin mentioned, the Datadog exporter leverages the "hostname", "host.name" (and equivalent) on the Resource a Metric is attached to, to create the corresponding Hosts in Datadog. It doesn't use the attributes of the Metric itself (which makes sense).

To solve the problem raised here, #6232 (Add support for metrics in groupbyattrsprocessor) must be implemented so that Prometheus metrics can be attached to Resources that are not just the exporter host itself.