New component: Queue Processor

[akats7]

The purpose and use-cases of the new component

The purpose of a queue processor would be to decouple the queuing mechanism from the exporterhelper util. This stems from a discussion in #33007, where the ask was for the failover connector to support queueing directly in the component. As the failover connector relies on backwards propagated errors to trigger failover but with queueing enabled on the exporters will only failover once the queue is full, leading to delayed failover and without a properly configured queue dropped data. For that reason certain users have resorted to disabling the queue.

Instead of enabling a queue mechanism directly in the failover connector, it was discussed if it might be beneficial to develop a queue processor that could be re-usable for similar uses cases in the future.

The queue processor would be a component that sits directly in front of the exporters(last in the processor chain) in the pipeline and would be responsible for providing an async export queue to the nextConsumer exporters to consume from without enabling the queue on the exporter itself.

I'd be happy to contribute this component if we are able to get it sponsored.

cc. @djaglowski @sinkingpoint

Example configuration for the component

receivers:
  otlp:

processors:
  batch:
  queue:

exporters:
  otlp:


service:
  pipelines:
    traces:
       receivers: [otlp]
       processors: [batch, queue]
       exporters: [otlp]

Telemetry data types supported

traces->traces
metrics->metrics
logs->logs

Is this a vendor-specific component?

• This is a vendor-specific component
• If this is a vendor-specific component, I am a member of the OpenTelemetry organization.
• If this is a vendor-specific component, I am proposing to contribute and support it as a representative of the vendor.

Code Owner(s)

akats7

Sponsor (optional)

No response

Additional context

No response