Rails OpenTelemetry error: Unable to export spans

[RahulMahale]

Share details about your runtime

Operating system details: Linux, Ubuntu 20.04 LTS
RUBY_ENGINE: "ruby"
RUBY_VERSION: "2.7.6"
RUBY_DESCRIPTION: "2.7.6p219 (2022-02-18 revision 53f5fc4236) [arm64-darwin21]"

Share a simplified reproduction if possible

Gemfile contents

gem 'opentelemetry-sdk'
gem 'opentelemetry-exporter-otlp'
gem 'opentelemetry-instrumentation-rails', '~> 0.24.0'
gem 'opentelemetry-instrumentation-mysql2', '~> 0.21.1'
gem 'opentelemetry-instrumentation-sidekiq', '~> 0.22.0'
end

config/environment.rb contents

require 'opentelemetry/sdk'
require_relative "application"
require_relative "application"


otel_exporter = OpenTelemetry::Exporter::OTLP::Exporter.new
processor = OpenTelemetry::SDK::Trace::Export::SimpleSpanProcessor.new(otel_exporter)
OpenTelemetry::SDK.configure do |c|
    c.add_span_processor(processor)
    c.use 'OpenTelemetry::Instrumentation::Rails'
    c.use 'OpenTelemetry::Instrumentation::Mysql2'
    c.use 'OpenTelemetry::Instrumentation::Sidekiq'
  end

Error while starting the app.

[39] * Listening on unix:///data/circle/shared/tmp/sockets/puma.sock?backlog=1024
[39] ! WARNING: Detected 3 Thread(s) started in app boot:
[39] ! #<Thread:0x00007fd49b4f1b00@FFI Callback Dispatcher sleep> -
[39] ! #<Thread:0x00007fd49b3dcb48 /usr/local/bundle/gems/rdkafka-0.12.0/lib/rdkafka/config.rb:19 sleep_forever> - /usr/local/bundle/gems/rdkafka-0.12.0/lib/rdkafka/config.rb:21:in `pop'
[39] ! #<Thread:0x00007fd48afe6058 /usr/local/bundle/gems/opentelemetry-sdk-1.2.0/lib/opentelemetry/sdk/trace/export/batch_span_processor.rb:180 sleep> - /usr/local/bundle/gems/opentelemetry-sdk-1.2.0/lib/opentelemetry/sdk/trace/export/batch_span_processor.rb:162:in `sleep'
[39] Use Ctrl-C to stop
[39] * Starting control server on unix:///tmp/puma-status-1671463568694-39
[39] - Worker 0 (PID: 48) booted in 0.01s, phase: 0
[39] - Worker 1 (PID: 58) booted in 0.01s, phase: 0
[39] - Worker 2 (PID: 66) booted in 0.01s, phase: 0
[39] * Starting Yabeda Prometheus metrics exporter on http://0.0.0.0:5100/metrics
10.0.19.41 - - [19/Dec/2022:15:26:37 +0000] "GET /metrics HTTP/1.1" 200 - 0.0053
[d1a10944-670f-4ab1-9068-fad724c1d927] Started GET "/healthcheck" for 127.0.0.1 at 2022-12-19 15:26:46 +0000
[d1a10944-670f-4ab1-9068-fad724c1d927] Processing by HealthcheckController#index as */*
10.0.7.38 - - [19/Dec/2022:15:26:47 +0000] "GET /healthcheck HTTP/1.1" 200 19 "-" "kube-probe/1.21+"
[d1a10944-670f-4ab1-9068-fad724c1d927] [ahoy] Visit excluded
[d1a10944-670f-4ab1-9068-fad724c1d927]   Rendered text template (Duration: 0.1ms | Allocations: 9)
[d1a10944-670f-4ab1-9068-fad724c1d927] Completed 200 OK in 422ms (Views: 1.9ms | ActiveRecord: 6.4ms | Allocations: 266382)
{"identifier":"text template","layout":null,"name":"render_template.action_view","transaction_id":"86aa8d1a9b02622db78f","duration":0.24,"request_id":"d1a10944-670f-4ab1-9068-fad724c1d927","source":"unknown","tags":[],"@timestamp":"2022-12-19T15:26:47.297Z","@version":"1"}
{"method":"GET","path":"/healthcheck","format":"*/*","controller":"healthcheck","action":"index","status":200,"duration":421.79,"view":1.9,"db":6.42,"ip":"127.0.0.1","route":"healthcheck#index","request_id":"d1a10944-670f-4ab1-9068-fad724c1d927","source":"unknown","tags":["request"],"@timestamp":"2022-12-19T15:26:47.298Z","@version":"1"}
E, [2022-12-19T15:26:52.260670 #48] ERROR -- : OpenTelemetry error: Unable to export 5 spans
[a764c92d-0d90-4d72-952d-1460e879d0d2] Started GET "/healthcheck" for 127.0.0.1 at 2022-12-19 15:26:56 +0000
[a764c92d-0d90-4d72-952d-1460e879d0d2] Processing by HealthcheckController#index as */*
10.0.7.38 - - [19/Dec/2022:15:26:57 +0000] "GET /healthcheck HTTP/1.1" 200 19 "-" "kube-probe/1.21+"
[a764c92d-0d90-4d72-952d-1460e879d0d2] [ahoy] Visit excluded
[a764c92d-0d90-4d72-952d-1460e879d0d2]   Rendered text template (Duration: 0.1ms | Allocations: 9)
[a764c92d-0d90-4d72-952d-1460e879d0d2] Completed 200 OK in 189ms (Views: 1.8ms | ActiveRecord: 5.6ms | Allocations: 140114)
E, [2022-12-19T15:27:02.025610 #66] ERROR -- : OpenTelemetry error: Unable to export 5 spans

[arielvalentin]

@RahulMahale This is not an issue that I think we can qualify as a bug since the exporter is reporting that it is unable to export spans and looks like you are looking for help debugging your collector integration, therefore I am going to transfer this to a discussion.

[arielvalentin]

@RahulMahale your log output does not match the code the example code you are referencing as it looks like there is a default batch span processor attempting to export spans to the default OTLP Exporter endpoint.

It also looks like you are configuring the SDK the config/environment.rb, however I guides recommend you use a custom initializer:

https://opentelemetry.io/docs/instrumentation/ruby/getting-started/#initialization

Without more information about your collector configurations or deployment environment it will be difficult to help you diagnose what is happening.

[RahulMahale]

@arielvalentin thanks for checking.

There was a typo in the OTEL exporter URL, Just pre-fixed the http before the Kubernetes service name and it worked.