[WIP] OTEL / Prom metrics benchmark

[yurishkuro]

Note that the benchmarks are designed to minimize the perf impact of counter bumps using "bounded instruments". Jaeger code always caches the instruments when some of the labels are dynamic (e.g. counts of received spans labeled with emitting service name)

$ go test -benchmem -benchtime=2s -bench=Benchmark ./internal/metrics/
goos: darwin
goarch: arm64
pkg: github.com/jaegertracing/jaeger/internal/metrics
BenchmarkPrometheusCounter-10       	342003924	         6.984 ns/op	       0 B/op	       0 allocs/op
BenchmarkOTELCounter-10             	33299455	        71.73 ns/op	       0 B/op	       0 allocs/op
BenchmarkOTELCounterWithLabel-10    	12442818	       190.6 ns/op	      16 B/op	       1 allocs/op
PASS
ok  	github.com/jaegertracing/jaeger/internal/metrics	8.415s

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 96.36%. Comparing base (afdd311) to head (c8735f6).
Report is 3 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #5676   +/-   ##
=======================================
  Coverage   96.36%   96.36%           
=======================================
  Files         329      329           
  Lines       16060    16060           
=======================================
  Hits        15477    15477           
  Misses        405      405           
  Partials      178      178           

Flag	Coverage Δ
badger_v1	8.04% <ø> (ø)
badger_v2	1.92% <ø> (ø)
cassandra-3.x-v1	16.60% <ø> (ø)
cassandra-3.x-v2	1.84% <ø> (ø)
cassandra-4.x-v1	16.60% <ø> (ø)
cassandra-4.x-v2	1.84% <ø> (ø)
elasticsearch-7.x-v1	18.88% <ø> (ø)
elasticsearch-8.x-v1	19.08% <ø> (ø)
elasticsearch-8.x-v2	19.08% <ø> (ø)
grpc_v1	9.47% <ø> (+0.01%)	⬆️
grpc_v2	7.49% <ø> (ø)
kafka	9.76% <ø> (ø)
opensearch-1.x-v1	18.93% <ø> (+0.01%)	⬆️
opensearch-2.x-v1	18.92% <ø> (-0.02%)	⬇️
opensearch-2.x-v2	18.92% <ø> (ø)
unittests	94.22% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[yurishkuro]

TODO: add Opencensus as implementation, for comparison

https://github.com/open-telemetry/opentelemetry-collector/blob/227fb823dbf33e6a62b842aef5a70e10ed1db84f/service/internal/proctelemetry/config.go#L70

[jmacd]

Curious how you feel about these two lines, and why you've excluded them from the benchmark? Ergonomically speaking, would you cache the attrOpt value after computing it? If so, you'd be better off with bound instruments. If not, you should measure it. I ask because the introduction of functional options adds a bunch of allocations, so unless you compute and re-use the []Option slice, you've either got an ergonomics problem or a performance problem. We stopped using the OTel-Go API as a result of this and installed a more-efficient functional-option-free bypass. lightstep/otel-launcher-go#446

[yurishkuro]

Ergonomically speaking, would you cache the attrOpt value after computing it?

@jmacd yes, that's exactly what Jaeger is doing, we always used "bound" instruments.

jaeger/internal/metrics/otelmetrics/counter.go

Lines 12 to 20 in 8a33800

	type otelCounter struct {
	counter metric.Int64Counter
	fixedCtx context.Context
	option metric.AddOption
	}
	func (c *otelCounter) Inc(value int64) {
	c.counter.Add(c.fixedCtx, value, c.option)
	}

But this does not help to completely avoid allocations. It's not the passing of vararg options that's causing the allocations, it's something deeper in the implementation.