How to ignore some PostgreSQL commands ?

[kedare]

Hello

I am looking to disable spans for specific SQL commands on the PostgreSQL integration,
Typically I get a lot of SET spans before each queries.

Looks like there is no "clean" way to do this ? So far I think the only way to do it would be to remove SET from the constants ? (not tested yet) ( https://github.com/open-telemetry/opentelemetry-ruby-contrib/blob/main/instrumentation/pg/lib/opentelemetry/instrumentation/pg/constants.rb#L58 )

Thanks

[arielvalentin]

@kedare There isn't anything in the API or SDK specification that provides granular suppression of spans so there isn't an easy way to do this in the SDK.

One option you have is to write a custom sampler that samples out specific commands however that will be quite slow and I would not recommend it.

The only supported option, AFAIK, is to filter the spans in the OTLP collector: https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/processor/filterprocessor/README.md#ottl-examples

Hope that helps.

[ahayworth]

@kedare Removing it from that array won't actually suppress a span. The span just won't have SET in the name. 😆

Can you share the statements that are actually generating this? One thing we could do is actually implement some filters that skip really noisy things - if these statements aren't really helpful in the vast majority of cases, we might consider unilaterally suppressing such spans. But, without seeing the statements in front of me it's hard to know what value they might provide to others.

(This instrumentation is a little chatty, especially when you're using it with Rails. That's been something I've worried about since we initially wrote it, but you're one of the first to mention that it's troublesome.)

[kedare]

The kind of SET spans I get a lot are typically the ones (I think) ran automatically when the client take a connection from the ActiveRecord pool.

SET client_min_messages TO 'warning'
SET standard_conforming_strings = on
SET intervalstyle = iso_8601
SET SESSION timezone TO 'UTC'

[ahayworth]

@kedare Gotcha, that makes sense. I definitely think these kinds of spans are not super important for the most part (unless you're trying to debug AR connection pool behavior).

I wonder if a good answer might be a sampler that the ActiveRecord instrumentation installs that would suppress such spans from Postgres?

Curious what other @open-telemetry/ruby-contrib-approvers and @open-telemetry/ruby-contrib-maintainers think about that...

[akiellor]

Hey folks, I'm running into a similar problem where the pg instrumentation is creating a lot of spans when a connection is checked out of a connection pool. ActiveRecord execs the query ; to check if the connection is active.

Would adding an untraced_queries option to the instrumentation suffice? There is some prior art of this pattern in the net_http instrumentation in the form of untraced_hosts.

I'd be happy to implement such an option, I'd just like to get some feedback on the direction before committing too much effort.

[zacheryph]

I see this as well with DEALLOCATE queries. They are root spans with zero children.

This is a small app and these account for 6% of root spans / traces. Not a big deal but its just unnecessary noise.

Rack has a similar untraced_endpoints option.

[zacheryph]

Made a PR for this; #988

[akiellor]

I did something very similar and got some feedback that there is a desire to have fewer options on instrumentations.

[robbkidd]

For others coming across this issue here in a discussion, I added a comment to #988 with an option for filtering out unwanted spans with the OpenTelemetry Collector. If the spans from the PG instrumentation are leaf spans in your traces (i.e. they have no child spans), then it's safe to trim them off with the filterprocessor.