Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

otel: add support for per tenant configuration for mapping otlp data to loki format #11143

Merged
merged 7 commits into from
Jan 24, 2024

Conversation

sandeepsukhani
Copy link
Contributor

@sandeepsukhani sandeepsukhani commented Nov 6, 2023

What this PR does / why we need it:
In OTEL, we pick select Resource Attributes to identify the streams and store all the other attributes as Structured Metadata, as explained here. The problem however is that the list of Resource Attributes that are picked as Stream labels are hardcoded, and there is no way to drop unwanted data. This PR adds support for configuring how data is mapped from OTEL to Loki format per tenant. It also adds support for dropping unwanted data. We decided to make the config look similar to Prometheus's relabling config to make it familiar.

Special notes for your reviewer:
Opening a draft PR to get some initial feedback. I will add documentation in a separate PR.

Checklist

  • Tests updated
  • CHANGELOG.md updated

Comment on lines +61 to +63
type Limits interface {
OTLPConfig(userID string) OTLPConfig
}
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure why a new interface here is helpful? Can't we use the existing overrides?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It makes it easier to write tests, and we can't reference overrides since that would cause circular dependency.

@slim-bean
Copy link
Collaborator

This looks good to me @sandeepsukhani!

@github-actions github-actions bot added the type/docs Issues related to technical documentation; the Docs Squad uses this label across many repositories label Jan 24, 2024
@sandeepsukhani sandeepsukhani marked this pull request as ready for review January 24, 2024 09:41
@sandeepsukhani sandeepsukhani requested a review from a team as a code owner January 24, 2024 09:41
Copy link
Contributor

@salvacorts salvacorts left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@sandeepsukhani sandeepsukhani merged commit e625d7f into main Jan 24, 2024
9 of 10 checks passed
@sandeepsukhani sandeepsukhani deleted the per-tenant-otlp-config branch January 24, 2024 11:36
Gordejj pushed a commit to Gordejj/loki that referenced this pull request Jan 29, 2024
…to loki format (grafana#11143)

**What this PR does / why we need it**:
In OTEL, we pick select Resource Attributes to identify the streams and
store all the other attributes as Structured Metadata, as explained
here. The problem however is that the list of Resource Attributes that
are picked as Stream labels are hardcoded, and there is no way to drop
unwanted data. This PR adds support for configuring how data is mapped
from OTEL to Loki format per tenant. It also adds support for dropping
unwanted data. We decided to make the config look similar to
Prometheus's relabling config to make it familiar.

**Special notes for your reviewer**:
Opening a draft PR to get some initial feedback. I will add
documentation in a separate PR.

**Checklist**
- [x] Tests updated
- [x] `CHANGELOG.md` updated
sandeepsukhani added a commit that referenced this pull request Feb 5, 2024
**What this PR does / why we need it**:
In PR #11143 we added support for per tenant otlp config. This PR adds
the relevant documentation to explain how the config looks and works.

**Checklist**
- [x] Documentation added

---------

Co-authored-by: J Stickler <[email protected]>
rhnasc pushed a commit to inloco/loki that referenced this pull request Apr 12, 2024
…to loki format (grafana#11143)

**What this PR does / why we need it**:
In OTEL, we pick select Resource Attributes to identify the streams and
store all the other attributes as Structured Metadata, as explained
here. The problem however is that the list of Resource Attributes that
are picked as Stream labels are hardcoded, and there is no way to drop
unwanted data. This PR adds support for configuring how data is mapped
from OTEL to Loki format per tenant. It also adds support for dropping
unwanted data. We decided to make the config look similar to
Prometheus's relabling config to make it familiar.

**Special notes for your reviewer**:
Opening a draft PR to get some initial feedback. I will add
documentation in a separate PR.

**Checklist**
- [x] Tests updated
- [x] `CHANGELOG.md` updated
rhnasc pushed a commit to inloco/loki that referenced this pull request Apr 12, 2024
**What this PR does / why we need it**:
In PR grafana#11143 we added support for per tenant otlp config. This PR adds
the relevant documentation to explain how the config looks and works.

**Checklist**
- [x] Documentation added

---------

Co-authored-by: J Stickler <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
size/XL type/docs Issues related to technical documentation; the Docs Squad uses this label across many repositories
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants