Add `otel.receiver.aws_firehose` component #6005

Obito1903 · 2023-12-19T10:14:00Z

PR Description

Add a new otel.receiver.awsfirehose based on the AWS Kinesis Data Firehose Receiver available in the opentelemetry collector.

Which issue(s) this PR fixes

Fixes grafana/alloy#287

Notes to the Reviewer

PR Checklist

CHANGELOG.md updated
Documentation added
Tests updated
Config converters updated

CLAassistant · 2023-12-19T10:14:07Z

All committers have signed the CLA.

tpaschalis

Thanks for opening your first PR on the grafana/agent repo @Obito1903 🙌

I've left some comments, mainly documentation nits as well as a comment about not needing to wire up the static mode; we're on a good path here!

tpaschalis · 2023-12-19T10:17:40Z

component/all/all.go

@@ -83,6 +83,7 @@ import (
 	_ "github.com/grafana/agent/component/otelcol/processor/span"                   // Import otelcol.processor.span
 	_ "github.com/grafana/agent/component/otelcol/processor/tail_sampling"          // Import otelcol.processor.tail_sampling
 	_ "github.com/grafana/agent/component/otelcol/processor/transform"              // Import otelcol.processor.transform
+	_ "github.com/grafana/agent/component/otelcol/receiver/awsfirehose"         


nit: can you add a comment next to this import like other components do?

tpaschalis · 2023-12-19T10:19:39Z

pkg/traces/config.go

@@ -901,6 +902,7 @@ func tracingFactories() (otelcol.Factories, error) {
 	}

 	receivers, err := receiver.MakeFactoryMap(
+		awsfirehosereceiver.NewFactory(),


I think we don't need this line; it would only make sense for static mode where we haven't wired up the rest of the code or documented as an option. cc @ptodev

component/otelcol/receiver/awsfirehose/awsfirehose.go

tpaschalis · 2023-12-19T10:28:30Z

docs/sources/flow/reference/components/otelcol.receiver.awsfirehose.md

+
+Name | Type | Description | Default | Required
+---- | ---- | ----------- | ------- | --------
+`record_type` | `string` | The type of record being received from the delivery stream. | `cwmetrics` | no


Can we also document AccessKey here?

tpaschalis · 2023-12-19T10:33:27Z

docs/sources/flow/reference/components/otelcol.receiver.awsfirehose.md

+`otelcol.receiver.awsfirehose` accepts cloudwatch json-formatted traces over the network and
+forwards it to other `otelcol.*` components.


The upstream component mentions this as its intro:

Receiver for ingesting AWS Kinesis Data Firehose delivery stream messages and parsing the records received based on the configured record type.

Are the two statements compatible (I'm not familiar with firehose tbh). Also, just to make sure, does this receiver receive JSON-formatted traces and forwards metrics generated out of them?

Finally, the upstream docs mention that:

The AWS Kinesis Data Firehose Delivery Streams currently only support HTTPS endpoints using port 443. This can be potentially circumvented using a Load Balancer.

Is it applicable to this component as well? Should we copy this over?

To answer your first question, yes, the two statements are compatible, as AWS Kinesis Data Firehose Delivery Streams is the service used by Cloudwatch to send metrics streams to an HTTP endpoint. I will clarify it in the doc.

For the second point, I did not explain it clearly. The receiver receives JSON-formatted metrics from AWS and forward them to other otel components.

As for the HTTPS-only endpoint, since it's a requirement from the AWS side, I will add the information to the doc.

clayton-cornell

Some documentation suggestions...

docs/sources/flow/reference/components/otelcol.receiver.awsfirehose.md

clayton-cornell · 2024-01-08T20:25:15Z

@tpaschalis Is this one ready to move to the next stage of approvals?

tpaschalis · 2024-01-09T09:16:54Z

@clayton-cornell yes, LGTM from a code perspective

tpaschalis · 2024-01-09T09:17:34Z

Will have to look into the checksum mismatch being reported here, it's weird

docs/sources/flow/reference/components/otelcol.receiver.awsfirehose.md

component/otelcol/receiver/awsfirehose/awsfirehose.go

docs/sources/flow/reference/components/otelcol.receiver.awsfirehose.md

Co-authored-by: Paschalis Tsilias <[email protected]>

…ehose.md Co-authored-by: Clayton Cornell <[email protected]>

…ehose.md Co-authored-by: Paulin Todev <[email protected]>

Signed-off-by: Paschalis Tsilias <[email protected]>

tpaschalis

LGTM. I've taken the liberty to fix various chores around documentation to make this easier to merge.

I'd like to let @ptodev take a final look and merge this.

ptodev

I noted a few issues which need fixing. Other than that, it looks ok to me.

docs/sources/flow/reference/components/otelcol.receiver.aws_firehose.md

ptodev · 2024-02-21T12:29:16Z

docs/sources/flow/reference/components/loki.source.awsfirehose.md

Why does this PR change the loki.source component docs? It's probably by accident?

Yeah, I accidentally did this as I was trying to make the check for compatible components pass, I'll revert.

docs/sources/flow/reference/components/otelcol.receiver.aws_firehose.md

Signed-off-by: Paschalis Tsilias <[email protected]>

ptodev

This PR will also need a change similar to the one here:
https://github.com/grafana/agent/pull/6475/files

…rehose.md Co-authored-by: Paulin Todev <[email protected]>

Signed-off-by: Paschalis Tsilias <[email protected]>

clayton-cornell

Just a note on aliases. I don't think we really need to add them on a new topic.. only if the topic is renamed/moved after publishing.

All the Cloud aliases are for historical mount points for topics that were published when things got shuffled in Cloud. Since this is a new topic... we can drop them until things get shuffled again. :-)

clayton-cornell · 2024-03-01T17:05:14Z

docs/sources/flow/reference/components/otelcol.receiver.aws_firehose.md

+aliases:
+- /docs/grafana-cloud/agent/flow/reference/components/otelcol.receiver.aws_firehose/
+- /docs/grafana-cloud/monitor-infrastructure/agent/flow/reference/components/otelcol.receiver.aws_firehose/
+- /docs/grafana-cloud/monitor-infrastructure/integrations/agent/flow/reference/components/otelcol.receiver.aws_firehose/
+- /docs/grafana-cloud/send-data/agent/flow/reference/components/otelcol.receiver.aws_firehose/


New topic? We don't really need to add all these aliases. We only need them if the topic moves or is renamed after it has been published at least once. It's not hurting anything to add them, but they aren't necessary.

clayton-cornell · 2024-06-03T17:01:35Z

@ptodev This one seems to have been missed... is it still valid/relevant?

Obito1903 requested a review from clayton-cornell as a code owner December 19, 2023 10:14

tpaschalis reviewed Dec 19, 2023

View reviewed changes

clayton-cornell reviewed Dec 19, 2023

View reviewed changes

clayton-cornell added the type/docs Docs Squad label across all Grafana Labs repos label Dec 19, 2023

tpaschalis force-pushed the otel-awsfirehose branch from d4b38ef to f98b484 Compare January 9, 2024 09:20

ptodev reviewed Jan 9, 2024

View reviewed changes

Obito1903 and others added 15 commits February 20, 2024 15:41

Add otel.receiver.awsfirehose component

1732e07

Update component/otelcol/receiver/awsfirehose/awsfirehose.go

8b6deaa

Co-authored-by: Paschalis Tsilias <[email protected]>

Update component/otelcol/receiver/awsfirehose/awsfirehose.go

dfb3482

Co-authored-by: Paschalis Tsilias <[email protected]>

Update according pr suggestions

afc64f7

Update docs/sources/flow/reference/components/otelcol.receiver.awsfir…

2b1f5d9

…ehose.md Co-authored-by: Clayton Cornell <[email protected]>

Update docs/sources/flow/reference/components/otelcol.receiver.awsfir…

d217b03

…ehose.md Co-authored-by: Clayton Cornell <[email protected]>

Update docs/sources/flow/reference/components/otelcol.receiver.awsfir…

ee32667

…ehose.md Co-authored-by: Clayton Cornell <[email protected]>

Update docs/sources/flow/reference/components/otelcol.receiver.awsfir…

3158090

…ehose.md Co-authored-by: Clayton Cornell <[email protected]>

Update docs/sources/flow/reference/components/otelcol.receiver.awsfir…

aaa0b62

…ehose.md Co-authored-by: Clayton Cornell <[email protected]>

Update docs/sources/flow/reference/components/otelcol.receiver.awsfir…

69bc356

…ehose.md Co-authored-by: Clayton Cornell <[email protected]>

Update docs/sources/flow/reference/components/otelcol.receiver.awsfir…

8262acb

…ehose.md Co-authored-by: Paulin Todev <[email protected]>

Update docs/sources/flow/reference/components/otelcol.receiver.awsfir…

d50287e

…ehose.md Co-authored-by: Paulin Todev <[email protected]>

Update docs/sources/flow/reference/components/otelcol.receiver.awsfir…

041d825

…ehose.md Co-authored-by: Paulin Todev <[email protected]>

Update docs/sources/flow/reference/components/otelcol.receiver.awsfir…

38a1866

…ehose.md Co-authored-by: Paulin Todev <[email protected]>

Tidy up, regenerate go.sum, docs

4279489

Signed-off-by: Paschalis Tsilias <[email protected]>

tpaschalis force-pushed the otel-awsfirehose branch from a9ab0f4 to 4279489 Compare February 20, 2024 13:53

tpaschalis and others added 3 commits February 20, 2024 16:02

Fix test

94d4533

Signed-off-by: Paschalis Tsilias <[email protected]>

Add changelog entry

b79ad9d

Signed-off-by: Paschalis Tsilias <[email protected]>

Merge branch 'main' into otel-awsfirehose

4fbbae8

tpaschalis approved these changes Feb 20, 2024

View reviewed changes

tpaschalis changed the title ~~Add otel.receiver.awsfirehose component~~ Add otel.receiver.aws_firehose component Feb 20, 2024

ptodev reviewed Feb 21, 2024

View reviewed changes

tpaschalis self-requested a review February 21, 2024 12:47

tpaschalis added 3 commits February 21, 2024 14:51

Fix correct docs

486b376

Signed-off-by: Paschalis Tsilias <[email protected]>

Fix debug metrics docs

6b061ee

Signed-off-by: Paschalis Tsilias <[email protected]>

Remove unused example outputs

31a2b6f

Signed-off-by: Paschalis Tsilias <[email protected]>

ptodev requested changes Feb 22, 2024

View reviewed changes

tpaschalis and others added 2 commits February 26, 2024 12:44

Update docs/sources/flow/reference/components/otelcol.receiver.aws_fi…

d6181f1

…rehose.md Co-authored-by: Paulin Todev <[email protected]>

Fix docs, set default high cardinality labels

74c3a09

Signed-off-by: Paschalis Tsilias <[email protected]>

tpaschalis requested a review from ptodev February 26, 2024 12:02

Merge branch 'main' into otel-awsfirehose

ac76d87

clayton-cornell reviewed Mar 1, 2024

View reviewed changes

rfratto added the variant/flow Relatd to Grafana Agent Flow. label Apr 9, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add `otel.receiver.aws_firehose` component #6005

Add `otel.receiver.aws_firehose` component #6005

Obito1903 commented Dec 19, 2023

CLAassistant commented Dec 19, 2023 •

edited

Loading

tpaschalis left a comment

tpaschalis Dec 19, 2023

tpaschalis Dec 19, 2023

tpaschalis Dec 19, 2023

tpaschalis Dec 19, 2023

Obito1903 Dec 19, 2023

clayton-cornell left a comment

clayton-cornell commented Jan 8, 2024

tpaschalis commented Jan 9, 2024

tpaschalis commented Jan 9, 2024

tpaschalis left a comment

ptodev left a comment

ptodev Feb 21, 2024

tpaschalis Feb 21, 2024

ptodev left a comment

clayton-cornell left a comment

clayton-cornell Mar 1, 2024

clayton-cornell commented Jun 3, 2024

		`otelcol.receiver.awsfirehose` accepts cloudwatch json-formatted traces over the network and
		forwards it to other `otelcol.*` components.

Add otel.receiver.aws_firehose component #6005

Are you sure you want to change the base?

Add otel.receiver.aws_firehose component #6005

Conversation

Obito1903 commented Dec 19, 2023

PR Description

Which issue(s) this PR fixes

Notes to the Reviewer

PR Checklist

CLAassistant commented Dec 19, 2023 • edited Loading

tpaschalis left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

clayton-cornell left a comment

Choose a reason for hiding this comment

clayton-cornell commented Jan 8, 2024

tpaschalis commented Jan 9, 2024

tpaschalis commented Jan 9, 2024

tpaschalis left a comment

Choose a reason for hiding this comment

ptodev left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ptodev left a comment

Choose a reason for hiding this comment

clayton-cornell left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

clayton-cornell commented Jun 3, 2024

Add `otel.receiver.aws_firehose` component #6005

Add `otel.receiver.aws_firehose` component #6005

CLAassistant commented Dec 19, 2023 •

edited

Loading