OpenTelemetry OTLP setup for tracing, take 2 #697
+611
−198
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Replace the "movement_timing" tracing target and the logging layer it targeted with an optionally installed OpenTelemetry OTLP exporter. The name of the tracing target matched to send OpenTelemetry events is "movement_telemetry".
Add the telemetry overlay enabling OTLP telemetry export in suzuka-full-node and m1-da-light-node.. In the telemetry overlay for suzuka-full-node, add an OTLP collector start job running a docker container.
Provide telemetry API as separate from tracing rather than a globally installed layer. Installing an OpenTelemetry layer into the global tracing subscriber raises nasty reentrancy issues because the OTLP exporter stack also uses tracing under the hood.
Install an OpenTelemetryLayer configured with the OTLP exporter. The tracing spans and events to export are selected by target "movement_telemetry".
mzabaluev
requested review from
l-monninger,
andyjsbell and
0xmovses
as code owners
The implementation of shutdown in the opentelemetry_sdk exporter calls futures_executor::block_on, which does not play well with the multithreaded Tokio runtime.
mzabaluev
force-pushed
the
mikhail/opentelemetry-through-tracing
branch
from
3a27069
to
f713d71
Compare
OpenTelemetry needs spans at the top level of its log event model at least.
Emit telemetry events detailing the success or failure
In the transaction_ingress task of suzuka-full-node and executor's transaction_pipe, add comments detailing which metrics the telemetry events are contributing to.
At the points where transaction is dropped in the submit_transaction method, emit telemetry events. These will help compute the transaction failure rate.
l-monninger
requested changes
Oct 18, 2024
| environment: | ||
|
|
||
| processes: | ||
| otlp-collector: |
There was a problem hiding this comment.
I don't seem to be getting anything served by this and the e2e tests for the basic simulation are failing. Is this how I should be using this?:
just suzuka-full-node native build.setup.eth-local.celestia-local.test.telemetry --keep-project
There was a problem hiding this comment.
This is all I'm seeing from the collector process when I start with the above.
There was a problem hiding this comment.
I added a feed overlay in case for some reason this is not in fact staying open with the --keep-project flag.
just suzuka-full-node native build.setup.eth-local.celestia-local.feed.telemetry --keep-project
|
@mzabaluev Last commit adds a test. It would be good to add to CI if you like it. |
Apply the telemetry overlay when running tests in the local setup.
I have added the overlays to the local test job, as we need some tests to drive the exporting. Do you think it would be better to test separately? |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Resolves: #555 #556 #559
Summary
protocol-units,networks,util.Integrate an OTLP exporter for tracing events matching the "movement_tracing" target.
The OTLP gRPC endpoint is configured with the MOVEMENT_OTLP environment variable.
This replaces the earlier "movement_timing" tracing layer, as analysis of OpenTelemetry data should be more versatile.
Changelog
MOVEMENT_OTLPenvironment variable.MOVEMENT_TIMING.Testing
The
telemetryoverlay for process-compose runs the jaeger all-in-one Docker container as a local OpenTelemetry collector.Unresolved issues
Add docker-compose setup for jaeger or similar.