Add a timeout alignment property to batch config #10733
Open
+158
−18
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add a boolean timeout alignment property to batch processor config and batch sender config. This is to indicate if a batch submission should be aligned at the timeout boundary.
Dropbox is adopting Open Telemetry for their internal metrics system. The system uses 10s sampling rate and expects data submission to be more or less at the sampling rate boundary. This is to ensure metric data is ingested and streamed in time to various backend processes.
Without the timeout alignment feature, metric data can be submitted quite late. Consider the case when timeout is 10s and the batch processor happens to start at 00:00:07 mark. A request{ts=00:00:09} is sent to the batch processor. This request is associated with ts.truncateUp(samplingRate)=00:00:10 system timestamp. However, it is not until 00:00:07 + 10s = 00:00:17 that a batch is submitted. This data submission is quite late.
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #10733 +/- ##
=======================================
Coverage ? 92.06%
=======================================
Files ? 404
Lines ? 18833
Branches ? 0
=======================================
Hits ? 17339
Misses ? 1134
Partials ? 360 ☔ View full report in Codecov by Sentry. |
|
@evan-bradley Hi Evan, can I get a review on this PR please? |
|
This PR was marked stale due to lack of activity. It will be closed in 14 days. |
|
This PR was marked stale due to lack of activity. It will be closed in 14 days. |
|
This PR was marked stale due to lack of activity. It will be closed in 14 days. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Add a boolean timeout alignment property to batch processor config and batch sender config. This is to indicate if a batch submission should be aligned at the timeout boundary.
Dropbox is adopting Open Telemetry for their internal metrics system. The system uses 10s sampling rate and expects data submission to be more or less at the sampling rate boundary. This is to ensure metric data is ingested and streamed in time to various backend processes.
Without the timeout alignment feature, metric data can be submitted quite late. Consider the case when timeout is 10s and the batch processor happens to start at 00:00:07 mark. A request{ts=00:00:09} is sent to the batch processor. This request is associated with ts.truncateUp(samplingRate)=00:00:10 system timestamp. However, it is not until 00:00:07 + 10s = 00:00:17 that a batch is submitted. This data submission is quite late.
Link to tracking issue
No tracking issue #
Testing
Unit tests added to verify the batch processor and batch sender behavior when timeout alignment is used.
Documentation
Inline comment to indicate the usage.