Releases: aws/aws-for-fluent-bit
AWS for Fluent Bit 2.31.10
2.31.10
This release includes:
- Fluent Bit 1.9.10
- Amazon CloudWatch Logs for Fluent Bit 1.9.3
- Amazon Kinesis Streams for Fluent Bit 1.10.2
- Amazon Kinesis Firehose for Fluent Bit 1.7.2
Compared to 2.31.9
this release adds:
- Bug - Fix crash that occurs with tail plugin and debug logging enabled aws-for-fluent-bit:637 aws-for-fluent-bit:639
- Bug - Fix init tag parsing of ECS Cluster name. ECS Cluster name can be returned as ARN or name, now init will always parse it and set the
ECS_CLUSTER
env var as just the name. aws-for-fluent-bit:641
We’ve run the new released image in our ECS load testing framework and here is the result. This testing result provides benchmarks of aws-for-fluent-bit
under different input load. Learn more about the load test.
plugin | source | 20 MB/s | 25 MB/s | 30 MB/s | |
---|---|---|---|---|---|
kinesis_firehose | stdstream | Log Loss | ✅ | ✅ | ✅ |
Log Duplication | ✅ | 0%(5500) | ✅ | ||
kinesis_firehose | tcp | Log Loss | ✅ | ✅ | 0%(17906) |
Log Duplication | ✅ | ✅ | 0%(33500) | ||
kinesis_streams | stdstream | Log Loss | ✅ | ✅ | 0%(4735) |
Log Duplication | ✅ | 0%(5640) | 0%(71809) | ||
kinesis_streams | tcp | Log Loss | ✅ | ✅ | ✅ |
Log Duplication | ✅ | ✅ | 0%(26632) | ||
s3 | stdstream | Log Loss | ✅ | ✅ | ✅ |
Log Duplication | ✅ | ✅ | ✅ | ||
s3 | tcp | Log Loss | ✅ | ✅ | ✅ |
Log Duplication | ✅ | ✅ | ✅ |
plugin | source | 1 MB/s | 2 MB/s | 3 MB/s | |
---|---|---|---|---|---|
cloudwatch_logs | stdstream | Log Loss | ✅ | ✅ | 19%(346317) |
Log Duplication | ✅ | ✅ | ✅ | ||
cloudwatch_logs | tcp | Log Loss | ✅ | ✅ | ✅ |
Log Duplication | ✅ | ✅ | ✅ |
Note:
- The green check ✅ in the table means no log loss or no log duplication.
- Number in parentheses means the number of records out of total records. For example, 0%(1064/1.8M) under 30Mb/s throughput means 1064 duplicate records out of 18M input records by which log duplication percentage is 0%.
- Log loss is the percentage of data lost and log duplication is the percentage of duplicate logs received at the destination. Your results may differ because they can be influenced by many factors like different configs and environment settings. Log duplication is exclusively caused by partially succeeded batches that were retried, which means it is random.
AWS for Fluent Bit 2.31.9
2.31.9
This release includes:
- Fluent Bit 1.9.10
- Amazon CloudWatch Logs for Fluent Bit 1.9.3
- Amazon Kinesis Streams for Fluent Bit 1.10.2
- Amazon Kinesis Firehose for Fluent Bit 1.7.2
Compared to 2.31.8
this release adds:
- Enhancement - Add clear info message when chunks are removed because
storage.total_limit_size
is reached fluent-bit:6719 - Bug - Fix S3 ARN parsing in init image that prevents it from being used in US Gov Cloud and China partitions aws-for-fluent-bit:617
- Bug - Fix SIGSEGV on shutdown when multiple instances of the same go plugin are configured aws-for-fluent-bit:613
- Bug - Fix off by one error that can lead to SDS string truncation fluent-bit:7143
- Bug - fix minor memory leak in cloudwatch_logs that leads no more than ~1KB of un-freed memory when the
log_stream_name
option is configured. - Bug - Fix SIGSEGV on shutdown when multiple instances of the same go plugin are configured aws-for-fluent-bit:613
We’ve run the new released image in our ECS load testing framework and here is the result. This testing result provides benchmarks of aws-for-fluent-bit
under different input load. Learn more about the load test.
plugin | source | 20 MB/s | 25 MB/s | 30 MB/s | |
---|---|---|---|---|---|
kinesis_firehose | stdstream | Log Loss | ✅ | ✅ | ✅ |
Log Duplication | ✅ | 0%(3000) | 0%(500) | ||
kinesis_firehose | tcp | Log Loss | ✅ | ✅ | 0%(9623) |
Log Duplication | ✅ | ✅ | 0%(16500) | ||
kinesis_streams | stdstream | Log Loss | ✅ | ✅ | ✅ |
Log Duplication | ✅ | ✅ | 0%(21162) | ||
kinesis_streams | tcp | Log Loss | ✅ | ✅ | 0%(5850) |
Log Duplication | ✅ | ✅ | 0%(62574) | ||
s3 | stdstream | Log Loss | ✅ | ✅ | ✅ |
Log Duplication | ✅ | ✅ | ✅ | ||
s3 | tcp | Log Loss | ✅ | ✅ | ✅ |
Log Duplication | ✅ | ✅ | ✅ |
plugin | source | 1 MB/s | 2 MB/s | 3 MB/s | |
---|---|---|---|---|---|
cloudwatch_logs | stdstream | Log Loss | ✅ | ✅ | ✅ |
Log Duplication | ✅ | ✅ | ✅ | ||
cloudwatch_logs | tcp | Log Loss | ✅ | 1%(15080) | ✅ |
Log Duplication | ✅ | ✅ | ✅ |
Note:
- The green check ✅ in the table means no log loss or no log duplication.
- Number in parentheses means the number of records out of total records. For example, 0%(1064/1.8M) under 30Mb/s throughput means 1064 duplicate records out of 18M input records by which log duplication percentage is 0%.
- Log loss is the percentage of data lost and log duplication is the percentage of duplicate logs received at the destination. Your results may differ because they can be influenced by many factors like different configs and environment settings. Log duplication is exclusively caused by partially succeeded batches that were retried, which means it is random.
AWS for Fluent Bit 2.31.8
2.31.8
This release includes:
- Fluent Bit 1.9.10
- Amazon CloudWatch Logs for Fluent Bit 1.9.3
- Amazon Kinesis Streams for Fluent Bit 1.10.2
- Amazon Kinesis Firehose for Fluent Bit 1.7.2
Compared to 2.31.7
this release adds:
- New images - Added debug images to Amazon ECR Public Gallery, Docker Hub and Amazon ECR. For debug images, we update the
debug-latest
tag and add a tag asdebug-<Version>
.
This release is a CVE patch release for 2.31.7. It contains the same contents but re-built to uptake the most recent Amazon Linux packages with patches.
We’ve run the new released image in our ECS load testing framework and here is the result. This testing result provides benchmarks of aws-for-fluent-bit
under different input load. Learn more about the load test.
plugin | source | 20 MB/s | 25 MB/s | 30 MB/s | |
---|---|---|---|---|---|
kinesis_firehose | stdstream | Log Loss | ✅ | ✅ | ✅ |
Log Duplication | ✅ | 0%(3000) | 0%(14511) | ||
kinesis_firehose | tcp | Log Loss | ✅ | ✅ | ✅ |
Log Duplication | ✅ | ✅ | 0%(5000) | ||
kinesis_streams | stdstream | Log Loss | ✅ | ✅ | ✅ |
Log Duplication | ✅ | 0%(23170) | 0%(41590) | ||
kinesis_streams | tcp | Log Loss | ✅ | ✅ | 0%(28) |
Log Duplication | ✅ | ✅ | 0%(37752) | ||
s3 | stdstream | Log Loss | ✅ | ✅ | ✅ |
Log Duplication | ✅ | ✅ | ✅ | ||
s3 | tcp | Log Loss | ✅ | ✅ | ✅ |
Log Duplication | ✅ | ✅ | ✅ |
plugin | source | 1 MB/s | 2 MB/s | 3 MB/s | |
---|---|---|---|---|---|
cloudwatch_logs | stdstream | Log Loss | ✅ | ✅ | ✅ |
Log Duplication | ✅ | ✅ | ✅ | ||
cloudwatch_logs | tcp | Log Loss | ✅ | ✅ | ✅ |
Log Duplication | ✅ | ✅ | ✅ |
Note:
- The green check ✅ in the table means no log loss or no log duplication.
- Number in parentheses means the number of records out of total records. For example, 0%(1064/1.8M) under 30Mb/s throughput means 1064 duplicate records out of 18M input records by which log duplication percentage is 0%.
- Log loss is the percentage of data lost and log duplication is the percentage of duplicate logs received at the destination. Your results may differ because they can be influenced by many factors like different configs and environment settings. Log duplication is exclusively caused by partially succeeded batches that were retried, which means it is random.
AWS for Fluent Bit 2.28.5
2.28.5
This release includes:
- Fluent Bit 1.9.9
- Amazon CloudWatch Logs for Fluent Bit 1.9.1
- Amazon Kinesis Streams for Fluent Bit 1.10.1
- Amazon Kinesis Firehose for Fluent Bit 1.7.1
This release is a CVE patch release for 2.28.4. It contains the same contents but re-built to uptake the most recent Amazon Linux packages with patches.
We’ve run the new released image in our ECS load testing framework and here is the result. This testing result provides benchmarks of aws-for-fluent-bit
under different input load. Learn more about the load test.
plugin | source | 20 MB/s | 25 MB/s | 30 MB/s | |
---|---|---|---|---|---|
kinesis_firehose | stdstream | Log Loss | ✅ | ✅ | ✅ |
Log Duplication | 0%(2500) | 0%(2000) | ✅ | ||
kinesis_firehose | tcp | Log Loss | ✅ | ✅ | ✅ |
Log Duplication | ✅ | 0%(2500) | 0%(11000) | ||
kinesis_streams | stdstream | Log Loss | ✅ | 0%(9) | 0%(1922) |
Log Duplication | ✅ | 0%(18536) | 0%(62308) | ||
kinesis_streams | tcp | Log Loss | ✅ | ✅ | ✅ |
Log Duplication | ✅ | ✅ | 0%(28273) | ||
s3 | stdstream | Log Loss | ✅ | ✅ | ✅ |
Log Duplication | ✅ | ✅ | ✅ | ||
s3 | tcp | Log Loss | ✅ | ✅ | ✅ |
Log Duplication | ✅ | ✅ | ✅ |
plugin | source | 1 MB/s | 2 MB/s | 3 MB/s | |
---|---|---|---|---|---|
cloudwatch_logs | stdstream | Log Loss | ✅ | ✅ | ✅ |
Log Duplication | ✅ | ✅ | ✅ | ||
cloudwatch_logs | tcp | Log Loss | ✅ | ✅ | ✅ |
Log Duplication | ✅ | ✅ | ✅ |
Note:
- The green check ✅ in the table means no log loss or no log duplication.
- Number in parentheses means the number of records out of total records. For example, 0%(1064/1.8M) under 30Mb/s throughput means 1064 duplicate records out of 18M input records by which log duplication percentage is 0%.
- Log loss is the percentage of data lost and log duplication is the percentage of duplicate logs received at the destination. Your results may differ because they can be influenced by many factors like different configs and environment settings. Log duplication is exclusively caused by partially succeeded batches that were retried, which means it is random.
AWS for Fluent Bit 2.31.7
2.31.7
This release includes:
- Fluent Bit 1.9.10
- Amazon CloudWatch Logs for Fluent Bit 1.9.2
- Amazon Kinesis Streams for Fluent Bit 1.10.1
- Amazon Kinesis Firehose for Fluent Bit 1.7.1
Compared to 2.23.6
this release adds:
- Bug - Fixed S3 Key Tag Corruption with
preserve_data_ordering
option aws-for-fluent-bit:541 fluent-bit:6933
This release includes the same fixes and features that we are working on getting accepted upstream as AWS for Fluent Bit 2.31.6.
We’ve run the new released image in our ECS load testing framework and here is the result. This testing result provides benchmarks of aws-for-fluent-bit
under different input load. Learn more about the load test.
plugin | source | 20 MB/s | 25 MB/s | 30 MB/s | |
---|---|---|---|---|---|
kinesis_firehose | stdstream | Log Loss | ✅ | 0%(4695) | ✅ |
Log Duplication | ✅ | 0%(6000) | ✅ | ||
kinesis_firehose | tcp | Log Loss | ✅ | ✅ | 0%(672) |
Log Duplication | ✅ | 0%(4500) | 0%(26896) | ||
kinesis_streams | stdstream | Log Loss | ✅ | ✅ | ✅ |
Log Duplication | 0%(500) | 0%(7490) | 0%(1986) | ||
kinesis_streams | tcp | Log Loss | ✅ | ✅ | ✅ |
Log Duplication | ✅ | ✅ | 0%(25960) | ||
s3 | stdstream | Log Loss | ✅ | ✅ | ✅ |
Log Duplication | ✅ | ✅ | ✅ | ||
s3 | tcp | Log Loss | ✅ | ✅ | ✅ |
Log Duplication | ✅ | ✅ | ✅ |
plugin | source | 1 MB/s | 2 MB/s | 3 MB/s | |
---|---|---|---|---|---|
cloudwatch_logs | stdstream | Log Loss | ✅ | ✅ | ✅ |
Log Duplication | ✅ | ✅ | ✅ | ||
cloudwatch_logs | tcp | Log Loss | ✅ | 8%(102321) | 2%(49719) |
Log Duplication | ✅ | ✅ | ✅ |
Note:
- The green check ✅ in the table means no log loss or no log duplication.
- Number in parentheses means the number of records out of total records. For example, 0%(1064/1.8M) under 30Mb/s throughput means 1064 duplicate records out of 18M input records by which log duplication percentage is 0%.
- Log loss is the percentage of data lost and log duplication is the percentage of duplicate logs received at the destination. Your results may differ because they can be influenced by many factors like different configs and environment settings. Log duplication is exclusively caused by partially succeeded batches that were retried, which means it is random.
AWS for Fluent Bit 2.31.6
2.31.6
This release includes:
- Fluent Bit 1.9.10
- Amazon CloudWatch Logs for Fluent Bit 1.9.2
- Amazon Kinesis Streams for Fluent Bit 1.10.1
- Amazon Kinesis Firehose for Fluent Bit 1.7.1
Compared to 2.23.5
this release adds:
- Bug - Fixed Log Loss can occur when log group creation or retention policy API calls fail cloudwatch:314
This release includes the same fixes and features that we are working on getting accepted upstream as AWS for Fluent Bit 2.31.5.
plugin | source | 20 MB/s | 25 MB/s | 30 MB/s | |
---|---|---|---|---|---|
kinesis_firehose | stdstream | Log Loss | ✅ | ✅ | ✅ |
Log Duplication | ✅ | 0%(6000) | ✅ | ||
kinesis_firehose | tcp | Log Loss | ✅ | ✅ | ✅ |
Log Duplication | 0%(11000) | 0%(5500) | 0%(2500) | ||
kinesis_streams | stdstream | Log Loss | ✅ | ✅ | ✅ |
Log Duplication | ✅ | 0%(3969) | 0%(7601) | ||
kinesis_streams | tcp | Log Loss | ✅ | ✅ | ✅ |
Log Duplication | ✅ | ✅ | 0%(29274) | ||
s3 | stdstream | Log Loss | ✅ | ✅ | ✅ |
Log Duplication | ✅ | ✅ | ✅ | ||
s3 | tcp | Log Loss | ✅ | ✅ | ✅ |
Log Duplication | ✅ | ✅ | ✅ |
plugin | source | 1 MB/s | 2 MB/s | 3 MB/s | |
---|---|---|---|---|---|
cloudwatch_logs | stdstream | Log Loss | ✅ | ✅ | 16%(299626) |
Log Duplication | ✅ | ✅ | ✅ | ||
cloudwatch_logs | tcp | Log Loss | ✅ | ✅ | 16%(300137) |
Log Duplication | ✅ | ✅ | ✅ |
AWS for Fluent Bit 2.31.5
2.31.5
This release includes:
- Fluent Bit 1.9.10
- Amazon CloudWatch Logs for Fluent Bit 1.9.1
- Amazon Kinesis Streams for Fluent Bit 1.10.1
- Amazon Kinesis Firehose for Fluent Bit 1.7.1
Compared to 2.23.4
this release adds:
- Feature - In the init image, extract Availability Zone from ECS Task Metadata and inject into Fluent Bit container as
AWS_AVAILABILITY_ZONE
environment variable aws-for-fluent-bit:539
Compared to 2.31.4
, this release removes the following fixes that may introduce instabilities to the S3 plugin:
- Enhancement - Transition S3 to fully synchronous file uploads to improve plugin stability fluent-bit:6573
Same as 2.31.4
, this release removes the following fixes that may introduce instabilities to the S3 plugin:
- Bug - Support Retry_Limit option in S3 plugin to set retries fluent-bit:6475
- Bug - Format S3 filename with timestamp from the first log in uploaded file, rather than the time the first log was buffered by the s3 output aws-for-fluent-bit:459
Same as 2.31.4
, this release includes the following fixes and features that we are working on getting accepted upstream:
- Bug - Resolve cloudwatch_logs duplicate tag match SIGSEGV issue introduced in 2.29.0 aws-for-fluent-bit:542
- Feature - Add
kinesis_firehose
andkinesis_streams
support fortime_key_format
milliseconds with%3N
option, and nanoseconds9N
and%L
options fluent-bit:2831 - Feature - Support OpenSearch Serverless data ingestion via OpenSearch plugin fluent-bit:6448
- Bug - Mitigate Datadog output plugin issue by reverting recent PR aws-for-fluent-bit:491
- Bug - Resolve S3 logic to display
log_key
missing warning message if the configuredlog_key
field is missing from log payload fluent-bit:6557 - Bug - ECS Metadata filter gracefuly handle task metadata query errors and cache metadata processing state to improve performance aws-for-fluent-bit:505
- Bug - Resolve a rare Datadog segfault that occurs when remapping tags aws-for-fluent-bit:491
- Bug - Resolve
net.keepalive
SIGSEGV issue and priority event loop corruption bug aws-for-fluent-bit:542
We’ve run the new released image in our ECS load testing framework and here is the result. This testing result provides benchmarks of aws-for-fluent-bit
under different input load. Learn more about the load test.
plugin | source | 20 MB/s | 25 MB/s | 30 MB/s | |
---|---|---|---|---|---|
kinesis_firehose | stdstream | Log Loss | ✅ | ✅ | ✅ |
Log Duplication | 0%(31089) | 0%(6000) | ✅ | ||
kinesis_firehose | tcp | Log Loss | ✅ | ✅ | ✅ |
Log Duplication | ✅ | ✅ | ✅ | ||
kinesis_streams | stdstream | Log Loss | ✅ | ✅ | 0%(43769) |
Log Duplication | 0%(13562) | 0%(45658) | 0%(108603) | ||
kinesis_streams | tcp | Log Loss | ✅ | ✅ | ✅ |
Log Duplication | 0%(1500) | ✅ | 0%(27809) | ||
s3 | stdstream | Log Loss | ✅ | ✅ | ✅ |
Log Duplication | ✅ | ✅ | ✅ | ||
s3 | tcp | Log Loss | ✅ | ✅ | ✅ |
Log Duplication | ✅ | ✅ | ✅ |
plugin | source | 1 MB/s | 2 MB/s | 3 MB/s | |
---|---|---|---|---|---|
cloudwatch_logs | stdstream | Log Loss | ✅ | ✅ | ✅ |
Log Duplication | ✅ | ✅ | ✅ | ||
cloudwatch_logs | tcp | Log Loss | ✅ | ✅ | ✅ |
Log Duplication | ✅ | ✅ | ✅ |
AWS for Fluent Bit 2.31.4
2.31.4
This release includes:
- Fluent Bit 1.9.10
- Amazon CloudWatch Logs for Fluent Bit 1.9.1
- Amazon Kinesis Streams for Fluent Bit 1.10.1
- Amazon Kinesis Firehose for Fluent Bit 1.7.1
Compared to 2.31.3
, this release removes the following fixes that may introduce instabilities to the S3 plugin:
- Bug - Support Retry_Limit option in S3 plugin to set retries fluent-bit:6475
- Bug - Format S3 filename with timestamp from the first log in uploaded file, rather than the time the first log was buffered by the s3 output aws-for-fluent-bit:459
Same as 2.31.3
, this release includes the following fixes and features that we are working on getting accepted upstream:
- Bug - Resolve cloudwatch_logs duplicate tag match SIGSEGV issue introduced in 2.29.0 aws-for-fluent-bit:542
- Feature - Add
kinesis_firehose
andkinesis_streams
support fortime_key_format
milliseconds with%3N
option, and nanoseconds9N
and%L
options fluent-bit:2831 - Feature - Support OpenSearch Serverless data ingestion via OpenSearch plugin fluent-bit:6448
- Enhancement - Transition S3 to fully synchronous file uploads to improve plugin stability fluent-bit:6573
- Bug - Mitigate Datadog output plugin issue by reverting recent PR aws-for-fluent-bit:491
- Bug - Resolve S3 logic to display
log_key
missing warning message if the configuredlog_key
field is missing from log payload fluent-bit:6557 - Bug - ECS Metadata filter gracefuly handle task metadata query errors and cache metadata processing state to improve performance aws-for-fluent-bit:505
- Bug - Resolve a rare Datadog segfault that occurs when remapping tags aws-for-fluent-bit:491
- Bug - Resolve
net.keepalive
SIGSEGV issue and priority event loop corruption bug aws-for-fluent-bit:542
We’ve run the new released image in our ECS load testing framework and here is the result. This testing result provides benchmarks of aws-for-fluent-bit
under different input load. Learn more about the load test.
plugin | source | 20 MB/s | 25 MB/s | 30 MB/s | |
---|---|---|---|---|---|
kinesis_firehose | stdstream | Log Loss | ✅ | ✅ | ✅ |
Log Duplication | ✅ | ✅ | ✅ | ||
kinesis_firehose | tcp | Log Loss | ✅ | ✅ | ✅ |
Log Duplication | ✅ | ✅ | ✅ | ||
kinesis_streams | stdstream | Log Loss | ✅ | ✅ | ✅ |
Log Duplication | ✅ | 0%(18360) | 0%(32342) | ||
kinesis_streams | tcp | Log Loss | ✅ | ✅ | 0%(24) |
Log Duplication | 0%(500) | ✅ | 0%(26068) | ||
s3 | stdstream | Log Loss | ✅ | 7%(1085784) | 27%(4931846) |
Log Duplication | ✅ | ✅ | ✅ | ||
s3 | tcp | Log Loss | ✅ | ✅ | 14%(2636934) |
Log Duplication | ✅ | ✅ | ✅ |
plugin | source | 1 MB/s | 2 MB/s | 3 MB/s | |
---|---|---|---|---|---|
cloudwatch_logs | stdstream | Log Loss | ✅ | ✅ | ✅ |
Log Duplication | ✅ | ✅ | ✅ | ||
cloudwatch_logs | tcp | Log Loss | ✅ | ✅ | ✅ |
Log Duplication | ✅ | ✅ | ✅ |
AWS for Fluent Bit 2.31.3
2.31.3
This release includes:
- Fluent Bit 1.9.10
- Amazon CloudWatch Logs for Fluent Bit 1.9.1
- Amazon Kinesis Streams for Fluent Bit 1.10.1
- Amazon Kinesis Firehose for Fluent Bit 1.7.1
Compared to 2.31.2
, this release adds the following fix that we are working on getting accepted upstream:
- Bug - Resolve
net.keepalive
SIGSEGV issue and priority event loop corruption bug aws-for-fluent-bit:542
Same as 2.31.2
, this release includes the following fixes and features that we are working on getting accepted upstream:
- Bug - Resolve cloudwatch_logs duplicate tag match SIGSEGV issue introduced in 2.29.0 aws-for-fluent-bit:542
- Feature - Add
kinesis_firehose
andkinesis_streams
support fortime_key_format
milliseconds with%3N
option, and nanoseconds9N
and%L
options fluent-bit:2831 - Feature - Support OpenSearch Serverless data ingestion via OpenSearch plugin fluent-bit:6448
- Enhancement - Transition S3 to fully synchronous file uploads to improve plugin stability fluent-bit:6573
- Bug - Mitigate Datadog output plugin issue by reverting recent PR aws-for-fluent-bit:491
- Bug - Format S3 filename with timestamp from the first log in uploaded file, rather than the time the first log was buffered by the s3 output aws-for-fluent-bit:459
- Bug - Resolve S3 logic to display
log_key
missing warning message if the configuredlog_key
field is missing from log payload fluent-bit:6557 - Bug - ECS Metadata filter gracefuly handle task metadata query errors and cache metadata processing state to improve performance aws-for-fluent-bit:505
- Bug - Support Retry_Limit option in S3 plugin to set retries fluent-bit:6475
- Bug - Resolve a rare Datadog segfault that occurs when remapping tags aws-for-fluent-bit:491
We’ve run the new released image in our ECS load testing framework and here is the result. This testing result provides benchmarks of aws-for-fluent-bit
under different input load. Learn more about the load test.
plugin | source | 20 MB/s | 25 MB/s | 30 MB/s | |
---|---|---|---|---|---|
kinesis_firehose | stdstream | Log Loss | ✅ | ✅ | ✅ |
Log Duplication | ✅ | ✅ | 0%(932) | ||
kinesis_firehose | tcp | Log Loss | ✅ | ✅ | ✅ |
Log Duplication | ✅ | ✅ | ✅ | ||
kinesis_streams | stdstream | Log Loss | ✅ | ✅ | ✅ |
Log Duplication | ✅ | 0%(9098) | 0%(26069) | ||
kinesis_streams | tcp | Log Loss | ✅ | ✅ | ✅ |
Log Duplication | ✅ | 0%(1500) | 0%(25373) | ||
s3 | stdstream | Log Loss | ✅ | 12%(1862790) | 23%(4223974) |
Log Duplication | ✅ | ✅ | ✅ | ||
s3 | tcp | Log Loss | ✅ | ✅ | 15%(2783857) |
Log Duplication | ✅ | ✅ | ✅ |
plugin | source | 1 MB/s | 2 MB/s | 3 MB/s | |
---|---|---|---|---|---|
cloudwatch_logs | stdstream | Log Loss | ✅ | ✅ | 11%(207189) |
Log Duplication | ✅ | ✅ | ✅ | ||
cloudwatch_logs | tcp | Log Loss | ✅ | ✅ | ✅ |
Log Duplication | ✅ | ✅ | ✅ |
AWS for Fluent Bit 2.31.2
2.31.2
This release includes:
- Fluent Bit 1.9.10
- Amazon CloudWatch Logs for Fluent Bit 1.9.1
- Amazon Kinesis Streams for Fluent Bit 1.10.1
- Amazon Kinesis Firehose for Fluent Bit 1.7.1
Compared to 2.31.1
, this release adds the following feature that we are working on getting accepted upstream:
- Bug - Resolve cloudwatch_logs duplicate tag match SIGSEGV issue introduced in 2.29.0 aws-for-fluent-bit:542
Same as 2.31.1
, this release includes the following fixes and features that we are working on getting accepted upstream:
- Feature - Add
kinesis_firehose
andkinesis_streams
support fortime_key_format
milliseconds with%3N
option, and nanoseconds9N
and%L
options fluent-bit:2831 - Feature - Support OpenSearch Serverless data ingestion via OpenSearch plugin fluent-bit:6448
- Enhancement - Transition S3 to fully synchronous file uploads to improve plugin stability fluent-bit:6573
- Bug - Mitigate Datadog output plugin issue by reverting recent PR aws-for-fluent-bit:491
- Bug - Format S3 filename with timestamp from the first log in uploaded file, rather than the time the first log was buffered by the s3 output aws-for-fluent-bit:459
- Bug - Resolve S3 logic to display
log_key
missing warning message if the configuredlog_key
field is missing from log payload fluent-bit:6557 - Bug - ECS Metadata filter gracefuly handle task metadata query errors and cache metadata processing state to improve performance aws-for-fluent-bit:505
- Bug - Support Retry_Limit option in S3 plugin to set retries fluent-bit:6475
- Bug - Resolve a rare Datadog segfault that occurs when remapping tags aws-for-fluent-bit:491
We’ve run the new released image in our ECS load testing framework and here is the result. This testing result provides benchmarks of aws-for-fluent-bit
under different input load. Learn more about the load test.
plugin | source | 20 MB/s | 25 MB/s | 30 MB/s | |
---|---|---|---|---|---|
kinesis_firehose | stdstream | Log Loss | ✅ | ✅ | ✅ |
Log Duplication | ✅ | ✅ | ✅ | ||
kinesis_firehose | tcp | Log Loss | ✅ | ✅ | ✅ |
Log Duplication | 0%(500) | ✅ | ✅ | ||
kinesis_streams | stdstream | Log Loss | ✅ | ✅ | 0%(5795) |
Log Duplication | 0%(500) | 0%(32694) | 0%(63482) | ||
kinesis_streams | tcp | Log Loss | ✅ | ✅ | ✅ |
Log Duplication | ✅ | 0%(500) | 0%(6445) | ||
s3 | stdstream | Log Loss | ✅ | 12%(1874773) | 30%(5479384) |
Log Duplication | ✅ | ✅ | ✅ | ||
s3 | tcp | Log Loss | ✅ | ✅ | 15%(2818466) |
Log Duplication | ✅ | ✅ | ✅ |
plugin | source | 1 MB/s | 2 MB/s | 3 MB/s | |
---|---|---|---|---|---|
cloudwatch_logs | stdstream | Log Loss | ✅ | ✅ | ✅ |
Log Duplication | ✅ | ✅ | ✅ | ||
cloudwatch_logs | tcp | Log Loss | ✅ | ✅ | ✅ |
Log Duplication | ✅ | ✅ | ✅ |
Note:
- The green check ✅ in the table means no log loss or no log duplication.
- Number in parentheses means the number of records out of total records. For example, 0%(1064/1.8M) under 30Mb/s throughput means 1064 duplicate records out of 18M input records by which log duplication percentage is 0%.
- Log loss is the percentage of data lost and log duplication is the percentage of duplicate logs received at the destination. Your results may differ because they can be influenced by many factors like different configs and environment settings. Log duplication is exclusively caused by partially succeeded batches that were retried, which means it is random.