You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Hello there!
I use fluent-plugin-s3 and according my configuration it should send logs from few files to MinIO once per day.
Hers is the configuration:
<match projectname>
@type s3
aws_key_id "#{ENV['MINIO_ROOT_USER']}"
aws_sec_key "#{ENV['MINIO_ROOT_PASSWORD']}"
s3_bucket tenants
format json
force_path_style true
s3_endpoint "http://#{ENV['MINIO_HOST']}:#{ENV['MINIO_PORT']}/"
path "#{ENV['TENANT_ID']}/logs/projectname-" # This prefix is added to each file
time_slice_format %Y%m%d%H%M # This timestamp is added to each file name
#s3_object_key_format %{path}%{time_slice}.%{file_extension} # Should be commented because target log file will be overwritten few times and logs wil be lost.
<buffer tag,time>
@type file
path /fluentd/logs/
timekey 1440m
timekey_wait 10m
flush_mode lazy
timekey_use_utc true
chunk_limit_size 256m
</buffer>
</match>
I have some misunderstanding why one day the logs are grouped into one file and it is created on MinIO, and on another day several log files are created on MinIO. Could someone explain it, please?
Example,
projectname-202401240532_3.gz
projectname-202401240532_2.gz
projectname-202401240532_1.gz
projectname-202401240532_0.gz
I found the following in the internet.
Uploading more than one file to s3 when timekey condition has been met:
Buffering: Fluentd buffers logs before writing them to the file. Sometimes, when the buffer is not completely full, Fluentd might decide to upload the existing logs to S3 and start a new file for the remaining logs. This can result in multiple files being uploaded.
Flushing: Fluentd has a flushing mechanism that triggers the upload of logs based on certain conditions. If the flushing condition is met before the timekey condition, Fluentd might upload the logs even if they are not part of the complete time range. This can also lead to multiple files being uploaded.
Delayed Write: Fluentd might experience a delay in writing logs to file due to various factors, such as a high load on the system or network issues. In such cases, Fluentd might start a new file even if the timekey condition has been met, resulting in additional files being uploaded.
Is amount of created files on MinIO (per day) related to amount of source log files?
Only for these reasons can several files be created on the MinIO?
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
-
Hello there!
I use fluent-plugin-s3 and according my configuration it should send logs from few files to MinIO once per day.
Hers is the configuration:
I have some misunderstanding why one day the logs are grouped into one file and it is created on MinIO, and on another day several log files are created on MinIO. Could someone explain it, please?
Example,
projectname-202401240532_3.gz
projectname-202401240532_2.gz
projectname-202401240532_1.gz
projectname-202401240532_0.gz
I found the following in the internet.
Uploading more than one file to s3 when timekey condition has been met:
Buffering: Fluentd buffers logs before writing them to the file. Sometimes, when the buffer is not completely full, Fluentd might decide to upload the existing logs to S3 and start a new file for the remaining logs. This can result in multiple files being uploaded.
Flushing: Fluentd has a flushing mechanism that triggers the upload of logs based on certain conditions. If the flushing condition is met before the timekey condition, Fluentd might upload the logs even if they are not part of the complete time range. This can also lead to multiple files being uploaded.
Delayed Write: Fluentd might experience a delay in writing logs to file due to various factors, such as a high load on the system or network issues. In such cases, Fluentd might start a new file even if the timekey condition has been met, resulting in additional files being uploaded.
Is amount of created files on MinIO (per day) related to amount of source log files?
Only for these reasons can several files be created on the MinIO?
Beta Was this translation helpful? Give feedback.
All reactions