Skip to content

Commit

Permalink
Add recommendation to use volume/VolumeMount for filesystem as storag…
Browse files Browse the repository at this point in the history 
…e layer for fluentbit

Signed-off-by: karan k <[email protected]>
  • Loading branch information
@karan56625
karan56625 committed Jul 10, 2023
1 parent 0cb2240 commit 95a000b
Show file tree
Hide file tree
Showing 12 changed files with 261 additions and 233 deletions.
2 changes: 1 addition & 1 deletion apis/fluentbit/v1alpha2/clusterfluentbitconfig_types.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -101,7 +101,7 @@ type Service struct {
LogLevel string `json:"logLevel,omitempty"`
// Optional 'parsers' config file (can be multiple)
ParsersFile string `json:"parsersFile,omitempty"`
// Configure a global environment for the storage layer in Service
// Configure a global environment for the storage layer in Service. It is recommended to configure the volume and volumeMount separately for this storage. The hostPath type should be used for that Volume in Fluentbit daemon set.
Storage *Storage `json:"storage,omitempty"`
}

Expand Down
4 changes: 3 additions & 1 deletion ...ent-operator/charts/fluent-bit-crds/crds/fluentbit.fluent.io_clusterfluentbitconfigs.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -299,7 +299,9 @@ spec:
type: string
storage:
description: Configure a global environment for the storage layer
in Service
in Service. It is recommended to configure the volume and volumeMount
separately for this storage. The hostPath type should be used
for that Volume in Fluentbit daemon set.
properties:
backlogMemLimit:
description: This option configure a hint of maximum value
Expand Down
56 changes: 28 additions & 28 deletions charts/fluent-operator/charts/fluent-bit-crds/crds/fluentbit.fluent.io_clusteroutputs.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2218,116 +2218,116 @@ spec:
s3:
description: S3 defines S3 Output configuration.
properties:
auto_retry_requests:
AutoRetryRequests:
description: Immediately retry failed requests to AWS services
once.
type: boolean
bucket:
Bucket:
description: S3 Bucket name
type: string
canned_acl:
CannedAcl:
description: Predefined Canned ACL Policy for S3 objects.
type: string
compression:
Compression:
description: Compression type for S3 objects.
type: string
content_type:
ContentType:
description: A standard MIME type for the S3 object; this will
be set as the Content-Type HTTP header.
type: string
endpoint:
Endpoint:
description: Custom endpoint for the S3 API.
type: string
external_id:
ExternalId:
description: Specify an external ID for the STS API, can be used
with the role_arn parameter if your role requires an external
ID.
type: string
json_date_format:
JsonDateFormat:
description: 'Specify the format of the date. Supported formats
are double, epoch, iso8601 (eg: 2018-05-30T09:39:52.000681Z)
and java_sql_timestamp (eg: 2018-05-30 09:39:52.000681)'
type: string
json_date_key:
JsonDateKey:
description: Specify the name of the time key in the output record.
To disable the time key just set the value to false.
type: string
log_key:
LogKey:
description: By default, the whole log record will be sent to
S3. If you specify a key name with this option, then only the
value of that key will be sent to S3.
type: string
preserve_data_ordering:
PreserveDataOrdering:
description: Normally, when an upload request fails, there is
a high chance for the last received chunk to be swapped with
a later chunk, resulting in data shuffling. This feature prevents
this shuffling by using a queue logic for uploads.
type: boolean
region:
Region:
description: The AWS region of your S3 bucket
type: string
retry_limit:
RetryLimit:
description: Integer value to set the maximum number of retries
allowed.
format: int32
type: integer
role_arn:
RoleArn:
description: ARN of an IAM role to assume
type: string
s3_key_format:
S3KeyFormat:
description: Format string for keys in S3.
type: string
s3_key_format_tag_delimiters:
S3KeyFormatTagDelimiters:
description: A series of characters which will be used to split
the tag into 'parts' for use with the s3_key_format option.
type: string
send_content_md5:
SendContentMd5:
description: Send the Content-MD5 header with PutObject and UploadPart
requests, as is required when Object Lock is enabled.
type: boolean
static_file_path:
StaticFilePath:
description: Disables behavior where UUID string is automatically
appended to end of S3 key name when $UUID is not provided in
s3_key_format. $UUID, time formatters, $TAG, and other dynamic
key formatters all work as expected while this feature is set
to true.
type: boolean
storage_class:
StorageClass:
description: Specify the storage class for S3 objects. If this
option is not specified, objects will be stored with the default
'STANDARD' storage class.
type: string
store_dir:
StoreDir:
description: Directory to locally buffer data before sending.
type: string
store_dir_limit_size:
StoreDirLimitSize:
description: The size of the limitation for disk usage in S3.
type: string
sts_endpoint:
StsEndpoint:
description: Custom endpoint for the STS API.
type: string
total_file_size:
TotalFileSize:
description: Specifies the size of files in S3. Minimum size is
1M. With use_put_object On the maximum size is 1G. With multipart
upload mode, the maximum size is 50G.
type: string
upload_chunk_size:
UploadChunkSize:
description: 'The size of each ''part'' for multipart uploads.
Max: 50M'
type: string
upload_timeout:
UploadTimeout:
description: Whenever this amount of time has elapsed, Fluent
Bit will complete an upload and create a new file in S3. For
example, set this value to 60m and you will get a new file every
hour.
type: string
use_put_object:
UsePutObject:
description: Use the S3 PutObject API, instead of the multipart
upload API.
type: boolean
required:
- bucket
- region
- Bucket
- Region
type: object
splunk:
description: Splunk defines Splunk Output Configuration
Expand Down
56 changes: 28 additions & 28 deletions charts/fluent-operator/charts/fluent-bit-crds/crds/fluentbit.fluent.io_outputs.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2218,116 +2218,116 @@ spec:
s3:
description: S3 defines S3 Output configuration.
properties:
auto_retry_requests:
AutoRetryRequests:
description: Immediately retry failed requests to AWS services
once.
type: boolean
bucket:
Bucket:
description: S3 Bucket name
type: string
canned_acl:
CannedAcl:
description: Predefined Canned ACL Policy for S3 objects.
type: string
compression:
Compression:
description: Compression type for S3 objects.
type: string
content_type:
ContentType:
description: A standard MIME type for the S3 object; this will
be set as the Content-Type HTTP header.
type: string
endpoint:
Endpoint:
description: Custom endpoint for the S3 API.
type: string
external_id:
ExternalId:
description: Specify an external ID for the STS API, can be used
with the role_arn parameter if your role requires an external
ID.
type: string
json_date_format:
JsonDateFormat:
description: 'Specify the format of the date. Supported formats
are double, epoch, iso8601 (eg: 2018-05-30T09:39:52.000681Z)
and java_sql_timestamp (eg: 2018-05-30 09:39:52.000681)'
type: string
json_date_key:
JsonDateKey:
description: Specify the name of the time key in the output record.
To disable the time key just set the value to false.
type: string
log_key:
LogKey:
description: By default, the whole log record will be sent to
S3. If you specify a key name with this option, then only the
value of that key will be sent to S3.
type: string
preserve_data_ordering:
PreserveDataOrdering:
description: Normally, when an upload request fails, there is
a high chance for the last received chunk to be swapped with
a later chunk, resulting in data shuffling. This feature prevents
this shuffling by using a queue logic for uploads.
type: boolean
region:
Region:
description: The AWS region of your S3 bucket
type: string
retry_limit:
RetryLimit:
description: Integer value to set the maximum number of retries
allowed.
format: int32
type: integer
role_arn:
RoleArn:
description: ARN of an IAM role to assume
type: string
s3_key_format:
S3KeyFormat:
description: Format string for keys in S3.
type: string
s3_key_format_tag_delimiters:
S3KeyFormatTagDelimiters:
description: A series of characters which will be used to split
the tag into 'parts' for use with the s3_key_format option.
type: string
send_content_md5:
SendContentMd5:
description: Send the Content-MD5 header with PutObject and UploadPart
requests, as is required when Object Lock is enabled.
type: boolean
static_file_path:
StaticFilePath:
description: Disables behavior where UUID string is automatically
appended to end of S3 key name when $UUID is not provided in
s3_key_format. $UUID, time formatters, $TAG, and other dynamic
key formatters all work as expected while this feature is set
to true.
type: boolean
storage_class:
StorageClass:
description: Specify the storage class for S3 objects. If this
option is not specified, objects will be stored with the default
'STANDARD' storage class.
type: string
store_dir:
StoreDir:
description: Directory to locally buffer data before sending.
type: string
store_dir_limit_size:
StoreDirLimitSize:
description: The size of the limitation for disk usage in S3.
type: string
sts_endpoint:
StsEndpoint:
description: Custom endpoint for the STS API.
type: string
total_file_size:
TotalFileSize:
description: Specifies the size of files in S3. Minimum size is
1M. With use_put_object On the maximum size is 1G. With multipart
upload mode, the maximum size is 50G.
type: string
upload_chunk_size:
UploadChunkSize:
description: 'The size of each ''part'' for multipart uploads.
Max: 50M'
type: string
upload_timeout:
UploadTimeout:
description: Whenever this amount of time has elapsed, Fluent
Bit will complete an upload and create a new file in S3. For
example, set this value to 60m and you will get a new file every
hour.
type: string
use_put_object:
UsePutObject:
description: Use the S3 PutObject API, instead of the multipart
upload API.
type: boolean
required:
- bucket
- region
- Bucket
- Region
type: object
splunk:
description: Splunk defines Splunk Output Configuration
Expand Down
2 changes: 1 addition & 1 deletion charts/fluent-operator/charts/fluentd-crds/crds/fluentd.fluent.io_fluentds.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -1310,7 +1310,7 @@ spec:
x-kubernetes-map-type: atomic
defaultOutputSelector:
description: Select cluster output plugins used to send all logs that
did not match a route to the matching outputs
did not match any route to the matching outputs
properties:
matchExpressions:
description: matchExpressions is a list of label selector requirements.
Expand Down
20 changes: 20 additions & 0 deletions charts/fluent-operator/values.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -134,6 +134,10 @@ fluentbit:
# - name: hostSys
# hostPath:
# path: /sys/
# Uncomment the code if you intend to create the volume for buffer storage in case the storage type "filesystem" is being used in the configuration of the fluentbit service.
# - name: hostBuffer
# hostPath:
# path: /tmp/
# additionalVolumesMounts:
# - mountPath: /host/sys
# mountPropagation: HostToContainer
Expand All @@ -143,6 +147,11 @@ fluentbit:
# mountPropagation: HostToContainer
# name: hostProc
# readOnly: true
# Uncomment the code if you intend to mount the volume for buffer storage in case the storage type "filesystem" is being used in the configuration of the fluentbit service.
# - mountPath: /host/buffer
# mountPropagation: HostToContainer
# name: hostBuffer


namespaceFluentBitCfgSelector: {}

Expand All @@ -160,6 +169,7 @@ fluentbit:
path: "/var/log/containers/*.log"
skipLongLines: true
readFromHead: false
# Use storageType as "filesystem" if you want to use filesystem as the buffering mechanism for tail input.
storageType: memory
pauseOnChunksOverlimit: "off"
systemd:
Expand All @@ -170,6 +180,7 @@ fluentbit:
path: "/var/log/journal"
includeKubelet: true
stripUnderscores: "off"
# Use storageType as "filesystem" if you want to use filesystem as the buffering mechanism for systemd input.
storageType: memory
pauseOnChunksOverlimit: "off"
nodeExporterMetrics: {}
Expand Down Expand Up @@ -219,6 +230,15 @@ fluentbit:
enable: false
service:
storage: {}
# Remove the above storage section and uncomment below section if you want to configure file-system as storage for buffer
# storage:
# path: "/host/buffer/"
# backlogMemLimit: "50MB"
# checksum: "off"
# deleteIrrecoverableChunks: "on"
# maxChunksUp: 128
# metrics: "on"
# sync: normal

# Configure the default filters in FluentBit.
# The `filter` will filter and parse the collected log information and output the logs into a uniform format. You can choose whether to turn this on or not.
Expand Down
4 changes: 3 additions & 1 deletion config/crd/bases/fluentbit.fluent.io_clusterfluentbitconfigs.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -299,7 +299,9 @@ spec:
type: string
storage:
description: Configure a global environment for the storage layer
in Service
in Service. It is recommended to configure the volume and volumeMount
separately for this storage. The hostPath type should be used
for that Volume in Fluentbit daemon set.
properties:
backlogMemLimit:
description: This option configure a hint of maximum value
Expand Down
Loading

0 comments on commit 95a000b

Please sign in to comment.