From 34264925e52eea9a200e9fc344b90354ad7c7009 Mon Sep 17 00:00:00 2001 From: Eldar Nash Date: Sat, 23 Sep 2023 11:05:22 +0200 Subject: [PATCH] S3 destination skeleton --- .../destination-s3/_index.md | 203 ++++++++++++++++++ 1 file changed, 203 insertions(+) create mode 100644 content/chapter-destinations/destination-s3/_index.md diff --git a/content/chapter-destinations/destination-s3/_index.md b/content/chapter-destinations/destination-s3/_index.md new file mode 100644 index 00000000..1a37ef2c --- /dev/null +++ b/content/chapter-destinations/destination-s3/_index.md @@ -0,0 +1,203 @@ +--- +title: "s3: Amazon S3" +weight: 2700 +--- + + +The `s3()` destination sends log messages to the [Amazon Simple Storage Service (Amazon S3)](https://aws.amazon.com/s3/) object storage service. You can send log messages over TCP, or encrypted with TLS. + + + + + +## Prerequisites + +- An existing S3 bucket configured for programmatic access, and the related `ACCESS_KEY` and `SECRET_KEY` of a user that can access it. + + + + +To use the `s3()` driver, the `scl.conf` file must be included in your {{% param "product.abbrev" %}} configuration: + +```shell + @include "scl.conf" +``` + +The `s3()` driver is actually a reusable configuration snippet. For details on using or writing such configuration snippets, see {{% xref "/chapter-configuration-file/large-configs/config-blocks/_index.md" %}}. You can find the source of this configuration snippet on [GitHub](https://github.com/syslog-ng/syslog-ng/blob/master/modules/python-modules/syslogng/modules/s3/s3_destination.py). + +## Options + +The following options are specific to the `s3()` destination. + + + +## access-key() + +| | | +| -------- | -------------------------- | +| Type: | string | +| Default: | | + +*Description:* The `ACCESS_KEY` of the service account used to access the S3 bucket. (Together with [`secret-key()`](#secret-key).) + +## bucket() + +| | | +| -------- | -------------------------- | +| Type: | string | +| Default: | | + +*Description:* The name of the S3 bucket, for example, `my-bucket` + +## canned-acl() + +| | | +| -------- | -------------------------- | +| Type: | string | +| Default: | `empty` | + +*Description:* The ACL assigned to the object, if specified, for example, `bucket-owner-read`. The following values are valid: + +`authenticated-read`, `aws-exec-read`, `bucket-owner-full-control`, `bucket-owner-read`, `log-delivery-write`, `private`, `public-read`, `public-read-write` + +If you configure an invalid value, the default is used. + +## chunk-size() + +| | | +| -------- | -------------------------- | +| Type: | string | +| Default: | `5MiB` | + +*Description:* The size of log messages that {{% param "product_name" %}} writes to the S3 object in a batch. + + +## compression() + +| | | +| -------- | -------------------------- | +| Type: | boolean | +| Default: | `no` | + +*Description:* + +## compresslevel() + +| | | +| -------- | -------------------------- | +| Type: | integer | +| Default: | `9` | + +*Description:* + +## flush-grace-period() + +| | | +| -------- | -------------------------- | +| Type: | integer (seconds) | +| Default: | `60` | + +*Description:* After the grace period expires and no new messages are routed to the destination, {{% param "product_name" %}} flushes the contents of the buffer to the S3 object even if the volume of the messages in the buffer is lower than [`chunk-size()`](#chunk-size). + +## max-object-size() + +| | | +| -------- | -------------------------- | +| Type: | string | +| Default: | `5120GiB` | + +*Description:* The maximal size of the S3 object. + + +## max-pending-uploads() + +| | | +| -------- | -------------------------- | +| Type: | integer | +| Default: | `32` | + +*Description:* + +## object-key() + +| | | +| -------- | -------------------------- | +| Type: | string | +| Default: | | + +*Description:* The [object key](https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-keys.html) (or key name), which uniquely identifies the object in an Amazon S3 bucket. + +## object-key-timestamp() + +| | | +| -------- | -------------------------- | +| Type: | string | +| Default: | | + +*Description:* The timestamp added to the object. + + +## region() + +| | | +| -------- | -------------------------- | +| Type: | string | +| Default: | | + +*Description:* The [regional endpoint](https://docs.aws.amazon.com/general/latest/gr/rande.html#regional-endpoints) where the bucket is stored. For example, `us-east-1` + +## secret-key() + +| | | +| -------- | -------------------------- | +| Type: | string | +| Default: | | + +*Description:* The `SECRET_KEY` of the service account used to access the S3 bucket. (Together with [`access-key()`](#access-key).) + +## storage-class() + +| | | +| -------- | -------------------------- | +| Type: | string | +| Default: | `STANDARD` | + +*Description:* The [storage class of the object](https://docs.aws.amazon.com/AmazonS3/latest/userguide/storage-class-intro.html), for example, `REDUCED_REDUNDANCY`. The following values are valid: + +`DEEP_ARCHIVE`, `GLACIER`, `GLACIER_IR`, `INTELLIGENT_TIERING`, `ONEZONE_IA`, `OUTPOSTS`, `REDUCED_REDUNDANCY`, `SNOW`, `STANDARD`, `STANDARD_IA` + +If you configure an invalid value, the default is used. + + + +## upload-threads() + +| | | +| -------- | -------------------------- | +| Type: | integer | +| Default: | `8` | + +*Description:* The number of {{% param "product_name" %}} worker threads that are used to upload data to S3 from this destination. + +## template() + +| | | +| -------- | ----------------------------- | +| Type: | template or template function | +| Default: | `${MESSAGE}\n` | + +*Description:* The message as written to the Amazon S3 object. You can use templates and [template functions]({{< relref "/chapter-manipulating-messages/customizing-message-format/reference-template-functions/_index.md" >}}) to format the message. + + + +## url() + +| | | +| -------- | -------------------------- | +| Type: | string | +| Default: | | + +*Description:* The URL of the S3 bucket, for example, `https://my-bucket.s3.us-west-2.amazonaws.com` \ No newline at end of file