parse_exception: unknown key [data_stream] in the template

[ng-bsy]

(check apply)

• read the contribution guideline
• (optional) already reported 3rd party upstream repository or mailing list if you use k8s addon or helm charts.

Steps to replicate

fluentd.yaml

# ...
 - match:
    $tag: '**'
    $type: opensearch
    hosts: oshost1,oshost2
    user: fluentd
    password: XXXX
    logstash_format: false
    include_timestamp: true
    index_name: fluentd.${tag}
    templates: !fluent/json {'fluentd.json': '/fluentd/etc/os_templates/fluentd.json'}
    template_overwrite: true
    compression_level: default_compression
    with_transporter_log: true
    log_os_400_reason: true

fluentd.json

{
    "index_patterns" : [
        "fluentd.*"
    ],
    "priority": 0,
    "data_stream": {}
}

error message

 #0 Could not communicate to OpenSearch, resetting connection and trying again. [400]
{
    "error": {
        "root_cause": [
            {
                "type": "parse_exception",
                "reason": "unknown key [data_stream] in the template"
            }
        ],
        "type": "parse_exception",
        "reason": "unknown key [data_stream] in the template"
    },
    "status": 400
}

Expected Behavior or What you need to ask

I expect the template be created and used as defined, like e.g. the logstash opensearch plugin

Out of interest: Why is it, that you had to jump through such hoops, creating a separate data stream plugin/type, when the logstash plugin makes it look so easy, providing both functionalities in one with just a little addition to the template?

Using Fluentd and OpenSearch plugin versions

Ubuntu 22.04 LTS
Docker fluentd:v1.16-1
fluentd 1.16.2
fluent-plugin-opensearch 1.1.4
OpenSearch version 2.5
OpenSearch template - provided above

[ng-bsy]

And another one, I could solve by myself - I should have configured the parameter use_legacy_template: false - just like in logstash :-)

Maybe the README / Documentation of the parameters should be made a bit more clear like e.g. documenting the default values