Could not load sniffer class Fluent::Plugin::ElasticsearchSimpleSniffer

[PScharrenberg]

Describe the bug:
When configuring the Fluent::Plugin::ElasticsearchSimpleSniffer sniffer class for the elasticsearch plugin using the sniffer_class_name setting the config checker fails with:

[error]: config error file="/fluentd/etc/fluent.conf" error_class=Fluent::ConfigError error="Could not load sniffer class Fluent::Plugin::ElasticsearchSimpleSniffer: uninitialized constant Fluent::Plugin::ElasticsearchSimpleSniffer\nDid you mean? Fluent::ElasticsearchIndexTemplate"

Expected behaviour:
We expect fluentd to use the configured sniffer class

Steps to reproduce the bug:
Add Fluent::Plugin::ElasticsearchSimpleSniffer sniffer class as value for sniffer_class_name to elasticsearch output in a (cluster)output resource.

Additional context:
We use elasticsearch behind an ssl offloading nginx proxy. After a while of successfull log shipping to elasticsearch, log shipping suddenly stops.

The root cause for that is, that the fluentd-elasticsearch-plugin uses the "default sniffer class" to refresh the elasticsearch node config after 10000 operations. Unfortunately the default sniffer class does not respect the nginx proxy in between and configures a wrong port number (elasticsearch port 9200 instead of proxy port 443). We have to use the Fluent::Plugin::ElasticsearchSimpleSniffer sniffer class instead to just keep our configured settings in place.
We configured this class using the documented setting in the clusteroutput resource.

After configuring this sniffer class we see this error in the config-checker:
[error]: config error file="/fluentd/etc/fluent.conf" error_class=Fluent::ConfigError error="Could not load sniffer class Fluent::Plugin::ElasticsearchSimpleSniffer: uninitialized constant Fluent::Plugin::ElasticsearchSimpleSniffer\nDid you mean? Fluent::ElasticsearchIndexTemplate"

(partial) solution:
As described in the elasticsearch plugin documentation the sniffer class file needs to get passed as a command line parameter to fluentd (fuentd -r <sniffer-class.rb>, e.g. fluentd -r $(fluent-gem contents fluent-plugin-elasticsearch|grep elasticsearch_simple_sniffer.rb))

I could not find a way to add this command line parameter to fluentd when using the logging-operator helm chart.
In the best case I would expect that this parameter would be added as soon as sniffer_class_name is set in the configuration.

Environment details:

• Kubernetes version: v1.24.10
• logging-operator version: 3.17.10
• Install method: helm
• Logs from the misbehaving component:
  [error]: config error file="/fluentd/etc/fluent.conf" error_class=Fluent::ConfigError error="Could not load sniffer class Fluent::Plugin::ElasticsearchSimpleSniffer: uninitialized constant Fluent::Plugin::ElasticsearchSimpleSniffer\nDid you mean? Fluent::ElasticsearchIndexTemplate"
• Resource definition (possibly in YAML format) that caused the issue, without sensitive data:

metadata:
  name: es-output
spec:
  elasticsearch:
    host: elasticsearch-proxy-host.example.com
    port: 443
    scheme: https
    ssl_verify: true
    ssl_version: TLSv1_2
    user: es-user
    password:
      valueFrom:
        secretKeyRef:
          name: es-password
          key: password
    buffer:
      timekey: 1m
      timekey_wait: 30s
      timekey_use_utc: true
    sniffer_class_name: Fluent::Plugin::ElasticsearchSimpleSniffer

/kind bug

[genofire]

If you could have access to your ingress-controller, you could enable routing by SNI by enable-ssl-passthrough, helm values are:

controller:
  extraArgs:
    enable-ssl-passthrough: "true"

[pepov]

@PScharrenberg you can add extra arguments to fluentd with the extraArgs field under fluentd in the logging spec.

[genofire]

Documentation:
https://kube-logging.github.io/docs/configuration/crds/v1beta1/fluentd_types/#fluentdspec-extraargs

This example should work:

apiVersion: logging.banzaicloud.io/v1beta1
kind: Logging
metadata:
  name: "logging-operator-logging"
spec:
  fluentd:
    extraArgs:
      - "-r"
      - "/usr/lib/ruby/gems/2.7.0/gems/fluent-plugin-elasticsearch-5.2.5/lib/fluent/plugin/elasticsearch_simple_sniffer.rb"

PS: Path could change on newer fluentd images.

[PScharrenberg]

Thank you @genofire. That was the missing piece in the puzzle and I got it working by specifying the right versioned path to the sniffer class file.

The remaining question is:
Is it possible make this configuration a bit more dynamic?
Statically adding a filename which depends on a changing version is not a viable path for a production system.

I tried:

• $(fluent-gem contents fluent-plugin-elasticsearch|grep elasticsearch_simple_sniffer.rb) and
• globbing

but it seem that the extraArgs strings get passed as shell escaped strings.

In the best case I would expect that this parameter would be added as soon as sniffer_class_name is set in the configuration but I would also go with any other solution that requires a one-time manual configuration and would be stable during updates.

[pepov]

What I don't understand is: why does the plugin has to load that class runtime, when the class is part of the same plugin? I understand that it would need to load a custom class, but it is not the case here.

Sorry but I'm totally against adding this kind of complexity to the operator until I understand why this mechanism is necessary.

[genofire]

myself i understand, that this should be solved on plugin level (and i do not understand, why this class is delieverd but not loaded):
https://github.com/uken/fluent-plugin-elasticsearch#sniffer-class-name

so i made the suggestion to change the behavier of the ingress (to support SNI routing).

---

Sorry @PScharrenberg i understand your problems, that it is very unconverdable.

[pepov]

I also understand the frustration, but I believe we cannot solve it on this level properly.