[Q&A]Trying to use opensearch plugin with fluentd using docker facing issue "error="Unknown output plugin 'opensearch'"

[vaibhavahire99]

What is a problem?

Hi, I'm running to run fluentd using docker with opensearch plugin but facing error="Unknown output plugin 'opensearch'
Can anyone please help me resolve this?
Below is my docker and config:
docker-compose.yml :
version: '3'
services:
fluentd:
build: ./fluentd # This will build Fluentd with the necessary plugins installed
container_name: earthid-fluentd
platform: linux/amd64
volumes:
- ./fluentd/fluent.conf:/fluentd/etc/fluent.conf
ports:
- "24224:24224"
- "24224:24224/udp"
networks:
- earthid-net
depends_on:
- opensearch

opensearch:
image: opensearchproject/opensearch:latest
container_name: earthid-opensearch
environment:
- discovery.type=single-node
- "plugins.security.disabled=true"
- OPENSEARCH_INITIAL_ADMIN_PASSWORD=Digital@2022 # Set the admin password here
ports:
- "9200:9200"
- "9600:9600"
networks:
- earthid-net

opensearch-dashboards:
image: opensearchproject/opensearch-dashboards:latest
container_name: earthid-dashboards
environment:
- OPENSEARCH_HOSTS=http://opensearch:9200
ports:
- "5601:5601"
networks:
- earthid-net
depends_on:
- opensearch

prometheus:
image: prom/prometheus:latest
container_name: earthid-prometheus
volumes:
- ./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml
ports:
- "9090:9090"
networks:
- earthid-net

grafana:
image: grafana/grafana:latest
container_name: earthid-grafana
ports:
- "3000:3000"
networks:
- earthid-net
depends_on:
- prometheus

elastalert:
image: bitsensor/elastalert:latest
container_name: earthid-elastalert
platform: linux/amd64
volumes:
- ./elastalert/config.yaml:/opt/elastalert/config.yaml
- ./elastalert/rules:/opt/elastalert/rules
depends_on:
- opensearch
networks:
- earthid-net

networks:
earthid-net:
driver: bridge

/fluentd/Dockerfile :
FROM fluent/fluentd:v1.14-1

USER root

Install dependencies for building Ruby from source using Alpine's apk package manager

RUN apk update &&
apk add --no-cache bash build-base openssl-dev zlib-dev readline-dev yaml-dev libxml2-dev libxslt-dev libffi-dev git curl

Install rbenv and Ruby 3.3.5

RUN git clone https://github.com/rbenv/rbenv.git ~/.rbenv &&
echo 'export PATH="$HOME/.rbenv/bin:$PATH"' >> ~/.bashrc &&
echo 'eval "$(rbenv init -)"' >> ~/.bashrc &&
git clone https://github.com/rbenv/ruby-build.git ~/.rbenv/plugins/ruby-build &&
bash -c "source ~/.bashrc && ~/.rbenv/bin/rbenv install 3.3.5 && ~/.rbenv/bin/rbenv global 3.3.5"

Update RubyGems to the latest version

RUN ~/.rbenv/shims/gem update --system

Install the OpenSearch plugin for Fluentd

RUN ~/.rbenv/shims/gem install fluent-plugin-opensearch --no-document

Copy Fluentd configuration

COPY fluent.conf /fluentd/etc/fluent.conf

Set permissions and run Fluentd

USER fluent

Please let me know if any corrections or additional steps to follow

Describe the configuration of Fluentd

@type forward port 24224 bind 0.0.0.0 @type opensearch host localhost port 9200 logstash_format true

Describe the logs of Fluentd

2024-10-16 13:09:37 +0000 [info]: parsing config file is succeeded path="/fluentd/etc/fluent.conf"
2024-10-16 13:09:38 +0000 [info]: gem 'fluentd' version '1.14.6'
2024-10-16 13:09:38 +0000 [error]: config error file="/fluentd/etc/fluent.conf" error_class=Fluent::NotFoundPluginError error="Unknown output plugin 'opensearch'. Run 'gem search -rd fluent-plugin' to find plugins"

Environment

- Fluentd version:
- TD Agent version:
- Fluent Package version:
- Docker image (tag):
- Operating system:
- Kernel version: