Logstash Elasticsearch Filter for OpenSearch

[vnil1994]

Is your feature request related to a problem? Please describe.
We are running OpenSearch 1.2.2 and are looking to use the logstash filter plugin "elasticsearch" for lookup and enrichment in other indices, before we send the logs to Opensearch and it's corresponding index. However, it seems that the logstash filter plugin "Elasticsearch" does not support OpenSearch. It complains with the following error message:

[2021-12-17T15:17:16,883][ERROR][logstash.javapipeline ][main] Pipeline error {:pipeline_id=>“main”, :exception=>#<LogStash::ConfigurationError: Could not connect to a compatible version of Elasticsearch>

I originally created a forum post here for this problem and was asked to create a feature request for it.

Describe the solution you'd like
Attempt to let Logstash connect to an OpenSearch instance without failing the license check, or perhaps build a logstash plugin with features similar to the existing Elasticsearch filter plugin that is able to connect to Elasticsearch OSS and OpenSearch instances.

Describe alternatives you've considered
We have tried running this command in the cluster:

PUT _cluster/settings

{
  "persistent": {
    "compatibility": {
      "override_main_response_version": true
    }
  }
}

but the license check still fails.

Additional context
The documentation for elasticsearch-filter-plugin: https://www.elastic.co/guide/en/logstash/current/plugins-filters-elasticsearch.html

[dblock]

I moved this to opensearch-clients.

[steve-offutt]

I would also like to see an OpenSearch filter plugin implemented in Logstash. Is this feature being worked on yet?

[brijos]

We don't have anything in the works at the moment. Is anyone interested in picking up the work?

[jgough]

I've created a fork of the plugin that can talk to OpenSearch at https://github.com/jgough/logstash-filter-opensearch
If someone wants to pick this up and pull it under the umbrella of the OpenSearch project I would be more than grateful.

You can download the gem from the releases there and it should hopefully work. Install it using logstash-plugin install logstash-filter-opensearch-0.1.0.gem

Then you can use it as you would expect:

filter {
    opensearch {
        hosts => ["http://opensearch:9200"]
        index => "my-index"
        sort => "id:asc"
        query => "id: %{id}"
        fields => {
            "data" => "[my_index_field]"
        }
    }
}

[jgough]

Should now be able to install my plugin with the command logstash-plugin install logstash-filter-opensearch

[dblock]

@jgough Thanks! Would you be so kind to enable issues in your fork, and open (or I can open) an issue in your repo similar to opensearch-project/opensearch-plugin-template-java#4 about moving that repo into the org?

Also we don't have to - if you're looking for more work to do, I would pickup a bunch of things from https://github.com/opensearch-project/.github to help grow a community around it (CONTRIBUTING, README, etc.).

[jgough]

@dblock I've enabled issues so please go ahead. Let me know what I can do to help with this

[dblock]

jgough/logstash-filter-opensearch#1

[pushanDev]

Hi,

This new opensearch filter plugin for logstash is still not working although installed successfully using bin/logstash-plugin install logstash-filter-opensearch.

Can you please help?

Created a new topic in opensearch community for the same- https://forum.opensearch.org/t/logstash-opensearch-filter-plugin/17756

Below are some relevant config and logs-

filter {
	opensearch {
		hosts => ["https://localhost:9200"]
		#ssl => true
		#ssl_certificate_verification => false
		index => "students"
		user => "admin"
		password => "admin"
		query_template => "es-query/student_id_query.json"
		docinfo_fields => {
			"_id" => "student_id"
		}
	}
}

Here the host is secured- https://localhost:9200, ssl config is disabled (as host directly contains https) and ssl_certificate_verification is disabled-

Error log:

[ERROR][logstash.javapipeline ][main] Pipeline error {:pipeline_id=>"main", :exception=>#<Faraday::SSLError wrapped=#<OpenSSL::SSL::SSLError: certificate verify failed>>

Now, with the same config, if I enable ssl_certificate_verification as ssl_certificate_verification => false, i.e., the below,

filter {
	opensearch {
		hosts => ["https://localhost:9200"]
		#ssl => true
		ssl_certificate_verification => false
		index => "students"
		user => "admin"
		password => "admin"
		query_template => "es-query/student_id_query.json"
		docinfo_fields => {
			"_id" => "student_id"
		}
	}
}

then the below error is occurring-

Error log:

[ERROR][logstash.filters.opensearch] Unknown setting 'ssl_certificate_verification' for opensearch

FYI, I have used ssl => true and host => ["localhost:9200"] also, but got the same error.

Please note that the same settings works fine for ElasticSearch cluster v8.x-

https://www.elastic.co/guide/en/logstash/current/plugins-filters-elasticsearch.html

Any suggestion will help me a lot.

Thanks and regards,
Pushan.

[dblock]

@pushanDev I am afraid of not being of much help, if https://github.com/jgough/logstash-filter-opensearch is not working then you should open bugs there; looks like 7 months since last time a commit was made in that repo, so it will take someone motivated to pick it up

[pushanDev]

@pushanDev I am afraid of not being of much help, if https://github.com/jgough/logstash-filter-opensearch is not working then you should open bugs there; looks like 7 months since last time a commit was made in that repo, so it will take someone motivated to pick it up

Okay @dblock, thank you, I shall do that.

[jgough]

@pushanDev Hi, I don't have much time myself (nor much ruby experience) to maintain the plugin and was hoping someone official from the project would step in to take over on this. It seems strange to me that logstash-output-opensearch is maintained in the Opensearch Project but logstash-filter-opensearch is not.

@dblock Still hoping someone official can pick this up and give it the attention it needs jgough/logstash-filter-opensearch#1

[pushanDev]

@pushanDev Hi, I don't have much time myself (nor much ruby experience) to maintain the plugin and was hoping someone official from the project would step in to take over on this. It seems strange to me that logstash-output-opensearch is maintained in the Opensearch Project but logstash-filter-opensearch is not.

@dblock Still hoping someone official can pick this up and give it the attention it needs jgough/logstash-filter-opensearch#1

Thank you @jgough!

[dblock]

My last attempt at finding someone to pick it up and do work on it hasn't succeeded, so I don't think moving it into the org makes much sense. But if it does get maintainers outside we'll happily move it in.

[bugblasterX]

@pushanDev Hi, I don't have much time myself (nor much ruby experience) to maintain the plugin and was hoping someone official from the project would step in to take over on this. It seems strange to me that logstash-output-opensearch is maintained in the Opensearch Project but logstash-filter-opensearch is not.
@dblock Still hoping someone official can pick this up and give it the attention it needs jgough/logstash-filter-opensearch#1

Thank you @jgough!

Hey, have you managed to solve the issue?