Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for OpenSearch Serverless collections to the opensearch s… #3288

Merged
merged 1 commit into from
Aug 31, 2023

Conversation

graytaylor0
Copy link
Member

@graytaylor0 graytaylor0 commented Aug 29, 2023

…ource

Description

This change adds a serverless flag in the opensearch source to support Amazon OpenSearch Serverless collections

aws:
  serverless: true

Serverless collections do not support point in time APIs, but I tested with both search_after and scroll and they worked as intended (the scroll deletion still fails from the client but is likely related to this issue (opensearch-project/opensearch-java#521)

Issues Resolved

Related to #1985

Check List

  • New functionality includes testing.
  • New functionality has a documentation issue. Please link to it in this PR.
    • New functionality has javadoc added
  • Commits are signed with a real name per the DCO

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@@ -31,7 +32,6 @@ public class SearchAccessorStrategy {
static final String OPENSEARCH_DISTRIBUTION = "opensearch";
static final String ELASTICSEARCH_DISTRIBUTION = "elasticsearch";
static final String ELASTICSEARCH_OSS_BUILD_FLAVOR = "oss";
static final String OPENDISTRO_DISTRIUBTION = "opendistro";
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a particular reason to remove this distribution?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I removed it because it doesn't seem to exist. I think the distribution will be just like "elasticsearch"

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems that ES 6.x and 7.x could be opendistro. But, ES versions beyond 7.10.2 would be elasticsearch only.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think elasticsearch or opendistro versions even publish their distribution. the root endpoint for elasticsearch does not return a distribution, and I think this was added for opensearch to differentiate it. I will download an opendistro and run it if I can to see what gets returned

Copy link
Member Author

@graytaylor0 graytaylor0 Aug 30, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed that opendistro does not have a distribution returned, which is same behavior as elasticsearch. We are just hard-coding the distribution to be elasticsearch when no distribution is returned in the info() request.

curl -XGET https://localhost:9200 -u 'admin:admin' --insecure' --insecure
{
  "name" : "22654e359636",
  "cluster_name" : "docker-cluster",
  "cluster_uuid" : "rGtPyarcQ5-tJHR6TsSxJw",
  "version" : {
    "number" : "7.10.2",
    "build_flavor" : "oss",
    "build_type" : "tar",
    "build_hash" : "747e1cc71def077253878a59143c1f785afa92b9",
    "build_date" : "2021-01-13T00:42:12.435326Z",
    "build_snapshot" : false,
    "lucene_version" : "8.7.0",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

@graytaylor0 graytaylor0 merged commit efe5834 into opensearch-project:main Aug 31, 2023
24 checks passed
asifsmohammed pushed a commit to asifsmohammed/data-prepper that referenced this pull request Sep 27, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants