AWS OpenSearch (Source Connector)

[poshkaran04]

I’d like to have AWS OpenSearch source connector

• AWS OpenSearch source and OpenSearch destination
• I doubt if there is any specific version that would be needed. However we are currently using opensearch-py 2.0.0 version within Python for same.

Describe the context around this new connector

• Use Case: We are a FinTech startup just like CRED in India and in order to access customer's financial data we need to make sure customers journey of connecting to our product where we fetch their financial data with there consent. Now many a times this establishing connection fails. So in order to make sure all these events of making a connection call to connecting of their finance account are captured we are using OpenSearch. Then we structure this JSON data into a tabular dataset and load it into our Data warehouse for further reporting.
• How often do you want to run syncs? Once a Day (Batch Processing and Incremental)
• If this is an API source connector, which entities/endpoints do you need supported? I am not sure about this.
• If the connector is for a paid service, can we name you as a mutual user when we subscribe for an account? Which company should we name? Currently I am doing POC on my own and then I need to make case for my management. At this point I would not want to name my company.

Describe the alternative you are considering or using

Currently I am using Python program in order to achieve this. I am thinking of using Airbyte because it not only has growing community as compared to it's peers but also has pretty decent list of connector that are useful for traditional as well as modern data architecture. In addition so far in terms of speed and ease of use I am liking it!!

[sajarin]

@poshkaran04, let us know if you're willing to work on this, otherwise I'm sure someone in our community will pick it up soon (if our team doesn't pick it up sooner!)

[poshkaran04]

I would not be able to work on it as I am not sure How it can be done. However I am willing to test it.

[a1tair6]

Can i develop it?

[sajarin]

@a1tair6 please feel free!

[a1tair6]

@sajarin could you review it? #23657