New Source: AWS Cloudwatch

[marcosmarxm]

Tell us about the new connector you’d like to have

• Which source and which destination?
• Do you need a specific version of the underlying data source e.g: you specifically need support for an older version of the API or DB?

Describe the context around this new connector

• Which team in your company wants this integration, what for? This helps us understand the use case.
• How often do you want to run syncs?
• If this is an API source connector, which entities/endpoints do you need supported?

Describe the alternative you are considering or using

What are you considering doing if you don’t have this integration through Airbyte?

Are you willing to submit a PR?

Remove this with your answer :-)

[eamontaaffe]

I'm mostly interested in using this to pull a CloudWatch log group into our warehouse (snowflake). However since the CloudWatch service also includes metrics I think this would be nice to have (although I don't have an immediate use case).

[karinakuz]

  Integration Vetting

Webhook-based? (no/partially/yes)
if partially, mention which endpoints are webhook-based
no

Available authentication modes (API key/Oauth/other)
AWS access key, which consists of an access key ID and secret access key
https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/auth-and-access-control-cw.htm

Connecting to the data source
what kind of API is it? (SOAP/GraphQL/REST/other)
CloudWatch Logs Query API (Query requests used with CloudWatch Logs are HTTP or HTTPS requests that use the HTTP verb GET or POST and a Query parameter named Action or Operation)
https://docs.aws.amazon.com/AmazonCloudWatchLogs/latest/APIReference/making-api-requests.html

is there an SDK available? Can we connect with HTTP requests directly?
we can use one of the AWS SDKs to access CloudWatch Logs using an API tailored to our programming language
https://aws.amazon.com/tools/#sdk

If this is a database/datalake/pubsub source, does it have a JDBC driver or Java SDK?
database source, Java SDK

Creating an account
Select one of the below options
self-service free account available?_ If yes then create it using [email protected] (you should be getting emails sent to this email address)
yes

How to populate the account with data?
Can it be done entirely through the API?
no

Will it require some work through the UI? If so, what is required?
activities for metrics and logs

Will it require performing some paid action like running an Ad campaign? If so, please note the action needed, estimated cost, and estimated effort in hours. If a task will take more than 4 hours, please break it down into smaller tasks each of which take no more than 4 hours
no

Do you need any actions from Airbyte to populate the account with data? If so, please tag @ sherifnada for support
no

Available streams for sync
If this is an API connector:

1. Provide a link  to the reference docs

Logs Events https://docs.aws.amazon.com/AmazonCloudWatchLogs/latest/APIReference/API_GetLogEvents.html

Metrics Data https://docs.aws.amazon.com/AmazonCloudWatch/latest/APIReference/API_GetMetricData.html

2. Please mention if any of the endpoints are asynchronous (meaning we have to make a request, wait for the server to compute a response, then make another request to obtain the result)
   no

if this is a source where the schema is dynamic/not fixed e.g: database, mention the data model of the connector i.e: relational, document-based, graph-based, etc
yes

Do any streams support incremental sync?
not necessary to mention all of them, just saying “none”, “some”, or “all” is sufficient. If only some streams, please mention which ones support incremental sync
no

[karinakuz]

Preparation
Creating the account
[ ] create an AWS IAM account , get access key ID and secret access key automatically, add notes to LastPass Description with name "CloudWatch Amazon AWS (45m)

Populating the account with data
[ ] Do activities on the account. Create API queries in Postman to get logs and metrics (22h).

API with required data for the main streams https://docs.google.com/document/d/1vB78slBjZsEjzzamH0Qyrj7uGAfIPZZ2NUXNFoBO1B0/edit?usp=sharing

[karinakuz]

Describing the output schema (15m) https://drive.google.com/drive/folders/19DubbHDHL9jno0ATlpillMY0tjoAY87-?usp=sharing