-
Notifications
You must be signed in to change notification settings - Fork 206
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Merge pull request #5922 from confluentinc/18092024
wip
- Loading branch information
Showing
15 changed files
with
165 additions
and
15 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,39 @@ | ||
| # Fully Managed Google Cloud Functions Gen 2 Sink connector | ||
|
|
||
|
|
||
|
|
||
| ## Objective | ||
|
|
||
| Quickly test [Fully Managed Google Cloud Functions Gen 2 Sink](https://docs.confluent.io/cloud/current/connectors/cc-google-cloud-functions-gen2-sink.html) connector. | ||
|
|
||
|
|
||
| * Active Google Cloud Platform (GCP) account with authorization to create resources | ||
|
|
||
| ## Google Cloud Functions Setup | ||
|
|
||
| * Navigate to the [Google Cloud Console](https://console.cloud.google.com/) | ||
|
|
||
| * Go to the [Cloud Functions](https://console.cloud.google.com/functions) tab. | ||
|
|
||
|  | ||
|
|
||
| * Create a new function. Use the default code that is provided. | ||
|
|
||
|  | ||
|
|
||
| Note down the project id, the region (example `us-central1`), and the function name (example `function-1`) as they will be used later. | ||
|
|
||
| N.B: make sure service account has `Cloud Run Invoker` role. | ||
|
|
||
| ## Prerequisites | ||
|
|
||
| See [here](https://kafka-docker-playground.io/#/how-to-use?id=%f0%9f%8c%a4%ef%b8%8f-confluent-cloud-examples) | ||
|
|
||
|
|
||
| ## How to run | ||
|
|
||
| Simply run: | ||
|
|
||
| ```bash | ||
| $ just use <playground run> command and search for fully-managed-google-cloud-functions<use tab key to activate fzf completion (see https://kafka-docker-playground.io/#/cli?id=%e2%9a%a1-setup-completion), otherwise use full path, or correct relative path> <GCP_FUNCTION_REGION> <GCP_FUNCTION_FUNCTION> .sh in this folder | ||
| ``` |
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
98 changes: 98 additions & 0 deletions
98
ccloud/fm-gcp-cloud-functions-gen2-sink/fully-managed-google-cloud-functions-gen2-sink.sh
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,98 @@ | ||
| #!/bin/bash | ||
| set -e | ||
|
|
||
| DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null && pwd )" | ||
| source ${DIR}/../../scripts/utils.sh | ||
| if [ -z "$GCP_PROJECT" ] | ||
| then | ||
| logerror "GCP_PROJECT is not set. Export it as environment variable or pass it as argument" | ||
| exit 1 | ||
| fi | ||
|
|
||
| GCP_FUNCTION_REGION=${1:-europe-west2} | ||
| GCP_FUNCTION_FUNCTION=${2:-function-gen2} | ||
|
|
||
| cd ../../ccloud/fm-gcp-cloud-functions-gen2-sink | ||
| GCP_KEYFILE="${DIR}/keyfile.json" | ||
| if [ ! -f ${GCP_KEYFILE} ] && [ -z "$GCP_KEYFILE_CONTENT" ] | ||
| then | ||
| logerror "ERROR: either the file ${GCP_KEYFILE} is not present or environment variable GCP_KEYFILE_CONTENT is not set!" | ||
| exit 1 | ||
| else | ||
| if [ -f ${GCP_KEYFILE} ] | ||
| then | ||
| GCP_KEYFILE_CONTENT=$(cat keyfile.json | jq -aRs . | sed 's/^"//' | sed 's/"$//') | ||
| else | ||
| log "Creating ${GCP_KEYFILE} based on environment variable GCP_KEYFILE_CONTENT" | ||
| echo -e "$GCP_KEYFILE_CONTENT" | sed 's/\\"/"/g' > ${GCP_KEYFILE} | ||
| fi | ||
| fi | ||
| cd - | ||
|
|
||
| log "Doing gsutil authentication" | ||
| set +e | ||
| docker rm -f gcloud-config | ||
| set -e | ||
| docker run -i -v ${GCP_KEYFILE}:/tmp/keyfile.json --name gcloud-config google/cloud-sdk:latest gcloud auth activate-service-account --project ${GCP_PROJECT} --key-file /tmp/keyfile.json | ||
|
|
||
| bootstrap_ccloud_environment | ||
|
|
||
|
|
||
|
|
||
| set +e | ||
| playground topic delete --topic functions-gen2-messages | ||
| sleep 3 | ||
| playground topic create --topic functions-gen2-messages | ||
| set -e | ||
|
|
||
|
|
||
| log "Produce test data to the functions-gen2-messages topic in Kafka" | ||
| playground topic produce -t functions-gen2-messages --nb-messages 10 --forced-value '{"f1":"value%g"}' << 'EOF' | ||
| { | ||
| "type": "record", | ||
| "name": "myrecord", | ||
| "fields": [ | ||
| { | ||
| "name": "f1", | ||
| "type": "string" | ||
| } | ||
| ] | ||
| } | ||
| EOF | ||
|
|
||
| connector_name="GoogleCloudFunctionsGen2Sink_$USER" | ||
| set +e | ||
| playground connector delete --connector $connector_name > /dev/null 2>&1 | ||
| set -e | ||
|
|
||
| log "Creating fully managed connector" | ||
| playground connector create-or-update --connector $connector_name << EOF | ||
| { | ||
| "connector.class": "GoogleCloudFunctionsGen2Sink", | ||
| "name": "$connector_name", | ||
| "kafka.auth.mode": "KAFKA_API_KEY", | ||
| "kafka.api.key": "$CLOUD_KEY", | ||
| "kafka.api.secret": "$CLOUD_SECRET", | ||
| "topics": "functions-gen2-messages", | ||
| "gcf.name": "$GCP_FUNCTION_FUNCTION", | ||
| "gcf.project.id": "$GCP_PROJECT", | ||
| "gcf.region.name": "$GCP_FUNCTION_REGION", | ||
| "gcp.credentials.json" : "$GCP_KEYFILE_CONTENT", | ||
| "input.data.format" : "AVRO", | ||
| "tasks.max" : "1" | ||
| } | ||
| EOF | ||
| wait_for_ccloud_connector_up $connector_name 180 | ||
|
|
||
| sleep 10 | ||
|
|
||
|
|
||
| connectorId=$(get_ccloud_connector_lcc $connector_name) | ||
|
|
||
| log "Verifying topic success-$connectorId" | ||
| playground topic consume --topic success-$connectorId --min-expected-messages 10 --timeout 60 | ||
|
|
||
| log "Do you want to delete the fully managed connector $connector_name ?" | ||
| check_if_continue | ||
|
|
||
| playground connector delete --connector $connector_name |
File renamed without changes.
File renamed without changes.
4 changes: 2 additions & 2 deletions
4
ccloud/fm-gcp-cloud-functions-sink/README.md → ...gcp-cloud-functions-legacy-sink/README.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
File renamed without changes
File renamed without changes
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,8 @@ | ||
| #!/bin/bash | ||
|
|
||
|
|
||
|
|
||
| DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null && pwd )" | ||
| source ${DIR}/../../scripts/utils.sh | ||
|
|
||
| maybe_delete_ccloud_environment |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters