diff --git a/airbyte-integrations/connectors/destination-glide/README.md b/airbyte-integrations/connectors/destination-glide/README.md index 9115f521d6e7..5a0c9343eca9 100644 --- a/airbyte-integrations/connectors/destination-glide/README.md +++ b/airbyte-integrations/connectors/destination-glide/README.md @@ -47,6 +47,12 @@ There are a set of simple integration tests that Airbyte provides that can be tr These simply call commands that Airbyte provides in their connector template. The dev-write one appears to be the most comprehensive, but I've struggled to get that one to consistently run (see TODO). +There are additional tests you can run against the live API in `integration_tests/` that you can run with: + +```sh +./scripts/test-integration.sh +``` + ### Build & Deployment The Airbyte destination is packed as Docker image. This script uses Airbyte-provided tooling named `airbyte-ci` that leverages the same tooling they use in their CI pipeline to build the container. @@ -57,7 +63,22 @@ To install the tooling see [`airbyte-ci` README in this repo](https://github.com ./scripts/build-docker-image.sh ``` -We are currently deploying this to a public repository for ease of access from an Airbyte OSS instance. To deploy it to a docker container registry use the script at: +We are deploying this to Glide's docker repository in Google Artifact Registry (GAR) for ease of access from an Airbyte OSS instance. To deploy it to a docker container registry you need to authenticate docker to GAR and push it. The steps are: + +#### Glide Google Artifact Registry Docker Repo: + +Our repo is at https://console.cloud.google.com/artifacts/docker/glide-connectors/us-central1/airbyte-glide-destination?project=glide-connectors + +#### Pushing a Docker Image to Google Artifact Registry + +Read access is available for all, but to push you have to authenticate. Authenticate docker by running the following command which adds credential helper to `~/.docker/config.json`: + +``` +gcloud auth configure-docker \ + us-central1-docker.pkg.dev +``` + +Then you can use gcloud and docker commands as normal. To push a new image version run: ```sh ./scripts/push-docker-image.sh @@ -69,13 +90,48 @@ To install Airbyte follow the guide at https://docs.airbyte.com/deploying-airbyt Once install it should be available at http://localhost:8000/. You should have been prompted for username/pw during install. -### Installing Glide Destination in Airbyte OSS +### Installing Glide Destination in Airbyte OSS (on Kubernetes) + +Install kind (macOS): + +NOTE: abctl installs kind, but it doesn't install the kind CLI, so to work with the airbyte kubernetes cluster follow these steps: + +```sh +brew install kind +``` + +Then you can list the clusters and you should see one named `airbyte-abctl`: + +```sh +$ kind get clusters +airbyte-abctl +``` + +To use it with kubectl (or k9s) use: + +```sh +# this updates ~/.kube/config to add the cluster +kind export kubeconfig --name airbyte-abctl + +# set the context to the cluster context that kind added: +kubectl config set-context kind-airbyte-abctl + +# now kubectl works! +kubectl get namespaces +kubectl get -n airbyte-abctl pods +``` + +Now we follow the course guidance at https://docs.airbyte.com/operator-guides/using-custom-connectors/#for-kubernetes-airbyte-deployments ... + +We made our docker registry public so you don't have to authenticate airbyte's kubernetes deployment. To install the destination into Airbyte OSS follow these steps: 1. Click on **Settings** on the far left then select **Destinations** in the sub-panel. You should see a list of **Available destination connectors**. 2. At the top click the **+ New Connector** button fill in the fields. The **Docker repository name** and **Docker image tag** are the important bits. +For repository name use `us-central1-docker.pkg.dev/glide-connectors/airbyte-glide-destination/destination-glide` and for tag, run `gcloud artifacts docker tags list us-central1-docker.pkg.dev/glide-connectors/airbyte-glide-destination/destination-glide` to get the available tags and choose the latest. + Once installed, you can upgrade it to a new version by visiting the same settings page and changing the tag in the **Change to** box and clicking the **Change** button. --- diff --git a/airbyte-integrations/connectors/destination-glide/destination_glide/spec.json b/airbyte-integrations/connectors/destination-glide/destination_glide/spec.json index f32a6b07840a..8f8f8dbe4ee2 100644 --- a/airbyte-integrations/connectors/destination-glide/destination_glide/spec.json +++ b/airbyte-integrations/connectors/destination-glide/destination_glide/spec.json @@ -6,16 +6,16 @@ "$schema": "http://json-schema.org/draft-07/schema#", "title": "Destination Glide", "type": "object", - "required": ["api_host", "api_path_root", "api_key"], + "required": ["api_key"], "additionalProperties": false, "properties": { "api_host": { "type": "string", - "description": "The protocol and host name (e.g. 'https://api.glideapp.io'). of the Glide API destination" + "description": "The protocol and host name of the Glide API destination. Defaults to 'https://api.glideapps.com'." }, "api_path_root": { "type": "string", - "description": "The path root of the Glide API destination" + "description": "The path root of the Glide API destination. Defaults to empty." }, "api_key": { "type": "string", diff --git a/airbyte-integrations/connectors/destination-glide/scripts/build-docker-image.sh b/airbyte-integrations/connectors/destination-glide/scripts/build-docker-image.sh index bdebc21ea741..fe64db0fb333 100755 --- a/airbyte-integrations/connectors/destination-glide/scripts/build-docker-image.sh +++ b/airbyte-integrations/connectors/destination-glide/scripts/build-docker-image.sh @@ -2,5 +2,6 @@ this_dir=$(cd $(dirname "$0"); pwd) # this script's directory this_script=$(basename $0) -airbyte-ci connectors --name=destination-glide build +# segment sometimes fails, so AIRBYTE_CI_DISABLE_TELEMETRY=true to disable it per airbyte-ci/connectors/pipelines/README.md +AIRBYTE_CI_DISABLE_TELEMETRY=true airbyte-ci connectors --name=destination-glide build diff --git a/airbyte-integrations/connectors/destination-glide/scripts/push-docker-image.sh b/airbyte-integrations/connectors/destination-glide/scripts/push-docker-image.sh index 37054bc1500c..53fb62033fe6 100755 --- a/airbyte-integrations/connectors/destination-glide/scripts/push-docker-image.sh +++ b/airbyte-integrations/connectors/destination-glide/scripts/push-docker-image.sh @@ -3,15 +3,22 @@ this_dir=$(cd $(dirname "$0"); pwd) # this script's directory this_script=$(basename $0) #!/usr/bin/env bash +LOCAL_BUILT_IMAGE_NAME=airbyte/destination-glide:dev +GLIDE_DOCKER_IMAGE_NAME=us-central1-docker.pkg.dev/glide-connectors/airbyte-glide-destination/destination-glide + +echo "Pushing to remote image repository: $GLIDE_DOCKER_IMAGE_NAME" +docker inspect --format='Pushing the local image "{{index .RepoTags 0}}" created at "{{.Created}}"' $LOCAL_BUILT_IMAGE_NAME + # Fetch the list of tags from Docker Hub -tags=$(wget -q -O - "https://hub.docker.com/v2/namespaces/activescott/repositories/destination-glide/tags?page_size=10" | grep -o '"name": *"[^"]*' | grep -o '[^"]*$' | grep -E '([0-9]+\.)+[0-9]+' ) -`` -echo "Found tags: $tags" +#tags=$(wget -q -O - "https://hub.docker.com/v2/namespaces/activescott/repositories/destination-glide/tags?page_size=10" | grep -o '"name": *"[^"]*' | grep -o '[^"]*$' | grep -E '([0-9]+\.)+[0-9]+' ) +# Get the list of tags using gcloud with the appropriate format +tags=$(gcloud artifacts docker tags list $GLIDE_DOCKER_IMAGE_NAME --format="get(tag)") + # Sort the tags and get the highest one -highest_tag=$(echo "$tags" | sort -V | tail -n 1) +highest_tag=$(echo "$tags" | awk -F'/' '{print $NF}' | sort -V | tail -n 1) -echo "found highest tag: $highest_tag" +echo "found highest tag on remote: $highest_tag" # Increment the version IFS='.' read -ra ADDR <<< "$highest_tag" @@ -28,12 +35,10 @@ else exit 1 fi - # Tag the local image with the new version -# TODO: airbyte/destination-glide is the local docker image that airbyte's CI builds it and names it as locally. Can't we change this ?? -docker image tag airbyte/destination-glide:dev activescott/destination-glide:$new_version - -# Push the image with the new tag -docker push activescott/destination-glide:$new_version +# NOTE: airbyte/destination-glide is the local docker image that airbyte's CI builds and names locally. +docker image tag $LOCAL_BUILT_IMAGE_NAME $GLIDE_DOCKER_IMAGE_NAME:$new_version +# Push the image with the new tag +docker push $GLIDE_DOCKER_IMAGE_NAME:$new_version