From 76977480a6c069471b7e47cc53d63c3f99eaa43a Mon Sep 17 00:00:00 2001
From: Scott Willeke <scott@willeke.com>
Date: Thu, 27 Jun 2024 07:58:12 -0700
Subject: [PATCH] docs: Readme update to help others dev-test locally

---
 .../connectors/destination-glide/README.md    | 86 +++++++++++++++++--
 .../connectors/destination-glide/todo.md      | 22 ++---
 2 files changed, 92 insertions(+), 16 deletions(-)

diff --git a/airbyte-integrations/connectors/destination-glide/README.md b/airbyte-integrations/connectors/destination-glide/README.md
index 44afb91a9e47..9115f521d6e7 100644
--- a/airbyte-integrations/connectors/destination-glide/README.md
+++ b/airbyte-integrations/connectors/destination-glide/README.md
@@ -3,23 +3,93 @@
 This is the repository for the Glide destination connector, written in Python.
 For information about how to use this connector within Airbyte, see [the documentation](https://docs.airbyte.com/integrations/destinations/glide).
 
-## Local development
+## Development
+
+The active todo list is at [./todo.md].
+The gist of the Glide-specific code is in `/destination_glide/destination.py` and `/destination_glide/glide.py`.
 
 ### Prerequisites
 
-* Python (`^3.9`)
-* Poetry (`^1.7`) - installation instructions [here](https://python-poetry.org/docs/#installation)
+- Python (`^3.9`, tested recently with `3.12.3`)
+- Poetry (`^1.7`, tested recently with `1.8.3_1`)
+
+I used homebrew for installing these prerequisites on macOS.
+
+### Unit Tests
+
+The unit tests for that code are in `/destination-glide/unit_tests`. To run them run:
+
+```sh
+./scripts/test-unit.sh
+```
+
+### Integration Tests
+
+The destination has a configuration in `/secrets/config.json`. That file must confirm to the configuration specification in `/destination_glide/spec.json`. It should be something like:
+
+```json
+{
+  "api_host": "http://localhost:5005",
+  "api_path_root": "api",
+  "api_key": "decafbad-1234-1234-1234-decafbad"
+}
+```
+
+The spec also specifies the configuration UI within the Airbyte product itself for configuring the destination.
+
+There are a set of simple integration tests that Airbyte provides that can be triggered with the following scripts:
+
+```sh
+./scripts/dev-check.sh
+./scripts/dev-spec.sh
+./scripts/dev-write.sh
+```
+
+These simply call commands that Airbyte provides in their connector template. The dev-write one appears to be the most comprehensive, but I've struggled to get that one to consistently run (see TODO).
+
+### Build & Deployment
+
+The Airbyte destination is packed as Docker image. This script uses Airbyte-provided tooling named `airbyte-ci` that leverages the same tooling they use in their CI pipeline to build the container.
+
+To install the tooling see [`airbyte-ci` README in this repo](https://github.com/airbytehq/airbyte/blob/master/airbyte-ci/connectors/pipelines/README.md) This worked for me other than to make it work on macOS with homebrew-provided python, I don't have `python` on my path only `python3` so I had to change that in a couple places in the `Makefile` in the root of this repo.
 
+```sh
+./scripts/build-docker-image.sh
+```
+
+We are currently deploying this to a public repository for ease of access from an Airbyte OSS instance. To deploy it to a docker container registry use the script at:
+
+```sh
+./scripts/push-docker-image.sh
+```
+
+### Running in Airbyte OSS Locally
+
+To install Airbyte follow the guide at https://docs.airbyte.com/deploying-airbyte/quickstart. On macOS this uses homebrew to install k8s kind locally and get an airbyte cluster running. It took a while but worked smoothly for me. I am currently using `airbytehq/tap/abctl (0.5.0)`.
+
+Once install it should be available at http://localhost:8000/. You should have been prompted for username/pw during install.
+
+### Installing Glide Destination in Airbyte OSS
+
+To install the destination into Airbyte OSS follow these steps:
 
+1. Click on **Settings** on the far left then select **Destinations** in the sub-panel. You should see a list of **Available destination connectors**.
+2. At the top click the **+ New Connector** button fill in the fields. The **Docker repository name** and **Docker image tag** are the important bits.
+
+Once installed, you can upgrade it to a new version by visiting the same settings page and changing the tag in the **Change to** box and clicking the **Change** button.
+
+---
+
+## Old (from Airbyte's Template)
 
 ### Installing the connector
 
 From this connector directory, run:
+
 ```bash
 poetry install --with dev
 ```
 
-
 #### Create credentials
 
 **If you are a community contributor**, follow the instructions in the [documentation](https://docs.airbyte.com/integrations/destinations/glide)
@@ -31,6 +101,7 @@ See `integration_tests/sample_config.json` for a sample config file.
 and place them into `secrets/config.json`.
 
 ### Locally running the connector
+
 ```
 poetry run destination-glide spec
 poetry run destination-glide check --config secrets/config.json
@@ -49,6 +120,7 @@ poetry run pytest tests
 
 1. Install [`airbyte-ci`](https://github.com/airbytehq/airbyte/blob/master/airbyte-ci/connectors/pipelines/README.md)
 2. Run the following command to build the docker image:
+
 ```bash
 airbyte-ci connectors --name=destination-glide build
 ```
@@ -58,6 +130,7 @@ An image will be available on your host with the tag `airbyte/destination-glide:
 ### Running as a docker container
 
 Then run any of the connector commands as follows:
+
 ```
 docker run --rm airbyte/destination-glide:dev spec
 docker run --rm -v $(pwd)/secrets:/secrets airbyte/destination-glide:dev check --config /secrets/config.json
@@ -91,10 +164,11 @@ Please commit the changes to `pyproject.toml` and `poetry.lock` files.
 ## Publishing a new version of the connector
 
 You've checked out the repo, implemented a million dollar feature, and you're ready to share your changes with the world. Now what?
+
 1. Make sure your changes are passing our test suite: `airbyte-ci connectors --name=destination-glide test`
 2. Bump the connector version (please follow [semantic versioning for connectors](https://docs.airbyte.com/contributing-to-airbyte/resources/pull-requests-handbook/#semantic-versioning-for-connectors)):
-    - bump the `dockerImageTag` value in in `metadata.yaml`
-    - bump the `version` value in `pyproject.toml`
+   - bump the `dockerImageTag` value in in `metadata.yaml`
+   - bump the `version` value in `pyproject.toml`
 3. Make sure the `metadata.yaml` content is up to date.
 4. Make sure the connector documentation and its changelog is up to date (`docs/integrations/destinations/glide.md`).
 5. Create a Pull Request: use [our PR naming conventions](https://docs.airbyte.com/contributing-to-airbyte/resources/pull-requests-handbook/#pull-request-title-convention).
diff --git a/airbyte-integrations/connectors/destination-glide/todo.md b/airbyte-integrations/connectors/destination-glide/todo.md
index 3fbd6440a431..50a863aa3b2b 100644
--- a/airbyte-integrations/connectors/destination-glide/todo.md
+++ b/airbyte-integrations/connectors/destination-glide/todo.md
@@ -1,18 +1,20 @@
-- [+] feat: choose a strategy based on config
-- [+] chore: stop writing any record data to logs
-- [+] chore: cleanup logs generally
-- [+] fix: "add rows failed with status 400: {"message":"More than 500 mutations"}" in mutation api (batch them to ~100 or something)
-- [+] fix: batch row adds in rest api
-- [+] chore: unify test framework across unit/integration tests (remove pytest?)
-- [+] feat: snuggle rows in a batch
-- [+] feat: support multiple streams from source by using a naming convention and discovering existing tables with GET /tables
+- [x] feat: choose a strategy based on config
+- [x] chore: stop writing any record data to logs
+- [x] chore: cleanup logs generally
+- [x] fix: "add rows failed with status 400: {"message":"More than 500 mutations"}" in mutation api (batch them to ~100 or something)
+- [x] fix: batch row adds in rest api
+- [x] chore: unify test framework across unit/integration tests (remove pytest?)
+- [x] feat: snuggle rows in a batch
+- [x] feat: support multiple streams from source by using a naming convention and discovering existing tables with GET /tables
+- [ ] feat: add airbyte\_ prefix to all tables to prevent user from inadvertently overwriting existing table.
 - [ ] fix: replace "hostname" and "path" configs with "baseUrl"
+- [ ] fix: use friendly names for the properties on the config page
 - [ ] feat: verify actual host/api/auth connection in check: https://docs.airbyte.com/understanding-airbyte/airbyte-protocol#check
 - [ ] feat: add a default timeout everywhere per https://requests.readthedocs.io/en/latest/user/quickstart/#timeouts
-- [ ] fix: use friendly names for the properties on the config page
 
 - [ ] chore: update readme
+- [ ] feat: incremental update support for airbyte (explore what APIs we need to do this)
 - [ ] chore: unit test for Column to ensure it serializes to json as expected
-- [ ] chore: remove GlideBigTableMutationsStrategy?
+- [x] chore: remove GlideBigTableMutationsStrategy?
 - [ ] chore: clean up todos
 - [ ] chore: figure out why the dev-write script stopped working (freezes)