diff --git a/airbyte-integrations/connectors/destination-glide/README.md b/airbyte-integrations/connectors/destination-glide/README.md index 44afb91a9e47..9115f521d6e7 100644 --- a/airbyte-integrations/connectors/destination-glide/README.md +++ b/airbyte-integrations/connectors/destination-glide/README.md @@ -3,23 +3,93 @@ This is the repository for the Glide destination connector, written in Python. For information about how to use this connector within Airbyte, see [the documentation](https://docs.airbyte.com/integrations/destinations/glide). -## Local development +## Development + +The active todo list is at [./todo.md]. +The gist of the Glide-specific code is in `/destination_glide/destination.py` and `/destination_glide/glide.py`. ### Prerequisites -* Python (`^3.9`) -* Poetry (`^1.7`) - installation instructions [here](https://python-poetry.org/docs/#installation) +- Python (`^3.9`, tested recently with `3.12.3`) +- Poetry (`^1.7`, tested recently with `1.8.3_1`) + +I used homebrew for installing these prerequisites on macOS. + +### Unit Tests + +The unit tests for that code are in `/destination-glide/unit_tests`. To run them run: + +```sh +./scripts/test-unit.sh +``` + +### Integration Tests + +The destination has a configuration in `/secrets/config.json`. That file must confirm to the configuration specification in `/destination_glide/spec.json`. It should be something like: + +```json +{ + "api_host": "http://localhost:5005", + "api_path_root": "api", + "api_key": "decafbad-1234-1234-1234-decafbad" +} +``` + +The spec also specifies the configuration UI within the Airbyte product itself for configuring the destination. + +There are a set of simple integration tests that Airbyte provides that can be triggered with the following scripts: + +```sh +./scripts/dev-check.sh +./scripts/dev-spec.sh +./scripts/dev-write.sh +``` + +These simply call commands that Airbyte provides in their connector template. The dev-write one appears to be the most comprehensive, but I've struggled to get that one to consistently run (see TODO). + +### Build & Deployment + +The Airbyte destination is packed as Docker image. This script uses Airbyte-provided tooling named `airbyte-ci` that leverages the same tooling they use in their CI pipeline to build the container. + +To install the tooling see [`airbyte-ci` README in this repo](https://github.com/airbytehq/airbyte/blob/master/airbyte-ci/connectors/pipelines/README.md) This worked for me other than to make it work on macOS with homebrew-provided python, I don't have `python` on my path only `python3` so I had to change that in a couple places in the `Makefile` in the root of this repo. +```sh +./scripts/build-docker-image.sh +``` + +We are currently deploying this to a public repository for ease of access from an Airbyte OSS instance. To deploy it to a docker container registry use the script at: + +```sh +./scripts/push-docker-image.sh +``` + +### Running in Airbyte OSS Locally + +To install Airbyte follow the guide at https://docs.airbyte.com/deploying-airbyte/quickstart. On macOS this uses homebrew to install k8s kind locally and get an airbyte cluster running. It took a while but worked smoothly for me. I am currently using `airbytehq/tap/abctl (0.5.0)`. + +Once install it should be available at http://localhost:8000/. You should have been prompted for username/pw during install. + +### Installing Glide Destination in Airbyte OSS + +To install the destination into Airbyte OSS follow these steps: +1. Click on **Settings** on the far left then select **Destinations** in the sub-panel. You should see a list of **Available destination connectors**. +2. At the top click the **+ New Connector** button fill in the fields. The **Docker repository name** and **Docker image tag** are the important bits. + +Once installed, you can upgrade it to a new version by visiting the same settings page and changing the tag in the **Change to** box and clicking the **Change** button. + +--- + +## Old (from Airbyte's Template) ### Installing the connector From this connector directory, run: + ```bash poetry install --with dev ``` - #### Create credentials **If you are a community contributor**, follow the instructions in the [documentation](https://docs.airbyte.com/integrations/destinations/glide) @@ -31,6 +101,7 @@ See `integration_tests/sample_config.json` for a sample config file. and place them into `secrets/config.json`. ### Locally running the connector + ``` poetry run destination-glide spec poetry run destination-glide check --config secrets/config.json @@ -49,6 +120,7 @@ poetry run pytest tests 1. Install [`airbyte-ci`](https://github.com/airbytehq/airbyte/blob/master/airbyte-ci/connectors/pipelines/README.md) 2. Run the following command to build the docker image: + ```bash airbyte-ci connectors --name=destination-glide build ``` @@ -58,6 +130,7 @@ An image will be available on your host with the tag `airbyte/destination-glide: ### Running as a docker container Then run any of the connector commands as follows: + ``` docker run --rm airbyte/destination-glide:dev spec docker run --rm -v $(pwd)/secrets:/secrets airbyte/destination-glide:dev check --config /secrets/config.json @@ -91,10 +164,11 @@ Please commit the changes to `pyproject.toml` and `poetry.lock` files. ## Publishing a new version of the connector You've checked out the repo, implemented a million dollar feature, and you're ready to share your changes with the world. Now what? + 1. Make sure your changes are passing our test suite: `airbyte-ci connectors --name=destination-glide test` 2. Bump the connector version (please follow [semantic versioning for connectors](https://docs.airbyte.com/contributing-to-airbyte/resources/pull-requests-handbook/#semantic-versioning-for-connectors)): - - bump the `dockerImageTag` value in in `metadata.yaml` - - bump the `version` value in `pyproject.toml` + - bump the `dockerImageTag` value in in `metadata.yaml` + - bump the `version` value in `pyproject.toml` 3. Make sure the `metadata.yaml` content is up to date. 4. Make sure the connector documentation and its changelog is up to date (`docs/integrations/destinations/glide.md`). 5. Create a Pull Request: use [our PR naming conventions](https://docs.airbyte.com/contributing-to-airbyte/resources/pull-requests-handbook/#pull-request-title-convention). diff --git a/airbyte-integrations/connectors/destination-glide/todo.md b/airbyte-integrations/connectors/destination-glide/todo.md index 3fbd6440a431..50a863aa3b2b 100644 --- a/airbyte-integrations/connectors/destination-glide/todo.md +++ b/airbyte-integrations/connectors/destination-glide/todo.md @@ -1,18 +1,20 @@ -- [+] feat: choose a strategy based on config -- [+] chore: stop writing any record data to logs -- [+] chore: cleanup logs generally -- [+] fix: "add rows failed with status 400: {"message":"More than 500 mutations"}" in mutation api (batch them to ~100 or something) -- [+] fix: batch row adds in rest api -- [+] chore: unify test framework across unit/integration tests (remove pytest?) -- [+] feat: snuggle rows in a batch -- [+] feat: support multiple streams from source by using a naming convention and discovering existing tables with GET /tables +- [x] feat: choose a strategy based on config +- [x] chore: stop writing any record data to logs +- [x] chore: cleanup logs generally +- [x] fix: "add rows failed with status 400: {"message":"More than 500 mutations"}" in mutation api (batch them to ~100 or something) +- [x] fix: batch row adds in rest api +- [x] chore: unify test framework across unit/integration tests (remove pytest?) +- [x] feat: snuggle rows in a batch +- [x] feat: support multiple streams from source by using a naming convention and discovering existing tables with GET /tables +- [ ] feat: add airbyte\_ prefix to all tables to prevent user from inadvertently overwriting existing table. - [ ] fix: replace "hostname" and "path" configs with "baseUrl" +- [ ] fix: use friendly names for the properties on the config page - [ ] feat: verify actual host/api/auth connection in check: https://docs.airbyte.com/understanding-airbyte/airbyte-protocol#check - [ ] feat: add a default timeout everywhere per https://requests.readthedocs.io/en/latest/user/quickstart/#timeouts -- [ ] fix: use friendly names for the properties on the config page - [ ] chore: update readme +- [ ] feat: incremental update support for airbyte (explore what APIs we need to do this) - [ ] chore: unit test for Column to ensure it serializes to json as expected -- [ ] chore: remove GlideBigTableMutationsStrategy? +- [x] chore: remove GlideBigTableMutationsStrategy? - [ ] chore: clean up todos - [ ] chore: figure out why the dev-write script stopped working (freezes)