Skip to content

Commit

Permalink
Merge remote-tracking branch 'origin/master' into travjenkins/docs/up…
Browse files Browse the repository at this point in the history
…date-title-and-descriptions
  • Loading branch information
travjenkins committed Aug 21, 2024
2 parents 30611f4 + 27cf5c7 commit 040cd21
Show file tree
Hide file tree
Showing 29 changed files with 2,382 additions and 2,278 deletions.
57 changes: 33 additions & 24 deletions site/docs/concepts/connectors.md
Original file line number Diff line number Diff line change
Expand Up @@ -54,7 +54,7 @@ or any other public image registry provider.
To interface with a connector, the Flow runtime needs to know:

1. The specific image to use, through an image name such as `ghcr.io/estuary/source-postgres:dev`.
Notice that the image name also conveys the specific image registry and version tag to use.
Notice that the image name also conveys the specific image registry and version tag to use.

2. Endpoint configuration such as a database address and account, with meaning that is specific to the connector.

Expand Down Expand Up @@ -153,8 +153,9 @@ user: postgres
</Tabs>
Storing configuration in separate files serves two important purposes:
* Re-use of configuration across multiple captures or materializations
* The ability to protect sensitive credentials
- Re-use of configuration across multiple captures or materializations
- The ability to protect sensitive credentials
### Protecting secrets
Expand Down Expand Up @@ -208,13 +209,13 @@ host: ENC[AES256_GCM,data:K/clly65pThTg2U=,iv:1bNmY8wjtjHFBcXLR1KFcsNMGVXRl5LGTd
password: ENC[AES256_GCM,data:IDDY+fl0/gAcsH+6tjRdww+G,iv:Ye8st7zJ9wsMRMs6BoAyWlaJeNc9qeNjkkjo6BPp/tE=,tag:EPS9Unkdg4eAFICGujlTfQ==,type:str]
user: ENC[AES256_GCM,data:w+F7MMwQhw==,iv:amHhNCJWAJnJaGujZgjhzVzUZAeSchEpUpBau7RVeCg=,tag:62HguhnnSDqJdKdwYnj7mQ==,type:str]
sops:
# Some items omitted for brevity:
gcp_kms:
- resource_id: projects/your-project-id/locations/us-central1/keyRings/your-ring/cryptoKeys/your-key-name
created_at: "2022-01-05T15:49:45Z"
enc: CiQAW8BC2GDYWrJTp3ikVGkTI2XaZc6F4p/d/PCBlczCz8BZiUISSQCnySJKIptagFkIl01uiBQp056c
lastmodified: "2022-01-05T15:49:45Z"
version: 3.7.1
# Some items omitted for brevity:
gcp_kms:
- resource_id: projects/your-project-id/locations/us-central1/keyRings/your-ring/cryptoKeys/your-key-name
created_at: "2022-01-05T15:49:45Z"
enc: CiQAW8BC2GDYWrJTp3ikVGkTI2XaZc6F4p/d/PCBlczCz8BZiUISSQCnySJKIptagFkIl01uiBQp056c
lastmodified: "2022-01-05T15:49:45Z"
version: 3.7.1
```
You then use this `config.yaml` within your Flow specification.
Expand All @@ -235,6 +236,7 @@ which is:
```
[email protected]
```

:::

#### Example: Protect portions of a configuration
Expand All @@ -259,21 +261,21 @@ Next, encrypt only values which have that suffix:
$ sops --encrypt --in-place --encrypted-suffix "_sops" --gcp-kms projects/your-project-id/locations/us-central1/keyRings/your-ring/cryptoKeys/your-key-name config.yaml
```

`sops` re-writes the file, wrapping only values having a "_sops" suffix and adding its `sops` metadata section:
`sops` re-writes the file, wrapping only values having a "\_sops" suffix and adding its `sops` metadata section:

```yaml title="config.yaml"
host: my.hostname
password_sops: ENC[AES256_GCM,data:dlfidMrHfDxN//nWQTPCsjoG,iv:DHQ5dXhyOOSKI6ZIzcUM67R6DD/2MSE4LENRgOt6GPY=,tag:FNs2pTlzYlagvz7vP/YcIQ==,type:str]
user: my-user
sops:
# Some items omitted for brevity:
encrypted_suffix: _sops
gcp_kms:
- resource_id: projects/your-project-id/locations/us-central1/keyRings/your-ring/cryptoKeys/your-key-name
created_at: "2022-01-05T16:06:36Z"
enc: CiQAW8BC2Au779CGdMFUjWPhNleCTAj9rL949sBvPQ6eyAC3EdESSQCnySJKD3eWX8XrtrgHqx327
lastmodified: "2022-01-05T16:06:37Z"
version: 3.7.1
# Some items omitted for brevity:
encrypted_suffix: _sops
gcp_kms:
- resource_id: projects/your-project-id/locations/us-central1/keyRings/your-ring/cryptoKeys/your-key-name
created_at: "2022-01-05T16:06:36Z"
enc: CiQAW8BC2Au779CGdMFUjWPhNleCTAj9rL949sBvPQ6eyAC3EdESSQCnySJKD3eWX8XrtrgHqx327
lastmodified: "2022-01-05T16:06:37Z"
version: 3.7.1
```

You then use this `config.yaml` within your Flow specification.
Expand All @@ -286,7 +288,14 @@ In some cases, your source or destination endpoint may be within a secure networ
to allow direct access to its port due to your organization's security policy.

:::tip
If permitted by your organization, a quicker solution is to whitelist the Estuary IP address, `34.121.207.128`.
If permitted by your organization, a quicker solution is to whitelist the Estuary IP addresses:

```
34.121.207.128
35.226.75.135
34.68.62.148
```

For help completing this task on different cloud hosting platforms,
see the documentation for the [connector](../reference/Connectors/README.md) you're using.
:::
Expand Down Expand Up @@ -352,10 +361,10 @@ Users are empowered to write their own connectors for esoteric systems not alrea

Furthermore, implementing a Docker-based community specification brings other important qualities to Estuary connectors:

* Cross-platform interoperability between Flow, Airbyte, and any other platform that supports the protocol
* The abilities to write connectors in any language and run them on any machine
* Built-in solutions for version management (through image tags) and distribution
* The ability to integrate connectors from different sources at will, without the centralized control of a single company, thanks to container image registries
- Cross-platform interoperability between Flow, Airbyte, and any other platform that supports the protocol
- The abilities to write connectors in any language and run them on any machine
- Built-in solutions for version management (through image tags) and distribution
- The ability to integrate connectors from different sources at will, without the centralized control of a single company, thanks to container image registries

:::info
In order to be reflected in the Flow web app and used on the managed Flow platform,
Expand Down
Loading

0 comments on commit 040cd21

Please sign in to comment.