Skip to content

Commit

Permalink
Added folder structure for DB capture docs (#1394)
Browse files Browse the repository at this point in the history
* Added folder structure for DB capture docs

* Merged Postgres docs for RDS and GCP; removed Postgres heroku docs

* Removed duplicate MySQL docs

* Removed duplicate SQL Server docs, fixed links

* Altered folder structure for DB materializations

* Reorganized materialization docs for consistency
  • Loading branch information
jonwihl authored Feb 27, 2024
1 parent 316cf3c commit 77d25a4
Show file tree
Hide file tree
Showing 24 changed files with 519 additions and 866 deletions.
8 changes: 8 additions & 0 deletions site/docs/getting-started/installation.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -134,6 +134,14 @@ Be sure to replace `YOUR-S3-BUCKET` with the actual name of your bucket.
},
"Action": "s3:ListBucket",
"Resource": "arn:aws:s3:::YOUR-S3-BUCKET"
},
{
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::789740162118:user/flow-aws"
},
"Action": "s3:GetBucketPolicy",
"Resource": "arn:aws:s3:::YOUR-S3-BUCKET"
}
]
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ sidebar_position: 3
# MariaDB

This is a change data capture (CDC) connector that captures change events from a MariaDB database via the [Binary Log](https://mariadb.com/kb/en/overview-of-the-binary-log/).
It's derived from the [MySQL capture connector](./MySQL.md),
It's derived from the [MySQL capture connector](../MySQL/MySQL.md),
so the same configuration applies, but the setup steps look somewhat different.

This connector is available for use in the Flow web application. For local development or open-source workflows, [`ghcr.io/estuary/source-mariadb:dev`](https://github.com/estuary/connectors/pkgs/container/source-mariadb) provides the latest version of the connector as a Docker image. You can also follow the link in your browser to see past image versions.
Expand All @@ -28,6 +28,10 @@ To use this connector, you'll need a MariaDB database setup with the following.
* If the table(s) to be captured include columns of type `DATETIME`, the `time_zone` system variable
must be set to an IANA zone name or numerical offset or the capture configured with a `timezone` to use by default.

:::tip Configuration Tip
To configure this connector to capture data from databases hosted on your internal network, you must set up SSH tunneling. For more specific instructions on setup, see [configure connections with SSH tunneling](../../../../guides/connect-network/).
:::

### Setup
To meet these requirements, do the following:

Expand Down Expand Up @@ -93,7 +97,7 @@ In this case, you may turn of backfilling on a per-table basis. See [properties]

## Configuration
You configure connectors either in the Flow web app, or by directly editing the catalog specification file.
See [connectors](../../../concepts/connectors.md#using-connectors) to learn more about using connectors. The values and specification sample below provide configuration details specific to the MariaDB source connector.
See [connectors](../../../../concepts/connectors.md#using-connectors) to learn more about using connectors. The values and specification sample below provide configuration details specific to the MariaDB source connector.

### Properties

Expand Down Expand Up @@ -150,7 +154,7 @@ captures:
Your capture definition will likely be more complex, with additional bindings for each table in the source database.
[Learn more about capture definitions.](../../../concepts/captures.md#pull-captures)
[Learn more about capture definitions.](../../../../concepts/captures.md#pull-captures)
## MariaDB on managed cloud platforms
Expand All @@ -176,10 +180,10 @@ You're able to apply the connector directly to the primary instance if you'd lik

:::info
Alternatively, you can allow secure connections via SSH tunneling. To do so:
* Follow the guide to [configure an SSH server for tunneling](../../../../guides/connect-network/)
* Follow the guide to [configure an SSH server for tunneling](../../../../../guides/connect-network/)
* When you configure your connector as described in the [configuration](#configuration) section above,
including the additional `networkTunnel` configuration to enable the SSH tunnel.
See [Connecting to endpoints on secure networks](../../../concepts/connectors.md#connecting-to-endpoints-on-secure-networks)
See [Connecting to endpoints on secure networks](../../../../concepts/connectors.md#connecting-to-endpoints-on-secure-networks)
for additional details and a sample.
:::

Expand Down Expand Up @@ -245,10 +249,10 @@ You can use this connector for MariaDB instances on Azure Database for MariaDB u

:::info
Alternatively, you can allow secure connections via SSH tunneling. To do so:
* Follow the guide to [configure an SSH server for tunneling](../../../../guides/connect-network/)
* Follow the guide to [configure an SSH server for tunneling](../../../../../guides/connect-network/)
* When you configure your connector as described in the [configuration](#configuration) section above,
including the additional `networkTunnel` configuration to enable the SSH tunnel.
See [Connecting to endpoints on secure networks](../../../concepts/connectors.md#connecting-to-endpoints-on-secure-networks)
See [Connecting to endpoints on secure networks](../../../../concepts/connectors.md#connecting-to-endpoints-on-secure-networks)
for additional details and a sample.
:::

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ sidebar_position: 3
# Amazon RDS for MariaDB

This is a change data capture (CDC) connector that captures change events from a MariaDB database via the [Binary Log](https://mariadb.com/kb/en/overview-of-the-binary-log/).
It's derived from the [MySQL capture connector](./MySQL.md),
It's derived from the [MySQL capture connector](../MySQL/MySQL.md),
so the same configuration applies, but the setup steps look somewhat different.

This connector is available for use in the Flow web application. For local development or open-source workflows, [`ghcr.io/estuary/source-mariadb:dev`](https://github.com/estuary/connectors/pkgs/container/source-mariadb) provides the latest version of the connector as a Docker image. You can also follow the link in your browser to see past image versions.
Expand Down Expand Up @@ -40,10 +40,10 @@ To use this connector, you'll need a MariaDB database setup with the following.

:::info
Alternatively, you can allow secure connections via SSH tunneling. To do so:
* Follow the guide to [configure an SSH server for tunneling](../../../../guides/connect-network/)
* Follow the guide to [configure an SSH server for tunneling](../../../../../guides/connect-network/)
* When you configure your connector as described in the [configuration](#configuration) section above,
including the additional `networkTunnel` configuration to enable the SSH tunnel.
See [Connecting to endpoints on secure networks](../../../concepts/connectors.md#connecting-to-endpoints-on-secure-networks)
See [Connecting to endpoints on secure networks](../../../../concepts/connectors.md#connecting-to-endpoints-on-secure-networks)
for additional details and a sample.
:::

Expand Down Expand Up @@ -109,7 +109,7 @@ In this case, you may turn of backfilling on a per-table basis. See [properties]

## Configuration
You configure connectors either in the Flow web app, or by directly editing the catalog specification file.
See [connectors](../../../concepts/connectors.md#using-connectors) to learn more about using connectors. The values and specification sample below provide configuration details specific to the MariaDB source connector.
See [connectors](../../../../concepts/connectors.md#using-connectors) to learn more about using connectors. The values and specification sample below provide configuration details specific to the MariaDB source connector.

### Properties

Expand Down Expand Up @@ -166,7 +166,7 @@ captures:
Your capture definition will likely be more complex, with additional bindings for each table in the source database.
[Learn more about capture definitions.](../../../concepts/captures.md#pull-captures)
[Learn more about capture definitions.](../../../../concepts/captures.md#pull-captures)
## Troubleshooting Capture Errors
Expand Down
Loading

0 comments on commit 77d25a4

Please sign in to comment.