Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Wb/misc docs #1517

Merged
merged 2 commits into from
Jul 11, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 14 additions & 0 deletions site/docs/reference/Connectors/capture-connectors/mongodb.md
Original file line number Diff line number Diff line change
Expand Up @@ -107,3 +107,17 @@ or [set a minimum retention
period](https://www.mongodb.com/docs/manual/reference/command/replSetResizeOplog/#minimum-oplog-retention-period)
for your oplog to be able to reliably capture data. The recommended minimum retention period is at
least 24 hours, but we recommend higher values to improve reliability.

## Change Event Pre- and Post-Images

Captured documents for change events from `update` operations will always
include a full post-image, since the change stream is configured with the [`{
fullDocument: 'updateLookup' }`
setting](https://www.mongodb.com/docs/manual/changeStreams/#lookup-full-document-for-update-operations).

Pre-images for `update`, `replace`, and `delete` operations will be captured if
they are available. For these pre-images to be captured, the source MongoDB
collection must have `changeStreamPreAndPostImages` enabled. See the [official
MongoDB
documentation](https://www.mongodb.com/docs/manual/changeStreams/#change-streams-with-document-pre--and-post-images)
for more information on how to enable this setting.
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,7 @@ more of your Flow collections to your desired tables in the database.
| Property | Title | Description | Type | Required/Default |
|------------------|--------------------|---------------------------------------------------------------------------------------------------------------|---------|------------------|
| **`/table`** | Table | Name of the database table. | string | Required |
| `/delta_updates` | Delta Update | Should updates to this table be done via delta updates. Currently this connector only supports delta updates. | boolean | `true` |
| `/delta_updates` | Delta Update | Should updates to this table be done via delta updates. | boolean | |
| `/schema` | Alternative Schema | Alternative schema for this table (optional). | string | |

### Sample
Expand All @@ -75,5 +75,23 @@ materializations:

## Delta updates

This connector currently supports only [delta updates](../../../../concepts/materialization/#delta-updates).
Future support for standard updates is planned.
This connector supports both standard (merge) and [delta
updates](../../../concepts/materialization.md#delta-updates). The default is to
use standard updates.

Enabling delta updates will prevent Flow from querying for documents in your
MotherDuck table, which can reduce latency and costs for large datasets. If you're
certain that all events will have unique keys, enabling delta updates is a
simple way to improve performance with no effect on the output. However,
enabling delta updates is not suitable for all workflows, as the resulting table
in MotherDuck won't be fully reduced.

You can enable delta updates on a per-binding basis:

```yaml
bindings:
- resource:
table: ${table_name}
delta_updates: true
source: ${PREFIX}/${COLLECTION_NAME}
```
Loading