feature: Teach the combiner to keep track of a running inferred schema and log whenever it changes. #1128
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
jshearer
force-pushed
the
feature/shape_widening
branch
from
04eba63
to
c895a0e
Compare
jshearer
force-pushed
the
feature/combiner_emit_schemas
branch
from
e6eb307
to
483f11e
Compare
jshearer
force-pushed
the
feature/combiner_emit_schemas
branch
2 times, most recently
from
713009d
to
f185673
Compare
jshearer
force-pushed
the
refactor/schema_builder
branch
from
218b8af
to
aa55bfa
Compare
jshearer
force-pushed
the
feature/combiner_emit_schemas
branch
2 times, most recently
from
ec9aec8
to
25b6025
Compare
|
I'm also done force-pushing here too, sorry 🤦 |
jgraettinger
reviewed
Aug 4, 2023
There was a problem hiding this comment.
Two major notes:
-
We want to selectively enable this widening / logging. Capture tasks definitely should do it, but materializations don't need to and also don't want to pay the overhead. How about a new boolean to enable the behavior in the protobuf config API?
-
The logs must include the collection name, which also needs to be plumbed through.
Other than that and comments below LGTM 👍
|
Ok, I added |
jshearer
force-pushed
the
feature/combiner_emit_schemas
branch
from
73e1103
to
9b2066c
Compare
|
…a and log whenever it changes.
…ew Combiner, otherwise keep the schema around to reduce spurious logging
… `Config` message, and plumb through the appropriate values.
jshearer
force-pushed
the
feature/combiner_emit_schemas
branch
from
9b2066c
to
0f50e31
Compare
…n `enable_schema_inference`
jshearer
force-pushed
the
feature/combiner_emit_schemas
branch
from
0f50e31
to
f536e96
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is the second step of #1115.
This change is