Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

end-to-end continuous schema inference #1178

Merged
merged 4 commits into from
Sep 12, 2023
Merged

end-to-end continuous schema inference #1178

merged 4 commits into from
Sep 12, 2023

Commits on Sep 12, 2023

  1. rustfmt: remove because group_imports is not stable

    @jgraettinger
    jgraettinger committed Sep 12, 2023
    Configuration menu
    Copy the full SHA
    efb3761 View commit details
    Browse the repository at this point in the history

  2. validation & flowctl: directly fetch built specs from the control-plane 

    Remove the migration shim which dynamically builds collections from
their user-level specs.

Update flowctl to directly retrieve built specs rather than the
user-level spec.
    @jgraettinger
    jgraettinger committed Sep 12, 2023
    Configuration menu
    Copy the full SHA
    2e36b11 View commit details
    Browse the repository at this point in the history

  3. end-to-end schema inference 

    Introduce well-known URIs `flow://write-schema` and
`flow://inferred-schema` which are available to collection read schemas.

These URIs may be `$ref`'d from within a read schema, and having done
so, their definitions are inlined into the effective read schema of the
built collection with every publication.

Extend the `validation::ControlPlane` trait to enable resolution of
collections to their current inferred schemas. Implement within
`flowctl` through our API, and within `agent` via out-of-transaction SQL
lookups done using the agent's Postgres pool.

Update the `agent` discovers handler to map the existing
`x-infer-schema: true` annotation into a synthesized initial read
schema, that composes the write & inferred schemas.

Issue #1103
    @jgraettinger
    jgraettinger committed Sep 12, 2023
    Configuration menu
    Copy the full SHA
    9036dc9 View commit details
    Browse the repository at this point in the history

  4. cleanup: remove deprecated derivation property of collection spec 

    It's been deprecated a while now, and no specs in production use it any
longer.
    @jgraettinger
    jgraettinger committed Sep 12, 2023
    Configuration menu
    Copy the full SHA
    543c258 View commit details
    Browse the repository at this point in the history