Feature/UUID timestamp extraction

[jshearer]

Description:

Customers were asking for the ability to include a timestamp column in their materializations that represents the point at which Flow saw that piece of data, effectively a "last modified" or "valid as of" field. We actually always had this data encoded as part of each document's UUID, and this PR allows you to get access to it via a "virtual projection" at the location /_meta/uuid/timestamp.

Workflow steps:

In order to materialize this timestamp, you have to:

1. Create a projection referencing /_meta/uuid/timestamp
2. Include that projection in your materialization's field selection

Unfortunately the UI does not support projections yet, so there is still some CLI work involved. It does technically support field selection by way of the "Advanced Specification Editor".

Given that, the workflow to materialize the meta uuid timestamp is:

1. Pull down an existing capture: flowctl catalog pull-specs
2. Add the projection:

   projections:
     meta_uuid_timestamp: /_meta/uuid/timestamp

3. Publish your change: flowctl catalog publish
4. Create a materialization for that collection
5. Include your projection in the field selection inside the advanced spec editor:

   "bindings": [
       {
         "source": "my/collection/name",
         "resource": {
           "foo": "bar"
         },
         "fields": {
           "recommended": true,
           "include": {
             "meta_uuid_timestamp": {}
           }
         }
       }
   ],

If everything works, you should that field materialized! Example of materializing a hello world capture to postgres with the meta timestamp projection:

---

This change is 

[jshearer]

@jgraettinger see here for my explanation of uuid_ptr and inject_uuid_placeholder:

flow/crates/derive/src/combine_api.rs

Lines 65 to 70 in 93f8855

	// NOTE about UUID pointers:
	// As a practical matter, materializations don’t ask the combiner to set a UUID
	// only captures and derivations do. Since `inject_uuid_placeholder` implies that the
	// UUID does not actually exist in the document yet, only when it's false (and `uuid_ptr` is Some)
	// can we reasonably expect to resolve the virtual field <uuid_ptr>/timestamp,
	// otherwise we'll just return null for that field.

[jgraettinger]

FYI you can update Go snapshots by running UPDATE_SNAPSHOTS=1 make go-test-fast

[jgraettinger]

nit: avoid extra space here. Have a single block of use statements, so that they're automatically sorted by the formatter.

[jgraettinger]

not thrilled with adding new dependency on derive. That crate is slatted to be taken out back to the woodshed...

[jgraettinger]

i don't love the introduced dependence on derive ... that crate is slated to be taken out behind the woodshed.

[jgraettinger]

Most of the reason for Param is to provide separation from the details of CollectionSpec representation and the way we actually use parameters for SQLite.

I'm thinking we need to introduce a is_uuid_timestamp boolean on flow::Projection.
Then, there would be projections like field: _meta/uuid/timestamp (or something custom) ptr: /_meta/uuid is_uuid_timestamp: true

That allows resolution of all this stuff from only the flow::Projection instance, and we don't need to put logic for UUID_PTR+'/timestamp' anywhere except the validation logic that makes a flow::Projection.

For this crate specifically, derive::PointerExt is backed out and instead, within the existing :

 Some(Node::String(s)) => {
           if *is_format_integer {

block, you also test for is_uuid_timestamp and extract it accordingly.

[jgraettinger]

🤔 I'm thinking the bulk of this should live in flow-proto -- perhaps an impl of flow::UuidParts::from_uuid(uuid: uuid::Uuid) -> Self or even impl From<uuid::Uuid> for flow::UuidParts etc would definitely make sense there.

[jgraettinger]

extrapolating from my comment above that is_uuid_timestamp belongs in flow::Projection, that implies that we need to pass flow::Projection instances in for key_ptrs and field_ptrs instead of raw JSON pointers.

That's a shift, but I do think that's probably the right call. if there are other circumstances that arise where we want to condition extraction behavior then standardizing on the passed-around flow::Projection greatly simplifies things.

come to think of it, default values are another good example! drain_chunk uses it's own parsed Shape below, but the flow::Projection also bakes in the default value. And there was another circumstance this came up, i think the extract API, where the source schema shape couldn't easily be plumbed through...

[jshearer]

Superseded by #1098