Review UX for --defer

[jtcohen6]

We're making a subtle-yet-significant change to deferral in v0.19 (#2946, #2954), which is already a complex and under-appreciated feature. It's well and good to document those changes, but there are also things we can do within this codebase to make the feature more intuitive for users.

Naming

Should we still call this defer?

Here's what we're trying to get across:

• You don't need to re-do work if you don't have to. You can stand on the shoulders of giants, i.e. your own past self.
• The flexibility of "failing over" onto a backup set. Missing a resource you need? No worries, let's just grab it from somewhere we know it already exists.
• Cloning without actually cloning. We all get read-only versions of the same data, and I only need to run the pieces I'm tinkering with.

What's good about defer?

• I like the semantic affinity between defer (deference) and ref (reference). Any closer and it'd be really confusing (imagine calling this --refer)
• "defer to" is really the sense we're after: I don't have all the tools for a full answer, so I'll defer to the experts (=production), defer to tradition (=past run). We could do a simple, more explicit rename: --defer-to-state
• One of my favorite high school English teachers used the phrase "a pleasure deferred" when describing, with gentle sarcasm, an unpleasant task that was too time-consuming for the current moment. ("Ah, yes, I was going to have Jeremy restate the major points of the Grand Inquisitor, but with scarce minutes remaining, I suppose it will have to be a pleasure deferred.") I think of Mr. Ward often, as I breeze past long-running parent models in my dev runs.

What's bad about defer?

• kind of a filler word, could mean anything
• carries a temporal aspect ("do it later"), rather than a spatial one ("do it over here instead")

Alternative metaphors:

• backup, safety net, entrust, trust fall
• shortcut, jump ahead, starting over from a save point in a platformer level
• look upstream, switch upstream
• borrow, mooch off of
• pinch hitting (player's out? put me in coach)
• in loco parentis (literally true)
• pulling off a tablecloth without stuff breaking

Any of those do anything for anyone?

Developer experience

How can we make it clear to users which models/resources have been deferred, and which haven't?

@jtcohen6: We currently have a debug log that lists the number of resources being deferred, and a sample of up to 5 (though the current wording is Merged {x} items from state). We could log that to stdout instead. We could also take it one step further, and determine which deferred resources are actually relevant to (upstream of) the models or tests being run. That would take some extra work, but it feels worthwhile:

$ dbt run -m model_a model_b
Running with dbt=0.19.0
Found 7 models, 4 tests, 1 snapshot, 0 analyses, 138 macros, 0 operations, 4 seed files, 1 source

DEFERRED 1 upstream relation: analytics.model_a

21:52:09 | Concurrency: 1 threads (target='dev')
21:52:09 |
21:52:09 | 1 of 1 START view model dbt_jcohen.model_b........................... [RUN]
21:52:09 | 1 of 1 OK created view model dbt_jcohen.model_b...................... [CREATE VIEW in 0.06s]
21:52:09 |
21:52:09 | Finished running 1 view model in 0.32s.

@drewbanin: I'm almost picturing that those nodes show up in the stdout logs as though we were running them, but they have a status like DEFERRED or UPSTREAM:

$ dbt run -m model_a model_b
Running with dbt=0.19.0
Found 7 models, 4 tests, 1 snapshot, 0 analyses, 138 macros, 0 operations, 4 seed files, 1 source

21:52:09 | Concurrency: 1 threads (target='dev')
21:52:09 |
21:52:09 | 1 of 2 SKIP relation dbt_jcohen.model_a.............................. [DEFERRED]
21:52:09 | 2 of 2 START view model dbt_jcohen.model_b........................... [RUN]
21:52:09 | 2 of 2 OK created view model dbt_jcohen.model_b...................... [CREATE VIEW in 0.06s]
21:52:09 |
21:52:09 | Finished running 1 view model in 0.32s.

We do store deferred as a node attribute in the manifest. Is there value in surfacing that information in the docs site? It's already sort of there, implicit in each model's compiled SQL, since deferred nodes will have rendered their references into a different namespace.

[gshank]

I like putting it in the standard out logs. None of the other terms proposed for it grabbed me. Most of them felt slightly more awkward and not more intuitively obvious. But I'll think about it.

[jtcohen6]

UX

• Printing to stdout feels like a good thing
• Not a breaking change, we can (and should!) do this any time

Naming

After some internal conversation:

• While we don't love --defer, there isn't a precisely better name for the highly specialized one-off thing it does today
• The better version of this feature requires both a name change and a functional expansion. The current behavior should be just one option among several.

Imagine, for instance (thanks @drewbanin):

dbt run --unbuilt-upstream=<MODE>, where <MODE> is one of:

• rewrite_refs (current implementation)
• clone (Snowflake/BQ only)
• pointer_views (cheap-o clones)

Nothing final about --unbuilt-upstream as a name, though I agree that it does the right thing by optimizing for clarity.

IMO we should change the name at the same time we expand functionality. We can do that with backwards compatibility—i.e. --defer would map to --unbuilt-upstream=rewrite_refs, DBT_DEFER_TO_STATE to DBT_UNBUILT_UPSTREAM=rewrite_refs)—so it wouldn't be a breaking change.

tl;dr

All of the above is worthwhile, but none of it needs to happen before v0.19.0-rc1, so I'm going to remove this issue from the Kiyoshi Kuromiya milestone.

[clausherther]

Just my 2 cents on naming: I'm working on an in-house dbt deployment tool that basically wraps dbt run and dbt test with a bunch of options to make build and deployment workflows a bit easier. I'm wrapping the idea of using a different upstream source simply in an --upstream-source or -u parameter. You just supply the name of the target you want to use as your upstream source and we handle generation of manifests if needed and --defer and --state syntax for you.

[panasenco]

I like "defer" just fine personally.

[ucg8j]

Hello - this is a great issue thread, I love the engagement with the community on opinions re naming. Also, @jtcohen6 "a pleasure deferred" got a chuckle from me - also a helpful reminder I need to read more Dostoevsky - however I keep putting this off, a pleasure deferred!

On naming
On our go implementation of the dbt cli, we have the following:

--upstream=prod or -u prod

We also built an in-house version of this to extend the dbt cli prior to the defer feature existing. Referred to in this blog. We invoke a model run, for example, we can read from ‘prod’ data, by using dbt monzo upstream prod -m modelA. ‘prod’ can be any target, so you could point it to another developers dev area if you were collaborating on different PRs. Theo, who built this a few years back, posted on the dbt community page more details on how it works.

Here's the CLI print out from our upstream

👷‍♀️  These models will be run in your dev dataset:

    -  modelC

🔄  These models will be substituted with data from the target 'prod

    -  modelA
    -  modelB

In short - my vote would be for –upstream with a nice short -u option too.

On behaviour
I wonder whether this should be the default behaviour? Or at least configurable to be the default behaviour at the target level.

When I was owning pipelines, I found myself most of the time using upstream=prod. And
at a previous data platform company I worked at, the default behaviour was to read from production data as data developers in most cases want to know what their pipeline modifications will look like production.

[ran-eh]

This may be a bit of a leap, but bare with me. I keep hoping for dbt to adopt the Make metaphor, and this may be the opportunity to start on that path. I expect this was considered in the past, but let's revisit this pandora's box.

Forget defer for a moment: imagine a dev cycle where you change the source file for a bunch of models/tests/etc, and then dbt make identifies and runs only the models/tests you changed, and possibly their downstream dependents. Rinse, repeat, until you are ready to commit your feature.

Make opens the door to a tone of useful features: you configure a model to be considered out-of-date if it has not run for 12 hours, and run make only runs out-of-date models and their dependents. Now instead of having a daily job and an hourly job and a job that runs every other Wednesday at 3:22 AM, you configure the models themselves for how frequently they need to be updated, and a single dbt make job takes care of everything for you.

In fact, if you properly define out-of-dateness for sources, you may never need to run a model that is actually current, just because it happens to be an upstream dependency of the model you need.

Defer is a natural extension of this: you define your baseline and development profiles in profiles.yml and do

dbt make --baseline-profile=my-prod --target-profile=my-test

This obviously brings up a ton of questions about how to maintain state, but we may be at the right time to break the taboo on out-of-database state, and go into the weeds on this one. Once this is figured out you have a much more natural way to think about data dependencies.

A difficulty in naming a feature sometimes signals that a rethink of underlying conceptual frameworks is due, and this may be the case here. Thoughts? @drewbanin ?

P.S. under the current framework, I add my vote for --upstream replacing --defer.

[jtcohen6]

Copying from dbt-labs/dbt-snowflake#315

@jtcohen6

IMO the best answer here looks:

• teaching dbt about clones, as a materialization/build strategy (especially now that BigQuery supports them, too: [CT-1044] Leverage BigQuery table clones for a Low Cost CI environment dbt-bigquery#270)
• an "alternative deferral mode" in CI (Review UX for --defer #5095)

Let's imagine a CI environment, and a form of dbt run --select state:modified --defer --state ..., where dbt identifies that 1 model out of 500 has changed, and 9 other models are downstream of the changed model. Currently, dbt will select only those 10 models to run, and implicitly rewrite the ref() within those models to select from production versions of the unchanged upstream models (since they won't exist in your CI schema). Instead, what if the first step were for dbt to "run" those 490 unchanged models — by cloning each and every one of them, in an independent thread, with parallelism up to your configured number of --threads?

To be clear, this would require a change within dbt-core. But I think it is the right way to solve for this use case, in keeping with the existing constructs and (good) constraints that we have in place for dbt today.

@jaysobel

I think what I'm hearing is dbt clone -s @state:modified? Which sounds 🔥. Would be faster on account of threads, and on account of the more restrictive selection (can even --exclude state:modified if first step is a full refresh of state:modified. Hopefully that would realize about the same gains as the async one... Not sure what the path forward would be if it was only 50% faster, vs 95%.

@jtcohen6

I think what I'm hearing is dbt clone -s @state:modified? Which sounds 🔥.

Yes, this is what I'm thinking! Sorry for not being clearer in the leadup; this doesn't yet exist today.

You could picture it as something like:

$ dbt clone -s -exclude state:modified+ --threads 10
$ dbt build -s state:modified+

Except, what if dbt did the first step for you implicitly? And what you actually run is a more configurable version of "deferral," where rewriting refs for unbuilt upstream models is one option, and cloning is another:

$ dbt build -s state:modified+ --unbuilt-upstream=clone

Hopefully that would realize about the same gains as the async one... Not sure what the path forward would be if it was only 50% faster, vs 95%.

Fair - we'd want to do some testing / benchmarking to confirm.

@jaysobel

Both of these seem like great options.

The --unbuilt-upstream sounds a little more novel, since I think dbt throws a SQL exception when an upstream doesn't exist. But maybe you're suggesting that under the hood it's doing a clone for you, and not actually DAG'ing in the clones as-needed?