Feature Idea: PyAirbyte CLI

[aaronsteers]

Let's add a CLI for PyAirbyte...

Decision 1: Entrypoint Name

We could use pyairbyte or airbyte as the entrypoint (CLI) name.

While airbyte matches the library name, I think I slightly prefer pyairbyte or another CLI name, to be clear that users are invoking pyairbyte and more clearly distinguish from Airbyte Platform, Terraform, abctl, and any other Airbyte REST API wrapper.

I like pyairbyte with the more concise alias pyab providing the same functionality.

Decision 2: Verb selection

I think I lean towards pairing these to match the classic verbs: read, write, discover, etc. - except that we're going to be streamlining the invocation for people quite a lot.

Decision 3: Workload Descriptions: Yaml vs CLI args

We can save users some headache and make the CLI more concise by letting them use yaml files to describe jobs. Since our first use case for the CLI will likely match to what we already have for acceptance tests, I think we should make sure we at least support the acceptance-test-config.yaml format or the new inlining of this into metadata.yaml files.

I lean towards the following implementation:

• Let users provide a workload name, which loosely correlates to the same name used in a config file, subtracting /secrets/ path prefix and the .json suffix.
• If no workload name is provided, we'll default to the first definition with the default config file name: config.json.
• If more than one workload is defined, and no workload name is provided, and none are named config.json, then we'll fail and ask for a workload name.
• We'll also let users override specific inputs by providing them via the command line.
• CLI args take precedence over yaml config. Neither is required if the other is fully declarative.

Examples

Run the first 'full_refresh' workload defined in acceptance-test-config.yaml

pyab run --source=source-faker --source-job="acceptance-test-config.yaml:tests.full_refresh[0]" --destination=destination-snowflake --destination-config=../destination-snowflake/secrets/config.json

Run a performance benchmark using the same workload info.

pyab benchmark --source=source-faker --job="acceptance-test-config.yaml:tests.full_refresh[0]"

[aaronsteers]

Resolved (first iteration) in:

• Feat: Add sync CLI #417