Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feat: MotherDuck cache support #43

Merged
merged 15 commits into from
Feb 23, 2024
Merged

Conversation

aaronsteers
Copy link
Contributor

@aaronsteers aaronsteers commented Feb 16, 2024

I put this together really quick but have not yet tested it.

This appears to be working well. Would appreciate additional testers and feedback.

I referenced the MotherDuck/LangChain docs here:

@aaronsteers
Copy link
Contributor Author

aaronsteers commented Feb 23, 2024

Interesting error if database name and schema name are both the same ('airbyte_test', below).

sqlalchemy.exc.ProgrammingError: (duckdb.duckdb.BinderException) Binder Error: Ambiguous reference to catalog or schema "airbyte_test" - use a fully qualified path like "airbyte_test.airbyte_test"
[SQL: 
CREATE TABLE airbyte_test._airbyte_streams (
        stream_name VARCHAR, 
        source_name VARCHAR, 
        table_name VARCHAR NOT NULL, 
        catalog_metadata VARCHAR, 
        PRIMARY KEY (table_name)
)

@aaronsteers
Copy link
Contributor Author

Looks like we have a new rendering bug in code unrelated to this PR. But on the plus side - it looks like the read was successful 🙌 :

AJ-Steers-MacBook-Pro---GGHWM7QWPJ:airbyte-lib-private-beta ajsteers$ poetry run python examples/run_faker_to_motherduck.py 
Connection check succeeded for `source-faker`.
Started `source-faker` read operation at 09:06:36...
<IPython.core.display.Markdown object>
<IPython.core.display.Markdown object>
<IPython.core.display.Markdown object>
<IPython.core.display.Markdown object>
<IPython.core.display.Markdown object>
<IPython.core.display.Markdown object>
<IPython.core.display.Markdown object>
<IPython.core.display.Markdown object>
<IPython.core.display.Markdown object>
<IPython.core.display.Markdown object>
<IPython.core.display.Markdown object>
<IPython.core.display.Markdown object>
<IPython.core.display.Markdown object>
<IPython.core.display.Markdown object>
<IPython.core.display.Markdown object>
<IPython.core.display.Markdown object>
Completed `source-faker` read operation at 09:06:48.
Stream products: 100 records
Stream users: 10000 records
Stream purchases: 10000 records

@aaronsteers aaronsteers marked this pull request as ready for review February 23, 2024 19:12
@aaronsteers aaronsteers changed the title (Draft) MotherDuck cache support Feat: MotherDuck cache support Feb 23, 2024
Copy link
Contributor

@bindipankhudi bindipankhudi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great! Much less cache code after the refactor you did.

@aaronsteers aaronsteers enabled auto-merge (squash) February 23, 2024 21:31
@aaronsteers aaronsteers merged commit 797e657 into main Feb 23, 2024
9 checks passed
@aaronsteers aaronsteers deleted the feat/motherduck-cache-support branch February 23, 2024 21:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants