Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Source Github: Add dbt converter #49

Open
wants to merge 5 commits into
base: main
Choose a base branch
from

Conversation

aazam-gh
Copy link

@aazam-gh aazam-gh commented Aug 10, 2024

Fixes #11

Fivertran's Streams were matched against Airbyte streams and differential tables were identified

@aazam-gh
Copy link
Author

aazam-gh commented Aug 10, 2024

Fivetran Airbyte
stg_github__issue.sql issues
stg_github__issue_assignee.sql Is a combination of the issue table and user table
stg_github__issue_closed_history.sql Does not exist
stg_github__issue_comment.sql Transformation from issues
stg_github__issue_label.sql Does not exist
stg_github__issue_merged.sql Does not exist
stg_github__label.sql Exists issue_labels
stg_github__pull_request.sql pull_requests
stg_github__pull_request_review.sql reviews
stg_github__repo_team.sql Need to add from new stream repo_team
stg_github__repository.sql Exists repositories
stg_github__requested_review_history.sql Does not exist
stg_github__team.sql Exists teams
stg_github__user.sql Need to add from the new stream auth_users

@marcosmarxm
Copy link
Member

@aazam-gh do you need any help?

@aazam-gh
Copy link
Author

@marcosmarxm Yes wanted to confirm the next step is finding the primary columns correct?

@marcosmarxm
Copy link
Member

No, you must create the transformation models doesn't exist in Airbyte.

@aazam-gh
Copy link
Author

I found that the fivetran github_issue equivalent, issues stream in Airbyte does not have a repository_id parameter.
How should I handle such cases?

@marcosmarxm
Copy link
Member

@aazam-gh there is a missing stream for Airbyte which must be implemented in the Github Source connector: https://docs.github.com/en/rest/teams/teams?apiVersion=2022-11-28#list-team-repositories this links the team and repository used by repo_team table.

@marcosmarxm
Copy link
Member

I found that the fivetran github_issue equivalent, issues stream in Airbyte does not have a repository_id parameter. How should I handle such cases?

You can join issues with repository table.

@aazam-gh
Copy link
Author

aazam-gh commented Sep 4, 2024

I found that the fivetran github_issue equivalent, issues stream in Airbyte does not have a repository_id parameter. How should I handle such cases?

You can join issues with repository table.

Can you elaborate on how that is to be done? I can similarly see a lot of tables have foreign key references

@aazam-gh
Copy link
Author

aazam-gh commented Sep 4, 2024

In the https://github.com/fivetran/dbt_github_source/blob/main/models/src_github.yml#L32
there is a parameter fivetran_synced how do we resolve that?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Create dbt converter for Airbyte Github source connector
2 participants