✨ Source Instagram: Migrate Instagram sources Media, User, UserLiftimelnsights and Stories to low-code

[aldogonzalez8]

What

We want to migrate certified connectors to low-code; in this case, it is Instagram Connector. We are initially migrating the following streams:

• Media
• Stories
• Users
• UserLifetimeInsights

Follow up in the next PR:

• StortInsights
• MediaInsights

Important: The original implementation used an api object to slice by user business accounts on all the Streams. To achieve this functionality I have created a new Api source that gets this information as pass it to other streams as a parent so they will slice by each account

How

I have followed this guide.

TLDR is the following

• Fill this template with information about the connector.
• Model the streams through connector builder.
• If some feature is missing, add a custom component.
• Update base code:
  1. Put the manifest
  2. update source file
  3. add testing around custom components.
  4. Run CAT

Review guide

1. Manifest.yml: The manifest now includes the streams and schemas, so schemas for the scope of this PR were deleted.
2. Source.py: Source was updated to YamlDeclarativeSource, and there is an override on streams to include non-migrated streams
3. components.py: custom components created to handle:
   a. Fetch data for children field of Media; children are an array of media IDs that require their own data, so we go through the list to update the original children's object.
   b. Clear URL params "_nc_rid" and "ccb"
   c. convert breakdown list to an object
4. tests_components.py: test for the new components
5. streams.py: removed the old Python streams in the scope of this PR and updated parent classes of left streams.

User Impact

Can this PR be safely reverted and rolled back?

• YES 💚
• NO ❌

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Ignored Deployment

Name	Status	Preview	Comments	Updated (UTC)
airbyte-docs	⬜️ Ignored (Inspect)	Visit Preview		Jun 24, 2024 6:03am

[aldogonzalez8]

Acceptance tests results

[natikgadzhi]

@maxi297 @bazarnov what's your take on version bump when we're completely rewriting the connector? Should this be 4.0.0? I get that it's compatible, buuuuut.

[aldogonzalez8]

I've run live tests on a few connections without a problem. I don't know if this implies that my changes wouldn't need to be breaking changes. The new thing is the Api parent Stream (description), which passes the buz accounts to (now) child Streams. Anyway, I will hear experience call on this matter.

[maxi297]

If the state, config and record outputs all have the same format and it does not add relatively big features, I would consider this a patch

[bazarnov]

Once there are no regressions in terms of a breaking change, there is no need to make a major update, but the patch. @aldogonzalez8 FYI

[natikgadzhi]

@btkcodedev see? When you make a connector in Builder, it's supposed to use the new CDK ;)

[btkcodedev]

Sure! Will prune Docker volumes and fetch new for next one

[aldogonzalez8]

For live tests, here you can find an analysis I did of the failing test test_catalog_are_the_same

https://www.loom.com/share/3d994c40315c4dc78cfb6f2f904da1d1
catalog_diff_iterable_item_added.json
catalog_diff_iterable_item_removed.json

Cc @maxi297 in case you are reviewing and saw those results

[aldogonzalez8]

Today's run from live tests looks good, sharing some screenshots.

There are a few diff of data related to the cdn servers which change on every request and live data like impressions; this is expected.

[bazarnov]

LGTM! Nice!

[aldogonzalez8]

This will be merged on June 24th, early in the Morning.