GA4 connector with many properties

[Talalqaddoumi]

We're currently facing a significant challenge with the integration and reading of GA4 data. Our setup involves multiple Google accounts, and for each account, we may have more than 200 properties. The issue arises because the GA4 connector generates a stream per table per property, potentially resulting in over 3000 streams per source.

This situation is causing complications as it appears there are limitations on the maximum number of streams per connection. Consequently, we are unable to create a single source that encompasses 200+ GA4 properties IDs. Our alternative approach of creating a source per account per property ID could also pose challenges, potentially leading to the management of hundreds of connections and sources. Moreover, there might be limitations associated with this strategy.

We are reaching out to seek guidance and explore potential solutions to overcome this limitation. Do you have any recommendations or alternative options that could help us efficiently manage GA4 data integration in scenarios with a high number of properties?

[sajarin]

I understand your concern about managing a large number of GA4 properties with Airbyte. Currently, the Google Analytics Data API Source in Airbyte supports multiple input property IDs, but it produces one stream per property. This means if you have 200 properties, you will end up with 200 streams, which can be overwhelming to manage

One solution that has been proposed is to extend stream schemas with an additional column with a property ID as a value. This would allow a constant number of streams independent of the number of properties

In the meantime, if you are syncing more than one property, any property after the first will have the property ID appended to the stream name. This is to prevent the duplication of stream names. If you add an additional property ID in the future, all new streams will append the property ID to the stream name without affecting existing streams

Please note that if you are syncing more than one property ID, you will need to reset those streams to ensure syncing continues accurately. Also, please ensure that the 'Google Analytics Data API' is enabled in the service account for GA4 connections

I hope this information helps. We understand the need for a more efficient way to manage multiple GA4 properties and we are working on improving this.

[jwright12]

+1