Initial Load Timeout Issue for Large Tables (>24h) with Retry Configuration Not Working as Expected

[xurtasun]

Hello Airbyte Team,

I’ve been working with Airbyte for several months, synchronizing data from a database with over 2TB of data. Some of the tables are extremely large, which has caused me a few challenges along the way. I recently upgraded to Airbyte 1.0 in hopes of resolving some of these issues, but I am still encountering a major blocker.

The Problem:

• Initial Load Timeout: My sync process for large tables exceeds the 24-hour maximum “initial load timeout.” As far as I can tell, this timeout is not configurable to extend beyond 24 hours (saw your discussion here), which poses a significant limitation for handling large datasets like mine.
• Retry Behavior: I’ve attempted to configure retries for when the sync fails. My expectation was that the retry mechanism would resume the sync process from where it left off. However, when the sync reaches the timeout and fails, the retry starts the process all over again, instead of continuing from the last point of failure. This leads to a loop of incomplete syncs, which is unmanageable for such large tables.

What I’ve Tried:

• Setting up retries after failures, but the sync restarts from the beginning rather than resuming.
• Source db (PostgreSQL)
• Destination (Snowflake)

Questions:

1. Is there any plan or current workaround to allow configuration of the initial load timeout beyond 24 hours for very large datasets?
2. Is there a way to configure the retry mechanism to pick up where the sync left off, instead of restarting from scratch?
3. Are there any other recommended strategies for syncing large tables that take more than 24 hours, considering the current limitations?

Any guidance or feedback would be greatly appreciated. I’m happy to provide more technical details or logs if needed.

Thank you for your help, and I look forward to hearing back from the team!