Postgres destination accepts only 10k rows from a table containing json

[D1111111]

Howdy ✋

I have a Postgres -> Postgres connection that fails to load more than 10k rows. It is stuck in some kind of loop. After several hours or days of loading, I need to cancel it manually. Job history shows a larger number of rows extracted (around 30k), but only 10k inserted.

The problem is a json column inside the source table, which is not too big, let's say a couple of kB per row. When I remove that json column from the pipeline rows are loaded as expected.

I tried all versions of Postgres destination connector - same problem.

This is the example of the log, these lines are repeated over and over...

Namespace: raw Stream: prs__public__xxx -- remaining records: **68020**

97475

2024-11-07 19:03:56 destination > INFO main i.a.c.i.d.a.FlushWorkers(close):227 Waiting for all streams to flush.

97476

2024-11-07 19:03:56 destination > INFO pool-5-thread-1 i.a.c.i.d.a.DetectStreamToFlush(getNextStreamToFlush):109 flushing: trigger info: raw - prs__public__xxx, time trigger: false , size trigger: true current threshold b: 0 bytes, queue size b: 240.43 MB, penalty b: 0 bytes, after penalty b: 240.43 MB

97477

2024-11-07 19:03:56 destination > INFO pool-5-thread-1 i.a.c.i.d.a.DetectStreamToFlush(getNextStreamToFlush):109 flushing: trigger info: raw - prs__public__xxx, time trigger: false , size trigger: true current threshold b: 0 bytes, queue size b: 240.43 MB, penalty b: 25 MB, after penalty b: 215.43 MB

97478

2024-11-07 19:03:56 destination > INFO pool-4-thread-1 i.a.c.i.d.a.FlushWorkers(flush$lambda$6):135 Flush Worker (acbe7) -- Worker picked up work.

97479

2024-11-07 19:03:56 destination > INFO pool-4-thread-1 i.a.c.i.d.a.FlushWorkers(flush$lambda$6):141 Flush Worker (acbe7) -- Attempting to read from queue namespace: raw, stream: prs__public__xxx.

97480

2024-11-07 19:03:56 destination > INFO pool-4-thread-2 i.a.c.i.d.a.FlushWorkers(flush$lambda$6):135 Flush Worker (54150) -- Worker picked up work.

97481

2024-11-07 19:03:56 destination > INFO pool-4-thread-1 i.a.c.i.d.a.FlushWorkers(flush$lambda$6):159 Flush Worker (acbe7) -- Batch contains: 0 records, 0 bytes bytes.

97482

2024-11-07 19:03:56 destination > INFO pool-4-thread-2 i.a.c.i.d.a.FlushWorkers(flush$lambda$6):141 Flush Worker (54150) -- Attempting to read from queue namespace: raw, stream: prs__public__xxx.

97483

2024-11-07 19:03:56 destination > INFO pool-4-thread-1 i.a.c.i.d.a.s.GlobalAsyncStateManager(flushStates):153 Flushing states

97484

2024-11-07 19:03:56 destination > INFO pool-4-thread-2 i.a.c.i.d.a.FlushWorkers(flush$lambda$6):159 Flush Worker (54150) -- Batch contains: 0 records, 0 bytes bytes.

97485

2024-11-07 19:03:56 destination > INFO pool-4-thread-1 i.a.c.i.d.a.s.GlobalAsyncStateManager(flushStates):207 Flushing states complete

97486

2024-11-07 19:03:56 destination > INFO pool-4-thread-2 i.a.c.i.d.a.s.GlobalAsyncStateManager(flushStates):153 Flushing states

97487

2024-11-07 19:03:56 destination > INFO pool-4-thread-1 i.a.c.i.d.a.GlobalMemoryManager(free):78 Freeing 0 bytes..

97488

2024-11-07 19:03:56 destination > INFO pool-4-thread-2 i.a.c.i.d.a.s.GlobalAsyncStateManager(flushStates):207 Flushing states complete

97489

2024-11-07 19:03:56 destination > INFO pool-4-thread-1 i.a.c.i.d.a.GlobalMemoryManager(free):78 Freeing 0 bytes..

97490

2024-11-07 19:03:56 destination > INFO pool-4-thread-2 i.a.c.i.d.a.GlobalMemoryManager(free):78 Freeing 0 bytes..

97491

2024-11-07 19:03:56 destination > INFO pool-4-thread-1 i.a.c.i.d.a.FlushWorkers(flush$lambda$6):170 Flush Worker (acbe7) -- Worker finished flushing. Current queue size: 68020

97492

2024-11-07 19:03:56 destination > INFO pool-4-thread-2 i.a.c.i.d.a.GlobalMemoryManager(free):78 Freeing 0 bytes..

97493

2024-11-07 19:03:56 destination > INFO pool-4-thread-2 i.a.c.i.d.a.FlushWorkers(flush$lambda$6):170 Flush Worker (54150) -- Worker finished flushing. Current queue size: 68020

97494

2024-11-07 19:03:57 destination > INFO main i.a.c.i.d.a.FlushWorkers(close):226 REMAINING_BUFFERS_INFO

97495

  Namespace: raw Stream: prs__public_xxx -- remaining records: 68020

97496

2024-11-07 19:03:57 destination > INFO main i.a.c.i.d.a.FlushWorkers(close):227 Waiting for all streams to flush.

97497

2024-11-07 19:03:57 destination > INFO pool-5-thread-1 i.a.c.i.d.a.DetectStreamToFlush(getNextStreamToFlush):109 flushing: trigger info: raw - prs__public__xxx, time trigger: false , size trigger: true current threshold b: 0 bytes, queue size b: 240.43 MB, penalty b: 0 bytes, after penalty b: 240.43 MB

97498

2024-11-07 19:03:57 destination > INFO pool-5-thread-1 i.a.c.i.d.a.DetectStreamToFlush(getNextStreamToFlush):109 flushing: trigger info: raw - prs__public__xxx, time trigger: false , size trigger: true current threshold b: 0 bytes, queue size b: 240.43 MB, penalty b: 25 MB, after penalty b: 215.43 MB

97499

2024-11-07 19:03:57 destination > INFO pool-4-thread-1 i.a.c.i.d.a.FlushWorkers(flush$lambda$6):135 Flush Worker (f8a3c) -- Worker picked up work.

97500

2024-11-07 19:03:57 destination > INFO pool-4-thread-1 i.a.c.i.d.a.FlushWorkers(flush$lambda$6):141 Flush Worker (f8a3c) -- Attempting to read from queue namespace: raw, stream: prs__public__xxx.

97501

2024-11-07 19:03:57 destination > INFO pool-4-thread-2 i.a.c.i.d.a.FlushWorkers(flush$lambda$6):135 Flush Worker (dbe81) -- Worker picked up work.

97502

2024-11-07 19:03:57 destination > INFO pool-4-thread-1 i.a.c.i.d.a.FlushWorkers(flush$lambda$6):159 Flush Worker (f8a3c) -- Batch contains: 0 records, 0 bytes bytes.

97503

2024-11-07 19:03:57 destination > INFO pool-4-thread-2 i.a.c.i.d.a.FlushWorkers(flush$lambda$6):141 Flush Worker (dbe81) -- Attempting to read from queue namespace: raw, stream: prs__public__xxx.

97504

2024-11-07 19:03:57 destination > INFO pool-4-thread-1 i.a.c.i.d.a.s.GlobalAsyncStateManager(flushStates):153 Flushing states

97505

2024-11-07 19:03:57 destination > INFO pool-4-thread-2 i.a.c.i.d.a.FlushWorkers(flush$lambda$6):159 Flush Worker (dbe81) -- Batch contains: 0 records, 0 bytes bytes.

97506

2024-11-07 19:03:57 destination > INFO pool-4-thread-1 i.a.c.i.d.a.s.GlobalAsyncStateManager(flushStates):207 Flushing states complete

97507

2024-11-07 19:03:57 destination > INFO pool-4-thread-2 i.a.c.i.d.a.s.GlobalAsyncStateManager(flushStates):153 Flushing states

97508

2024-11-07 19:03:57 destination > INFO pool-4-thread-1 i.a.c.i.d.a.GlobalMemoryManager(free):78 Freeing 0 bytes..

97509

2024-11-07 19:03:57 destination > INFO pool-4-thread-2 i.a.c.i.d.a.s.GlobalAsyncStateManager(flushStates):207 Flushing states complete

97510

2024-11-07 19:03:57 destination > INFO pool-4-thread-1 i.a.c.i.d.a.GlobalMemoryManager(free):78 Freeing 0 bytes..

97511

2024-11-07 19:03:57 destination > INFO pool-4-thread-2 i.a.c.i.d.a.GlobalMemoryManager(free):78 Freeing 0 bytes..

97512

2024-11-07 19:03:57 destination > INFO pool-4-thread-1 i.a.c.i.d.a.FlushWorkers(flush$lambda$6):170 Flush Worker (f8a3c) -- Worker finished flushing. Current queue size: 68020

97513

2024-11-07 19:03:57 destination > INFO pool-4-thread-2 i.a.c.i.d.a.GlobalMemoryManager(free):78 Freeing 0 bytes..

97514

2024-11-07 19:03:57 destination > INFO pool-4-thread-2 i.a.c.i.d.a.FlushWorkers(flush$lambda$6):170 Flush Worker (dbe81) -- Worker finished flushing. Current queue size: 68020

97515

2024-11-07 19:03:58 destination > INFO main i.a.c.i.d.a.FlushWorkers(close):226 REMAINING_BUFFERS_INFO

97516

  Namespace: raw Stream: prs__public__xxx -- remaining records: **68020**

97517