[FEAT] Monotonically Increasing Id for Swordfish

[colin-ho]

Implements monotonically increasing id as a streaming sink with max_concurrency = 1.

I tested multithreaded and single threaded implementations and found that there was no performance gain in multithreaded. This is because monotonically increasing id is a memory bound operator, all it does is allocate an array of ints for the id. Multiple threads trying to do this in parallel are bottlenecked by memory bandwidth.

It is actually also much simpler to implement this as a single threaded operation, as we just need to keep a running count of the lengths of morsels seen so far. This is effectively just row_number.

Note:

• There was a bug in the pyfunc_into_table_iter function, which consumes python iterators in scan tasks (used in read_lance and read_generator), where the consumer only calls next() on the iterator once. This PR fixes that.

[codspeed-hq]

CodSpeed Performance Report

Merging #3180 will improve performances by ×2.2

_{Comparing colin/swordfish-mono-id (509e645) with main (84db665)}

Summary

⚡ 2 improvements
✅ 15 untouched benchmarks

Benchmarks breakdown

	Benchmark	main	colin/swordfish-mono-id	Change
⚡	test_iter_rows_first_row[100 Small Files]	421.4 ms	375 ms	+12.36%
⚡	test_show[100 Small Files]	32.7 ms	14.9 ms	×2.2

[codecov]

Codecov Report

Attention: Patch coverage is 84.84848% with 20 lines in your changes missing coverage. Please review.

Project coverage is 74.96%. Comparing base (84db665) to head (509e645).

Files with missing lines	Patch %	Lines
...execution/src/sinks/monotonically_increasing_id.rs	77.01%	20 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3180      +/-   ##
==========================================
- Coverage   77.55%   74.96%   -2.60%     
==========================================
  Files         668      669       +1     
  Lines       82268    86911    +4643     
==========================================
+ Hits        63807    65151    +1344     
- Misses      18461    21760    +3299     

Files with missing lines	Coverage Δ
src/daft-local-execution/src/pipeline.rs	95.51% <100.00%> (+0.09%)	⬆️
src/daft-local-plan/src/plan.rs	96.72% <100.00%> (+0.14%)	⬆️
src/daft-local-plan/src/translate.rs	94.15% <100.00%> (+0.24%)	⬆️
src/daft-micropartition/src/python.rs	78.32% <100.00%> (+0.04%)	⬆️
...execution/src/sinks/monotonically_increasing_id.rs	41.10% <77.01%> (ø)

... and 39 files with indirect coverage changes