refactor(rust): Parallel linearize in new streaming engine

[orlp]

No description provided.

[codecov]

Codecov Report

Attention: Patch coverage is 2.60870% with 112 lines in your changes missing coverage. Please review.

Project coverage is 80.86%. Comparing base (8a410ca) to head (0d612cc).
Report is 67 commits behind head on main.

❗ Current head 0d612cc differs from pull request most recent head c30937a

Please upload reports for the commit c30937a to get more accurate results.

Files	Patch %	Lines
...tes/polars-stream/src/utils/in_memory_linearize.rs	0.00%	85 Missing ⚠️
crates/polars-utils/src/priority.rs	0.00%	9 Missing ⚠️
crates/polars-stream/src/morsel.rs	0.00%	6 Missing ⚠️
crates/polars-stream/src/nodes/filter.rs	0.00%	6 Missing ⚠️
crates/polars-stream/src/nodes/in_memory_sink.rs	0.00%	6 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #17050      +/-   ##
==========================================
- Coverage   80.95%   80.86%   -0.10%     
==========================================
  Files        1448     1457       +9     
  Lines      190554   191042     +488     
  Branches     2723     2723              
==========================================
+ Hits       154265   154487     +222     
- Misses      35786    36051     +265     
- Partials      503      504       +1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[ritchie46]

Can we have a comment why we have increments of 2?

[orlp]

That's what the comment on new is for, the idea is to indicate the 'last morsel with this sequence id' using the final bit, so the sequence 1, 2, 3, 3, 3, 4 can become 2, 4, 6, 6, 7, 8 and as soon as we see 7 we know there won't be any 6s or 7s anymore.