GH-41225: [C#] Slice value buffers when writing sliced list or binary arrays in IPC format

[adamreeve]

Rationale for this change

This reduces file sizes when writing sliced binary or list arrays to IPC format.

What changes are included in this PR?

Changes ArrowStreamWriter to write only the subset of the values that is needed rather than the full value buffer when writing a ListArray or BinaryArray, and compute shifted value offset buffers.

Are these changes tested?

This code is covered by existing tests and the change doesn't introduce any difference in the observed array values, so I haven't added new tests or checks.

I did change how list arrays are compared though as we can no longer compare the value and value offset buffers directly, so the tests now get list items as arrays and create a new ArrayComparer to compare them. This meant that array offsets are no longer always zero, so I've changed the offset assertions to only be used in strict mode.

Are there any user-facing changes?

Yes, this might reduce IPC file sizes for users writing sliced data.

• GitHub Issue: [C#] Slice value buffers when writing slices of list arrays and binary arrays #41225

[github-actions]

⚠️ GitHub issue #41225 has been automatically assigned in GitHub to PR creator.

[adamreeve]

This change was needed because otherwise tests that create zero-length arrays would fail when doing a strict comparison, as the values array would be sliced so would not exactly match the input. It seems logical to only add an extra item to the last list if there is actually a list to add to.

The similar change to the ListViewType above wasn't strictly needed but was just made for consistency.

The ListType data generation also had the same behaviour before but I've made that generate more complex lists with nulls and a range of list sizes including empty lists, for more confidence that the code is correct.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 5 benchmarking runs that have been run so far on merge-commit 56186b9.

There was 1 benchmark result indicating a performance regression:

• Commit Run on arm64-m6g-linux-compute at 2024-04-17 00:13:57Z
  • dataset-read (Python) with async=True, dataset=nyctaxi_multi_parquet_s3, pre_buffer=true

The full Conbench report has more details. It also includes information about 1 possible false positive for unstable benchmarks that are known to sometimes produce them.