GH-41263: [C#][Integration] Ensure offset is considered in all branches of the bitmap comparison

[paleolimbot]

Rationale for this change

The optimization for validity buffers was still failing after #41259 (sorry!).

What changes are included in this PR?

There were still two problems:

• The offset of the actual array was not considered in the "optimized" branch
• When this offset was considered, it became clear that zero-length arrays were not going to work in that branch

Are these changes tested?

I added the integration workflow to also run for C# additions. This might be a heavy CI job and I'm not sure if you want to keep it there (but running it is useful for this PR to ensure I actually fix things).

For future me (or maybe future others), the integration tests are pretty easy to check:

dotnet build
archery integration --run-c-data --with-csharp=true

Are there any user-facing changes?

No.

• GitHub Issue: [C#][Integration] Integration CI job is still failing for C# consumers #41263

[github-actions]

⚠️ GitHub issue #41263 has been automatically assigned in GitHub to PR creator.

[paleolimbot]

I'm not sure what's going on with the zerolength case, but it seems to be failing for Java and C# producing (with C# consuming):

################# FAILURES #################
FAILED TEST: primitive_zerolength Java producing,  C# consuming
<class 'subprocess.CalledProcessError'>: Command '/arrow/csharp/artifacts/Apache.Arrow.IntegrationTest/Debug/net7.0/Apache.Arrow.IntegrationTest --mode stream-to-file -a /tmp/tmpaibchz2w/28737e0c_generated_primitive_zerolength.consumer_stream_as_file < /tmp/tmpaibchz2w/28737e0c_generated_primitive_zerolength.producer_file_as_stream' returned non-zero exit status 1.

FAILED TEST: union Java producing,  C# consuming
<class 'subprocess.CalledProcessError'>: Command '/arrow/csharp/artifacts/Apache.Arrow.IntegrationTest/Debug/net7.0/Apache.Arrow.IntegrationTest --mode stream-to-file -a /tmp/tmpaibchz2w/3795be39_generated_union.consumer_stream_as_file < /tmp/tmpaibchz2w/3795be39_generated_union.producer_file_as_stream' returned non-zero exit status 1.

FAILED TEST: primitive_zerolength C# producing,  C# consuming
<class 'subprocess.CalledProcessError'>: Command '/arrow/csharp/artifacts/Apache.Arrow.IntegrationTest/Debug/net7.0/Apache.Arrow.IntegrationTest --mode stream-to-file -a /tmp/tmpaibchz2w/fd9d0ef3_0.14.1_primitive_zerolength.gold.consumer_stream_as_file < /arrow/testing/data/arrow-ipc-stream/integration/0.14.1/generated_primitive_zerolength.stream' returned non-zero exit status 1.

[adamreeve]

Hi, sorry for breaking this! I didn't think to also run the integration tests.

The Java tests are unrelated to the validity buffer validation but are throwing an error when writing zero length binary arrays to IPC format, due to my more recent changes in #41230. When we create a zero-length binary or list array in C# we get an offsets buffer with length 1, but the arrays read from Java have zero-length offsets.

I've pushed a fix for this a branch in my fork: adamreeve@b2bfb74. Do you want to include that in this PR or should I make a separate one for that?

[paleolimbot]

Do you want to include that in this PR or should I make a separate one for that?

I think that you should do it and I'll rebase! I am not all that confident in my C# 😬

[adamreeve]

I've opened #41303 to fix the issue writing empty binary arrays. With that change plus the ones here, the integration tests all pass for me when running with C#, Java and C++.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 5 benchmarking runs that have been run so far on merge-commit f8ef09a.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details. It also includes information about 43 possible false positives for unstable benchmarks that are known to sometimes produce them.