GH-42005: [Java][Integration][CI] Fix ARROW_BUILD_ROOT Path to find pom.xml

[llama90]

Rationale for this change

This PR aims to fix the issue where the integration tests are failing due to the missing /java/pom.xml file. It appears that the current code incorrectly determines the path to ARROW_BUILD_ROOT, leading to the failure in locating the pom.xml file.

What changes are included in this PR?

• Updating the ARROW_BUILD_ROOT path determination logic in tester_java.py to correctly reference the project root.

Are these changes tested?

Maybe, Yes.

Are there any user-facing changes?

No.

• GitHub Issue: [Java][Integration][CI] Integration tests are failing due to missing /java/pom.xml #42005

[github-actions]

⚠️ GitHub issue #42005 has been automatically assigned in GitHub to PR creator.

[llama90]

I verified that the path to find the java directory needs to be set to 4 levels up. If this resolves the issue, we may also need to update the paths for js to ensure consistency.

# from /Users/lama/workspace/arrow-new/dev/archery/archery/utils
# Here is the path that was referenced before this PR is merged. 
ARROW_ROOT: /Users/lama/workspace/arrow-new
# from /Users/lama/workspace/arrow-new/dev/archery/archery/integration
ARROW_BUILD_ROOT: /Users/lama/workspace

[llama90]

@raulcd Could you please execute the command as you mentioned?

[raulcd]

@github-actions crossbow submit verify-rc-source-integration-linux-ubuntu-20.04-amd64

[github-actions]

Revision: 849707f

Submitted crossbow builds: ursacomputing/crossbow @ actions-0e1aa06ee4

Task	Status
verify-rc-source-integration-linux-ubuntu-20.04-amd64

[llama90]

@kou Could you please let me know if there might have been another intention in the PR you wrote?

From my review of the code, it seems that in this file, the path needs to go up 4 levels to correctly refer java directory.

[llama90]

It seems that the java tests have passed, but as expected, there is an error with the js tests. I will fix it too.

UPDATE: I attached the error message.

################# FAILURES #################
FAILED TEST: primitive_no_batches C++ producing,  JS consuming
<class 'FileNotFoundError'>: [Errno 2] No such file or directory: '/js'

FAILED TEST: primitive C++ producing,  JS consuming
<class 'FileNotFoundError'>: [Errno 2] No such file or directory: '/js'

FAILED TEST: primitive_zerolength C++ producing,  JS consuming
<class 'FileNotFoundError'>: [Errno 2] No such file or directory: '/js'

FAILED TEST: primitive_large_offsets C++ producing,  JS consuming
<class 'FileNotFoundError'>: [Errno 2] No such file or directory: '/js'

FAILED TEST: null C++ producing,  JS consuming
<class 'FileNotFoundError'>: [Errno 2] No such file or directory: '/js'

FAILED TEST: null_trivial C++ producing,  JS consuming
<class 'FileNotFoundError'>: [Errno 2] No such file or directory: '/js'

FAILED TEST: decimal C++ producing,  JS consuming
<class 'FileNotFoundError'>: [Errno 2] No such file or directory: '/js'

FAILED TEST: datetime C++ producing,  JS consuming
<class 'FileNotFoundError'>: [Errno 2] No such file or directory: '/js'

FAILED TEST: duration C++ producing,  JS consuming
<class 'FileNotFoundError'>: [Errno 2] No such file or directory: '/js'

FAILED TEST: map C++ producing,  JS consuming
<class 'FileNotFoundError'>: [Errno 2] No such file or directory: '/js'

FAILED TEST: map_non_canonical C++ producing,  JS consuming
<class 'FileNotFoundError'>: [Errno 2] No such file or directory: '/js'

FAILED TEST: nested C++ producing,  JS consuming
<class 'FileNotFoundError'>: [Errno 2] No such file or directory: '/js'

FAILED TEST: recursive_nested C++ producing,  JS consuming
<class 'FileNotFoundError'>: [Errno 2] No such file or directory: '/js'

FAILED TEST: union C++ producing,  JS consuming
<class 'FileNotFoundError'>: [Errno 2] No such file or directory: '/js'

FAILED TEST: custom_metadata C++ producing,  JS consuming
<class 'FileNotFoundError'>: [Errno 2] No such file or directory: '/js'

FAILED TEST: dictionary C++ producing,  JS consuming
<class 'FileNotFoundError'>: [Errno 2] No such file or directory: '/js'

FAILED TEST: dictionary_unsigned C++ producing,  JS consuming
<class 'FileNotFoundError'>: [Errno 2] No such file or directory: '/js'
FAILED TEST: nested JS producing,  Go consuming
<class 'FileNotFoundError'>: [Errno 2] No such file or directory: '/js'

FAILED TEST: recursive_nested JS producing,  Go consuming
<class 'FileNotFoundError'>: [Errno 2] No such file or directory: '/js'

FAILED TEST: union JS producing,  Go consuming
<class 'FileNotFoundError'>: [Errno 2] No such file or directory: '/js'

FAILED TEST: custom_metadata JS producing,  Go consuming
<class 'FileNotFoundError'>: [Errno 2] No such file or directory: '/js'

FAILED TEST: dictionary JS producing,  Go consuming
<class 'FileNotFoundError'>: [Errno 2] No such file or directory: '/js'

FAILED TEST: dictionary_unsigned JS producing,  Go consuming
<class 'FileNotFoundError'>: [Errno 2] No such file or directory: '/js'

FAILED TEST: nested_dictionary JS producing,  Go consuming
<class 'FileNotFoundError'>: [Errno 2] No such file or directory: '/js'

FAILED TEST: extension JS producing,  Go consuming
<class 'FileNotFoundError'>: [Errno 2] No such file or directory: '/js'

FAILED TEST: primitive_no_batches Go producing,  JS consuming
<class 'FileNotFoundError'>: [Errno 2] No such file or directory: '/js'

FAILED TEST: primitive Go producing,  JS consuming
<class 'FileNotFoundError'>: [Errno 2] No such file or directory: '/js'

FAILED TEST: primitive_zerolength Go producing,  JS consuming
<class 'FileNotFoundError'>: [Errno 2] No such file or directory: '/js'

FAILED TEST: primitive_large_offsets Go producing,  JS consuming
<class 'FileNotFoundError'>: [Errno 2] No such file or directory: '/js'

FAILED TEST: null Go producing,  JS consuming
<class 'FileNotFoundError'>: [Errno 2] No such file or directory: '/js'

FAILED TEST: null_trivial Go producing,  JS consuming
<class 'FileNotFoundError'>: [Errno 2] No such file or directory: '/js'

FAILED TEST: decimal Go producing,  JS consuming
<class 'FileNotFoundError'>: [Errno 2] No such file or directory: '/js'

FAILED TEST: datetime Go producing,  JS consuming
<class 'FileNotFoundError'>: [Errno 2] No such file or directory: '/js'

FAILED TEST: duration Go producing,  JS consuming
<class 'FileNotFoundError'>: [Errno 2] No such file or directory: '/js'

FAILED TEST: map Go producing,  JS consuming
<class 'FileNotFoundError'>: [Errno 2] No such file or directory: '/js'

FAILED TEST: map_non_canonical Go producing,  JS consuming
<class 'FileNotFoundError'>: [Errno 2] No such file or directory: '/js'

FAILED TEST: nested Go producing,  JS consuming
<class 'FileNotFoundError'>: [Errno 2] No such file or directory: '/js'

FAILED TEST: recursive_nested Go producing,  JS consuming
<class 'FileNotFoundError'>: [Errno 2] No such file or directory: '/js'

FAILED TEST: union Go producing,  JS consuming
<class 'FileNotFoundError'>: [Errno 2] No such file or directory: '/js'

FAILED TEST: custom_metadata Go producing,  JS consuming
<class 'FileNotFoundError'>: [Errno 2] No such file or directory: '/js'

FAILED TEST: dictionary Go producing,  JS consuming
<class 'FileNotFoundError'>: [Errno 2] No such file or directory: '/js'

FAILED TEST: dictionary_unsigned Go producing,  JS consuming
<class 'FileNotFoundError'>: [Errno 2] No such file or directory: '/js'

FAILED TEST: nested_dictionary Go producing,  JS consuming
<class 'FileNotFoundError'>: [Errno 2] No such file or directory: '/js'

FAILED TEST: extension Go producing,  JS consuming
<class 'FileNotFoundError'>: [Errno 2] No such file or directory: '/js'

127 failures, 200 skips
Failed to verify release candidate. See /tmp/arrow-HEAD.hhs4k for details.
Error: docker run --rm -e VERIFY_VERSION= -e VERIFY_RC= -e TEST_DEFAULT=0 -e TEST_INTEGRATION=1 --shm-size 1073741824 -e CCACHE_COMPILERCHECK=content -e CCACHE_COMPRESS=1 -e CCACHE_COMPRESSLEVEL=6 -e CCACHE_DIR=/ccache -e CCACHE_MAXSIZE=1G -e CMAKE_GENERATOR=Ninja -e GITHUB_ACTIONS=true -e TEST_APT=0 -e TEST_YUM=0 -v /home/runner/work/crossbow/crossbow/arrow:/arrow -v ubuntu-ccache:/ccache apache/arrow-dev:amd64-ubuntu-20.04-verify-rc /bin/bash -c   git config --global --add safe.directory /arrow &&
  /arrow/dev/release/verify-release-candidate.sh ${VERIFY_VERSION} ${VERIFY_RC} exited with non-zero exit code 1
Error: Process completed with exit code 1.

[llama90]

@raulcd Could you please command one more?

Also, I understand that PR was aimed at improving documentation. However, I'm not sure why it included updates to the integration paths for Java and JS.

If these integration paths are not related to improving documentation, I think the code needs to be rolled back.

• Is there any method to execute commands with archery as integration tests on macOS?

[raulcd]

@github-actions crossbow submit verify-rc-source-integration-linux-ubuntu-20.04-amd64

[github-actions]

Revision: e5a8205

Submitted crossbow builds: ursacomputing/crossbow @ actions-7777758971

Task	Status
verify-rc-source-integration-linux-ubuntu-20.04-amd64

[raulcd]

I haven't tested locally but the verification script just uses

  LD_LIBRARY_PATH=$ARROW_CPP_EXE_PATH:$LD_LIBRARY_PATH archery integration \
    --run-ipc --run-flight --run-c-data \
    --with-cpp=${TEST_INTEGRATION_CPP} \
    --with-java=${TEST_INTEGRATION_JAVA} \
    --with-js=${TEST_INTEGRATION_JS} \
    --with-go=${TEST_INTEGRATION_GO} \
    $INTEGRATION_TEST_ARGS

This is the relevant piece: https://github.com/apache/arrow/blob/main/dev/release/verify-release-candidate.sh#L986-L1013

[llama90]

Umm... the error message is changed. It seems that the js tests failed.

################# FAILURES #################
FAILED TEST: primitive_zerolength Go producing,  JS consuming
<class 'RuntimeError'>: Command failed: /arrow/js/bin/integration.ts -a /tmp/tmp5mhokyv3/dfcb2e35_generated_primitive_zerolength.consumer_stream_as_file -j /tmp/arrow-integration-ima0onkg/generated_primitive_zerolength.json --mode VALIDATE
With output:
--------------


#
# Fatal error in , line 0
# unreachable code
#
#
#
#FailureMessage Object: 0x7fdeb7fb6a10
----- Native stack trace -----

 1: 0xd38851  [node]
 2: 0x216e041 V8_Fatal(char const*, ...) [node]
 3: 0x1566cec int v8::internal::Deserializer<v8::internal::Isolate>::ReadSingleBytecodeData<v8::internal::SlotAccessorForHeapObject>(unsigned char, v8::internal::SlotAccessorForHeapObject) [node]
 4: 0x1567e91 v8::internal::Deserializer<v8::internal::Isolate>::ReadData(v8::internal::Handle<v8::internal::HeapObject>, int, int) [node]
 5: 0x1568158 v8::internal::Deserializer<v8::internal::Isolate>::ReadObject(v8::internal::SnapshotSpace) [node]
 6: 0x1566d5b int v8::internal::Deserializer<v8::internal::Isolate>::ReadSingleBytecodeData<v8::internal::SlotAccessorForHeapObject>(unsigned char, v8::internal::SlotAccessorForHeapObject) [node]
 7: 0x1567e91 v8::internal::Deserializer<v8::internal::Isolate>::ReadData(v8::internal::Handle<v8::internal::HeapObject>, int, int) [node]
 8: 0x1568158 v8::internal::Deserializer<v8::internal::Isolate>::ReadObject(v8::internal::SnapshotSpace) [node]
 9: 0x1566d5b int v8::internal::Deserializer<v8::internal::Isolate>::ReadSingleBytecodeData<v8::internal::SlotAccessorForHeapObject>(unsigned char, v8::internal::SlotAccessorForHeapObject) [node]
10: 0x1567e91 v8::internal::Deserializer<v8::internal::Isolate>::ReadData(v8::internal::Handle<v8::internal::HeapObject>, int, int) [node]
11: 0x1568158 v8::internal::Deserializer<v8::internal::Isolate>::ReadObject(v8::internal::SnapshotSpace) [node]
12: 0x1566d5b int v8::internal::Deserializer<v8::internal::Isolate>::ReadSingleBytecodeData<v8::internal::SlotAccessorForHeapObject>(unsigned char, v8::internal::SlotAccessorForHeapObject) [node]
13: 0x1567e91 v8::internal::Deserializer<v8::internal::Isolate>::ReadData(v8::internal::Handle<v8::internal::HeapObject>, int, int) [node]
14: 0x1568158 v8::internal::Deserializer<v8::internal::Isolate>::ReadObject(v8::internal::SnapshotSpace) [node]
15: 0x1566d5b int v8::internal::Deserializer<v8::internal::Isolate>::ReadSingleBytecodeData<v8::internal::SlotAccessorForHeapObject>(unsigned char, v8::internal::SlotAccessorForHeapObject) [node]
16: 0x1567e91 v8::internal::Deserializer<v8::internal::Isolate>::ReadData(v8::internal::Handle<v8::internal::HeapObject>, int, int) [node]
17: 0x1568158 v8::internal::Deserializer<v8::internal::Isolate>::ReadObject(v8::internal::SnapshotSpace) [node]
18: 0x1566d5b int v8::internal::Deserializer<v8::internal::Isolate>::ReadSingleBytecodeData<v8::internal::SlotAccessorForHeapObject>(unsigned char, v8::internal::SlotAccessorForHeapObject) [node]
19: 0x1567e91 v8::internal::Deserializer<v8::internal::Isolate>::ReadData(v8::internal::Handle<v8::internal::HeapObject>, int, int) [node]
20: 0x1568158 v8::internal::Deserializer<v8::internal::Isolate>::ReadObject(v8::internal::SnapshotSpace) [node]
21: 0x1566d5b int v8::internal::Deserializer<v8::internal::Isolate>::ReadSingleBytecodeData<v8::internal::SlotAccessorForHeapObject>(unsigned char, v8::internal::SlotAccessorForHeapObject) [node]
22: 0x1567e91 v8::internal::Deserializer<v8::internal::Isolate>::ReadData(v8::internal::Handle<v8::internal::HeapObject>, int, int) [node]
23: 0x1568158 v8::internal::Deserializer<v8::internal::Isolate>::ReadObject(v8::internal::SnapshotSpace) [node]
24: 0x1568439 int v8::internal::Deserializer<v8::internal::Isolate>::ReadSingleBytecodeData<v8::internal::SlotAccessorForHandle<v8::internal::Isolate> >(unsigned char, v8::internal::SlotAccessorForHandle<v8::internal::Isolate>) [node]
25: 0x1568d3f v8::internal::Deserializer<v8::internal::Isolate>::ReadObject() [node]
26: 0x156d8d2 v8::internal::ObjectDeserializer::Deserialize() [node]
27: 0x156dae4 v8::internal::ObjectDeserializer::DeserializeSharedFunctionInfo(v8::internal::Isolate*, v8::internal::SerializedCodeData const*, v8::internal::Handle<v8::internal::String>) [node]
28: 0x1560822 v8::internal::CodeSerializer::Deserialize(v8::internal::Isolate*, v8::internal::AlignedCachedData*, v8::internal::Handle<v8::internal::String>, v8::ScriptOriginOptions, v8::internal::MaybeHandle<v8::internal::Script>) [node]
29: 0xfb9ccb  [node]
30: 0xfb9fad v8::internal::Compiler::GetSharedFunctionInfoForScriptWithCachedData(v8::internal::Isolate*, v8::internal::Handle<v8::internal::String>, v8::internal::ScriptDetails const&, v8::internal::AlignedCachedData*, v8::ScriptCompiler::CompileOptions, v8::ScriptCompiler::NoCacheReason, v8::internal::NativesFlag) [node]
31: 0xf19dbc  [node]
32: 0xf19ec8 v8::ScriptCompiler::CompileUnboundScript(v8::Isolate*, v8::ScriptCompiler::Source*, v8::ScriptCompiler::CompileOptions, v8::ScriptCompiler::NoCacheReason) [node]
33: 0xca8421 node::contextify::ContextifyScript::New(v8::FunctionCallbackInfo<v8::Value> const&) [node]
34: 0xf57eaf v8::internal::FunctionCallbackArguments::Call(v8::internal::CallHandlerInfo) [node]
35: 0xf58465  [node]
36: 0xf58b83 v8::internal::Builtin_HandleApiCall(int, unsigned long*, v8::internal::Isolate*) [node]
37: 0x1963df6  [node]

--------------


1 failures, 200 skips
Failed to verify release candidate. See /tmp/arrow-HEAD.cpSRw for details.
Error: docker run --rm -e VERIFY_VERSION= -e VERIFY_RC= -e TEST_DEFAULT=0 -e TEST_INTEGRATION=1 --shm-size 1073741824 -e CCACHE_COMPILERCHECK=content -e CCACHE_COMPRESS=1 -e CCACHE_COMPRESSLEVEL=6 -e CCACHE_DIR=/ccache -e CCACHE_MAXSIZE=1G -e CMAKE_GENERATOR=Ninja -e GITHUB_ACTIONS=true -e TEST_APT=0 -e TEST_YUM=0 -v /home/runner/work/crossbow/crossbow/arrow:/arrow -v ubuntu-ccache:/ccache apache/arrow-dev:amd64-ubuntu-20.04-verify-rc /bin/bash -c   git config --global --add safe.directory /arrow &&
  /arrow/dev/release/verify-release-candidate.sh ${VERIFY_VERSION} ${VERIFY_RC} exited with non-zero exit code 1
Error: Process completed with exit code 1.
0s

[raulcd]

Umm... the error message is changed. It seems that the js tests failed.

I've re-run the job in case it was a fluke, otherwise we might need to open a different issue for that test.

[llama90]

I think we need to review it more to solve the problem.

[kou]

@github-actions crossbow submit preview-docs -g wheel verify-rc-source-integration*

[github-actions]

Revision: e5a8205

Submitted crossbow builds: ursacomputing/crossbow @ actions-e708b89fd9

Task	Status
preview-docs
verify-rc-source-integration-linux-almalinux-8-amd64
verify-rc-source-integration-linux-conda-latest-amd64
verify-rc-source-integration-linux-ubuntu-20.04-amd64
verify-rc-source-integration-linux-ubuntu-22.04-amd64
verify-rc-source-integration-macos-amd64
verify-rc-source-integration-macos-arm64
verify-rc-source-integration-macos-conda-amd64
wheel-macos-big-sur-cp310-arm64
wheel-macos-big-sur-cp311-arm64
wheel-macos-big-sur-cp312-arm64
wheel-macos-big-sur-cp38-arm64
wheel-macos-big-sur-cp39-arm64
wheel-macos-catalina-cp310-amd64
wheel-macos-catalina-cp311-amd64
wheel-macos-catalina-cp312-amd64
wheel-macos-catalina-cp38-amd64
wheel-macos-catalina-cp39-amd64
wheel-manylinux-2-28-cp310-amd64
wheel-manylinux-2-28-cp310-arm64
wheel-manylinux-2-28-cp311-amd64
wheel-manylinux-2-28-cp311-arm64
wheel-manylinux-2-28-cp312-amd64
wheel-manylinux-2-28-cp312-arm64
wheel-manylinux-2-28-cp38-amd64
wheel-manylinux-2-28-cp38-arm64
wheel-manylinux-2-28-cp39-amd64
wheel-manylinux-2-28-cp39-arm64
wheel-manylinux-2014-cp310-amd64
wheel-manylinux-2014-cp310-arm64
wheel-manylinux-2014-cp311-amd64
wheel-manylinux-2014-cp311-arm64
wheel-manylinux-2014-cp312-amd64
wheel-manylinux-2014-cp312-arm64
wheel-manylinux-2014-cp38-amd64
wheel-manylinux-2014-cp38-arm64
wheel-manylinux-2014-cp39-amd64
wheel-manylinux-2014-cp39-arm64
wheel-windows-cp310-amd64
wheel-windows-cp311-amd64
wheel-windows-cp312-amd64
wheel-windows-cp38-amd64
wheel-windows-cp39-amd64

[kou]

+1

Let's open a new issue for the JS failures.

Windows wheel failures are #42017.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 4 benchmarking runs that have been run so far on merge-commit 41ae29e.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details. It also includes information about 3 possible false positives for unstable benchmarks that are known to sometimes produce them.