GraphBLAS: Add support for generating the JITpackage when cross-compiling

[mmuetzel]

Official CMake documentation suggests that CMake should be run multiple times when building a native binary is needed to generate files when cross-compiling. First: build native (including the native generator binary). Second: cross-build (using the native generator binary built in the first step)
But that is inconvenient and doesn't integrate well into SuiteSparse which consists of many subprojects.

This PR tries to implement an approach where calling cmake twice (once for the host and then again for the target) shouldn't be necessary. It still needs to configure twice: once before the native generator executable was built, and then again after it has been built as an external project. But it does that automatically.

I didn't actually test if this works. (I don't have a cross-toolchain here.) But I hope it does. Keeping this as a draft for now.

I'd very much appreciate if someone could test and report potential issues.
@Desperado17: Would you be up for that?

[Desperado17]

Do you have a branch with zipped source? Otherwise I'll have to try to modify our build system to work with a branch. Should be publicly accessible though.

I'm away from work now so I could try in 12 hours or so.

[mmuetzel]

Maybe here: https://codeload.github.com/mmuetzel/SuiteSparse/zip/refs/heads/graphblas

You can also use GitHub's interface on the "Code" tab and click on the green "Code" button right above the list of files in the repository and click on "Download ZIP" there. That's how I got the above link...

[Desperado17]

Okay, I checked out dev and applied the changes from this pull request. Unfortunately it fails with some strange messages:

[ 12%] Performing build step for 'jitpackage_generator_native' -- Could NOT find GrBJITpackageGenerator (missing: GrBJITpackageGenerator_DIR) -- Configuring done (0.0s) -- Generating done (0.0s) -- Build files have been written to: /proprietary/build/system/ext-suitesparse-dev/1/workspace/GraphBLAS/JITpackage/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native [ 12%] Performing configure step for 'jitpackage_generator_native' Current working directory cannot be established. make[833]: *** [CMakeFiles/jitpackage_generator_native.dir/build.make:92: jitpackage_generator_native-prefix/src/jitpackage_generator_native-stamp/jitpackage_generator_native-configure] Error 1 make[832]: *** [CMakeFiles/Makefile2:85: CMakeFiles/jitpackage_generator_native.dir/all] Error 2 make[831]: *** [Makefile:91: all] Error 2 make[830]: *** [CMakeFiles/jitpackage_generator_native.dir/build.make:86: jitpackage_generator_native-prefix/src/jitpackage_generator_native-stamp/jitpackage_generator_native-build] Error 2 make[829]: *** [CMakeFiles/Makefile2:85: CMakeFiles/jitpackage_generator_native.dir/all] Error 2 make[828]: *** [Makefile:91: all] Error 2

[mmuetzel]

Oof. That looks like an infinite recursion. Most likely a think-o in some of the conditions.

Do you have access to the full build logs? Could you please share them?

[Desperado17]

I have to see if I can do that because there might be proprietary stuff in the logs. Will have to ask for permission. Do you have a discreet way to send or upload that log?

[mmuetzel]

I've found a typo in the target name that might have caused this. It should have been GrBJITPackageGenerator, but I wrote GrBJITpackageGenerator in one place...

I rebased on a current head of the dev2 branch and fixed that typo.

I also added the option to pass additional arguments to the native CMake build (e.g., a toolchain file for the build platform).
If that should be necessary, you could configure SuiteSparse with something like -DSUITESPARSE_CROSS_TOOLCHAIN_FLAGS_NATIVE="-DCMAKE_TOOLCHAIN_FILE=/path/to/your-native-toolchain.cmake;-DCMAKE_C_COMPILER=gcc". (The name for that flag is heavily "inspired" by a similar name that is used by the LLVM project.)

I don't actually know if that caused the infinite regression.

If you are uncomfortable sharing the log here, you could also send it to me via email. You should find my email address as the author of a couple of commits in this repository.

[Desperado17]

To my surprise, the beta7 branch built successfully with -DSUITEPARSE_USE_FORTRAN=OFF and replacing the find_package calls. Is there anything in this branch that replaces the changes above?

[mmuetzel]

The change that removed the generated file was reverted for that beta. But as @DrTimothyAldenDavis pointed out: That can only be a temporary change. So, we need a more permanent solution for cross-compiling.

[mmuetzel]

Given that we might want to upstream this, I renamed the new flag to GRAPHBLAS_CROSS_TOOLCHAIN_FLAGS_NATIVE.

I tested this in MXE Octave (cross-building from Linux to Windows). That seems to be working.
Marking as ready for review.

[DrTimothyAldenDavis]

The change that removed the generated file was reverted for that beta. But as @DrTimothyAldenDavis pointed out: That can only be a temporary change. So, we need a more permanent solution for cross-compiling.

There's no rush, however. I can keep the GB_JITpackage.c file in the final SuiteSparse 7.4.0 stable release.

[mmuetzel]

There's no rush, however. I can keep the GB_JITpackage.c file in the final SuiteSparse 7.4.0 stable release.

I thought this was going to be much more of a pain to implement. (It still was tricky. But a fun-project anyway.)
Imho, if this is working for me (it is), and if it is also working for @Desperado17's use-case (to be seen), and if there is still time, it would be nice to have this in 7.4.0.

There have been a lot of changes to the build system between 7.3 and now. There was already very useful feedback from packagers and the community. I very much appreciate that feedback. 👍
But to be honest, I wouldn't be surprised if we missed some corner-cases (or even if there is a gaping hole somewhere that we didn't think about).

If something breaks horribly, we could probably follow up with a 7.4.1 release. (When it comes to this PR, reverting the removal of the JIT package file would be an easy work-around.)

But we should still wait for @Desperado17's test results before merging this.

[Desperado17]

Ok so I checked out dev2 branch and replaced the two files with the ones from the pull request. Recursion still there.

[mmuetzel]

Could you please show the log? Or at least the last 20 lines or so until the error?

[Desperado17]

It's a recursion. It just keeps printing larger messages. How do I pinpoint the ciritcal part?

[mmuetzel]

Just the last 20 lines or so are ok. Similar to what you showed before...

[Desperado17]

A few more. After these, I aborted manually.

[ 4%] Built target mongoose_test_io
[ 75%] Performing build step for 'jitpackage_generator_native'
[ 4%] Building CXX object Mongoose/CMakeFiles/mongoose_test_memory.dir/Tests/Mongoose_Test_Memory.o
[ 4%] Building C object CXSparse/CMakeFiles/CXSparse_static.dir/Source/cs_ci_pvec.c.o
[ 12%] Creating directories for 'jitpackage_generator_native'
[ 25%] No download step for 'jitpackage_generator_native'
[ 37%] No update step for 'jitpackage_generator_native'
[ 4%] Building C object CXSparse/CMakeFiles/CXSparse.dir/Source/cs_ci_permute.c.o
[ 50%] No patch step for 'jitpackage_generator_native'
[ 62%] Performing configure step for 'jitpackage_generator_native'
[ 4%] Building CXX object Mongoose/CMakeFiles/mongoose_test_edgesep.dir/Tests/Mongoose_Test_EdgeSeparator_exe.o
[ 4%] Building C object SPEX/CMakeFiles/SPEX.dir/SPEX_Util/Source/SPEX_check_solution.c.o
[ 4%] Building C object SPEX/CMakeFiles/SPEX_static.dir/SPEX_Util/Source/SPEX_LU_analyze.c.o
-- The C compiler identification is GNU 9.3.0
[ 4%] Building C object CXSparse/CMakeFiles/CXSparse_static.dir/Source/cs_ci_qr.c.o
-- Detecting C compiler ABI info
[ 4%] Building C object SPEX/CMakeFiles/SPEX_static.dir/SPEX_Util/Source/SPEX_calloc.c.o
[ 4%] Building C object CXSparse/CMakeFiles/CXSparse.dir/Source/cs_ci_pinv.c.o
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /proprietary/dist/toolchain/aarch64-linux-gnu/1/workspace/gcc-9.3.0-2.31-edfe925f87-aarch64-linux-gnu/bin/aarch64-linux-gnu-gcc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Could NOT find GrBJITPackageGenerator (missing: GrBJITPackageGenerator_DIR)
-- Configuring done (0.4s)
-- Generating done (0.0s)
-- Build files have been written to: /proprietary/build/system/ext-suitesparse-dev/1/workspace/GraphBLAS/JITpackage/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native
[ 4%] Building C object CXSparse/CMakeFiles/CXSparse.dir/Source/cs_ci_post.c.o
[ 4%] Building C object SPEX/CMakeFiles/SPEX.dir/SPEX_Util/Source/SPEX_create_default_options.c.o
[ 75%] Performing build step for 'jitpackage_generator_native'
[ 12%] Creating directories for 'jitpackage_generator_native'
[ 4%] Building C object SPEX/CMakeFiles/SPEX_static.dir/SPEX_Util/Source/SPEX_check_solution.c.o
[ 25%] No download step for 'jitpackage_generator_native'
[ 37%] No update step for 'jitpackage_generator_native'
[ 50%] No patch step for 'jitpackage_generator_native'
[ 62%] Performing configure step for 'jitpackage_generator_native'
[ 4%] Building C object SPEX/CMakeFiles/SPEX.dir/SPEX_Util/Source/SPEX_cumsum.c.o
[ 4%] Building C object CXSparse/CMakeFiles/CXSparse_static.dir/Source/cs_ci_qrsol.c.o
[ 4%] Building C object SPEX/CMakeFiles/SPEX.dir/SPEX_Util/Source/SPEX_finalize.c.o
-- The C compiler identification is GNU 9.3.0
[ 4%] Building C object CXSparse/CMakeFiles/CXSparse.dir/Source/cs_ci_print.c.o
-- Detecting C compiler ABI info
[ 4%] Building C object SPEX/CMakeFiles/SPEX.dir/SPEX_Util/Source/SPEX_free.c.o
[ 4%] Building C object SPEX/CMakeFiles/SPEX.dir/SPEX_Util/Source/SPEX_gmp.c.o
[ 4%] Building C object CXSparse/CMakeFiles/CXSparse_static.dir/Source/cs_ci_randperm.c.o
-- Detecting C compiler ABI info - done
[ 4%] Building C object SPEX/CMakeFiles/SPEX_static.dir/SPEX_Util/Source/SPEX_create_default_options.c.o
[ 4%] Building C object CXSparse/CMakeFiles/CXSparse.dir/Source/cs_ci_pvec.c.o
-- Check for working C compiler: /proprietary/dist/toolchain/aarch64-linux-gnu/1/workspace/gcc-9.3.0-2.31-edfe925f87-aarch64-linux-gnu/bin/aarch64-linux-gnu-gcc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Could NOT find GrBJITPackageGenerator (missing: GrBJITPackageGenerator_DIR)
-- Configuring done (0.6s)
[ 4%] Building C object CXSparse/CMakeFiles/CXSparse.dir/Source/cs_ci_qr.c.o
-- Generating done (0.0s)
-- Build files have been written to: /proprietary/build/system/ext-suitesparse-dev/1/workspace/GraphBLAS/JITpackage/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native
[ 75%] Performing build step for 'jitpackage_generator_native'
[ 5%] Building C object CXSparse/CMakeFiles/CXSparse_static.dir/Source/cs_ci_reach.c.o
[ 12%] Creating directories for 'jitpackage_generator_native'
[ 25%] No download step for 'jitpackage_generator_native'
[ 37%] No update step for 'jitpackage_generator_native'
[ 50%] No patch step for 'jitpackage_generator_native'
[ 62%] Performing configure step for 'jitpackage_generator_native'
[ 5%] Building C object SPEX/CMakeFiles/SPEX_static.dir/SPEX_Util/Source/SPEX_cumsum.c.o
[ 5%] Building C object CXSparse/CMakeFiles/CXSparse.dir/Source/cs_ci_qrsol.c.o
-- The C compiler identification is GNU 9.3.0
[ 5%] Building C object CXSparse/CMakeFiles/CXSparse_static.dir/Source/cs_ci_scatter.c.o
-- Detecting C compiler ABI info
[ 5%] Building CXX object Mongoose/CMakeFiles/mongoose_test_memory.dir/Tests/Mongoose_Test_Memory_exe.o
[ 5%] Linking CXX executable tests/mongoose_test_edgesep
[ 5%] Building C object SPEX/CMakeFiles/SPEX_static.dir/SPEX_Util/Source/SPEX_finalize.c.o
[ 5%] Building C object SPEX/CMakeFiles/SPEX_static.dir/SPEX_Util/Source/SPEX_free.c.o
-- Detecting C compiler ABI info - done
[ 5%] Built target mongoose_test_edgesep
[ 5%] Building C object SPEX/CMakeFiles/SPEX_static.dir/SPEX_Util/Source/SPEX_gmp.c.o
-- Check for working C compiler: /proprietary/dist/toolchain/aarch64-linux-gnu/1/workspace/gcc-9.3.0-2.31-edfe925f87-aarch64-linux-gnu/bin/aarch64-linux-gnu-gcc - skipped
-- Detecting C compile features
[ 5%] Building C object CXSparse/CMakeFiles/CXSparse_static.dir/Source/cs_ci_scc.c.o
-- Detecting C compile features - done
[ 5%] Building C object CXSparse/CMakeFiles/CXSparse.dir/Source/cs_ci_randperm.c.o
-- Could NOT find GrBJITPackageGenerator (missing: GrBJITPackageGenerator_DIR)
-- Configuring done (0.5s)
-- Generating done (0.0s)
-- Build files have been written to: /proprietary/build/system/ext-suitesparse-dev/1/workspace/GraphBLAS/JITpackage/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native
[ 75%] Performing build step for 'jitpackage_generator_native'
[ 12%] Creating directories for 'jitpackage_generator_native'
[ 5%] Building C object CXSparse/CMakeFiles/CXSparse_static.dir/Source/cs_ci_schol.c.o
[ 25%] No download step for 'jitpackage_generator_native'
[ 37%] No update step for 'jitpackage_generator_native'
[ 50%] No patch step for 'jitpackage_generator_native'
[ 62%] Performing configure step for 'jitpackage_generator_native'
[ 5%] Building CXX object Mongoose/CMakeFiles/mongoose_test_performance.dir/Tests/Mongoose_Test_Performance.o
[ 5%] Building C object CXSparse/CMakeFiles/CXSparse.dir/Source/cs_ci_reach.c.o
-- The C compiler identification is GNU 9.3.0
[ 5%] Building C object CXSparse/CMakeFiles/CXSparse_static.dir/Source/cs_ci_spsolve.c.o
-- Detecting C compiler ABI info
[ 5%] Building C object CXSparse/CMakeFiles/CXSparse_static.dir/Source/cs_ci_sqr.c.o
[ 5%] Building C object CXSparse/CMakeFiles/CXSparse.dir/Source/cs_ci_scatter.c.o
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /proprietary/dist/toolchain/aarch64-linux-gnu/1/workspace/gcc-9.3.0-2.31-edfe925f87-aarch64-linux-gnu/bin/aarch64-linux-gnu-gcc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Could NOT find GrBJITPackageGenerator (missing: GrBJITPackageGenerator_DIR)
-- Configuring done (0.5s)
-- Generating done (0.0s)
-- Build files have been written to: /proprietary/build/system/ext-suitesparse-dev/1/workspace/GraphBLAS/JITpackage/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native
[ 5%] Building C object CXSparse/CMakeFiles/CXSparse_static.dir/Source/cs_ci_symperm.c.o
[ 75%] Performing build step for 'jitpackage_generator_native'
[ 12%] Creating directories for 'jitpackage_generator_native'
[ 5%] Building C object CXSparse/CMakeFiles/CXSparse.dir/Source/cs_ci_scc.c.o
[ 25%] No download step for 'jitpackage_generator_native'
[ 37%] No update step for 'jitpackage_generator_native'
[ 50%] No patch step for 'jitpackage_generator_native'
[ 62%] Performing configure step for 'jitpackage_generator_native'
[ 5%] Building C object CXSparse/CMakeFiles/CXSparse_static.dir/Source/cs_ci_tdfs.c.o
-- The C compiler identification is GNU 9.3.0
-- Detecting C compiler ABI info
[ 5%] Building C object SPEX/CMakeFiles/SPEX.dir/SPEX_Util/Source/SPEX_initialize.c.o
[ 5%] Building C object CXSparse/CMakeFiles/CXSparse_static.dir/Source/cs_ci_transpose.c.o
[ 5%] Building C object CXSparse/CMakeFiles/CXSparse.dir/Source/cs_ci_schol.c.o
[ 5%] Building C object CXSparse/CMakeFiles/CXSparse_static.dir/Source/cs_ci_updown.c.o
[ 5%] Linking CXX executable tests/mongoose_test_memory
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /proprietary/dist/toolchain/aarch64-linux-gnu/1/workspace/gcc-9.3.0-2.31-edfe925f87-aarch64-linux-gnu/bin/aarch64-linux-gnu-gcc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Could NOT find GrBJITPackageGenerator (missing: GrBJITPackageGenerator_DIR)
-- Configuring done (0.4s)
[ 5%] Building C object SPEX/CMakeFiles/SPEX.dir/SPEX_Util/Source/SPEX_initialize_expert.c.o
-- Generating done (0.0s)
-- Build files have been written to: /proprietary/build/system/ext-suitesparse-dev/1/workspace/GraphBLAS/JITpackage/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native
[ 75%] Performing build step for 'jitpackage_generator_native'
[ 5%] Built target mongoose_test_memory
[ 5%] Building C object CXSparse/CMakeFiles/CXSparse_static.dir/Source/cs_ci_usolve.c.o
[ 12%] Creating directories for 'jitpackage_generator_native'
[ 25%] No download step for 'jitpackage_generator_native'
[ 5%] Building C object CXSparse/CMakeFiles/CXSparse.dir/Source/cs_ci_spsolve.c.o
[ 5%] Building CXX object Mongoose/CMakeFiles/mongoose_test_reference.dir/Tests/Mongoose_Test_Reference.o
[ 37%] No update step for 'jitpackage_generator_native'
[ 50%] No patch step for 'jitpackage_generator_native'
[ 62%] Performing configure step for 'jitpackage_generator_native'
[ 5%] Building C object SPEX/CMakeFiles/SPEX.dir/SPEX_Util/Source/SPEX_malloc.c.o
[ 5%] Building CXX object Mongoose/CMakeFiles/mongoose_test_reference.dir/Tests/Mongoose_Test_Reference_exe.o
-- The C compiler identification is GNU 9.3.0
-- Detecting C compiler ABI info
[ 5%] Building C object CXSparse/CMakeFiles/CXSparse_static.dir/Source/cs_ci_util.c.o
[ 5%] Building C object SPEX/CMakeFiles/SPEX_static.dir/SPEX_Util/Source/SPEX_initialize.c.o
[ 5%] Building C object CXSparse/CMakeFiles/CXSparse.dir/Source/cs_ci_sqr.c.o
[ 5%] Building C object SPEX/CMakeFiles/SPEX.dir/SPEX_Util/Source/SPEX_matrix_allocate.c.o
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /proprietary/dist/toolchain/aarch64-linux-gnu/1/workspace/gcc-9.3.0-2.31-edfe925f87-aarch64-linux-gnu/bin/aarch64-linux-gnu-gcc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Could NOT find GrBJITPackageGenerator (missing: GrBJITPackageGenerator_DIR)
-- Configuring done (0.5s)
-- Generating done (0.0s)
-- Build files have been written to: /proprietary/build/system/ext-suitesparse-dev/1/workspace/GraphBLAS/JITpackage/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native/native
[ 5%] Building C object SPEX/CMakeFiles/SPEX_static.dir/SPEX_Util/Source/SPEX_initialize_expert.c.o

[mmuetzel]

Which command do you use to configure SuiteSparse?

[Desperado17]

In theory only the -DSUITESPARSE_USE_FORTRAN=OFF FLAG. Plus the parameters the toolchain sets of course.

[mmuetzel]

Which parameters does the toolchain set?

[Desperado17]

I can try sending an anonymized version of cmakecache.txt if that helps.

[mmuetzel]

Is there no way you can check how cmake was called for configuration?

The cmakecache.txt might help. (But please make clear which parts you anonymized.)

[mmuetzel]

Could you please also show which build platform and which target platform you are using?

[Desperado17]

CMakeCache.txt

Host is an Ubuntu 22.04 x64 . Compiler is a gcc-9.3.0-2.31 aarch 64 cross compiler by linaro. Anonymization is only the prefix of the file path of the source/build dir which has been replaced by /proprietary/. Target platform is an aarch64 based poky linux.

[Desperado17]

Need to leave for two hours. Will read messages later.

[mmuetzel]

It looks like it never picks up a native compiler in your configuration. In that case, you'd need to explicitly let CMake know which compilers it should use to build a native binary.
A toolchain file like the following might suffice:

set ( CMAKE_SYSTEM_NAME Linux )
set ( CMAKE_SYSTEM_PROCESSOR x86_64 )
set ( CMAKE_C_COMPILER gcc )

You might need to adjust the path to a native C compiler for your build system.

Save a file with that content, e.g., to /some/path/my-native-toolchain.cmake.

Then configure with -DGRAPHBLAS_CROSS_TOOLCHAIN_FLAGS_NATIVE="-DCMAKE_TOOLCHAIN_FILE=/some/path/my-native-toolchain.cmake".

If that doesn't help yet, you might need to override more settings in the toolchain file (e.g., CMAKE_FIND_ROOT_PATH, CMAKE_EXE_LINKER_FLAGS or others).
Check the CMakeCache.txt in build/GraphBLAS/JITpackage/native for what it picks up from the target toolchain. That file should only contain flags that are valid for your build toolchain.

If you like, please upload that CMakeCache.txt here. And I could try and point to which values would need to be set.

[mmuetzel]

For comparison, here is the file that I find in the folder build/GraphBLAS/JITpackage/native when cross-building on Linux AMD64 to Windows:
CMakeCache.txt
Note that there is no notion of the cross toolchain for the Windows target in that file.

[Desperado17]

I removed the lines again and those cross flags did not reappear only your error message. Something seems wrong here.

[mmuetzel]

I removed the lines again and those cross flags did not reappear only your error message. Something seems wrong here.

Please clarify: What did you remove from where and which flags did not reappear where? And what do you mean by "your error message"?

[Desperado17]

With "your error message" I mean "Native toolchain not configured correctly"

I removed the following lines again from the linux_host.toolchain
set ( CMAKE_C_FLAGS "" )
set ( CMAKE_EXE_LINKER_FLAGS "" )
set ( CMAKE_MODULE_LINKER_FLAGS "" )
set ( CMAKE_SHARED_LINKER_FLAGS "" )

And it still printed Native toolchain not configured correctly although I expected the error from

#616 (comment)

to reappear. I had to delete the build directory to bring it back. Readding the flags yields again "Native toolchain not configured correctly".

[mmuetzel]

Could you please delete the current content of /media/user/Ext4Disc/Projects4/bob_nav/dev/build/system/ext-suitesparse-dev/1/workspace/GraphBLAS/JITpackage/native?
Then use the following toolchain file (remove the .txt extension) and build again:
linux_host.toolchain.txt

After that, upload the new CMakeCache.txt from build/GraphBLAS/JITpackage/native.

[Desperado17]

Native toolchain not configured correctly

CMakeCacheNative4.txt

[mmuetzel]

What does /proprietary/dev/dist/toolchain/aarch64-linux-gnu/1/workspace/defaults.cmake contain? Is that the source of those C_FLAGS?

[mmuetzel]

Maybe try the following in linux_host.toolchain:

# CMake settings for build host
set ( CMAKE_SYSTEM_NAME "Linux" )
set ( CMAKE_SYSTEM_PROCESSOR "x86_64" )
set ( CMAKE_C_COMPILER "gcc" )
set ( CMAKE_C_FLAGS "" CACHE STRING "" FORCE )
set ( CMAKE_EXE_LINKER_FLAGS "" CACHE STRING "" FORCE )
set ( CMAKE_MODULE_LINKER_FLAGS "" CACHE STRING "" FORCE )
set ( CMAKE_SHARED_LINKER_FLAGS "" CACHE STRING "" FORCE )
# do not look into package registries
SET(CMAKE_FIND_PACKAGE_NO_PACKAGE_REGISTRY ON)
SET(CMAKE_FIND_PACKAGE_NO_SYSTEM_PACKAGE_REGISTRY ON)

Then delete the native folder again and try to build.

[Desperado17]

As far as I can tell these CFLAGS are an environment variable our build system sets.

[mmuetzel]

Does CMake still use the environment variables when you FORCE override the cache (like in the comment above)?

[Desperado17]

Native toolchain not configured correctly

CMakeCacheNative5.txt

[mmuetzel]

Good. Those wrong flags are gone now. 👍

Could you please copy the exact error message that you see now?

[Desperado17]

---

-- SuiteSparse CMAKE report for: SuiteSparse

---

-- build type: MinSizeRel
-- BUILD_SHARED_LIBS: ON
-- BUILD_STATIC_LIBS: ON
-- use OpenMP: no
-- C compiler: /proprietary/dist/toolchain/aarch64-linux-gnu/1/workspace/gcc-9.3.0-2.31-edfe925f87-aarch64-linux-gnu/bin/aarch64-linux-gnu-gcc
-- C flags: -pipe -feliminate-unused-debug-types -march=armv8-a+crypto -mtune=cortex-a57.cortex-a53 -ffunction-sections -fdata-sections -Wno-error=deprecated
-- C++ compiler: /proprietary/dist/toolchain/aarch64-linux-gnu/1/workspace/gcc-9.3.0-2.31-edfe925f87-aarch64-linux-gnu/bin/aarch64-linux-gnu-g++
-- C++ flags: -pipe -feliminate-unused-debug-types -march=armv8-a+crypto -mtune=cortex-a57.cortex-a53 -ffunction-sections -fdata-sections -std=c++11 -Wno-error=deprecated
-- C Flags release: -O3 -DNDEBUG
-- C++ Flags release: -O3 -DNDEBUG
-- Fortran compiler: none
-- compile definitions:

---

-- Configuring done (3.7s)
-- Generating done (1.4s)
CMake Warning:
Manually-specified variables were not used by the project:

CMAKE_EXPORT_NO_PACKAGE_REGISTRY

-- Build files have been written to: /proprietary/build/system/ext-suitesparse-dev/1/workspace

• touch .cmake-done
• echo afcdd11b3be79fa17a66dd13e296706433391dc9
• makeParallel
• '[' xnproc = xnproc ']'
  ++ nproc
• MAKE_JOBS=8
• '[' ' ' = nproc ']'
• make -j 8 -l
  [ 0%] Building C object SuiteSparse_config/CMakeFiles/SuiteSparseConfig.dir/SuiteSparse_config.c.o
  [ 0%] Building C object SuiteSparse_config/CMakeFiles/SuiteSparseConfig_static.dir/SuiteSparse_config.c.o
  [ 0%] Building C object BTF/CMakeFiles/BTF.dir/Source/btf_l_maxtrans.c.o
  [ 0%] Building C object LDL/CMakeFiles/LDL.dir/Source/ldl.c.o
  [ 0%] Building C object BTF/CMakeFiles/BTF.dir/Source/btf_l_order.c.o
  [ 0%] Building C object BTF/CMakeFiles/BTF_static.dir/Source/btf_l_maxtrans.c.o
  [ 0%] Building C object LDL/CMakeFiles/LDL_static.dir/Source/ldl.c.o
  [ 0%] Creating directories for 'jitpackage_generator_native'
  [ 0%] No download step for 'jitpackage_generator_native'
  [ 0%] No update step for 'jitpackage_generator_native'
  [ 0%] No patch step for 'jitpackage_generator_native'
  [ 0%] Performing configure step for 'jitpackage_generator_native'
  [ 0%] Building C object LDL/CMakeFiles/LDL.dir/Source/ldll.c.o
  [ 0%] Building C object BTF/CMakeFiles/BTF.dir/Source/btf_l_strongcomp.c.o
  -- The C compiler identification is GNU 11.4.0
  [ 0%] Building C object BTF/CMakeFiles/BTF_static.dir/Source/btf_l_order.c.o
  -- Detecting C compiler ABI info
  [ 0%] Linking C static library libsuitesparseconfig.a
  [ 0%] Linking C shared library libsuitesparseconfig.so
  [ 0%] Built target SuiteSparseConfig_static
  [ 1%] Building C object BTF/CMakeFiles/BTF.dir/Source/btf_maxtrans.c.o
  [ 1%] Built target SuiteSparseConfig
  [ 1%] Building C object BTF/CMakeFiles/BTF.dir/Source/btf_order.c.o
  -- Detecting C compiler ABI info - done
  -- Check for working C compiler: /usr/bin/gcc - skipped
  -- Detecting C compile features
  -- Detecting C compile features - done
  CMake Error at CMakeLists.txt:24 (message):
  Native toolchain not configured correctly

-- Configuring incomplete, errors occurred!
make[2]: *** [GraphBLAS/JITpackage/CMakeFiles/jitpackage_generator_native.dir/build.make:92: GraphBLAS/JITpackage/jitpackage_generator_native-prefix/src/jitpackage_generator_native-stamp/jitpackage_generator_native-configure] Error 1
make[1]: *** [CMakeFiles/Makefile2:2926: GraphBLAS/JITpackage/CMakeFiles/jitpackage_generator_native.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....

[mmuetzel]

Hmmm. Maybe try again after adding the following line to the toolchain file:

set ( CMAKE_CROSSCOMPILING OFF )

[Desperado17]

It seems to proceed much further now. Will tell you if overall build is success.

[mmuetzel]

If compilation succeeds, could you please show the content of the native toolchain file that finally worked? That might be helpful for others that might want to cross-compile GraphBLAS.

[Desperado17]

Success! This is the content of the final linux_host.toolchain:

# CMake settings for build host
set ( CMAKE_SYSTEM_NAME "Linux" )
set ( CMAKE_SYSTEM_PROCESSOR "x86_64" )
set ( CMAKE_C_COMPILER "gcc" )
set ( CMAKE_C_FLAGS "" CACHE STRING "" FORCE )
set ( CMAKE_EXE_LINKER_FLAGS "" CACHE STRING "" FORCE )
set ( CMAKE_MODULE_LINKER_FLAGS "" CACHE STRING "" FORCE )
set ( CMAKE_SHARED_LINKER_FLAGS "" CACHE STRING "" FORCE )
# do not look into package registries
SET(CMAKE_FIND_PACKAGE_NO_PACKAGE_REGISTRY ON)
SET(CMAKE_FIND_PACKAGE_NO_SYSTEM_PACKAGE_REGISTRY ON)

set ( CMAKE_CROSSCOMPILING OFF )

[mmuetzel]

I'll try and add some documentation with hints for cross-compilers. That'll have to wait until after the holidays though.

[DrTimothyAldenDavis]

Sounds great. Thanks for all your help with this, @mmuetzel and @Desperado17 .

Shall I merge this in now? Or shall I wait until you've had a chance to add some instructions? Presumably in the GraphBLAS/JITpackage/README.txt file?

It would be fine with me if the instructions were specific to an x86 linux host. Someone else who is cross compiling with another host could probably use those instructions to figure out a reasonable host toolchain of their own.

I'd like to post a stable 7.4.0 sometime before the end of the year, if possible.

[mmuetzel]

I've added some hints for cross-compiling to the README file, and changed its format to mark-down.

[DrTimothyAldenDavis]

Great! Is it ready to merge? I could do that and then post a new beta release. My goal for a stable 7.4.0 release is Dec 30.

[DrTimothyAldenDavis]

I've merged it in. I haven't tested it in a cross compilation mode though.

I will post another beta release first. If that looks good, with this change, I'd like to call it a stable 7.4.0, on Dec 30 if possible.

Does that sound ok?

[mmuetzel]

I will post another beta release first. If that looks good, with this change, I'd like to call it a stable 7.4.0, on Dec 30 if possible.

Does that sound ok?

That sounds ok. In case there are major issues, they could probably be fixed in a version 7.4.1...

[DrTimothyAldenDavis]

OK. I will release the stable SuiteSparse 7.4.0 tomorrow morning, my time. I can easily and quickly do a 7.4.1 if needed.