Support resiliency tests and Docker rules on macOS

[fmorency]

macOS binaries are created using the *-apple-darwin triplet. Since there's no apple-darwin docker image, Docker requires binaries to be built using the *-unknown-linux-* triplet.

Add support for targeting *-unknown-linux-* from macOS (e.g., using Docker sandboxing) and running resiliency tests (Linux Docker) on macOS.

[fmorency]

The issue is more complex than initially estimated.

Using Bazel docker sandboxing fails because Bazel's process_wrapper is built for the HOST system, namely aarch64-apple-darwin, and fails to run in the container, which is aarch64-unknown-linux-gnu. The exact Bazel error is

exec bazel-out/darwin_arm64-opt-exec-2B5CBBC6/bin/external/rules_rust/util/process_wrapper/process_wrapper: exec format error

I made progress with cross-compilation to aarch64-unknown-linux-gnu using zig-cc, but the final linking step failed because it tried to link with HOST libraries (SSL, libusb & libudev) on a different architecture.

One potential solution would be integrating OpenSSL & libhidapi into our Bazel build. However, the openssl-sys crate fails to build in Bazel when using the vendored feature, which builds openssl from the source (bazelbuild/rules_rust#1519).

More investigation is needed.

[fmorency]

Let's see if we can make bazel-contrib/rules_oci#77 work for our project.

[fmorency]

The current rules_oci version cannot pull from the docker hub.

bazel-contrib/rules_oci#85

[fmorency]

This post wraps up my Bazel macOS + Docker adventure. This project took much longer than initially estimated and still needs to be completed.

Initial goal: Build and run the many-rs Docker images on macOS using Bazel.
Problem: The macOS host system is Darwin (aarch64), and the Docker images are Linux (aarch64). The Docker images are created by Bazel using the binaries built for Darwin, resulting in the wrong executable format.

The latest attempt can be found at https://github.com/fmorency/many-rs-cross. I tried to cross-compile from Linux x86_64 to Linux aarch64.

Attempts

Bazel docker-sandbox

The Bazel process-wrapper is built for Darwin and fails to run inside the docker sandbox. I was unable to find a solution to this problem.

Build inside Docker

I adapted the Bazel docker-container tutorial from 1. I built my own Docker build environment 2. The solution worked sometime 3 and was slow. Future work could use an alternative storage driver 4, such as zfs.

One must also address the Docker inside Docker (dind) challenge of building the Docker image from inside the Docker container. I needed more time to tackle this challenge.

Cross-compiling

My last attempt involved cross-compilation using Grail`s hermetic LLVM 5.

The many-rs project depends on the openssl crate; the openssl-sys crate cannot be built with Bazel when using the vendored feature 6; we need to use a sysroot.

Sysroot

You will find in 7 the scripts used to create a Debian aarch64 sysroot. Those scripts come from the Chromium project 8. Both Debian bullseye and bookworm sysroots are supported.

Pre-built sysroots are found in https://github.com/fmorency/test-sysroot/releases/tag/v1.0.0

Note: After creating the sysroot archive, extract it and copy the openssl headers from usr/include/aarch64-linux-gnu/openssl to usr/include/openssl. Then re-create the archive and use the result as the sysroot. This step is required to avoid compilation issues with the openssl-sys crate.

$ cp -Lr usr/include/aarch64-linux-gnu/openssl/configuration.h usr/include/aarch64-linux-gnu/openssl/opensslconf.h usr/include/openssl

Openssl

You will find in 9 an example project cross-compiling against the openssl crate and using a sysroot OpenSSL libraries. The sysroot used in this example is based in Debian bullseye.

Please take a look at the Note in the section above if you happen to encounter an openssl-related compilation issue.

Debian bullseye is based on OpenSSL 1.1.1. The example project builds just fine, but the many-rs project requires OpenSSL 3.x because of the webauthn-authenticator-rs crate 10. OpenSSL 3 is available from Debian bookworm.

webauthn-authenticator-rs

As mentioned in the section above, the webauthn-authenticator-rs crate requires OpenSSL 3.x. In addition, I used revision e9ebff3123263073c3668228fa088f007f3fa555, which defaults to linux-static-hidraw instead of linux-static-libusb like in v0.4.9.

libc

I was unable to link the cross-compiled project against glibc. The error is

  = note: ld.lld: error: undefined symbol: fstat64
          >>> referenced by fs.rs:1009 (library/std/src/sys/unix/fs.rs:1009)
          >>>               std-5e17e5117b93a26c.std.bc4647e0-cgu.0.rcgu.o:(std::fs::buffer_capacity_required::h87a0125b04f49ca2) in archive /home/fmorency/.cache/bazel/_bazel_fmorency/1b9b0f09909ec2a9cffa7ce4feab1a7a/external/rust-linux-aarch64__aarch64-unknown-linux-gnu__nightly_tools/lib/rustlib/aarch64-unknown-linux-gnu/lib/libstd-5e17e5117b93a26c.rlib
          >>> referenced by fs.rs:1009 (library/std/src/sys/unix/fs.rs:1009)
          >>>               std-5e17e5117b93a26c.std.bc4647e0-cgu.0.rcgu.o:(std::backtrace_rs::symbolize::gimli::mmap::hb24f5017b37ff140) in archive /home/fmorency/.cache/bazel/_bazel_fmorency/1b9b0f09909ec2a9cffa7ce4feab1a7a/external/rust-linux-aarch64__aarch64-unknown-linux-gnu__nightly_tools/lib/rustlib/aarch64-unknown-linux-gnu/lib/libstd-5e17e5117b93a26c.rlib
          >>> did you mean: fstat64@GLIBC_2.33
          >>> defined in: external/org_ll_sysroot_linux_aarch64/lib/aarch64-linux-gnu/libc.so.6
          
          ld.lld: error: undefined symbol: stat64
          >>> referenced by fs.rs:1482 (library/std/src/sys/unix/fs.rs:1482)
          >>>               std-5e17e5117b93a26c.std.bc4647e0-cgu.0.rcgu.o:(std::sys::common::small_c_string::run_with_cstr_allocating::h56d025d60e5c6ba9) in archive /home/fmorency/.cache/bazel/_bazel_fmorency/1b9b0f09909ec2a9cffa7ce4feab1a7a/external/rust-linux-aarch64__aarch64-unknown-linux-gnu__nightly_tools/lib/rustlib/aarch64-unknown-linux-gnu/lib/libstd-5e17e5117b93a26c.rlib
          >>> referenced by fs.rs:1482 (library/std/src/sys/unix/fs.rs:1482)
          >>>               std-5e17e5117b93a26c.std.bc4647e0-cgu.0.rcgu.o:(std::sys::unix::fs::stat::h2b943d7f1c11da2f) in archive /home/fmorency/.cache/bazel/_bazel_fmorency/1b9b0f09909ec2a9cffa7ce4feab1a7a/external/rust-linux-aarch64__aarch64-unknown-linux-gnu__nightly_tools/lib/rustlib/aarch64-unknown-linux-gnu/lib/libstd-5e17e5117b93a26c.rlib
          >>> did you mean: stat64@GLIBC_2.33
          >>> defined in: external/org_ll_sysroot_linux_aarch64/lib/aarch64-linux-gnu/libc.so.6
          clang: error: linker command failed with exit code 1 (use -v to see invocation)

and I'm still determining where it comes from. Perhaps I need to rebuild Rust's libstd, e.g., cargo +nightly -Zbuild-std?

I tried building against musl but discovered that the Grail`s LLVM Bazel rule doesn't support it 11.

I would have needed more time to try the gcc-toolchain Bazel rule.

Additional references

• Broken rules_docker transitions: Unable to build rust images on macOS bazelbuild/rules_docker#2108
• rules_docker alternative: https://github.com/bazel-contrib/rules_oci