Compile binutils with --enable-deterministic-archives

[marc-hb]

zephyrproject-rtos/zephyr@b4078c557ddcd7 has just been merged. Long story short it changes the ar invocation to pass -D which creates reproducible .a files by hardcoding the filemodes, timestamps and owners of the files inside the archive.

If no one/nothing complains about the change above after "some" time, a simpler way to achieve the same result is to flip the default when building the SDK. This is apparently possible with some --enable-deterministic-archives option in the binutils build. This what Debian has been doing for a long time (see discussion in https://salsa.debian.org/reproducible-builds/strip-nondeterminism/issues/3) and can be observed with /usr/bin/ar on any Ubuntu >= 18.04, maybe even earlier. I've seen rumours OpenSuSE didn't like it though, I haven't researched why.

[galak]

I'm a little concerned about enabling this since we support other GNU toolchains like the ARM embedded that don't set this, so I think its better to try and handle this in the build system if doable.

[marc-hb]

@galak maybe there is a misunderstanding: this requested change cannot possibly affect toolchains others than the Zephyr SDK, it's what provides the most isolation. Out of all other possibilities and unlike what I'm trying to do at the build system level, this is the one that cannot break other toolchains even if it tried to.

[galak]

I understand, I meant that by changing the Zephyr SDK toolchain behavior we wouldn't notice that we need to support something in the build system for other toolchains.

[marc-hb]

Considering Zephyr's ambitious goal to support an unlimited number of toolchains(zephyrproject-rtos/zephyr#16031, @mped-oticon), builds will never be fully deterministic for all files and all configurations across all the toolchains that Zephyr support. There will always be different levels of quality. Build reproduction issues are not different from other sorts of bugs, see examples at zephyrproject-rtos/zephyr#14593 and build reproduction quality is no different from general quality: how many bugs? Affecting what/who?

[marc-hb]

This what Debian has been doing for a long time

In fact I just confirmed that anyone building native_posix[_64] on Ubuntu and probably any other Debian-based distribution is already getting this new feature request "for free".

I noticed this while doing something else: testing PR #17632. On Fedora, Zephyr SDK PR #17632 fixes everything except native_posix, because native_posix doesn't use the Zephyr SDK. On Ubuntu native_posix is "already fixed" by Debian's --enable-deterministic-archives default.

builds will never be fully deterministic for all files and all configurations across all the toolchains that Zephyr support.

I hope the very basic example above is enough to proves this point.

[marc-h38]

I've seen rumours OpenSuSE didn't like it though, I haven't researched why.

Apparently OpenSuSE experienced some delay in getting a related GNU make fix: https://build.opensuse.org/request/show/509329

Searching for this option returns (among others) tickets equivalent to this one but for other projects:
https://www.google.com/search?q=--enable-deterministic-archives

For instance one can very easily find that FreeBSD turned this on by default in mid 2015
https://reviews.freebsd.org/D3190

[marc-hb]

Ping @galak ? This is hopefully just a one-line change.

[galak]

I'm not sure I understand what the benefit is to turning this flag on in the zephyr SDK as we still will need to support the -D option in the Zephyr Build system, so what does this accomplish.

[marc-hb]

as we still will need to support the -D option in the Zephyr Build system,

The code to pass -D is complex and had to be reverted immediately.

so what does this accomplish.

It replaces something complex and brittle (and... reverted) by something super simple and robust that everyone else does already.

[marc-hb]

Looks like crosstool-ng finally made this the default in December 2019: crosstool-ng/crosstool-ng#1259

To confirm:

1. Build a toolchain with a crosstool-ng past this version (obviously)
2. Build any Zephyr example. Move the build directory, build again. All .a files should be strictly identical.
3. Close this issue.

[galak]

Will pick this up when we update crosstool-ng.

[marc-hb]

Confirmed fixed in Zephyr 0.12.2 (and probably much earlier)

west build
for i in $(find build/ -name *.a); do ar tv $i; done

rw-r--r-- 0/0   2936 Jan  1 00:00 1970 main.c.obj
rw-r--r-- 0/0   3492 Jan  1 00:00 1970 cbprintf.c.obj
rw-r--r-- 0/0   4840 Jan  1 00:00 1970 crc32_sw.c.obj
rw-r--r-- 0/0   6096 Jan  1 00:00 1970 crc16_sw.c.obj
rw-r--r-- 0/0   5124 Jan  1 00:00 1970 crc8_sw.c.obj
rw-r--r-- 0/0   3728 Jan  1 00:00 1970 crc7_sw.c.obj
rw-r--r-- 0/0   4056 Jan  1 00:00 1970 dec.c.obj
rw-r--r-- 0/0  16204 Jan  1 00:00 1970 fdtable.c.obj
rw-r--r-- 0/0   6252 Jan  1 00:00 1970 hex.c.obj
rw-r--r-- 0/0   5576 Jan  1 00:00 1970 notify.c.obj

etc.