meta issue for reproducible builds (was: tests/determinism.sh)

[marc-hb]

These tests are in a "work4me" state. They all proved valuable because they all found issues and all passed a fair number of times in some configuration(s)/environment(s). However for either passing or finding issues they may require:

• magic test parameters
• fixes not merged or not submitted yet
• additional tools like disorderedfs
• additional test steps like comparing checksums across different environments

I'm sharing this very early draft for two purposes:

1. Provide more detailed reproduction information and transparency for related fixes. There's only so much test code that can be put in a commit message.

2. Gather feedback and ideas about what could be an acceptable, high-level test design so CI can hopefully catch the most basic regressions some day. Please don't spend time reviewing any implementation detail because I doubt the final test design (if any) will look similar to this prototype.

Here a list of fixes that some past, present or future version of (some of) these tests have helped with:

• cmake: more deterministic git describe --abbrev=12 #13608

• scripts/*syscalls.py: sort os.walk() for a more deterministic build #13446

• Sort a few python dicts for deterministic builds, even with python 3.5 and earlier. #14119

• Sort a few python dicts for deterministic builds, even with python 3.5 and earlier. #14119

• git.cmake: let the environment override BUILD_VERSION #14594

• host-gcc/target.cmake: stop discarding CMAKE_<tool>_FLAGS for x86 #14595

• sanitycheck: don't generate the top-level Makefile in random order #14775

• extensions.cmake: need unique strings, not random ones #14776

• zephyr_library_compile_options(): de-duplicate #15094

• file2hex.py: switch from gzip.compress() to GzipFile() #14804

• file2hex.py: new --gzip-mtime option that defaults to zero + test #15043

• cmake: atomic rename to fix toolchain cache creation race #15128

• sanitycheck: order results.csv and discards.csv deterministically #15313

• cmake: zephyr_cc_option(-fmacro-prefix-map=${ZEPHYR_BASE}=.) #15376

• gen_kobject_list.py: better comments and --help. Zero code change. #15578

• qemu_x86 's SeaBIOS clears the screen every time it runs #11717

• extensions.cmake: don't leak absolute paths in snippets-*.ld comment #16368

• sanitycheck: CONFIG_TEST_USERSPACE / userspace tag cleanup #16466

• CMakeLists.txt: -fmacro-prefix-map=${CMAKE_SOURCE_DIR}=CMAKE_SOURCE_DIR #16536

• BOOT_BANNER: show KERNEL_VERSION and BUILD_VERSION differently #16917

• tests: minor FCB re-ordering not to leave a random flash.bin behind #16965

• cmake: strip_nondeterminism_if_found() on all .a library files #17494
  strip-nondeterminism MR4 ar.pm: Don't corrupt tables of symbols and long filenames

• toolchain/zephyr: invoke ar with -D for deterministic .a files #17495

• scripts: dts: Output paths relative to $ZEPHYR_BASE #19444

• cmake: log ${CMAKE_VERSION} #53286

• scripts/build: make struct_tags.json deterministic #66009

• elf_parser.py: make dependency graph output deterministic #66014

• list_hardware.py: sort rglob(SOC_YML) HWMv2 results #70132

diffoscope MR 29 Catch failures to disassemble and rescue all other differences
diffoscope issue 64 Remove elf.StaticLibFile: it's a serious design flaw

• XCC RG-2017.8-linux: generated object code is affected by -g2 level (the default level) + longer source paths thesofproject/sof#7114
• zephyr/CMakeLists.txt: remove '..' in include paths  thesofproject/sof#7428
• Windows/Linux disassembly differences when using #include ../something.h? thesofproject/sof#9034

[zephyrbot]

Found the following issues, please fix and resubmit:

Codeowners issues

New files added that are not covered in CODEOWNERS:

tests/determinism.sh

Please add one or more entries in the CODEOWNERS file to cover those files

checkpatch issues

-:244: ERROR:TRAILING_WHITESPACE: trailing whitespace
#244: FILE: tests/determinism.sh:238:
+    ninja -C "$bld"  # obj_list # kobj_types_h_target $

- total: 1 errors, 0 warnings, 264 lines checked

NOTE: For some of the reported defects, checkpatch may be able to
      mechanically convert to the typical style using --fix or --fix-inplace.

NOTE: Whitespace errors detected.
      You may wish to use scripts/cleanpatch or scripts/cleanfile

Your patch has style problems, please review.

NOTE: Ignored message types: AVOID_EXTERNS BRACES CONFIG_EXPERIMENTAL CONST_STRUCT DATE_TIME FILE_PATH_CHANGES MINMAX NETWORKING_BLOCK_COMMENT_STYLE PRINTK_WITHOUT_KERN_LEVEL SPLIT_STRING VOLATILE

NOTE: If any of the errors are false positives, please report
      them to the maintainers.

[nashif]

determinism might be a bit confusing name, how about reproducible_builds or something that is closer to what this is about.

[carlescufi]

Just a general comment, without getting into the particulars of this change (which by the way looks pretty interesting as a concept to me). Can we write this in Python? This would be good for 2 reasons:

1. Most people contributing to Zephyr are more or less comfortable with Python at this point. Less so with bash.
2. It would work on Windows, which is a supported platform

[marc-hb]

Thanks @carlescufi , yes python shouldn't be a problem. Just a bit more verbose that's all :-)

But even before looking at the programming language I would like ideas and directions about where this could/should fit in the grand testing architecture scheme of things. Like: still sitting on top of sanitycheck like this? Or as additional --feature(s) embedded inside sanitycheck itself instead? Other? If not embedded in sanitycheck then how could CI run the most basic of these tests and catch the most basic regressions? Additional github check?

[nashif]

But even before looking at the programming language I would like ideas and directions about where this could/should fit in the grand testing architecture scheme of things. Like: still sitting on top of sanitycheck like this? Or as additional --feature(s) embedded inside sanitycheck itself instead? Other? If not embedded in sanitycheck then how could CI run the most basic of these tests and catch the most basic regressions? Additional github check?

this will run on its own in a nightly job, not per PR and not with check_compliance.

[marc-hb]

Thanks @nashif , this answers my main question.

Here's how I see the next steps:

1. Share here my most recent updates and fixes to this shell script
2. Rewrite it in tests/build_reproduction.py and submit a new PR
3. Archive this shell script PR.

[nashif]

are you planning for this to be merged?

[marc-hb]

No, I will push another one. This script has pretty much been entirely rewritten. I will just keep using the description for bookmarks

EDIT 3 years later: I never shared the newer version. It's been bitrotting on a private branch somewhere. The high-level logic was still the same:

• build once
• change as many things as possible
• build again
• diff