Reduce kernel boot time to <1s on comma 3/3X

[adeebshihadeh]

systemd-analyze reports this for the latest AGNOS:

Startup finished in 3.650s (kernel) + 7.343s (userspace) = 10.994s
graphical.target reached after 7.059s in userspace

Get the kernel part of that down to <1s in a self-contained way that doesn't delay the rest of the openpilot.

• can't just defer stuff to later and cost boot time at later stages
• can turn off stuff openpilot doesn't use (NFC, bluetooth, etc.)

AGNOS is built with https://github.com/commaai/agnos-builder

[andiradulescu]

I'm planning to do the following (most of them extracted from the boot-time-slides.pdf from the documentation links):

• Measure and optimize the kernel initialization functions:
  • Analyze boot.svg from Boot time speedup agnos-builder#110 (comment)
  • Remove unnecessary functionality:
    • Find which kernel configuration parameter compiles the code, by looking at the Makefile in the corresponding source directory.
  • Postpone:
    • Find which module (if any) the function belongs to. Load this module later if possible.
  • Optimize necessary functionality:
    • Look for parameters which could be used to reduce probe time, looking for the module_param macro.
    • Look for delay loops and calls to functions containing delay in their name, which could take more time than needed. You could reduce such delays, and see whether the code still works or not.
• Cut out unused drivers.
• Reduce kernel size:
  • Compile everything that is not needed at boot time as a module.
  • Find the biggest symbols in the compiled kernel, for those which could be unnecessary, study the corresponding Makefile to see how not to compile them, if possible.
• Check which kernel compression option is the fastest (gzip/lzma/xz/lzo/lz4/zstd or uncompressed - I love zstd but most likely lzo or lz4 is the winner, since pretty unclear if zstd support can easily be added and if the benefit is worth it).
• Deferring drivers and initcalls - If you can’t compile a feature as a module (e.g. networking or block subsystem), you can try to defer its execution.
• Scale the CPU frequency to maximum (or some thermally limited maximum) during boot. The faster the CPU, the faster the boot.
• Turning off console output (Disable it by passing the quiet argument on the kernel command line).

Documentation links:

• https://medium.com/smartmechatronics/embedded-linux-boot-time-reduction-eda5f5b4c014
• https://bootlin.com/doc/training/boot-time/boot-time-slides.pdf
• https://elinux.org/images/6/64/Chris-simmonds-boot-time-elce-2017_0.pdf
• https://www.e-consystems.com/articles/Product-Design/Linux-Boot-Time-Optimization-Techniques.asp
• https://www.toradex.com/blog/embedded-linux-boot-time-optimization
• https://source.android.com/docs/core/architecture/kernel/boot-time-opt

[adeebshihadeh]

We already use quiet to get the initial speedup of like 11s -> 3s. I suspect we'll get the most savings from disabling and deferring. Anything openpilot doesn't use is fine to disable. Also keep in mind that deferring something critical but it costs time later in bootup doesn't count, but something non-essential to driving like WiFi is fine.