-
Notifications
You must be signed in to change notification settings - Fork 1
Creating a 32bit container on an AArch64 server
This is a list of the problems (and solutions to most of them) I had when trying to set up a 32bits container on a fast AArch64 server for 32bits ARM development.
I used systemd-nspawn when setting up a 32bits (i686) container on a x64 host.
It was convenient on x64 since it supports a --personality option which can be used to set the uname -m to
i686 (same as setarch i686 or linux32).
However, when I tried to use it on AArch64, the container is killed by SYS when I tried to run it
(see here for someone else who has the same problem).
The --personality optional doesn't seem to work on AArch64 either.
The problem could be related to some of the issues below but I haven't tested it again after fixing those
problems yet.
The alternative I've tried is lxc, which seems to work much better on AArch64. It also comes with templates for many distros making it easier to setup a cantainer for a different distro.
Local template is deprecated so the way to create a container is to use the download template.
ArchLinux container for amd64 (x86_64), arm64 (aarch64) and armhf (armv7h) are supported.
The command to use for an armv7 container is
lxc-create --name=<container_name> -t download -- --dist archlinux --release current --arch armhf
Old version of lxc requires manually setting the arch by changing the init in the container. As of version 4, this does not seem to be a problem anymore and the ELF arch is set correctly by default.
Since I'm not really interested in isolating the network in the container I set lxc.net.0.type to none,
which makes the container using the same network as the host.
(This can probably cause security issues for other use case.)
Some distros (CentOS and Debian both seem to do this) turns off the network on shutdown causing rebooting the
container to reset the host network and in general, I only want to share the network to the container in a
read-only way.
This is controlled by the NET_ADMIN capability so adding net_admin to lxc.cap.drop in the
LXC solves this issue.
There seem to be multiple way to achieve this. The method I used is to use lxc-autostart.
This is a service (lxc-auto.service) that automatically (re?)start containers marked as lxc.start.auto = 1.
Simply adding the option to the LXC config file and enable/start the lxc-auto service will make sure that
the container is started automatically at host boot time.
An example config file for a 32bits archlinux container
# Template used to create this container: /usr/share/lxc/templates/lxc-download
# Parameters passed to the template: --dist archlinux --release current --arch armhf
# Template script checksum (SHA-1): 9893b2e0dba7be0d74cf38537bebe0af939c269c
# For additional config options, please look at lxc.container.conf(5)
# Uncomment the following line to support nesting containers:
#lxc.include = /usr/share/lxc/config/nesting.conf
# (Be aware this has security implications)
# Distribution configuration
lxc.include = /usr/share/lxc/config/common.conf
lxc.arch = armv7l
# Container specific configuration
lxc.rootfs.path = dir:/var/lib/lxc/arch32/rootfs
lxc.uts.name = arch32
# Network configuration
lxc.net.0.type = none
lxc.start.auto = 1
lxc.cap.drop = setfcap sys_nice sys_pacct sys_rawio net_admin
Certain kernels (noticeably the one ships with GigaByte MP30-AR0 and
the stock ArchLinux ARM linux-aarch64 kernel) do not support 32bits ELF file.
This is controlled by the kernel option CONFIG_COMPAT
which requires either CONFIG_EXPERT or 4k page size (more on this later).
The default (only?) page size on ARM (32bits) is 4k and some programs/binaries assumes this. In particular, the dynamic loader / kernel refuses to load any executable who's segment alignments are not a multiple of the page size. Latest binutils always uses at least 64k alignments but the binaries compiled with earlier versions of binutils do not (some in archlinux and most in debian 7) so it is better to set the page size to 4k for maximum compatibility with 32bits applications.
When setting the the page size, also make sure to decrease the minimum allowed mmap address to be the same
as the page size. Otherwise, non-privilege process will not be able to map executables compiled with a low
load address and can cause SegFault at exec time. (errno is permission denied).
Some armv6 instructions are not supported by the hardware anymore and the kernel emulation for those instructions needs to be turned on in order to run those applications.
The options to enable are CONFIG_SWP_EMULATION, CONFIG_CP15_BARRIER_EMULATION and CONFIG_SETEND_EMULATION
under CONFIG_ARMV8_DEPRECATION.
Ref this question on ARM community.
Thanks to wookey from the linarno IRC for providing the options to enable instruction emulation and 32bits ELF support.
These are not necessarily related to creating a 32bit container on an aarch64 machine but are some issues I had when using LXC in general.
On Ubuntu lxc-auto doesn't seem to exist but lxc.server does which seems to cover the function of lxc-auto.
On Ubuntu 18.04 with the Quancomm kernel for the RB5 board and lxc 3.0.2, using schedtune cgroup causes an error when trying to start an lxc instance since the kernel does not like nested cgroup for schedtune which lxc wants to do (LXC creates an lxc sub group and a group for the container under that). The error code was printed as a memory allocation error which is quite confusing though the dmesg output was more reasonable: "Nested SchedTune boosting groups not allowed". Disabling the use of this cgroup globally in /etc/lxc/lxc.conf using lxc.cgroup.use key worksaround the problem.
It causes some issues with logind etc in the container so I disabled it on the RB5 board. To be fair, some services in the host also failed so it might not be a apparmor problem rather bad config from thundercomm... I worked around this by disabling apparmor in the kernel command line.
When trying to create nested environment (e.g. when building a chroot package), systemd-nspawn by default (called by makechrootpkg) copied the static DNS config to the chroot which talks to resolved on 127.0.0.53. On the RB5 board when using the none network, the resolved from the parent disabled the version in the lxc which somehow causes the chroot DNS resolving to fail (the listened port is visible from the container but somehow the chroot cannot use it...) Disabling the DNSStubListener=no in /etc/systemd/resolved.conf fixed this for me... (probably not the best solution).