-
Notifications
You must be signed in to change notification settings - Fork 1
Creating a 32bit container on an AArch64 server
This is a list of the problems (and solutions to most of them) I had when trying to set up a 32bits container on a fast AArch64 server for 32bits ARM development.
I used systemd-nspawn
when setting up a 32bits (i686) container on a x64 host.
It was convenient on x64 since it supports a --personality
option which can be used to set the uname -m
to
i686
(same as setarch i686
or linux32
).
However, when I tried to use it on AArch64, the container is killed by SYS
when I tried to run it
(see here for someone else who has the same problem).
The --personality
optional doesn't seem to work on AArch64 either.
The problem could be related to some of the issues below but I haven't tested it again after fixing those
problems yet.
The alternative I've tried is lxc
, which seems to work much better on AArch64. It also comes with templates
for many distros making it easier to setup a cantainer for a different distro.
lxc
doesn't seem to have a builtin option for setting uname -m
and the standard way to do that seems to be replacing the init
in the container.
The init command can be configured using the lxc.init_cmd
option
so a simple solution to set uname -m
is to create an executable script
(/sbin/init-32
for example) with the content
#!/bin/bash
exec linux32 /sbin/init
and change lxc.init_cmd
in the LXC config file to the path of the script.
Since I'm not really interested in isolating the network in the container I set lxc.network.type
to none
,
which makes the container using the same network as the host.
(This can probably cause security issues for other use case.)
Some distros (CentOS and Debian both seem to do this) turns off the network on shutdown causing rebooting the
container to reset the host network and in general, I only want to share the network to the container in a
read-only way.
This is controlled by the NET_ADMIN
capability so adding net_admin
to lxc.cap.drop
in the
LXC solves this issue.
There seem to be multiple way to achieve this. The method I used is to use lxc-autostart
.
This is a service (lxc-auto.service
) that automatically (re?)start containers marked as lxc.start.auto = 1
.
Simply adding the option to the LXC config file and enable/start the lxc-auto
service will make sure that
the container is started automatically at host boot time.
An example config file for a 32bits archlinux container
lxc.network.type = none
lxc.rootfs = /var/lib/lxc/arch32/rootfs
lxc.rootfs.backend = dir
lxc.utsname = arch32
lxc.arch = armv7h
lxc.include = /usr/share/lxc/config/archlinux.common.conf
lxc.start.auto = 1
lxc.init_cmd = /sbin/init-32
lxc.cap.drop = setfcap sys_nice sys_pacct sys_rawio net_admin
Certain kernels (noticeably the one ships with GigaByte MP30-AR0
and
the stock ArchLinux ARM linux-aarch64
kernel) do not support 32bits ELF file.
This is controlled by the kernel option CONFIG_COMPAT
which requires either CONFIG_EXPERT
or 4k page size (more on this later).
The default (only?) page size on ARM (32bits) is 4k and some programs/binaries assumes this. In particular, the dynamic loader / kernel refuses to load any executable who's segment alignments are not a multiple of the page size. Latest binutils always uses at least 64k alignments but the binaries compiled with earlier versions of binutils do not (some in archlinux and most in debian 7) so it is better to set the page size to 4k for maximum compatibility with 32bits applications.
When setting the the page size, also make sure to decrease the minimum allowed mmap address to be the same
as the page size. Otherwise, non-privilege process will not be able to map executables compiled with a low
load address and can cause SegFault at exec
time. (errno
is permission denied).
Some armv6 instructions are not supported by the hardware anymore and the kernel emulation for those instructions needs to be turned on in order to run those applications.
The options to enable are CONFIG_SWP_EMULATION
, CONFIG_CP15_BARRIER_EMULATION
and CONFIG_SETEND_EMULATION
under CONFIG_ARMV8_DEPRECATION
.
Ref this question on ARM community.
Thanks to wookey
from the linarno IRC for providing the options to enable instruction emulation and 32bits ELF support.
LLVM handles one of the relocation incorrectly in the JIT causing segfault for certain 64bit immediates. This seems to be triggered only by some kernel configurations (likely due to the address ranges the kernel loads libraries into).
Should be fixed in LLVM 4.0 by https://reviews.llvm.org/D27609