Creating a 32bit container on an AArch64 server

This is a list of the problems (and solutions to most of them) I had when trying to set up a 32bits container on a fast AArch64 server for 32bits ARM development.

LXC vs systemd-nspawn

I used systemd-nspawn when setting up a 32bits (i686) container on a x64 host. It was convenient on x64 since it supports a --personality option which can be used to set the uname -m to i686 (same as setarch i686 or linux32).

However, when I tried to use it on AArch64, the container is killed by SYS when I tried to run it (see here for someone else who has the same problem). The --personality optional doesn't seem to work on AArch64 either. The problem could be related to some of the issues below but I haven't tested it again after fixing those problems yet.

The alternative I've tried is lxc, which seems to work much better on AArch64. It also comes with templates for many distros making it easier to setup a cantainer for a different distro.

LXC init

lxc doesn't seem to have a builtin option for setting uname -m and the standard way to do that seems to be replacing the init in the container.

The init command can be configured using the lxc.init_cmd option so a simple solution to set uname -m is to create an executable script (/sbin/init-32 for example) with the content

#!/bin/bash
exec linux32 /sbin/init

and change lxc.init_cmd in the LXC config file to the path of the script.

LXC network

Since I'm not really interested in isolating the network in the container I set lxc.network.type to none, which makes the container using the same network as the host. (This can probably cause security issues for other use case.)

Some distros (CentOS and Debian both seem to do this) turns off the network on shutdown causing rebooting the container to reset the host network and in general, I only want to share the network to the container in a read-only way. This is controlled by the NET_ADMIN capability so adding net_admin to lxc.cap.drop in the LXC solves this issue.

LXC autostart

There seem to be multiple way to achieve this. The method I used is to use lxc-autostart. This is a service (lxc-auto.service) that automatically (re?)start containers marked as lxc.start.auto = 1. Simply adding the option to the LXC config file and enable/start the lxc-auto service will make sure that the container is started automatically at host boot time.

An example config file for a 32bits archlinux container

lxc.network.type = none
lxc.rootfs = /var/lib/lxc/arch32/rootfs
lxc.rootfs.backend = dir
lxc.utsname = arch32
lxc.arch = armv7h
lxc.include = /usr/share/lxc/config/archlinux.common.conf
lxc.start.auto = 1
lxc.init_cmd = /sbin/init-32
lxc.cap.drop = setfcap sys_nice sys_pacct sys_rawio net_admin

32bits ELF

Certain kernels (noticeably the one ships with GigaByte MP30-AR0 and the stock ArchLinux ARM linux-aarch64 kernel) do not support 32bits ELF file. This is controlled by the kernel option CONFIG_COMPAT which requires either CONFIG_EXPERT or 4k page size (more on this later).

Page size, min mmap address

The default (only?) page size on ARM (32bits) is 4k and some programs/binaries assumes this. In particular, the dynamic loader / kernel refuses to load any executable who's segment alignments are not a multiple of the page size. Latest binutils always uses at least 64k alignments but the binaries compiled with earlier versions of binutils do not (some in archlinux and most in debian 7) so it is better to set the page size to 4k for maximum compatibility with 32bits applications.

When setting the the page size, also make sure to decrease the minimum allowed mmap address to be the same as the page size. Otherwise, non-privilege process will not be able to map executables compiled with a low load address and can cause SegFault at exec time. (errno is permission denied).

Deprecated instructions

Some armv6 instructions are not supported by the hardware anymore and the kernel emulation for those instructions needs to be turned on in order to run those applications.

The options to enable are CONFIG_SWP_EMULATION, CONFIG_CP15_BARRIER_EMULATION and CONFIG_SETEND_EMULATION under CONFIG_ARMV8_DEPRECATION.

Ref this question on ARM community. Thanks to wookey from the linarno IRC for providing the options to enable instruction emulation and 32bits ELF support.

LLVM Codegen error

LLVM handles one of the relocation incorrectly in the JIT causing segfault for certain 64bit immediates. This seems to be triggered only by some kernel configurations (likely due to the address ranges the kernel loads libraries into).

Should be fixed in LLVM 4.0 by https://reviews.llvm.org/D27609