vmtest
enables you to quickly and programmatically run tests inside a virtual
machine.
This can be useful in the following, non-exhaustive, list of scenarios:
- You ship a virtual machine image and you want to programmatically test the image during development, both locally and in CI.
- You develop eBPF-powered applications and you want to run your application tests on a variety of kernels your application supports, both locally and in CI.
- You are a kernel developer and you want to quickly iterate on changes.
A key feature is that the root host userspace can-be/is transparently mapped
into the guest VM. This makes dropping vmtest
into existing CI workflows
easy, as dependencies installed on the root host can also be effortlessly
reused inside the guest VM.
The following are required dependencies, grouped by location:
Host machine:
Virtual machine image:
qemu-guest-agent
- Kernel 9p filesystem support, either compiled in or as modules (see kernel
dependencies)
- Most (if not all) distros already ship support as modules or better
Kernel:
CONFIG_VIRTIO=y
CONFIG_VIRTIO_PCI=y
CONFIG_VIRTIO_CONSOLE=y
CONFIG_NET_9P=y
CONFIG_NET_9P_VIRTIO=y
CONFIG_9P_FS=y
Note the virtual machine image dependencies are only required if you're using
the image
target parameter. Likewise, the same applies for kernel
dependencies.
Assuming you have a rust toolchain
installed, simply
run:
$ cargo install vmtest
Alternatively, vmtest
publishes statically linked binaries in its release
assets. Currently only x86-64-linux
is published.
The config file interface is more powerful and unlocks all vmtest
features.
However it can be a bit heavyweight if you're just trying to do something
one-off. For such lighter-weight cases, vmtest
has a one-liner interface.
For example, to run an arbitrary command in the guest VM with a different kernel:
$ vmtest -k ./bzImage-v6.2 "uname -r"
=> bzImage-v6.2
===> Booting
===> Setting up VM
===> Running command
6.2.0
To run an arbitrary command in a guest VM with a different kernel and rootfs:
$ vmtest -k ./bzImage-v6.2 -r ./rootfs "uname -r"
=> bzImage-v6.2
===> Booting
===> Setting up VM
===> Running command
6.2.0
To run an arbitrary command from a kernel from another architecture in a guest VM:
$ vmtest -k ./kernels/Image-arm64 -r ./rootfs/ubuntu-lunar-arm64 -a aarch64 "uname -r"
=> Image-arm64
===> Booting
===> Setting up VM
===> Running command
6.6.0-rc5-ga4a0c99f10ca-dirty
It is also possible to get an interactive shell prompt in the guest by using the command -
:
vmtest -k ./bzImage-v6.2 "-"
...
...
root@(none):/#
See vmtest --help
for all options and flags.
vmtest
by default reads from vmtest.toml
in the current working directory.
vmtest.toml
, in turn, describes which targets should be run.
For example, consider the following vmtest.toml
:
[[target]]
name = "AWS kernel"
kernel = "./bzImage-5.15.0-1022-aws"
command = "uname -r | grep -e aws$"
[[target]]
name = "OCI image"
image = "./oci-stage-6/oci-stage-6-disk001.qcow2"
command = "ls -l /mnt/vmtest && cat /proc/thiswillfail"
[[target]]
name = "Foreign Architecture"
kernel = "./kernels/Image-arm64"
arch = "aarch64"
rootfs = "./rootfs/ubuntu-lunar-arm64"
command = "uname -m | grep aarch64"
In the above config, two see two defined targets: "AWS kernel" and "OCI image".
In plain english, the "AWS kernel" target tells vmtest to run command
in a VM
with the same userspace environment as the host, except with the specified
kernel
.
"OCI image", on the other hand, tells vmtest to run command
inside the
provided VM image. The image completely defines the environment command
is
run in with the exception of /mnt/vmtest
. /mnt/vmtest
(as we will see
below) contains the full directory tree of the host machine rooted at the
directory containing vmtest.toml
. This directory tree is shared - not
copied - with both readable and writable permissions.
Running vmtest with the above config yields the following results:
$ vmtest
=> AWS kernel
PASS
=> OCI image
===> Booting
===> Setting up VM
===> Running command
total 2057916
drwxr-xr-x 1 ubuntu ubuntu 200 Nov 14 20:41 avx-gateway-oci-stage-6
-rw-r--r-- 1 ubuntu ubuntu 11631520 Feb 1 00:33 bzImage-5.15.0-1022-aws
-rw-r--r-- 1 ubuntu ubuntu 359 Feb 4 01:41 vmtest.toml
cat: /proc/thiswillfail: No such file or directory
Command failed with exit code: 1
FAILED
=> Foreign Architecture
===> Booting
===> Setting up VM
===> Running command
aarch64
For full configuration documentation, see config.md.
vmtest-action is a convenient
wrapper around vmtest
that is designed to run inside Github Actions. See
vmtest-action
documentation for more details.
For general architecture notes, see architecture.md.
Many thanks to drgn
's
vmtest by Omar Sandoval and
Andy Lutomirski's most excellent virtme
for providing both ideas and technical exploration.