SNO seed image creation and restoration using Image Base Upgrade (IBU)

Note that this repo is just a proof-of-concept. This repo is for debugging / experimenting with single node OpenShift **

Note that single node OpenShift relocation is currently unsupported.

Prerequisites

• NMState v2.2.10 or above, this is required due to the nmstate config used in the agent-config.yaml

sudo dnf copr enable nmstate/nmstate-git
sudo dnf install nmstate
- Set `PULL_SECRET` environment variable to the contents of your cluster pull secret
- Set `BACKUP_SECRET` environment variable with the credentials needed to push/pull the seed image, in standard pull-secret format

• virt-install

sudo dnf install virt-install

Fedora prerequisites

If using systemd-resolved as a system DNS resolver

Show more info

Add the NetworkManager dnsmasq instance as a DNS server for resolved:

sudo mkdir /etc/systemd/resolved.conf.d

Then create /etc/systemd/resolved.conf.d/dns_servers.conf with:

[Resolve]
DNS=127.0.0.1
Domains=~.

And finally restart systemd-resolved:

sudo systemctl restart systemd-resolved

If using authselect as nsswitch manager

Show more info

Install libvirt-nss

sudo dnf install libvirt-nss

Add authselect libvirt feature

sudo authselect enable-feature with-libvirt

This makes it so that libvirt guest names resolve to IP addresses

Procedure

Generate the seed image template

To generate a seed image we want to:

• Provision a VM and install SNO in it

make seed-vm-create wait-for-seed

• (OPTIONAL) Modify that installation to suit the use-case that we want to have in the seed image. In this example we install the components of a vDU profile

make vdu

• Prepare the seed cluster to have a couple of needed extras

make dnsmasq-workaround seed-varlibcontainers

• Create a seed image from that SNO cluster

make seed-image-create SEED_IMAGE=quay.io/whatever/repo:tag

This will run ibu-imager to create an OCI seed image

WARNING Once we create a seed-image, some changes will be applied to seed VM that will "break" the node. If you intend on reusing that seed VM, it would be wise to create a backup using make seed-vm-backup that you can restore afterwards

Restore seed image

To restore a seed image we will use LifeCycle Agent, and manage everything with the CR ImageBasedUpgrade

The steps will be as follow:

• Provision a VM and install SNO in it

make recipient-vm-create wait-for-recipient

• Prepare the recipient cluster to use /var/lib/containers in a directory that can be shared among different deployments

make recipient-varlibcontainers

• Restore the seed image

make seed-image-restore SEED_IMAGE=quay.io/whatever/repo:tag

• Reboot into the new deployment

virsh reboot recipient

Extra goodies

Descriptive help for the Makefile

You can run

make help

and get a description of the main Makefile targets that you can use

Backup and reuse VM qcow2 files

To be able to reuse the VMs, we can backup the qcow2 files of both seed and recipient VM This will allow us to skip the initial provision, allowing for faster iterations when testing To create a backup run:

make seed-vm-backup

or

make recipient-vm-backup

To restore an image, we run the complementary restore command

make seed-vm-restore

or

make recipient-vm-restore

IMPORTANT: Remember that certificates expire, so if a backed up image is old, certificates will expire and openshift wont be usable TODO: Apply recert after restoring the image

vDU profile

A vDU profile can be applied to the image before baking with

make vdu

Use shared directorty for /var/lib/containers

A shared directory /sysroot/containers can be used to mount and share /var/lib/containers among ostree deployments Run:

make seed-varlibcontainers

or

make recipient-varlibcontainers

This will create a /sysroot/containers in the SNO (when not specifying the cluster with the CLUSTER variable, it defaults to the seed image) to be mounted in /var/lib/containers The use case for this is to easily precache all the images that the cluster in the seed image will need, while original recipient cluster is still running

WARNING

It is important to note that for precaching to work, this change must be applied both in seed image and recipient cluster

Examples

Full run with vDU profile

Prerequisites

Let's first define a few environment variables:

SEED_IMAGE=quay.io/whatever/ostbackup:seed
export PULL_SECRET=$(jq -c . /path/to/my/pull-secret.json)
export BACKUP_SECRET=$(jq -c . /path/to/my/repo/credentials.json)

Creation of relocatable image

• Create seed VM and image

make seed-vm-create wait-for-seed vdu dnsmasq-workaround ostree-shared-containers seed-image-create SEED_IMAGE=$SEED_IMAGE

• Create recipient SNO and restore seed image

make recipient-vm-create wait-for-recipient ostree-shared-containers seed-image-restore SEED_IMAGE=$SEED_IMAGE
virsh reboot recipient

Installing the backup into some running SNO

make seed-image-restore SNO_KUBECONFIG=path/to/recipient/sno/kubeconfig SEED_IMAGE=$SEED_IMAGE

• Reboot the recipient host