Add testing on OpenShift 4

[michaelsauter]

Right now we have a public AMI which runs OpenShift 3.11 using oc cluster up. The code for this is in ods-devenv.

We need to have E2E tests for ODS on OpenShift 4 as well. The question is how we want to achieve this.

OpenShift 4 does not have oc cluster up anymore. Instead, https://github.com/code-ready/crc should be used. Unfortunately, the standard crc needs an image pull secret, which is personal and therefore cannot be shared.

Since a few weeks, there is an OKD variant of crc available at https://www.okd.io/crc.html, which does not need the image pull secret. However, there might be (significant) differences to an actual OpenShift installation.

Further, I tried to run the OKD variant of crc in GitHub Actions in https://github.com/michaelsauter/crc-okd-action-test/runs/1478942209?check_suite_focus=true, which fails due to not enough memory:

CodeReady Containers version: 1.19.0-OKD+deb39a0
OpenShift version: 4.5.0-0.okd-2020-10-15-235428 (embedded in executable)
...
crc requires at least 9.664GB to run
Error: Process completed with exit code 1.

FYI @clemensutschig @metmajer @georgfedermann

[stitakis]

@michaelsauter regarding

OpenShift 4 does not have oc cluster up anymore. Instead, https://github.com/code-ready/crc should be used. Unfortunately, the standard crc needs an image pull secret, which is personal and therefore cannot be shared.

We could manually download the image and push it to a S3 bucket everytime there is a new version. By using always the same resource name and having S3 versioning enabled we could simulate a kind of image stream. I think S3 resources are accessible for the packer AMI build. This could solve the problem. What do you think?

[michaelsauter]

@stitakis Hmm, could be part of the solution. I need to read up in which places this image pull secret is used. I guess it is also used to pull base container images, not just images for OpenShift itself.

[clemensutschig]

@michaelsauter - could we just build (silently) - and not produce an AMI? at least we would have a solid way to test ocp4

[stitakis]

@clemensutschig, we don't need to produce an image to test ocp4 with ods at first instance. The image is currently produced for the corporate devs to be able to create an instance of ODS (called ODSBox/EDPBox). What we need is to identify a ocp4 distribution that we could install with a script and configure with a script.
@michaelsauter, this image that you suggested https://www.okd.io/crc.html (Run a developer instance of OKD4 on your local workstation with CodeReady Containers built for OKD No Pull Secret Required!) should be the correct one, have you tried it?

[clemensutschig]

@stitakis - I think it's worth a try ... https://code-ready.github.io/crc/ - seems pretty easy to make this happen ... can you try to give this a shot on a new ami instance ...

looks like many of the small things we don't need anymore - because they are part of the image I guess? (e.g. dnsmasq or alike) ... but this needs a lot of investigation ..

steps:

1. run setup from the deploy.sh to get all the atlassian stuff ... (or use ods in a box :))
2. download crc onto the ods in a box instance - and start it up ...
3. see if you can connect from the ods install on the crc vm to a process running on the AMI /e.g. bitbucket thru installation ods ..

[stitakis]

@clemensutschig sure, I'll give a try and come back to you

[stitakis]

@clemensutschig @michaelsauter an update on this... I wasn't able to install CRC in an ec2 instance (ODSBox) due to this error: You need to enable virtualization in BIOS

[openshift@ip-172-31-2-109 ~]$ ./crc version
CodeReady Containers version: 1.20.0-OKD+91e49901
OpenShift version: 4.6.0-0.okd-2020-12-12-135354 (embedded in executable)
[openshift@ip-172-31-2-109 ~]$ ./crc setup
INFO Checking if podman remote executable is cached
INFO Checking if admin-helper executable is cached
INFO Caching admin-helper executable             
INFO Will use root access: change ownership of /home/openshift/.crc/bin/admin-helper
[sudo] password for openshift:
INFO Will use root access: set suid for /home/openshift/.crc/bin/admin-helper
INFO Checking if CRC bundle is extracted in '$HOME/.crc'
INFO Extracting bundle from the CRC executable   
crc.qcow2: 11.97 GiB / 11.97 GiB [------------------------------------------------------------------------------------------------------------------] 100.00%
INFO Checking minimum RAM requirements           
INFO Checking if running as non-root             
INFO Checking if Virtualization is enabled       
INFO Setting up virtualization                   
You need to enable virtualization in BIOS

We can't enable virtualization in an virtualized environment like ec2.

The `crc documentation (https://code-ready.github.io/crc/) explains:

CodeReady Containers includes the crc command-line interface (CLI) to interact with the CodeReady Containers virtual machine running the OpenShift cluster.

Also this page (https://github.com/code-ready/crc) says that crc is for use on local laptop or desktop computer.

Conclusion, crc is not meant to be installed in any virtualized environment like ec2.

It looks to me that we will need to setup a oc4 cluster (e.g. deployed AWS) to be able to test ods.

[metmajer]

@stitakis this comes unexpected and would render ODS in a Box useless 👎

[michaelsauter]

Hmm. Looks like CRC would require something like https://aws.amazon.com/about-aws/whats-new/2018/05/announcing-general-availability-of-amazon-ec2-bare-metal-instances/. Maybe we could try this?

On the other hand, maybe the actual OpenShift installer is easier to use now and for AWS it would be best to run that (instead of CRC)?

[stitakis]

@metmajer @michaelsauter another alternative would be to figure out how to install a single-node cluster on ec2. However the main problem is that RedHat doesn't offer an oc4 distribution that we could easily install in an ec2 instance. This is the main issue and big concern regarding the ODS in a Box. Do we have any promising way to address this on RedHat?

[michaelsauter]

Another piece I found on this is https://gist.github.com/tmckayus/8e843f90c44ac841d0673434c7de0c6a ... not sure if helpful?

[felipecruz91]

I've played around with CRC a few weeks ago in my laptop. If it's just for testing purposes, provisioning an EC2 bare metal instance (check Spot instance to save money) should be enough to get CRC up and running (that should come with VT-x enabled).

[metmajer]

Cool @felipecruz91, that helps. Development and testing environment only.

[stitakis]

@felipecruz91 @michaelsauter @ I'll give EC2 bare metal a try

[stitakis]

@clemensutschig @felipecruz91 @michaelsauter @metmajer good news before the weekend starts... I managed to install crc in a clean centos 7 bare metal ec2 instance. It looks like future of odsbox is saved!
This is the log of crc start:

centos@ip-172-31-24-128 ~]$ crc status
CRC VM:          Running
OpenShift:       Running (v4.6.0-0.okd-2020-12-12-135354)
Disk Usage:      13.98GB of 32.74GB (Inside the CRC VM)
Cache Usage:     16.18GB
Cache Directory: /home/centos/.crc/cache

[metmajer]

@stitakis @jorge-romero @felipecruz91 we are approaching the release of ODS 4.x and plan to go live in July. Without a working ODS in a Box including CRC, we are unable to provide training environments around the EDP based on ODS 4.x. @stitakis can you please dedicate some time to this? Dedicated hardware in AWS for ODS in a Box is a very pricey option.

FYI @michaelsauter @clemensutschig

[felipecruz91]

@metmajer @stitakis After some research, I see the following alternatives:

• Customize the OKD 4 installation process sping up a single-node cluster in AWS:

You'll be prompted to choose a platform to install to - AWS is currently the best place to start with OKD4 while we get Fedora CoreOS machine images set up in the other clouds.

The good news is, in this video OKD 4 is installed in an m5.xlarge machine which is far cheaper than a bare-metal one.

However, this approach would mean we need to maintain and communicate 2 systems in AWS that need to talk to each other (EC2 instance with the Atlassian Stack and an OKD 4 instance), thus vanishing the "all-in-a Box" model.

• Given that the main limitation is that OKD 4 needs to run on machines with nested virtualization enabled, we might as well look into other cloud providers that offer a better price? For instance, GCP offers the general-purpose n1 machines e.g n1-standard-16 (16vCPU, 60GB RAM) at $0.9792/hour or $508.57/month in Frankfurt and create a single-node cluster in it.

[segator]

AWS is nice for native cloud applications but if you just need a bunch of VM's have a look on hetzner.com, super cheap VM's + nested virtualtization.

dedicated cpu
166€
32c
64g ram

[michaelsauter]

Based on @felipecruz91's comment, I googled a bit with @stitakis and we came across https://github.com/danielmenezesbr/crc-on-gcp-terraform. Maybe a direction to investigate? I also saw a comment that someone installed CRC on CentOS on Azure. So maybe only AWS has this limitation?

[tbugfinder]

Spot pricing for AWS metal instance types drops prices by ~70% (e.g. for c5n.metal).
ARM a1.metal (16vCPUs, 32GB) would be cheapest entrypoint, though.

Azure offers different types with VT-x enabled hardware.
E.g. https://docs.microsoft.com/en-us/azure/virtual-machines/dv3-dsv3-series

[metmajer]

Thanks @tbugfinder, I guess that price on AWS is quite a bit more expensive than the instance type we're using today @jorge-romero? Switching to Azure would mean quite a few changes to adapt EDP in a Box creation and management.

[tbugfinder]

My understanding is that an ODS installation on top of CodeReadyContainers (CRC) is most beneficial option.

• it enables local development environments (PC HW or Macs)
• it does not depend on a cloud provider (AWS, Azure, GCP, ...)

Therefore, using ec2 metal for developing the setup now, is definitely an option - ec2 metal should be still in budget.
After that, running it locally or in AWS or Azure is just a way of using it.

[stitakis]

@metmajer @tbugfinder maybe could help us to decide on the roadmap if we do an analysis that take into account a budget limit and a cost/benefit analysis. Migrating to Azure have the additional effort of migrating the ODS Box build infrastructure including also the EDP CI infrastructure.
Regarding the ODS Box build infrastructure, there are some optimizations that we have in the backlog that could help us to opmitize the utilization and reduce overall cost.

[BraisVQ]

@victorpablosceruelo @metmajer we have found that node18 cannot be build in Centos7 and Rhel7 here, this makes it not compatible with our current ODSBox which uses Centos7

[victorpablosceruelo]

@braisvq1996 Centos8 died (stopped having security upgrades) in December 2021 and Centos7 will die in June 2024. We still have ~2 years, but we should stop spending efforts in Centos7.
Some important issues we should take into account:

Centos End Of Life:
https://www.centos.org/centos-linux-eol/

Graphical representation of the end of life differences between RH and Centos:
https://linuxlifecycle.com/

And here, what is Centos Stream (no more an old but stable version of RH):
https://www.centos.org/centos-stream/

Nevertheless, I think we should split this discussion in 2: the Linux os distro we have in the boxes (which if I am right needs to have ocp3 or ocp4) and the Linux os distro of the Jenkins slaves (which in my humble opinion should be upgraded as soon as possible).

For the first ones, the evaluation above lacks a conclusion which might not be easy.
For the second ones maybe we should decide to start using UBI 8 for containers. See: https://developers.redhat.com/products/rhel/ubi

And this might become the "good answer" to all our problems, but it is still being evaluated:
https://github.com/opendevstack/ods-pipeline

[michaelsauter]

Nevertheless, I think we should split this discussion in 2: the Linux os distro we have in the boxes (which if I am right needs to have ocp3 or ocp4) and the Linux os distro of the Jenkins slaves (which in my humble opinion should be upgraded as soon as possible).
For the first ones, the evaluation above lacks a conclusion which might not be easy.
For the second ones maybe we should decide to start using UBI 8 for containers. See: https://developers.redhat.com/products/rhel/ubi

I think this is really one discussion only. For OpenShift 4, only UBI makes sense and ODS already has UBI images for everything. And at the same time, you need a development environment that uses OpenShift 4.

So in my view, the direction should simply be:

1. Move development environment to OpenShift 4
2. Drop all RHEL/Centos images