Unable to build SNC for OKD using snc.sh

[rohanKanojia]

Describe the Issue

⚠️ I'm not 100% sure whether it's an actual issue or whether some problem in snc. Most probably it's due to some misconfiguration on my side.

When I'm trying to follow the instructions provided on Building SNC for OKD 4, my script seems to be stuck at this phase since forever:

+ sudo virt-install --name crc --vcpus 6 --memory 14336 --arch=x86_64 --disk path=/var/lib/libvirt/crc/crc.qcow2,size=31 --network network=crc,mac=52:54:00:ee:42:e1 --os-variant rhel9-unknown --nographics --cdrom /var/lib/libvirt/crc/rhcos-live.iso --events on_reboot=restart --autoconsole none --boot uefi --wait

Starting install...
Allocating 'crc.qcow2'                                                                          | 3.4 MB  00:00:00 ... 
Creating domain...                                                                              |    0 B  00:00:00     

Domain is still running. Installation may be in progress.
Waiting for the installation to complete.

I'm trying this for the first time, so it could be due to some misconfiguration from my side.

Check virsh seems to indicate that crc is still running:

snc : $ virsh -c qemu:///system list --all
 Id   Name       State
---------------------------
 1    crc        running
 -    minikube   shut off

SNC version

I was trying to build OKD bundle based on this branch 1628dfa (it already got merged to main)

Component

I'm not sure which component this issue belongs to, perhaps it could be part of release pipeline.

Steps to reproduce

Here are the steps I followed:

• Create Pull Secret file

cat << EOF > /tmp/pull_secret.json
{"auths":{"fake":{"auth": "Zm9vOmJhcgo="}}}
EOF

• Set environment for OKD version export OKD_VERSION=4.16.0-0.okd-scos-2024-08-21-155613
• Set environment for OpenShift Pull Secret Path export OPENSHIFT_PULL_SECRET_PATH="/tmp/pull_secret.json"
• Execute SNC script ./snc.sh

Expected behavior

The expected behavior for the script is to terminate at some point, the script could be improved to show some verbose logs so that the user knows what's happening in the background.

Environment

Linux (Fedora 40)

Linux fedora 6.10.11-200.fc40.x86_64 #1 SMP PREEMPT_DYNAMIC Wed Sep 18 21:09:58 UTC 2024 x86_64 GNU/Linux

SNC Script Complete Logs

+ export LC_ALL=C.UTF-8
+ LC_ALL=C.UTF-8
+ export LANG=C.UTF-8
+ LANG=C.UTF-8
+ source tools.sh
++ JQ=jq
++ QEMU_IMG=qemu-img
++ VIRT_FILESYSTEMS=virt-filesystems
++ GUESTFISH=guestfish
++ VIRSH=virsh
++ VIRT_INSTALL=virt-install
++ XMLLINT=xmllint
++ DIG=dig
++ UNZIP=unzip
++ ZSTD=zstd
++ CRC_ZSTD_EXTRA_FLAGS='--ultra -22'
++ HTPASSWD=htpasswd
++ PATCH=patch
+++ uname -m
++ ARCH=x86_64
++ case "${ARCH}" in
++ yq_ARCH=amd64
++ SNC_GENERATE_MACOS_BUNDLE=1
++ SNC_GENERATE_WINDOWS_BUNDLE=1
++ SNC_GENERATE_LINUX_BUNDLE=1
++ test -z
++ echo 'Downloading yq binary to manipulate yaml files'
Downloading yq binary to manipulate yaml files
++ curl -L https://github.com/mikefarah/yq/releases/download/v4.5.1/yq_linux_amd64 -o yq
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100 6906k  100 6906k    0     0  4376k      0  0:00:01  0:00:01 --:--:-- 15.9M
++ chmod +x yq
++ YQ=./yq
++ which jq
/usr/bin/jq
++ which virt-filesystems
/usr/bin/virt-filesystems
++ which guestfish
/usr/bin/guestfish
++ which virsh
/usr/bin/virsh
++ which qemu-img
/usr/bin/qemu-img
++ which virt-install
/usr/bin/virt-install
++ rpm -q libguestfs-xfs
libguestfs-xfs-1.52.2-2.fc40.x86_64
++ '[' 1 '!=' 0 -o 1 '!=' 0 ']'
++ which unzip
/usr/bin/unzip
++ which xmllint
/usr/bin/xmllint
++ which dig
/usr/bin/dig
++ which zstd
/usr/bin/zstd
++ which htpasswd
/usr/bin/htpasswd
++ which patch
/usr/bin/patch
+ source snc-library.sh
++ set -exuo pipefail
+ trap 'jobs=($(jobs -p)); [ -n "${jobs-}" ] && ((${#jobs})) && kill "${jobs[@]}" || true' EXIT
+ OKD_VERSION=4.16.0-0.okd-scos-2024-08-21-155613
+ BUNDLE_TYPE=snc
+ [[ 4.16.0-0.okd-scos-2024-08-21-155613 != \n\o\n\e ]]
+ OPENSHIFT_VERSION=4.16.0-0.okd-scos-2024-08-21-155613
+ MIRROR=https://github.com/okd-project/okd-scos/releases/download
+ BUNDLE_TYPE=okd
+ INSTALL_DIR=crc-tmp-install-data
+ SNC_PRODUCT_NAME=crc
+ SNC_CLUSTER_MEMORY=14336
+ SNC_CLUSTER_CPUS=6
+ CRC_VM_DISK_SIZE=31
+ BASE_DOMAIN=testing
+ CRC_PV_DIR=/mnt/pv-data
+ SSH='ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -i id_ecdsa_crc'
+ SCP='scp -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -i id_ecdsa_crc'
+ MIRROR=https://github.com/okd-project/okd-scos/releases/download
+ CERT_ROTATION=enabled
+ USE_PATCHED_RELEASE_IMAGE=disabled
+ HTPASSWD_FILE=users.htpasswd
+ run_preflight_checks okd
+ local bundle_type=okd
+ '[' -z /tmp/pull_secret.json ']'
+ '[' '!' -f /tmp/pull_secret.json ']'
+ echo 'Checking libvirt and DNS configuration'
Checking libvirt and DNS configuration
+ LIBVIRT_URI=qemu:///system
+ sudo virsh -c qemu:///system uri
[sudo] password for rokumar: 
+ sudo virsh -c qemu:///system net-info default
+ echo 'default libvirt network is available'
default libvirt network is available
++ sudo virsh -c qemu:///system net-info default
++ awk '{print $2}'
++ sed '3q;d'
+ [[ yes == \n\o ]]
+ case $ARCH in
+ echo 'The host arch is x86_64.'
The host arch is x86_64.
+ sudo virsh -c qemu:///system capabilities
+ xmllint --xpath '/capabilities/guest/arch[@name='\''x86_64'\'']/domain[@type='\''kvm'\'']' -
+ '[' okd == snc ']'
+ '[' okd == okd ']'
+ local ping_status
++ ping -c1 api.crc.testing
++ head -1
++ true
+ ping_status='PING jkube-helm-maven-default.apps-crc.testing (127.0.0.1) 56(84) bytes of data.'
+ echo PING jkube-helm-maven-default.apps-crc.testing '(127.0.0.1)' '56(84)' bytes of data.
+ grep 'PING api.crc.testing ('
+ echo 'libvirt and DNS configuration successfully checked'
libvirt and DNS configuration successfully checked
+ test -n 4.16.0-0.okd-scos-2024-08-21-155613
+ OPENSHIFT_RELEASE_VERSION=4.16.0-0.okd-scos-2024-08-21-155613
+ echo 'Using release 4.16.0-0.okd-scos-2024-08-21-155613 from OPENSHIFT_VERSION'
Using release 4.16.0-0.okd-scos-2024-08-21-155613 from OPENSHIFT_VERSION
+ OC=./openshift-clients/linux/oc
+ download_oc
+ local current_oc_version=
+ '[' -f ./openshift-clients/linux/oc ']'
++ ./openshift-clients/linux/oc version --client -o json
++ jq -r .releaseClientVersion
+ current_oc_version=4.16.0-0.okd-scos-2024-08-21-155613
+ '[' 4.16.0-0.okd-scos-2024-08-21-155613 = 4.16.0-0.okd-scos-2024-08-21-155613 ']'
+ echo 'No need to download oc, local oc is already version 4.16.0-0.okd-scos-2024-08-21-155613'
No need to download oc, local oc is already version 4.16.0-0.okd-scos-2024-08-21-155613
+ return
+ test -z ''
++ curl -L https://github.com/okd-project/okd-scos/releases/download/4.16.0-0.okd-scos-2024-08-21-155613/release.txt
++ sed -n 's/^Pull From: //p'
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100 29160  100 29160    0     0  17966      0  0:00:01  0:00:01 --:--:--  171k
+ OPENSHIFT_INSTALL_RELEASE_IMAGE_OVERRIDE=quay.io/okd/scos-release@sha256:2d20d7770f511ca89d92b37812fdab66483d3d8cc2b1e9fcd33a341350e4f827
+ echo 'Setting OPENSHIFT_INSTALL_RELEASE_IMAGE_OVERRIDE to quay.io/okd/scos-release@sha256:2d20d7770f511ca89d92b37812fdab66483d3d8cc2b1e9fcd33a341350e4f827'
Setting OPENSHIFT_INSTALL_RELEASE_IMAGE_OVERRIDE to quay.io/okd/scos-release@sha256:2d20d7770f511ca89d92b37812fdab66483d3d8cc2b1e9fcd33a341350e4f827
+ test -z
+ echo 'Extracting OpenShift installer binary'
Extracting OpenShift installer binary
+ ./openshift-clients/linux/oc adm release extract -a /tmp/pull_secret.json quay.io/okd/scos-release@sha256:2d20d7770f511ca89d92b37812fdab66483d3d8cc2b1e9fcd33a341350e4f827 --command=openshift-install --to .
+ OPENSHIFT_INSTALL=./openshift-install
+ [[ disabled == \e\n\a\b\l\e\d ]]
+ test -z ''
+ OPENSHIFT_INSTALL_EXTRA_ARGS='--log-level debug'
+ ./openshift-install --dir crc-tmp-install-data destroy cluster --log-level debug
DEBUG OpenShift Installer 4.16.0-0.okd-scos-2024-08-21-155613 
DEBUG Built from commit deb993e72a920fb1c68a578d1b4c598071fefea4 
FATAL Failed while preparing to destroy cluster: no platform configured in metadata 
+ echo 'failed to destroy previous cluster.  Continuing anyway'
failed to destroy previous cluster.  Continuing anyway
+ rm id_ecdsa_crc id_ecdsa_crc.pub
+ ssh-keygen -t ecdsa -b 521 -N '' -f id_ecdsa_crc -C core
Generating public/private ecdsa key pair.
Your identification has been saved in id_ecdsa_crc
Your public key has been saved in id_ecdsa_crc.pub
The key fingerprint is:
SHA256:BkQaD8FFf9y7qMSlSO+FUjHu9kUxPmSN147vKrsOQoY core
The key's randomart image is:
+---[ECDSA 521]---+
|   .+==          |
|    .* . . . o . |
|    . o + o B o .|
|       + + + = o |
|      E S . = . .|
|     . X + o o . |
|      o X + o   .|
|       = = o.  . |
|        o ..++...|
+----[SHA256]-----+
+ grep -iqR dns=dnsmasq /etc/NetworkManager/conf.d/
+ '[' -f /etc/NetworkManager/dnsmasq.d/openshift.conf ']'
+ destroy_libvirt_resources rhcos-live.iso
+ local iso=rhcos-live.iso
+ sudo virsh destroy crc
error: Failed to destroy domain 'crc'
error: Requested operation is not valid: domain is not running

+ true
+ sudo virsh undefine crc --nvram
Domain 'crc' has been undefined

+ sudo virsh vol-delete --pool crc crc.qcow2
Vol crc.qcow2 deleted

+ sudo virsh vol-delete --pool crc rhcos-live.iso
Vol rhcos-live.iso deleted

+ sudo virsh pool-destroy crc
Pool crc destroyed

+ sudo virsh pool-undefine crc
Pool crc has been undefined

+ sudo virsh net-destroy crc
error: failed to get network 'crc'
error: Network not found: no network with matching name 'crc'

+ true
+ sudo virsh net-undefine crc
error: failed to get network 'crc'
error: Network not found: no network with matching name 'crc'

+ true
+ create_libvirt_resources
+ sudo virsh pool-define-as crc --type dir --target /var/lib/libvirt/crc
Pool crc defined

+ sudo virsh pool-start --build crc
Pool crc started

+ sudo virsh pool-autostart crc
Pool crc marked as autostarted

+ sed -e 's|NETWORK_NAME|crc|' -e 's|CLUSTER_NAME|crc|' -e 's|BASE_DOMAIN|testing|' host-libvirt-net.xml.template
+ sudo virsh net-create host-libvirt-net.xml
Network crc created from host-libvirt-net.xml

+ rm -fr host-libvirt-net.xml
+ cat
+ sudo tee /etc/NetworkManager/dnsmasq.d/crc-snc.conf
server=/crc.testing/192.168.126.1
address=/apps-crc.testing/192.168.126.11
+ sudo systemctl reload NetworkManager
+ [[ enabled == \e\n\a\b\l\e\d ]]
+ sudo timedatectl set-ntp off
+ sudo date -s '-1 day'
Sun Sep 29 10:26:30 IST 2024
+ rm -fr crc-tmp-install-data
+ mkdir crc-tmp-install-data
+ cp install-config.yaml crc-tmp-install-data
+ ./yq eval --inplace '.controlPlane.architecture = "amd64"' crc-tmp-install-data/install-config.yaml
+ ./yq eval --inplace '.baseDomain = "testing"' crc-tmp-install-data/install-config.yaml
+ ./yq eval --inplace '.metadata.name = "crc"' crc-tmp-install-data/install-config.yaml
+ replace_pull_secret crc-tmp-install-data/install-config.yaml
+ set +x
++ cat id_ecdsa_crc.pub
+ ./yq eval '.sshKey = "ecdsa-sha2-nistp521 AAAAE2VjZHNhLXNoYTItbmlzdHA1MjEAAAAIbmlzdHA1MjEAAACFBADqD9z6TnMRZ+V9NC/gm8bC9h3E6J9r/DdPOfYFG/X38g5oID/gUj2/MTXCkZ95XpgWnG+YUBFJ4ZGZIiGj8uldiQC+Irmzw/o4KqA8XFbJIdx1xGrn238TRKLEen3zkiLcAwP0LVKLsCUXVLoDH7eO7EIMmMlvNsc2CbN1FCGknPw8uA== core"' --inplace crc-tmp-install-data/install-config.yaml
+ OPENSHIFT_INSTALL_RELEASE_IMAGE_OVERRIDE=quay.io/okd/scos-release@sha256:2d20d7770f511ca89d92b37812fdab66483d3d8cc2b1e9fcd33a341350e4f827
+ ./openshift-install --dir crc-tmp-install-data create manifests
INFO Consuming Install Config from target directory 
WARNING Making control-plane schedulable by setting MastersSchedulable to true for Scheduler cluster settings 
INFO Manifests created in: crc-tmp-install-data/manifests and crc-tmp-install-data/openshift 
+ ./yq eval-all --inplace 'select(fileIndex == 0) * select(filename == "cvo-overrides.yaml")' crc-tmp-install-data/manifests/cvo-overrides.yaml cvo-overrides.yaml
+ ./yq eval --inplace '.spec.domain = "apps-crc.testing"' crc-tmp-install-data/manifests/cluster-ingress-02-config.yml
+ cp cluster-network-03-config.yaml crc-tmp-install-data/manifests/
+ cp 99_master-chronyd-mask.yaml crc-tmp-install-data/openshift/
+ cp 99-openshift-machineconfig-master-dummy-networks.yaml crc-tmp-install-data/openshift/
++ base64 -w0 node-sizing-enabled.env
+ DYNAMIC_DATA=Tk9ERV9TSVpJTkdfRU5BQkxFRD1mYWxzZQpTWVNURU1fUkVTRVJWRURfTUVNT1JZPTM1ME1pClNZU1RFTV9SRVNFUlZFRF9DUFU9MjAwbQpTWVNURU1fUkVTRVJWRURfRVM9MzUwTWkK
+ envsubst
+ export OPENSHIFT_INSTALL_INVOKER=codeReadyContainers
+ OPENSHIFT_INSTALL_INVOKER=codeReadyContainers
+ export KUBECONFIG=crc-tmp-install-data/auth/kubeconfig
+ KUBECONFIG=crc-tmp-install-data/auth/kubeconfig
+ OPENSHIFT_INSTALL_RELEASE_IMAGE_OVERRIDE=quay.io/okd/scos-release@sha256:2d20d7770f511ca89d92b37812fdab66483d3d8cc2b1e9fcd33a341350e4f827
+ ./openshift-install --dir crc-tmp-install-data create single-node-ignition-config --log-level debug
DEBUG OpenShift Installer 4.16.0-0.okd-scos-2024-08-21-155613 
DEBUG Built from commit deb993e72a920fb1c68a578d1b4c598071fefea4 
DEBUG Fetching Kubeconfig Admin Client...          
DEBUG Loading Kubeconfig Admin Client...           
DEBUG   Loading Certificate (admin-kubeconfig-client)... 
DEBUG     Loading Certificate (admin-kubeconfig-signer)... 
DEBUG   Loading Certificate (kube-apiserver-complete-server-ca-bundle)... 
DEBUG     Loading Certificate (kube-apiserver-localhost-ca-bundle)... 
DEBUG       Loading Certificate (kube-apiserver-localhost-signer)... 
DEBUG     Loading Certificate (kube-apiserver-service-network-ca-bundle)... 
DEBUG       Loading Certificate (kube-apiserver-service-network-signer)... 
DEBUG     Loading Certificate (kube-apiserver-lb-ca-bundle)... 
DEBUG       Loading Certificate (kube-apiserver-lb-signer)... 
DEBUG   Loading Install Config...                  
DEBUG     Loading SSH Key...                       
DEBUG     Loading Base Domain...                   
DEBUG       Loading Platform...                    
DEBUG     Loading Cluster Name...                  
DEBUG       Loading Base Domain...                 
DEBUG       Loading Platform...                    
DEBUG     Loading Pull Secret...                   
DEBUG     Loading Platform...                      
DEBUG   Using Install Config loaded from state file 
DEBUG   Fetching Certificate (admin-kubeconfig-client)... 
DEBUG     Fetching Certificate (admin-kubeconfig-signer)... 
DEBUG     Generating Certificate (admin-kubeconfig-signer)... 
DEBUG   Generating Certificate (admin-kubeconfig-client)... 
DEBUG   Fetching Certificate (kube-apiserver-complete-server-ca-bundle)... 
DEBUG     Fetching Certificate (kube-apiserver-localhost-ca-bundle)... 
DEBUG       Fetching Certificate (kube-apiserver-localhost-signer)... 
DEBUG       Generating Certificate (kube-apiserver-localhost-signer)... 
DEBUG     Generating Certificate (kube-apiserver-localhost-ca-bundle)... 
DEBUG     Fetching Certificate (kube-apiserver-service-network-ca-bundle)... 
DEBUG       Fetching Certificate (kube-apiserver-service-network-signer)... 
DEBUG       Generating Certificate (kube-apiserver-service-network-signer)... 
DEBUG     Generating Certificate (kube-apiserver-service-network-ca-bundle)... 
DEBUG     Fetching Certificate (kube-apiserver-lb-ca-bundle)... 
DEBUG       Fetching Certificate (kube-apiserver-lb-signer)... 
DEBUG       Generating Certificate (kube-apiserver-lb-signer)... 
DEBUG     Generating Certificate (kube-apiserver-lb-ca-bundle)... 
DEBUG   Generating Certificate (kube-apiserver-complete-server-ca-bundle)... 
DEBUG   Fetching Install Config...                 
DEBUG   Reusing previously-fetched Install Config  
DEBUG Generating Kubeconfig Admin Client...        
DEBUG Fetching Kubeadmin Password...               
DEBUG Loading Kubeadmin Password...                
DEBUG Using Kubeadmin Password loaded from state file 
DEBUG Reusing previously-fetched Kubeadmin Password 
DEBUG Fetching Worker Ignition Config...           
DEBUG Loading Worker Ignition Config...            
DEBUG   Loading Install Config...                  
DEBUG   Loading Machine Config Server Root CA...   
DEBUG   Using Machine Config Server Root CA loaded from state file 
DEBUG Using Worker Ignition Config loaded from state file 
DEBUG Reusing previously-fetched Worker Ignition Config 
DEBUG Fetching Single node bootstrap In Place Ignition Config... 
DEBUG Loading Single node bootstrap In Place Ignition Config... 
DEBUG   Loading Ironic bootstrap credentials...    
DEBUG   Loading CVO Ignore...                      
DEBUG     Loading Common Manifests...              
DEBUG       Loading Cluster ID...                  
DEBUG         Loading Install Config...            
DEBUG       Using Cluster ID loaded from state file 
DEBUG       Loading Install Config...              
DEBUG       Loading Ingress Config...              
DEBUG         Loading Install Config...            
DEBUG       Using Ingress Config loaded from state file 
DEBUG       Loading DNS Config...                  
DEBUG         Loading Install Config...            
DEBUG         Loading Cluster ID...                
DEBUG         Loading Platform Credentials Check... 
DEBUG           Loading Install Config...          
DEBUG         Using Platform Credentials Check loaded from state file 
DEBUG       Using DNS Config loaded from state file 
DEBUG       Loading Infrastructure Config...       
DEBUG         Loading Cluster ID...                
DEBUG         Loading Install Config...            
DEBUG         Loading Cloud Provider Config...     
DEBUG           Loading Install Config...          
DEBUG           Loading Cluster ID...              
DEBUG           Loading Platform Credentials Check... 
DEBUG         Using Cloud Provider Config loaded from state file 
DEBUG         Loading Additional Trust Bundle Config... 
DEBUG           Loading Install Config...          
DEBUG         Using Additional Trust Bundle Config loaded from state file 
DEBUG       Using Infrastructure Config loaded from state file 
DEBUG       Loading Network Config...              
DEBUG         Loading Install Config...            
DEBUG       Using Network Config loaded from state file 
DEBUG       Loading Proxy Config...                
DEBUG         Loading Install Config...            
DEBUG         Loading Network Config...            
DEBUG       Using Proxy Config loaded from state file 
DEBUG       Loading Scheduler Config...            
DEBUG         Loading Install Config...            
DEBUG       Using Scheduler Config loaded from state file 
DEBUG       Loading Image Content Source Policy... 
DEBUG         Loading Install Config...            
DEBUG       Using Image Content Source Policy loaded from state file 
DEBUG       Loading Cluster CSI Driver Config...   
DEBUG         Loading Install Config...            
DEBUG         Loading Cluster ID...                
DEBUG       Using Cluster CSI Driver Config loaded from state file 
DEBUG       Loading Image Digest Mirror Set...     
DEBUG         Loading Install Config...            
DEBUG       Using Image Digest Mirror Set loaded from state file 
DEBUG       Loading Machine Config Server Root CA... 
DEBUG       Loading Certificate (mcs)...           
DEBUG         Loading Machine Config Server Root CA... 
DEBUG         Loading Install Config...            
DEBUG       Using Certificate (mcs) loaded from state file 
DEBUG       Loading CVOOverrides...                
DEBUG       Using CVOOverrides loaded from state file 
DEBUG       Loading KubeCloudConfig...             
DEBUG       Using KubeCloudConfig loaded from state file 
DEBUG       Loading KubeSystemConfigmapRootCA...   
DEBUG       Using KubeSystemConfigmapRootCA loaded from state file 
DEBUG       Loading MachineConfigServerTLSSecret... 
DEBUG       Using MachineConfigServerTLSSecret loaded from state file 
DEBUG       Loading OpenshiftConfigSecretPullSecret... 
DEBUG       Using OpenshiftConfigSecretPullSecret loaded from state file 
DEBUG     Loading Common Manifests from both state file and target directory 
DEBUG     Using Common Manifests loaded from target directory 
DEBUG     Loading Openshift Manifests...           
DEBUG       Loading Install Config...              
DEBUG       Loading Cluster ID...                  
DEBUG       Loading Kubeadmin Password...          
DEBUG       Loading OpenShift Install (Manifests)... 
DEBUG       Loading OpenShift Install (Manifests) from both state file and target directory 
DEBUG       On-disk OpenShift Install (Manifests) matches asset in state file 
DEBUG       Using OpenShift Install (Manifests) loaded from state file 
DEBUG       Loading Feature Gate Config...         
DEBUG         Loading Install Config...            
DEBUG       Using Feature Gate Config loaded from state file 
DEBUG       Loading CloudCredsSecret...            
DEBUG       Using CloudCredsSecret loaded from state file 
DEBUG       Loading KubeadminPasswordSecret...     
DEBUG       Using KubeadminPasswordSecret loaded from state file 
DEBUG       Loading RoleCloudCredsSecretReader...  
DEBUG       Using RoleCloudCredsSecretReader loaded from state file 
DEBUG       Loading Baremetal Config CR...         
DEBUG       Using Baremetal Config CR loaded from state file 
DEBUG       Loading Image...                       
DEBUG         Loading Install Config...            
DEBUG       Using Image loaded from state file     
DEBUG       Loading AzureCloudProviderSecret...    
DEBUG       Using AzureCloudProviderSecret loaded from state file 
DEBUG     Loading Openshift Manifests from both state file and target directory 
DEBUG     Using Openshift Manifests loaded from target directory 
DEBUG   Loading Install Config...                  
DEBUG   Loading Kubeconfig Admin Internal Client... 
DEBUG     Loading Certificate (admin-kubeconfig-client)... 
DEBUG     Loading Certificate (kube-apiserver-complete-server-ca-bundle)... 
DEBUG     Loading Install Config...                
DEBUG   Loading Kubeconfig Kubelet...              
DEBUG     Loading Certificate (kube-apiserver-complete-server-ca-bundle)... 
DEBUG     Loading Certificate (kubelet-client)...  
DEBUG       Loading Certificate (kubelet-bootstrap-kubeconfig-signer)... 
DEBUG     Loading Install Config...                
DEBUG   Loading Kubeconfig Admin Client (Loopback)... 
DEBUG     Loading Certificate (admin-kubeconfig-client)... 
DEBUG     Loading Certificate (kube-apiserver-localhost-ca-bundle)... 
DEBUG     Loading Install Config...                
DEBUG   Loading Master Ignition Customization Check... 
DEBUG     Loading Install Config...                
DEBUG     Loading Machine Config Server Root CA... 
DEBUG     Loading Master Ignition Config...        
DEBUG       Loading Install Config...              
DEBUG       Loading Machine Config Server Root CA... 
DEBUG     Using Master Ignition Config loaded from state file 
DEBUG   Loading Worker Ignition Customization Check... 
DEBUG     Loading Install Config...                
DEBUG     Loading Machine Config Server Root CA... 
DEBUG     Loading Worker Ignition Config...        
DEBUG   Loading Master Machines...                 
DEBUG     Loading Cluster ID...                    
DEBUG     Loading Platform Credentials Check...    
DEBUG     Loading Install Config...                
DEBUG     Loading Image...                         
DEBUG     Loading Master Ignition Config...        
DEBUG CPMS file missing. Ignoring it while loading machine asset. 
DEBUG   Loading Master Machines from both state file and target directory 
DEBUG   Using Master Machines loaded from target directory 
DEBUG   Loading Worker Machines...                 
DEBUG     Loading Cluster ID...                    
DEBUG     Loading Platform Credentials Check...    
DEBUG     Loading Install Config...                
DEBUG     Loading Image...                         
DEBUG     Loading Release...                       
DEBUG       Loading Install Config...              
DEBUG     Using Release loaded from state file     
DEBUG     Loading Worker Ignition Config...        
DEBUG   Loading Worker Machines from both state file and target directory 
DEBUG   Using Worker Machines loaded from target directory 
DEBUG   Loading Common Manifests...                
DEBUG   Loading Openshift Manifests...             
DEBUG   Loading Proxy Config...                    
DEBUG   Loading Certificate (admin-kubeconfig-ca-bundle)... 
DEBUG     Loading Certificate (admin-kubeconfig-signer)... 
DEBUG   Loading Certificate (aggregator)...        
DEBUG   Loading Certificate (aggregator-ca-bundle)... 
DEBUG     Loading Certificate (aggregator-signer)... 
DEBUG   Loading Certificate (system:kube-apiserver-proxy)... 
DEBUG     Loading Certificate (aggregator-signer)... 
DEBUG   Loading Certificate (aggregator-signer)... 
DEBUG   Loading Certificate (system:kube-apiserver-proxy)... 
DEBUG     Loading Certificate (aggregator)...      
DEBUG   Loading Bootstrap SSH Key Pair...          
DEBUG   Loading User-provided Service Account Signing key... 
DEBUG   Loading Cloud Provider CA Bundle...        
DEBUG     Loading Install Config...                
DEBUG   Loading Certificate (journal-gatewayd)...  
DEBUG     Loading Machine Config Server Root CA... 
DEBUG   Loading Certificate (kube-apiserver-lb-ca-bundle)... 
DEBUG   Loading Certificate (kube-apiserver-external-lb-server)... 
DEBUG     Loading Certificate (kube-apiserver-lb-signer)... 
DEBUG     Loading Install Config...                
DEBUG   Loading Certificate (kube-apiserver-internal-lb-server)... 
DEBUG     Loading Certificate (kube-apiserver-lb-signer)... 
DEBUG     Loading Install Config...                
DEBUG   Loading Certificate (kube-apiserver-lb-signer)... 
DEBUG   Loading Certificate (kube-apiserver-localhost-ca-bundle)... 
DEBUG   Loading Certificate (kube-apiserver-localhost-server)... 
DEBUG     Loading Certificate (kube-apiserver-localhost-signer)... 
DEBUG   Loading Certificate (kube-apiserver-localhost-signer)... 
DEBUG   Loading Certificate (kube-apiserver-service-network-ca-bundle)... 
DEBUG   Loading Certificate (kube-apiserver-service-network-server)... 
DEBUG     Loading Certificate (kube-apiserver-service-network-signer)... 
DEBUG     Loading Install Config...                
DEBUG   Loading Certificate (kube-apiserver-service-network-signer)... 
DEBUG   Loading Certificate (kube-apiserver-complete-server-ca-bundle)... 
DEBUG   Loading Certificate (kube-apiserver-complete-client-ca-bundle)... 
DEBUG     Loading Certificate (admin-kubeconfig-ca-bundle)... 
DEBUG     Loading Certificate (kubelet-client-ca-bundle)... 
DEBUG       Loading Certificate (kubelet-signer)... 
DEBUG     Loading Certificate (kube-control-plane-ca-bundle)... 
DEBUG       Loading Certificate (kube-control-plane-signer)... 
DEBUG       Loading Certificate (kube-apiserver-lb-signer)... 
DEBUG       Loading Certificate (kube-apiserver-localhost-signer)... 
DEBUG       Loading Certificate (kube-apiserver-service-network-signer)... 
DEBUG     Loading Certificate (kube-apiserver-to-kubelet-ca-bundle)... 
DEBUG       Loading Certificate (kube-apiserver-to-kubelet-signer)... 
DEBUG     Loading Certificate (kubelet-bootstrap-kubeconfig-ca-bundle)... 
DEBUG       Loading Certificate (kubelet-bootstrap-kubeconfig-signer)... 
DEBUG   Loading Certificate (kube-apiserver-to-kubelet-ca-bundle)... 
DEBUG   Loading Certificate (kube-apiserver-to-kubelet-client)... 
DEBUG     Loading Certificate (kube-apiserver-to-kubelet-signer)... 
DEBUG   Loading Certificate (kube-apiserver-to-kubelet-signer)... 
DEBUG   Loading Certificate (kube-control-plane-ca-bundle)... 
DEBUG   Loading Certificate (kube-control-plane-kube-controller-manager-client)... 
DEBUG     Loading Certificate (kube-control-plane-signer)... 
DEBUG   Loading Certificate (kube-control-plane-kube-scheduler-client)... 
DEBUG     Loading Certificate (kube-control-plane-signer)... 
DEBUG   Loading Certificate (kube-control-plane-signer)... 
DEBUG   Loading Certificate (kubelet-bootstrap-kubeconfig-ca-bundle)... 
DEBUG   Loading Certificate (kubelet-client-ca-bundle)... 
DEBUG   Loading Certificate (kubelet-client)...    
DEBUG   Loading Certificate (kubelet-signer)...    
DEBUG   Loading Certificate (kubelet-serving-ca-bundle)... 
DEBUG     Loading Certificate (kubelet-signer)...  
DEBUG   Loading Certificate (mcs)...               
DEBUG   Loading Machine Config Server Root CA...   
DEBUG   Loading Key Pair (service-account.pub)...  
DEBUG   Loading Release Image Pull Spec...         
DEBUG   Loading Image...                           
DEBUG   Fetching Ironic bootstrap credentials...   
DEBUG   Generating Ironic bootstrap credentials... 
DEBUG   Fetching CVO Ignore...                     
DEBUG     Fetching Common Manifests...             
DEBUG     Reusing previously-fetched Common Manifests 
DEBUG     Fetching Openshift Manifests...          
DEBUG     Reusing previously-fetched Openshift Manifests 
DEBUG   Generating CVO Ignore...                   
DEBUG   Fetching Install Config...                 
DEBUG   Reusing previously-fetched Install Config  
DEBUG   Fetching Kubeconfig Admin Internal Client... 
DEBUG     Fetching Certificate (admin-kubeconfig-client)... 
DEBUG     Reusing previously-fetched Certificate (admin-kubeconfig-client) 
DEBUG     Fetching Certificate (kube-apiserver-complete-server-ca-bundle)... 
DEBUG     Reusing previously-fetched Certificate (kube-apiserver-complete-server-ca-bundle) 
DEBUG     Fetching Install Config...               
DEBUG     Reusing previously-fetched Install Config 
DEBUG   Generating Kubeconfig Admin Internal Client... 
DEBUG   Fetching Kubeconfig Kubelet...             
DEBUG     Fetching Certificate (kube-apiserver-complete-server-ca-bundle)... 
DEBUG     Reusing previously-fetched Certificate (kube-apiserver-complete-server-ca-bundle) 
DEBUG     Fetching Certificate (kubelet-client)... 
DEBUG       Fetching Certificate (kubelet-bootstrap-kubeconfig-signer)... 
DEBUG       Generating Certificate (kubelet-bootstrap-kubeconfig-signer)... 
DEBUG     Generating Certificate (kubelet-client)... 
DEBUG     Fetching Install Config...               
DEBUG     Reusing previously-fetched Install Config 
DEBUG   Generating Kubeconfig Kubelet...           
DEBUG   Fetching Kubeconfig Admin Client (Loopback)... 
DEBUG     Fetching Certificate (admin-kubeconfig-client)... 
DEBUG     Reusing previously-fetched Certificate (admin-kubeconfig-client) 
DEBUG     Fetching Certificate (kube-apiserver-localhost-ca-bundle)... 
DEBUG     Reusing previously-fetched Certificate (kube-apiserver-localhost-ca-bundle) 
DEBUG     Fetching Install Config...               
DEBUG     Reusing previously-fetched Install Config 
DEBUG   Generating Kubeconfig Admin Client (Loopback)... 
DEBUG   Fetching Master Ignition Customization Check... 
DEBUG     Fetching Install Config...               
DEBUG     Reusing previously-fetched Install Config 
DEBUG     Fetching Machine Config Server Root CA... 
DEBUG     Reusing previously-fetched Machine Config Server Root CA 
DEBUG     Fetching Master Ignition Config...       
DEBUG     Reusing previously-fetched Master Ignition Config 
DEBUG   Generating Master Ignition Customization Check... 
DEBUG   Fetching Worker Ignition Customization Check... 
DEBUG     Fetching Install Config...               
DEBUG     Reusing previously-fetched Install Config 
DEBUG     Fetching Machine Config Server Root CA... 
DEBUG     Reusing previously-fetched Machine Config Server Root CA 
DEBUG     Fetching Worker Ignition Config...       
DEBUG     Reusing previously-fetched Worker Ignition Config 
DEBUG   Generating Worker Ignition Customization Check... 
DEBUG   Fetching Master Machines...                
DEBUG   Reusing previously-fetched Master Machines 
DEBUG   Fetching Worker Machines...                
DEBUG   Reusing previously-fetched Worker Machines 
DEBUG   Fetching Common Manifests...               
DEBUG   Reusing previously-fetched Common Manifests 
DEBUG   Fetching Openshift Manifests...            
DEBUG   Reusing previously-fetched Openshift Manifests 
DEBUG   Fetching Proxy Config...                   
DEBUG   Reusing previously-fetched Proxy Config    
DEBUG   Fetching Certificate (admin-kubeconfig-ca-bundle)... 
DEBUG     Fetching Certificate (admin-kubeconfig-signer)... 
DEBUG     Reusing previously-fetched Certificate (admin-kubeconfig-signer) 
DEBUG   Generating Certificate (admin-kubeconfig-ca-bundle)... 
DEBUG   Fetching Certificate (aggregator)...       
DEBUG   Generating Certificate (aggregator)...     
DEBUG   Fetching Certificate (aggregator-ca-bundle)... 
DEBUG     Fetching Certificate (aggregator-signer)... 
DEBUG     Generating Certificate (aggregator-signer)... 
DEBUG   Generating Certificate (aggregator-ca-bundle)... 
DEBUG   Fetching Certificate (system:kube-apiserver-proxy)... 
DEBUG     Fetching Certificate (aggregator-signer)... 
DEBUG     Reusing previously-fetched Certificate (aggregator-signer) 
DEBUG   Generating Certificate (system:kube-apiserver-proxy)... 
DEBUG   Fetching Certificate (aggregator-signer)... 
DEBUG   Reusing previously-fetched Certificate (aggregator-signer) 
DEBUG   Fetching Certificate (system:kube-apiserver-proxy)... 
DEBUG     Fetching Certificate (aggregator)...     
DEBUG     Reusing previously-fetched Certificate (aggregator) 
DEBUG   Generating Certificate (system:kube-apiserver-proxy)... 
DEBUG   Fetching Bootstrap SSH Key Pair...         
DEBUG   Generating Bootstrap SSH Key Pair...       
DEBUG   Fetching User-provided Service Account Signing key... 
DEBUG   Generating User-provided Service Account Signing key... 
DEBUG   Fetching Cloud Provider CA Bundle...       
DEBUG     Fetching Install Config...               
DEBUG     Reusing previously-fetched Install Config 
DEBUG   Generating Cloud Provider CA Bundle...     
DEBUG   Fetching Certificate (journal-gatewayd)... 
DEBUG     Fetching Machine Config Server Root CA... 
DEBUG     Reusing previously-fetched Machine Config Server Root CA 
DEBUG   Generating Certificate (journal-gatewayd)... 
DEBUG   Fetching Certificate (kube-apiserver-lb-ca-bundle)... 
DEBUG   Reusing previously-fetched Certificate (kube-apiserver-lb-ca-bundle) 
DEBUG   Fetching Certificate (kube-apiserver-external-lb-server)... 
DEBUG     Fetching Certificate (kube-apiserver-lb-signer)... 
DEBUG     Reusing previously-fetched Certificate (kube-apiserver-lb-signer) 
DEBUG     Fetching Install Config...               
DEBUG     Reusing previously-fetched Install Config 
DEBUG   Generating Certificate (kube-apiserver-external-lb-server)... 
DEBUG   Fetching Certificate (kube-apiserver-internal-lb-server)... 
DEBUG     Fetching Certificate (kube-apiserver-lb-signer)... 
DEBUG     Reusing previously-fetched Certificate (kube-apiserver-lb-signer) 
DEBUG     Fetching Install Config...               
DEBUG     Reusing previously-fetched Install Config 
DEBUG   Generating Certificate (kube-apiserver-internal-lb-server)... 
DEBUG   Fetching Certificate (kube-apiserver-lb-signer)... 
DEBUG   Reusing previously-fetched Certificate (kube-apiserver-lb-signer) 
DEBUG   Fetching Certificate (kube-apiserver-localhost-ca-bundle)... 
DEBUG   Reusing previously-fetched Certificate (kube-apiserver-localhost-ca-bundle) 
DEBUG   Fetching Certificate (kube-apiserver-localhost-server)... 
DEBUG     Fetching Certificate (kube-apiserver-localhost-signer)... 
DEBUG     Reusing previously-fetched Certificate (kube-apiserver-localhost-signer) 
DEBUG   Generating Certificate (kube-apiserver-localhost-server)... 
DEBUG   Fetching Certificate (kube-apiserver-localhost-signer)... 
DEBUG   Reusing previously-fetched Certificate (kube-apiserver-localhost-signer) 
DEBUG   Fetching Certificate (kube-apiserver-service-network-ca-bundle)... 
DEBUG   Reusing previously-fetched Certificate (kube-apiserver-service-network-ca-bundle) 
DEBUG   Fetching Certificate (kube-apiserver-service-network-server)... 
DEBUG     Fetching Certificate (kube-apiserver-service-network-signer)... 
DEBUG     Reusing previously-fetched Certificate (kube-apiserver-service-network-signer) 
DEBUG     Fetching Install Config...               
DEBUG     Reusing previously-fetched Install Config 
DEBUG   Generating Certificate (kube-apiserver-service-network-server)... 
DEBUG   Fetching Certificate (kube-apiserver-service-network-signer)... 
DEBUG   Reusing previously-fetched Certificate (kube-apiserver-service-network-signer) 
DEBUG   Fetching Certificate (kube-apiserver-complete-server-ca-bundle)... 
DEBUG   Reusing previously-fetched Certificate (kube-apiserver-complete-server-ca-bundle) 
DEBUG   Fetching Certificate (kube-apiserver-complete-client-ca-bundle)... 
DEBUG     Fetching Certificate (admin-kubeconfig-ca-bundle)... 
DEBUG     Reusing previously-fetched Certificate (admin-kubeconfig-ca-bundle) 
DEBUG     Fetching Certificate (kubelet-client-ca-bundle)... 
DEBUG       Fetching Certificate (kubelet-signer)... 
DEBUG       Generating Certificate (kubelet-signer)... 
DEBUG     Generating Certificate (kubelet-client-ca-bundle)... 
DEBUG     Fetching Certificate (kube-control-plane-ca-bundle)... 
DEBUG       Fetching Certificate (kube-control-plane-signer)... 
DEBUG       Generating Certificate (kube-control-plane-signer)... 
DEBUG       Fetching Certificate (kube-apiserver-lb-signer)... 
DEBUG       Reusing previously-fetched Certificate (kube-apiserver-lb-signer) 
DEBUG       Fetching Certificate (kube-apiserver-localhost-signer)... 
DEBUG       Reusing previously-fetched Certificate (kube-apiserver-localhost-signer) 
DEBUG       Fetching Certificate (kube-apiserver-service-network-signer)... 
DEBUG       Reusing previously-fetched Certificate (kube-apiserver-service-network-signer) 
DEBUG     Generating Certificate (kube-control-plane-ca-bundle)... 
DEBUG     Fetching Certificate (kube-apiserver-to-kubelet-ca-bundle)... 
DEBUG       Fetching Certificate (kube-apiserver-to-kubelet-signer)... 
DEBUG       Generating Certificate (kube-apiserver-to-kubelet-signer)... 
DEBUG     Generating Certificate (kube-apiserver-to-kubelet-ca-bundle)... 
DEBUG     Fetching Certificate (kubelet-bootstrap-kubeconfig-ca-bundle)... 
DEBUG       Fetching Certificate (kubelet-bootstrap-kubeconfig-signer)... 
DEBUG       Reusing previously-fetched Certificate (kubelet-bootstrap-kubeconfig-signer) 
DEBUG     Generating Certificate (kubelet-bootstrap-kubeconfig-ca-bundle)... 
DEBUG   Generating Certificate (kube-apiserver-complete-client-ca-bundle)... 
DEBUG   Fetching Certificate (kube-apiserver-to-kubelet-ca-bundle)... 
DEBUG   Reusing previously-fetched Certificate (kube-apiserver-to-kubelet-ca-bundle) 
DEBUG   Fetching Certificate (kube-apiserver-to-kubelet-client)... 
DEBUG     Fetching Certificate (kube-apiserver-to-kubelet-signer)... 
DEBUG     Reusing previously-fetched Certificate (kube-apiserver-to-kubelet-signer) 
DEBUG   Generating Certificate (kube-apiserver-to-kubelet-client)... 
DEBUG   Fetching Certificate (kube-apiserver-to-kubelet-signer)... 
DEBUG   Reusing previously-fetched Certificate (kube-apiserver-to-kubelet-signer) 
DEBUG   Fetching Certificate (kube-control-plane-ca-bundle)... 
DEBUG   Reusing previously-fetched Certificate (kube-control-plane-ca-bundle) 
DEBUG   Fetching Certificate (kube-control-plane-kube-controller-manager-client)... 
DEBUG     Fetching Certificate (kube-control-plane-signer)... 
DEBUG     Reusing previously-fetched Certificate (kube-control-plane-signer) 
DEBUG   Generating Certificate (kube-control-plane-kube-controller-manager-client)... 
DEBUG   Fetching Certificate (kube-control-plane-kube-scheduler-client)... 
DEBUG     Fetching Certificate (kube-control-plane-signer)... 
DEBUG     Reusing previously-fetched Certificate (kube-control-plane-signer) 
DEBUG   Generating Certificate (kube-control-plane-kube-scheduler-client)... 
DEBUG   Fetching Certificate (kube-control-plane-signer)... 
DEBUG   Reusing previously-fetched Certificate (kube-control-plane-signer) 
DEBUG   Fetching Certificate (kubelet-bootstrap-kubeconfig-ca-bundle)... 
DEBUG   Reusing previously-fetched Certificate (kubelet-bootstrap-kubeconfig-ca-bundle) 
DEBUG   Fetching Certificate (kubelet-client-ca-bundle)... 
DEBUG   Reusing previously-fetched Certificate (kubelet-client-ca-bundle) 
DEBUG   Fetching Certificate (kubelet-client)...   
DEBUG   Reusing previously-fetched Certificate (kubelet-client) 
DEBUG   Fetching Certificate (kubelet-signer)...   
DEBUG   Reusing previously-fetched Certificate (kubelet-signer) 
DEBUG   Fetching Certificate (kubelet-serving-ca-bundle)... 
DEBUG     Fetching Certificate (kubelet-signer)... 
DEBUG     Reusing previously-fetched Certificate (kubelet-signer) 
DEBUG   Generating Certificate (kubelet-serving-ca-bundle)... 
DEBUG   Fetching Certificate (mcs)...              
DEBUG   Reusing previously-fetched Certificate (mcs) 
DEBUG   Fetching Machine Config Server Root CA...  
DEBUG   Reusing previously-fetched Machine Config Server Root CA 
DEBUG   Fetching Key Pair (service-account.pub)... 
DEBUG   Generating Key Pair (service-account.pub)... 
DEBUG   Fetching Release Image Pull Spec...        
DEBUG   Generating Release Image Pull Spec...      
WARNING Found override for release image (quay.io/okd/scos-release@sha256:2d20d7770f511ca89d92b37812fdab66483d3d8cc2b1e9fcd33a341350e4f827). Please be warned, this is not advised 
DEBUG   Fetching Image...                          
DEBUG   Reusing previously-fetched Image           
DEBUG Generating Single node bootstrap In Place Ignition Config... 
INFO Consuming Common Manifests from target directory 
DEBUG Purging asset "Common Manifests" from disk   
INFO Consuming OpenShift Install (Manifests) from target directory 
DEBUG Purging asset "OpenShift Install (Manifests)" from disk 
INFO Consuming Openshift Manifests from target directory 
DEBUG Purging asset "Openshift Manifests" from disk 
INFO Consuming Master Machines from target directory 
DEBUG Purging asset "Master Machines" from disk    
INFO Consuming Worker Machines from target directory 
DEBUG Purging asset "Worker Machines" from disk    
DEBUG Fetching Metadata...                         
DEBUG Loading Metadata...                          
DEBUG   Loading Cluster ID...                      
DEBUG   Loading Install Config...                  
DEBUG   Loading Bootstrap Ignition Config...       
DEBUG     Loading Ironic bootstrap credentials...  
DEBUG     Loading CVO Ignore...                    
DEBUG     Loading Install Config...                
DEBUG     Loading Kubeconfig Admin Internal Client... 
DEBUG     Loading Kubeconfig Kubelet...            
DEBUG     Loading Kubeconfig Admin Client (Loopback)... 
DEBUG     Loading Master Ignition Customization Check... 
DEBUG     Loading Worker Ignition Customization Check... 
DEBUG     Loading Master Machines...               
DEBUG     Loading Worker Machines...               
DEBUG     Loading Common Manifests...              
DEBUG     Loading Openshift Manifests...           
DEBUG     Loading Proxy Config...                  
DEBUG     Loading Certificate (admin-kubeconfig-ca-bundle)... 
DEBUG     Loading Certificate (aggregator)...      
DEBUG     Loading Certificate (aggregator-ca-bundle)... 
DEBUG     Loading Certificate (system:kube-apiserver-proxy)... 
DEBUG     Loading Certificate (aggregator-signer)... 
DEBUG     Loading Certificate (system:kube-apiserver-proxy)... 
DEBUG     Loading Bootstrap SSH Key Pair...        
DEBUG     Loading User-provided Service Account Signing key... 
DEBUG     Loading Cloud Provider CA Bundle...      
DEBUG     Loading Certificate (journal-gatewayd)... 
DEBUG     Loading Certificate (kube-apiserver-lb-ca-bundle)... 
DEBUG     Loading Certificate (kube-apiserver-external-lb-server)... 
DEBUG     Loading Certificate (kube-apiserver-internal-lb-server)... 
DEBUG     Loading Certificate (kube-apiserver-lb-signer)... 
DEBUG     Loading Certificate (kube-apiserver-localhost-ca-bundle)... 
DEBUG     Loading Certificate (kube-apiserver-localhost-server)... 
DEBUG     Loading Certificate (kube-apiserver-localhost-signer)... 
DEBUG     Loading Certificate (kube-apiserver-service-network-ca-bundle)... 
DEBUG     Loading Certificate (kube-apiserver-service-network-server)... 
DEBUG     Loading Certificate (kube-apiserver-service-network-signer)... 
DEBUG     Loading Certificate (kube-apiserver-complete-server-ca-bundle)... 
DEBUG     Loading Certificate (kube-apiserver-complete-client-ca-bundle)... 
DEBUG     Loading Certificate (kube-apiserver-to-kubelet-ca-bundle)... 
DEBUG     Loading Certificate (kube-apiserver-to-kubelet-client)... 
DEBUG     Loading Certificate (kube-apiserver-to-kubelet-signer)... 
DEBUG     Loading Certificate (kube-control-plane-ca-bundle)... 
DEBUG     Loading Certificate (kube-control-plane-kube-controller-manager-client)... 
DEBUG     Loading Certificate (kube-control-plane-kube-scheduler-client)... 
DEBUG     Loading Certificate (kube-control-plane-signer)... 
DEBUG     Loading Certificate (kubelet-bootstrap-kubeconfig-ca-bundle)... 
DEBUG     Loading Certificate (kubelet-client-ca-bundle)... 
DEBUG     Loading Certificate (kubelet-client)...  
DEBUG     Loading Certificate (kubelet-signer)...  
DEBUG     Loading Certificate (kubelet-serving-ca-bundle)... 
DEBUG     Loading Certificate (mcs)...             
DEBUG     Loading Machine Config Server Root CA... 
DEBUG     Loading Key Pair (service-account.pub)... 
DEBUG     Loading Release Image Pull Spec...       
DEBUG     Loading Image...                         
DEBUG   Fetching Cluster ID...                     
DEBUG   Reusing previously-fetched Cluster ID      
DEBUG   Fetching Install Config...                 
DEBUG   Reusing previously-fetched Install Config  
DEBUG   Fetching Bootstrap Ignition Config...      
DEBUG     Fetching Ironic bootstrap credentials... 
DEBUG     Reusing previously-fetched Ironic bootstrap credentials 
DEBUG     Fetching CVO Ignore...                   
DEBUG     Reusing previously-fetched CVO Ignore    
DEBUG     Fetching Install Config...               
DEBUG     Reusing previously-fetched Install Config 
DEBUG     Fetching Kubeconfig Admin Internal Client... 
DEBUG     Reusing previously-fetched Kubeconfig Admin Internal Client 
DEBUG     Fetching Kubeconfig Kubelet...           
DEBUG     Reusing previously-fetched Kubeconfig Kubelet 
DEBUG     Fetching Kubeconfig Admin Client (Loopback)... 
DEBUG     Reusing previously-fetched Kubeconfig Admin Client (Loopback) 
DEBUG     Fetching Master Ignition Customization Check... 
DEBUG     Reusing previously-fetched Master Ignition Customization Check 
DEBUG     Fetching Worker Ignition Customization Check... 
DEBUG     Reusing previously-fetched Worker Ignition Customization Check 
DEBUG     Fetching Master Machines...              
DEBUG     Reusing previously-fetched Master Machines 
DEBUG     Fetching Worker Machines...              
DEBUG     Reusing previously-fetched Worker Machines 
DEBUG     Fetching Common Manifests...             
DEBUG     Reusing previously-fetched Common Manifests 
DEBUG     Fetching Openshift Manifests...          
DEBUG     Reusing previously-fetched Openshift Manifests 
DEBUG     Fetching Proxy Config...                 
DEBUG     Reusing previously-fetched Proxy Config  
DEBUG     Fetching Certificate (admin-kubeconfig-ca-bundle)... 
DEBUG     Reusing previously-fetched Certificate (admin-kubeconfig-ca-bundle) 
DEBUG     Fetching Certificate (aggregator)...     
DEBUG     Reusing previously-fetched Certificate (aggregator) 
DEBUG     Fetching Certificate (aggregator-ca-bundle)... 
DEBUG     Reusing previously-fetched Certificate (aggregator-ca-bundle) 
DEBUG     Fetching Certificate (system:kube-apiserver-proxy)... 
DEBUG     Reusing previously-fetched Certificate (system:kube-apiserver-proxy) 
DEBUG     Fetching Certificate (aggregator-signer)... 
DEBUG     Reusing previously-fetched Certificate (aggregator-signer) 
DEBUG     Fetching Certificate (system:kube-apiserver-proxy)... 
DEBUG     Reusing previously-fetched Certificate (system:kube-apiserver-proxy) 
DEBUG     Fetching Bootstrap SSH Key Pair...       
DEBUG     Reusing previously-fetched Bootstrap SSH Key Pair 
DEBUG     Fetching User-provided Service Account Signing key... 
DEBUG     Reusing previously-fetched User-provided Service Account Signing key 
DEBUG     Fetching Cloud Provider CA Bundle...     
DEBUG     Reusing previously-fetched Cloud Provider CA Bundle 
DEBUG     Fetching Certificate (journal-gatewayd)... 
DEBUG     Reusing previously-fetched Certificate (journal-gatewayd) 
DEBUG     Fetching Certificate (kube-apiserver-lb-ca-bundle)... 
DEBUG     Reusing previously-fetched Certificate (kube-apiserver-lb-ca-bundle) 
DEBUG     Fetching Certificate (kube-apiserver-external-lb-server)... 
DEBUG     Reusing previously-fetched Certificate (kube-apiserver-external-lb-server) 
DEBUG     Fetching Certificate (kube-apiserver-internal-lb-server)... 
DEBUG     Reusing previously-fetched Certificate (kube-apiserver-internal-lb-server) 
DEBUG     Fetching Certificate (kube-apiserver-lb-signer)... 
DEBUG     Reusing previously-fetched Certificate (kube-apiserver-lb-signer) 
DEBUG     Fetching Certificate (kube-apiserver-localhost-ca-bundle)... 
DEBUG     Reusing previously-fetched Certificate (kube-apiserver-localhost-ca-bundle) 
DEBUG     Fetching Certificate (kube-apiserver-localhost-server)... 
DEBUG     Reusing previously-fetched Certificate (kube-apiserver-localhost-server) 
DEBUG     Fetching Certificate (kube-apiserver-localhost-signer)... 
DEBUG     Reusing previously-fetched Certificate (kube-apiserver-localhost-signer) 
DEBUG     Fetching Certificate (kube-apiserver-service-network-ca-bundle)... 
DEBUG     Reusing previously-fetched Certificate (kube-apiserver-service-network-ca-bundle) 
DEBUG     Fetching Certificate (kube-apiserver-service-network-server)... 
DEBUG     Reusing previously-fetched Certificate (kube-apiserver-service-network-server) 
DEBUG     Fetching Certificate (kube-apiserver-service-network-signer)... 
DEBUG     Reusing previously-fetched Certificate (kube-apiserver-service-network-signer) 
DEBUG     Fetching Certificate (kube-apiserver-complete-server-ca-bundle)... 
DEBUG     Reusing previously-fetched Certificate (kube-apiserver-complete-server-ca-bundle) 
DEBUG     Fetching Certificate (kube-apiserver-complete-client-ca-bundle)... 
DEBUG     Reusing previously-fetched Certificate (kube-apiserver-complete-client-ca-bundle) 
DEBUG     Fetching Certificate (kube-apiserver-to-kubelet-ca-bundle)... 
DEBUG     Reusing previously-fetched Certificate (kube-apiserver-to-kubelet-ca-bundle) 
DEBUG     Fetching Certificate (kube-apiserver-to-kubelet-client)... 
DEBUG     Reusing previously-fetched Certificate (kube-apiserver-to-kubelet-client) 
DEBUG     Fetching Certificate (kube-apiserver-to-kubelet-signer)... 
DEBUG     Reusing previously-fetched Certificate (kube-apiserver-to-kubelet-signer) 
DEBUG     Fetching Certificate (kube-control-plane-ca-bundle)... 
DEBUG     Reusing previously-fetched Certificate (kube-control-plane-ca-bundle) 
DEBUG     Fetching Certificate (kube-control-plane-kube-controller-manager-client)... 
DEBUG     Reusing previously-fetched Certificate (kube-control-plane-kube-controller-manager-client) 
DEBUG     Fetching Certificate (kube-control-plane-kube-scheduler-client)... 
DEBUG     Reusing previously-fetched Certificate (kube-control-plane-kube-scheduler-client) 
DEBUG     Fetching Certificate (kube-control-plane-signer)... 
DEBUG     Reusing previously-fetched Certificate (kube-control-plane-signer) 
DEBUG     Fetching Certificate (kubelet-bootstrap-kubeconfig-ca-bundle)... 
DEBUG     Reusing previously-fetched Certificate (kubelet-bootstrap-kubeconfig-ca-bundle) 
DEBUG     Fetching Certificate (kubelet-client-ca-bundle)... 
DEBUG     Reusing previously-fetched Certificate (kubelet-client-ca-bundle) 
DEBUG     Fetching Certificate (kubelet-client)... 
DEBUG     Reusing previously-fetched Certificate (kubelet-client) 
DEBUG     Fetching Certificate (kubelet-signer)... 
DEBUG     Reusing previously-fetched Certificate (kubelet-signer) 
DEBUG     Fetching Certificate (kubelet-serving-ca-bundle)... 
DEBUG     Reusing previously-fetched Certificate (kubelet-serving-ca-bundle) 
DEBUG     Fetching Certificate (mcs)...            
DEBUG     Reusing previously-fetched Certificate (mcs) 
DEBUG     Fetching Machine Config Server Root CA... 
DEBUG     Reusing previously-fetched Machine Config Server Root CA 
DEBUG     Fetching Key Pair (service-account.pub)... 
DEBUG     Reusing previously-fetched Key Pair (service-account.pub) 
DEBUG     Fetching Release Image Pull Spec...      
DEBUG     Reusing previously-fetched Release Image Pull Spec 
DEBUG     Fetching Image...                        
DEBUG     Reusing previously-fetched Image         
DEBUG   Generating Bootstrap Ignition Config...    
DEBUG Generating Metadata...                       
INFO Single-Node-Ignition-Config created in: crc-tmp-install-data and crc-tmp-install-data/auth 
+ cat
++ jq '.systemd.units += [{"mask": true, "name": "chronyd.service"}]' crc-tmp-install-data/bootstrap-in-place-for-live-iso.ign
++ OPENSHIFT_INSTALL_RELEASE_IMAGE_OVERRIDE=quay.io/okd/scos-release@sha256:2d20d7770f511ca89d92b37812fdab66483d3d8cc2b1e9fcd33a341350e4f827
++ ./openshift-install coreos print-stream-json
++ grep location
++ grep x86_64
++ grep iso
++ cut '-d"' -f4
+ ISO_URL=https://builds.coreos.fedoraproject.org/prod/streams/stable/builds/39.20231101.3.0/x86_64/fedora-coreos-39.20231101.3.0-live.x86_64.iso
+ curl -L https://builds.coreos.fedoraproject.org/prod/streams/stable/builds/39.20231101.3.0/x86_64/fedora-coreos-39.20231101.3.0-live.x86_64.iso -o crc-tmp-install-data/rhcos-live.iso
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  765M  100  765M    0     0  15.9M      0  0:00:47  0:00:47 --:--:-- 15.0M
+ podman run --privileged --pull always --rm -v /dev:/dev -v /run/udev:/run/udev -v /home/rokumar/go/src/github.com/crc-org/snc:/data -w /data quay.io/coreos/coreos-installer:release iso ignition embed --force --ignition-file crc-tmp-install-data/bootstrap-in-place-for-live-iso.ign crc-tmp-install-data/rhcos-live.iso
Trying to pull quay.io/coreos/coreos-installer:release...
Getting image source signatures
Copying blob 0244ee8ecc67 skipped: already exists  
Copying blob e681b1c25a38 skipped: already exists  
Copying blob 6c72606fec66 skipped: already exists  
Copying config ec69049e84 done   | 
Writing manifest to image destination
+ sudo mv -Z crc-tmp-install-data/rhcos-live.iso /var/lib/libvirt/crc/rhcos-live.iso
+ create_vm rhcos-live.iso
+ local iso=rhcos-live.iso
+ sudo virt-install --name crc --vcpus 6 --memory 14336 --arch=x86_64 --disk path=/var/lib/libvirt/crc/crc.qcow2,size=31 --network network=crc,mac=52:54:00:ee:42:e1 --os-variant rhel9-unknown --nographics --cdrom /var/lib/libvirt/crc/rhcos-live.iso --events on_reboot=restart --autoconsole none --boot uefi --wait

Starting install...
Allocating 'crc.qcow2'                                                                          | 3.4 MB  00:00:00 ... 
Creating domain...                                                                              |    0 B  00:00:00     

Domain is still running. Installation may be in progress.
Waiting for the installation to complete.

[cfergeau]

sudo virt-install --name crc --vcpus 6 --memory 14336 --arch=x86_64 --disk path=/var/lib/libvirt/crc/crc.qcow2,size=31 --network network=crc,mac=52:54:00:ee:42:e1 --os-variant rhel9-unknown --nographics --cdrom /var/lib/libvirt/crc/rhcos-live.iso --events on_reboot=restart --autoconsole none --boot uefi --wait

We could change -nographics to -graphics vnc to make it easier to debug such situations. --console pty,target_type=virtio or -serial pty would also be useful to have for debugging purposes if they are not there by default.

[praveenkumar]

okd-project/okd#2015 looks like OKD is broken and workaround as of now use RHCOS. Will try that also.

[cfergeau]

okd-project/okd#2015 looks like OKD is broken and workaround as of now use RHCOS. Will try that also.

The whole point of OKD is to not depend on RH-content, I'm not sure it's really useful to get it to work with RHCOS base images.

[praveenkumar]

The whole point of OKD is to not depend on RH-content, I'm not sure it's really useful to get it to work with RHCOS base images.

Yes, but this is only for initial start and then it would pivot to SCOS (centos stream core os) so our final bundle wouldn't be having any RH-content.

[cfergeau]

The whole point of OKD is to not depend on RH-content, I'm not sure it's really useful to get it to work with RHCOS base images.

Yes, but this is only for initial start and then it would pivot to SCOS (centos stream core os) so our final bundle wouldn't be having any RH-content.

Ah ok, that's better, thanks for the additional details!

[praveenkumar]

So when trying with RHCOS instead FCOS is also not working and it is hang during pivoting (will check without uefi also) :(