operator pod CrashLoopBackOff because “Liveness probe failed” after deploy

[BackMountainDevil]

the operator code can work by make run。 And I can make docker-build(make docker-build docker-push IMG=loverobot/operator:v1). Things get wrong when I deploy it.

For network reason, I edit the Makefile to save the yaml into pod.yml(quite lone, 9633 Lines).

.PHONY: deploy
deploy: manifests kustomize ## Deploy controller to the K8s cluster specified in ~/.kube/config.
	cd config/manager && $(KUSTOMIZE) edit set image controller=${IMG}
	$(KUSTOMIZE) build config/default >pod.yml
## $(KUSTOMIZE) build config/default | kubectl create -f -

Then I change a image called gcr.io/kubebuilder/kube-rbac-proxy:v0.13.0 into bitnami/kube-rbac-proxy:0.13.0. gcr image could not be reached and I find a same name image at docker hub. Then I deply the operator by kubectl create -f pod.yml

At first, the pod status is Running. But when I do test with kubectl apply -f config/samples. The operator has no responce to the test. The test result is that 2 pods will be created to run python code and save log on disk(make run pass the test).

Then I should check the pod inside. It got Liveness probe failed and Readiness probe failed. I tried to search deploy, healthz, readyz, docker in repo disscussion and issues. Any advice is welcome.

$ kubectl create -f pod.yml

$ kubectl get pods -n operator-system
NAME                                           READY   STATUS    RESTARTS   AGE
operator-controller-manager-758d4888bc-rx4w8   2/2     Running   0          96s

$ kubectl get pods -n operator-system
NAME                                           READY   STATUS             RESTARTS   AGE
operator-controller-manager-758d4888bc-rx4w8   1/2     CrashLoopBackOff   3          4m59s

$ kubectl describe pod  operator-controller-manager-758d4888bc-rx4w8  -n operator-system
Name:             operator-controller-manager-758d4888bc-rx4w8
Namespace:        operator-system
Priority:         0
Service Account:  operator-controller-manager
Node:             node04/192.168.0.122
Start Time:       Sat, 04 Feb 2023 18:26:03 +0800
Labels:           control-plane=controller-manager
                  pod-template-hash=758d4888bc
Annotations:      cni.projectcalico.org/podIP: 10.42.248.207/32
                  cni.projectcalico.org/podIPs: 10.42.248.207/32
                  kubectl.kubernetes.io/default-container: manager
Status:           Running
IP:               10.42.248.207
IPs:
  IP:           10.42.248.207
Controlled By:  ReplicaSet/operator-controller-manager-758d4888bc
Containers:
  kube-rbac-proxy:
    Container ID:  docker://fd88d07770e8402d12a3e797558dca7710d26ba0b2e7ad520826d2c8de8eba11
    Image:         bitnami/kube-rbac-proxy:0.13.0
    Image ID:      docker://sha256:e50b67fef52272242a9e434951c52f98d1e46f2456cca1118505cba3f2c88700
    Port:          8443/TCP
    Host Port:     0/TCP
    Args:
      --secure-listen-address=0.0.0.0:8443
      --upstream=http://127.0.0.1:8080/
      --logtostderr=true
      --v=0
    State:          Running
      Started:      Sat, 04 Feb 2023 18:26:05 +0800
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:     500m
      memory:  128Mi
    Requests:
      cpu:        5m
      memory:     64Mi
    Environment:  <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from operator-controller-manager-token-z527w (ro)
  manager:
    Container ID:  docker://0470b91df965461a79ad64b6fe38141a7195aee0216b0af3b700fdb0ff5c09b8
    Image:         loverobot/operator:v1
    Image ID:      docker://sha256:f75245ae928fc8614a0f7723aec3faf4625cf4997be61a1e9338ca789f3f7e48
    Port:          <none>
    Host Port:     <none>
    Command:
      /manager
    Args:
      --health-probe-bind-address=:8081
      --metrics-bind-address=127.0.0.1:8080
      --leader-elect
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Sat, 04 Feb 2023 18:31:29 +0800
      Finished:     Sat, 04 Feb 2023 18:31:32 +0800
    Ready:          False
    Restart Count:  4
    Limits:
      cpu:     500m
      memory:  128Mi
    Requests:
      cpu:        10m
      memory:     64Mi
    Liveness:     http-get http://:8081/healthz delay=15s timeout=1s period=20s #success=1 #failure=3
    Readiness:    http-get http://:8081/readyz delay=5s timeout=1s period=10s #success=1 #failure=3
    Environment:  <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from operator-controller-manager-token-z527w (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  operator-controller-manager-token-z527w:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  operator-controller-manager-token-z527w
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                 node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason     Age                       From               Message
  ----     ------     ----                      ----               -------
  Normal   Scheduled  4m21s                     default-scheduler  Successfully assigned operator-system/operator-controller-manager-758d4888bc-rx4w8 to node04
  Normal   Pulled     4m55s                     kubelet            Container image "bitnami/kube-rbac-proxy:0.13.0" already present on machine
  Normal   Created    4m55s                     kubelet            Created container kube-rbac-proxy
  Normal   Started    4m54s                     kubelet            Started container kube-rbac-proxy
  Warning  Unhealthy  2m53s                     kubelet            Liveness probe failed: Get "http://10.42.248.207:8081/healthz": dial tcp 10.42.248.207:8081: connect: connection refused
  Warning  Unhealthy  113s                      kubelet            Liveness probe failed: Get "http://10.42.248.207:8081/healthz": dial tcp 10.42.248.207:8081: connect: invalid argument
  Normal   Pulled     48s (x4 over 4m54s)       kubelet            Container image "loverobot/operator:v1" already present on machine
  Normal   Created    48s (x4 over 4m54s)       kubelet            Created container manager
  Normal   Started    48s (x4 over 4m54s)       kubelet            Started container manager
  Warning  Unhealthy  18s (x3 over 108s)        kubelet            Readiness probe failed: Get "http://10.42.248.207:8081/readyz": dial tcp 10.42.248.207:8081: connect: invalid argument
  Warning  BackOff    <invalid> (x8 over 103s)  kubelet            Back-off restarting failed container

[camilamacedo86]

See the error CrashLoopBackOff. My 2 cents here is that the error is because was not possible to get the image. You must build the image and push for a registry where you can docker pull the image in your local environment. If you are looking at it for experimental purposes, you can push in docker.io and make your repository public.

From the quickStart: https://book.kubebuilder.io/quick-start.html

However, just so you know, if you want to test it locally with make run you do not need to have the image published. However, you will not either run make deploy. make deploy will test out on cluster. You can check the quick start for further information.

[BackMountainDevil]

Thanks for your kindly reply.

If the image not exist, it does will result in CrashLoopBackOff and the Event will be Error: ImagePullBackOff. So I put the two image on the cluster(only 4 nodes. quite easy to do that). Actually, I did meet the Error: ImagePullBackOff at first, so I put the two images on every nodes of the cluster.

make run means the code run on my pc, isn't it? it shows that operator works fine. So I go to nest step: test on cluster. And I do publish to docker hub.

Thank you for helping me.

[BackMountainDevil]

The problem caused by Liveness probe， I did not set probe in code while the kustomize generate probe things. Just remove the probe things the error will be go away. This is the answer... Thank you all the same.

[BackMountainDevil]

emm I tried to add probe httpHandle in main.go follow server.go. Then the output of running code told me that the port is in used, which means kubebuilder has the part of probe. So only a few possible: my code wrong or cluster gets sick.

[BackMountainDevil]

well, since our engineer fix the k8s cluster, the operator work fine and no more any probe problem. So the problem is caused by the internal failure of the cluster cause I run the same code.

[camilamacedo86]

Closing since it is answered.