Challenges encountered when deploying to DIY K8s Cluster

[tth37]

Deploying Apache OpenWhisk on a DIY Kubernetes cluster can throw a couple of curveballs. Here's a quick rundown of the snags I hit and how I untangled them.

1. GitHub cloning hiccups in initCouchDB and installPackages

   The Problem: Due to network issues, the setup scripts couldn't clone from GitHub.

   The Fix: Tweak the scripts initdb.sh and myTask.sh under helm/openwhisk/configMapFiles/initCouchDB/ to point to a mirror instead of github.com/apache/openwhisk.

2. JVM Bus Error in Controller and Invoker Pods

   The Problem: JVM threw a bus error raised by JIT compilation, crashing both the controller and invoker pods.

   The Fix: Disable JIT and boost the heap size to 16GB to stabilize the JVM.

   # mycluster.yaml
   controller:
     jvmHeapMB: "16384"
     jvmOptions: "-Xint"
   
   invoker:
     jvmHeapMB: "16384"
     jvmOptions: "-Xint"

3. Probes for controller getting axed by Kubernetes

   The Problem: Kubernetes was terminating controller pod for taking too long, as we disabled JIT.

   The Fix: Set the readiness probe timeout to a generous 180 seconds to avoid premature termination.

   # mycluster.yaml
   probes:
     controller:
       livenessProbe:
         initialDelaySeconds: 180
         periodSeconds: 180
         timeoutSeconds: 10
       readinessProbe:
         initialDelaySeconds: 180
         periodSeconds: 180
         timeoutSeconds: 10

4. Custom Kubernetes DNS Configuration

   The Problem: Needed to align DNS settings with DIY cluster setup, otherwise will fail Nginx pod.

   The Fix: Specify the DNS service (RKE2 in my case) in the cluster config.

   # mycluster.yaml
   k8s:
     dns: rke2-coredns-rke2-coredns.kube-system

[lucky0218]

I was confused by this error for a whole day. Really appreciate your sol !