Check Readiness, Liveness and graceful shutdown of submission, client and continuum-adaptor

[erkannt]

Currently all our deployments query /health to determine Readiness and Liveness of the container.

• Readiness signals that the app is ready to receive requests
• failing Liveness will trigger a restart of the container

It is not recommended to use the same check for readiness and liveness.

From the official docs:

If the process in your container is able to crash on its own whenever it encounters an issue or becomes unhealthy, you do not necessarily need a liveness probe; the kubelet will automatically perform the correct action in accordance with the Pod's restartPolicy.
If you'd like your container to be killed and restarted if a probe fails, then specify a liveness probe, and specify a restartPolicy of Always or OnFailure.

I'm creating this ticket as I don't have sufficient understanding of our components to assess the questions raised by the DoD.

DoD:

• only return 200 on /health when deployment can receive and parse requests
  • client
  • submission
  • continuum-adaptor
• deployment crashes and terminate itself or has a Liveness Probe that can detect crashes/hangs/deadlocks etc.
  • client
  • submission
  • continuum-adaptor
• submission and continuum-adaptor run a single process (no nodejs cluster-mode etc)
• submission handles SIGTERM gracefully (see Properly handle sigterm reviewer-submission#223)
• default terminationGracePeriodSeconds is set to a value in the Chart that avoids submission receiving a SIGKILL (see

  reviewer/charts/libero-reviewer/values.yaml

  Line 21 in 356dbe4

  terminationGracePeriodSeconds: 120

  )

[hdrury1]

This still relevant as currently out deployments go against recommended practice regarding liveness/readiness probes. IMHO the important thing about this ticket is that whoever owns the deployment understands the issue and then can make up their mind if/how to address this. The current setup shouldn't break anything but could lead to weird side affects/issues later on. It might be fully sufficient to just remove the liveness probe if our deployments are expected to be well behaved and crash if they run into any problem.