Disable APF

[johnbelamaric]

Fixes #4023

The upstream bug is currently patched in kubernetes/kubernetes but has not been cherry-picked to release branches. We can try re-enabling this after that is done or we have upgraded to the 1.29 packages. Given that Porch is an aggregated API server, it may be that APF in the primary API server already protects Porch, but that has not been verified.

[johnbelamaric]

Note: I am verifying that this fixes the problem, we should know by tomorrow. Without this change, we see a crash about 2-3 times per day.

[johnbelamaric]

Ok, this sort of worked. No more APF errors. But I still saw one restart. Investigating further, memory spikes so my guess is the restart is an OOM kill. Here is the instance with APF disabled:

Looking back in time, we see these regular spikes, like this

Memory:

CPU:

Is there something that happens periodically in Porch? I don't see anything particularly unusual in the logs, but I do see a lot of the "overnotifying" and "sending watch" messages. Here is the histogram of log entries:

This does seem to correlate with the other spikes.

[johnbelamaric]

So my current hypothesis is this:

• we saw APF errors because APF was actually getting triggered and had this bug
• it was getting triggered because we start hammering the api server, probably due to a watch storm of some kind

The watch events being sent just before the crash are 85%+ of the log entries - we're talking over 400k in the 30m before the crash. So, there is something wrong in that watch code somewhere.

[johnbelamaric]

I don't think this is needed, #4048 is better.

/close