[V2] Fix log spamming from cache misses when updating status repeatedly

[rquitales]

Proposed Changes

This PR addresses the issue of excessive log spam caused by reconcile loops being triggered in quick succession on the same Workspace object due to changes in its dependent resources (e.g., services, statefulsets). Previously, each reconcile loop attempted to update the Status subresource, but because the fetched object had an outdated resourceVersion (due to a stale cache), it would result in a conflict error.

While r.Update() invalidates the cache, r.Status().Update() does not, leading to this conflict when only the status was updated in consecutive reconcile loops. The failed reconcile loop would eventually succeed once the cache was updated, but the log spam remained an issue.

This PR resolves the problem by patching the status subresource instead of performing an update, which avoids the need for the latest resourceVersion and prevents the conflict errors from occurring.

Refs #700

Commits

1. Patch status subresource instead of using Update to prevent conflicts caused by cache miss.
2. Add a field manager to all Kubernetes API calls.

[rquitales]

We could alternatively do something like the following if we wanted to keep using .Update and reduce log spam:

err := r.Status().Update(...)
if err != nil && isResourceVersionOld(err) {
    return ctrl.Result{Requeue: true}, nil
}
return ctrl.Result{}, err

This will allow the reconcile loop to handle and act on a Stack object that is current. In this scenario, using Patch should be safe, as the spec is up-to-date, it's just that updating a status does not trigger the informer cache to be invalidated.

[blampe]

Looking at UpdateStatus makes me think patching (or more specifically put'ing) the status is fairly typical, so I'm OK with this. Although it does seem like we've still got some cache issues. Total stab in the dark but I wonder if a call to mgr.GetCache().WaitForCacheSync() before we start doing work would help?

[codecov]

Codecov Report

Attention: Patch coverage is 50.00000% with 2 lines in your changes missing coverage. Please review.

Project coverage is 33.26%. Comparing base (ab6c570) to head (bfdf78d).
Report is 30 commits behind head on v2.

Files with missing lines	Patch %	Lines
...ator/internal/controller/auto/update_controller.go	0.00%	1 Missing ⚠️
...tor/internal/controller/pulumi/stack_controller.go	0.00%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##               v2     #679      +/-   ##
==========================================
+ Coverage   33.24%   33.26%   +0.01%     
==========================================
  Files          25       25              
  Lines        3949     3950       +1     
==========================================
+ Hits         1313     1314       +1     
  Misses       2474     2474              
  Partials      162      162              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[EronWright]

I think this could lead to .status.observedGeneration regressing, e.g. from 2 to 1, if the object is cached.

The system is basically racing between the status change on the owner and a write to the owned. My theory of the root cause is:

1. The reconciler is seeing a conflict when attempting to write the status, due to optimistic concurrency.
2. The reconciler may see a cached object if reconciliation runs just after a write to the object's status. There's no guarantee that, when reconciliation begins, the cache of stacks is fresh.
3. Writes to the API Server do not update the cache.
4. There's many watch triggers for other objects, that cause a stack to be enqueued for reconciliation. When they trigger too eagerly, we race.

i would prefer we try to tune the watches on the child objects, to be less aggressive in enqueuing the owner. For example, we watch the StatefulSet but are we filtering on the relevant status change?