How to monitor external resources periodically

[anekdoti]

To monitor external resources (like databases) in the controller, e.g., to update the status of a controlled entity accordingly, events need to be requeued periodically. For example, by requeueing events when leaving the handler methods in the controller with 10 seconds delay, we can be sure that a handler is called at least every 10 seconds.

However, it seems that this way one is collecting more and more events in the queue of requeued events, which each are requeued again every 10 seconds. For example, if there were updates to the controlled entity at 10:00:01, 10:00:02, ... 10:00:09, then we will have approriate handler methods in the controller called every second forever.

What is the best way to requeue events to allow periodical actions without ending in a scenario as described above? Or do I misunderstand the event handling?

[erin-allison]

There's a couple of different (fairly easy) options that I see here:

1. Run a separate IHostedService with a timer that checks your external resources on whatever interval you desire.
2. (Ab)use the fact that the SDK's resource watcher reconnects every 60 seconds and will process a notmodified event for all watched resources at that time, and use that event to check status of the external resources.

[buehler]

In addition to what @erin-allison proposed:

What you describe is the idea of https://github.com/smartive/hosted-database-operator/. In https://github.com/smartive/hosted-database-operator/blob/master/src/HostedDatabaseOperator/Controller/HostController.cs#L42-L58 we check the connection on "created", "updated", and "not-modified" events and requeue them with 1 minute delay.

Then the status / events are updated on the entity.

But: Now if I'm thinking about, you are correct. The requeue mechanism does not check if the given resource (uuid) is already requeued which could lead to such a situation where you pile up requeued entities.

[anekdoti]

Do I understand right, that the piling up therefore just happens because reconnecting the watcher always adds a new event to the queue?

[buehler]

Exactly

[anekdoti]

Would it be possible to only add a new event during reconnect when there is no event pending? And why is there a new event added at all?

[buehler]

I guess somehow this should be possible. The reason for the new event is the watcher. The watcher (from Kubernetes, not the SDK) is initialized and returns a list of "new" objects. So when a watcher is started, all found objects are "created" for the watcher. And when Kubernetes terminates the connection to the watcher and a new one is created, the found objects are "new" to the new watcher.

[anekdoti]

Thank you very much for the clarification! But when I understand right, we should be safe when we choose a requeue delay that is shorter than the watcher timeout, right?