DNSRecord Heartbeat mechanism

[mikenairn]

Implement a heartbeat mechanism which will keep track of when each DNSRecord that is contributing to a distributed record set last successfully reconciled. The heartbeat of all other DNSRecords that are contributing to the record set (owners) should be monitored on each reconcile and a degraded status indicated in the status if any are falling below a predefined threshold.

Heartbeat storage

The registry will be modified in order to keep track of the heartbeat timestamps. The registry currently contains details of the owners of each record, this will be extended to optionally add further entries to indicate the last successful reconcile time (heartbeat) of each known owner:

Example TXT record value:

"heritage=external-dns,external-dns/owner=owner1;;owner2"
"kuadrant/heartbeat/owner1=zxy"
"kuadrant/heartbeat/owner2=zxy"

The registry creates an entry per endpoint, however we do not need a hearbeat per endpoint only the defined rootHost endpoint. As such the heartbeat mechanism will handle both registry entries with and without heartbeat entries and the dnsrecord controller via configuration of the registry which endpoints should include them.

Note: This will cause an update to a record on every reconcile, we will need to ensure that TXT registry entries are not included in the plans calculations when deciding if a plan has changes or not.

Heartbeat check

The dnsrecord controller will be updated to check each endpoint for heartbeat metadata. Any endpoint that has any entries for heartbeat will be considered "monitored" by the heartbeat mechanism and therefore all known owners are expected to have a heartbeat entry. Each owner will be checked for an entry, if there is no entry or the entry value is greater than a predefined duration, the record will be considered "failing".

Heartbeat status

Any heartbeat checks that are considered "failing", will result in a new status on the DNSRecord. This status condition will detail the failing record owner id and the last time it was successfully reconciled, if at all.

If previously failing heartbeats are considered healthy at a later stage the status will updated to remove the failing heartbeat status for that owner.

ToDo mnairn: define the status condition

Updates to providers based on heartbeat checks

The heartbeat mechanism, at least initially, will only be used for informing users of potential issues and will not remove or update records from the DNS provider.

Considerations:

• Default validation time will directly influence how long we should wait before considering a record failing.
• Can only report status if at least one controller is still functional.