You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
ControllerReconcileErrorRatioHigh and ControllerReconcileErrorRatioCritical should fire when a controller is genuinely failing to make progress, and should stay quiet otherwise. Today a controller with a steady, known set of failures flaps in and out of critical as unrelated healthy traffic rises and falls, and the notification doesn't say which cluster it came from.
Why this matters
Measured on the prod infra control plane for dnsrecordset-powerdns, per 10-minute window:
result
count
error
193
success
604
requeue
0
requeue_after
0
total
797
Ratio at time of measurement: 24.2%.
Six customer records cannot be written because PowerDNS rejects them as coexistence conflicts (datum-cloud/infra#3716, #315). Those retries produce a roughly constant error floor. The ratio therefore tracks the denominator — when legitimate DNS work is busy, 600-odd successes dilute the floor below threshold and the alert resolves; when real activity goes quiet, the same unchanged error floor reads as 96.77% or 100% and it fires again.
The result is an alert that flips on traffic patterns rather than on a change in controller health, firing and clearing roughly 120 times per 24h into slack-critical. On 2026-07-29 it paged the on-call engineer seven times before being silenced, and re-fired seven minutes after the silence lapsed.
Separately, sum by (controller) discards cluster, namespace and pod. The notification names the reconciler but not where it is running, and both prod and staging fire the same alert — today the only way to tell them apart is the Alertmanager source URL.
Note
An earlier version of this issue claimed the alert fires on negligible reconcile volume, citing "30 of 31 reconciles". That was wrong — it was inferred from the reported percentage rather than measured. Real volume is ~800 reconciles per 10-minute window. A minimum-volume guard would not have prevented any of the pages described above. See the comment below.
Desired outcome
A controller with a stable set of known-failing objects does not flap in and out of critical as unrelated traffic varies. Either the alert reflects sustained inability to progress, or terminal failures stop contributing to it.
The notification identifies the cluster and namespace, so whoever is paged knows where to look without inferring it from the source link.
The promtool fixtures under test/prometheus-rules/controllers/ cover the flapping case — a fixed error rate with a varying success rate — so a future change cannot reintroduce it.
dns-operator#75 will make this specific alert go quiet by eliminating the error floor. It does not fix the alert's behaviour — any controller that accumulates a steady set of permanently-failing objects will flap the same way. Worth keeping this open on its own merits.
What needs to happen
ControllerReconcileErrorRatioHighandControllerReconcileErrorRatioCriticalshould fire when a controller is genuinely failing to make progress, and should stay quiet otherwise. Today a controller with a steady, known set of failures flaps in and out of critical as unrelated healthy traffic rises and falls, and the notification doesn't say which cluster it came from.Why this matters
Measured on the prod infra control plane for
dnsrecordset-powerdns, per 10-minute window:Ratio at time of measurement: 24.2%.
Six customer records cannot be written because PowerDNS rejects them as coexistence conflicts (datum-cloud/infra#3716, #315). Those retries produce a roughly constant error floor. The ratio therefore tracks the denominator — when legitimate DNS work is busy, 600-odd successes dilute the floor below threshold and the alert resolves; when real activity goes quiet, the same unchanged error floor reads as 96.77% or 100% and it fires again.
The result is an alert that flips on traffic patterns rather than on a change in controller health, firing and clearing roughly 120 times per 24h into
slack-critical. On 2026-07-29 it paged the on-call engineer seven times before being silenced, and re-fired seven minutes after the silence lapsed.Separately,
sum by (controller)discardscluster,namespaceandpod. The notification names the reconciler but not where it is running, and both prod and staging fire the same alert — today the only way to tell them apart is the Alertmanager source URL.Note
An earlier version of this issue claimed the alert fires on negligible reconcile volume, citing "30 of 31 reconciles". That was wrong — it was inferred from the reported percentage rather than measured. Real volume is ~800 reconciles per 10-minute window. A minimum-volume guard would not have prevented any of the pages described above. See the comment below.
Desired outcome
A controller with a stable set of known-failing objects does not flap in and out of critical as unrelated traffic varies. Either the alert reflects sustained inability to progress, or terminal failures stop contributing to it.
The notification identifies the cluster and namespace, so whoever is paged knows where to look without inferring it from the source link.
The promtool fixtures under
test/prometheus-rules/controllers/cover the flapping case — a fixed error rate with a varying success rate — so a future change cannot reintroduce it.Related
Important
dns-operator#75 will make this specific alert go quiet by eliminating the error floor. It does not fix the alert's behaviour — any controller that accumulates a steady set of permanently-failing objects will flap the same way. Worth keeping this open on its own merits.