Skip to content

Domain verification backoff has no recovery path once the record is fixed #290

Description

@ecv

Summary

A customer who fixes their domain verification record has no way to get re-verified promptly once the retry backoff has matured. The verification retry interval is keyed to elapsed time since the domain first went pending (verifiedCondition.LastTransitionTime), and nothing resets or expedites it: not the verification record becoming answerable, and not a user request.

Registration refresh already solves this — spec.desiredRegistrationRefreshAttempt lets a user expedite the next registry lookup. Verification has no equivalent knob.

Customer impact

A common journey: create a Domain, fumble the DNS record, come back later and fix it. By then the domain's elapsed time has walked down the configured retryIntervals schedule, and the next verification attempt can be hours or a day away. The fix is invisible until the timer fires.

In Datum's production deployment, the entire unverified fleet (152 domains) currently has a median next verification attempt ~16 hours out, worst ~35 hours. The platform convergence threshold for this path is 300s once the record is in place (https://github.com/datum-cloud/infra/issues/3692), so mature-backoff domains sit two orders of magnitude outside it.

Where the behavior lives

  • internal/controller/domain_controller.goreconcileVerification computes requeueAfter from Config.DomainVerification.GetRetryInterval(elapsed), where elapsed is time since the Verified condition's last transition. It only counts up; a record appearing does not reset it.
  • internal/config/config.goDomainVerificationConfig.RetryIntervals (deployment-supplied schedule, 5m fallback).
  • Contrast: reconcileRegistration honors spec.desiredRegistrationRefreshAttempt and sets expedite when the desired time is due.

Proposed fix

Give verification the same shape registration has:

  1. spec.desiredVerificationAttempt (or equivalent) — user- or portal-triggered "check now", bounded by rate limits, mirroring the registration expedite semantics.
  2. Optionally, reset the backoff tier when a verification attempt observes partial progress (for example, the record exists but content mismatches), so near-miss fixes converge on the early tiers instead of the mature ones.

The deployed schedule itself is deployment configuration and is tracked on the deployment side; this issue is about the missing recovery path, which no schedule tuning can fully fix.

Related: datum-cloud/engineering#43 tracks the same cadence problem for nameserver/registration data freshness.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Fields

    Priority

    None yet

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions