Recreate can take a whole shard (or Keeper quorum) down when a pod is slow to delete

[atsarevskiy]

Impact

During a routine rolling Recreate (for example a version upgrade, where every host's object-version changes), the operator can delete a host's StatefulSet and then fail to recreate it, leaving the host at Replicas=0. Recovery is deferred to a later reconcile pass; on a large installation (hundreds of hosts per cluster) that pass may not return to the host for a long time, so it stays down.

A shard reaches zero serving replicas like this:

1. The roll recreates replica A of a shard. A's pod will not terminate in time (e.g. it is wedged on Keeper and hangs on shutdown), so the delete times out and the recreate aborts — replica A is left at Replicas=0 and is not recreated until a much later reconcile pass.
2. The shard is now serving on replica B alone (single-replica, no redundancy).
3. The roll continues and reaches replica B — either as B's own turn in the rollout, or because B is independently disrupted (restart, its own wedge). Nothing checks that A is still down, so B is deleted too.
4. A and B are now both down at the same time → the shard has zero serving replicas and its data is unavailable until some later pass finally recreates them.

As the reconcile walks the whole fleet this repeats, so shards are progressively left single-replica and any one of them can hit step 4. Distributed queries that reach a zero-replica shard fail outright (DNS_ERROR: Not found address of host, ALL_CONNECTION_TRIES_FAILED), surfacing to clients as query errors. No manual action triggers this; a normal upgrade/recreate is enough, and once it starts it does not self-heal promptly.

The delete-and-recreate path is shared by both ClickHouse-server and Keeper StatefulSets, so the same sequence applies to Keeper: strand enough replicas of an ensemble (steps 1-4 across its members) and it drops below Raft quorum. Losing keeper quorum is what makes ClickHouse hosts go readonly and hang on shutdown (KEEPER_EXCEPTION) — which is exactly what makes their deletes time out at step 1, so the failure is self-reinforcing across the roll.

What happens

In the Recreate path the operator scales the StatefulSet to 0 and deletes it, then waits for the StatefulSet to actually be gone, bounded by reconcile.statefulSet.update.timeout. If the pod does not terminate within that window — for example a ClickHouse pod slow to shut down because it is wedged on Keeper (KEEPER_EXCEPTION: Connection loss / Session expired, tables in readonly) — the delete times out with poll(delete StatefulSet ...) - wait timeout, and the recreate aborts without recreating the StatefulSet. The host is left at Replicas=0 until a future re-enqueue.

Code path

Shared reconcile path (used for both ClickHouse-server and Keeper StatefulSet

• pkg/controller/common/statefulset/statefulset-reconciler.go — recreateStlete error instead of retrying, and returns before createStatefulSet()`:

  if err := r.doDeleteStatefulSet(ctx, host); err != nil {
      // ... Warning("Recreate aborted: StatefulSet delete failed %s/%s")
      // Host stays at Replicas=0 until [re-enqueue]
      return err            // <-- returns here; createStatefulSet is never r
  }
  _ = r.storage.ReconcilePVCs(ctx, host, api.DesiredStatefulSet)
  return r.createStatefulSet(ctx, host, register, opts)

• doDeleteStatefulSet() (same file) — scales the StatefulSet to 0, best-effn calls r.sts.Delete(ctx, namespace, name) and returns its error verbatim.

• pkg/controller/chi/kube/statesfulset.go — STS.Delete() deletes the Stattreturns NotFound**; the poll is what emitspoll(delete StatefulSet:/) - wait timeout. A pod stuck terminating keeps the StatefulSet present, so the poll runs to timeout. (The Keeper adapter under pkg/controller/chk/kube/` mirrors this.)

• pkg/controller/common/poller/poller.go — returns poll(%s) - wait timeout once time.Since(start) >= opts.Timeout.

• pkg/controller/common/poller/poller-options.go — the delete poll's Options.Timeout is set from config.Reconcile.StatefulSet.Update.Timeout.

• pkg/apis/clickhouse.altinity.com/v1/type_configuration_chop.go — defaultStatefulSetUpdateTimeout = 300 (seconds).

Net: a pod that will not terminate within update.timeout makes STS.Delete time out → doDeleteStatefulSet returns the error → recreateStatefulSet aborts and leaves the host at Replicas=0.

Expected

A Recreate whose graceful pod deletion can't complete should never strand thecalate the deletion (force-delete the stuck pod so StatefulSet deletioncompletes) or retry on the next pass, so a host — and therefore a shard or a Keeper ensemble — is never left at zero replicas.

Note

Distinct from #1704 (defer disrupting the last healthy replica / recovery-first ordering), and not addressed by #2070 (defer disrupting an already-healthy Keeper when quorum headroom
is tight). Neither changes the abort-on-delete-failure path, so a host or a Kranded by a delete that times out even with #1704 and #2070 applied.

Version: observed on 0.27.3.

[alex-zaitsev]

@atsarevskiy ,this part is has been significantly re-written in 0.27.4 by #2070 . We will review if problem is still present.

[atsarevskiy]

Thanks for taking a look, Alex!

I think it still should happen. We have merged most of changes from 0.27.4 in the test branch. #2070 is scoped to the CHK controller and the new Raft-safety helper (worker-raft-safety.go, worker-reconciler-chk.go, plus the CHK status types), so it doesn't touch the shared recreate path this issue is about.

The stranding is in pkg/controller/common/statefulset/statefulset-reconciler.go, in recreateStatefulSet, which both the ClickHouse-server and Keeper reconcilers call. On the 0.27.4 branch it still deletes first and returns on a delete failure before it ever reaches createStatefulSet:

if err := r.doDeleteStatefulSet(ctx, host); err != nil {
    ...
    Warning("Recreate aborted: StatefulSet delete failed %s/%s", namespace, name)
    ...
    // Host stays at Replicas=0 until then.
    return err
}
_ = r.storage.ReconcilePVCs(ctx, host, api.DesiredStatefulSet)
return r.createStatefulSet(ctx, host, register, opts)

So a pod that won't terminate within reconcile.statefulSet.update.timeout still makes doDeleteStatefulSet time out, and the host is left at Replicas=0 until a later re-enqueue, same as 0.27.3.

[alex-zaitsev]

@atsarevskiy , please confirm that #2080 fixes your issue

[alex-zaitsev]

Fixed in https://github.com/Altinity/clickhouse-operator/releases/tag/release-0.27.4