fix(relay): harden community-deletion lifecycle interactions - #7015
Open
navenduagarwal wants to merge 1 commit into
Open
fix(relay): harden community-deletion lifecycle interactions#7015navenduagarwal wants to merge 1 commit into
navenduagarwal wants to merge 1 commit into
Conversation
Two production-observed issues on deployments using the community deletion control plane, both benign-looking but permanent: 1. The NIP-43 membership reconciler sweeps every community, including quiescing/fenced/tombstone ones — whose database write fence correctly rejects the snapshot write. Result: after any community deletion reaches its fence step, every reconciliation sweep logs a 'community write fenced' failure per retired community, forever. Sweep only lifecycle-'active' communities (new usage::writable_community_hosts). 2. The boot-time deletion serving-fence validation is fail-closed with no retry, so a transient DB pool timeout — routine during rolling deploys when old and new pods briefly share the connection budget — flaps a fresh pod into CrashLoopBackOff. Retry 3x with backoff before aborting; persistent failure still fails the boot. Both observed and fixed in a ~7.7k-user production deployment; the reconciler loop produced continuous warnings after retiring three communities, and the boot flap reproduced under connection pressure during rolling restarts. Signed-off-by: Navendu Agarwal <navendu.agarwal+ola@olacabs.com>
🔐 Codex Security Review
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two production-observed issues on a deployment (~7.7k users) using the community deletion control plane. Both look benign and are permanent:
1. The NIP-43 membership reconciler sweeps deleted communities forever. The sweep iterates every community, including quiescing/fenced/tombstone ones — whose database write fence correctly rejects the snapshot write. After any community deletion reaches its fence step, every reconciliation cycle logs a
community write fencedfailure per retired community, indefinitely (we observed ~18 log lines/min after retiring three communities). Fix: sweep only lifecycle-activecommunities via a newwritable_community_hostsquery.2. Boot-time serving-fence validation flaps fresh pods under rolling deploys. The validation is fail-closed with no retry, so a transient DB pool timeout — routine while old and new pods briefly share the connection budget — exits the pod into CrashLoopBackOff. We reproduced this repeatedly under connection pressure. Fix: retry 3x with exponential backoff before aborting; persistent failure still fails the boot, preserving the fail-closed contract.
Both fixes are running in our production deployment.