fix(relay): edge-tolerant WS keepalive + connection lifecycle hardening - #7016
Open
navenduagarwal wants to merge 1 commit into
Open
fix(relay): edge-tolerant WS keepalive + connection lifecycle hardening#7016navenduagarwal wants to merge 1 commit into
navenduagarwal wants to merge 1 commit into
Conversation
Four related production-hardening changes to the WebSocket connection
lifecycle, from a ~7.7k-user deployment behind a CDN edge:
1. Keepalive retune: ping every 15s, disconnect after 8 missed pongs
(~2min) — was 30s/3 (90s). Some edges drop inbound WS Pong control
frames, so the old policy tore down LIVE clients (persistent
'Reconnecting' loops). Frequent pings keep the socket warm through any
idle-timeout; the tolerant pong budget only reaps genuinely dead peers.
2. Subscription-registry leak fix: REQ/EVENT/COUNT handlers are spawned
detached, so a handler still parked on auth/DB work when the socket
closed could register its subscription AFTER the teardown sweep —
orphaning it for the life of the pod (buzz_subscriptions_active leaks).
Handlers now spawn on a per-connection TaskTracker that teardown closes
and awaits before sweeping.
3. Close-reason classification: terminal log + buzz_ws_disconnects_total
{reason=pong_timeout|client_close|unauthenticated|community_deleted} —
turns 'why did connections drop' from archaeology into a query.
4. Admission-rejection observability: handler-semaphore rejections now log
pubkey/limit-type and count into buzz_admission_rejections_total
{transport,reason} instead of a bare NOTICE.
Signed-off-by: Navendu Agarwal <navendu.agarwal+ola@olacabs.com>
🔐 Codex Security Review
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Four related WebSocket lifecycle hardenings from a ~7.7k-user production deployment behind a CDN edge:
1. Edge-tolerant keepalive retune (15s ping / 8 missed pongs ≈ 2min). The current 30s/3-missed policy (90s) tears down live clients behind edges that drop inbound WS Pong control frames — users see persistent "Reconnecting" loops. Frequent pings keep idle sockets warm through any proxy read-timeout; the tolerant pong budget only reaps genuinely dead peers.
2. Subscription-registry leak fix. REQ/EVENT/COUNT handlers spawn detached, so a handler still parked on auth/DB work when its socket closes can register its subscription after the teardown sweep — orphaning it for the life of the pod (
buzz_subscriptions_activeclimbs forever). Handlers now spawn on a per-connectionTaskTrackerthat teardown closes and awaits before sweeping. The wait is bounded by any single in-flight DB query, never full historical delivery.3. Close-reason classification — terminal log +
buzz_ws_disconnects_total{reason=pong_timeout|client_close|unauthenticated|community_deleted}. Turns "why are connections dropping" from log archaeology into a metric query.4. Admission-rejection observability — handler-semaphore rejections log pubkey/limit-type and count into
buzz_admission_rejections_total{transport,reason}instead of a bare NOTICE.All four are running in our production deployment.