Skip to content

MAYBE: Drop benign SolidQueue shutdown connection errors from Honeybadger - #2192

Open
maebeale wants to merge 1 commit into
mainfrom
maebeale/san-jose
Open

MAYBE: Drop benign SolidQueue shutdown connection errors from Honeybadger#2192
maebeale wants to merge 1 commit into
mainfrom
maebeale/san-jose

Conversation

@maebeale

Copy link
Copy Markdown
Collaborator

🤖 suggested review level: 3 Read 📖 small before_notify filter with one guard condition, plus unit spec

What & why

  • A terminating worker pod raises ActiveRecord::ConnectionNotEstablished (TRILOGY_CLOSED_CONNECTION) in at_exit while SolidQueue deregisters its Process record after the DB pool has closed. Pages on every deploy/restart, 0 users affected, nothing lost.
  • Adds a Honeybadger before_notify hook that halts these — but only when raised outside any request (component blank), so a genuine mid-request/job DB outage still reports.

Notes

  • Kept narrow on purpose: requires the closed-connection sentinel and no component. Infra-side fix (pod terminationGracePeriod / stop ordering) left out as riskier; suppression first.
  • Spec exercises the real registered hook: drops the shutdown case, keeps in-request + "connection refused".

A terminating worker pod raises ActiveRecord::ConnectionNotEstablished
(TRILOGY_CLOSED_CONNECTION) in at_exit while deregistering its SolidQueue
Process record after the DB pool has closed. It pages on every deploy/restart,
affects no users, and loses no work. Halt only closed-connection errors raised
outside any request so genuine mid-request DB outages still report.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Copilot AI lite review requested due to automatic review settings August 13, 2026 22:38

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@maebeale
maebeale marked this pull request as ready for review August 13, 2026 22:59
@maebeale
maebeale requested a review from jmilljr24 August 13, 2026 22:59

@jmilljr24 jmilljr24 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same comment as before on the other error. This is the first seeing it. I don't know what this code does but I know honeybadger is wrapped around basically everything so if this is incorrect it could take the whole site down.

I'll accept a tiny bit of noise on the hb notifications.

@maebeale maebeale changed the title Drop benign SolidQueue shutdown connection errors from Honeybadger MAYBE: Drop benign SolidQueue shutdown connection errors from Honeybadger Aug 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants