fix(core-amqp): stop leaking the response queue of a cancelled management operation - #7387
Open
Sagar Patel (sagar0207) wants to merge 1 commit into
Open
Conversation
|
Azure Pipelines: Successfully started running 2 pipeline(s). 8 pipeline(s) were filtered out due to trigger conditions. There may be pipelines that require an authorized user to comment /azp run to run. |
Contributor
There was a problem hiding this comment.
Pull request overview
Fixes cancelled uAMQP management operations leaking response queues and synchronizes queue-map access.
Changes:
- Adds scope-based queue cleanup.
- Locks queue lookup before waiting.
- Adds cancellation coverage and changelog documentation.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
management.cpp |
Implements synchronized lookup and cleanup. |
management_tests.cpp |
Adds a cancellation test. |
CHANGELOG.md |
Documents the fix. |
💡 Add a code-review agent skill for context-aware, tailored reviews. Learn more in the docs.
…ment operation
`ManagementClientImpl::ExecuteOperation` creates one response queue for each
request, keyed by a fresh UUID. It removed that queue on the two exits that
return a result, but not on the exit that throws for a cancelled wait, and not
on the rethrow that follows a failure. A management client lives as long as the
connection that owns it, and the `$cbs` client runs a put-token for every
authentication, so a long running client grew that map by one entry for every
cancelled operation.
The removal now runs from a scope guard, so every exit clears it, including any
exit added later. The request identifier is a UUID, so a leaked entry could
never be matched to a later request; the cost was memory and a map on the
receive path that only grows.
`ExecuteOperation` also took the queue out of the map without holding the lock
that guards it:
auto result = m_messageQueues.at(requestId)->WaitForResult(context);
Every other access to that map takes `m_messageQueuesLock`, including
`OnMessageReceived`, which inserts and removes entries from the polling thread.
The lock cannot be held across the wait, because the receive path needs the same
lock to find the queue it must complete, so the pointer is now taken under the
lock and the wait runs without it. Only this function removes this request's
queue, and it does so after the wait, so the pointer stays valid.
Adds a test that drives a failing operation. The existing cancellation tests
cancel `Open` and `Close`; none of them reach `ExecuteOperation`.
Fixes Azure#7386
Copilot-Session: 7ae0b8be-ee18-4b04-8ddf-158d6ffbd1c9
Sagar Patel (sagar0207)
force-pushed
the
fix/management-operation-queue-leak
branch
from
August 29, 2026 02:01
ffdba1b to
a10f352
Compare
Sagar Patel (sagar0207)
marked this pull request as ready for review
August 29, 2026 03:43
|
Azure Pipelines: Successfully started running 2 pipeline(s). 8 pipeline(s) were filtered out due to trigger conditions. There may be pipelines that require an authorized user to comment /azp run to run. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #7386
1. The per-request queue leaks when an operation is cancelled
ManagementClientImpl::ExecuteOperationcreates one response queue per request, keyed by a fresh UUID. It had three exits and only two cleaned up:WaitForResultreturned null →throw OperationCancelledExceptioncatch (...)handlerA management client lives as long as the connection that owns it, and the
$cbsclient runs a put-token for every authentication, so a long running client grew that map by one entry for every cancelled operation.The removal now runs from a scope guard, so every exit clears it — including any exit added later. The request identifier is a
Uuid, so a leaked entry could never be matched to a later request; the cost was memory and a receive-path map that only grows.2.
m_messageQueueswas read without its lockEvery other access to that map takes
m_messageQueuesLock, includingOnMessageReceived, which inserts and erases entries from the polling thread. Thisat()read it with no lock.The lock cannot be held across the wait:
OnMessageReceivedneeds the same lock to find the queue it must complete, so holding it would deadlock. The pointer is now taken under the lock and the wait runs without it. Only this function removes this request's queue, and it does so after the wait, so the pointer stays valid.Testing
Adds
TestManagement.ManagementExecuteOperationWithCancelledContext. The existing cancellation tests cancelOpenandClose; none of them reachExecuteOperation, so both exits changed here had no coverage at all.The test asserts that a cancelled context makes the operation fail rather than succeed, hang, or crash. It deliberately does not assert which exit is taken: where the cancellation lands decides whether the send fails and returns a non-Ok status or the wait ends and throws, and both exits now go through the same guard.
What it does not cover: the leak itself is not observable from outside the library —
m_messageQueuesis private and there is no testing accessor on this type. Asserting the map is empty would mean adding one. The leak is reported from reading the code, and the fix is a scope guard whose correctness does not depend on which exit runs.azure-core-amqp-tests: 227 / 229 pass, matching the baseline on this commit's parent.TestCbs.CbsOpenNoListener,TestMessageSendReceive.SenderCloseWhileUnsettledSendIgnoresLateDisposition) reproduce on cleanmainat9dacd081with none of this change applied. They are pre-existing and are not addressed here.Where this came from, and what it is not
While investigating claims based security failures at scale — 450 pods, 200 connections each, ~22.6 hours, ~2M connection rebuilds driven by a 30 minute idle link timeout — the receive path reported:
That comes from
OnMessageReceivedwhen a response arrives whose request id is no longer inm_messageQueues. It occurred twice in 13.7M log lines, so it is not a meaningful failure source, and it is not necessarily caused by either defect here — a send that reports a non-Ok status can still have been delivered, and the response then arrives after that path has already removed the queue.This PR is a correctness and robustness fix found by reading that code. It is not a fix for the claims based security failures in that run; those were overwhelmingly DNS related, and a
ndots:1change reduced them from roughly 10,000-84,000 per hour to 2.4 per hour across the same workload.