Skip to content

fix(core-amqp): stop leaking the response queue of a cancelled management operation - #7387

Open
Sagar Patel (sagar0207) wants to merge 1 commit into
Azure:mainfrom
sagar0207:fix/management-operation-queue-leak
Open

fix(core-amqp): stop leaking the response queue of a cancelled management operation#7387
Sagar Patel (sagar0207) wants to merge 1 commit into
Azure:mainfrom
sagar0207:fix/management-operation-queue-leak

Conversation

@sagar0207

Copy link
Copy Markdown
Member

Fixes #7386

1. The per-request queue leaks when an operation is cancelled

ManagementClientImpl::ExecuteOperation creates one response queue per request, keyed by a fresh UUID. It had three exits and only two cleaned up:

Exit Removed the queue
send failed yes
result received yes
WaitForResult returned null → throw OperationCancelledException no
rethrow from the catch (...) handler no

A management client lives as long as the connection that owns it, and the $cbs client runs a put-token for every authentication, so a long running client grew that map by one entry for every cancelled operation.

The removal now runs from a scope guard, so every exit clears it — including any exit added later. The request identifier is a Uuid, so a leaked entry could never be matched to a later request; the cost was memory and a receive-path map that only grows.

2. m_messageQueues was read without its lock

auto result = m_messageQueues.at(requestId)->WaitForResult(context);

Every other access to that map takes m_messageQueuesLock, including OnMessageReceived, which inserts and erases entries from the polling thread. This at() read it with no lock.

The lock cannot be held across the wait: OnMessageReceived needs the same lock to find the queue it must complete, so holding it would deadlock. The pointer is now taken under the lock and the wait runs without it. Only this function removes this request's queue, and it does so after the wait, so the pointer stays valid.

Testing

Adds TestManagement.ManagementExecuteOperationWithCancelledContext. The existing cancellation tests cancel Open and Close; none of them reach ExecuteOperation, so both exits changed here had no coverage at all.

The test asserts that a cancelled context makes the operation fail rather than succeed, hang, or crash. It deliberately does not assert which exit is taken: where the cancellation lands decides whether the send fails and returns a non-Ok status or the wait ends and throws, and both exits now go through the same guard.

What it does not cover: the leak itself is not observable from outside the library — m_messageQueues is private and there is no testing accessor on this type. Asserting the map is empty would mean adding one. The leak is reported from reading the code, and the fix is a scope guard whose correctness does not depend on which exit runs.

  • azure-core-amqp-tests: 227 / 229 pass, matching the baseline on this commit's parent.
  • The two failures (TestCbs.CbsOpenNoListener, TestMessageSendReceive.SenderCloseWhileUnsettledSendIgnoresLateDisposition) reproduce on clean main at 9dacd081 with none of this change applied. They are pre-existing and are not addressed here.

Where this came from, and what it is not

While investigating claims based security failures at scale — 450 pods, 200 connections each, ~22.6 hours, ~2M connection rebuilds driven by a 30 minute idle link timeout — the receive path reported:

AMQP Error processing ClaimsBasedSecurity: Error {Condition =amqp:internal-error,
  Description=Message Delivery Rejected: Received message correlation ID does not match request ID., Info={}}

That comes from OnMessageReceived when a response arrives whose request id is no longer in m_messageQueues. It occurred twice in 13.7M log lines, so it is not a meaningful failure source, and it is not necessarily caused by either defect here — a send that reports a non-Ok status can still have been delivered, and the response then arrives after that path has already removed the queue.

This PR is a correctness and robustness fix found by reading that code. It is not a fix for the claims based security failures in that run; those were overwhelmingly DNS related, and a ndots:1 change reduced them from roughly 10,000-84,000 per hour to 2.4 per hour across the same workload.

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 2 pipeline(s).
8 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Fixes cancelled uAMQP management operations leaking response queues and synchronizes queue-map access.

Changes:

  • Adds scope-based queue cleanup.
  • Locks queue lookup before waiting.
  • Adds cancellation coverage and changelog documentation.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.

File Description
management.cpp Implements synchronized lookup and cleanup.
management_tests.cpp Adds a cancellation test.
CHANGELOG.md Documents the fix.

💡 Add a code-review agent skill for context-aware, tailored reviews. Learn more in the docs.

Comment thread sdk/core/azure-core-amqp/test/ut/management_tests.cpp Outdated
Comment thread sdk/core/azure-core-amqp/src/impl/uamqp/amqp/management.cpp Outdated
Comment thread sdk/core/azure-core-amqp/CHANGELOG.md Outdated
…ment operation

`ManagementClientImpl::ExecuteOperation` creates one response queue for each
request, keyed by a fresh UUID. It removed that queue on the two exits that
return a result, but not on the exit that throws for a cancelled wait, and not
on the rethrow that follows a failure. A management client lives as long as the
connection that owns it, and the `$cbs` client runs a put-token for every
authentication, so a long running client grew that map by one entry for every
cancelled operation.

The removal now runs from a scope guard, so every exit clears it, including any
exit added later. The request identifier is a UUID, so a leaked entry could
never be matched to a later request; the cost was memory and a map on the
receive path that only grows.

`ExecuteOperation` also took the queue out of the map without holding the lock
that guards it:

    auto result = m_messageQueues.at(requestId)->WaitForResult(context);

Every other access to that map takes `m_messageQueuesLock`, including
`OnMessageReceived`, which inserts and removes entries from the polling thread.
The lock cannot be held across the wait, because the receive path needs the same
lock to find the queue it must complete, so the pointer is now taken under the
lock and the wait runs without it. Only this function removes this request's
queue, and it does so after the wait, so the pointer stays valid.

Adds a test that drives a failing operation. The existing cancellation tests
cancel `Open` and `Close`; none of them reach `ExecuteOperation`.

Fixes Azure#7386

Copilot-Session: 7ae0b8be-ee18-4b04-8ddf-158d6ffbd1c9
@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 2 pipeline(s).
8 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG] core-amqp: ManagementClientImpl::ExecuteOperation leaks its response queue on cancellation and reads m_messageQueues without its lock

2 participants