Skip to content

fix(bigtable): fix session creation leaks#13887

Open
mutianf wants to merge 1 commit into
googleapis:mainfrom
mutianf:fix/session-budget-goaway-leak
Open

fix(bigtable): fix session creation leaks#13887
mutianf wants to merge 1 commit into
googleapis:mainfrom
mutianf:fix/session-budget-goaway-leak

Conversation

@mutianf

@mutianf mutianf commented Jul 24, 2026

Copy link
Copy Markdown
Contributor

Fix budget leaks:

  1. GO_AWAY before open handshake — session goes STARTING → WAIT_SERVER_CLOSE and closes with prevState == WAIT_SERVER_CLOSE, so the release never fired.
  2. Synchronous failure in createSession — factory.createNew()/constructor/metadata-merge throws before any listener is wired, so no terminal callback ever runs to release the slot.
  3. Session wedged in STARTING forever — a stream that connects but never sends OpenSession or a GO_AWAY leaves the session stuck with no terminal callback.

@mutianf
mutianf requested review from a team as code owners July 24, 2026 16:30

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request fixes a budget slot leak in SessionPoolImpl when a session receives a GO_AWAY before completing the open handshake. It introduces a set to track session handles holding a budget reservation, ensuring exactly-once release on success or failure. A test case and fake server support are added to verify this scenario. The feedback suggests using System.nanoTime() instead of System.currentTimeMillis() in the test's polling loop to avoid potential flakiness from system clock adjustments.

Comment on lines +694 to +695
long deadlineMs = System.currentTimeMillis() + 5_000;
while (System.currentTimeMillis() < deadlineMs) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Using System.currentTimeMillis() to measure elapsed time or timeouts can make tests flaky if the system clock is adjusted (e.g., via NTP synchronization) during execution. It is safer to use System.nanoTime() for monotonic time measurements.

Suggested change
long deadlineMs = System.currentTimeMillis() + 5_000;
while (System.currentTimeMillis() < deadlineMs) {
long deadlineNs = System.nanoTime() + 5_000_000_000L;
while (System.nanoTime() < deadlineNs) {

@mutianf
mutianf force-pushed the fix/session-budget-goaway-leak branch from 48a2217 to 2e2cd0b Compare July 24, 2026 16:40
@mutianf
mutianf requested review from a team as code owners July 24, 2026 16:40
@snippet-bot

snippet-bot Bot commented Jul 24, 2026

Copy link
Copy Markdown

No region tags are edited in this PR.

This comment is generated by snippet-bot.
If you find problems with this result, please file an issue at:
https://github.com/googleapis/repo-automation-bots/issues.
To update this comment, add snippet-bot:force-run label or use the checkbox below:

  • Refresh this comment

@mutianf
mutianf force-pushed the fix/session-budget-goaway-leak branch from 2e2cd0b to 3a1200c Compare July 24, 2026 21:01
@mutianf mutianf changed the title fix(bigtable): release session-creation budget on GO_AWAY during STAR… fix(bigtable): fix session creation leaks Jul 24, 2026
@mutianf

mutianf commented Jul 24, 2026

Copy link
Copy Markdown
Contributor Author

/gemini review

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces an open-handshake timeout to prevent sessions from remaining in the STARTING state indefinitely, which would otherwise leak session-creation-budget slots. It also improves budget tracking by associating reservations directly with session handles in a new sessionsHoldingBudget set, ensuring that budget slots are released exactly once during both normal and abnormal session lifecycles (such as receiving a GO_AWAY before the handshake completes or encountering synchronous failures during creation). The review feedback identifies two critical concurrency issues in SessionPoolImpl.java where non-thread-safe collections and budget operations are accessed or modified without acquiring the poolLock, which could lead to data races.

Comment on lines +484 to +487
handle = sessions.newHandle(session);
// Bind the budget reservation made by tryReserveSession() above to this handle so it is
// released exactly once when the session becomes READY or terminates.
sessionsHoldingBudget.add(handle);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

The sessions list and sessionsHoldingBudget set are non-thread-safe collections guarded by poolLock. Since createSession runs asynchronously and does not hold poolLock during stream creation, calling sessions.newHandle(session) and sessionsHoldingBudget.add(handle) directly here introduces a data race with other threads that access these collections under poolLock (e.g., in onSessionReady or onSessionClose).

Please wrap these registration steps in a poolLock critical section.

        poolLock.lock();
        try {
          handle = sessions.newHandle(session);
          // Bind the budget reservation made by tryReserveSession() above to this handle so it is
          // released exactly once when the session becomes READY or terminates.
          sessionsHoldingBudget.add(handle);
        } finally {
          poolLock.unlock();
        }

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

wrong. createSession is guarded by pool lock.

Comment on lines +521 to +526
if (handle == null || sessionsHoldingBudget.remove(handle)) {
budget.onSessionCreationFailure();
}
if (handle != null) {
sessions.removeUnstartedHandle(handle);
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

The operations in this catch block modify sessionsHoldingBudget, budget, and sessions (via removeUnstartedHandle), all of which are non-thread-safe resources guarded by poolLock. Since createSession runs asynchronously without holding poolLock, executing these modifications without synchronization introduces a critical data race with other threads (e.g., those executing onSessionClose or onSessionReady).

Please wrap these state updates in a poolLock critical section.

      poolLock.lock();
      try {
        if (handle == null || sessionsHoldingBudget.remove(handle)) {
          budget.onSessionCreationFailure();
        }
        if (handle != null) {
          sessions.removeUnstartedHandle(handle);
        }
      } finally {
        poolLock.unlock();
      }

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same as above. CreateSession is already guarded by poolLock.

// session-creation-budget slot it reserved and never firing the terminal callback that would
// release it. Force-closing routes STARTING -> WAIT_SERVER_CLOSE -> onSessionClose, which frees
// the budget.
static final Duration OPEN_SESSION_TIMEOUT = Duration.ofSeconds(90);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe even make it 10-30 seconds

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants