HDDS-11646. Intermittent timeout in TestXceiverClientMetrics

[shuan1026]

What changes were proposed in this pull request?

TestXceiverClientMetrics#testMetrics was tagged @Flaky("HDDS-11646") because of an intermittent waitFor TimeoutException.

Root cause: with the default STAND_ALONE pipeline, writes go through XceiverClientGrpc, whose sendCommandAsync already blocks the caller until the response arrives (shouldBlockAndWaitAsyncReply returns true for non-read-only requests). The test used a single background thread that fired 10 "async" writes serially and then slept 1s, so at most 1 request was ever in flight at a time, PendingOps floated back to 0 for most of each cycle, and the main thread's 100ms poll frequently missed the narrow non-zero window, causing the observed timeout.

This PR fixes the root cause by:

• Replaces the single serialized sender thread with 10 concurrent sender threads, each continuously sending blocking writes until the pending spike is observed. With genuine concurrent in-flight requests, PendingOps stays non-zero for a sustained window instead of spiking for microseconds, so the poll reliably catches it.

• computeResults is wrapped in Collections.synchronizedList for thread-safe concurrent writes, and each sender thread now calls countDown() on the shared CountDownLatch in a finally block so latch.await() can't hang if a thread exits via an exception.

• The pending count increased poll now readsXceiverClientManager.getXceiverClientMetrics().getPendingContainerOpCountMetrics(...) directly instead of going through MetricsAsserts.getMetrics(SOURCE_NAME) on every tick; the poll interval is also tightened from 100ms to 10ms.

• Adds a short retry before the final PendingOps == 0 / numPendingPutSmallFile == 0 assertions, absorbing the small window between a response future completing and its pending-metric decrement (two separate, non-atomic steps in XceiverClientGrpc's gRPC callback).

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-11646

How was this patch tested?

• Confirmed with .github/workflows/intermittent-test-check.yml all green https://github.com/shuan1026/ozone/actions/runs/31959022505

[chihsuan]

Thanks for digging into this one @shuan1026! Overall looks good.

I left a few inline comments, mainly about two behaviours the old test had that this version drops.

I also considered making this deterministic, but that seems to require a test hook in XceiverClientGrpc or the datanode. That feels beyond this Jira, so I think this direction is reasonable.

[chihsuan]

I wonder if we should keep the try inside the loop? With the current structure, the first transient error permanently stops the sender, whereas the previous code retried.

[shuan1026]

Done. try/catch is back inside the while loop.

[chihsuan]

Since we are already touching this block, would it make sense to keep the first failure somewhere and surface it if the wait times out? Right now, nothing is logged.

[shuan1026]

Done. the first sender exception is kept in an AtomicReference.

[chihsuan]

The old predicate required both pendingOps > 0 and numPendingPutSmallFile > 0 but now only the per-type counter is checked. So a regression that stops incrementing the aggregate while still incrementing the per-type counter would pass. Should we keep both?

[shuan1026]

Done. restored PendingOps > 0 && numPendingPutSmallFile > 0 via getMetrics(SOURCE_NAME), same as the original test.

The getSource NPE problem (the comment in HDDS-11646) only appears if this method is re-entered in the same JVM after close(), which was a side effect of the timeout + test-flaky rerun. With the flaky timeout fixed that path should not run, so I kept the original snapshot API. We can follow up on the metrics unregister lifecycle separately.

[chihsuan]

What do you think about waiting for both PendingOps and numPendingPutSmallFile to reach zero? We assert both below, and this avoids relying on their internal decrement order.

[shuan1026]

Done. the drain waitFor now waits for both counters to reach 0.