Skip to content

[Perf] Capture async-health signals within the performance benchmarks - #5068

Draft
mdaigle wants to merge 4 commits into
mainfrom
dev/mdaigle/automatic-openasync-perf-coverage
Draft

mdaigle wants to merge 4 commits into
mainfrom
dev/mdaigle/automatic-openasync-perf-coverage

Conversation

@mdaigle

@mdaigle mdaigle commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Description

Capture async-health regression signals inside the existing BenchmarkDotNet performance-test flow. The previously added standalone pre-BDN sweep, CLI experiment harness, network-shaping helpers, custom sweep gates and their tests have been removed from this PR. Pipeline timeout defaults are unchanged; there is no separate ramp or sweep before the perf tests.

  1. Async-health diagnoser: observe BDN's own startup/warmup and extra diagnostic iteration separately. Normal timing iterations do not run the probe producer or operation recorder. Startup includes fixture initialization; the extra-iteration window excludes successful fixture setup/cleanup. FirstBenchmarkInProcess identifies the first case rather than claiming every empty-pool case is process-cold.
  2. Automatic pipeline measurements: both shared entrypoints enable the diagnoser and burst unit, preserve existing tuning, retain per-side/per-repetition diagnostic artifacts even after failure, and produce benchmark-async-health.md. Existing triggers and BDN timing-regression policies remain unchanged.
  3. Regression cases represented as benchmarks: instrument existing single/parallel async opens, async pool contention, and the saturated synchronous ThreadPool control. Add eight OpenAsyncBurst cases: 64 queued workers against empty, warm, oversubscribed 16-slot and non-pooled connections, each performing open/close or open/query/close. Worker minimum 16 is explicitly set/read back and restored afterward.

Benchmark operations record offer-to-worker-start delay, open duration, offer-to-open latency, useful throughput, inline/pending opens, failures/cancellations, queued/in-flight work and missing first-worker outcomes. Every worker's first record survives the normal retention cap; dropped-record counts remain explicit. Probe/resource data includes dispatch and producer delay, missing probes, ThreadPool counts and all process threads, plus allocation/GC counters. Exception messages and connection strings are not retained.

Short/censored windows are insufficient, not healthy. Failed or incomplete instrumented work fails explicitly. The 100ms dispatch finding is diagnostic, not a new unconditional pipeline gate. Diagnostic resource/allocation data includes instrumentation bookkeeping; use identical harnesses for comparisons. BDN's batch-normalized mean is amortized cost, not individual request latency; the operation records supply those latency distributions.

Candidate-owned benchmark/diagnoser sources are compiled against both driver versions. Source baselines select their driver and matching TestCommon via MdsSourceRoot; package baselines retain MdsPackageVersion. Sequential mode rebuilds the candidate after a baseline build. Pool/SNI settings remain process-wide selections; no experimental driver switch or production driver/API change is included.

Test-only BenchmarkDotNet packages use 0.16.0-preview.1 for the required lifecycle hooks; per-case timeouts remain enforced. The shared job explicitly uses JitTieringMode.Skip to avoid the preview's additional automatic tier-promotion workload. This does not disable runtime tiered compilation or reduce configured warmups, measurement counts, or diagnostic iterations. Existing run strategies remain unchanged; marking additional long-running benchmarks as Monitoring is deferred to a separate pass.

The VM runner reserves the final ten minutes of the existing task timeout for completion polling and artifact collection. Build/setup elapsed time reduces the remaining benchmark budget. Budget expiry terminates owned benchmark processes and fails explicitly while retaining raw logs/reports under results/runs and per-unit elapsed/exit-status JSON. This protects benchmark-timeout evidence, not an indefinitely hung SDK installation or an externally killed agent/VM.

See eng/pipelines/perf/async-health.md. Automated pipelines use zero added RTT. The 100ms added-RTT reproduction below was local Linux validation only; the exact latency-injected scenario does not automatically run. No TUI, study data, standalone sweep, or artificial RTT machinery is added to the pipeline.

Issues

Related to #5014. The experiments remain preserved there; this PR does not replace or close it.

Testing

Actual local SQL reproduction

The BDN benchmarks reproduced the reported Linux scheduler stall in all three fresh-process repetitions. Tested benchmark revision f2ad8a688, local Ubuntu 24.04 WSL, .NET 10.0.12, managed SNI, 16 logical CPUs, 64 workers, worker minimum/maximum 16/32767, pool 100, SELECT 1, SQL authentication, Encrypt=true/TrustServerCertificate=true, 15-second connection timeout, and 100ms added RTT (50ms each direction). The subsequent JIT-preparation fix does not alter this unit's existing Monitoring strategy, which already bypasses that preparation stage.

Four conditions x three fresh processes produced 12 processes and 24 diagnostic windows. Each window completed 64 operations with zero failures, cancellations, queued/in-flight work or missing first-worker outcomes.

Condition / observation cohort Observed dispatch p99 across 3 runs
V2 empty pool, Startup 2723.6–2767.3ms
V1 empty pool, Startup 0.351–0.616ms
Non-pooled, Startup 1.522–3.179ms
Warm V2, ExtraIteration (fixture excluded) 0.129–0.208ms

The V2 startup result matches the earlier sweep's 2690–2791ms symptom. Its first-worker queue p99 was 2754.5–2791.4ms; offer-to-open p99 was 3451.7–3597.9ms, versus 1137.0–1229.5ms within-worker open p99. Thus successful-open timing alone misses the scheduling component. V1 completed 9.17–9.22 operations/s versus V2's 17.29–18.01 in these first batches, retaining the scheduler/throughput tradeoff. These batch-throughput values are not claimed numerically equivalent to an 8-second continuous sweep.

Cohort/censoring qualifications: cold V2 Startup windows lasted 3.672–3.821s, with 100–107 executed probes and 46–48 capacity-dropped probes explicitly retained in the counters (no producer misses or outstanding callbacks). Their observed percentile is a failure/lower-bound finding, not a qualified healthy estimate. Warm Startup includes about 39s of synchronous fixture prewarming; its large healthy probe cohort is not used as a workload-only warm/cold comparison. The table instead shows the warm ExtraIteration cohort, which has only 15–16 probes and correctly remains Insufficient, not a statistical health pass. Its fully observed per-operation offer-to-open p99 was 2.908–3.350ms. Operation throughput excludes fixture time.

Process-thread costs also remain visible: non-pooled Startup peaked at 45–46 process threads while ThreadPool count was 16; V2 Startup peaked at 31 process threads/18 ThreadPool threads. Raw phase, duration, counters and samples are preserved. Local shaping recorded traffic with zero packet drops, cleanup reported Cleaned, and a live tc check confirmed loopback returned to noqueue. The credential-bearing config existed only in a memfd and the validation process exited; no password was saved.

Normal execution ordering was tested too: three fresh Windows integrated-auth, managed-SNI, zero-added-RTT runs executed the entire unfiltered OpenAsyncBurst unit (all 8 cases each). The normal first case, OpenClose(State=Empty), captured 281–319ms startup dispatch p99. Capture therefore does not depend on the special reproduction filter. Separate Windows query-case controls reproduced V2-empty 363–408ms versus V1-empty 24–27ms across three repetitions.

The local validation helper only launches the real BDN executable and is not in this PR/pipeline. Linux used one warmup, one normal timing iteration and the normal diagnostic extra iteration per case; Windows full-unit checks used the default three timing iterations. The actual BDN commands were:

$env:PERF_BENCHMARK = 'OpenAsyncBurst'
# RUNNER_CONFIG selected the documented pool/auth settings; DATATYPES_CONFIG selected datatypes.json.
dotnet $binary --filter '*OpenQueryClose*Empty*'
dotnet $binary --filter '*OpenQueryClose*Warm*'
dotnet $binary --filter '*OpenQueryClose*NonPooled*'
# Passed across 3 fresh processes per Linux condition; V1/V2 chosen by runner config.

dotnet $binary
# Passed 3 Windows fresh-process runs of all 8 cases, without a case filter.

Preparation overhead and timeout fix

Run 192544 reached the 180-minute outer task timeout during its first pass, before OpenAsyncBurst and before result collection. VM teardown succeeded; detailed result artifacts were unavailable. Timestamped unit-start markers show approximately 59.07 minutes for the DataTypeReader pair, 32.39 for AlwaysEncryptedDataTypeReader, 30.71 for ConnectionPoolContention, and 23.43 for SqlCommand. Previous successful full executions took approximately 105–121 minutes; their logs bulk-flushed timestamps, so no per-unit timing comparison is claimed.

The preview's automatic tier-promotion stage repeatedly invokes full benchmark workloads and fixtures beyond configured warmup. A real local SQL ExecuteScalar Text/MARS=false A/B confirmed the effect:

Preparation mode Elapsed JIT operations Warmup operations Actual operations Extra operations
Auto 22.6s 48,128 1,024 20,480 (20 iterations) 1,024
Skip 10.15s 1,024 1,024 20,480 (20 iterations) 1,024

The new production-config regression test executes BDN and checks bounded preparation plus unchanged explicit warmup/actual iteration counts. Timeout tests cover shared deadlines, real owned-process termination, partial retention, invalid budgets and both shell entrypoints.

Automated checks

dotnet test src\Microsoft.Data.SqlClient\tests\UnitTests\Microsoft.Data.SqlClient.UnitTests.csproj -c Release -f net10.0 --list-tests --filter 'FullyQualifiedName~BenchmarkConfigTests|FullyQualifiedName~AsyncHealth' --nologo -v minimal
# Passed: intended preparation/health tests discovered; test project rebuilt.

dotnet test src\Microsoft.Data.SqlClient\tests\UnitTests\Microsoft.Data.SqlClient.UnitTests.csproj -c Release -f net10.0 --no-build --filter 'FullyQualifiedName~BenchmarkConfigTests|FullyQualifiedName~AsyncHealth|FullyQualifiedName~ThreadPoolHealthObserver|FullyQualifiedName~ThreadPoolProbeRecorder' --nologo -v minimal
# Passed: 16 tests, no skips. Real BDN preparation/lifecycle, persisted startup/extra operation
# counters, unobserved timing invocations, failure retention, late records and queue/open timings.

python -B -m unittest discover -s eng\pipelines\perf\scripts\tests -p test_perf_budget.py -q
# Passed: 9 tests.
python -B -m unittest discover -s eng\pipelines\perf\scripts\tests -p test_benchmark_health.py -q
# Passed: 6 tests.
python -B -m unittest discover -s eng\pipelines\perf\scripts\tests -p test_runner_config.py -q
# Passed: 5 tests across both shells.

dotnet build src\Microsoft.Data.SqlClient\tests\PerformanceTests\Microsoft.Data.SqlClient.PerformanceTests.csproj -c Release -f net10.0 --nologo -v minimal
# Passed: 0 warnings/errors.

dotnet build src\Microsoft.Data.SqlClient\tests\PerformanceTests\Microsoft.Data.SqlClient.PerformanceTests.csproj -c Release -f net10.0 --nologo -v minimal -p:ReferenceType=Package -p:MdsPackageVersion=7.1.2 -o artifacts\benchmark-only-baseline
# Passed at f2ad8a688: 0 warnings/errors.

dotnet build src\Microsoft.Data.SqlClient\tests\UnitTests\Microsoft.Data.SqlClient.UnitTests.csproj -c Release -f net462 --nologo -v minimal
# Passed: 0 warnings/errors.

git diff --cached --check
# Passed. Complete bash/PowerShell entrypoints also passed native syntax parsing.

Three targeted timeout tests also passed under local WSL, including actual POSIX process-group termination and bash budget grace/expiry. The f2ad8a6 source was exported into an isolated local WSL directory, dotnet tool restore was run there, and the BDN host built with -c Release -f net10.0 -p:DisableSourceLink=true: passed with 0 warnings/errors.

Known broader-suite baseline failure: python -B -m unittest discover -s eng\pipelines\perf\scripts\tests -q ran 49 tests: 46 passed and the same 3 pre-existing Git Bash test_tool_restore temp-path assertions failed. Previously reproduced on untouched main; not modified or suppressed.

Remote validation: replacement run 192609 is in progress on exact head d64140e0c1e4d5d4ec119a2f21758c581b4ea68c, Linux, baseline 7.1.2, Kusto ingestion disabled. It has not yet passed. Run 192544 timed out as described above. The obsolete sweep-bearing run 192494 was cancelled and its VM teardown succeeded. The feed issue was repaired by saving missing Abstractions/Logging 7.1.2 dependencies, then verifying anonymous downloads and clean-cache restores.

  • Tests added or updated
  • Public API changes documented (not applicable; production API unchanged)
  • Verified against customer repro (actual local Linux 100ms RTT BDN cases reproduce the reported stall in 3/3 fresh processes)
  • Ensure no breaking changes introduced (production driver unchanged)

Guidelines

Please review the contribution guidelines before submitting a pull request:

Build one candidate-owned harness against both drivers and retain matched regression, baseline-debt and invalid-evidence outcomes.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@mdaigle
mdaigle requested a review from a team as a code owner November 13, 2026 21:36
Copilot AI balanced review requested due to automatic review settings November 13, 2026 21:36
@github-project-automation github-project-automation Bot moved this to To triage in SqlClient Board Nov 13, 2026
@mdaigle mdaigle added this to the 8.1.0-preview1 milestone Nov 13, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Sequential package comparisons in both scripts can reuse baseline binaries for the current pass, hiding candidate regressions, and the health build fails for the specified preview baseline because required dependency overrides are missing.

3 open issues
What changed in this PR

Adds a bounded OpenAsync health phase before BenchmarkDotNet in SqlClient’s existing performance runners, without changing production driver code or pipeline triggers. This is separate from the broader experiments in #5014.

Changes:

  • Adds 20 workload cases with three paired repetitions, using the same harness against candidate and baseline drivers.
  • Introduces matched regression gating, raw evidence, summaries, and optional Linux network-delay testing; default runs add no delay.
  • Integrates both runners, increases timeout defaults, and adds tests and documentation.
File Description
src/​Microsoft.Data.SqlClient/​tests/​UnitTests/​Microsoft.Data.SqlClient.UnitTests.csproj Links harness sources into modern .NET unit tests.
src/​Microsoft.Data.SqlClient/​tests/​UnitTests/​Measurements/​ThreadPoolProbeRecorderTests.cs Tests bounded probe accounting.
src/​Microsoft.Data.SqlClient/​tests/​UnitTests/​Measurements/​ThreadPoolHealthObserverTests.cs Tests observer failure and incomplete evidence.
src/​Microsoft.Data.SqlClient/​tests/​UnitTests/​Measurements/​OpenAsyncHealthSettingsTests.cs Tests workload-setting validation.
src/​Microsoft.Data.SqlClient/​tests/​UnitTests/​Measurements/​OpenAsyncHealthRecorderTests.cs Tests outcome retention and counter snapshots.
src/​Microsoft.Data.SqlClient/​tests/​PerformanceTests/​OpenAsyncHealth/​Program.cs Adds the standalone command-line entrypoint.
src/​Microsoft.Data.SqlClient/​tests/​PerformanceTests/​OpenAsyncHealth/​OpenAsyncHealth.csproj Supports source and package driver references.
src/​Microsoft.Data.SqlClient/​tests/​PerformanceTests/​Microsoft.Data.SqlClient.PerformanceTests.csproj Excludes standalone harness sources from BenchmarkDotNet.
src/​Microsoft.Data.SqlClient/​tests/​PerformanceTests/​Measurements/​ThreadPoolProbeRecorder.cs Records probe timing and missing callbacks.
src/​Microsoft.Data.SqlClient/​tests/​PerformanceTests/​Measurements/​ThreadPoolHealthObserver.cs Samples scheduling and process resources.
src/​Microsoft.Data.SqlClient/​tests/​PerformanceTests/​Measurements/​OpenAsyncHealthRecorder.cs Retains operation outcomes and counters.
src/​Microsoft.Data.SqlClient/​tests/​PerformanceTests/​Measurements/​OpenAsyncHealthExperiment.cs Executes bounded connection workloads.
eng/​pipelines/​perf/​sqlclient-perf-pr-pipeline.yml Adds 30 minutes to the PR timeout default.
eng/​pipelines/​perf/​sqlclient-perf-pipeline.yml Adds 30 minutes to the standard timeout default.
eng/​pipelines/​perf/​sqlclient-perf-experiment.yml Adds 30 minutes to the experiment timeout default.
eng/​pipelines/​perf/​scripts/​tests/​test_openasync_health.py Tests evidence validation, comparisons, and orchestration.
eng/​pipelines/​perf/​scripts/​tests/​test_openasync_health_runners.py Tests both runner integrations and exit handling.
eng/​pipelines/​perf/​scripts/​tests/​test_openasync_health_process.py Adds opt-in executable lifecycle tests.
eng/​pipelines/​perf/​scripts/​tests/​test_openasync_health_netem.py Tests network-rule ownership and cleanup.
eng/​pipelines/​perf/​scripts/​run-perf-tests.sh Runs health coverage before Linux benchmarks.
eng/​pipelines/​perf/​scripts/​run-perf-tests.ps1 Runs health coverage before Windows benchmarks.
eng/​pipelines/​perf/​scripts/​openasync_health.py Builds, pairs, validates, and reports health runs.
eng/​pipelines/​perf/​scripts/​netem.py Adds optional SQL-only loopback delay.
eng/​pipelines/​perf/​README.md Documents automatic coverage and timeout changes.
eng/​pipelines/​perf/​openasync-health.md Explains workloads, gates, artifacts, and limitations.

Review effort: Balanced

@mdaigle
mdaigle marked this pull request as draft November 13, 2026 22:14
Observe marked BDN workloads in extra iterations, run burst regression cases automatically, and build matched candidate benchmark sources against source baselines.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings November 16, 2026 17:48
@mdaigle mdaigle changed the title [Perf] Automatically run matched OpenAsync health coverage [Perf] Add async-health diagnoser, automated measurements, and regression benchmarks Nov 16, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The new performance gates and BDN lifecycle changes need human review of authenticated SQL pipeline runs on Linux and Windows, since SQL-free checks do not establish end-to-end measurement validity for this revision.

1 open issue
2 resolved since last review

Review effort: Balanced

@codecov

codecov Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 72.51%. Comparing base (17bef7b) to head (d64140e).
⚠️ Report is 11 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #5068      +/-   ##
==========================================
+ Coverage   68.99%   72.51%   +3.52%     
==========================================
  Files         314      314              
  Lines       45857    71062   +25205     
==========================================
+ Hits        31638    51533   +19895     
- Misses      14219    19529    +5310     
Flag Coverage Δ
CI-SqlClient ?
sqlclient-pr 72.51% <ø> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Remove the standalone pre-benchmark sweep and network shaping. Record startup/warmup and extra-iteration queue/open/outcome signals in the perf tests, retaining missing work and resource evidence.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings November 16, 2026 19:12
@paulmedynski paulmedynski changed the title [Perf] Add async-health diagnoser, automated measurements, and regression benchmarks [Perf] Add async-health diagnoser, automated measurements, and regression benchmarks [workflow replay] Nov 16, 2026
@paulmedynski paulmedynski changed the title [Perf] Add async-health diagnoser, automated measurements, and regression benchmarks [workflow replay] [Perf] Add async-health diagnoser, automated measurements, and regression benchmarks Nov 16, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The oversubscribed benchmark may never create pool waiters, leaving the advertised waiter-wakeup regression untested, and the PR description claims health checks and gates that are not implemented.

2 open issues
1 resolved since last review

Review effort: Balanced

Comment on lines +3 to +5
Async-health measurements are part of the ordinary BenchmarkDotNet perf tests.
There is no standalone sweep, ramp prerequisite, separate case harness, or network shaping.
The existing pipeline flow and trigger conventions are unchanged.
Comment on lines +100 to +121
var invocation = AsyncHealthDiagnoser.BeginInvocation(Workers);
var tasks = new Task[Workers];
for (int i = 0; i < tasks.Length; i++)
{
var operation = invocation?.Offer(i);
tasks[i] = Task.Run(async () =>
{
operation?.Start();
try
{
using (var connection = new SqlConnection(_connectionString))
{
Task open = connection.OpenAsync();
operation?.OpenReturned(open.IsCompleted);
await open.ConfigureAwait(false);
operation?.Opened();
}
operation?.Complete();
}
catch (Exception error)
{
operation?.Fail(error);
@mdaigle mdaigle changed the title [Perf] Add async-health diagnoser, automated measurements, and regression benchmarks [Perf] Capture async-health signals within the performance benchmarks Nov 16, 2026
Skip preview-only automatic JIT tier-promotion workload, retaining configured warmups and measurement counts. Reserve collection grace inside the pipeline task budget and retain partial per-unit logs and timings.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings November 16, 2026 23:56

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Human assessment is needed because multi-warmup timing includes setup and cleanup between batches, potentially distorting the reported workload throughput.

2 open issues

Review effort: Balanced

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: To triage

Development

Successfully merging this pull request may close these issues.

4 participants