Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 15 additions & 5 deletions eng/pipelines/perf/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,14 @@ dedicated performance test lab (Azure Dedicated Hosts), compare the branch under
released NuGet baseline, and optionally ingest the results into an Azure Data Explorer (Kusto)
database.

BDN runs [AsyncHealthDiagnoser](async-health.md) for marked single/parallel opens, async pool
contention, the saturated synchronous control, and the new `OpenAsyncBurst` benchmarks.
The pipeline enables the diagnoser and burst unit even with an older private runner config.
Startup/warmup and the extra diagnostic iteration are observed separately; normal timing
iterations are not observed. Raw queue/open/outcome/probe/resource evidence is retained alongside
each side/repetition and summarized in `benchmark-async-health.md`. There is no separate sweep
or ramp before the benchmarks, and pipeline trigger conventions remain unchanged.

## Contents

| Path | Purpose |
Expand All @@ -17,6 +25,7 @@ database.
| `scripts/run-perf-tests.ps1` | Windows equivalent (ProcessorAffinity instead of `taskset`). |
| `scripts/interleave_perf.py` | Interleaved + best-of-N orchestrator: runs each unit baseline↔candidate back-to-back and confirms regressions across N passes. |
| `scripts/compare_perf.py` | Compares baseline vs current BenchmarkDotNet JSON → delta (md + json). Reused by the orchestrator. |
| `scripts/benchmark_health_report.py` | Summarizes BDN's workload-only diagnostic windows separately from timing comparisons. |
| `scripts/perf_to_kusto.py` | Translates BenchmarkDotNet "full" JSON → Kusto `PerfRun` + `PerfBenchmarkResult` NDJSON. |
| `scripts/ingest_kusto.py` | Queued Kusto ingestion (az CLI auth, runs on the agent). |

Expand Down Expand Up @@ -60,7 +69,7 @@ context variables are available (the VM is behind NAT and lacks the pipeline ide
| --------- | ------- | ----------- |
| `platform` | `linux` | `linux` or `windows` VM + client. |
| `dotnetFramework` | `net10.0` | TFM the benchmarks run against (`net10.0` only). |
| `testTimeoutMinutes` | `180` | Template timeout waiting for the VM run. |
| `testTimeoutMinutes` | `180` (`210` for PR) | Outer task timeout; the VM runner reserves its final 10 minutes for completion polling and artifact collection. |
| `baselineVersion` | `latest` | Highest stable MDS version in the governed feed, resolved at run time. Pin an explicit version to override. Empty = current-only (no baseline pass / comparison). |
| `regressionThreshold` | `10` | Percent slowdown (current vs baseline mean) flagged as a regression. |
| `failOnRegression` | `false` | When `true`, a candidate-slower regression **fails** the run (gate). In interleaved mode only **confirmed** regressions (best-of-N majority) fail. Default off. |
Expand Down Expand Up @@ -195,7 +204,7 @@ Switch-pipeline parameters that differ from the tables above:
| --------- | ------- | ----------- |
| `switchUnderTest` | `UseConnectionPoolV2` | The single switch to A/B. Baseline forces it `false`, current forces it `true`. |
| `failIfSwitchSlower` | `false` | Maps onto the scripts' `--fail-on-regression` gate, but means something different here: "switch on is slower" is usually the *result* you queued the run to measure, not a defect. Enable it only when asserting the switch must not be a slowdown (e.g. before flipping its default). |
| `testTimeoutMinutes` | `180` | Same as the main pipeline, not the PR pipeline's `210`: both sides are the same source, so only **one** driver build is needed. |
| `testTimeoutMinutes` | `180` | Same as the main pipeline, not the PR pipeline's `210`: both sides are the same source. |

The pipeline deliberately does **not** expose `baselineVersion` / `baselineSourceRef` (the run
scripts reject combining those with `--switch-under-test`, since a simultaneous source change would
Expand Down Expand Up @@ -289,9 +298,10 @@ The `PerformanceTests` project references Microsoft.Data.SqlClient two ways, sel
- **Baseline (package)**: `ReferenceType=Package` turns the reference into a `PackageReference`. Because the
repo uses **Central Package Management (CPM)**, the version is pinned with `VersionOverride` via
`-p:MdsPackageVersion=<version>` (a plain `Version` is ignored under CPM).
- **Baseline (source)**: no reference switching at all — the baseline ref's own copy of the perf
project is built from `../sqlclient-perf-baseline-src`, keeping its default `ProjectReference` to
that ref's driver source. Used by the PR pipeline.
- **Baseline (source)**: the candidate's perf project is built with
`-p:MdsSourceRoot=../sqlclient-perf-baseline-src`. Its driver and matching TestCommon references
come from that tree, but benchmark definitions, diagnoser and BDN version stay identical on
both sides. Baselines do not need the new benchmark unit or diagnostic CLI. Used by perf-pr.
- **Baseline (switch experiment)**: no second build at all — `--switch-under-test` measures one
source tree twice, so the scripts build the `current` variant once and point both passes at it,
differing only in the runner config each pass is handed. Used by the experiment pipeline.
Expand Down
121 changes: 121 additions & 0 deletions eng/pipelines/perf/async-health.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,121 @@
# Async health within the performance benchmarks

Async-health measurements are part of the ordinary BenchmarkDotNet perf tests.
There is no standalone sweep, ramp prerequisite, separate case harness, or network shaping.
The existing pipeline flow and trigger conventions are unchanged.
Comment on lines +3 to +5

## Benchmark coverage

| Unit | Regression signals |
| --- | --- |
| `SqlConnection` | Single OpenAsync, pooled/non-pooled and MARS off/on |
| `ParallelAsyncConnection` | Concurrent OpenAsync at 10/50/100 operations, pooled/non-pooled |
| `ConnectionPoolContention` | Async open/query/close across spare capacity through oversubscription |
| `ConnectionPoolThreadPoolPressure` | Synchronous blocking on ThreadPool workers as a pressure control |
| `OpenAsyncBurst` | 64 queued workers against empty, warm, oversubscribed (16-slot), and non-pooled connections, with separate open/close and open/query/close methods |

`OpenAsyncBurst` adds eight benchmark cases, not a preliminary suite. Each has one warmup,
three timing iterations and one invocation per iteration by default. Each completed batch
contains 64 connections; BDN normalizes timing/allocation results using `OperationsPerInvoke`.
It pins the worker minimum to 16 during each case and restores it afterward; actual minimum,
maximum and observed thread counts are retained with the diagnostic evidence.
Warm setup holds all connections simultaneously before returning them. Empty-pool setup
clears only that benchmark's pool. Query work keeps contention and network-related effects
visible; open-only work isolates acquisition/return overhead.

Pool version, SNI and optimized async behavior come from the normal runner config.
Use separate process configurations to compare V1/V2; the cached pool switch is not flipped
between cases. A requested V2 diagnostic fails explicitly if the loaded driver cannot apply it.
No experimental LongRunning driver switch is added.

## Measurement boundaries

`AsyncHealthDiagnoser` observes two parts of BDN's existing lifecycle separately:

1. **Startup/warmup:** starts before the engine runs and ends before actual timing iterations.
This retains first-open scheduling and thread-growth evidence that warmup would otherwise
discard. This window includes fixture initialization, and is labeled `Startup`.
2. **Extra diagnostic iteration:** observes BDN's extra iteration after timing. This window
excludes iteration setup/cleanup on successful execution and is labeled `ExtraIteration`.

Neither window adds a separate workload sweep. Probe production and operation recording are
disabled during normal timing iterations. Instrumented benchmark code takes a null-recorder
branch during timing; baseline and candidate use identical instrumentation.
Diagnostic resource/allocation counts include observer bookkeeping; they are not driver-only
allocation counts. For isolated allocation profiling, disable async health locally.

`FirstBenchmarkInProcess` identifies the first BDN case in that process. Only that case can
expose initial process/driver behavior; subsequent cases may reuse initialized driver/JIT/worker
state even when their SQL pool is empty. Startup is not a claim that every case is process-cold.
Interleaved pipeline execution already starts a fresh process for each benchmark unit and side.

## Captured signals

The shared dedicated-thread observer records intended probe cadence, actual posting, actual
execution, missing/rejected/dropped/outstanding probes, and a bounded recovery interval.
Raw resources include all process threads (including dedicated threads), ThreadPool threads,
queue depth, allocated bytes and Gen2 collection counts.

Marked benchmark methods also record their real operations during those diagnostic windows:

- Offer-to-worker-start queue delay, open duration and offer-to-open acquisition latency.
- Completed work throughput; started/opened/completed/failed/cancelled counters.
- Inline versus pending open completion, failed-open duration and cancelled-operation duration.
- Queued/in-flight operations and missing first-worker outcomes.
- Invocation and worker identities, so repeated invocations do not overwrite early evidence.

The recorder retains every worker's first record, including queued or incomplete work, even
after the normal 100,000-record cap. Dropped-record counts remain explicit. Snapshots freeze
the evidence so late completions cannot mutate a saved result. Exception types and SQL error
numbers are retained, never exception messages or connection strings.

Successful-only latency cannot establish health when failures or missing work exist.
Operation evidence is explicitly complete, truncated, incomplete, failed or not instrumented.
Short scheduler windows (less than 2.5 seconds or 100 probes) and censored windows are
insufficient, not healthy. The 100 ms dispatch-p99 finding is an investigative diagnostic
budget, not a product SLA or a new unconditional pipeline gate.

## Pipeline integration and artifacts

Both shared entrypoints explicitly enable the diagnoser and `OpenAsyncBurst`, including with
an older private runner config. Existing burst tuning values are preserved. BDN still runs
through the existing interleaved/sequential machinery; there is no preceding health phase.
The existing `failOnRegression` timing-comparison policy is unchanged.

Candidate-owned benchmark and diagnoser sources are compiled against both driver versions.
Source baselines use `MdsSourceRoot` to reference the baseline driver and matching TestCommon;
older baselines need not already contain the new benchmark unit or diagnostic hooks.
Package baselines use the existing `MdsPackageVersion` override.

Per-side/per-repetition `async-health/*.json` contains schema-versioned benchmark identity,
phase, settings, actual driver hash, raw probes/resources/operation records and derived summaries.
These are retained in the normal perf-results artifact, even after a failed interleaved run.
Successful runs also produce `benchmark-async-health.md` at the artifact root. Diagnostic
records are not extra timing rows in BDN comparison or Kusto ingestion.

Benchmark exceptions, missing diagnostic windows, observer faults and incomplete instrumented
work fail explicitly. Failed-window evidence and execution-failure markers are separate files.
The normal per-case timeout remains enforced. The test-only BDN packages use
`0.16.0-preview.1` because `0.15.8` lacks the necessary in-process lifecycle hooks.
SQL jobs explicitly use `JitTieringMode.Skip`: the preview's automatic tier-promotion stage
otherwise adds about 90 complete workload invocations, including repeated fixtures, beyond
the configured warmup. This does not disable runtime tiered compilation or reduce configured
warmups, timing iterations, operations, or diagnostic iterations. Reclassifying long-running
jobs as `Monitoring` is separate work; existing strategies are unchanged.
For local native/interactive profiling, disable `EnableAsyncHealth`; pipeline runs enable it.

The VM runner reserves the final ten minutes of `testTimeoutMinutes` for the template's
completion poll and artifact collection. Build/setup elapsed time counts against the remaining
benchmark budget. An expired benchmark budget terminates owned benchmark processes and fails
explicitly, but leaves the collector alive. Interleaved raw logs/reports live under
`perf-results/runs`; per-unit `.timing.json` records retain elapsed time and exit status.
This protects benchmark-timeout evidence; a hung SDK installation or an externally killed
agent/VM can still prevent collection.

## Limits

These are modern-.NET benchmarks on the selected endpoint/authentication/SNI configuration.
They do not cover every authentication, timeout, failover, transaction or Framework branch.
No artificial RTT is added. A short batch may have insufficient probe samples while still
providing useful queue/open/outcome measurements. Process thread peaks can occur between the
25 ms resource samples; raw data and explicit missing evidence must inform interpretation.
74 changes: 74 additions & 0 deletions eng/pipelines/perf/scripts/benchmark_health_report.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,74 @@
#!/usr/bin/env python3
# Licensed to the .NET Foundation under one or more agreements.
# The .NET Foundation licenses this file to you under the MIT license.

"""Render BDN startup and extra-iteration evidence separately from timing comparisons/Kusto."""

import argparse
import json
from pathlib import Path


def render(root):
"""Keep side/repetition provenance and explicit failures/insufficient windows in the summary."""
lines = [
"# BenchmarkDotNet async health", "",
"Startup/warmup and the extra diagnostic iteration are separate observations of BDN's "
"own workloads. Normal timing iterations are not observed.",
"Startup includes fixture initialization. Short/censored windows are not health passes.",
"",
"| Evidence | Benchmark | Parameters | Phase | Status | Scheduler | Dispatch p99 (ms) | "
"First queue p99 (ms) | First offer-to-open p99 (ms) | Completed/s | Failed/cancelled | "
"Queued/in-flight | Missing first outcomes | Operation evidence | Peak process threads |",
"| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |",
]
paths = sorted(root.glob("**/async-health/*.json"))
if not paths:
raise ValueError("No BDN async-health artifacts were produced.")
seen = set()
failed = False
for path in paths:
data = json.loads(path.read_text(encoding="utf-8-sig"))
if data.get("SchemaVersion") != 2 or data.get("Kind") != "BenchmarkAsyncHealth":
raise ValueError(f"Unknown BDN async-health schema: {path.name}")
# Sequential collection retains both a full artifacts tree and copied timing reports;
# identify health windows by their unique case id and result/marker filename.
identity = (data["CaseId"], path.name)
if identity in seen:
continue
seen.add(identity)
status = data["Status"]
if status not in ("Completed", "Failed"):
raise ValueError("Invalid BDN diagnostic status.")
failed |= status == "Failed"
summary = data.get("Summary")
if status == "Completed" and (not summary or not data.get("Measurement")):
raise ValueError("Completed BDN diagnostic has no measurement evidence.")
summary = summary or {}
operations = data.get("Operations") or {}
values = [
str(path.relative_to(root)), data["Benchmark"], data["Parameters"], data["Phase"], status,
summary.get("Health", "Invalid"), summary.get("DispatchP99Ms"),
summary.get("FirstWorkerQueueP99Ms"), summary.get("FirstWorkerOfferToOpenP99Ms"),
summary.get("CompletedOperationsPerSecond"),
f"{operations.get('Failed', '-')}/{operations.get('Cancelled', '-')}",
f"{operations.get('Queued', '-')}/{operations.get('InFlight', '-')}",
operations.get("FirstWorkerOutcomesMissing"),
summary.get("OperationEvidence", "Unknown"), summary.get("ProcessThreadsPeak")]
lines.append("| " + " | ".join(
str(value).replace("|", "\\|").replace("\n", " ") if value is not None else "-"
for value in values) + " |")
return "\n".join(lines) + "\n", failed


def main(argv=None):
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--results-dir", type=Path, required=True)
args = parser.parse_args(argv)
markdown, failed = render(args.results_dir)
(args.results_dir / "benchmark-async-health.md").write_text(markdown, encoding="utf-8")
return 1 if failed else 0


if __name__ == "__main__":
raise SystemExit(main())
Loading
Loading