diff --git a/eng/pipelines/perf/README.md b/eng/pipelines/perf/README.md index fcd4fa7b2c..581afa8bc1 100644 --- a/eng/pipelines/perf/README.md +++ b/eng/pipelines/perf/README.md @@ -6,6 +6,36 @@ dedicated performance test lab (Azure Dedicated Hosts), compare the branch under released NuGet baseline, and optionally ingest the results into an Azure Data Explorer (Kusto) database. +BDN runs [AsyncHealthDiagnoser](async-health.md) only for the eight new `OpenAsyncBurst` cases. +Existing benchmarks retain their ordinary measurements without async-health instrumentation. +The pipeline enables the diagnoser and burst unit even with an older private runner config. +Startup/warmup and the extra diagnostic iteration are observed separately; normal timing +iterations are not observed. Raw queue/open/outcome/probe/resource evidence is retained alongside +each side/repetition and summarized in `benchmark-async-health.md`. There is no separate sweep +or ramp before the benchmarks, and pipeline trigger conventions remain unchanged. + +The compact async-health report pairs baseline/current worst-observed burst metrics across +phases and repetitions. Its leading dispatch-change indicator uses the same configured +`regressionThreshold`, but is an observed relative change, not best-of-N confirmation or an +absolute health verdict. Coverage qualifications and raw cohort provenance remain separate. + +All runner configs support explicit `ProcessIsolation` (`PerRunner` by default, or `PerCase`) +and optional `AdditionalRttMilliseconds` (zero by default). These are independent of diagnoser +annotations. The shared executable dispatcher handles direct CLI and pipeline runs, retains +per-process artifacts, and verifies owned Linux SQL-only shaping when RTT is requested. +Execution conditions also participate in ordinary timing comparisons. See the +[execution protocol and calibrated opt-in profile](async-health.md#execution-and-workload-settings). + +To queue the calibrated latency workload, select `platform=linux` and +`runnerProfile=openasync-extended-100ms`. This loads the checked-in burst-only profile instead +of merging it into the full suite. Set `useConnectionPoolV2=true` to retain the calibrated V2 +configuration; queue-time behaviour switches still take precedence over profile values. +`confirmationRuns=1` runs one matched baseline/current pass without additional confirmation +passes, preserving the profile's warmup and timing iteration counts. +The default `runnerProfile=default` retains the existing live `runnerconfig.jsonc`. +Both entrypoints accept the corresponding `--runner-profile` / `-RunnerProfile` option; +the Windows entrypoint rejects the Linux-only 100ms profile before setup. + ## Contents | Path | Purpose | @@ -17,6 +47,7 @@ database. | `scripts/run-perf-tests.ps1` | Windows equivalent (ProcessorAffinity instead of `taskset`). | | `scripts/interleave_perf.py` | Interleaved + best-of-N orchestrator: runs each unit baseline↔candidate back-to-back and confirms regressions across N passes. | | `scripts/compare_perf.py` | Compares baseline vs current BenchmarkDotNet JSON → delta (md + json). Reused by the orchestrator. | +| `scripts/benchmark_health_report.py` | Pairs worst-observed burst metrics, relative status, and separate coverage/provenance evidence. | | `scripts/perf_to_kusto.py` | Translates BenchmarkDotNet "full" JSON → Kusto `PerfRun` + `PerfBenchmarkResult` NDJSON. | | `scripts/ingest_kusto.py` | Queued Kusto ingestion (az CLI auth, runs on the agent). | @@ -60,7 +91,7 @@ context variables are available (the VM is behind NAT and lacks the pipeline ide | --------- | ------- | ----------- | | `platform` | `linux` | `linux` or `windows` VM + client. | | `dotnetFramework` | `net10.0` | TFM the benchmarks run against (`net10.0` only). | -| `testTimeoutMinutes` | `180` | Template timeout waiting for the VM run. | +| `testTimeoutMinutes` | `180` (`210` for PR) | Outer task timeout; the VM runner reserves its final 10 minutes for completion polling and artifact collection. | | `baselineVersion` | `latest` | Highest stable MDS version in the governed feed, resolved at run time. Pin an explicit version to override. Empty = current-only (no baseline pass / comparison). | | `regressionThreshold` | `10` | Percent slowdown (current vs baseline mean) flagged as a regression. | | `failOnRegression` | `false` | When `true`, a candidate-slower regression **fails** the run (gate). In interleaved mode only **confirmed** regressions (best-of-N majority) fail. Default off. | @@ -195,7 +226,7 @@ Switch-pipeline parameters that differ from the tables above: | --------- | ------- | ----------- | | `switchUnderTest` | `UseConnectionPoolV2` | The single switch to A/B. Baseline forces it `false`, current forces it `true`. | | `failIfSwitchSlower` | `false` | Maps onto the scripts' `--fail-on-regression` gate, but means something different here: "switch on is slower" is usually the *result* you queued the run to measure, not a defect. Enable it only when asserting the switch must not be a slowdown (e.g. before flipping its default). | -| `testTimeoutMinutes` | `180` | Same as the main pipeline, not the PR pipeline's `210`: both sides are the same source, so only **one** driver build is needed. | +| `testTimeoutMinutes` | `180` | Same as the main pipeline, not the PR pipeline's `210`: both sides are the same source. | The pipeline deliberately does **not** expose `baselineVersion` / `baselineSourceRef` (the run scripts reject combining those with `--switch-under-test`, since a simultaneous source change would @@ -289,9 +320,10 @@ The `PerformanceTests` project references Microsoft.Data.SqlClient two ways, sel - **Baseline (package)**: `ReferenceType=Package` turns the reference into a `PackageReference`. Because the repo uses **Central Package Management (CPM)**, the version is pinned with `VersionOverride` via `-p:MdsPackageVersion=` (a plain `Version` is ignored under CPM). -- **Baseline (source)**: no reference switching at all — the baseline ref's own copy of the perf - project is built from `../sqlclient-perf-baseline-src`, keeping its default `ProjectReference` to - that ref's driver source. Used by the PR pipeline. +- **Baseline (source)**: the candidate's perf project is built with + `-p:MdsSourceRoot=../sqlclient-perf-baseline-src`. Its driver and matching TestCommon references + come from that tree, but benchmark definitions, diagnoser and BDN version stay identical on + both sides. Baselines do not need the new benchmark unit or diagnostic CLI. Used by perf-pr. - **Baseline (switch experiment)**: no second build at all — `--switch-under-test` measures one source tree twice, so the scripts build the `current` variant once and point both passes at it, differing only in the runner config each pass is handed. Used by the experiment pipeline. diff --git a/eng/pipelines/perf/async-health.md b/eng/pipelines/perf/async-health.md new file mode 100644 index 0000000000..52ae6d5198 --- /dev/null +++ b/eng/pipelines/perf/async-health.md @@ -0,0 +1,274 @@ +# Async health within the performance benchmarks + +Async-health measurements are part of the ordinary BenchmarkDotNet perf tests. +There is no standalone sweep, ramp prerequisite, separate case harness, or network shaping. +The existing pipeline flow and trigger conventions are unchanged. + +## Benchmark coverage + +| Unit | Regression signals | +| --- | --- | +| `OpenAsyncBurst` | 64 queued workers against empty, warm, oversubscribed (16-slot), and non-pooled connections, with separate open/close and open/query/close methods | + +Only the newly added `OpenAsyncBurstRunner` carries async-health instrumentation. +Existing single/parallel connection, pool-contention and synchronous-pressure benchmarks +retain their original workloads and ordinary BDN measurements, without async-health +annotations or operation-recorder wrappers. + +`OpenAsyncBurst` adds eight benchmark cases, not a preliminary suite. Each has one warmup, +three timing iterations and one invocation per iteration by default. Each completed batch +contains 64 connections; BDN normalizes timing/allocation results using `OperationsPerInvoke`. +It pins the worker minimum to 16 during each case and restores it afterward; actual minimum, +maximum and observed thread counts are retained with the diagnostic evidence. +Warm setup holds all connections simultaneously before returning them. Empty-pool setup +clears only that benchmark's pool. Query work keeps contention and network-related effects +visible; open-only work isolates acquisition/return overhead. + +Pool version, SNI and optimized async behavior come from the normal runner config. +Use separate process configurations to compare V1/V2; the cached pool switch is not flipped +between cases. A requested V2 diagnostic fails explicitly if the loaded driver cannot apply it. +No experimental LongRunning driver switch is added. + +## Execution and workload settings + +Every `RunnerJob`, including command and large-data subclasses, supports: + +| Setting | Default | Meaning | +| --- | --- | --- | +| `ProcessIsolation` | `PerRunner` | `PerRunner` shares a benchmark process across the selected cases of one unit; `PerCase` starts a fresh process for each selected method/parameter case. There is no `Auto` and no diagnoser override. | +| `AdditionalRttMilliseconds` | `0` | Additional round-trip SQL delay. Positive values require supported Linux loopback TCP routing and successful owned netem setup. | + +The checked-in burst job explicitly selects `PerCase`. The same dispatcher is used by direct +CLI and pipeline runs. It verifies an exact selected-case fingerprint in the child, retains +per-process manifests and separate artifact directories, and stops on child failure. +BenchmarkDotNet still uses its in-process toolchain inside each child, so AppContext switches +apply to the process doing the work. + +`OpenAsyncBurstRunnerJob` also supports `SamplingProfile`: + +- `Burst` (default) retains one 64-worker wave and one operation per worker. +- `Extended` preserves that single wave for `Empty`. Other states run 16 successive waves + of 64 concurrent workers, giving 1,024 distinct worker starts per invocation. + `SteadyOperationsPerWorker` supplies a fixed positive count keyed by + `OpenClose/Warm`, `OpenClose/Oversubscribed`, `OpenClose/NonPooled`, and the corresponding + `OpenQueryClose/...` cases. Missing counts fail rather than being silently calibrated. + +Burst jobs require `Monitoring`, one launch, one warmup and one invocation per iteration. +Additional BDN warmups/invocations would mix later warm work into the startup cohort; +invalid settings fail explicitly. Timing iteration count remains configurable. + +The same workload definition drives execution and BDN's `OperationsPerInvoke`. For example, +16 waves with 20 operations per worker execute 20,480 operations, not 64. +First records for every logical worker start remain retained even when ordinary operation +records reach their cap. Other dropped records are still reported as truncation. +Interleaved extended runs use all configured repetitions even without a timing regression; +there is no additional nested confirmation loop. + +Fixed counts must be selected for the intended RTT and compared unchanged across both sides. +Sustained non-pooled work can exhaust local TCP source ports; that is a real failed workload, +not a reason to ignore socket errors or modify a shared host's TCP settings automatically. + +### Calibrated 100ms example + +[`profiles/openasync-extended-100ms.jsonc`](../../../src/Microsoft.Data.SqlClient/tests/PerformanceTests/profiles/openasync-extended-100ms.jsonc) +is an opt-in, burst-only example. It does not change the default pipeline profile. +Select it at queue time with `runnerProfile=openasync-extended-100ms`, `platform=linux`, +and `useConnectionPoolV2=true`. It replaces the full-suite config; normal credential injection +and redacted config publication still apply. Use `confirmationRuns=1` for one matched pass, +without changing the configured BDN iterations. +On the local 16-CPU Linux test host, both the current driver and 7.1.2 produced +25-second-plus ExtraIteration windows with no missing probes for all six steady cases: + +| Method | State | Operations per worker per wave | ExtraIteration duration, baseline/current | +| --- | --- | ---: | --- | +| OpenClose | Warm | 32768 | 36.0s / 35.2s | +| OpenClose | Oversubscribed | 23114 | 31.6s / 31.4s | +| OpenClose | NonPooled | 1 | 50.9s / 51.3s | +| OpenQueryClose | Warm | 20 | 32.9s / 32.9s | +| OpenQueryClose | Oversubscribed | 5 | 32.8s / 32.8s | +| OpenQueryClose | NonPooled | 1 | 52.6s / 52.5s | + +These windows retained 1,258-2,104 dispatch probes and all 1,024 first-worker starts each. +The open-only warm/oversubscribed cases still reported scheduler-health failures on both +drivers: sustained hot-loop scheduling/CPU pressure is not the same scenario as the isolated +cold-start reproduction. Their ordinary operation records also reached the retention cap; +first-worker records remained complete. Adequacy is not a promise of health or a confidence +interval. Always retain those qualifications when comparing the results. + +Calibration used one warmup, one timing iteration and the normal diagnostic extra iteration, +with identical fixed work on both sides. The example retains three timing iterations; its +full three-repetition runtime was not measured. Added RTT also lengthens fixture prewarming, +which took much more time than short open-only observations during initial calibration. +The zero-RTT extended matrix and additional cold-repeat/observer-overhead study were deferred. +Do not treat this example as a calibrated zero-RTT profile or extrapolate its evidence to a +different host without checking the recorded adequacy. + +## Optional SQL-only latency + +The outer dispatcher applies netem before starting the measured child and removes its owned +rules afterward, including failure and cooperative cancellation. A 100ms setting adds 50ms +per direction; odd millisecond values are represented exactly as half-delays in microseconds. +Only TCP packets to/from the selected local SQL address and port use the delayed band. + +The initial implementation supports Linux loopback routes, not remote endpoints, named-instance +discovery or Windows shaping. An explicitly requested unsupported condition fails. A zero value +does not require Python, `tc`, elevated permissions, or any network mutation. +Positive values require Python 3, iproute2 and noninteractive permission to invoke `tc`. +Use an exclusive SQL benchmark environment: other processes using the same endpoint would also +experience its shaped traffic while the case runs. + +The helper refuses foreign queues, serializes queue mutation, persists ownership before changing +the interface, verifies the applied delay, waits for queued packets to drain, and checks for +matched traffic, changed settings and packet drops. It never deletes a replacement owned by +another run. Forced process termination cannot guarantee `finally` execution; retained +`network-state.json` supports explicit recovery with `netem.py stop --state ... --output ...`. +An unrecovered queue is a conflict, not permission for the next run to overwrite it. + +Requested/applied RTT, requested/effective isolation, workload sizes and observer settings are +recorded as comparison conditions. Rule handles and traffic counters remain per-process +`network.json` evidence. Credentials are not passed to the helper or written to these artifacts. +Ordinary BDN timing comparisons also use the retained execution conditions, including for +unmarked runners. Different RTT/isolation/workload settings remain unpaired; missing manifests +in the new isolated-process layout fail explicitly rather than allowing a misleading delta. + +## Measurement boundaries + +`AsyncHealthDiagnoser` observes two parts of BDN's existing lifecycle separately: + +1. **Startup/warmup:** starts before the engine runs and ends before actual timing iterations. + This retains first-open scheduling and thread-growth evidence that warmup would otherwise + discard. This window includes fixture initialization, and is labeled `Startup`. +2. **Extra diagnostic iteration:** observes BDN's extra iteration after timing. This window + excludes iteration setup/cleanup on successful execution and is labeled `ExtraIteration`. + +Neither window adds a separate workload sweep. Probe production and operation recording are +disabled during normal timing iterations. Instrumented burst benchmark code takes a null-recorder +branch during timing; baseline and candidate use identical instrumentation. +Diagnostic resource/allocation counts include observer bookkeeping; they are not driver-only +allocation counts. For isolated allocation profiling, disable async health locally. + +`FirstBenchmarkInProcess` identifies the first BDN case in that process. Only that case can +expose initial process/driver behavior; subsequent cases may reuse initialized driver/JIT/worker +state even when their SQL pool is empty. Startup is not a claim that every case is process-cold. +Interleaved pipeline execution already starts a fresh process for each benchmark unit and side. + +## Captured signals + +The shared dedicated-thread observer records intended probe cadence, actual posting, actual +execution, missing/rejected/dropped/outstanding probes, and a bounded recovery interval. +Raw resources include all process threads (including dedicated threads), ThreadPool threads, +queue depth, allocated bytes and Gen2 collection counts. + +Marked benchmark methods also record their real operations during those diagnostic windows: + +- Offer-to-worker-start queue delay, open duration and offer-to-open acquisition latency. +- Completed work throughput; started/opened/completed/failed/cancelled counters. +- Inline versus pending open completion, failed-open duration and cancelled-operation duration. +- Queued/in-flight operations and missing first-worker outcomes. +- Invocation and worker identities, so repeated invocations do not overwrite early evidence. + +The recorder retains every worker's first record, including queued or incomplete work, even +after the normal 100,000-record cap. Dropped-record counts remain explicit. Snapshots freeze +the evidence so late completions cannot mutate a saved result. Exception types and SQL error +numbers are retained, never exception messages or connection strings. + +Successful-only latency cannot establish health when failures or missing work exist. +Operation evidence is explicitly complete, truncated, incomplete, failed or not instrumented. +Scheduler p99 adequacy requires at least 25 seconds, 1,000 executed probes and no missing +dispatch evidence. Resource completeness remains a separate requirement for a health pass. +First-worker queue p99 records its own sample count and adequacy: all expected starts must be +observed and at least 1,000 samples retained. A cold 64-worker wave's nearest-rank p99 is its +maximum; it remains explicitly sparse rather than a precise population-p99 estimate. +These are evidence floors, not confidence-interval guarantees. +The producer retains its 25ms cadence and allows at most 256 outstanding probes. +Short or censored windows are insufficient, not healthy. The 100 ms dispatch-p99 finding is an investigative diagnostic +budget, not a product SLA or a new unconditional pipeline gate. + +## Pipeline integration and artifacts + +Both shared entrypoints explicitly enable the diagnoser and `OpenAsyncBurst`, including with +an older private runner config. Existing burst tuning values are preserved. BDN still runs +through the existing interleaved/sequential machinery; there is no preceding health phase. +The existing `failOnRegression` timing-comparison policy is unchanged. + +Candidate-owned benchmark and diagnoser sources are compiled against both driver versions. +Source baselines use `MdsSourceRoot` to reference the baseline driver and matching TestCommon; +older baselines need not already contain the new benchmark unit or diagnostic hooks. +Package baselines use the existing `MdsPackageVersion` override. + +Per-side/per-repetition `async-health/*.json` contains schema-versioned benchmark identity, +phase, settings, actual driver hash, raw probes/resources/operation records and derived summaries. +These are retained in the normal perf-results artifact, even after a failed interleaved run. +Successful runs also produce `benchmark-async-health.md` at the artifact root. Diagnostic +records are not extra timing rows in BDN comparison or Kusto ingestion. + +### Compact comparison + +The report contains one baseline/current row per burst method, parameters and compatible +recorded configuration. It reduces Startup, ExtraIteration and available repetitions to +independent maxima for dispatch p99, first-worker queue p99, peak ThreadPool threads and peak +total process threads. It compares existing per-window percentiles; it does not pool samples +or calculate a new percentile. Different metrics may attain their maxima in different cohorts. + +Each metric cell shows `current maximum (signed percentage delta; signed raw difference)`. +The difference is current minus baseline; the percentage divides that difference by the +baseline. Missing values and every zero baseline, including zero-to-zero, produce `N/A` +percentages. Rows sort by descending unrounded dispatch-p99 percentage delta, undefined last, +with deterministic ties. + +The leading status uses the regular comparison's indicators and the pipeline's actual +`regressionThreshold` (default 10%). Dispatch increases strictly above the threshold are red, +decreases strictly below its negative are green, and boundary equality is neutral/white. +Current-only, baseline-only and undefined comparisons retain the new/removed/unknown indicators. +This is an **observed relative change**, not the normal timing report's best-of-N majority +confirmation and not an absolute scheduler-health verdict. It adds no regression failure gate. + +Pool/SNI/async selections, runtime, OS, CPU/worker counts, process-cold position and recorded +workload settings must match. Incompatible configurations remain unpaired, with explicit +new/removed rows rather than misleading deltas. This also applies to intentional switch +experiments whose settings differ. Process IDs and driver identities may differ across sides; +mixing multiple driver identities within one side/configuration is rejected. + +Coverage warnings remain outside the compact table. Expanded provenance retains each side's +health classifications, phase completeness, sample/missing-probe counts, operation problems, +driver identity, repetition, process ID, raw artifact aliases and the cohort(s) supplying every +maximum. Identical aliases are counted once; conflicting data/origins fail explicitly. A green +relative indicator never hides insufficient, censored, incomplete or failed evidence. + +To regenerate a historical preview without replacing the original report: + +```powershell +python eng\pipelines\perf\scripts\benchmark_health_report.py ` + --results-dir --threshold 10 ` + --out-md \historical-comparison.md ` + --source-label "Historical run data, regenerated with the current report format" +``` + +### Execution and collection + +Benchmark exceptions, missing diagnostic windows, observer faults and incomplete instrumented +work fail explicitly. Failed-window evidence and execution-failure markers are separate files. +The normal per-case timeout remains enforced. The test-only BDN packages use +`0.16.0-preview.1` because `0.15.8` lacks the necessary in-process lifecycle hooks. +SQL jobs explicitly use `JitTieringMode.Skip`: the preview's automatic tier-promotion stage +otherwise adds about 90 complete workload invocations, including repeated fixtures, beyond +the configured warmup. This does not disable runtime tiered compilation or reduce configured +warmups, timing iterations, operations, or diagnostic iterations. Reclassifying long-running +jobs as `Monitoring` is separate work; existing strategies are unchanged. +For local native/interactive profiling, disable `EnableAsyncHealth`; pipeline runs enable it. + +The VM runner reserves the final ten minutes of `testTimeoutMinutes` for the template's +completion poll and artifact collection. Build/setup elapsed time counts against the remaining +benchmark budget. An expired benchmark budget terminates owned benchmark processes and fails +explicitly, but leaves the collector alive. Interleaved raw logs/reports live under +`perf-results/runs`; per-unit `.timing.json` records retain elapsed time and exit status. +This protects benchmark-timeout evidence; a hung SDK installation or an externally killed +agent/VM can still prevent collection. + +## Limits + +These are modern-.NET benchmarks on the selected endpoint/authentication/SNI configuration. +They do not cover every authentication, timeout, failover, transaction or Framework branch. +No artificial RTT is added by default. A short batch may have insufficient probe samples while still +providing useful queue/open/outcome measurements. Process thread peaks can occur between the +25 ms resource samples; raw data and explicit missing evidence must inform interpretation. diff --git a/eng/pipelines/perf/scripts/benchmark_health_report.py b/eng/pipelines/perf/scripts/benchmark_health_report.py new file mode 100644 index 0000000000..544d4ad93c --- /dev/null +++ b/eng/pipelines/perf/scripts/benchmark_health_report.py @@ -0,0 +1,356 @@ +#!/usr/bin/env python3 +# Licensed to the .NET Foundation under one or more agreements. +# The .NET Foundation licenses this file to you under the MIT license. + +"""Compare worst-observed OpenAsyncBurst health metrics, retaining raw cohort provenance.""" + +import argparse +from collections import Counter, defaultdict +import json +import math +from pathlib import Path +import re + +import compare_perf + + +METRICS = { + "DispatchP99Ms": "Dispatch p99 (ms)", + "FirstWorkerQueueP99Ms": "First worker queue p99 (ms)", + "ThreadPoolThreadsPeak": "Peak ThreadPool threads", + "ProcessThreadsPeak": "Peak process threads", +} +PHASES = {"Startup", "ExtraIteration"} +CONFIG_FIELDS = { + "UseConnectionPoolV2", "UseManagedSni", "UseOptimizedAsync", "Runtime", "OS", + "ProcessorCount", "MinimumWorkerThreads", "MaximumWorkerThreads", + "DiagnosticRunMode", "FirstBenchmarkInProcess", "ProcessId", "DriverVersion", "DriverSha256", +} +PROCESS_FIELDS = {"ProcessId", "DriverVersion", "DriverSha256"} +OUTCOME_FIELDS = ( + "Failed", "Cancelled", "OpenFailures", "Queued", "InFlight", + "FirstWorkerOutcomesMissing", "ObservationsDropped", +) + + +def _configuration(data): + """Match recorded workload/settings, allowing process IDs and driver identity to differ.""" + config = data.get("Configuration") + if not isinstance(config, dict) or any(config.get(key) is None for key in CONFIG_FIELDS): + raise ValueError(f"Missing BDN matching configuration: {data['CaseId']}") + result = {key: value for key, value in config.items() if key not in PROCESS_FIELDS} + for key in ("ConfiguredInvocationsPerIteration", "OperationsPerInvoke"): + value = data.get(key) + if type(value) is not int or value <= 0: + raise ValueError(f"Invalid BDN workload configuration: {key}") + result[key] = value + return json.dumps(result, sort_keys=True, separators=(",", ":")) + + +def _origin(path): + """Read side and optional repetition from either canonical or copied artifact layouts.""" + sides = [part for part in path.parts[:-1] if part in ("baseline", "current")] + repetitions = {int(match[1]) for part in path.parts[:-1] + if (match := re.fullmatch(r"rep([1-9][0-9]*)", part))} + if len(sides) != 1 or len(repetitions) > 1: + raise ValueError(f"Ambiguous BDN artifact origin: {path}") + return sides[0], next(iter(repetitions), None) + + +def _load(root): + """Deduplicate aliases, rejecting conflicting contents or origin instead of choosing one.""" + seen, owners = {}, {} + for path in sorted(root.glob("**/async-health/*.json")): + data = json.loads(path.read_text(encoding="utf-8-sig")) + if data.get("SchemaVersion") != 2 or data.get("Kind") != "BenchmarkAsyncHealth": + raise ValueError(f"Unknown BDN async-health schema: {path.name}") + benchmark = data.get("Benchmark") + if not isinstance(benchmark, str): + raise ValueError(f"Missing BDN benchmark identity: {path.name}") + if benchmark.rpartition(".")[0].rsplit(".", 1)[-1] != "OpenAsyncBurstRunner": + continue + if (not isinstance(data.get("CaseId"), str) or not data["CaseId"] or + not isinstance(data.get("Parameters"), str)): + raise ValueError(f"Invalid BDN case identity: {path.name}") + status = data["Status"] + if status not in ("Completed", "Failed"): + raise ValueError("Invalid BDN diagnostic status.") + if (not isinstance(data.get("Summary"), (dict, type(None))) or + not isinstance(data.get("Measurement"), (dict, type(None)))): + raise ValueError("Invalid BDN measurement evidence.") + if status == "Completed" and (not data.get("Summary") or not data.get("Measurement")): + raise ValueError("Completed BDN diagnostic has no measurement evidence.") + execution = data.get("Measurement") is None + if data["Phase"] not in PHASES and not (execution and status == "Failed"): + raise ValueError("Invalid BDN diagnostic phase.") + signature = _configuration(data) + relative = path.relative_to(root) + side, repetition = _origin(relative) + owner = (side, benchmark, data["Parameters"]) + if owners.setdefault(data["CaseId"], owner) != owner: + raise ValueError(f"Conflicting BDN case origins: {data['CaseId']}") + identity = (data["CaseId"], "execution" if execution else data["Phase"]) + if identity in seen: + previous = seen[identity] + if (previous["data"] != data or previous["side"] != side or + (repetition is not None and previous["repetition"] is not None and + repetition != previous["repetition"])): + raise ValueError(f"Conflicting BDN artifact aliases: {data['CaseId']}") + if repetition is not None: + previous["repetition"] = repetition + previous["sources"].append(relative.as_posix()) + else: + seen[identity] = dict(data=data, side=side, repetition=repetition, + sources=[relative.as_posix()], configuration=signature) + if not seen: + raise ValueError("No BDN OpenAsyncBurst async-health artifacts were produced.") + return list(seen.values()) + + +def _number(value): + if value is not None and (type(value) not in (int, float) or + not math.isfinite(value) or value < 0): + raise ValueError(f"Invalid nonnegative BDN metric: {value!r}") + return value + + +def _reduce(records): + """Take independent metric maxima, retaining every tied origin and incomplete cohort.""" + drivers = {(r["data"]["Configuration"]["DriverVersion"], + r["data"]["Configuration"]["DriverSha256"]) for r in records} + if len(drivers) != 1: + raise ValueError("Multiple driver identities in one BDN side/configuration.") + metrics, origins, missing_values = {}, {}, {} + for metric in METRICS: + values = [_number((r["data"].get("Summary") or {}).get(metric)) for r in records] + metrics[metric] = max((v for v in values if v is not None), default=None) + origins[metric] = [r for r, value in zip(records, values) + if value is not None and value == metrics[metric]] + missing_values[metric] = values.count(None) + + phases = defaultdict(set) + issues, unknown = Counter(), set() + samples, missing_probes = [], [] + for record in records: + data = record["data"] + observed = phases[data["CaseId"]] + if data.get("Measurement") is not None: + observed.add(data["Phase"]) + summary = data.get("Summary") or {} + samples.append(_number(summary.get("Samples"))) + missing_probes.append(_number(summary.get("Missing"))) + # Execution-failure markers can repeat the last window's snapshot. + for field in OUTCOME_FIELDS: + value = _number((data.get("Operations") or {}).get(field)) + if value is None: + unknown.add(field) + else: + issues[field] += value + return dict( + records=records, metrics=metrics, origins=origins, driver=next(iter(drivers)), + missingValues=missing_values, issues=issues, unknownCounters=sorted(unknown), + samples=samples, missingProbes=missing_probes, + missingPhases={case: sorted(PHASES - observed) for case, observed in sorted(phases.items()) + if observed != PHASES}, + statuses=Counter(r["data"]["Status"] for r in records), + health=Counter((r["data"].get("Summary") or {}).get("Health", "Invalid") for r in records), + operationEvidence=Counter((r["data"].get("Summary") or {}).get( + "OperationEvidence", "Unknown") for r in records), + dispatchEvidence=Counter((r["data"].get("Summary") or {}).get( + "DispatchEvidence", "Unknown") for r in records), + firstQueueEvidence=Counter((r["data"].get("Summary") or {}).get( + "FirstWorkerQueueEvidence", "Unknown") for r in records), + firstQueueSamples=[(r["data"].get("Summary") or {}).get("FirstWorkerQueueSamples") + for r in records]) + + +def build_comparison(root, threshold=compare_perf.DEFAULT_THRESHOLD_PCT): + """Pair only compatible settings; percentages compare maxima, never pooled sample percentiles.""" + if not math.isfinite(threshold) or threshold < 0: + raise ValueError("The comparison threshold must be finite and nonnegative.") + groups = defaultdict(lambda: defaultdict(list)) + for record in _load(Path(root)): + data = record["data"] + key = (data["Benchmark"], data["Parameters"], record["configuration"]) + groups[key][record["side"]].append(record) + + entries = [] + for (benchmark, parameters, signature), sides in sorted(groups.items()): + baseline = _reduce(sides["baseline"]) if "baseline" in sides else None + current = _reduce(sides["current"]) if "current" in sides else None + metrics = {} + for metric in METRICS: + before = baseline["metrics"][metric] if baseline else None + after = current["metrics"][metric] if current else None + metrics[metric] = dict( + baseline=before, current=after, + delta=after - before if before is not None and after is not None else None, + deltaPct=compare_perf._pct(before, after) if before not in (None, 0) else None) + delta = metrics["DispatchP99Ms"]["deltaPct"] + status = (compare_perf.classify_change(delta, threshold) if baseline and current else + "current-only" if current else "baseline-only") + entries.append(dict(benchmark=benchmark, parameters=parameters, + configuration=json.loads(signature), baseline=baseline, current=current, + metrics=metrics, status=status, key=(benchmark, parameters, signature))) + entries.sort(key=lambda e: (e["metrics"]["DispatchP99Ms"]["deltaPct"] is None, + -(e["metrics"]["DispatchP99Ms"]["deltaPct"] or 0), e["key"])) + return entries + + +def _text(value): + return str(value).replace("|", "\\|").replace("\r", " ").replace("\n", " ") + + +def _value(value, metric, signed=False): + if value is None: + return "N/A" + precision = 4 if metric.endswith("Ms") else 0 + return format(value, f"{'+' if signed else ''}.{precision}f") + + +def _cell(values, metric): + pct = values["deltaPct"] + percent = f"{pct:+.2f}%" if pct is not None else "N/A" + return (f"{_value(values['current'], metric)} " + f"({percent}; {_value(values['delta'], metric, signed=True)})") + + +def _counts(values): + return ", ".join(f"{key}={value}" for key, value in sorted(values.items())) or "none" + + +def _provenance(entry, index): + """Keep coverage and per-metric origins outside the compact metric table.""" + lines = [f"### {index}. {_text(entry['benchmark'])} {_text(entry['parameters'])}", "", + "Matching configuration: `" + _text(json.dumps( + entry["configuration"], sort_keys=True)) + "`", ""] + for side in ("baseline", "current"): + group = entry[side] + if group is None: + lines.extend([f"**{side}: unmatched configuration** (no compatible side).", ""]) + continue + lines.extend([ + f"**{side}**: {len(group['records'])} unique records; " + f"status: {_counts(group['statuses'])}; scheduler: {_counts(group['health'])}; " + f"operation evidence: {_counts(group['operationEvidence'])}.", + f"Dispatch p99 sampling: {_counts(group['dispatchEvidence'])}; " + f"first-worker p99 sampling: {_counts(group['firstQueueEvidence'])}; " + f"first-worker sample counts: `{json.dumps(group['firstQueueSamples'])}`.", + "Probe sample counts by window: `" + json.dumps(group["samples"]) + + "`; missing/censored probe counts by window: `" + + json.dumps(group["missingProbes"]) + "` (null means unavailable).", + "Missing phases: `" + _text(json.dumps(group["missingPhases"], sort_keys=True)) + "`.", + "Outcome/retention issues: " + _counts( + {key: value for key, value in group["issues"].items() if value}) + + "; unknown counters: " + (", ".join(group["unknownCounters"]) or "none") + ".", + "Missing metric values: " + _counts( + {key: value for key, value in group["missingValues"].items() if value}) + ".", + f"Driver: `{_text(group['driver'][0])}`; SHA-256 `{_text(group['driver'][1])}`.", "", + ]) + for metric in METRICS: + origins = [ + f"{r['data']['CaseId']}/{r['data']['Phase']} " + f"(rep {r['repetition'] if r['repetition'] is not None else 'unlabelled'})" + for r in group["origins"][metric]] + lines.append(f"- {METRICS[metric]} maximum " + f"{_value(group['metrics'][metric], metric)}: " + + _text("; ".join(origins) or "N/A")) + lines.extend(["", "Raw cohort provenance (all duplicate aliases retained):", ""]) + for record in group["records"]: + data = record["data"] + phase = data["Phase"] if data.get("Measurement") is not None else "execution" + sources = "; ".join(f"`{_text(path)}`" for path in record["sources"]) + lines.append(f"- `{_text(data['CaseId'])}/{phase}`; " + f"rep {record['repetition'] or 'unlabelled'}; " + f"PID {data['Configuration']['ProcessId']}; " + f"{data['Status']}; {sources}") + lines.append("") + return lines + + +def render(root, threshold=compare_perf.DEFAULT_THRESHOLD_PCT, source_label=None): + """Render observed relative status separately from absolute health/coverage qualifications.""" + entries = build_comparison(root, threshold) + lines = ["# OpenAsyncBurst async-health comparison", ""] + if source_label: + lines.extend(["Source: " + _text(source_label), ""]) + lines.extend([ + f"Observed relative dispatch-p99 status; threshold **{threshold:g}%**. " + "Increases strictly above the threshold are regressions, decreases strictly below its " + "negative are improvements, and equality is unchanged.", + "These indicators are not best-of-N-confirmed regressions and not an absolute " + "scheduler-health verdict.", + "Each metric cell is **current maximum (signed percentage delta; signed raw difference)** " + "to the baseline maximum. Startup, ExtraIteration and available repetitions are reduced " + "independently per metric. Existing cohort p99 values are not pooled or recomputed.", + "Percentages are N/A when either value is missing or the baseline is zero, including 0 to 0.", + "", + "| Status | Method | Parameters | " + " | ".join(METRICS.values()) + " |", + "| --- | --- | --- | --- | --- | --- | --- |", + ]) + identities = Counter((e["benchmark"], e["parameters"]) for e in entries) + for index, entry in enumerate(entries, 1): + parameters = entry["parameters"] + if identities[(entry["benchmark"], parameters)] > 1: + parameters += f" (configuration {index})" + values = [compare_perf.STATUS_LABELS[entry["status"]], + entry["benchmark"].rsplit(".", 1)[-1], parameters] + values.extend(_cell(entry["metrics"][metric], metric) for metric in METRICS) + lines.append("| " + " | ".join(_text(value) for value in values) + " |") + groups = [entry[side] for entry in entries for side in ("baseline", "current") if entry[side]] + health = sum((group["health"] for group in groups), Counter()) + missing_phases = sum(len(group["missingPhases"]) for group in groups) + failures = sum(group["statuses"].get("Failed", 0) for group in groups) + lines.extend([ + "", "## Coverage and provenance", "", + f"Unique records: {sum(len(group['records']) for group in groups)}; " + f"scheduler classifications: {_counts(health)}; " + f"cases with missing phases: {missing_phases}; failed records: {failures}.", + "**Coverage warning:** relative improvement or neutrality is not proof of absolute " + "scheduler health. Short/censored cohorts remain insufficient; missing phases, missing " + "outcomes and failures are not waived when both sides are unhealthy.", + "Startup includes fixture initialization; ExtraIteration excludes successful fixture " + "setup/cleanup. Raw phase JSON is unchanged. Individual maxima can come from different " + "phases, processes and repetitions; tied origins and all artifact aliases are listed below.", + "", + ]) + for side in ("baseline", "current"): + side_groups = [entry[side] for entry in entries if entry[side]] + evidence = sum((group["operationEvidence"] for group in side_groups), Counter()) + dispatch = sum((group["dispatchEvidence"] for group in side_groups), Counter()) + first_queue = sum((group["firstQueueEvidence"] for group in side_groups), Counter()) + issues = sum((group["issues"] for group in side_groups), Counter()) + unknown = sorted({field for group in side_groups for field in group["unknownCounters"]}) + lines.append(f"- **{side} coverage:** operation evidence: {_counts(evidence)}; " + f"dispatch p99 sampling: {_counts(dispatch)}; " + f"first-worker p99 sampling: {_counts(first_queue)}; " + f"outcome/retention issues: {_counts(issues)}; " + f"unknown counters: {', '.join(unknown) or 'none'}.") + lines.extend(["", "
", + "Per-row coverage and raw cohort provenance", ""]) + failed = False + for index, entry in enumerate(entries, 1): + lines.extend(_provenance(entry, index)) + failed |= any(group and group["statuses"].get("Failed", 0) + for group in (entry["baseline"], entry["current"])) + lines.extend(["
", ""]) + return "\n".join(lines) + "\n", failed + + +def main(argv=None): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--results-dir", type=Path, required=True) + parser.add_argument("--threshold", type=float, default=compare_perf.DEFAULT_THRESHOLD_PCT) + parser.add_argument("--out-md", type=Path, + help="Alternate output path; useful for preserving historical reports.") + parser.add_argument("--source-label", help="Identify historical evidence in a regenerated report.") + args = parser.parse_args(argv) + markdown, failed = render(args.results_dir, args.threshold, args.source_label) + output = args.out_md or args.results_dir / "benchmark-async-health.md" + output.parent.mkdir(parents=True, exist_ok=True) + output.write_text(markdown, encoding="utf-8") + return 1 if failed else 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/eng/pipelines/perf/scripts/compare_perf.py b/eng/pipelines/perf/scripts/compare_perf.py index 609af7e8bd..feab5251e1 100644 --- a/eng/pipelines/perf/scripts/compare_perf.py +++ b/eng/pipelines/perf/scripts/compare_perf.py @@ -15,12 +15,60 @@ import argparse import glob +import hashlib import json import os import sys +from pathlib import Path NS_PER_MS = 1_000_000.0 +DEFAULT_THRESHOLD_PCT = 10.0 +STATUS_LABELS = { + "regression": "\U0001f534 regression", + "improvement": "\U0001f7e2 improvement", + "unchanged": "\u26aa unchanged", + "current-only": "\U0001f195 new", + "baseline-only": "\u2796 removed", + "unknown": "\u2754", +} + + +def classify_change(delta, threshold_pct): + """Classify an unrounded relative change; equality with either boundary is neutral.""" + if delta is None: + return "unknown" + if delta > threshold_pct: + return "regression" + if delta < -threshold_pct: + return "improvement" + return "unchanged" + + +def _execution_conditions(path, directory): + """Keep new runner outputs from silently pairing across different execution conditions.""" + root = Path(directory).resolve() + parent = Path(path).resolve().parent + while parent == root or root in parent.parents: + manifest = parent / "process.json" + if manifest.is_file(): + data = json.loads(manifest.read_text(encoding="utf-8-sig")) + conditions = data.get("ExecutionConditions") + if not isinstance(conditions, dict): + raise ValueError(f"Missing execution conditions: {manifest}") + requested = conditions.get("AdditionalRttMilliseconds") + applied = data.get("AppliedAdditionalRttMilliseconds") + if requested is None or applied != requested: + raise ValueError(f"Mismatched requested/applied RTT: {manifest}") + if data.get("EffectiveProcessIsolation") != conditions.get("ProcessIsolation"): + raise ValueError(f"Mismatched process isolation: {manifest}") + return conditions + if parent == root: + break + parent = parent.parent + if "processes" in Path(path).resolve().relative_to(root).parts: + raise ValueError(f"Missing process manifest for isolated benchmark report: {path}") + return None def _load_benchmarks(directory): @@ -32,6 +80,9 @@ def _load_benchmarks(directory): records = {} pattern = os.path.join(directory, "**", "*-report-full.json") for path in sorted(glob.glob(pattern, recursive=True)): + conditions = _execution_conditions(path, directory) + signature = (hashlib.sha256(json.dumps(conditions, sort_keys=True).encode()).hexdigest()[:16] + if conditions is not None else None) try: with open(path, "r", encoding="utf-8-sig") as handle: data = json.load(handle) @@ -51,12 +102,16 @@ def _load_benchmarks(directory): alloc = memory.get("BytesAllocatedPerOperation") key = f"{btype}.{method}({params})" + if signature is not None: + key += f"[conditions={signature}]" records[key] = { "benchmarkName": btype, "methodName": method, "parameterSignature": params, "meanNs": float(mean_ns), "allocatedBytes": float(alloc) if alloc is not None else None, + "executionConditions": conditions, + "conditionSignature": signature, } return records @@ -86,6 +141,8 @@ def build_comparison(baseline_dir, current_dir, threshold_pct): "benchmarkName": ref["benchmarkName"], "methodName": ref["methodName"], "parameterSignature": ref["parameterSignature"], + "executionConditions": ref["executionConditions"], + "conditionSignature": ref["conditionSignature"], "baselineMeanMs": (b["meanNs"] / NS_PER_MS) if b else None, "currentMeanMs": (c["meanNs"] / NS_PER_MS) if c else None, "baselineAllocBytes": b["allocatedBytes"] if b else None, @@ -103,15 +160,7 @@ def build_comparison(baseline_dir, current_dir, threshold_pct): entry["allocDeltaPct"] = _pct(b["allocatedBytes"], c["allocatedBytes"]) else: entry["allocDeltaPct"] = None - delta = entry["meanDeltaPct"] - if delta is None: - entry["status"] = "unknown" - elif delta > threshold_pct: - entry["status"] = "regression" - elif delta < -threshold_pct: - entry["status"] = "improvement" - else: - entry["status"] = "unchanged" + entry["status"] = classify_change(entry["meanDeltaPct"], threshold_pct) elif c and not b: entry["meanDeltaPct"] = None entry["meanRatio"] = None @@ -161,6 +210,13 @@ def _fmt_alloc(entry): return "-" +def _fmt_parameters(entry): + parameters = entry["parameterSignature"] or "-" + if entry["status"] in ("current-only", "baseline-only") and entry.get("conditionSignature"): + parameters += f" [conditions={entry['conditionSignature']}]" + return parameters + + def render_markdown(entries, baseline_version, threshold_pct): regressions = [e for e in entries if e["status"] == "regression"] improvements = [e for e in entries if e["status"] == "improvement"] @@ -169,33 +225,25 @@ def render_markdown(entries, baseline_version, threshold_pct): lines.append("# SqlClient Performance Comparison") lines.append("") lines.append(f"Baseline: **{baseline_version}**  |  " - f"Regression threshold: **{threshold_pct:.0f}%**") + f"Regression threshold: **{threshold_pct:g}%**") lines.append("") lines.append(f"- Benchmarks compared: **{len(entries)}**") - lines.append(f"- Regressions (slower > {threshold_pct:.0f}%): **{len(regressions)}**") - lines.append(f"- Improvements (faster > {threshold_pct:.0f}%): **{len(improvements)}**") + lines.append(f"- Regressions (slower > {threshold_pct:g}%): **{len(regressions)}**") + lines.append(f"- Improvements (faster > {threshold_pct:g}%): **{len(improvements)}**") lines.append("") lines.append("| Status | Benchmark | Method | Params | Baseline (ms) | " "Current (ms) | Mean Δ | Alloc Δ |") lines.append("| ------ | --------- | ------ | ------ | ------------- | " "------------ | ------ | ------- |") - icon = { - "regression": "🔴 regression", - "improvement": "🟢 improvement", - "unchanged": "⚪ unchanged", - "current-only": "🆕 new", - "baseline-only": "➖ removed", - "unknown": "❔", - } for e in entries: lines.append( "| {status} | {name} | {method} | {params} | {base} | {cur} | " "{delta} | {alloc} |".format( - status=icon.get(e["status"], e["status"]), + status=STATUS_LABELS.get(e["status"], e["status"]), name=e["benchmarkName"], method=e["methodName"], - params=e["parameterSignature"] or "-", + params=_fmt_parameters(e), base=_fmt_ms(e["baselineMeanMs"]), cur=_fmt_ms(e["currentMeanMs"]), delta=_fmt_pct(e["meanDeltaPct"]), @@ -213,7 +261,7 @@ def main(argv=None): parser.add_argument("--out-md", required=True) parser.add_argument("--out-json", required=True) parser.add_argument("--baseline-version", default="baseline") - parser.add_argument("--threshold", type=float, default=10.0, + parser.add_argument("--threshold", type=float, default=DEFAULT_THRESHOLD_PCT, help="Percent slowdown that counts as a regression.") parser.add_argument("--fail-on-regression", action="store_true", help="Exit non-zero if any regression is detected.") diff --git a/eng/pipelines/perf/scripts/interleave_perf.py b/eng/pipelines/perf/scripts/interleave_perf.py index 16999939b5..c53080e9a5 100644 --- a/eng/pipelines/perf/scripts/interleave_perf.py +++ b/eng/pipelines/perf/scripts/interleave_perf.py @@ -29,10 +29,13 @@ import argparse import glob import json +import math import os import shutil +import signal import subprocess import sys +import time SCRIPT_DIR = os.path.dirname(os.path.abspath(__file__)) @@ -109,7 +112,30 @@ def apply_affinity(proc, cpus): # -------------------------------------------------------------------------------------------------- # Running one unit and collecting its artifacts. # -------------------------------------------------------------------------------------------------- -def run_unit_process(exe_dir, assembly, unit, cwd, cpus, log_path, env_overrides=None): +def stop_process_tree(proc): + """Stop only this benchmark's process tree, leaving the collector alive.""" + if proc.poll() is not None: + return + if os.name == "nt": + stopped = subprocess.run( + ["taskkill", "/PID", str(proc.pid), "/T", "/F"], + capture_output=True, text=True, timeout=30) + if stopped.returncode and proc.poll() is None: + stopped.check_returncode() + else: + try: + os.killpg(proc.pid, signal.SIGTERM) + except ProcessLookupError: + pass + try: + proc.wait(timeout=10) + except subprocess.TimeoutExpired: + os.killpg(proc.pid, signal.SIGKILL) + proc.wait(timeout=30) + + +def run_unit_process(exe_dir, assembly, unit, cwd, cpus, log_path, env_overrides=None, + timeout=None): """Run one benchmark *unit* from the build at *exe_dir* in *cwd*. Returns the subprocess return code. Kept as a small seam so tests can substitute @@ -128,9 +154,18 @@ def run_unit_process(exe_dir, assembly, unit, cwd, cpus, log_path, env_overrides with open(log_path, "w", encoding="utf-8") as log: proc = subprocess.Popen(cmd, cwd=cwd, env=env, stdout=log, - stderr=subprocess.STDOUT) + stderr=subprocess.STDOUT, start_new_session=os.name != "nt") apply_affinity(proc, cpus) - return proc.wait() + try: + return proc.wait(timeout=timeout) + except subprocess.TimeoutExpired: + log.write("\nERROR: benchmark exceeded the remaining run budget.\n") + log.flush() + stop_process_tree(proc) + return 124 + except BaseException: + stop_process_tree(proc) + raise def _report_files(root): @@ -146,8 +181,19 @@ def collect_results(cwd, dest): os.makedirs(dest, exist_ok=True) types = set() artifacts = os.path.join(cwd, "BenchmarkDotNet.Artifacts") + for name in ("process.json", "network.json"): + for path in glob.glob(os.path.join(artifacts, "processes", "**", name), recursive=True): + target = os.path.join(dest, os.path.relpath(path, artifacts)) + os.makedirs(os.path.dirname(target), exist_ok=True) + shutil.copy2(path, target) + for health in glob.glob(os.path.join(artifacts, "**", "async-health"), recursive=True): + shutil.copytree(health, os.path.join(dest, "async-health"), dirs_exist_ok=True) for path in _report_files(artifacts): - shutil.copy2(path, os.path.join(dest, os.path.basename(path))) + relative = os.path.relpath(path, artifacts) + target = os.path.join(dest, relative if relative.startswith("processes" + os.sep) + else os.path.basename(path)) + os.makedirs(os.path.dirname(target), exist_ok=True) + shutil.copy2(path, target) try: with open(path, "r", encoding="utf-8-sig") as fh: for bench in json.load(fh).get("Benchmarks", []): @@ -157,7 +203,11 @@ def collect_results(cwd, dest): pass # Also keep the human-readable GitHub markdown reports alongside, best effort. for md in glob.glob(os.path.join(artifacts, "**", "*-report-github.md"), recursive=True): - shutil.copy2(md, os.path.join(dest, os.path.basename(md))) + relative = os.path.relpath(md, artifacts) + target = os.path.join(dest, relative if relative.startswith("processes" + os.sep) + else os.path.basename(md)) + os.makedirs(os.path.dirname(target), exist_ok=True) + shutil.copy2(md, target) return types @@ -168,7 +218,7 @@ class Runner: """Holds the invariant run parameters and performs interleaved unit passes.""" def __init__(self, baseline_dir, current_dir, assembly, work_dir, cpus, - baseline_runner_config=None, current_runner_config=None): + baseline_runner_config=None, current_runner_config=None, max_run_seconds=None): self.baseline_dir = baseline_dir self.current_dir = current_dir self.assembly = assembly @@ -179,14 +229,32 @@ def __init__(self, baseline_dir, current_dir, assembly, work_dir, cpus, # same build). None means "no override" -> both variants use the ambient RUNNER_CONFIG. self.baseline_runner_config = baseline_runner_config self.current_runner_config = current_runner_config + self.deadline = (time.monotonic() + max_run_seconds + if max_run_seconds is not None else None) + self.repeat_all_units = set() + + def remaining_seconds(self): + if self.deadline is None: + return None + remaining = self.deadline - time.monotonic() + if remaining <= 0: + raise TimeoutError("Benchmark run budget exhausted; partial artifacts are retained.") + return remaining def list_units(self): cmd = ["dotnet", os.path.join(self.current_dir, self.assembly)] env = dict(os.environ) - env["PERF_LIST_BENCHMARKS"] = "1" + env["PERF_LIST_BENCHMARKS"] = "json" env.pop("PERF_BENCHMARK", None) - out = subprocess.check_output(cmd, cwd=self.work_dir, env=env, text=True) - return [line.strip() for line in out.splitlines() if line.strip()] + out = subprocess.check_output(cmd, cwd=self.work_dir, env=env, text=True, + timeout=self.remaining_seconds()) + settings = json.loads(out) + if not isinstance(settings, list) or any( + not isinstance(item.get("Name"), str) or type(item.get("RepeatAll")) is not bool + for item in settings): + raise ValueError("Invalid benchmark execution metadata.") + self.repeat_all_units = {item["Name"] for item in settings if item["RepeatAll"]} + return [item["Name"] for item in settings] def _run_one(self, variant, exe_dir, unit, rep, agg_dir): cwd = os.path.join(self.work_dir, f"rep{rep}", variant, unit) @@ -197,12 +265,27 @@ def _run_one(self, variant, exe_dir, unit, rep, agg_dir): runner_config = (self.baseline_runner_config if variant == "baseline" else self.current_runner_config) env_overrides = {"RUNNER_CONFIG": runner_config} if runner_config else None - rc = run_unit_process(exe_dir, self.assembly, unit, cwd, self.cpus, log_path, - env_overrides=env_overrides) + started = time.monotonic() + rc = None + try: + rc = run_unit_process(exe_dir, self.assembly, unit, cwd, self.cpus, log_path, + env_overrides=env_overrides, timeout=self.remaining_seconds()) + finally: + elapsed = time.monotonic() - started + timing_path = os.path.join(cwd, "run-timing.json") + with open(timing_path + ".tmp", "w", encoding="utf-8") as timing: + json.dump(dict(unit=unit, variant=variant, repetition=rep, + elapsed_seconds=elapsed, exit_code=rc), timing) + os.replace(timing_path + ".tmp", timing_path) + # Preserve partial evidence even when launch, execution or the deadline fails. + types = collect_results(cwd, agg_dir) + shutil.copy2(timing_path, os.path.join(agg_dir, f"{unit}-rep{rep}.timing.json")) + if os.path.isfile(log_path): + shutil.copy2(log_path, os.path.join(agg_dir, f"{unit}-rep{rep}.log")) + print(f"[rep {rep}] {unit} ({variant}): {elapsed:.1f}s, exit {rc}", flush=True) if rc != 0: _tail(log_path) raise RuntimeError(f"benchmark unit '{unit}' ({variant}, rep {rep}) failed (exit {rc}).") - types = collect_results(cwd, agg_dir) if not types: _tail(log_path) raise RuntimeError( @@ -258,7 +341,8 @@ def orchestrate(runner, units, results_dir, threshold, reps): entries, reg_keys_1 = _regression_keys(baseline_agg, current_agg, threshold) # Which units contain a rep-1 regression? Those are the best-of-N candidates. - candidate_units = [] + candidate_units = [unit for unit in units + if unit in getattr(runner, "repeat_all_units", set())] for e in entries: if e["status"] == "regression": unit = type_to_unit.get(e["benchmarkName"]) @@ -307,13 +391,8 @@ def orchestrate(runner, units, results_dir, threshold, reps): # Rendering. # -------------------------------------------------------------------------------------------------- _ICON = { - "regression": "🔴 regression", - "regression-unconfirmed": "🟠 regression (unconfirmed)", - "improvement": "🟢 improvement", - "unchanged": "⚪ unchanged", - "current-only": "🆕 new", - "baseline-only": "➖ removed", - "unknown": "❔", + **compare_perf.STATUS_LABELS, + "regression-unconfirmed": "\U0001f7e0 regression (unconfirmed)", } @@ -323,13 +402,13 @@ def render_markdown(entries, confirmed, unconfirmed, baseline_version, threshold lines.append("# SqlClient Performance Comparison (interleaved, best-of-N)") lines.append("") lines.append(f"Baseline: **{baseline_version}**  |  " - f"Regression threshold: **{threshold:.0f}%**  |  " + f"Regression threshold: **{threshold:g}%**  |  " f"Confirmation runs: **N = {reps}**") lines.append("") lines.append(f"- Benchmarks compared: **{len(entries)}**") lines.append(f"- **Confirmed** regressions (majority of {reps}): **{len(confirmed)}**") lines.append(f"- Unconfirmed (flagged once, not reproduced): **{len(unconfirmed)}**") - lines.append(f"- Improvements (faster > {threshold:.0f}%): **{len(improvements)}**") + lines.append(f"- Improvements (faster > {threshold:g}%): **{len(improvements)}**") lines.append("") lines.append("| Status | Benchmark | Method | Params | Baseline (ms) | " "Current (ms) | Mean Δ | Alloc Δ | Confirm |") @@ -346,7 +425,7 @@ def render_markdown(entries, confirmed, unconfirmed, baseline_version, threshold status=_ICON.get(e["status"], e["status"]), name=e["benchmarkName"], method=e["methodName"], - params=e["parameterSignature"] or "-", + params=compare_perf._fmt_parameters(e), base=compare_perf._fmt_ms(e["baselineMeanMs"]), cur=compare_perf._fmt_ms(e["currentMeanMs"]), delta=compare_perf._fmt_pct(e["meanDeltaPct"]), @@ -371,8 +450,10 @@ def main(argv=None): parser.add_argument("--assembly", default="PerformanceTests.dll") parser.add_argument("--results-dir", required=True) parser.add_argument("--work-dir", default=None, - help="Scratch directory for per-unit run dirs (default: /../perf-interleave-work).") - parser.add_argument("--threshold", type=float, default=10.0) + help="Per-unit raw artifacts directory (default: /runs).") + parser.add_argument("--max-run-seconds", type=float, default=None, + help="Remaining execution budget, excluding pipeline collection grace.") + parser.add_argument("--threshold", type=float, default=compare_perf.DEFAULT_THRESHOLD_PCT) parser.add_argument("--reps", type=int, default=3, help="Total interleaved passes for a flagged unit (best-of-N). 1 disables confirmation.") parser.add_argument("--baseline-version", default="baseline") @@ -392,10 +473,13 @@ def main(argv=None): if args.reps < 1: parser.error("--reps must be >= 1") + if args.max_run_seconds is not None and ( + not math.isfinite(args.max_run_seconds) or args.max_run_seconds <= 0): + parser.error("--max-run-seconds must be a positive finite number") results_dir = os.path.abspath(args.results_dir) work_dir = os.path.abspath(args.work_dir) if args.work_dir \ - else os.path.join(os.path.dirname(results_dir), "perf-interleave-work") + else os.path.join(results_dir, "runs") os.makedirs(work_dir, exist_ok=True) os.makedirs(results_dir, exist_ok=True) @@ -409,7 +493,8 @@ def main(argv=None): os.path.abspath(args.current_exe_dir), args.assembly, work_dir, cpus, baseline_runner_config=args.baseline_runner_config, - current_runner_config=args.current_runner_config) + current_runner_config=args.current_runner_config, + max_run_seconds=args.max_run_seconds) units = runner.list_units() if not units: diff --git a/eng/pipelines/perf/scripts/netem.py b/eng/pipelines/perf/scripts/netem.py new file mode 100644 index 0000000000..151149b002 --- /dev/null +++ b/eng/pipelines/perf/scripts/netem.py @@ -0,0 +1,245 @@ +#!/usr/bin/env python3 +# Licensed to the .NET Foundation under one or more agreements. +# The .NET Foundation licenses this file to you under the MIT license. + +"""Own a latency-only SQL TCP loopback queue for one benchmark process.""" + +import argparse +from contextlib import contextmanager +import ipaddress +import json +import math +import os +from pathlib import Path +import re +import signal +import shutil +import socket +import subprocess +import sys +import tempfile +import time +import uuid + + +def nonnegative(value): + if not re.fullmatch(r"[0-9]+", str(value)): + raise argparse.ArgumentTypeError("RTT must be nonnegative integer milliseconds") + return int(value) + + +def write_json(path, data): + path.parent.mkdir(parents=True, exist_ok=True) + temporary = path.with_name(path.name + ".tmp") + temporary.write_text(json.dumps(data, indent=2) + "\n", encoding="utf-8") + temporary.replace(path) + + +@contextmanager +def locked(): + """Serialize inspection/mutation; an installed owned root reserves the run's network scope.""" + if sys.platform != "linux": + raise RuntimeError("SQL-only netem requires Linux") + import fcntl + lock = Path(tempfile.gettempdir()) / "sqlclient-perf-netem-lo.lock" + descriptor = os.open(lock, os.O_CREAT | os.O_RDWR | os.O_NOFOLLOW, 0o600) + try: + fcntl.flock(descriptor, fcntl.LOCK_EX | fcntl.LOCK_NB) + yield + finally: + os.close(descriptor) + + +class NetworkDelay: + """Never replace a foreign queue; persist ownership before any mutation.""" + + def __init__(self, state_path, output_path): + self.state_path = Path(state_path) + self.output_path = Path(output_path) + self.attempted_setup = False + + def save(self, data): + write_json(self.state_path, data) + write_json(self.output_path, data) + + def command(self, *args, privileged=False): + prefix = ["sudo", "-n"] if privileged and os.geteuid() != 0 else [] + executable = shutil.which(args[0]) + if executable is None and args[0] in ("tc", "ip"): + candidate = Path("/usr/sbin") / args[0] + executable = str(candidate) if candidate.is_file() else None + if executable is None: + raise RuntimeError(f"Required networking tool is unavailable: {args[0]}") + return subprocess.run([*prefix, executable, *args[1:]], check=True, + text=True, capture_output=True, + timeout=10).stdout + + def queues(self): + return json.loads(self.command("tc", "-s", "-j", "qdisc", "show", "dev", "lo")) + + def setup(self, rtt, server, port): + if self.state_path.exists(): + previous = json.loads(self.state_path.read_text(encoding="utf-8")) + if previous.get("RootHandle") and not previous.get("CleanupSucceeded"): + raise RuntimeError("Unrecovered netem state exists; stop that owned run first") + data = dict( + SchemaVersion=1, AdditionalRttMilliseconds=rtt, + DelayMicrosecondsPerDirection=rtt * 500, Interface="lo", TcpPort=port, + QueueLimitPackets=10000, ServerAddresses=[], Applied=False, + Status="Disabled" if rtt == 0 else "Preparing", + RootHandle=None, NetemHandle=None, CleanupSucceeded=False) + self.attempted_setup = True + self.save(data) + if rtt == 0: + return + + addresses = sorted({str(ipaddress.ip_address(info[4][0])) + for info in socket.getaddrinfo(server, port, + type=socket.SOCK_STREAM)}) + if not addresses: + raise RuntimeError("SQL endpoint did not resolve") + for address in addresses: + family = "-4" if ipaddress.ip_address(address).version == 4 else "-6" + routes = json.loads(self.command("ip", family, "-j", "route", "get", address)) + if not routes or any(route.get("dev") != "lo" for route in routes): + raise RuntimeError("Netem supports only local loopback SQL routes") + + before = self.queues() + if any(queue.get("root") and queue["kind"] != "noqueue" for queue in before): + raise RuntimeError("Refusing to overwrite an existing root queue on lo") + + number = uuid.uuid4().int % 0x7ffe + 1 + data.update(ServerAddresses=addresses, RootHandle=f"{number:x}:", + NetemHandle=f"{number + 0x8000:x}:", Before=before, Status="Installing") + self.save(data) + self.command("tc", "qdisc", "add", "dev", "lo", "root", "handle", data["RootHandle"], + "prio", "bands", "3", "priomap", *(["0"] * 16), privileged=True) + self.command("tc", "qdisc", "add", "dev", "lo", "parent", f"{number:x}:2", + "handle", data["NetemHandle"], "netem", "delay", f"{rtt * 500}us", + "limit", str(data["QueueLimitPackets"]), privileged=True) + for index, address in enumerate(addresses): + protocol = "ip" if ipaddress.ip_address(address).version == 4 else "ipv6" + for offset, direction in enumerate(("dst", "src")): + self.command("tc", "filter", "add", "dev", "lo", "parent", data["RootHandle"], + "protocol", protocol, "pref", str(index * 2 + offset + 1), + "flower", "ip_proto", "tcp", f"{direction}_ip", address, + f"{direction}_port", str(port), "classid", f"{number:x}:2", + privileged=True) + configured = self.queues() + netem = [queue for queue in configured if queue.get("handle") == data["NetemHandle"] + and queue["kind"] == "netem"] + if len(netem) != 1: + raise RuntimeError("Netem queue was not installed") + options = netem[0].get("options", {}) + delay = options.get("delay", {}) + if (not isinstance(delay, dict) or + not math.isclose(delay.get("delay", -1), rtt / 2000, abs_tol=0.000001) or + delay.get("jitter", 0) != 0 or delay.get("correlation", 0) != 0): + raise RuntimeError("Applied netem delay differs from the requested latency-only profile") + data.update(Status="Applied", Applied=True, Configured=configured) + self.save(data) + print(f"Netem applied: {rtt}ms added RTT for SQL TCP/{port} on lo.", flush=True) + + def cleanup(self, require_traffic=True): + if not self.state_path.exists(): + return + data = json.loads(self.state_path.read_text(encoding="utf-8")) + handle = data["RootHandle"] + if handle is None or data.get("CleanupSucceeded"): + return + if not re.fullmatch(r"[0-9a-f]{1,4}:", handle): + raise ValueError("Invalid owned netem queue handle") + + queues = self.queues() + owned = any(queue.get("root") and queue.get("handle") == handle + and queue["kind"] == "prio" for queue in queues) + if not owned: + if data["Applied"]: + raise RuntimeError("The owned netem root disappeared or changed") + return + + problems = [] + try: + deadline = time.monotonic() + min(10, max(1, data["AdditionalRttMilliseconds"] / 250)) + while any(queue.get("qlen", 0) for queue in queues + if queue.get("handle") in (handle, data["NetemHandle"])): + if time.monotonic() >= deadline: + problems.append("Netem did not drain before cleanup") + break + time.sleep(0.01) + queues = self.queues() + data["After"] = queues + data["Filters"] = json.loads(self.command( + "tc", "-s", "-j", "filter", "show", "dev", "lo", "parent", handle)) + if data["Applied"]: + netem = [queue for queue in queues if queue.get("handle") == data["NetemHandle"] + and queue["kind"] == "netem"] + configured = next(queue for queue in data["Configured"] + if queue.get("handle") == data["NetemHandle"]) + if len(netem) != 1 or netem[0].get("options") != configured.get("options"): + problems.append("Netem settings changed during execution") + if any(queue.get("drops", 0) for queue in queues + if queue.get("handle") in (handle, data["NetemHandle"])): + problems.append("Netem dropped packets; this was not a latency-only run") + if require_traffic and (not netem or not netem[0].get("packets", 0)): + problems.append("No SQL packets traversed netem") + finally: + self.command("tc", "qdisc", "del", "dev", "lo", "root", "handle", handle, + privileged=True) + data["AfterCleanup"] = self.queues() + data["CleanupSucceeded"] = not any( + queue.get("handle") == handle for queue in data["AfterCleanup"]) + data["Status"] = "Cleaned" if data["CleanupSucceeded"] and not problems else "Failed" + self.save(data) + if not data["CleanupSucceeded"]: + problems.append("The owned netem queue was not removed") + if problems: + raise RuntimeError("; ".join(problems)) + print("Removed the owned netem queue.", flush=True) + + +def interrupted(signum, frame): + raise InterruptedError("Netem operation interrupted") + + +def main(argv=None): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("action", choices=("start", "stop")) + parser.add_argument("--state", type=Path, required=True) + parser.add_argument("--output", type=Path, required=True) + parser.add_argument("--rtt-ms", type=nonnegative, default=0) + parser.add_argument("--server", default="localhost") + parser.add_argument("--port", type=int, default=1433) + args = parser.parse_args(argv) + if not 1 <= args.port <= 65535: + parser.error("port must be between 1 and 65535") + controller = NetworkDelay(args.state, args.output) + signal.signal(signal.SIGTERM, interrupted) + signal.signal(signal.SIGINT, interrupted) + entered = False + try: + with locked(): + entered = True + try: + if args.action == "start": + controller.setup(args.rtt_ms, args.server, args.port) + else: + controller.cleanup() + except (OSError, ValueError, RuntimeError, subprocess.SubprocessError): + if args.action == "start" and controller.attempted_setup: + controller.cleanup(require_traffic=False) + raise + except (OSError, ValueError, RuntimeError, subprocess.SubprocessError) as error: + detail = getattr(error, "stderr", None) or str(error) + print(f"ERROR: netem {args.action}: {detail}", file=sys.stderr) + if entered and args.state.exists() and ( + args.action == "stop" or controller.attempted_setup): + data = json.loads(args.state.read_text(encoding="utf-8")) + data.update(Status="Failed", Error=str(detail)) + controller.save(data) + return 1 + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/eng/pipelines/perf/scripts/run-perf-tests.ps1 b/eng/pipelines/perf/scripts/run-perf-tests.ps1 index 57aca288c6..d5fc9b88c1 100644 --- a/eng/pipelines/perf/scripts/run-perf-tests.ps1 +++ b/eng/pipelines/perf/scripts/run-perf-tests.ps1 @@ -24,6 +24,8 @@ param( [string]$Configuration = "Release", [string]$Framework = "net10.0", [string]$ResultsSubdir = "perf-results", + [ValidateRange(11, 1440)] + [int]$TestTimeoutMinutes = 180, [string]$BaselineVersion = "", # Alternative to -BaselineVersion: benchmark against Microsoft.Data.SqlClient built from ANOTHER # git ref of this repository (e.g. 'main') instead of a released NuGet package. Used by the PR @@ -46,6 +48,8 @@ param( # Best-of-N: total interleaved passes for a flagged unit before a regression is confirmed. [ValidateRange(1, [int]::MaxValue)] [int]$ConfirmationRuns = 3, + [ValidateSet("default", "openasync-extended-100ms")] + [string]$RunnerProfile = "default", # Optional SqlClient behaviour flags (true/false, or empty to leave the checked-in # runnerconfig.default.jsonc default untouched). Written into the runner config the benchmarks run # against and, via the pipeline's Kusto translation, recorded in PerfRun.Config. @@ -69,6 +73,19 @@ param( $ErrorActionPreference = "Stop" Set-StrictMode -Version Latest +$RunClock = [System.Diagnostics.Stopwatch]::StartNew() +if ($RunnerProfile -ne "default") { + throw "The openasync-extended-100ms profile requires the Linux runner." +} + +function Get-RemainingPerfSeconds { + # Leave time inside the ADO task deadline for the remote poll and artifact collection. + $remaining = [int][Math]::Floor(($TestTimeoutMinutes - 10) * 60 - $RunClock.Elapsed.TotalSeconds) + if ($remaining -le 0) { + throw "Benchmark execution budget exhausted; retaining partial results." + } + return $remaining +} #################################################################################################### # Native-command error handling @@ -135,6 +152,7 @@ Write-Host " Perf project : $PerfProject" Write-Host " Configuration : $Configuration" Write-Host " Framework : $Framework" Write-Host " Results dir : $ResultsDir" +Write-Host " Task budget : $TestTimeoutMinutes minutes (10 reserved for collection)" Write-Host " Run mode : $RunMode (confirmation runs: $ConfirmationRuns)" Write-Host " Baseline ver : $(if ($BaselineVersion) { $BaselineVersion } else { '' })" Write-Host " Baseline ref : $(if ($BaselineSourceRef) { $BaselineSourceRef } else { '' })" @@ -409,6 +427,10 @@ $RunnerConfig = Join-Path $RepoRoot "perf-runnerconfig.json" $env:RUNNER_CONFIG = $RunnerConfig $ResultsRunnerConfig = Join-Path $ResultsDir "runnerconfig.json" $SourceRunnerConfig = Join-Path $PerfDir "runnerconfig.jsonc" +if ($RunnerProfile -ne "default") { + $SourceRunnerConfig = Join-Path $PerfDir "profiles\$RunnerProfile.jsonc" +} +Write-Host "Runner profile: $RunnerProfile" # The perf app also loads datatypes.json via the DATATYPES_CONFIG env var, falling back to # "datatypes.json" in the working directory. Each pass runs from an otherwise-empty @@ -439,6 +461,17 @@ function Write-RunnerConfig { $rawConfig = ($rawConfig -split "`n" | ForEach-Object { $_ -replace '(?m)^\s*//.*$', '' }) -join "`n" $cfg = ConvertFrom-Json $rawConfig + $defaultsText = Get-Content (Join-Path $PerfDir "runnerconfig.default.jsonc") -Raw + $defaultsText = ($defaultsText -split "`n" | + ForEach-Object { $_ -replace '(?m)^\s*//.*$', '' }) -join "`n" + $defaults = ConvertFrom-Json $defaultsText + $cfg | Add-Member -Force -NotePropertyName EnableAsyncHealth -NotePropertyValue $true + if ($cfg.Benchmarks.PSObject.Properties.Name -notcontains "OpenAsyncBurstRunnerConfig") { + $cfg.Benchmarks | Add-Member -NotePropertyName OpenAsyncBurstRunnerConfig ` + -NotePropertyValue $defaults.Benchmarks.OpenAsyncBurstRunnerConfig + } + $cfg.Benchmarks.OpenAsyncBurstRunnerConfig.Enabled = $true + $cfg.ConnectionString = $BenchmarkConnectionString # Apply the optional SqlClient behaviour overrides supplied by the pipeline. An empty value # leaves the checked-in default untouched; otherwise the flag is forced to the requested boolean @@ -502,7 +535,7 @@ $BaselineSrcDir = Join-Path (Split-Path -Parent $RepoRoot) "sqlclient-perf-basel # Prefers the copied checkout's own 'origin' (no extra network round-trip, and it resolves the ref # exactly as the pipeline's repository does); falls back to a shallow clone of -BaselineRepoUrl when # the source tree arrived without .git, or when origin is unreachable/needs credentials. -# Returns a hashtable with the baseline perf project path and the '@' label. +# Returns the '@' label after validating source dependencies. # --- Non-interactive, time-bounded git ------------------------------------------------------------ # The checkout is copied to the VM WITHOUT credentials (ADO's checkout task defaults to # persistCredentials:false), so a fetch from an authenticated origin has no way to succeed. Every @@ -623,9 +656,10 @@ function Initialize-BaselineSource { $ErrorActionPreference = $previousPreference } - $baselineProject = Join-Path $BaselineSrcDir "src\Microsoft.Data.SqlClient\tests\PerformanceTests\Microsoft.Data.SqlClient.PerformanceTests.csproj" - if (-not (Test-Path $baselineProject)) { - throw "Baseline source tree at $BaselineSrcDir has no performance test project ($baselineProject)." + $driverProject = Join-Path $BaselineSrcDir "src\Microsoft.Data.SqlClient\src\Microsoft.Data.SqlClient.csproj" + $commonProject = Join-Path $BaselineSrcDir "src\Microsoft.Data.SqlClient\tests\Common\Microsoft.Data.SqlClient.TestCommon.csproj" + if (-not (Test-Path $driverProject) -or -not (Test-Path $commonProject)) { + throw "Baseline source tree requires the unified driver and matching TestCommon projects." } Restore-LocalTools $BaselineSrcDir @@ -634,7 +668,7 @@ function Initialize-BaselineSource { $label = "$Ref@$($sha.Trim())" Write-Host "Baseline source ready: $BaselineSrcDir ($label)" - return @{ Project = $baselineProject; Label = $label } + return @{ Label = $label } } # Convert a CPU range like "16-31" (or a comma list "16,17,18") into an affinity bitmask. @@ -717,8 +751,7 @@ function Build-CurrentDefaultOutput { # Runs one benchmark pass (build + run pinned to PERF_CLIENT_CPUS) from $Project and collects its # artifacts into results\