Skip to content

Refactor benchmarks#1203

Open
fdcastel wants to merge 37 commits into
FirebirdSQL:masterfrom
fdcastel:refactor-benchmarks
Open

Refactor benchmarks#1203
fdcastel wants to merge 37 commits into
FirebirdSQL:masterfrom
fdcastel:refactor-benchmarks

Conversation

@fdcastel
Copy link
Copy Markdown
Member

Based on #1200 for now. I will rebase it with latest master in the future.

  • Rename project Perf to FirebirdSql.Data.FirebirdClient.Benchmarks.
  • Update project to use .NET8.
  • Upgrade BenchmarkDotNet to version 0.14.0.
  • Update baseline nuget package to v10.3.1.
  • Add /BenchmarkDotNet.Artifacts folder to .gitignore.
  • Pass command-line arguments to BenchmarkDotNet engine.
  • Apply SQL Formatting. Use raw strings.
  • Add script run-benchmark.ps1.

Comment thread src/FirebirdSql.Data.FirebirdClient.Benchmarks/CommandBenchmark.cs Outdated
@fdcastel fdcastel force-pushed the refactor-benchmarks branch from bd2e3a0 to e5e4075 Compare May 18, 2025 01:19
@fdcastel
Copy link
Copy Markdown
Member Author

Rebase with master.

@fdcastel fdcastel force-pushed the refactor-benchmarks branch 2 times, most recently from 2a8789b to 1ec795d Compare March 23, 2026 03:31
@fdcastel
Copy link
Copy Markdown
Member Author

Following up on the initial refactoring, these additional changes were made on the branch:

Infrastructure

  • Extracted a shared BenchmarkConfig : ManualConfig class to eliminate the duplicated inner Config class that existed independently in each benchmark class. Job configuration is now defined in a single place.
  • Extracted a BenchmarkBase abstract class holding the ConnectionString, CreateDatabase(), and GlobalCleanup() members that were duplicated across benchmark classes.
  • The connection string can now be overridden via the FIREBIRD_BENCHMARK_CS environment variable (falls back to the existing localhost default), enabling use in environments with a non-default Firebird setup.

Configuration

  • Updated the benchmark toolchain from .NET 8 to .NET 10, matching the project's TargetFramework.
  • Removed WithPlatform(Platform.X64) and WithJit(Jit.RyuJit) — both are no-ops on .NET 8+ (x64 + RyuJIT is the only supported configuration) and were just noise.
  • Added MarkdownExporter.GitHub so each run automatically writes a GitHub-flavored Markdown table to BenchmarkDotNet.Artifacts/, ready to paste into issues or PRs.
  • Added DefaultOrderer(SummaryOrderPolicy.FastestToSlowest) to make the results table easier to scan.

Benchmark methods

  • Single-value [Params(100)] and [Params(100_000)] replaced with plain constants — a single-value [Params] adds a column to the results table without varying anything.
  • Fetch methods now return the last read value instead of discarding it, to make dead-code elimination by the JIT impossible.
  • Added async variants (ExecuteAsync, FetchAsync, OpenCloseAsync) for all sync benchmarks. Async ADO.NET paths have different allocation profiles (state machines, Task wrappers) and are the primary path in web workloads.

New benchmarks

  • ConnectionBenchmark — measures connection open/close throughput (sync and async), which exercises connection pool logic.
  • LargeFetchBenchmark — added to the committed tree (was previously an untracked file).

Script

  • run-benchmark.ps1 now accepts -Benchmark ConnectionBenchmark in addition to the existing values, and two new optional switches: -Disasm (enables --disasm) and -Profile (enables ETW profiling on Windows).

Documentation

  • Added docs/benchmark.md describing prerequisites, how to run, connection string configuration, available benchmarks, and where results are written.

@fdcastel
Copy link
Copy Markdown
Member Author

Future improvements:

  1. async benchmarks
  2. connection-pool benchmarks
  3. configurable connection string
  4. Remove Duplicated config (use a shared base)

fdcastel added 19 commits May 23, 2026 16:14
StartActivity() returns null when no listeners are interested in the
activity. The code accessed activity.IsAllDataRequested without a null
check, causing a NullReferenceException. Use the Npgsql-style pattern
'activity is not { IsAllDataRequested: true }' for early return.
Stopwatch ticks are not TimeSpan ticks when Stopwatch.IsHighResolution
is true (most systems). TimeSpan.FromTicks(stopwatchTicks) produces
incorrect durations. Use Stopwatch.GetElapsedTime() which correctly
handles the frequency conversion.
The semantic conventions renamed db.system to db.system.name. The
well-known value for Firebird is 'firebirdsql', not 'firebird'.
Updated in both FbActivitySource (spans) and FbConnection (metrics).
The semantic conventions mark db.namespace as Conditionally Required
when available. The connection's Database property is used as the value.
The semantic conventions mark error.type as Conditionally Required on
failure. For FbException, use the SQLSTATE code; for other exceptions,
use the full exception type name.
Per OTel semantic conventions, db.query.text and db.query.parameter.*
are Opt-In level attributes that may expose sensitive data. They should
not be collected by default. Added IsQueryTextTracingEnabled flag and
reuse IsParameterLoggingEnabled for parameter tracing.
Replace hardcoded '1.0.0' version with the assembly's informational
version, keeping the telemetry version in sync with the NuGet package.
…ecording

TraceCommandException now records metrics immediately and resets
_startedAtTicks so the later TraceCommandStop (called during Release)
does not record a second, inflated duration that includes cleanup time.
- Remove db.transaction_id custom tag (not in OTel spec, causes server
  round-trip per traced command)
- Remove commented-out db.snapshot_id code
- Remove deprecated exception.escaped attribute
- Remove NormalizeDbNull helper, inline DBNull check
- Set server.port on spans when using a non-default port
- Set db.stored_procedure.name for StoredProcedure command types
- Fix server.address in MetricsConnectionAttributes to not concatenate
  the port (server.port is now a separate attribute)
- Clean up leftover placeholder comments
Set db.query.summary as the low-cardinality activity name on all spans.
For Text commands, extract the first SQL verb (SELECT, INSERT, etc.)
and use it as db.operation.name and in the activity name.
Debug.Assert checks that TraceCommandStart was called before the stop
or exception path, helping catch lifecycle bugs during development.
Prevents potential NullReferenceException if metrics are recorded
before the connection string is set.
Change GetMetrics() to return IEnumerable of tuples via Select instead
of allocating a Dictionary via ToDictionary on every collection cycle.
Also simplify GetConnectionMax to use Select instead of SelectMany.
Suggest default histogram bucket boundaries for operation duration and
connection create time histograms, improving out-of-the-box resolution.
Guarded by NET9_0_OR_GREATER since InstrumentAdvice is a .NET 9 API.
Expose the telemetry source names as public constants so consumers can
easily subscribe: builder.AddSource(FbTelemetry.ActivitySourceName) and
builder.AddMeter(FbTelemetry.MeterName). Internal code updated to use
the constants.
fdcastel added 17 commits May 23, 2026 16:14
Document the ActivitySource and Meter names, available span attributes,
opt-in sensitive attributes, and available metrics with their types and
semantic convention references.
- Rename project 'Perf' to 'FirebirdSql.Data.FirebirdClient.Benchmarks'.
- Update project to use .net8.
- Upgrade BenchmarkDotNet to version 0.14.0.
- Update baseline nuget package to v10.3.1.
- Add /BenchmarkDotNet.Artifacts to .gitignore.
- Pass command-line arguments to BenchmarkDotNet engine.
- Apply SQL Formatting. Use raw strings.
- Add script run-benchmark.ps1.
@fdcastel fdcastel force-pushed the refactor-benchmarks branch from 533d57c to 9d42b35 Compare May 23, 2026 20:04
@fdcastel
Copy link
Copy Markdown
Member Author

Rebased onto latest master.

…tection

CommandBenchmark (the default suite) only covered BIGINT and VARCHAR UTF8, so
the fixed-length CHAR(n) UTF8 rune path was untested -- the exact path where
issue FirebirdSQL#1272 (~11.7x allocation regression) lives. That regression surfaced only
via the opt-in LargeFetchBenchmark.

- CommandBenchmark: add "CHAR(100) CHARACTER SET UTF8" param; add SampleValue()
  so Execute sends a column-sized string (exercising the write/validate rune
  count path) instead of a bare int.
- docs/benchmark.md: correct the NuGet baseline description (pinned version plus
  a bump-per-release note), add an "Interpreting results" section noting that
  Alloc Ratio is the primary regression signal (with FirebirdSQL#1272 as a worked example),
  and list the third data type.

Verified locally against the released baseline: CHAR UTF8 Fetch Alloc Ratio
~9.6x and Execute ~1.9x, while BIGINT and VARCHAR stay ~1.0.
@fdcastel
Copy link
Copy Markdown
Member Author

Added a couple of benchmark improvements prompted by #1272 (the UTF8 CHAR fetch allocation regression). The lesson from that investigation: the regression lives only in the fixed-length CHAR(n) CHARACTER SET UTF8 rune path, but the default CommandBenchmark only covered BIGINT and VARCHAR(10) UTF8, so it never surfaced there — it was visible only via the opt-in LargeFetchBenchmark. These changes make the default suite catch that class of regression and make the results easier to read.

What changed (commit 46db5a1)

  • CommandBenchmark: added CHAR(100) CHARACTER SET UTF8 to the DataType params. This covers the read/truncation path in Fetch/FetchAsync automatically.
  • CommandBenchmark.Execute: Execute/ExecuteAsync now send a column-sized string (via a small SampleValue() helper) instead of a bare integer, so the write/validate rune-count path is exercised too — previously it was never benchmarked.
  • docs/benchmark.md: corrected the NuGet baseline description (it claimed "latest published" but the project pins a specific released version; it now documents the pin plus a bump-per-release note), added an Interpreting results section explaining that for these I/O-bound benchmarks the Alloc Ratio (not the noisy Mean) is the primary regression signal — using Major performance degradation introduced with 10.3.3 (UTF8 char fetch) #1272 as a worked example — and listed the new data type.

Effect — the default benchmark now catches #1272

Running CommandBenchmark on this branch, local Core100 vs the released NuGet100 baseline:

DataType Allocated (baseline → local) Alloc Ratio
BIGINT — Fetch 46.3 KB → 50.9 KB 1.10
VARCHAR(10) UTF8 — Fetch 50.2 KB → 54.8 KB 1.09
CHAR(100) UTF8 — Fetch 187.7 KB → 1805.1 KB 9.61
CHAR(100) UTF8 — FetchAsync 209.6 KB → 1825.1 KB 8.71
CHAR(100) UTF8 — Execute 409.5 KB → 759.9 KB 1.86
CHAR(100) UTF8 — ExecuteAsync 977.9 KB → 1329.1 KB 1.36

Only the CHAR UTF8 cases regress; BIGINT and VARCHAR stay at ~1.0, which also matches the root cause (only DbDataType.Char goes through the rune-truncation path). These ratios reflect the current provider code on master — once the fix from #1252 lands they should return to ~1.0.

CI is green across FB30/FB40/FB50.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants