dba-runner

dbaai_bench runs one model against one task on servers you already have. This runs many models against many tasks on servers it creates itself, marks the results with its own commands, and destroys the servers afterwards.

uv sync                                     # installs ../dbaai_bench as an editable checkout
uv run python dbrun.py suites               # what tests exist
uv run python dbrun.py run -m sonnet,gpt-oss-120b -s mysql-install

One run leaves a directory like this:

runs/20260822-143512/
  run.json          what was asked for, and the exact command that asked
  results.jsonl     one line per finished cell, appended as it happens
  leaderboard.md    the table, the tables that explain it, and the tasks it scored
  results.csv       one row per stage, for a spreadsheet
  events.jsonl      droplets created and destroyed, with their ids
  known_hosts       this run's host keys, not yours
  cells/<model>/<suite>/01-<stage>/     <suite>-<os>-<db> for whichever axis is wide
      task.md         the task exactly as the model received it
      grade.json      every check, what it returned, and the score
      bench.stdout.txt
      bench/          the do-dba run directory: transcript.jsonl, report.md, secrets.json

What it is doing

• A cell is one model against one suite, on one operating system, for one database. It owns its own droplets, its own subprocess and its own directory, so cells run four at a time and a model that wedges costs one cell.
• A suite is one TOML file in suites/. It says how many servers the work needs and lists the stages in order.
• A stage is one task handed to the model, plus the checks that mark it. One stage means a fresh-droplet single task. Several stages mean the same droplets, in order - which is how you test work that builds on itself: set up replication, then upgrade both servers underneath it.
• A lease is the droplets a cell holds from its first stage to its last. --cloud aws makes them EC2 instances instead, which changes where they come from and nothing else on this page: this file says droplets throughout, and everything it says about them is true of both.

Each stage ends with two verdicts, and they are allowed to disagree:

status	what do-dba concluded from the model's own VERIFY commands - done, unverified, exhausted, ...
score	the fraction of check weight the runner's own commands found satisfied

A stage that reports done and scores 40% is the interesting case: the model convinced the harness and not the suite. The leaderboard lists every one of them under Said done, was not. The checks were written before any model saw the task, and they run over the runner's own SSH connection after the model's process has exited.

Getting set up

Two things are needed - a gateway, and something to make servers with - and the second one is optional in the sense that you can bring your own servers instead.

A model gateway. The same keys dbaai_bench uses - OPENROUTER_API_KEY, or DIGITALOCEAN_INFERENCE_KEY with --provider digitalocean. The runner reads its own .env first and the bench's second, so a key that already works there works here - this project's .env only needs the names it is actually overriding. A name left blank in it is dropped rather than read as an empty answer, so it falls through to the bench's file instead of shadowing the real key there.

A third gateway needs no key at all. --provider selfhosted - self: for short - talks to whatever OpenAI-compatible server $DBA_SELFHOSTED_BASE_URL points at, LM Studio or vLLM or llama.cpp or Ollama, and nothing it serves is billed per token: those cells count as $0 and the leaderboard says so rather than leaving a row that looks free by accident. Two things are worth knowing before pointing a matrix at one. A server that holds one model in memory at a time wants -j 1, or the run spends its droplet hours swapping weights instead of working. And the first request to each model waits while those weights load, which happens inside the stage's own timeout rather than beside it.

doctl, to create droplets.

winget install DigitalOcean.doctl        # or: scoop install doctl
doctl auth init                          # paste a personal access token
doctl account get                        # check it

A token in $DIGITALOCEAN_ACCESS_TOKEN is used in preference to whatever doctl auth init stored, so a run can be pointed at a different account without touching your own login. It is passed to doctl through the environment and never as an argument, because arguments are visible in the process list.

Or the aws CLI, to create EC2 instances instead. --cloud aws is the only difference to the run:

winget install Amazon.AWSCLI             # or: scoop install aws
aws configure                            # or: aws sso login --profile my-profile
aws sts get-caller-identity              # check it

The runner shells out to the CLI you already configured rather than talking to the API itself, so $AWS_PROFILE, $AWS_REGION, ~/.aws/config and an instance role all work here exactly as they do in your own shell. --region overrides the region for one run, $AWS_BINARY says where the CLI is if PATH does not reach it, and $DBRUN_CLOUD=aws makes the flag the default so you stop typing it.

Four things EC2 makes the runner do differently, all of them on the plan before anything is created:

• No image lets root in. The key pair is installed for the image's own account - ubuntu, ec2-user, admin, rocky, whoever the vendor chose - and root's authorized_keys holds the same key behind a forced command that says so. So the first thing done to a new instance is one login as that user, which copies the key over root's and takes the refusal off; everything after it, the bench and the suites and every check, is root over the runner's own connection exactly as on a droplet. The runner knows the names the big vendors use and the plan says which one it will try. A Marketplace or custom AMI is whoever built it, so --ssh-user NAME names it - without that the runner falls back to trying ten common names in turn, which works surprisingly often and costs a refused connection each time it does not. Whatever the account, it needs passwordless sudo; an image that does not give its own user that cannot be graded here, and the run says so rather than waiting.
• One key pair, not every key. EC2 installs exactly one, so --ssh-key NAME picks it and a region with a single key pair needs no flag. Its private half has to be somewhere the runner can reach - your agent, or -i ~/.ssh/that-key.pem, which the plan asks for if it cannot see one. -i is expanded and opened before anything is created: the tilde is yours, not the runner's - paramiko and dba.py never see a shell - and a path that is not a file is refused up front rather than at the first login, where a file that will not open is indistinguishable from a machine that is still booting and gets waited on for the whole boot timeout, per node, on instances that are already billing.
• Its own security group. A new instance is in nothing that would let SSH in, so the run creates a tagged group of its own - port 22 from --ssh-from (default 0.0.0.0/0, narrow it to your own address where you can) plus everything between the group's own members, which is what the replication suites need - and gives it back in teardown. --security-group ID uses one of yours instead and leaves it alone. --subnet and --vpc place the instances; the default is the roomiest subnet of the region's default VPC, and whatever you name needs a route to the internet, because the runner arrives over SSH and the model installs packages.
• The disk is billed separately, and the vendors' images ship 8 GB, which will not hold a database and a dump of it. Each instance gets a 100 GB gp3 root volume; --disk GB changes that.

Two more things read the same as on DigitalOcean and are answered differently underneath. --os rocky-9 has no account-wide catalogue to resolve against, so it asks each vendor's own published account for the newest image they released under their own naming scheme - an ami- id is taken as given. And --size s-4vcpu-8gb, which is what the suites here pin, is read as the capacity that slug names and answered with the cheapest instance type that meets it, so a suite saying what the task needs still gets it (the default is t3.xlarge).

Skip all of that to run against servers you already have:

uv run python dbrun.py run -m sonnet -s mysql-install --host root@203.0.113.10

Those are never destroyed, and every cell gets the same machines one after another - fine for trying the runner out, not a matrix.

Running a matrix

# every suite in suites/, three models, four cells at a time
uv run python dbrun.py run -m sonnet -m gpt-oss-120b -m qwen3-max

# one suite, models from a file, a hard cost ceiling
uv run python dbrun.py run --models-file models.txt -s mysql-replication --max-cost 20

# see what it would do and spend nothing
uv run python dbrun.py run -m sonnet --dry-run

# mix gateways in one run: provider:model overrides the default
uv run python dbrun.py run -m sonnet -m do:openai-gpt-oss-120b

# the box down the hall, one model in memory at a time
uv run python dbrun.py run -m self:qwen/qwen3.8-27b -s mysql-install -j 1

# the same models and suites on three operating systems, compared
uv run python dbrun.py run -m sonnet -s mysql-replication \
    --os ubuntu-26 --os debian-13 --os rocky-9

# and the same again across three databases
uv run python dbrun.py run -m sonnet -s mysql-replication \
    --db 9.7 --db percona-8.4 --db mariadb-11.4

# one task, three engines: the install suites are written check for check
uv run python dbrun.py run -m sonnet \
    -s mysql-install -s postgres-install -s mongodb-install

# the other cloud: EC2 instances rather than droplets, in London
uv run python dbrun.py run -m sonnet -s mysql-install \
    --cloud aws --region eu-west-2 -i ~/.ssh/laptop.pem

Model names are resolved against the gateway's catalogue before anything is created, the same partial-name matching dba.py -m does. A typo fails in two seconds instead of after forty droplets.

An id served by two gateways is two cells rather than one, which is the point of running both: -m qwen/qwen3.8-27b -m self:qwen/qwen3.8-27b gets its own droplets, its own directory and its own row on each side, and the rows are labelled or: and self: so the table says which of them was which. Only ids that appear on more than one gateway are labelled - everything else keeps its bare name, so a matrix that names each model once reads exactly as it did before.

Suites are matched by name, unique prefix or path, so -s postgres-t finds postgres-tune. A prefix that matches several suites is refused and says which, which is also the quickest way to remember the names: -s mongodb lists the three MongoDB suites rather than guessing between them. With no -s, every suite in suites/ runs.

The matrix is built suite-major: every model gets tried on the first suite before any model sees the second. A run stopped early is then a comparison rather than one model's complete row.

Ctrl+C sets a flag every cell polls. Running stages are killed, pending cells are cancelled, the droplets are destroyed, and the cells that already finished are tabulated - then --resume picks up the rest:

uv run python dbrun.py run -m sonnet -m opus --resume 20260822-143512

--resume skips the (model, gateway, suite, OS, database) cells that finished - the same pair on the run's other image, or its other database, or the same weights on another gateway, is still owed its droplets. A cell the runner broke - a timeout, a droplet that never came up - is not counted as finished, because there is nothing to learn from it.

Operating systems

--os (spelled --image if you prefer) is repeatable, and each one it is given is another axis of the matrix: cells become models x suites x OSes, one OS at a time within each suite, and one leaderboard compares them. Given one --os, or none (the default is ubuntu-26-04-x64), nothing changes - the OS is recorded but stays out of the names and the tables, because one OS is not a comparison.

• Friendly names. rocky-9 resolves against your own account's image list, so it finds rockylinux-9-x64 without you looking it up. A name that matches several images, or none, fails before a droplet exists and says which images it was choosing between. On --cloud aws the same name is answered by asking Rocky's own published account for its newest Rocky-9-EC2-Base-*, because an AMI is per region and there is no one catalogue to match against; every image the matrix may boot is resolved in preflight, so a third OS is not discovered hours in. An ami- id is taken as given, and a distribution neither vendor table knows is named that way.
• A suite may decline. os_family says which package manager a suite's shell was written for, and a suite that cannot run on one of the run's images is left out of that column with a not run: line on the plan, rather than being graded on a dnf it never had. If that empties the matrix, the run stops instead of starting.
• A suite may pin. A suite with its own image runs once, on that image, whatever --os says - it is testing something about that OS specifically.
• The OS is part of a cell's identity: droplet names, directory names, the row labels, results.jsonl, the CSV and --resume all carry it.

--os and --host are mutually exclusive: an operating system is something the runner chooses when it creates a server, and your own servers already have one.

Databases

Six products across three engines, and none of them is a drop-in for any other. --db (spelled --mysql if you prefer - the axis was called that when MySQL was all it had) is repeatable the same way --os is, and each value is another axis of the matrix:

uv run python dbrun.py run -m sonnet -s mysql-install \
    --db 9.7 --db percona-8.4 --db mariadb-11.4

# the same question, asked of the other two engines
uv run python dbrun.py run -m sonnet -s postgres-tune --db pg-18 --db pg-16
uv run python dbrun.py run -m sonnet -s mongodb-install --db mongodb-8.0 --db psmdb-8.0

Whatever you write is what the models are asked for. There is no list of versions to be on and no list of names to be on, because a benchmark whose vocabulary lags its subject cannot ask the only question worth asking the week MariaDB 12.3 ships.

written	means
mysql-9.7, percona-8.4, mariadb-11.4	that flavour at that version
pg-18, postgres-17, mongodb-8.0, psmdb-8.0	the other two engines, in the words their own documentation uses
mariadb-12.3, mysql-8.0.36, pg-19	the same, at any version you like - nothing is checked against a list
percona, mariadb, postgres, psmdb	that flavour at its newest version here, which is the one thing the built-in versions are still for
9.7, 8.4	that version of MySQL Community - a bare number is a MySQL, because that is where this axis started
"Percona Server 8.4", "MariaDB 11.4 rc"	your words, reaching the task exactly as written
"FerretDB 2.0"	a product this runner has never heard of

One token is the shorthand: mysql (Community), percona (Percona Server for MySQL), mariadb, postgres, mongodb (Community), percona-mongodb, the vendors' own words (percona-server-for-mysql, maria-db, postgresql, pgsql, psmdb, mongodb-org) and a version if you want one. It resolves to the vendor's own name for the product, so --db psmdb-8.0 asks the model for "Percona Server for MongoDB 8.0".

Anything with a space in it is prose, kept word for word: --db "Percona Server 8.4" asks for exactly that and is recorded as percona-server-8.4. The runner reads three things out of it and edits nothing - a trailing version if the last word looks like one, which of the six flavours it is, and which engine that flavour belongs to, because grading has to know both what a server should look like and which suites could honestly mark it. The words are read most-specific-first and the MySQL ones last, since "Percona Server for MongoDB" contains percona and is not a MySQL. A name with none of them in it is a flavour of its own, identified by grepping whatever the server says about itself - VERSION(), version(), db.version() - for the name itself, and offered to every suite on the axis, because which engine somebody meant by "FerretDB 2.0" is not something a runner should be guessing.

Two things are still refused, before anything is created: a value with no name in it, and two values that would be recorded as one row (percona-8.4 and "Percona 8.4" are both percona-8.4 on a leaderboard). Everything else the plan says rather than refuses, because a typo and a product released last Tuesday are the same string:

  db       myqsl-8.4
  myqsl-8.4: 'myqsl' is not a flavour anything here knows, so every suite on the
  --db axis - whichever database it is written for - will be asked for 'myqsl 8.4'
  and will identify what turns up by grepping the version it reports for 'myqsl'
  - did you mean mysql?

A version outside the built-in list gets the quieter version of that, naming the ones this runner has been kept up to date with. Both land above the confirmation prompt, which is the point: a database nobody can install grades every model at zero and looks exactly like every model failing.

One --db, or none, changes nothing about the matrix - the database is recorded on every row but stays out of the cell names and the table headings, which have nothing to distinguish. The default is one per engine (mysql-9.7, postgres-18, mongodb-8.0), so a bare dbrun run asks each suite for its own engine's. The plan still prints the db line whenever it is not the default, because one database is a decision even when it is not an axis.

• The axis is really one axis per engine. --db percona-8.4 --db psmdb-8.0 does not cross every suite with both: nothing can grade both, and a MySQL suite asked for MongoDB would score zero for having been asked the wrong question. So a suite takes the values on its own engine's line, and an engine the run never named falls back to that engine's default rather than dropping its suites - --db mariadb-11.4 says which MySQL to compare and says nothing at all about MongoDB, and reading it as "no MongoDB today" would shorten the matrix by six suites without a word.
• A suite may decline. db_flavour lists the flavours a suite's checks can honestly mark, and one the suite has not claimed is left out with a not run: line on the plan. mysql-tune's tuning stage rests on performance_schema.variables_info, which MariaDB has not got, so grading it there would score a MariaDB tuner zero for MariaDB not being MySQL. A flavour from outside the six is nobody's to decline - no suite could have named a product that did not exist when it was written - so every suite on the axis is asked for it, and the warning above is where the doubt goes.
• A suite may sit the axis out. No db_flavour at all means the suite is not on this axis: it runs once however many databases the run compares, and its rows record no database. That is mysql-restore, whose cloud-init installs whatever the distribution calls mysql-server. Note the asymmetry with os_family, where saying nothing means runs on anything - a suite that mentions the product but declares no flavour, or declares flavours and never mentions the product, is refused at load time rather than run.
• The database is part of a cell's identity: droplet names, directory names, the row labels, results.jsonl, the CSV and --resume all carry it.

--db and --host are not mutually exclusive, unlike --os: the product is what the task asks the model to install, not what the server already boots.

Money

Two meters run at once, and the runner shows both.

• Models. --max-cost USD is the run's ceiling: once spend reaches it no new cell starts, and the report says which were skipped. A cell already running is left alone - it is bounded by its own --max-cost-per-stage, and killing a cell halfway leaves a half-built server and a result worth nothing. Suites can set their own per-stage cap, and most of the ones here do. Self-hosted cells report no token cost, so a matrix made only of them never reaches any ceiling you set and the plan says as much instead of promising a guard that cannot fire. The droplets they drive are billed the same as anyone's.
• Droplets. Billed by the hour whether or not anybody remembers them. Every droplet is tagged dbaai-runner, dbrun-<runid> and suite-<name>, and its id is written to events.jsonl before the model is handed anything. So whatever happens to the process:

uv run python dbrun.py reap                        # anything this runner ever left behind
uv run python dbrun.py reap --run 20260822-143512  # just that run's
uv run python dbrun.py reap --dry-run              # list them and stop

On AWS the same tags are key and value - dbaai-runner=<runid>, dbrun-suite=<name> and a Name you can read in the console - and a reap takes back the run's security group as well as its instances. Two things differ enough to be worth typing:

uv run python dbrun.py reap --cloud aws --region eu-west-2   # EC2 tags are per region
uv run python dbrun.py reap --cloud both                     # both accounts, one pass

reap looks where --cloud says and nowhere else, and the default is DigitalOcean (or $DBRUN_CLOUD) - so a reap that says nothing was found has answered about one cloud. The rate on an EC2 plan is arithmetic rather than a quote: EC2 has no call that will price an instance without the pricing endpoint and its own permissions, so the figure is the on-demand list price in us-east-1 plus the volume, and the plan says "about". Your region, your agreement and any savings plan all say otherwise.

--keep leaves the droplets up for a post-mortem. It is the one way this runner spends money after it exits, so it says so on the way in and on the way out.

--keep-failed is that aimed at the cell that needs it. Some failures cannot be diagnosed from the log - an sshd that refuses the login, a cloud-init that wedged, a disk that filled - and destroying the server destroys the evidence, which is how the same failure gets diagnosed twice. So a cell the runner broke keeps its servers and everything else is destroyed as usual: a matrix of forty can be run for the price of the one that needs looking at. What it keeps, it tells you how to get into, as the cell fails and again as the last line of the run:

kept dbrun-20260825-074411-01-mysql-in 198.51.100.7 for diagnosis: could not get a root login
  ssh -i ~/.ssh/id_ed25519 ubuntu@198.51.100.7
  aws ec2 get-console-output --instance-id i-0a1b2c --region us-east-1 --output text
  dbrun.py reap --cloud aws --region us-east-1 --run 20260825-074411   # when you are done

Three details are the point of it. The login offered is the image's own user, not root - root is exactly what a boot failure on EC2 means the runner never managed to arrange, so the lease's own user is the one that will not work. The console output comes with it because it is the only thing that answers on an instance whose sshd never came up, and on an instance that never even reached running it is the whole of the diagnosis - so those are kept too, addressless as they are. And on AWS the run's security group stays behind with them, because it is what lets port 22 through: deleting it would leave servers that are still billing and can no longer be reached.

A cell the runner broke means a cell that never got servers, timed out, crashed, or failed its setup - not a model that scored badly. A model that fails its task is a result, and keeping servers for every bad answer would leave a forty-cell matrix holding forty servers. reap is how it all goes, and the exact command is printed.

--effort {low,medium,high} is the one flag that changes what a run costs without changing anything the plan counts. It reaches every stage as dba.py --effort, asking the model to think before each step, and the gateway turns that one word into whatever the model upstream wants - so it is worth comparing models on rather than a knob only one of them has. Thinking is billed as output tokens: expect several times the model spend of the same matrix without it, and expect the suites' own max_cost ($1.50 to $5 a stage across the ones here) to be reached sooner, which shows up as stages stopping on cost rather than on a verdict - so the plan warns about it above the confirmation prompt, beside the cost caps. Unset is the absence of the flag rather than a word meaning none: a reasoning model goes on reasoning at the gateway's default, exactly as in every run recorded so far. The effort is part of each stage's recorded command line in grade.json, so a cell that thought is distinguishable from one that did not long after the run.

Writing a suite

A suite is a TOML file in suites/. The short form is one task:

name = "mysql-install"
description = "Install MySQL 9.7, a database, and a login user for it"
nodes = 1
max_steps = 25
max_cost = 1.5

task = '''
Install MySQL 9.7 from the vendor's own apt repository on this server.
Create a database called `app` and a login user called `app_rw`.
'''

  [[check]]
  name = "mysql is running and starts at boot"
  weight = 2
  command = "systemctl is-active --quiet mysql && systemctl is-enabled --quiet mysql"

  [[check]]
  name = "the app database exists"
  command = "mysql -N -B -e \"SHOW DATABASES\" | grep -qx app"

The long form is [[stage]] tables with [[stage.check]] under each, which is what a suite that reuses its droplets looks like - see suites/mysql-repl-upgrade.toml.

Suite keys

key	meaning
name	the suite's name, and part of its droplets' hostnames
description	one line, shown in listings
nodes = N	N servers, unnamed. The harness calls them node1, node2, ... and the model works out the roles
names = [...]	N servers whose roles the suite has decided; the model is told these names and follows them
size / image / region	override the run's droplet settings for this suite. An image here takes the suite off the --os axis: it runs once, on that image. On --cloud aws a droplet slug is read as the capacity it names and a friendly image name as the vendor's newest, so a suite written for DigitalOcean still asks for what it meant
os_family	which OS families the suite's shell is written for - "debian" (apt, so Ubuntu and Debian both) or "rhel" (dnf), one or a list. --os leaves the suite out of an image it has not claimed
db_flavour	which databases the suite's checks can honestly mark - "mysql", "percona", "mariadb", "postgres", "mongodb", "percona-mongodb", one or a list. All of them must be the same engine, since one suite's checks cannot mark two. Saying nothing takes the suite off the --db axis rather than putting it on every column of it. Spelled mysql_flavour in a suite written before there was anything but MySQL, and still read
cloud_init / cloud_init_file	user-data for the droplets, for a fixture that should be in place before the model logs in
stop_on_fail	end the cell if a stage does not pass (off by default)

Stage keys

key	meaning
id	short, used as the directory name and in the tables
task / task_file	the prose the model is given. This is the specification: a vague task grades vaguely
title	one line for the progress display
setup	commands the runner runs before the model is told anything - the fixture the task starts from. A failure here is filed as setup-failed and never blamed on the model
setup_host	which servers the setup runs on (default: all)
weight	this stage's share of the cell's score
max_steps	steps the model gets (default 30)
max_cost	dollars this one task may spend
timeout	wall clock for the whole stage, in seconds (default 3600)
command_timeout	per command on the server, in seconds (default 300)

Check keys

key	meaning
name	what the tables call it
command	shell, run by the runner over its own connection. Exit 0 is a pass
host	"*" every server must pass (the default), "one" exactly one must, "any" at least one must, or a node's name
weight	its share of the stage score (default 1)
expect_exit	when success is not exit 0
contains / not_contains	when the exit code is not the answer - SHOW REPLICA STATUS exits 0 either way
timeout	seconds (default 60)

host = "one" and host = "any" are what make a role-agnostic suite gradeable: exactly one of these servers is a replica that is caught up, without the suite dictating which one the model should have chosen. The nodes that satisfied it are recorded, so the report can say which way round the model built it.

Use "one" for a role only one server can hold and "any" for one that several can. The difference is worth getting right: two servers that both take writes and both have binary logging on are two standalone databases, and under "any" they would collect the marks for a replication pair. Both replication suites use "one" for every check that names a role.

Every check and setup command is told about the lease it is running in:

DBRUN_SELF              this node's label
DBRUN_NODES             every label, space separated
DBRUN_PEERS             the other labels
DBRUN_HOST_<LABEL>      that node's public address
DBRUN_PRIVATE_<LABEL>   its private address (the public one if it has none)

which is how a check can require that replication reads from the peer's private address rather than merely from somewhere.

A suite on the --db axis is told which database this cell asked for as well:

DBRUN_DB                mysql-9.7, percona-8.4, pg-18, psmdb-8.0, ferretdb-2.0
DBRUN_DB_ENGINE         mysql | postgres | mongodb | empty for a product from outside
DBRUN_DB_FLAVOUR        the flavour: percona | mariadb | postgres | percona-mongodb | ...
DBRUN_DB_VERSION        9.7          empty if none was asked for
DBRUN_DB_NAME           Percona Server for MySQL      (the vendor's words, or yours)
DBRUN_DB_TITLE          Percona Server for MySQL 8.4  (exactly what the task asks for)
DBRUN_DB_VERSION_RE     ^9\.7        what to grep the reported version for, empty for "any"
DBRUN_DB_MATCH          percona      what the version and what it says about itself must match
DBRUN_DB_REJECT         percona|mariadb   what it must not, or empty

Every one of them except _ENGINE is exported under DBRUN_MYSQL_* as well, because the axis had that name when MySQL was all it had and a suite written then still greps $DBRUN_MYSQL_MATCH. Nine variables of duplication is a cheap price for an unset $DBRUN_MYSQL_VERSION_RE being impossible: an empty pattern is a version check that passes for every version.

A check written against those needs nothing else to grade a product this runner has never heard of: an empty _REJECT and an empty _VERSION_RE both mean "no constraint", which is what grep -q "" does anyway, so --db "MariaDB Enterprise" grades as MariaDB at any version without a line of the suite changing.

Patterns and not only names, because every suite that cares asks the same two questions - is this the version that was asked for, and is it the right vendor - and neither is a string comparison. VERSION() is not enough on its own: Percona's is a bare 8.4.3-3 and names the vendor only in @@version_comment, and Community is the one product that has to be identified by what it is not. So a check greps both fields together:

version=$(q "SELECT VERSION()"); comment=$(q "SELECT @@version_comment")
echo "version=${version:-none} comment=${comment:-none}"
[ -n "$version" ] || exit 1
printf '%s\n' "$version" | grep -q "$DBRUN_DB_VERSION_RE" || exit 1
printf '%s %s\n' "$version" "$comment" | grep -Eqi "$DBRUN_DB_MATCH" || exit 1
[ -z "$DBRUN_DB_REJECT" ] || ! printf '%s %s\n' "$version" "$comment" | grep -Eqi "$DBRUN_DB_REJECT"

The same three lines carry to the other engines, and only the two fields change. PostgreSQL is one project, so version() and server_version are all there is to read. MongoDB is the awkward one: Percona tracks upstream's numbering exactly, so db.version() cannot tell a community server from a Percona one and the mongodb suites grep the version together with the installed package names - which is the only place either vendor writes itself down plainly.

In the task - and only there - {{db}} writes the product in as it was asked for, with {{db_name}}, {{db_version}}, {{db_flavour}} and {{db_engine}} for the parts. {{mysql}}, {{mysql_name}}, {{mysql_version}} and {{mysql_flavour}} still render the same things. Commands are given the variables above instead, deliberately: check commands are full of docker ps --format '{{.Names}}' and other Go templates that nothing here may touch. A {{db_ver}} that nothing replaces is refused at load time rather than handed to a model as literal braces.

Give a check a product-neutral name. Where the points went groups by check name, and it answers as the database and version asked for is one comparable row where it answers as MySQL 9.7 would be three that are not.

Checks are read-only by convention, and the starter suites break that convention deliberately twice: one check writes a marker row on whichever server accepts writes, and the next looks for that row on a server that refuses them. Checks run in the order they are written, which is what makes that pair work - and a write that arrives is the only real evidence that replication replicates.

A stage with no checks is refused at load time. Grading a stage on the model's own account of itself is the one thing the harness under test already refuses to do.

The starter suites

suite	servers	OS	db	what it is
mysql-install	1	debian	all three MySQLs	install the database, a schema, a scoped user, not open to the world
mysql-restore	1	debian	-	a dropped database and last night's dump - cloud-init and setup build the situation
mysql-replication	2	debian, rhel	all three MySQLs	replication over the private network, roles left to the model
mysql-repl-upgrade	2	debian, rhel	mysql	two stages on the same pair: replicate, then move to Percona without breaking it
mysql-tune	1	debian	mysql, percona	two stages: install it, then size it to the machine - graded as ratios of the server's own memory, and on whether the settings would survive a restart
mysql-group-replication	1	debian, rhel	mysql, percona	three containers, three ONLINE members, writes crossing all of them
postgres-install	1	debian	postgres	the same task as mysql-install, in PostgreSQL's vocabulary: a cluster, a database, a login role that is not a superuser
postgres-tune	1	debian	postgres	install it, then size it - shared_buffers as a ratio of the machine, and pending_restart to catch an ALTER SYSTEM that was never restarted into
postgres-replication	2	debian, rhel	postgres	streaming replication over the private network, roles left to the model
mongodb-install	1	debian	mongodb, psmdb	install it, turn access control on, a database with a collection in it, a user scoped to it
mongodb-tune	1	debian	mongodb, psmdb	install it, then tune the server and its host: the WiredTiger cache, transparent huge pages, swappiness - and each of those in a way that outlives a reboot
mongodb-replication	2	debian, rhel	mongodb, psmdb	a two-member replica set with a keyfile, over the private network, roles left to the model

They are the tasks this benchmark was built for, and they are not cheap: the runs they were written from took 40-70 steps and half an hour each. Start with -s mysql-install and one model.

The OS column is each suite's os_family, and it is what --os rocky-9 picks up: the install and tune suites are written around a vendor apt repository, so they are left out of an rhel column rather than failed on it. The replication suites grade a thing the database does rather than a way of installing it, and run on either family.

The db column is each suite's db_flavour, and what --db mariadb-11.4 or --db psmdb-8.0 picks up. Three suites per engine, deliberately the same three tasks: install, tune and replication are written check for check against each other, so a model's PostgreSQL score is readable beside its MySQL one and the difference is the database rather than the marking. On the MySQL line, mysql-install and mysql-replication mark all three flavours; mysql-repl-upgrade is Community-only because becoming Percona is the task; mysql-tune and mysql-group-replication need something MariaDB has not got. On the MongoDB line every suite marks both builds, because what tells them apart is the installed package name and not a check - which makes --db mongodb-8.0 --db psmdb-8.0 the cleanest comparison in the whole set. PostgreSQL has one flavour: what Percona distributes is PostgreSQL, with the same version(), so a second column would be noise - the axis worth comparing there is the major version, and --db pg-18 --db pg-16 is two repositories, two data directories and one set of checks. mysql-restore has no db_flavour at all and runs once whatever the run compares.

Reading the results

uv run python dbrun.py report                     # rebuild the last run's tables
uv run python dbrun.py report 20260822-143512

results.jsonl is authoritative and append-only; the leaderboard and the CSV are derived from it and can be rebuilt at any time, including after a scoring change.

The leaderboard has five sections: the score matrix, Said done, was not, Where the points went (every check and how many attempts satisfied it - the fastest way to find a check that is wrong rather than hard), Cells that did not get a fair run, and The tasks - the prose the models were given, quoted once per distinct wording, because a suite's paragraphs get edited between runs and a percentage is not a result without them.

A run that compared operating systems gets a sixth, By operating system, and one that compared databases gets By database. Either way every table that names a cell grows that axis in its row labels: rows read `sonnet @ rockylinux-9` or `sonnet / mariadb-11.4`, or both at once, and Where the points went counts each check per OS and per database rather than pooling them - because a check that passes on one image and not the other, or on MySQL and not on MariaDB, is the whole reason to run both. The tasks splits the same way, naming the database whenever a stage's {{db}} made two wordings out of one paragraph.

Layout

dbrun.py              entry point
dba_runner/
  cli.py              subcommands, screens, preflight
  matrix.py           cells, the thread pool, the budget, cleanup
  suites.py           the TOML format and its loader
  products.py         the --db axis: engines, flavours, versions, what a check is told
  provision.py        doctl, leases, droplets, reap
  aws.py              the same on EC2: instance types, AMIs, security groups
  harness.py          running dba.py as a subprocess and reading it back
  grade.py            the checks, over the runner's own SSH connection
  results.py          results.jsonl, the leaderboard, the CSV
suites/               the tests
tests/                offline suites - no network, no token, no money
run_tests.py          runs them all, one line per suite

do_dba is imported for its SSH transport and terminal handling, and dba.py is run as a subprocess: the thing being measured is allowed to fail badly, and in a subprocess a wedged model costs one cell and a timeout.

Tests

uv run python run_tests.py
uv run python run_tests.py suites grade

Nothing there touches the network, a DigitalOcean account, an AWS account or a model. doctl and aws are fakes that answer from a script, the bench is a fake dba.py that writes a transcript, and the servers are fakes that answer the checks - so the whole matrix, including a cell that times out and a droplet that leaks, runs on a laptop with no token and no money.

ec2 is the suite for the other cloud, and it is mostly about the three things with no DigitalOcean counterpart: a security group that outlives its instances (deleted on the retry after termination catches up, and reported as leaked when it does not), the root login a vendor image refuses on purpose, and a size and an image named in this project's vocabulary rather than EC2's - s-4vcpu-8gb becoming the cheapest type that holds it, rocky-9 becoming a lookup against Rocky's own account.

Three of them grade shell rather than Python. suites loads all twelve shipped suites and puts every check and setup script through sh -n, renders every task once per flavour the suite claims and fails on a {{ that survived, and asserts that each suite accepts its own engine's default database - a suite that refused it would be silently skipped by an ordinary run. tune runs mysql-tune's tuning checks against nine fake servers - one tuned well, one untouched, one whose settings would not survive a restart, one with performance_schema switched off - and asserts which checks should fail on each. repl grades both MySQL replication suites against twelve pairs of servers that are really directories with a stub mysql on their PATH: a correct pair, the same pair the other way round, an older server that spells every field Slave_, a pair replicating over the public internet, a replica that still takes writes, a faultless pair of MariaDB, two standalone databases, a dead peer. Three of the twelve are the --db axis: a Percona pair graded as a Community run and a Community pair graded as a Percona one both fail on the version check alone, and the Percona pair graded as the Percona run it is comes out clean. The suite file is byte-identical across all three - only the exported environment differs, which is the whole claim the axis makes.

That stub is an executable and not a shell function on purpose, because the client is where a suite goes wrong: this one refuses \G when the statement arrives through -e, exactly as the real one does, and a helper that asked that way scored a working replica zero on every model of two whole runs. One scenario grades a correct pair with that old helper put back and asserts it still comes out at the 0.625 the run recorded - so the day the stub stops being awkward, the test says so instead of passing everything.

Both stub-server graders are MySQL, and that is the gap in this battery. The four PostgreSQL and MongoDB tune and replication suites are loaded, rendered and parsed by suites, but nothing offline runs their checks against a fake server yet, so their marking has been argued rather than tested. Closing it means a stub psql and a stub mongosh of the same kind as the mysql one - awkward in the same places the real clients are awkward, which for mongosh means an adminCommand that answers {ok: 0} instead of failing.

A suite that mis-marks does not look like a broken suite. It looks like a model that failed.

Notes

• runs/ holds real credentials. The bench's secrets.json and the servers' generated passwords end up under it. It is in .gitignore and should stay there.
• Its own known_hosts. Per run, inside the run directory. Both clouds reissue addresses, and neither your ~/.ssh/known_hosts filling with dead droplets nor a recycled address looking like an attack is wanted.
• Error reading SSH protocol banner is not a failure. Ubuntu socket-activates sshd, so port 22 answers while the host keys are still being generated and the first connection closes where the banner should be. The runner retries for up to 7 minutes and says what happened either way, so paramiko's own traceback for it is kept off stderr - DBRUN_SSH_DEBUG=1 puts it back when the connection is the thing you are debugging.
• The retries back off, because sshd punishes the ones that do not. OpenSSH from 9.8 keeps a penalty per source address and refuses connections from one whose attempts keep failing - 15 seconds at first, up to 10 minutes if it continues, and every refusal inside the penalty extends it. A loop knocking every 5 seconds can therefore lock itself out of a machine that is working, for longer than it was ever willing to wait, and the run dies with could not get a root login. So the wait doubles after each failure up to 45 seconds. On EC2 there is a better answer than waiting: the runner asks for the instance's own two status checks first and only knocks once they pass, so most of the time there is no penalty to lapse. That check is a courtesy and never the verdict - if EC2 has not decided within 2 minutes, or will not answer, the login goes ahead and decides.
• Every wait says which wait it is and how long it has been. Booting a cell is four waits in a row - the instance reaching running with an address, port 22 answering, the root login going through, cloud-init finishing - and any of them can legitimately take minutes. Until they reported themselves the whole thing was one silent gap, which reads as a hung runner and got read that way: EC2 reaching running takes about 8 seconds, but the 7 minutes of nothing afterwards looked like the cloud being slow to say so. Each wait now prints a line every 20 seconds naming what it is still waiting for and its elapsed seconds, rate-limited so four cells booting at once do not bury the log. The four share one BOOT_TIMEOUT between them rather than starting a fresh one each, so a cell cannot spend 7 minutes per wait and a slow port no longer buys the login extra time.
• cloud-init is waited for. A freshly booted image is still installing things for a minute or two, holding the apt or dnf lock, and a model whose first step is apt-get would meet "Could not get lock" through no fault of its own. That is the difference between grading a model and grading a race.
• Multi-stage suites rely on do-dba's server-side credential store. Stage one's generated passwords stay on the server in /etc/profile.d/dba-secrets.sh, which is how stage two can log in at all. --no-server-secrets in the bench would break that, so the runner does not pass it.