Skip to content

Add a /benchmark PR command for the adapter accuracy benchmark - #137

Open
AlonMalach wants to merge 1 commit into
mainfrom
feature/benchmark-command
Open

AlonMalach wants to merge 1 commit into
mainfrom
feature/benchmark-command

Conversation

@AlonMalach

Copy link
Copy Markdown
Collaborator

What

A maintainer's /benchmark comment on a pull request benchmarks the PR's head commit with the adapter accuracy benchmark, the way /gpu-test runs the GPU tests:

/benchmark                          the first model (granite-4.1-3b)
/benchmark --model granite-4.2-3b   another model
/benchmark --no-cache               re-run a commit that already has its row

The run publishes the commit's row to https://generative-computing.github.io/granite-switch/benchmarks/, sets a commit status, and posts the results on the PR.

Why only these two files

GitHub runs comment-triggered workflows only from the default branch. The benchmark itself (the harness, its workflow adapter-benchmark.yaml and the results page) lives on feature/adapter-benchmark and does not touch main. This PR adds only the launcher:

  • .github/workflows/benchmark-command.yaml reacts to /benchmark comments on a GitHub-hosted runner.
  • .github/scripts/benchmark_command.sh checks the commenter's role (Maintain or Admin, with the existing check_role.sh), reads the flags, and dispatches adapter-benchmark.yaml on feature/adapter-benchmark with the PR's head SHA.

Both follow gpu-test-command.yaml and gpu_test_command.sh: the same role check, comment text passed only through env and quoted arguments, the first line only, and unknown flags declined rather than ignored.

Security

  • The launcher's role check is fast-fail UX only. The dispatched workflow re-checks the role on the GPU runner with /opt/gsw/check_role.sh, like the GPU tests.
  • Like /gpu-test, this runs the PR's code on the cluster, so a maintainer should read a PR before commenting.
  • The workflow that runs on the GPU runner comes from feature/adapter-benchmark, so that branch should be protected like main.

Testing

  • The handler, run locally with a stub gh: valid commands dispatch to the benchmark branch with the right inputs; a mistyped flag (--nocache), /benchmarks and a non-maintainer are declined, with nothing dispatched.
  • The dispatched side already works: gh workflow run adapter-benchmark.yaml --ref feature/adapter-benchmark benchmarked b7e5421 on both models and published the rows.
  • The comment trigger itself can only run after merge, since GitHub reads it from main.

A maintainer's `/benchmark [--model <id>] [--no-cache]` comment on a pull
request benchmarks its head commit and publishes the row to
https://generative-computing.github.io/granite-switch/benchmarks/, the way
/gpu-test runs the GPU tests.

Only this launcher goes to main, because comment workflows run only from
the default branch. It checks the commenter's role and dispatches
adapter-benchmark.yaml, which stays on the benchmark's own branch
(feature/adapter-benchmark) with the rest of the benchmark. That workflow
re-checks the role on the GPU runner, sets a commit status and posts the
results on the pull request.

Signed-off-by: AlonMalach <alonmalach@gmail.com>
@codecov-commenter

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants