Repository navigation
Add a /benchmark PR command for the adapter accuracy benchmark - #137
Open
AlonMalach wants to merge 1 commit into
Open
AlonMalach wants to merge 1 commit into
AlonMalach wants to merge 1 commit into
Conversation
A maintainer's `/benchmark [--model <id>] [--no-cache]` comment on a pull request benchmarks its head commit and publishes the row to https://generative-computing.github.io/granite-switch/benchmarks/, the way /gpu-test runs the GPU tests. Only this launcher goes to main, because comment workflows run only from the default branch. It checks the commenter's role and dispatches adapter-benchmark.yaml, which stays on the benchmark's own branch (feature/adapter-benchmark) with the rest of the benchmark. That workflow re-checks the role on the GPU runner, sets a commit status and posts the results on the pull request. Signed-off-by: AlonMalach <alonmalach@gmail.com>
AlonMalach
requested review from
antonpibm,
aviv1ron1,
freunda and
yairallouche
as code owners
October 4, 2026 13:19
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
A maintainer's
/benchmarkcomment on a pull request benchmarks the PR's head commit with the adapter accuracy benchmark, the way/gpu-testruns the GPU tests:The run publishes the commit's row to https://generative-computing.github.io/granite-switch/benchmarks/, sets a commit status, and posts the results on the PR.
Why only these two files
GitHub runs comment-triggered workflows only from the default branch. The benchmark itself (the harness, its workflow
adapter-benchmark.yamland the results page) lives onfeature/adapter-benchmarkand does not touch main. This PR adds only the launcher:.github/workflows/benchmark-command.yamlreacts to/benchmarkcomments on a GitHub-hosted runner..github/scripts/benchmark_command.shchecks the commenter's role (Maintain or Admin, with the existingcheck_role.sh), reads the flags, and dispatchesadapter-benchmark.yamlonfeature/adapter-benchmarkwith the PR's head SHA.Both follow
gpu-test-command.yamlandgpu_test_command.sh: the same role check, comment text passed only through env and quoted arguments, the first line only, and unknown flags declined rather than ignored.Security
/opt/gsw/check_role.sh, like the GPU tests./gpu-test, this runs the PR's code on the cluster, so a maintainer should read a PR before commenting.feature/adapter-benchmark, so that branch should be protected like main.Testing
gh: valid commands dispatch to the benchmark branch with the right inputs; a mistyped flag (--nocache),/benchmarksand a non-maintainer are declined, with nothing dispatched.gh workflow run adapter-benchmark.yaml --ref feature/adapter-benchmarkbenchmarked b7e5421 on both models and published the rows.