[RNE Rewrite] feat: benchmark harness for every registry variant - #1363

Draft

msluszniak wants to merge 26 commits into

rne-rewritefrom

@ms/perf-benchmarks

Commits on Sep 4, 2026

test(bench): add an on-device performance regression harness
msluszniak
committed
66fac3e
test(bench): type case definitions against the pipelines they drive
msluszniak
committed
69fcdda
test(bench): make the synthetic waveform actually trigger the VAD
msluszniak
committed
62a6df9
test(bench): stop long runs from losing their results
msluszniak
committed
6eb9188
test(bench): record the ExecuTorch 1.3.1 baseline
msluszniak
committed
7e69a9b
test(bench): calibrate the comparator against two runs of identical code
msluszniak
committed
651c6a1
test(bench): detect thermal throttling, which invalidates a comparison
msluszniak
committed
fb5cc58
test(bench): follow the library's schema and constants exports
msluszniak
committed
348f13f
test(bench): cover instance segmentation, OCR and Kokoro TTS
msluszniak
committed
b5f8571
test(bench): pin the CPU clock for the duration of a run
msluszniak
committed
a623af0
test(bench): wait for the device to be cool instead of sleeping a guess
msluszniak
committed
0c3e34b
test(bench): segment with RF-DETR, not FastSAM
msluszniak
committed
febff66
test(bench): cover every published variant, 3 gated repeats each
msluszniak
committed
ef04fd9
test(bench): stop pinning clocks by default
msluszniak
committed
3d10b09
test(bench): take one measurement per variant, not three
msluszniak
committed
189c993
test(bench): hold the ceiling when the host cannot be reached
msluszniak
committed
9ef4bbb
test(bench): gate at 37C, and never write a measurement twice
msluszniak
committed
c9fc167
test(bench): stay plugged in, and refuse to measure in Battery Saver
msluszniak
committed
6cbdb63
test(bench): add a raw-output probe for device-only defects
msluszniak
committed
9dc8c16
feat(bench): measure ExecuTorch in-band, and build release
msluszniak
committed
98d8157
docs(bench): state that runs must be release builds
msluszniak
committed
c55e9f9
refactor(bench): derive the variant list at runtime, drop the replay pass
msluszniak
committed
9e1cf3f
fix(profiler): mark the JSI wrappers as worklets
msluszniak
committed
339490a
fix(bench): stop counting warmups in the ExecuTorch tally
msluszniak
committed
a0fb58b

Navigation Menu

Uh oh!

[RNE Rewrite] feat: benchmark harness for every registry variant - #1363

[RNE Rewrite] feat: benchmark harness for every registry variant#1363
msluszniak wants to merge 26 commits into
rne-rewritefrom
@ms/perf-benchmarks

Commits on Sep 4, 2026

test(bench): add an on-device performance regression harness

test(bench): type case definitions against the pipelines they drive

test(bench): make the synthetic waveform actually trigger the VAD

test(bench): stop long runs from losing their results

test(bench): record the ExecuTorch 1.3.1 baseline

test(bench): calibrate the comparator against two runs of identical code

test(bench): detect thermal throttling, which invalidates a comparison

test(bench): follow the library's schema and constants exports

test(bench): cover instance segmentation, OCR and Kokoro TTS

test(bench): pin the CPU clock for the duration of a run

test(bench): wait for the device to be cool instead of sleeping a guess

test(bench): segment with RF-DETR, not FastSAM

test(bench): cover every published variant, 3 gated repeats each

test(bench): stop pinning clocks by default

test(bench): take one measurement per variant, not three

test(bench): hold the ceiling when the host cannot be reached

test(bench): gate at 37C, and never write a measurement twice

test(bench): stay plugged in, and refuse to measure in Battery Saver

test(bench): add a raw-output probe for device-only defects

feat(bench): measure ExecuTorch in-band, and build release

docs(bench): state that runs must be release builds

refactor(bench): derive the variant list at runtime, drop the replay pass

fix(profiler): mark the JSI wrappers as worklets

fix(bench): stop counting warmups in the ExecuTorch tally

Commits on Sep 10, 2026

fix(bench): hold every iteration at the gate temperature, not just the case

fix(bench): judge a measurement on where its iterations started