[RNE Rewrite] feat: benchmark harness for every registry variant - #1363
Draft
msluszniak wants to merge 24 commits into
Draft
[RNE Rewrite] feat: benchmark harness for every registry variant#1363msluszniak wants to merge 24 commits into
msluszniak wants to merge 24 commits into
Conversation
msluszniak
added
test
msluszniak
force-pushed
the
@ms/perf-benchmarks
branch
2 times, most recently
from
August 27, 2026 10:44
383f54e to
df61e4d
Compare
Adds apps/benchmarks, a headless Expo app that runs the task pipelines against deterministic synthetic inputs and reports load time, inference latency and peak memory as JSON, plus a driver that collects a run and a comparator that diffs two runs and fails on regressions. Built to bracket an ExecuTorch bump: run the suite on 1.3.1, bump, run it again on the same device, compare. Three things the design turns on: - A raw-execute pass isolates ExecuTorch from the pipeline. Task timings fold model.execute together with preprocessing and post-processing, which are TypeScript and unaffected by a bump. The pass loads the .pte on its own and sizes its tensors from model.schema, so it covers every method a program exports and needs no per-model wiring. - Memory is sampled in a pass of its own. Reading total PSS on Android walks /proc/self/smaps and costs milliseconds, which would otherwise land in the inference numbers. - Inputs are pure functions of their parameters. Post-processing cost is input-dependent, so a harness reading a photo off the device would move for reasons unrelated to the change under test. The comparator refuses to diff runs from different devices, and reports a metric whose workload changed as INCOMPARABLE rather than as a delta. Refs #1078
The case list is heterogeneous, so the array can only be typed as BenchCase<any> — which let a case keep compiling after the pipeline it drives renamed the method it calls. Naming the pipeline's create as a separate leading parameter of defineCase makes it its own inference site, resolved before the case body is checked, so run is checked against the real instance type. Passing create inside the literal does not work: it is then inferred alongside run and TInstance collapses to its constraint. Also stop a failed build from leaving the collector waiting forever. It holds the port, so the next attempt could not start its own.
The first waveform was a harmonic stack, and on device the FSMN VAD scored none of it as speech: the case reported zero segments, so the segmentation path never ran and the comment claiming otherwise was wrong. Replaced with a glottal pulse train swept through three formant resonators plus aspiration noise. Still not speech and still fully deterministic, but close enough in spectral shape that the VAD now opens a segment per burst (10 over the 10 s waveform) and closes it on the gap. Also document that execute.<method> and pipeline.median are not comparable to each other: the raw pass takes dynamic dimensions at the top of their declared domain, so on all-MiniLM-L6-v2 it runs a 254-token forward while the pipeline runs a 20-token one.
A full-suite run died partway through and took every result with it. Three separate causes, all worth fixing: - The collector only wrote to disk on /end, so a run that stopped at case 10 of 12 threw away the nine cases it had already measured. Cases are now appended to a .partial file as they arrive, removed once the final report lands. - The adb reverse tunnel disappeared mid-run on a wireless connection. Every result posted after that was dropped while the run carried on looking healthy. The driver now re-establishes the tunnel on a timer, and the app retries a failed post rather than giving up on the first refusal. - The screen turned off, Android froze the app, and the run stopped dead. The app now holds a keep-awake lock for the duration. Also adds baselines/, where a run worth keeping is committed, with a README covering the naming convention and when to re-record.
Full suite on a Galaxy S26 Ultra (SM-S948B, Android 16), all 12 cases reporting. This is the reference for the 1.3.1 to 1.4.1 bump. The run covers the three paths nothing had exercised before: the privacy filter, Whisper (whose multi-method program gives separate encode and decode numbers from the raw-execute pass, 112 ms and 10.5 ms) and Supertonic, whose streaming generator has no synchronous entry point and is timed on the RN thread instead. Also adds a NOISY verdict to the comparator. Where a metric's own interquartile range is wider than the tolerance, it cannot resolve a regression of the size we care about, and reporting it as "same" overstates what the run knows. YOLO26's pipeline metric is the case in point: its IQR is around 38% of its median, from garbage collection during post-processing, and it moved 57% between two runs of identical code, while its execute.forward number over the same runs sits inside 1%.
Comparing the recorded 1.3.1 baseline against a second run of the same build produced six regressions. All six were false. The measured drift between those runs, by metric family: execute.* 7.6% worst case, most inside 4% pipeline.median 35% load.* 45% memory.* 2% Two causes, both fixed here. Load was timed exactly once per case, and one sample of a load is not a measurement: it is mostly filesystem cache state. Loads are now repeated (three cycles by default, load and dispose) and reported as a median with its spread, which is why the report schema goes to 2. And a single tolerance across every metric cannot work. A pipeline figure carries TypeScript pre- and post-processing and its garbage collection; a raw execute figure carries ExecuTorch and nothing else, which is what a version bump changes. They are now separate knobs, set from the numbers above rather than guessed: execute 10%, pipeline 30%, load 35%. The raw-execute rows also move to the top of each case block, since they are the ones to read.
Two full suites run fifteen seconds apart reported sixteen regressions, including every raw-execute metric, on identical code. The cause was the phone, not the build: all twelve execute metrics were slower in the second run, from 9% to 51%, median 22%. The device never got a chance to cool. That is not something a tolerance can absorb. Widening execute to 50% would hide it and destroy the only signal that reliably tracks ExecuTorch; the earlier well-separated pair had every execute metric inside 7.6%, so the 10% tolerance is right for runs taken cool. So the harness measures the confounder instead. bench-probe reports thermal state (PowerManager on Android with battery temperature, ProcessInfo on iOS), the runner records it per case and at run boundaries, and the comparator refuses outright to diff two runs where either was throttling. The driver grows --cooldown for the wait between runs.
The schema contract moved behind a `schema` namespace and its specs gained a dimension type parameter (#1327), and the task constants are re-exported flat rather than under a `constants` object. Track both so the harness typechecks against the current surface.
These landed on rne-rewrite after the harness was written, so the suite had no numbers for them. All three go in the `full` tier: FastSAM and PP-OCRv6 time their worklet entry point, while Kokoro streams chunks from the RN thread and carries the same caveat as Supertonic. LLM and SDXS text-to-image are still uncovered. Both pull multi-gigabyte artifacts, which changes what a full run costs, so they are worth a deliberate decision rather than being folded in here.
The biggest source of noise on a phone is the clock, not the code: a device boosts early and sags as it heats, which is how two runs of identical code came out 9% to 51% apart. Android exposes PowerManager's fixed-performance mode over `cmd power`, and vendors implement it as a hard frequency cap rather than a hint. On the S26 Ultra it takes every cluster from 3.19/3.40 GHz to about 1.98 GHz. Absolute numbers drop, which is the trade: the same clock in every run is worth more than a fast one. `--pin-clocks` defaults to `auto` (pin where supported), with `on` to require it and `off` to opt out. The driver reads the frequency back rather than trusting the call, since not every vendor implements the HAL, and restores normal clocks on exit, signal and crash alike so a capped device is never left behind. The run records whether it was pinned and the comparator refuses to diff a pinned run against an unpinned one. No iOS equivalent exists; nothing in the public API pins the clock, so runs there still depend on the thermal gate.
A fixed `--cooldown 420` is wrong in both directions: it burns seven minutes on a phone that is already cold, and it is not enough after a heavy suite. `--cooldown auto` polls `dumpsys` until the framework reports no throttling and the battery temperature has stopped falling. It waits on a plateau rather than an absolute threshold, because what counts as cool differs per device while "no longer dropping" does not, and it requires two consecutive settled samples so a flat reading mid-fall does not end the wait early. A 30s floor lets the heat of building and installing dissipate; `--cooldown-max` stops a warm room or a charging phone stalling the run forever. Charging is reported, since it keeps a device warm. On an idle S26 Ultra this releases after 30s where the fixed wait took 420s. Android only: iOS exposes no thermal readout to the host, so `auto` falls back to a fixed sleep there rather than pretending to measure.
The FastSAM case wedged a run. FastSAM pairs a 0.5 confidence threshold with an IoU of 0.9, and NMS at 0.9 suppresses almost nothing, so on a textured synthetic image nearly every candidate survives and each survivor materialises a full 640x640 mask in JS. The worklet thread stalled with no output. RF-DETR nano emits a fixed set of queries and runs NMS at 0.55, so its post-processing is bounded whatever the input looks like. Verified on an S26 Ultra: 215 ms median, 516 MB peak. This is the input-dependence the README already warns about, met head on: a model whose post-processing cost is unbounded in the number of detections does not belong in a suite fed deliberately adversarial synthetic images.
The suite measured 17 hand-written cases against a registry that publishes 261 variants. Extending it by hand does not scale and does not stay correct: a variant added to models.ts and not to the suite is a model that silently never gets benchmarked, which is the failure this harness exists to prevent. So the case list is derived rather than written. generate-variants.mjs evaluates models.ts under Node's type stripping and emits every concrete variant with its backend, precision, platforms and download size; suite.ts joins each to a per-task driver. 163 variants are runnable on Android and 234 on iOS. Adding a model or a variant now needs no change here at all; adding a task needs one driver; a task with no driver is reported as skipped rather than dropped. Each variant is measured three times, and each measurement starts with the device at or below 35C. The gate is an absolute ceiling rather than the plateau rule it replaces: a plateau answers "has it stopped cooling", which is the right question for two runs on one device and the wrong one for four devices, since a phone settling at 41C and one settling at 30C both pass it. The wait lives on the host because Android exposes battery temperature to adb and not to an app; iOS has no readout at all, so it falls back to thermalState plus a fixed settle and records that it did, rather than implying 35C. Repeats are not iterations. Iterations bound the noise inside one measurement; repeats expose the run-to-run spread that thermal state and clock drift produce, which on a phone is the larger of the two. The comparator folds repeats to a median and widens each metric's noise floor by the across-repeat range, so a thermal artefact stops reading as a regression. Measurements are appended to a JSONL as they land and --resume skips what it already holds: a whole-estate run is hours, and a report assembled only at the end loses all of it to a crash on the last case. Models are deleted after a variant's last repeat, so peak disk is one model rather than 119 GB, and a variant over --max-bytes is recorded as skipped with its size instead of failing halfway through its download. Also adds an LLM driver (decode pinned to 64 tokens with EOS ignored, since generation length is a property of the model and not of the runtime), a summarize script for the table people actually read, and BENCHMARK_SPEC.md as the protocol other devices run against.
Fixed-performance mode caps a Galaxy S26 Ultra from 3.19/3.40 GHz to about 1.98 GHz. That is the right trade for detecting a regression, where the same clock in both runs is worth more than a fast one, and the wrong one for publishing device numbers: every figure would understate the phone by roughly the ratio of the clocks and describe a state its governor would never choose. The flag stays, for A/B work against another build. What changes is which way it points when nobody says. With the clock free the thermal gate is the only control left over run-to-run drift, which is what the per-repeat gate is for. Also resolves a contradiction in the spec, which told people to charge the phone for a long run two sections after telling them charging stalls the gate.
Member
Author
Running this on your devicesgit submodule update --init --recursive # phonemis, CMake fails without it
yarn install
cd apps/benchmarks
yarn bench --platform android --suite quick --label v0.10.0 --max-temp-c 37
yarn bench --platform ios --suite quick --label v0.10.0
yarn bench:summary <out>.jsonlSend back the Notes:
|
Repeating a whole measurement was meant to expose the run-to-run spread that thermal state and clock drift produce, on the assumption it was larger than the spread inside one measurement. Measured on a Galaxy S26 Ultra it is not: 16.8% within against 12.1% across on EfficientNet int8, 21.7% against 19.6% on fp32. Twenty back-to-back iterations already show what three cold repeats show, at roughly a third of the wall clock, because each repeat waits at the gate again. The spread that remains belongs to the metric rather than to the sampling. pipeline carries garbage collection in its TypeScript post-processing and sits at 7-22%; the raw execute figure is ExecuTorch alone and sits near 2.5%. No repeat count changes that, which is why the summary now leads with where the time goes rather than with an error bar. --repeats 3 still does the old thing for a model worth pricing precisely. Also reports Model MB (peak minus the baseline taken just before the load) and Execute %, after finding that absolute peak charges a model measured late for the cases before it: every case shares one process, and its baseline crept from 292 MB to 350 MB over seven measurements.
Two ways a run could quietly measure under settings nobody chose. expo run:* attaches to a bundler already listening rather than starting one, and every EXPO_PUBLIC_BENCH_* value is inlined at transform time, so the settings live in that process. A run asked for one repeat under a new label and the app announced three repeats under the previous one. The driver now frees the dev-server port first, and the app echoes the label and repeat count it actually booted with so a mismatch is a hard stop rather than a footnote: a complete set of plausible numbers taken under the wrong configuration is worse than a crash. The device-side gate then ignored the ceiling it was given. It exists for iOS, where nothing exposes a temperature, but it also catches any blip in the adb tunnel — and on Android BenchProbe does report one. A lost /gate POST dropped a measurement onto that path and it started at 36.6C under a 35C gate, with the reading sitting in its own result. It now holds the ceiling wherever a temperature is readable, and keeps the blind settle only where one is not.
A ceiling below the device's idle floor never opens. A Galaxy S26 Ultra sits at 35.4C doing nothing with the harness in the foreground, because the screen is held on for the length of the run or Android freezes the app mid-suite. Under a 35C gate every measurement waited out its full 30 minute timeout and then measured warm regardless, which would have turned a 52 model tier into a day of waiting and made the timedOut flag meaningless by setting it on every row. 37C opens immediately on an idle device and still holds after a model that heated it. It stays an absolute number, since two phones gated differently are not comparable, which is the whole reason for a fixed ceiling over a plateau rule. --resume was also weaker than it read. The app asks the collector which measurements exist and skips them, over the same adb tunnel that has already been seen to drop a POST: a lost answer means everything is measured again and the file quietly grows a second copy of every row. The collector now holds the keys its output file already contains and refuses one it has, so the guarantee belongs to the writer rather than to a request surviving.
Stopping the charge was meant to keep the device under a 35C gate, since the cable adb needs holds it about 1.5C warmer. It worked, and it invalidated the measurements: dumpsys battery unplug convinces the framework the device is on battery, Samsung answers by engaging Battery Saver, and it sticks until the cell reaches 90%. Nothing about it looks like throttling — the CPU's maximum frequencies read normal — but EfficientNet int8 went from 66 ms to 116 ms with nothing else changed. A phone in a power-saving mode no user asked for is not the phone the numbers describe. With the gate at 37C the trade is unnecessary: charging costs 1.5C and the ceiling has room for it, so runs stay plugged in and the battery survives a long suite. --unplug is still there for a device whose idle floor needs it, and now clears low_power rather than leaving it set. The run also refuses to start in Battery Saver rather than trusting that nobody turned it on, because the harness itself turned it on once and the resulting numbers looked entirely plausible.
Member
Author
Android results: SM-S948BRelease build, ET 1.4.1, quick tier, 52 variants. 20 iterations after 3 warmups, gated to 37C before each measurement, clocks free.
|
The suite answers "how fast". This answers "what did the model actually compute on this device", which is a question a host cannot answer for #1406: Core ML fp16 draws shifted masks and boxes on an iPhone's ANE, and a Mac cannot reproduce it because its own ANE compiler rejects the model and Core ML falls back to CPU/GPU without saying so. Every host check therefore passed. The host writes the input as a raw tensor and serves it; the device feeds those bytes to execute untouched and sends the output tensors back as base64 of their own buffers. Neither side decodes an image or rounds through decimal text, so a difference in the output cannot be a difference in preprocessing or in serialisation - which matters when the signal being measured is a fraction of a pixel. iOS needs the ATS exception and a local-network usage string to reach a collector on the LAN at all; Android already had usesCleartextTraffic.
msluszniak
force-pushed
the
@ms/perf-benchmarks
branch
from
September 4, 2026 09:31
df61e4d to
9dc8c16
Compare
msluszniak
changed the title
Two defects made the execute share unusable, and both are fixed here. The share was derived from a standalone replay that sized every tensor at its schema maximum. For a model with a dynamic dimension that is not the work the pipeline did: a text embedder declaring 510 tokens was replayed at 510 while the pipeline ran the ~75 of the benchmark text, so execute exceeded the pipeline and the column was blanked for 21 of 52 variants. OCR was worse, since the pipeline calls `recognize` once per detected box and the replay called it once. `getExecutionProfile()` accumulates time inside `Model::execute` instead, so the figure covers exactly the shapes and call counts the pipeline used, and the share is always defined. The API is new public surface because a task pipeline owns its `Model` privately and callers had no way to ask. The runs were also debug builds. ExecuTorch is a prebuilt release library so `execute` was unaffected, but the library's own C++ compiles unoptimised and JS is served as a dev bundle, inflating everything around the model by roughly an order of magnitude. That does not merely add noise, it inverts the conclusion: EfficientNet measured 29% ExecuTorch in debug and 91% in release. Release is now the default, debug warns, and the build type is recorded in the report. Drops the ET 1.3.1 baseline: schema 1, debug, no thermal gate, and no execution data, so nothing the harness produces can be compared to it.
The spec is the contract other devices follow, and it did not say which build type to use. A debug run does not merely read slow, it inverts the verdict: EfficientNet is 29% ExecuTorch in debug and 91% in release.
…pass Cuts about 4,000 lines from the harness without losing a measurement. `src/variants.generated.ts` was 3,171 checked-in lines produced by a script, plus a CI check to catch it going stale. The registry is a plain nested object once imported and `variants()` only spreads a map, so the list is a walk, not something that needs generating. Everything in the generated file was derivable in-process except the download sizes, which need a network round trip, so those alone stay cached and the generator shrinks to the script that measures them. The replay pass goes with it. It loaded each `.pte` separately and sized tensors at the schema maximum, which is not what the pipeline ran: it is what made the execute share unusable for 21 of 52 variants and produced a 510-token forward to divide by a 75-token pipeline. The in-band profiler measures the real work, so the replay answered no question worth the code. The RF-DETR ANE probe moves out to its own branch. It rides on the collector but it is a device-only correctness investigation for #1406, not part of a benchmark harness.
Both functions wrap a native JSI call, so the repo's convention requires the directive or they cannot be serialized onto a worklet runtime. That is not hypothetical here: the profile is read around a measurement loop that runs inside one, which is also why the C++ side takes a mutex. Records the three new exports in the API surface snapshot.
The execution profile was zeroed before the warmups and then divided by iterations + warmup, so "Execute ms" was a mean over all 23 passes while "Inference ms" was the median of the 20 timed ones. A warmup pays one-off costs the timed median never sees, Vulkan shader compilation above all, so ExecuTorch time came out larger than the pipeline time that contains it on 23 of 52 variants - MiniLM on Vulkan fp16 by 1.47x, which works out to roughly 18 ms per warmup pass. The profile is now zeroed between the warmups and the timed loop, inside the worklet, so the tally covers exactly the iterations the durations cover. That works because resetExecutionProfile carries the 'worklet' directive. summarize.mjs compared that per-iteration mean against the pipeline median, mixing two statistics; it now compares mean to mean. It also clamped the share to 100% and floored JS ms at 0, which is how a physically impossible reading rendered as a tidy "100% / 0.00 ms" and went unnoticed. Both clamps are gone: a share over 100% is a measurement fault and has to be visible. Verified on the first re-run rows: mosaic-int8 101% -> 99%, and every row now sits under 100%.
Member
Author
|
Pushed a0fb58b: the The profile was zeroed before the warmups and divided by Fix: zero the profile between the warmups and the timed loop, inside the worklet.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Adds
apps/benchmarks, a headless Expo app that measures load time, inference latency and peak memory for every published registry variant (163 Android / 234 iOS, generated frommodels.ts), plus a collector, a summarizer and a comparator.Notes:
getExecutionProfile()API, soExecute %reflects the shapes and call counts the pipeline actually used..ptepages.Protocol for other devices:
apps/benchmarks/BENCHMARK_SPEC.md.Introduces a breaking change?
Type of change
Tested on
Testing instructions
Screenshots
Related issues
Closes #1078
Checklist
Additional notes
Not yet run on iOS.