ci: add manual workflow to benchmark LTO's impact per platform#74
Merged
Conversation
Adds .github/workflows/benchmark-lto.yml (workflow_dispatch only - JMH runs are too slow for every push/PR, and this is throughput exploration, not a correctness gate). For each of linux-x86_64, linux-aarch64, osx-aarch64, windows-x86_64: checks out main and an LTO variant branch (default experiment/lto-linux, overridable via the lto_ref input) side by side, builds+runs the CompressBenchmark/DecompressBenchmark suite (quick smoke settings: 1 fork, 2 warmup, 5 measurement, size=65536) against each, and writes a comparison table to the job summary via the new .github/scripts/compare-benchmarks.py. This turns the "-flto is Linux-only, no-op elsewhere" claim from #70/ #72 into CI evidence per platform instead of a manual assertion - linux-x86_64/linux-aarch64 should show a real delta, osx-aarch64/ windows-x86_64 should show ~0% (LTO_FLAG stays empty for those classifiers in scripts/build-zstd.sh). Verified locally: - actionlint clean - compare-benchmarks.py tested against real JMH JSON output; found and fixed a real bug along the way - scoreError comes through as the JSON string "NaN" (not a number) when JMH can't compute a confidence interval, e.g. with few iterations
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Follow-up to #72 (LTO experiment, Linux-only): a manual
workflow_dispatchworkflow that turns the "LTO is a no-op on macOS/Windows, real on Linux" claim into CI evidence per platform rather than a manual assertion.For each of
linux-x86_64,linux-aarch64,osx-aarch64,windows-x86_64:mainand an LTO variant branch (defaults toexperiment/lto-linux, overridable via thelto_refinput) side by side into separate directories.CompressBenchmark/DecompressBenchmark(quick smoke settings — 1 fork, 2 warmup, 5 measurement,size=65536) against each..github/scripts/compare-benchmarks.py.Not wired into
ci.yml— JMH runs are too slow for every push/PR, and this is throughput exploration, not a correctness gate.Expected outcome once run:
linux-x86_64/linux-aarch64show a real delta (matching the local +6.9%/+12-14% ballpark we measured for the-mcpuflag, though this workflow isolates LTO specifically),osx-aarch64/windows-x86_64show ~0% (sinceLTO_FLAGstays empty for those classifiers inscripts/build-zstd.sh).Test plan
actionlintclean on the new workflowcompare-benchmarks.pytested against real JMH JSON output (not just syntax-checked) — found and fixed a real bug:scoreErrorcomes through as the JSON string"NaN"(not a number) when JMH can't compute a confidence interval (e.g. too few iterations); fixed with an explicitfloat()castworkflow_dispatch) once merged — this PR doesn't include an actual run's output since it's manually triggered🤖 Generated with Claude Code