feat(serve): make BenchmarkResult sweep-aware for concurrency search runs#6098
Open
smouaa wants to merge 1 commit into
Open
feat(serve): make BenchmarkResult sweep-aware for concurrency search runs#6098smouaa wants to merge 1 commit into
smouaa wants to merge 1 commit into
Conversation
…runs A concurrency search / magic-list sweep (--search-recipe or --concurrency 1,10,100) does not produce a single top-level profile_export_aiperf.json. The benchmark container instead ships search_history.json at the artifact root plus per-trial profile exports nested in subdirs. The existing reader matched profile_export_aiperf.json by filename suffix, so for a sweep it would either fail (no such root file) or silently return one arbitrary level's per-trial metrics as if they were the whole benchmark. BenchmarkResult.from_s3 now checks for search_history.json first and, when present, parses the sweep outcome into a new BenchmarkSearchResult (winning swept value from boundary_summary.feasible_max, the first constraint breach, and the raw history). The single-run path is unchanged. Adds BenchmarkSearchResult (exported), a BenchmarkResult.search field + is_search property, and unit tests covering the sweep layout, the per-trial-name collision, null boundary summaries, and the single-run regression guard.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Issue / motivation
A concurrency search (--search-recipe) or magic-list sweep (--concurrency 1,10,100) benchmark does not produce a single top-level profile_export_aiperf.json. The benchmark container instead writes search_history.json at the artifact root plus per-trial profile exports nested in subdirectories.
BenchmarkResult.from_s3 matched profile_export_aiperf.json by filename suffix, so for a sweep it would either raise FileNotFoundError (no such root file) or — worse — silently return one arbitrary swept level's per-trial metrics as if they were the whole benchmark, since those per-trial files share the same name.
Changes
BenchmarkResult.from_s3 now checks for search_history.json first; when present it parses the sweep outcome instead of a headline profile.
New exported BenchmarkSearchResult dataclass: winning swept value (boundary_summary.feasible_max.value), the first constraint breach, and the raw history.
BenchmarkResult.search field + is_search property; str renders the sweep summary for search runs.
Single-run path is unchanged (regression-guarded by a test).
Testing
New unit tests: sweep layout parsing, the per-trial-filename collision, no-SLA-breach and null/absent boundary_summary edge cases, single-run regression guard, and str rendering.
pytest tests/unit/test_ai_inference_recommender/ → 154 passed; flake8 clean.