[docs] Benchmark suite and ruleset proposal - #468
Conversation
Signed-off-by: Rashid Kaleem <230885705+arekay-nv@users.noreply.github.com>
|
MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅ |
github-actions
Bot
added
the
size/normal
arav-agarwal2
left a comment
arav-agarwal2
left a comment
There was a problem hiding this comment.
I don't think I can approve it just yet as it's a proposal, but my main worry is that I don't see a policies PR suggesting the changes to the submission directory and submission checker rules for this system.
| - Pre-processing - executed to generate the complete configuration files, as well as perform any sanity checks (for instance current repo matches the SHA for the cohort tag ). | ||
| - Performance - measured execution of the performance dataset with the user specified concurrency. | ||
| - Accuracy - correctness checks performed over the accuracy dataset and scored by the designated scorer and aggregated to component scores. Each accuracy dataset can be run multiple times (repeats) and there can be different ways of aggregating the scores from different runs (best of K or average over K). As there can be multiple accuracy datasets, we can also specify aggregation across accuracy datasets (unweighted average), or alternatively leave the scores per dataset separate. | ||
| - Post-processing - performs integrity checks on the datasets (SHA verification), number of samples issued, model checkpoints. It will also ensure that the accuracy scores satisfies the gates specified in the benchmark specification. There can also be requirements on the performance phase such as minimum duration, minimum number of samples issued, time-to-first-token constraints etc. |
There was a problem hiding this comment.
The way I'm reading this, the document intends for the load generator to have a ton of submission-related checks.
Who is implementing these checks, and what's the goal of the submission checker given them? Is our goal to verify that these checks were made by the loadgen checker?
There was a problem hiding this comment.
We already have some post-processing steps in the existing setup (steady state checks) and some expected to be added (steady state). The goal is to ensure that a submission is as close to compliance as possible and there are no issues that are overlooked (incorrect checkpoint, insufficient number of samples etc).
Submission checker would independently make those checks based on the benchmark specification which is more reliable as it is done on MLC side compared to the loadgen side checks which can be bypassed.
| type: submission | ||
| submission_ref: | ||
| cohort: 2026-09-C0 # frozen selection (YYYY-MM-C0/C1, submission rules §4.2) | ||
| seed_set_id: B |
There was a problem hiding this comment.
Note: Target cohort should completely specify a seed set, so I'm unsure what seed set format you're expecting here. Also, what does "A" and "B" mean here? I assume it means the earlier and later available seed sets for a given submission, but it pays to be precise.
There was a problem hiding this comment.
There is a cohort definiton example below that includes seeds
seed_sets: # a submission selects exactly one, by id
- id: A
scheduler_rng_seed: 16159082839903944936
sample_index_rng_seed: 2747215439041700203
model_seed: 42
- id: B
scheduler_rng_seed: <generated at ruleset publication>
sample_index_rng_seed: <generated at ruleset publication>
model_seed: <generated at ruleset publication>
|
|
||
| ``` | ||
| benchmark_suite/ | ||
| ├── suite.yaml version; the loader globs the three directories below |
There was a problem hiding this comment.
I"m assuming this is all of the possible configuration for loadgen, which would be specified by the cohort-id.
Where would these details be stored? Do we know who has the responsibility for making sure that this is accurate?
There was a problem hiding this comment.
These are independent of cohort. A cohort only defines which of these is valid for a submission into that cohort.
A task force will work independently to update the model/dataset/benchmark specs for its benchmark, and then merge those changes in. Then propose to open it up for submission at which point it will be added to the next cohort.
| ``` | ||
| benchmark_suite/ | ||
| ├── suite.yaml version; the loader globs the three directories below | ||
| ├── models/<id>.yaml id, reference_id, tokenizer?, processor?, chat_template?, |
There was a problem hiding this comment.
What does refer to here? Is this a model-specific id and, if-so, where would that list be defined?
There was a problem hiding this comment.
Yes, this would be a model specific id. Note that this is the list. We would not need to maintain a separate document as the models specified here are what can be used in a benchmark, which is what can be specified in a cohort for submission.
| ├── suite.yaml version; the loader globs the three directories below | ||
| ├── models/<id>.yaml id, reference_id, tokenizer?, processor?, chat_template?, | ||
| │ metadata{} | ||
| ├── datasets/<id>.yaml id, description, samples, source{url, sha256}, |
There was a problem hiding this comment.
Same as above. Datasets defined here are what can be used in a benchmark and then in a cohort.
| ├── datasets/<id>.yaml id, description, samples, source{url, sha256}, | ||
| │ prompts{}, ground_truth_or_tests{}, evaluation_recipes{}?, | ||
| │ metadata{} | ||
| └── benchmarks/<id>.yaml id, family, model, api_type, |
There was a problem hiding this comment.
This is one level below a cohort. A cohort will define the set of benchmarks for submission.
| postprocessing: | ||
| gates: | ||
| [ | ||
| dataset_integrity, |
There was a problem hiding this comment.
If this needs to be verified by the submission checker after running the benchmark, these files should be present during submission. Is there a PR in the rules repo for getting these into the submission file structure? I couldn't find any.
There was a problem hiding this comment.
Do you mean dataset_integrity and peers? These are passes that will need to be implemented. But this is similar in structure to what we have for scorers/extractors for accuracy datasets.
| 2. The front end will perform initial validation on the submission metadata to check valid cohort, benchmark membership, and verify repository versions match the remote tags for that cohort. | ||
| 3. Once validated, the specification is inlined into the yaml file to reveal an expanded configuration file which specifies exactly what needs to be executed. This ensures that the benchmark execution is correct by construction. The generated yaml file will be saved as part of the output artifacts. | ||
| 4. The generated yaml is passed on to the execution phases which will run the phases in order and produce associated artifacts. For instance, the dataloader will produce an implementation specific SHA for the data loaded which will be logged. | ||
| 5. Towards the end of the execution of the benchmark phases, post-processing step will apply the checks needed to verify submission integrity: |
There was a problem hiding this comment.
Is post-processing something that is done by the submission checker as created by MLC, or is this done by loadgen? If the former, we need rules added to the rules repo so everyone knows what checks are being done on submissions.
|
Is this intended for 1.0? If so, this needs to be far more detailed in terms of checks and overall schema for each of the files, and there needs to be accompanying rules PRs that another organization agrees with. |
Yes, this is intended for 1.0. But this is purely an implementation change, nothing from the rules needs to change. The goal is to 1) centralize all the benchmark specification - right now we do not have a single source of truth for benchmarks in a given round/cohort 2) ensure that we can capture all aspects of what the existing rules specify to be automated and 3) enable us to add new benchmarks rapidly by updating the specification. There are two sides to the verification of a submission - client side (performed before submission) and MLC side (after submission). Both should use the same yaml configs but should be independent so that we can more robust and identify any disambiguation. This change is designed not have any rules PR and will be an implementation only change unless we find gaps in the specification and we can then file issues/PRs for rules. |
This shouldn't affect the submission directory structure or the submission checker rules. Those should remain the same. |
2 participants
What does this PR do?
Adds a proposed Benchmark Suite spec: a git-versioned catalog of models, datasets, and per-benchmark contracts, with a cohort ruleset that freezes the valid benchmark IDs, suite version, and seed sets. A submission names only cohort, seed set, and benchmark (plus submitter-owned concurrency/endpoints); the client inlines the suite contract and the checker validates the run against it. Worked YAML examples cover DeepSeek-R1 and gpt-oss-120b (including multi-dataset accuracy, generation overrides, and gates). No code changes — this is the design source of truth for a follow-on schema/POC.
Type of change
Related issues
Testing
Checklist