MiniMax-M3 H200 AgentX EAGLE3 tuning on vLLM v0.27.1 - #2565
Conversation
…n-AL synthetic acceptance Add minimaxm3-fp8-h200-vllm-agentic-mtp, the spec-decoding=mtp variant of the MiniMax-M3 MXFP8 H200 agentic-coding recipe: Inferact/MiniMax-M3-EAGLE3 draft head, 3 speculative tokens, FLASH_ATTN drafter, and synthetic acceptance pinned to the committed golden AL 2.83 (minimaxm3_eagle3.yaml, thinking_on, K=3). EVAL_ONLY keeps real target verification. Serve shape, Mooncake offload and search-space arms mirror the non-MTP sibling, trimmed at the top concurrencies. 中文：新增 minimaxm3-fp8-h200-vllm-agentic-mtp，即 MiniMax-M3 MXFP8 H200 智能体 编码配方的投机解码（spec-decoding=mtp）变体：Inferact/MiniMax-M3-EAGLE3 草稿头、 3 个投机 token、drafter 固定为 FLASH_ATTN，合成接受长度固定为黄金 AL 2.83 （minimaxm3_eagle3.yaml，thinking_on，K=3）。EVAL_ONLY 保留真实目标验证。服务参数、 Mooncake KV 卸载与搜索空间沿用非 MTP 版本，仅在高并发端裁剪。 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
cquil11
added
full-sweep-enabled
agentx-fast
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
There was a problem hiding this comment.
Looks good — straightforward addition of a new AgentX benchmark recipe (MiniMax-M3 H200 EAGLE3 MTP), config sweep entry, and changelog line, following the established pattern of the sibling non-MTP recipe.
What was reviewed: the new vllm serve invocation and speculative-decoding config block, the Mooncake KV-offload DRAM budget arithmetic, the model-download/flock-lock resolution logic (verified against the sibling script it mirrors), and the new nvidia-master.yaml sweep matrix and perf-changelog entry. All findings from this run are minor quality nits (duplicated helper code, a hardcoded value that duplicates a variable, and a stale header comment) with no functional or security impact.
Extended reasoning...
Overview
This PR adds a new single-node benchmark recipe (minimaxm3_fp8_h200_mtp.sh) for MiniMax-M3 on H200 with EAGLE3 speculative decoding (MTP), plus a corresponding sweep-matrix entry in configs/nvidia-master.yaml and a changelog entry in perf-changelog.yaml. The new script is derived almost line-for-line from the existing non-MTP sibling minimaxm3_fp8_h200.sh, adding only the speculative-decoding config block, synthetic-acceptance-length handling, and adjusted cudagraph/max-num-seqs sizing for the extra verified tokens per step.
Security risks
None identified. This is benchmark infrastructure — a shell script that downloads public HF models, launches a local vLLM server, and runs a benchmark client. There's no auth, secrets handling, or externally-reachable surface introduced beyond what the sibling script already does (same flock-guarded download pattern, same Mooncake local KV store config).
Level of scrutiny
Low-to-moderate is appropriate. This is a benchmark recipe, not production-serving code — correctness issues here affect benchmark numbers/CI, not a live service. The script closely mirrors an existing, presumably-vetted sibling, which reduces the risk of novel logic errors. The three findings from this run are all nits (code duplication, a hardcoded literal duplicating a variable, a stale comment) rather than functional defects — none affect correctness at the current parameter values.
Other factors
The PR includes its own validation notes (46 generated AgentX points, bash/YAML validation, 231 matrix/schema tests passing). No prior reviews or unaddressed comments exist on this PR's timeline. The changes are additive (new file, new YAML block, new changelog entry) and don't modify any existing recipe, shared library code, or config schema, limiting blast radius.
| resolve_complete_model_snapshot() { | ||
| python3 - "$1" <<'PY' | ||
| import json | ||
| import sys | ||
| from pathlib import Path | ||
|
|
||
| model_cache_dir = Path(sys.argv[1]) | ||
| try: | ||
| revision = model_cache_dir.joinpath("refs/main").read_text().strip() | ||
| except OSError: | ||
| raise SystemExit | ||
|
|
||
| if not revision or Path(revision).name != revision: | ||
| raise SystemExit | ||
|
|
||
| snapshot = model_cache_dir / "snapshots" / revision | ||
| index_path = snapshot / "model.safetensors.index.json" | ||
| required_files = ( | ||
| snapshot / "config.json", | ||
| snapshot / "tokenizer_config.json", | ||
| index_path, | ||
| ) | ||
| if not all(path.is_file() for path in required_files): | ||
| raise SystemExit | ||
| try: | ||
| weight_map = json.loads(index_path.read_text())["weight_map"] | ||
| except (KeyError, json.JSONDecodeError, OSError): | ||
| raise SystemExit | ||
| shards = {snapshot / filename for filename in weight_map.values()} | ||
| if shards and all(path.is_file() for path in shards): |
There was a problem hiding this comment.
🟡 The resolve_complete_model_snapshot helper and the entire MODEL_PATH resolution/download/flock-lock block (lines 31-89) are copied verbatim from benchmarks/single_node/agentic/minimaxm3_fp8_h200.sh:13-71. This is a pre-existing duplication pattern (the sibling script already has this logic inline) rather than something introduced by this PR's own design; consider hoisting it into benchmark_lib.sh, which both scripts already source, so future fixes to the stale-snapshot race or the flock -w 3600 timeout apply to both.
Extended reasoning...
What the duplication is: minimaxm3_fp8_h200_mtp.sh lines 31-89 contain the resolve_complete_model_snapshot() Python heredoc (which validates a cached HF snapshot has a resolved revision, config.json, tokenizer_config.json, and every shard referenced in model.safetensors.index.json) plus the surrounding MODEL_PATH resolution logic: direct-path handling, cache-root computation, the flock-guarded download-and-recheck retry, and the final export MODEL_PATH. This entire ~35-59 line block is byte-for-byte identical to benchmarks/single_node/agentic/minimaxm3_fp8_h200.sh:13-71.
Why it's not caught by existing shared code: Both scripts already source benchmark_lib.sh (see the check_env_vars, resolve_trace_source, install_agentic_deps, wait_for_server_ready calls right next to this block). benchmark_lib.sh has adjacent HF-related helpers but no resolve_complete_model_snapshot or model-download/lock helper — this specific snapshot-completeness + flock-lock logic exists in exactly these two files and nowhere else in the repo.
Concrete consequence — step-by-step:
- Today,
flock -w 3600(a 1-hour download-lock timeout) and the snapshot-completeness checks (revision resolution, required-file list, shard-set-from-index-map validation) are identical in both files. - Suppose a maintainer later discovers the stale-snapshot race needs a fix — e.g. the completeness check should also verify shard file sizes, or the lock timeout is too short for a 700GB+ MiniMax-M3 checkpoint on a slow filesystem.
- They fix it in
minimaxm3_fp8_h200.sh(the file they happen to be debugging) because that's the file in front of them. minimaxm3_fp8_h200_mtp.shsilently keeps the old, buggy logic — there's no compiler error, no test failure, and no obvious signal that a second copy exists, since the two files aren't otherwise linked.- The MTP variant now has a latent, already-known-to-be-wrong download race that will resurface on the next cold cache / concurrent job collision, and the fix has to be rediscovered and reapplied by hand.
Why the current approach doesn't prevent this: Nothing forces the two scripts to be updated together; they're independent shell scripts, and the shared benchmark_lib.sh — the natural place for exactly this kind of logic — doesn't have it, so there's no single source of truth to inherit a fix from.
Suggested fix: Extract resolve_complete_model_snapshot() and the MODEL_PATH resolution/download/lock block into benchmark_lib.sh (e.g. as resolve_or_download_model_path MODEL MODEL_CACHE_ROOT), and have both minimaxm3_fp8_h200.sh and minimaxm3_fp8_h200_mtp.sh call it. This is a quality/reuse cleanup, not a functional defect in the current PR — the duplicated code itself is correct — so it doesn't block merging, but it does create maintenance risk that grows every time a third MiniMax-M3 recipe is added with the same pattern.
| MOONCAKE_LOCAL_BUFFER_GIB=4 | ||
| if require_agentic_kv_offload_backend mooncake; then | ||
| TOTAL_CPU_DRAM_GIB=$((TOTAL_CPU_DRAM_GB * 1000000000 / 1073741824)) | ||
| PER_RANK_GB=$(((TOTAL_CPU_DRAM_GIB - MODEL_CHECKPOINT_PAGE_CACHE_GIB) / TP - MOONCAKE_LOCAL_BUFFER_GIB)) | ||
| if (( PER_RANK_GB <= 0 )); then | ||
| echo "Error: CPU DRAM budget is too small for checkpoint cache and KV offload" >&2 | ||
| exit 1 | ||
| fi | ||
| MOONCAKE_VERSION=0.3.11.post1 | ||
| agentic_pip_install --quiet --no-cache-dir --no-deps \ | ||
| --force-reinstall "mooncake-transfer-engine-cuda13==$MOONCAKE_VERSION" | ||
| python3 -c "from mooncake.store import MooncakeDistributedStore" >/dev/null | ||
| MOONCAKE_MASTER_PORT=$((PORT + 12000)) | ||
| MOONCAKE_CONFIG_PATH="$RESULT_DIR/mooncake_config.json" | ||
| cat > "$MOONCAKE_CONFIG_PATH" <<EOF | ||
| { | ||
| "mode": "embedded", | ||
| "metadata_server": "P2PHANDSHAKE", | ||
| "master_server_address": "127.0.0.1:$MOONCAKE_MASTER_PORT", | ||
| "global_segment_size": "${PER_RANK_GB}GB", | ||
| "local_buffer_size": "4GB", |
There was a problem hiding this comment.
🟡 Line 139 hardcodes "local_buffer_size": "4GB", duplicating the value of MOONCAKE_LOCAL_BUFFER_GIB (line 119) instead of interpolating it as line 138 does for PER_RANK_GB. The sibling recipe minimaxm3_fp8_h100.sh:58 correctly writes "local_buffer_size": "${MOONCAKE_LOCAL_BUFFER_GIB}GB" — apply the same interpolation here so a future change to the variable can't silently desync from the DRAM-budget guard.
Extended reasoning...
MOONCAKE_LOCAL_BUFFER_GIB is defined at line 119 (MOONCAKE_LOCAL_BUFFER_GIB=4) and used at line 122 to subtract the Mooncake local buffer reservation out of the per-rank DRAM budget: PER_RANK_GB=$(((TOTAL_CPU_DRAM_GIB - MODEL_CHECKPOINT_PAGE_CACHE_GIB) / TP - MOONCAKE_LOCAL_BUFFER_GIB)). That computed PER_RANK_GB is then written into the generated mooncake_config.json heredoc at line 138 as "global_segment_size": "${PER_RANK_GB}GB" — correctly interpolated. But the very next line, 139, hardcodes the local buffer size as a literal: "local_buffer_size": "4GB" instead of "local_buffer_size": "${MOONCAKE_LOCAL_BUFFER_GIB}GB".
The bug is a DRY violation: the same logical quantity (the reserved local buffer size) is expressed twice — once as the variable used in the capacity-guard arithmetic, and once as an independently-maintained literal in the generated config. Today both are 4, so there's no functional divergence at HEAD. But the two are no longer mechanically tied together: if a future maintainer bumps MOONCAKE_LOCAL_BUFFER_GIB (e.g. to accommodate a larger buffer for a different SKU or workload), the PER_RANK_GB budget check at line 122 will use the new value, while the actual Mooncake store config at line 139 will keep reserving the stale hardcoded 4GB. That silently desyncs the value the budget guard assumes is reserved from the value Mooncake is actually told to reserve — exactly the kind of gap that either wastes DRAM (if the literal ends up smaller than intended) or produces a config that's inconsistent with the capacity check that was supposed to prevent OOM (if the literal ends up smaller than what the guard subtracted, leaving more headroom than expected — or vice versa, larger than what was budgeted, defeating the guard).
Nothing in the surrounding code catches this: set -euo pipefail and the PER_RANK_GB <= 0 check only validate the arithmetic result, not that the heredoc literal matches the variable it was derived alongside. The heredoc is a plain bash cat > file <<EOF, so ${MOONCAKE_LOCAL_BUFFER_GIB} would interpolate exactly the same way ${PER_RANK_GB} does on the line above — there's no technical barrier, just an inconsistency introduced when the script was authored.
This is proven by direct comparison with the sibling recipe benchmarks/single_node/agentic/minimaxm3_fp8_h100.sh, which defines the identical MOONCAKE_LOCAL_BUFFER_GIB=4 pattern and at line 58 correctly writes "local_buffer_size": "${MOONCAKE_LOCAL_BUFFER_GIB}GB". That sibling demonstrates the intended, safer pattern already exists in this repo; this new h200-MTP recipe (and its non-MTP h200 sibling at line 102, which has the same hardcoded literal) simply didn't carry it over.
Step-by-step reproduction of the risk: (1) today MOONCAKE_LOCAL_BUFFER_GIB=4, so PER_RANK_GB is computed by subtracting 4, and local_buffer_size in the JSON is also 4GB — consistent by coincidence of both being hand-set to 4. (2) Suppose a future PR changes line 119 to MOONCAKE_LOCAL_BUFFER_GIB=8 because Mooncake's actual buffer needs grew. (3) PER_RANK_GB at line 122 now correctly subtracts 8GB from the budget, leaving less global_segment_size. (4) But line 139 still emits "local_buffer_size": "4GB" — Mooncake is configured with a 4GB buffer while the script's own capacity guard assumed 8GB was reserved. The actual runtime behavior and the budget check are now based on different numbers, and nothing fails loudly — it just produces a subtly wrong DRAM allocation.
Fix: change line 139 from "local_buffer_size": "4GB" to "local_buffer_size": "${MOONCAKE_LOCAL_BUFFER_GIB}GB", matching both the ${PER_RANK_GB}GB interpolation immediately above it and the h100 sibling's pattern. This is a pure quality/consistency nit — the values agree today so there is no active runtime bug — but it removes a latent trap for the next person who touches this variable.
| # Everything outside the speculative block mirrors the non-MTP agentic sibling | ||
| # (Mooncake host-DRAM KV offload, --block-size 128, --language-model-only, | ||
| # --kv-cache-dtype fp8, TRITON_ATTN, gmu 0.92, minimax_m3 parsers, vllm-router | ||
| # for DP-attention), so the spec-decode delta is readable at equal concurrency. |
There was a problem hiding this comment.
🟡 The header comment (lines 7-10) states this recipe mirrors the non-MTP sibling at gmu 0.92 specifically so the spec-decode delta is readable at equal concurrency, but line 218 actually sets --gpu-memory-utilization 0.90 (the sibling minimaxm3_fp8_h200.sh:145 does use 0.92). Either the comment is stale and should explain the 2% reduction (likely HBM headroom for the EAGLE3 draft head), or the value should be changed to 0.92 to match the stated invariant.
Extended reasoning...
The bug: The header comment block at the top of minimaxm3_fp8_h200_mtp.sh (lines 7-10) explicitly enumerates the settings this MTP recipe is supposed to mirror from its non-MTP sibling, agentic/minimaxm3_fp8_h200.sh: Mooncake host-DRAM KV offload, --block-size 128, --language-model-only, --kv-cache-dtype fp8, TRITON_ATTN, gmu 0.92, the minimax_m3 parsers, and vllm-router for DP-attention. The comment then states the purpose of this mirroring: so that 'the spec-decode delta is readable at equal concurrency' — i.e. the intent is to hold every non-speculative-decoding variable constant relative to the sibling, isolating the EAGLE3 contribution as the only difference between the two benchmark curves.\n\nThe code path that contradicts it: Line 218 of the same file sets --gpu-memory-utilization 0.90 in the actual vllm serve invocation. I verified the sibling script independently — benchmarks/single_node/agentic/minimaxm3_fp8_h200.sh:145 sets --gpu-memory-utilization 0.92. So the comment's claim that gmu is held at 0.92 to match the sibling is factually false for the script as written; the actual value differs by 2 percentage points of GPU memory.\n\nWhy nothing catches this: There's no test or lint that cross-checks a recipe's header comment against its own flag values, and the comment reads as authoritative documentation of the experimental design (which knobs are controlled vs. varied). A maintainer or reviewer reading only the header would reasonably conclude gmu is not a confound between the two curves, when in fact it is.\n\nImpact: This isn't a runtime bug — the script runs fine at 0.90, and 0.90 is very plausibly the correct value (leaving extra HBM headroom for the EAGLE3 draft head's KV cache and weights, which the sibling doesn't need). But the documented invariant of the recipe — 'only the speculative-decoding block differs; everything else is held equal so the delta is attributable to spec decode alone' — is violated by a variable the comment claims is not different. Concretely: at equal concurrency, the MTP run has 2% less KV-cache capacity than the sibling for a reason unrelated to speculative decoding. This slightly reduces max batched sequences / KV-cache headroom, which could itself shift throughput or latency at high concurrency independent of the EAGLE3 draft's effect — exactly the kind of confound the comment claims doesn't exist. Future maintainers relying on this comment to reason about what varies between the two curves will be misled.\n\nProof walkthrough:\n1. Read minimaxm3_fp8_h200_mtp.sh lines 7-10: comment lists 'gmu 0.92' as mirrored from the sibling, framed as necessary so the spec-decode delta is readable at equal concurrency.\n2. Read minimaxm3_fp8_h200_mtp.sh line 218: --gpu-memory-utilization 0.90.\n3. Read minimaxm3_fp8_h200.sh line 145 (the sibling referenced by the comment): --gpu-memory-utilization 0.92.\n4. 0.90 ≠ 0.92 — the comment's factual claim about what's held constant between the two recipes is wrong.\n\nFix: Either bump line 218 to --gpu-memory-utilization 0.92 if there's no real need for the reduction, or (more likely correct, given the EAGLE3 draft's extra memory footprint) update the header comment to say 'gmu 0.90 (2% lower than the sibling's 0.92 to leave HBM headroom for the EAGLE3 draft head)' so the documented invariant matches reality and doesn't mislead readers about what's actually held constant across the two curves.
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=31540120459 |
在 H200 MiniMax-M3 的拐点附近测试 vLLM SimpleCPUOffloadConnector。
…agentx-v27 # Conflicts: # perf-changelog.yaml
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=31558467483 |
将 vLLM SimpleCPUOffloadConnector 配置接入实际的 MiniMax-M3 MTP 运行时，并保留检查点缓存的主机内存预算。
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=31558981228 |
保留 TP8 常驻 c1/c2/c4/c6/c8/c10 和 Mooncake c12/c14，依据完整快速扫描形成严格全量扫点。
cquil11
added
full-sweep-enabled
and removed
full-sweep-enabled
agentx-fast
2 participants
Summary
Adds a current MiniMax-M3 MXFP8 AgentX EAGLE3-GQA submission on H200 using stable vLLM v0.27.1, strict vLLM metric validation, and a broad
agentx-fastdiscovery sweep.The fast results will prune dominated and post-cliff points before an exact-head full-duration Pareto sweep.
Supersedes the stale/conflicting experiment in #2424.
Initial search
Validation