Run fullsize bit-exact comparisons eager - #129
Open
ItzikVa wants to merge 1 commit into
Open
Conversation
tests/vllm/_granite4_fullsize_tests.py::test_logits_match[4.0-micro] fails on
the vllm20 CI leg and passes on vllm19, and has done so on unrelated PRs -- 117
(2f919be) is a docs-and-comments-only commit, which cannot move a logit. So the
assertion is measuring something other than the code under test.
What it measures is compilation. The test asserts atol=rtol=0 between two
separately launched vLLM engines: upstream GraniteMoeHybrid and GraniteSwitch
with num_adapters=0. Those are different graphs, so torch.compile and inductor
are free to fuse and tile them differently, and at production dimensions that
is enough to round one bf16 logit's last bit apart. vLLM logs its own per-engine
kernel choice: the failing engine reported IrOpPriorityConfig(rms_norm=
['native']) where its sibling reported ['vllm_c', 'native'] -- a different
RMSNorm kernel in 80 places.
The attribution rests on what does pass. 4.0-1b and 4.0-350m have the same 40
dense layers and are bit-exact on both legs; the same comparison on the HF
backend is bit-exact for all three; and real granite-4.0-micro weights are
bit-exact through vLLM in test_skinning_equivalence.py. A weight-transfer or
skinning bug would not spare all of those.
So run these eager. Bit-exactness is a property of the module tree, and eager is
where the module tree is what executes -- enforce_eager takes the compiler, the
CUDA-graph capture and vLLM's on-disk compile cache out of the comparison and
leaves the gate at atol=rtol=0 rather than loosening it. There was already
precedent for the flag in _granite4_mini_tests.py's _UPSTREAM_EAGER_CONFIGS. The
compiled path keeps the coverage it already had, from the two tests that are
deliberately tolerant of a flipped last bit: test_skinning_equivalence.py on
real weights and test_generation_equivalence.py on distributions.
Two supporting changes, because the failure was also unreadable:
* The bit-exact branch now calls this repo's assert_close instead of
torch.testing.assert_close. A msg= string handed to the latter *replaces*
its whole diagnostic, which is why four CI runs recorded "logprobs should be
bit-exact" and not one number -- there was no way to tell a last-bit drift
from a real regression without re-running 4 GPU-hours. assert_close reports
the count, the worst diff and its magnitude.
* assert_close now treats matching non-finite entries as equal. Logprob
tensors are padded with -inf for positions vLLM did not return, and
-inf - -inf is nan, which fails every comparison including diff <= 0.0. That
trap was latent while only the tolerance branch used the helper; the
bit-exact branch would have hit it as a bogus worst=nan.
The margin is printed on a pass too, so a gate that has quietly started riding
at one ULP is visible before it trips.
Verified on CPU: 35 tests in tests/hf/test_granite4_mini.py and
test_granitemoe_mini.py (the existing assert_close callers) still pass, the three
TestGranite4FullSizeWeightTransfer cases pass, and the five assert_close
semantics above were checked directly. The GPU legs are the real gate.
Signed-off-by: itzikvaisman <yizhak.vaisman@ibm.com>
Collaborator
Author
|
/gpu-test-multi |
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
✅ GPU tests passed —
|
✅ GPU tests passed —
|
ItzikVa
marked this pull request as ready for review
September 9, 2026 12:09
ItzikVa
requested review from
antonpibm,
aviv1ron1,
freunda and
yairallouche
as code owners
September 9, 2026 12:09
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
it, IrOpPriorityConfig(rms_norm=['vllm_c','native']) vs ['native'] gives a different rms_norm kernel on each side and the "bit-exact" case is not a statement about our code.
assert_close keeps counts and worst-diff numbers.
including diff <= 0.0.