Where it lives: not a known defect — an unexplained measurement that drives hardware choices. Found while sizing pods for feature/granite-turboctc-default (ef43e69).
A composed 4.2-3b engine reports:
Model loading took 12.81 GiB
Graph capturing took 0.64 GiB
Available KV cache 52.29 GiB (668,624 tokens)
against 6.85 GiB of bf16 weights — 3,675,312,640 params x 2 bytes, which matches the 6.86 GiB on disk exactly. That is a 5.96 GiB gap, a ratio of 1.87x.
Ruled out
- dtype. The engine logged
dtype=torch.bfloat16. Not a float32 fallback (that would be ~13.7 GiB and is tracked separately in the torch_dtype: null issue).
- Adapter duplication.
BUILD.md records base_param_count 3,659,737,600 -> composed_param_count 3,675,312,640, a +0.43% delta — 15.6M params for one rank-8 adapter, the switch layer and two vocab rows. A second backbone would be +100%. load_weights is a name-matched direct load per tensor (decoder/interface.py:196); the MoE fan-out branch never fires on a dense config.
- A dead process's allocator.
transfer_base_weights is composer-only and runs in a separate process that exits before the engine starts.
Leading hypothesis
The torch caching allocator's high-water mark during weight loading, retained as reserved-but-free. vLLM's "Model loading took" is a reserved-memory delta and torch does not return reserved blocks to the driver, so a per-shard staging buffer that is logically freed still counts against the KV budget. The largest shard is 4.63 GiB and the gap is 5.96 GiB — that plus fragmentation.
How to test
- Set
PYTORCH_ALLOC_CONF=expandable_segments:True and re-read the figure.
- Compare against the 8b (16.43 GiB on disk, 4 shards, largest 4.57 GiB): if its gap is also ~one-largest-shard rather than proportional to total weights, it is the staging buffer; if it scales with total size, something structural is being missed.
Why it matters
It is the difference between a 24 GB L4 and a 48 GB L40S for the 3b demo stack. Measured fixed floor including the ASR encoder (1.64 GiB) and a TTS model (1.44 GiB) was ~18.1 GiB before any KV; recovering most of 5.96 GiB brings that to ~12 GiB.
Where it lives: not a known defect — an unexplained measurement that drives hardware choices. Found while sizing pods for
feature/granite-turboctc-default(ef43e69).A composed 4.2-3b engine reports:
against 6.85 GiB of bf16 weights — 3,675,312,640 params x 2 bytes, which matches the 6.86 GiB on disk exactly. That is a 5.96 GiB gap, a ratio of 1.87x.
Ruled out
dtype=torch.bfloat16. Not a float32 fallback (that would be ~13.7 GiB and is tracked separately in thetorch_dtype: nullissue).BUILD.mdrecordsbase_param_count 3,659,737,600 -> composed_param_count 3,675,312,640, a +0.43% delta — 15.6M params for one rank-8 adapter, the switch layer and two vocab rows. A second backbone would be +100%.load_weightsis a name-matched direct load per tensor (decoder/interface.py:196); the MoE fan-out branch never fires on a dense config.transfer_base_weightsis composer-only and runs in a separate process that exits before the engine starts.Leading hypothesis
The torch caching allocator's high-water mark during weight loading, retained as reserved-but-free. vLLM's "Model loading took" is a reserved-memory delta and torch does not return reserved blocks to the driver, so a per-shard staging buffer that is logically freed still counts against the KV budget. The largest shard is 4.63 GiB and the gap is 5.96 GiB — that plus fragmentation.
How to test
PYTORCH_ALLOC_CONF=expandable_segments:Trueand re-read the figure.Why it matters
It is the difference between a 24 GB L4 and a 48 GB L40S for the 3b demo stack. Measured fixed floor including the ASR encoder (1.64 GiB) and a TTS model (1.44 GiB) was ~18.1 GiB before any KV; recovering most of 5.96 GiB brings that to ~12 GiB.