Where it lives: test isolation, observed on feature/granite-turboctc-default (ef43e69); the mechanism is not branch-specific.
A vLLM engine from an earlier test is still holding GPU memory when later tests start:
Process 26825 has 56.03 GiB memory in use
ValueError: Free memory on device cuda:0 (22.27/79.25 GiB) on startup is less than
desired GPU memory utilization (0.7, 55.48 GiB)
torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 6.13 GiB
Impact: 8 failures + 8 errors —
tests/integration/test_switch_e2e_compose.py — 7x test_hf_composed_adapter_indices[granite-4.1-3b-*] plus test_hf_vllm_argmax_equivalencetests/integration/test_audio_serving_smoke.py[granite-4.1-3b] — 5tests/integration/test_audio_uncached_processor.py — 3
Likely downstream of #121: the audio-smoke engines die mid-request when the leaked Mock raises, and the LLM object is never torn down. I have not pinned pid 26825 to a specific test, so that link is inference.
Fix: guarantee engine teardown between GPU tests — an explicit del llm + torch.cuda.empty_cache() in the fixture's finaliser is the minimum; better, assert free GPU memory has returned before the next engine starts, so a leak fails loudly at its source instead of as an OOM three tests later.
Whole-suite runs are where this bites; the tests pass in isolation.
Where it lives: test isolation, observed on
feature/granite-turboctc-default(ef43e69); the mechanism is not branch-specific.A vLLM engine from an earlier test is still holding GPU memory when later tests start:
Impact: 8 failures + 8 errors —
tests/integration/test_switch_e2e_compose.py— 7xtest_hf_composed_adapter_indices[granite-4.1-3b-*]plustest_hf_vllm_argmax_equivalencetests/integration/test_audio_serving_smoke.py[granite-4.1-3b]— 5tests/integration/test_audio_uncached_processor.py— 3Likely downstream of #121: the audio-smoke engines die mid-request when the leaked Mock raises, and the
LLMobject is never torn down. I have not pinned pid 26825 to a specific test, so that link is inference.Fix: guarantee engine teardown between GPU tests — an explicit
del llm+torch.cuda.empty_cache()in the fixture's finaliser is the minimum; better, assert free GPU memory has returned before the next engine starts, so a leak fails loudly at its source instead of as an OOM three tests later.Whole-suite runs are where this bites; the tests pass in isolation.