Skip to content

vllm: HasInnerState is declared but unhonoured, and excludes the model from the V2 model runner #125

Description

@aviv1ron1

Where it lives: main. Found while working on feature/granite-turboctc-default (ef43e69).

GraniteSwitchForCausalLM lists HasInnerState among its bases and implements none of it. Same inherited-marker mistake as IsHybrid (dropped on the TurboCTC branch): it arrived from GraniteMoeHybridConfig rather than by design.

Unlike IsHybrid, this one is not merely cosmetic. On vLLM 0.20.2:

# vllm/config/vllm.py:1797-1802
def _validate_v2_model_runner(self) -> None:
    """Check for features not yet supported by the V2 model runner."""
    unsupported: list[str] = []
    if self.model_config is not None and self.model_config.has_inner_state:
        unsupported.append("hybrid/mamba models")

So declaring it excludes the model from vLLM's V2 model runner — we are opted out of a code path on the strength of a claim that is not true. The check is gone from that file in 0.26.0+, so the cost is specific to the version line we are on.

Fix: drop the declaration, after checking nothing else reads has_inner_state for us. Plausibly a free win rather than a cost. Should be measured (does the V2 runner actually engage, and does anything change?) rather than assumed.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingfeature/granite-turboctc-defaultFound while landing the TurboCTC ASR default + transformers 5.16prio:lowLatent or investigativevllmvLLM backend

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions