From ef43e697423e4fef2d82986d8497c41716fb8de3 Mon Sep 17 00:00:00 2001 From: aviv ron Date: Tue, 8 Sep 2026 18:00:21 +0300 Subject: [PATCH 1/5] Default ASR to Granite Speech 5.0 TurboCTC, and raise transformers to 5.16 Re-applies the TurboCTC work onto public main by hand, rather than by cherry-picking the six commits it was developed as (09c52c2..9f1e5e8 on the staging branch). Public main has since absorbed #95, #104, #116 and #121, so several of the files had moved underneath the original patches. What changes, in five parts: * DEFAULT_ASR_MODEL_ID becomes ibm-granite/granite-speech-5.0-470m-turboctc, a 470M English conformer CTC encoder, replacing distil-whisper/distil-small.en. Being CTC it has no generate(), so decode kwargs (language/task) are dropped rather than forwarded, and transcripts are lowercase and unpunctuated. * ASR defaults retuned for a CTC backend: asr_self_chunks False and asr_chunk_length_s 120.0, since the encoder has no internal chunking and the HF pipeline's own CTC chunking mis-trims every seam (it rescales stride by inputs_to_logits_ratio, which this checkpoint does not publish). asr_device defaults to cuda and asr_dtype resolves to bfloat16 there. * transformers pinned >=5.5.1,<5.17.0 in the core package and >=5.16.0 on the audio extra, which is where granite_speech5_ctc landed. Only the audio path needs 5.16, so the requirement sits on the extra. * ATTENTION_LAYER_TYPES accepts both "attention" and "full_attention". transformers 5.16 renamed the value and rewrites it inside PreTrainedConfig.__init__, so comparing against the bare string silently dropped the attention LoRA target groups and left adapters with MLP targets only. * IsHybrid is no longer declared. It requires mamba-state hooks this model neither implements nor needs -- the composer normalizes every layer to attention, so a composed checkpoint has zero mamba layers. vLLM's escape hatch for exactly this shape compares the literal "attention", so the 5.16 rename flipped is_hybrid true and engine init began failing for the mamba dtype hook. Two places where main had moved and the original patch could not be taken as-is. Both are in compose_granite_switch.py's argparse help, and both would have been regressions if cherry-picked: * main established that --enable-audio is the only flag that switches audio on ("Requires --enable-audio; ignored without it"), replacing the older "Implies --enable-audio". The re-applied text keeps main's rule and carries only the TurboCTC facts across. * docs/AUDIO.md was hand-merged rather than overwritten, so main's confirmation from the Granite authors that <|unused_N|> ids are reserved survives alongside the new TurboCTC sections. uv.lock regenerated rather than patched: transformers 5.8.1 -> 5.16.1. Known limitation, carried over and unresolved: vLLM <=0.25 keys a layer-type table on the old "attention" spelling in granitemoehybrid.py, so in a venv built from this branch, serving a *stock* Granite 4.0 hybrid checkpoint raises KeyError: 'full_attention'. Composed Granite Switch checkpoints are unaffected -- they declare GraniteSwitchForCausalLM, which builds its own layers and never consults that table. vLLM fixed it in 0.26.0. Verified: ruff check and format clean; tests/unit/test_asr.py, test_config.py and test_config_edge_cases.py pass 90/90 (2 skipped for lack of a local vLLM). Signed-off-by: aviv ron --- docs/AUDIO.md | 88 +++++++---- pyproject.toml | 11 +- .../composer/compose_granite_switch.py | 32 ++-- src/granite_switch/config.py | 44 ++++-- src/granite_switch/vllm/audio/asr.py | 126 ++++++++++++---- src/granite_switch/vllm/audio/processor.py | 6 +- .../vllm/granite_switch_model.py | 17 ++- tests/integration/test_asr_ctc_default_gpu.py | 140 ++++++++++++++++++ tests/unit/test_asr.py | 124 +++++++++++++++- tests/unit/test_config.py | 8 +- tests/unit/test_config_edge_cases.py | 18 ++- tests/vllm/test_audio_processor.py | 14 +- tutorials/notebooks/granite_speech_demo.ipynb | 10 +- uv.lock | 79 +++++----- 14 files changed, 575 insertions(+), 142 deletions(-) create mode 100644 tests/integration/test_asr_ctc_default_gpu.py diff --git a/docs/AUDIO.md b/docs/AUDIO.md index 38359a6a..7bf536c3 100644 --- a/docs/AUDIO.md +++ b/docs/AUDIO.md @@ -15,7 +15,13 @@ The audio path needs vLLM's audio deps (`av`, `soundfile`, `resampy`, `scipy`) t decode and resample the incoming waveform. They come from vLLM's own `[audio]` extra, which the `audio` extra here pulls in (as `vllm[audio]`). A plain `uv sync --extra vllm` omits them, so it gives you a checkpoint that fails on any -non-16 kHz input: +non-16 kHz input. + +The `audio` extra also requires **transformers >= 5.16**, the release that added +`granite_speech5_ctc` — the architecture of the default ASR model. On an older +transformers the first transcription raises an `ImportError` naming the fix +(the rest of the package still works on an older release, which is why the +requirement sits on the extra rather than the core dependency). ```bash # Serving an audio-enabled checkpoint @@ -42,20 +48,29 @@ This adds the `<|audio|>` marker token to the tokenizer and writes the audio settings into `config.json` so the checkpoint is self-describing: ```json -{ "asr_enabled": true, "asr_model_id": null, "asr_device": "cpu" } +{ "asr_enabled": true, "asr_model_id": null, "asr_device": "cuda" } ``` -- `asr_model_id` — HF id of the speech-to-text model (default: a small built-in - `distil-whisper/distil-small.en`). Override with `--asr-model `, e.g. - `openai/whisper-small` for multilingual. -- `asr_device` — `cpu` (default) keeps vLLM's GPU KV-cache budget clean; set - `--asr-device cuda:0` to run transcription on GPU (watch GPU memory). +- `asr_model_id` — HF id of the speech-to-text model (default: + `ibm-granite/granite-speech-5.0-470m-turboctc`, a 470M English conformer CTC + encoder). Override with `--asr-model `, e.g. `openai/whisper-small` for + multilingual. +- `asr_device` — `cuda` (default): the default encoder is small and its speed + comes from running on GPU. Set `--asr-device cpu` to leave vLLM's whole GPU + memory budget to the KV cache — transcription is then several times slower + (measured ~3x realtime on a laptop CPU, i.e. a 10-minute clip takes minutes). + On GPU, mind that vLLM pre-allocates its KV cache first, so a tight + `--gpu-memory-utilization` can leave too little for the ASR weights. - `asr_dtype` — precision the ASR weights load in. Unset (default) derives it - from the device: `float16` on CUDA, `float32` on CPU. Half precision halves - the ASR weight footprint and is what the Whisper-family defaults expect, but - it is not universally safe — an encoder with **BatchNorm** layers raises + from the device: `bfloat16` on CUDA, `float32` on CPU. bfloat16 because it is + the default checkpoint's own dtype (no conversion implied) and because it keeps + float32's exponent range, which is the safer choice for an encoder carrying + **BatchNorm** in every conv block. Note that float16 is *not* rejected by this + model — measured on an A100 (torch 2.10 / transformers 5.16) it loads and + transcribes correctly — so bfloat16 is a considered default, not a hard + requirement. A different encoder may still hit `Expected weight to have type Float but got Half`, since BatchNorm will not - promote a float16 weight against float32 features. Such a checkpoint needs + promote a float16 weight against float32 features; such a checkpoint needs `--asr-dtype float32`. Accepted: `auto`, `float16`, `bfloat16`, `float32`. Audio capability is **gated per checkpoint** by `asr_enabled`: a checkpoint built @@ -78,7 +93,8 @@ needed to swap or steer any HF `automatic-speech-recognition` model: pipeline is built, so they are folded into the transcriber cache key. - `asr_generate_kwargs` — **decode-time** defaults applied on every transcription (e.g. `language`, `task` for a multilingual Whisper). Applied at call time, so - one loaded pipeline is reused. Ignored by non-generative backends (e.g. CTC). + one loaded pipeline is reused. Dropped for a CTC backend (the default), which + has no ``generate()`` to steer. Set them at compose time (JSON), which writes them into `config.json`: @@ -117,15 +133,25 @@ Shorten the audio or serve with a larger `--max-model-len`. Relevant config fiel **Long single clips** are handled two ways, selected by `asr_self_chunks`: -- `asr_self_chunks: true` (default) — the backend chunks internally. The Whisper - pipeline does this via `chunk_length_s` with timestamp-based stitching, so our - chunker is bypassed. -- `asr_self_chunks: false` — route audio through the **encoder-agnostic** chunker: - split into overlapping windows (`asr_chunk_length_s`, default `30.0`; - `asr_chunk_overlap_s`, default `5.0`), transcribe each, and merge with - overlap de-duplication. Use this for a backend with a fixed input window (e.g. a - speech encoder that cannot self-chunk); the transcript stitching then lives - above the backend so any backend inherits long-audio support. +- `asr_self_chunks: false` (default) — route audio through the + **encoder-agnostic** chunker: split into overlapping windows + (`asr_chunk_length_s`, default `120.0`; `asr_chunk_overlap_s`, default `5.0`), + transcribe each, and merge with overlap de-duplication. A clip at or under the + window is a single segment and reaches the backend whole, so the CTC default + handles everything up to two minutes in one pass and only longer clips are + split. The window is what bounds activation memory: measured on CPU, peak RSS + was ~1.4GB at 60s of audio, ~2.3GB at 300s and ~3.5GB at 600s. +- `asr_self_chunks: true` — the backend handles long audio itself. For a + generative backend that means its own timestamp-based stitching (Whisper), which + is more precise than our text-level merge. For a CTC backend it means feeding an + arbitrarily long clip in one pass — its block attention keeps cost linear in + duration, so this is a memory-for-accuracy trade rather than a hard limit. + +The HF pipeline's *own* CTC chunking is deliberately never used: it rescales chunk +stride by the model's `inputs_to_logits_ratio`, which the CTC default does not +publish, so the pipeline falls back to `1` and trims every seam at the wrong +offset. `chunk_length_s` therefore reaches only a generative backend, and only at +call time — once the pipeline exists and its kind is known. These are settable at compose time and are equally editable in `config.json`: @@ -269,9 +295,15 @@ ids, 4.2 has 72), so nothing should depend on a specific count or id range — - **Cascade, not end-to-end.** Prosody/emotion/uncertainty are lost; ASR errors propagate to the LLM. Two models run sequentially (ASR then LLM). -- **English by default** (`distil-whisper/distil-small.en`). Use `--asr-model` - with a multilingual model and set the language via `asr_generate_kwargs` (or +- **English only by default** (`ibm-granite/granite-speech-5.0-470m-turboctc`), + and being CTC it has no language/task knobs at all, so the per-request + `language` override is inert. For other languages use `--asr-model` with a + multilingual generative model and set the language via `asr_generate_kwargs` (or per request via `mm_processor_kwargs`; see *Tuning the ASR model* above). +- **Transcripts from the CTC default are lowercase and unpunctuated** + (`what is the capital of israel`). They are spliced into the prompt as ordinary + text, so the LLM reads them that way. A generative backend such as Whisper + restores case and punctuation. - **HF `pipeline` backends only.** Any `automatic-speech-recognition` pipeline model works via config alone; a non-pipeline backend (cloud STT, faster-whisper, a custom encoder) still needs a code-level plug point — tracked as future work. @@ -279,8 +311,10 @@ ids, 4.2 has 72), so nothing should depend on a specific count or id range — context split across the request's clips, so many/long clips together are bound by `max_model_len` (see *Long audio & multiple clips* above). - Chunk-merge de-duplication is text-level (word overlap at each seam); it can - mis-handle a phrase legitimately repeated across a window boundary. Whisper's - internal timestamp stitching (`asr_self_chunks: true`) is more precise. + mis-handle a phrase legitimately repeated across a window boundary. A generative + backend's internal timestamp stitching (`asr_self_chunks: true`) is more + precise, but is unavailable for the CTC default — hence the wide 120s window, + which leaves most clips seam-free. ## Audio + adapters @@ -309,5 +343,9 @@ pytest -m "audio and not gpu" -v -s --tb=short and per-request decode-kwargs resolution). No GPU/vLLM required. - `tests/unit/test_config.py` — round-trips `asr_pipeline_kwargs` / `asr_generate_kwargs` through save/load. +- `tests/integration/test_asr_ctc_default_gpu.py` (GPU, downloads the ~1GB + checkpoint) — the default CTC model through `ASRTranscriber`: bfloat16 on CUDA, + CTC classification, a correct transcript with client decode kwargs dropped, the + float16/BatchNorm guard, and the 120s single-pass/chunked boundary. - End-to-end (GPU): compose an `--enable-audio` checkpoint, then an audio request through vLLM produces an answer and text-only requests are unaffected. diff --git a/pyproject.toml b/pyproject.toml index 3796d113..4fe70f39 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -11,7 +11,10 @@ license = "Apache-2.0" requires-python = ">=3.11,<3.14" dependencies = [ "torch>=2.10.0", - "transformers>=5.5.1,<5.10.0", + # Ceiling raised from <5.10 so the audio extra can reach 5.16, which is where + # the default ASR model's architecture (granite_speech5_ctc) landed. The core + # package keeps the lower bound: only the audio path needs 5.16. + "transformers>=5.5.1,<5.17.0", ] [project.urls] @@ -25,8 +28,10 @@ vllm20 = ["vllm>=0.20.0,<0.21.0"] compose = ["huggingface_hub", "pyyaml", "tqdm", "safetensors"] build = ["huggingface_hub", "pyyaml", "tqdm", "safetensors"] # Backward compatibility alias for compose # Audio (ASR) decode + resample. Reuse vLLM's own audio deps (unversioned, so it -# tracks the active vLLM). -audio = ["vllm[audio]"] +# tracks the active vLLM). transformers 5.16 is where granite_speech5_ctc — the +# default ASR model's architecture — landed, so the audio path requires it while +# the rest of the package still works on an older release. +audio = ["vllm[audio]", "transformers>=5.16.0"] tutorials = [ "granite-switch[hf,vllm,compose]", "chromadb>=0.4.0", diff --git a/src/granite_switch/composer/compose_granite_switch.py b/src/granite_switch/composer/compose_granite_switch.py index b9e45df5..8e3015d2 100755 --- a/src/granite_switch/composer/compose_granite_switch.py +++ b/src/granite_switch/composer/compose_granite_switch.py @@ -840,14 +840,16 @@ def _compose_argparser(): type=str, default=None, help="HF id of the speech-to-text model the audio preprocessor loads. " - "Requires --enable-audio; ignored without it. Defaults to a small built-in model when unset.", + "Requires --enable-audio; ignored without it. Defaults to ibm-granite/" + "granite-speech-5.0-470m-turboctc (470M English CTC) when unset.", ) parser.add_argument( "--asr-device", type=str, - default="cpu", - help="Device the ASR model runs on (default: cpu). Use e.g. cuda:0 to " - "run transcription on GPU (watch vLLM's KV-cache memory budget).", + default="cuda", + help="Device the ASR model runs on (default: cuda). Use cpu to leave " + "vLLM's whole GPU memory budget to the KV cache, at the cost of much " + "slower transcription.", ) parser.add_argument( "--asr-dtype", @@ -855,8 +857,9 @@ def _compose_argparser(): default=None, choices=ASR_DTYPES, help="Precision the ASR weights load in. Default derives it from " - "--asr-device (float16 on CUDA, float32 on CPU); set float32 for an " - "encoder that cannot run in half precision (e.g. one with BatchNorm). " + "--asr-device (bfloat16 on CUDA, float32 on CPU); bfloat16 is the " + "default checkpoint's own dtype and keeps float32's exponent range. " + "float16 also works on that model and may be set explicitly. " "Requires --enable-audio; ignored without it.", ) parser.add_argument( @@ -873,8 +876,10 @@ def _compose_argparser(): default=None, help="JSON object of default decode kwargs applied on every " 'transcription, e.g. \'{"language": "de", "task": ' - '"transcribe"}\' for multilingual Whisper. Per-request ' - "mm_processor_kwargs override these. Requires --enable-audio; ignored without it.", + '"transcribe"}\' for a multilingual generative model such as Whisper. ' + "Ignored by a CTC backend (the default), which has no decoder to steer. " + "Per-request mm_processor_kwargs override these. Requires " + "--enable-audio; ignored without it.", ) parser.add_argument( "--asr-max-audio-clips", @@ -888,8 +893,10 @@ def _compose_argparser(): dest="asr_self_chunks", action="store_true", default=None, - help="Backend chunks long audio itself (Whisper default). Mutually " - "exclusive with --asr-no-self-chunks.", + help="Backend chunks long audio itself (a generative backend such as " + "Whisper stitches its own windows from timestamps), bypassing our " + "chunker. Also feeds an arbitrarily long clip to a CTC backend in one " + "pass. Mutually exclusive with --asr-no-self-chunks.", ) parser.add_argument( "--asr-no-self-chunks", @@ -902,8 +909,9 @@ def _compose_argparser(): "--asr-chunk-length-s", type=float, default=None, - help="Chunker window length in seconds (default 30.0). Only used when " - "the backend does not self-chunk. Requires --enable-audio; ignored without it.", + help="Chunker window length in seconds (default 120.0), which is also " + "the longest clip handed to the backend whole. Only used when the " + "backend does not self-chunk. Requires --enable-audio; ignored without it.", ) parser.add_argument( "--asr-chunk-overlap-s", diff --git a/src/granite_switch/config.py b/src/granite_switch/config.py index e75c937b..b46567ff 100644 --- a/src/granite_switch/config.py +++ b/src/granite_switch/config.py @@ -3,6 +3,14 @@ from transformers import GraniteMoeHybridConfig +# Layer-type names that mean "a full attention layer". transformers renamed +# "attention" to "full_attention" in 5.16 and rewrites the value inside +# PreTrainedConfig.__init__, so a config built with either spelling — or loaded +# from a checkpoint written by either version — must be recognized. Comparing +# against the bare string silently dropped the attention LoRA target groups on +# 5.16, leaving adapters with MLP targets only. +ATTENTION_LAYER_TYPES = frozenset({"attention", "full_attention"}) + # Accepted asr_dtype values. Keep in sync with vllm.audio.asr._ASR_DTYPE_NAMES. ASR_DTYPES = ("auto", "float16", "bfloat16", "float32") @@ -51,13 +59,17 @@ class GraniteSwitchConfig(GraniteMoeHybridConfig): asr_enabled (bool): Register the audio preprocessor that transcribes audio and splices the transcript into the prompt. Default: False. asr_model_id (Optional[str]): HF id of the speech-to-text model. None - falls back to a small built-in default. - asr_device (str): Device the ASR model runs on. Default "cpu" keeps - vLLM's GPU KV-cache budget clean. + falls back to the built-in default (Granite Speech 5.0 TurboCTC, + a 470M English CTC encoder). + asr_device (str): Device the ASR model runs on. Default "cuda" — the + default encoder is small and GPU-bound work is what makes it fast. + Set "cpu" to keep vLLM's GPU memory budget entirely for KV cache. asr_dtype (Optional[str]): Precision the ASR weights load in, one of - ASR_DTYPES. None/"auto" derives it from asr_device (float16 on - CUDA). An encoder with BatchNorm layers must set "float32". - Default: None. + ASR_DTYPES. None/"auto" derives it from asr_device (bfloat16 on + CUDA, float32 otherwise). bfloat16 because it is the default + checkpoint's own dtype and keeps float32's exponent range, which + suits an encoder with BatchNorm layers; float16 still works on the + default model and can be set explicitly. Default: None. asr_pipeline_kwargs (Optional[dict]): Extra kwargs merged into the ``transformers.pipeline(...)`` construction, e.g. ``{"chunk_length_s": 15}``. Baked into the transcriber cache key. @@ -70,15 +82,21 @@ class GraniteSwitchConfig(GraniteMoeHybridConfig): synchronous transcriptions one request can trigger and the startup profiling pass; ``--limit-mm-per-prompt`` may lower it, not raise it. Default: 32. - asr_chunk_length_s (float): Chunker window length in seconds. Only - used when asr_self_chunks is False. Default: 30.0. + asr_chunk_length_s (float): Chunker window length in seconds, and so + also the longest clip that reaches the backend in one piece (a + shorter clip is a single segment). Only used when asr_self_chunks + is False. Default: 120.0 — what the default CTC encoder handles in + one pass before activation memory dominates. asr_chunk_overlap_s (float): Overlap in seconds between chunker windows, de-duplicated by the transcript merge. Only used when asr_self_chunks is False. Default: 5.0. asr_self_chunks (bool): True when the backend chunks long audio itself (Whisper's timestamp stitching beats our text-level merge), bypassing our chunker. False routes audio through the - split/transcribe/merge chunker instead. Default: True. + split/transcribe/merge chunker instead. Default: False — the + default CTC backend does not self-chunk, and the HF pipeline's own + CTC chunking mis-trims seams for it (it needs the model to publish + inputs_to_logits_ratio, which this checkpoint does not). Shadow Residual (SR) parameters: dual_stream (bool): Whole-checkpoint decoder mode. ``False`` (default) = @@ -120,14 +138,14 @@ def __init__( # Audio (ASR) preprocessing parameters asr_enabled: bool = False, asr_model_id: str | None = None, - asr_device: str = "cpu", + asr_device: str = "cuda", asr_dtype: str | None = None, asr_pipeline_kwargs: dict | None = None, asr_generate_kwargs: dict | None = None, asr_max_audio_clips: int = 32, - asr_chunk_length_s: float = 30.0, + asr_chunk_length_s: float = 120.0, asr_chunk_overlap_s: float = 5.0, - asr_self_chunks: bool = True, + asr_self_chunks: bool = False, # Shadow Residual (SR) parameters cross_stream_rank: int | None = None, dual_stream: bool = False, @@ -340,7 +358,7 @@ def __init__( if self.num_adapters > 0: # Attention modules (present in all attention layers) - if any(lt == "attention" for lt in self.layer_types): + if any(lt in ATTENTION_LAYER_TYPES for lt in self.layer_types): lora_target_modules.extend( [ "qkv_proj", # Q/K/V fused diff --git a/src/granite_switch/vllm/audio/asr.py b/src/granite_switch/vllm/audio/asr.py index 948e63bf..1314d00b 100644 --- a/src/granite_switch/vllm/audio/asr.py +++ b/src/granite_switch/vllm/audio/asr.py @@ -5,8 +5,9 @@ CPU. The model loads lazily and is cached per (model_id, device, dtype, pipeline_kwargs), so a process loads each ASR model at most once. -Device defaults to CPU to keep vLLM's GPU KV-cache budget clean; dtype follows -the device unless a checkpoint sets ``asr_dtype``. See docs/AUDIO.md. +Device defaults to CUDA (the default CTC encoder is small and GPU-bound work is +what makes it fast); dtype follows the device unless a checkpoint sets +``asr_dtype``. See docs/AUDIO.md. """ from __future__ import annotations @@ -17,9 +18,32 @@ import numpy as np -# Small, CPU-friendly, English ASR model that emits text directly. Used when the -# checkpoint does not name its own (config.asr_model_id is None). -DEFAULT_ASR_MODEL_ID = "distil-whisper/distil-small.en" +# Default speech-to-text model: Granite Speech 5.0 TurboCTC, a 470M conformer +# CTC encoder. Used when the checkpoint does not name its own +# (config.asr_model_id is None). Non-autoregressive (one forward pass + greedy +# CTC collapse), so it cannot loop or hallucinate, but it also has no decoder to +# steer: output is lowercase and unpunctuated and language/task decode kwargs do +# not apply. English only. +DEFAULT_ASR_MODEL_ID = "ibm-granite/granite-speech-5.0-470m-turboctc" + +# Call-time chunk window handed to a *generative* (seq2seq) pipeline, which has a +# fixed input window and stitches its own chunks from timestamps. Never handed to +# a CTC pipeline: chunked CTC rescales stride by config.inputs_to_logits_ratio, +# which the CTC default does not publish, so the pipeline would fall back to 1 +# and trim every seam at the wrong offset. Long audio on a CTC backend goes +# through our own chunker instead (asr_self_chunks=False). +SEQ2SEQ_CHUNK_LENGTH_S = 30.0 + +# Longest clip the default CTC backend takes in one pass. Its block attention +# makes cost grow linearly with duration rather than quadratically, so a clip up +# to this length needs no splitting at all; past it, activation memory is the +# binding constraint and the caller's chunker takes over. Measured on CPU: +# ~1.4GB peak at 60s, ~2.3GB at 300s, ~3.5GB at 600s. +DEFAULT_CHUNK_LENGTH_S = 120.0 + +# Pipeline types that decode autoregressively, i.e. the ones the window above +# applies to. transformers sets pipeline.type at construction. +_CTC_PIPELINE_TYPES = frozenset({"ctc", "ctc_with_lm"}) ASR_DTYPE_AUTO = "auto" @@ -37,10 +61,13 @@ def _resolve_torch_dtype(dtype: str | None, device: str) -> Any: """Resolve an ``asr_dtype`` name to a ``torch.dtype``. - None/"auto" derives it from the device: float16 on CUDA, float32 elsewhere - (CPU float16 is slow and partly unimplemented). Name a dtype explicitly for - an encoder that cannot run in half precision — BatchNorm raises on a float16 - weight against float32 features rather than promoting. + None/"auto" derives it from the device: bfloat16 on CUDA, float32 elsewhere + (CPU half precision is slow and partly unimplemented). bfloat16 because it is + the default checkpoint's own dtype, so no conversion is implied, and because + it keeps float32's exponent range — the safer default for an encoder carrying + BatchNorm in every conv block. float16 is not rejected here: measured on an + A100 (torch 2.10 / transformers 5.16) it loads and transcribes correctly, so + it remains available as an explicit override. Name a dtype to override. """ import torch @@ -48,7 +75,7 @@ def _resolve_torch_dtype(dtype: str | None, device: str) -> Any: name = _ASR_DTYPE_ALIASES.get(name, name) if name == ASR_DTYPE_AUTO: on_cuda = isinstance(device, str) and device.startswith("cuda") - return torch.float16 if on_cuda else torch.float32 + return torch.bfloat16 if on_cuda else torch.float32 if name not in _ASR_DTYPE_NAMES: raise ValueError( f"Unsupported asr_dtype {dtype!r}. Expected {ASR_DTYPE_AUTO!r} (or " @@ -58,6 +85,28 @@ def _resolve_torch_dtype(dtype: str | None, device: str) -> Any: return getattr(torch, name) +def _unsupported_architecture_error(model_id: str, exc: Exception) -> Exception: + """Turn transformers' generic "unrecognized architecture" into a fix. + + The default CTC model's architecture landed in transformers 5.16 (which the + ``audio`` extra requires), so an install below that reports only that it does + not know ``granite_speech5_ctc`` — with a suggestion (trust_remote_code) that + does not apply, since the checkpoint carries no auto_map. Anything else is + re-raised untouched. + """ + if "does not recognize this architecture" not in str(exc): + return exc + import transformers + + return ImportError( + f"transformers {transformers.__version__} cannot load the ASR model " + f"{model_id!r}: its architecture requires transformers>=5.16, which the " + f"'audio' extra pins. Install it (uv sync --extra vllm --extra audio, or " + f"pip install 'transformers>=5.16'), or point the checkpoint at a model " + f"your version supports via asr_model_id (see docs/AUDIO.md)." + ) + + _CHUNKING = None @@ -86,7 +135,7 @@ def _load_chunking(): return _CHUNKING -# Sample rate expected by Whisper-family feature extractors. +# Sample rate every supported ASR front-end expects. _TARGET_SAMPLE_RATE = 16_000 # Audio item shapes vLLM may pass to a multimodal processor. @@ -104,7 +153,7 @@ class ASRTranscriber: def __init__( self, model_id: str = DEFAULT_ASR_MODEL_ID, - device: str = "cpu", + device: str = "cuda", pipeline_kwargs: Mapping[str, Any] | None = None, dtype: str | None = None, ) -> None: @@ -113,6 +162,9 @@ def __init__( self.dtype = dtype self.pipeline_kwargs: dict[str, Any] = dict(pipeline_kwargs or {}) self._pipeline = None + # Set by load(): whether the resolved backend decodes with CTC (no + # generation, so no decode kwargs and no pipeline-level chunking). + self._is_ctc = False self._load_lock = threading.Lock() def load(self) -> None: @@ -130,27 +182,40 @@ def load(self) -> None: "model": self.model_id, "device": self.device, "torch_dtype": _resolve_torch_dtype(self.dtype, self.device), - "chunk_length_s": 30, } # pipeline_kwargs last: a checkpoint may override any default above. kwargs.update(self.pipeline_kwargs) - self._pipeline = pipeline(**kwargs) + try: + built = pipeline(**kwargs) + except ValueError as exc: + raise _unsupported_architecture_error(self.model_id, exc) from exc + # transformers resolves .type from the model class; a CTC backend gets + # no chunk window (see SEQ2SEQ_CHUNK_LENGTH_S) and no decode kwargs. + self._is_ctc = getattr(built, "type", None) in _CTC_PIPELINE_TYPES + self._pipeline = built def transcribe( self, audio: AudioInput, sampling_rate: int | None = None, generate_kwargs: Mapping[str, Any] | None = None, - self_chunks: bool = True, - chunk_length_s: float = 30.0, + self_chunks: bool = False, + chunk_length_s: float = DEFAULT_CHUNK_LENGTH_S, chunk_overlap_s: float = 5.0, ) -> str: """Transcribe one audio clip, stripped. Resampled to 16 kHz as needed. ``sampling_rate`` is required unless ``audio`` is an ``(array, rate)`` - tuple. ``generate_kwargs`` is passed only when non-empty, so CTC backends - are unaffected. ``self_chunks=False`` routes long audio through - :mod:`.chunking` using ``chunk_length_s``/``chunk_overlap_s``. + tuple. ``generate_kwargs`` is passed only when non-empty and the backend + generates, so CTC backends are unaffected. + + ``self_chunks=False`` (the default, matching the CTC default model) routes + the waveform through :mod:`.chunking`: a clip at or under + ``chunk_length_s`` is one segment and reaches the backend whole, and only + a longer clip is split into overlapping windows and merged. Set + ``self_chunks=True`` for a backend that stitches its own windows from + timestamps (Whisper) or to feed an arbitrarily long clip to a CTC backend + in a single pass. """ samples, sr = _coerce_audio(audio, sampling_rate) samples = _to_mono_float32(samples) @@ -173,10 +238,21 @@ def _run_pipeline( samples: np.ndarray, generate_kwargs: Mapping[str, Any] | None = None, ) -> str: - """Run the loaded pipeline over an already-resampled mono waveform.""" + """Run the loaded pipeline over an already-resampled mono waveform. + + A generative backend gets ``chunk_length_s`` so it stitches its own + windows from timestamps (a seq2seq encoder has a fixed input window and + would otherwise silently truncate). A CTC backend gets neither that nor + ``generate_kwargs``: it consumes the whole waveform in one pass, and the + caller bounds the waveform's length instead (``asr_self_chunks=False``). + A window supplied in ``pipeline_kwargs`` is already bound into the + pipeline, so it is not repeated here. + """ call_kwargs: dict[str, Any] = {} - if generate_kwargs: + if generate_kwargs and not self._is_ctc: call_kwargs["generate_kwargs"] = dict(generate_kwargs) + if not self._is_ctc and "chunk_length_s" not in self.pipeline_kwargs: + call_kwargs["chunk_length_s"] = SEQ2SEQ_CHUNK_LENGTH_S result = self._pipeline( {"raw": samples, "sampling_rate": _TARGET_SAMPLE_RATE}, **call_kwargs, @@ -231,7 +307,7 @@ def _freeze(value: Any) -> Any: def get_transcriber( model_id: str | None = None, - device: str = "cpu", + device: str = "cuda", pipeline_kwargs: Mapping[str, Any] | None = None, dtype: str | None = None, ) -> ASRTranscriber: @@ -262,12 +338,12 @@ def transcribe( sampling_rate: int | None = None, *, model_id: str | None = None, - device: str = "cpu", + device: str = "cuda", pipeline_kwargs: Mapping[str, Any] | None = None, dtype: str | None = None, generate_kwargs: Mapping[str, Any] | None = None, - self_chunks: bool = True, - chunk_length_s: float = 30.0, + self_chunks: bool = False, + chunk_length_s: float = DEFAULT_CHUNK_LENGTH_S, chunk_overlap_s: float = 5.0, ) -> str: """Convenience wrapper: transcribe with the cached transcriber for the args.""" diff --git a/src/granite_switch/vllm/audio/processor.py b/src/granite_switch/vllm/audio/processor.py index ba6715e0..a47c81e0 100644 --- a/src/granite_switch/vllm/audio/processor.py +++ b/src/granite_switch/vllm/audio/processor.py @@ -89,7 +89,7 @@ def _asr_model_id(self) -> str: def _asr_device(self) -> str: cfg = self.get_hf_config() - return getattr(cfg, "asr_device", "cpu") or "cpu" + return getattr(cfg, "asr_device", "cuda") or "cuda" def _asr_dtype(self) -> str | None: cfg = self.get_hf_config() @@ -109,11 +109,11 @@ def _asr_max_audio_clips(self) -> int: def _asr_self_chunks(self) -> bool: cfg = self.get_hf_config() - return bool(getattr(cfg, "asr_self_chunks", True)) + return bool(getattr(cfg, "asr_self_chunks", False)) def _asr_chunk_length_s(self) -> float: cfg = self.get_hf_config() - return float(getattr(cfg, "asr_chunk_length_s", 30.0) or 30.0) + return float(getattr(cfg, "asr_chunk_length_s", 120.0) or 120.0) def _asr_chunk_overlap_s(self) -> float: cfg = self.get_hf_config() diff --git a/src/granite_switch/vllm/granite_switch_model.py b/src/granite_switch/vllm/granite_switch_model.py index 6011aa7b..64f7b263 100644 --- a/src/granite_switch/vllm/granite_switch_model.py +++ b/src/granite_switch/vllm/granite_switch_model.py @@ -34,7 +34,6 @@ ) from vllm.model_executor.models.interfaces import ( HasInnerState, - IsHybrid, SupportsLoRA, SupportsMultiModal, SupportsPP, @@ -414,7 +413,21 @@ class GraniteSwitchForCausalLM( SupportsLoRA, SupportsMultiModal, SupportsPP, - IsHybrid, + # IsHybrid deliberately NOT declared. It is a Protocol requiring + # get_mamba_state_shape_from_config and get_mamba_state_copy_func, neither of + # which this model implements or needs: the composer normalizes every layer + # to attention, so a composed checkpoint has zero mamba layers. The marker + # arrived by inheritance from GraniteMoeHybridConfig, not by design. + # + # Declaring it set ModelConfig._model_info.is_hybrid, and vLLM's escape hatch + # for exactly this case compares the literal string "attention": + # return layer_types is None or not all( + # layer == "attention" for layer in layer_types) # config/model.py + # transformers 5.16 remaps "attention" -> "full_attention" on load, so the + # hatch stopped matching and vLLM took the hybrid path, asking for the mamba + # dtype hook and failing engine init with AttributeError. Nine other vLLM + # sites gate on is_hybrid (mamba state allocation, block-size alignment, + # speculative decoding, KV sizing); none should apply here. ): """ Granite model with switch for causal language modeling. diff --git a/tests/integration/test_asr_ctc_default_gpu.py b/tests/integration/test_asr_ctc_default_gpu.py new file mode 100644 index 00000000..e87a828a --- /dev/null +++ b/tests/integration/test_asr_ctc_default_gpu.py @@ -0,0 +1,140 @@ +# SPDX-License-Identifier: Apache-2.0 +"""GPU checks for the default CTC ASR backend (Granite Speech 5.0 TurboCTC). + +Exercises the real model through our own :class:`ASRTranscriber`, so it covers +the path a served checkpoint takes rather than transformers in isolation: the +default device, the dtype auto-resolution, the CTC classification that suppresses +pipeline chunking and decode kwargs, and the window that decides whether a clip +reaches the model whole or through our chunker. + +Downloads the ~1GB checkpoint on first run. Opt in explicitly: +`pytest -m "audio and requires_model and gpu"`. +""" + +import importlib.util +import pathlib +import wave + +import numpy as np +import pytest + +pytestmark = [ + pytest.mark.audio, + pytest.mark.slow, + pytest.mark.requires_model, + pytest.mark.gpu, +] + +torch = pytest.importorskip("torch") +if not torch.cuda.is_available(): + pytest.skip("requires a CUDA GPU", allow_module_level=True) + +_REPO_ROOT = pathlib.Path(__file__).resolve().parents[2] + +# Load asr.py by path, as the CPU unit tests do: it has no vLLM dependency of its +# own, and the package __init__ above it imports vLLM. +_spec = importlib.util.spec_from_file_location( + "gs_asr_ctc_gpu", _REPO_ROOT / "src/granite_switch/vllm/audio/asr.py" +) +asr = importlib.util.module_from_spec(_spec) +_spec.loader.exec_module(asr) + +# tests/audio/test1.wav says this, and CTC emits it lowercase and unpunctuated. +_EXPECTED = "what is the capital of israel" + + +def _test_clip(): + """The committed test waveform as float32 mono, plus its sample rate.""" + with wave.open(str(_REPO_ROOT / "tests/audio/test1.wav")) as handle: + sample_rate, frames = ( + handle.getframerate(), + handle.readframes(handle.getnframes()), + ) + samples = np.frombuffer(frames, dtype=np.int16).astype(np.float32) / 32768.0 + return samples, sample_rate + + +@pytest.fixture(scope="module") +def transcriber(): + """The default model on CUDA with the default (auto -> bfloat16) dtype.""" + instance = asr.ASRTranscriber(model_id=asr.DEFAULT_ASR_MODEL_ID, device="cuda") + instance.load() + return instance + + +def test_auto_dtype_on_cuda_is_bfloat16(): + assert asr._resolve_torch_dtype(None, "cuda") is torch.bfloat16 + + +def test_backend_is_detected_as_ctc(transcriber): + # Drives everything else: a CTC pipeline takes neither a chunk window nor + # decode kwargs, and misclassifying it would surface as a TypeError per call. + assert transcriber._is_ctc is True + + +def test_transcribes_correctly_and_ignores_decode_kwargs(transcriber): + samples, sample_rate = _test_clip() + # `language` is what a client may send via mm_processor_kwargs. A CTC backend + # has no generate() to take it, so it must be dropped rather than forwarded. + text = transcriber.transcribe( + samples, sampling_rate=sample_rate, generate_kwargs={"language": "fr"} + ) + assert text == _EXPECTED + + +def test_float16_override_still_transcribes(): + """float16 is an available override, not a failure mode. + + The default resolves to bfloat16 on CUDA because that is the checkpoint's own + dtype and it keeps float32's exponent range next to this encoder's BatchNorm + layers. An earlier version of this test asserted float16 *raised*; measured on + an A100 (torch 2.10 / transformers 5.16) it does not, so what is worth + guarding is that the override keeps working and still produces the transcript. + """ + samples, sample_rate = _test_clip() + half = asr.ASRTranscriber( + model_id=asr.DEFAULT_ASR_MODEL_ID, device="cuda", dtype="float16" + ) + assert half.transcribe(samples, sampling_rate=sample_rate) == _EXPECTED + + +def _tile_to_seconds(samples, sample_rate, seconds): + reps = int(np.ceil(seconds * sample_rate / len(samples))) + return np.tile(samples, reps)[: int(seconds * sample_rate)] + + +@pytest.mark.parametrize( + "seconds,expected_calls", + [ + # At/under the window the clip must reach the model in one piece; past it + # our chunker splits, since the pipeline's own CTC chunking mis-trims + # seams for a model that publishes no inputs_to_logits_ratio. + (int(asr.DEFAULT_CHUNK_LENGTH_S) - 20, 1), + (int(asr.DEFAULT_CHUNK_LENGTH_S) * 2, None), + ], +) +def test_window_decides_single_pass_vs_chunked(transcriber, seconds, expected_calls): + samples, sample_rate = _test_clip() + audio = _tile_to_seconds(samples, sample_rate, seconds) + + calls: list[float] = [] + inner = transcriber._run_pipeline + + def counting(segment, generate_kwargs=None): + calls.append(len(segment) / asr._TARGET_SAMPLE_RATE) + return inner(segment, generate_kwargs) + + transcriber._run_pipeline = counting + try: + text = transcriber.transcribe(audio, sampling_rate=sample_rate) + finally: + transcriber._run_pipeline = inner + + print(f"{seconds}s -> {len(calls)} call(s) {[round(c) for c in calls]}") + if expected_calls is None: + assert len(calls) > 1 + assert max(calls) <= asr.DEFAULT_CHUNK_LENGTH_S + 1 + else: + assert len(calls) == expected_calls + # The phrase repeats throughout, so the merge must not collapse it to nothing. + assert _EXPECTED.split()[-1] in text diff --git a/tests/unit/test_asr.py b/tests/unit/test_asr.py index 8ccf492b..24010c0a 100644 --- a/tests/unit/test_asr.py +++ b/tests/unit/test_asr.py @@ -171,6 +171,120 @@ def test_short_clip_single_window(self): assert out == "seg80000" +class TestUnsupportedArchitectureError: + """A transformers too old for the default model must say so actionably.""" + + def test_unrecognized_architecture_becomes_actionable_importerror(self): + boom = mock.Mock( + side_effect=ValueError( + "The checkpoint you are trying to load has model type " + "`granite_speech5_ctc` but Transformers does not recognize this " + "architecture." + ) + ) + with _patched_pipeline(boom): + t = asr.ASRTranscriber(model_id=asr.DEFAULT_ASR_MODEL_ID, device="cpu") + with pytest.raises(ImportError) as excinfo: + t.load() + message = str(excinfo.value) + assert "transformers>=5.16" in message + assert "audio" in message # names the extra that pins it + + def test_other_value_errors_are_left_alone(self): + boom = mock.Mock(side_effect=ValueError("some unrelated pipeline problem")) + with _patched_pipeline(boom): + with pytest.raises(ValueError, match="some unrelated pipeline problem"): + asr.ASRTranscriber(model_id="m", device="cpu").load() + + +class TestBackendKindDrivesCallKwargs: + """A CTC backend gets neither a chunk window nor decode kwargs; a generative + one gets both. Guards the two ways handing chunk_length_s to a CTC pipeline + goes wrong: chunked CTC rescales stride by inputs_to_logits_ratio (absent on + the default checkpoint, so it silently falls back to 1 and mis-trims every + seam), and a CTC pipeline has no generate() to take decode kwargs at all.""" + + def _transcriber(self, pipeline_type, pipeline_kwargs=None): + factory = mock.Mock(return_value=mock.Mock(type=pipeline_type)) + with _patched_pipeline(factory): + t = asr.ASRTranscriber( + model_id="m", device="cpu", pipeline_kwargs=pipeline_kwargs + ) + t.load() + # Re-point at a recorder now that load() has classified the backend. + t._pipeline = mock.Mock(return_value={"text": "hi"}) + return t + + @pytest.mark.parametrize("pipeline_type", ["ctc", "ctc_with_lm"]) + def test_ctc_gets_no_chunk_window_and_no_decode_kwargs(self, pipeline_type): + t = self._transcriber(pipeline_type) + assert t._is_ctc is True + t.transcribe( + np.zeros(1600, dtype=np.float32), + sampling_rate=16000, + generate_kwargs={"language": "fr"}, + self_chunks=True, + ) + kwargs = t._pipeline.call_args.kwargs + assert "chunk_length_s" not in kwargs + assert "generate_kwargs" not in kwargs + + def test_seq2seq_gets_chunk_window_and_decode_kwargs(self): + t = self._transcriber("seq2seq_whisper") + assert t._is_ctc is False + t.transcribe( + np.zeros(1600, dtype=np.float32), + sampling_rate=16000, + generate_kwargs={"language": "fr"}, + self_chunks=True, + ) + kwargs = t._pipeline.call_args.kwargs + assert kwargs["chunk_length_s"] == asr.SEQ2SEQ_CHUNK_LENGTH_S + assert kwargs["generate_kwargs"] == {"language": "fr"} + + def test_explicit_pipeline_window_is_not_repeated_at_call_time(self): + # Already bound into the pipeline at construction; passing it again would + # override the checkpoint's own choice. + t = self._transcriber("seq2seq_whisper", pipeline_kwargs={"chunk_length_s": 15}) + t.transcribe(np.zeros(1600, dtype=np.float32), sampling_rate=16000) + assert "chunk_length_s" not in t._pipeline.call_args.kwargs + + def test_construction_passes_no_chunk_window(self): + # The window is a call-time decision now, since it depends on the backend + # kind, which is only known once the pipeline exists. + factory = mock.Mock(return_value=mock.Mock(type="ctc")) + with _patched_pipeline(factory): + asr.ASRTranscriber(model_id="m", device="cpu").load() + assert "chunk_length_s" not in factory.call_args.kwargs + + +class TestSinglePassCeiling: + """The default 120s window is the boundary between 'backend handles it whole' + and 'our chunker splits it'.""" + + def _recording_transcriber(self): + t = asr.ASRTranscriber(model_id="x", device="cpu") + t._is_ctc = True + t.calls = [] + t._pipeline = lambda inp, **k: ( + t.calls.append(len(inp["raw"])) or {"text": f"seg{len(inp['raw'])}"} + ) + return t + + def test_clip_at_the_ceiling_reaches_the_backend_whole(self): + t = self._recording_transcriber() + n = int(asr.DEFAULT_CHUNK_LENGTH_S) * 16000 + t.transcribe(np.zeros(n, dtype=np.float32), sampling_rate=16000) + assert t.calls == [n] + + def test_clip_past_the_ceiling_is_split(self): + t = self._recording_transcriber() + n = int(asr.DEFAULT_CHUNK_LENGTH_S * 2) * 16000 + t.transcribe(np.zeros(n, dtype=np.float32), sampling_rate=16000) + assert len(t.calls) > 1 + assert max(t.calls) <= int(asr.DEFAULT_CHUNK_LENGTH_S) * 16000 + + class TestTranscriberCache: def test_same_key_returns_same_instance(self): a = asr.get_transcriber("m", "cpu") @@ -298,12 +412,14 @@ def _patched_pipeline(factory): class TestResolveTorchDtype: - """asr_dtype resolution. float16-on-CUDA is the default, but overridable.""" + """asr_dtype resolution. bfloat16-on-CUDA is the default, but overridable.""" - def test_auto_on_cuda_is_float16(self): + def test_auto_on_cuda_is_bfloat16(self): + # bfloat16, not float16: it is the default checkpoint's own dtype and + # keeps float32's exponent range next to the encoder's BatchNorm layers. torch = pytest.importorskip("torch") - assert asr._resolve_torch_dtype(None, "cuda:0") is torch.float16 - assert asr._resolve_torch_dtype("auto", "cuda") is torch.float16 + assert asr._resolve_torch_dtype(None, "cuda:0") is torch.bfloat16 + assert asr._resolve_torch_dtype("auto", "cuda") is torch.bfloat16 def test_auto_on_cpu_is_float32(self): torch = pytest.importorskip("torch") diff --git a/tests/unit/test_config.py b/tests/unit/test_config.py index 1eab268f..1b134986 100644 --- a/tests/unit/test_config.py +++ b/tests/unit/test_config.py @@ -106,7 +106,7 @@ def test_asr_defaults_off(self): cfg = GraniteSwitchConfig(num_adapters=0) assert cfg.asr_enabled is False assert cfg.asr_model_id is None - assert cfg.asr_device == "cpu" + assert cfg.asr_device == "cuda" assert cfg.asr_dtype is None assert cfg.asr_pipeline_kwargs is None assert cfg.asr_generate_kwargs is None @@ -125,9 +125,11 @@ def test_invalid_asr_dtype_raises(self): def test_longaudio_defaults(self): cfg = GraniteSwitchConfig(num_adapters=0) assert cfg.asr_max_audio_clips == 32 - assert cfg.asr_chunk_length_s == 30.0 + assert cfg.asr_chunk_length_s == 120.0 assert cfg.asr_chunk_overlap_s == 5.0 - assert cfg.asr_self_chunks is True + # The default backend is CTC: it does not self-chunk, so long clips go + # through our chunker at a 120s window. + assert cfg.asr_self_chunks is False def test_longaudio_round_trip(self, tmp_path): cfg = GraniteSwitchConfig( diff --git a/tests/unit/test_config_edge_cases.py b/tests/unit/test_config_edge_cases.py index 4d541e3a..72c18ab6 100644 --- a/tests/unit/test_config_edge_cases.py +++ b/tests/unit/test_config_edge_cases.py @@ -1,7 +1,7 @@ # SPDX-License-Identifier: Apache-2.0 """Additional config edge case tests for GraniteSwitchConfig.""" -from granite_switch.config import GraniteSwitchConfig +from granite_switch.config import ATTENTION_LAYER_TYPES, GraniteSwitchConfig def _valid_kwargs(num_adapters=2, **overrides): @@ -44,11 +44,20 @@ def test_explicit_shared_intermediate_size_preserved(self): class TestLayerTypesDefault: - """layer_types defaults to all-attention with length == num_hidden_layers.""" + """layer_types defaults to all-attention with length == num_hidden_layers. + + The spelling is transformers', not ours: it renamed "attention" to + "full_attention" in 5.16 and rewrites whatever we pass inside + PreTrainedConfig.__init__. So these assert the *meaning* — every layer is a + full attention layer, and the list is as long as num_hidden_layers — against + ATTENTION_LAYER_TYPES, which is the same set the config consults when it + derives the default LoRA target groups. + """ def test_default_layer_types_when_omitted(self): cfg = GraniteSwitchConfig(num_adapters=0, num_hidden_layers=4) - assert cfg.layer_types == ["attention"] * 4 + assert len(cfg.layer_types) == 4 + assert set(cfg.layer_types) <= ATTENTION_LAYER_TYPES def test_explicit_layer_types_preserved(self): cfg = GraniteSwitchConfig( @@ -56,7 +65,8 @@ def test_explicit_layer_types_preserved(self): num_hidden_layers=3, layer_types=["attention", "attention", "attention"], ) - assert cfg.layer_types == ["attention", "attention", "attention"] + assert len(cfg.layer_types) == 3 + assert set(cfg.layer_types) <= ATTENTION_LAYER_TYPES class TestLoraTargetModulesDefault: diff --git a/tests/vllm/test_audio_processor.py b/tests/vllm/test_audio_processor.py index 367f6c1c..59051ecd 100644 --- a/tests/vllm/test_audio_processor.py +++ b/tests/vllm/test_audio_processor.py @@ -81,8 +81,8 @@ def test_model_id_explicit(self): info = _make_info(asr_enabled=True, asr_model_id="openai/whisper-small") assert info._asr_model_id() == "openai/whisper-small" - def test_device_default_cpu(self): - assert _make_info(asr_enabled=True)._asr_device() == "cpu" + def test_device_default_cuda(self): + assert _make_info(asr_enabled=True)._asr_device() == "cuda" def test_pipeline_and_generate_kwargs_default_empty(self): info = _make_info(asr_enabled=True) @@ -101,8 +101,10 @@ def test_pipeline_and_generate_kwargs_from_config(self): def test_longaudio_accessor_defaults(self): info = _make_info(asr_enabled=True) assert info._asr_max_audio_clips() == 32 - assert info._asr_self_chunks() is True - assert info._asr_chunk_length_s() == 30.0 + # The default backend is CTC: it does not self-chunk, so a clip longer + # than the 120s window goes through our chunker. + assert info._asr_self_chunks() is False + assert info._asr_chunk_length_s() == 120.0 assert info._asr_chunk_overlap_s() == 5.0 def test_longaudio_accessors_from_config(self): @@ -146,8 +148,8 @@ def transcribe( audio, sampling_rate=None, generate_kwargs=None, - self_chunks=True, - chunk_length_s=30.0, + self_chunks=False, + chunk_length_s=120.0, chunk_overlap_s=5.0, ): capture["sampling_rate"] = sampling_rate diff --git a/tutorials/notebooks/granite_speech_demo.ipynb b/tutorials/notebooks/granite_speech_demo.ipynb index bb0c8bb9..a79c3db5 100644 --- a/tutorials/notebooks/granite_speech_demo.ipynb +++ b/tutorials/notebooks/granite_speech_demo.ipynb @@ -20,7 +20,7 @@ "\n", "## Prerequisites\n", "\n", - "- **GPU runtime with ~16+ GiB free** (e.g. Colab A100/L4). The audio model = the LLM weights + a small embedded Whisper.\n", + "- **GPU runtime with ~16+ GiB free** (e.g. Colab A100/L4). The audio model = the LLM weights + a small embedded speech recognizer.\n", "- **A composed, audio-enabled checkpoint.** Our audio model isn't on the Hub — compose it once with `--enable-audio` and point `MODEL_PATH` at it (see the configuration cell).\n", "- **HuggingFace read token.** Free; create one at https://huggingface.co/settings/tokens. Add it as a Colab Secret named `HF_TOKEN`. Used for downloading model weights *and* minting per-session WebRTC TURN credentials.\n", "- **Browser:** Chrome, Edge, or Firefox. Safari may behave oddly with WebRTC.\n", @@ -194,7 +194,7 @@ "source": [ "## 4 · Launch the vLLM model server (~2-4 min cold, ~30s cached)\n", "\n", - "**One** vLLM process — your composed, audio-enabled **Granite Switch** checkpoint (`MODEL_PATH`) on **port 8000**. It transcribes the incoming audio *internally* (Whisper, on the `asr_device` baked into the checkpoint) and generates the answer in the same request, so there's no separate STT server.\n", + "**One** vLLM process — your composed, audio-enabled **Granite Switch** checkpoint (`MODEL_PATH`) on **port 8000**. It transcribes the incoming audio *internally* (Granite Speech CTC, on the `asr_device` baked into the checkpoint) and generates the answer in the same request, so there's no separate STT server.\n", "\n", "Runs in the background; logs stream to `logs/vllm-audio.log`. The cell blocks until the server responds on `/v1/models`." ] @@ -238,7 +238,7 @@ "if free_gib < 16:\n", " raise RuntimeError(\n", " f\"Only {free_gib:.1f} GiB free on the GPU — need >=16 for the audio model \"\n", - " \"(LLM + Whisper). Free the GPU (kill-switch cell) and retry.\"\n", + " \"(LLM + ASR). Free the GPU (kill-switch cell) and retry.\"\n", " )\n", "\n", "\n", @@ -284,10 +284,10 @@ "\n", "\n", "# ONE server now: the audio-enabled Granite Switch model. It transcribes the\n", - "# incoming audio internally (Whisper on the asr_device baked into the checkpoint\n", + "# incoming audio internally (the ASR model on the asr_device baked into the checkpoint\n", "# config) and generates the answer — so there's no separate STT server.\n", "audio_log = open(\"logs/vllm-audio.log\", \"w\")\n", - "print(\"⏳ Starting Granite Switch (audio) vLLM (loads weights + Whisper, ~2-4 min)...\")\n", + "print(\"⏳ Starting Granite Switch (audio) vLLM (loads weights + ASR, ~2-4 min)...\")\n", "switch_proc = subprocess.Popen(\n", " [\n", " VENV_VLLM,\n", diff --git a/uv.lock b/uv.lock index 71c7739a..94a16650 100644 --- a/uv.lock +++ b/uv.lock @@ -2450,6 +2450,7 @@ dependencies = [ [package.optional-dependencies] audio = [ + { name = "transformers" }, { name = "vllm", version = "0.19.1", source = { registry = "https://pypi.org/simple" }, extra = ["audio"], marker = "extra == 'extra-14-granite-switch-tutorials' or extra == 'extra-14-granite-switch-vllm' or extra == 'group-14-granite-switch-dev' or extra == 'group-14-granite-switch-test' or extra == 'group-14-granite-switch-vllm19'" }, { name = "vllm", version = "0.20.2", source = { registry = "https://pypi.org/simple" }, extra = ["audio"], marker = "extra == 'extra-14-granite-switch-vllm20' or extra == 'group-14-granite-switch-dev-vllm20' or (extra == 'extra-14-granite-switch-tutorials' and extra == 'group-14-granite-switch-vllm20') or (extra == 'extra-14-granite-switch-vllm' and extra == 'group-14-granite-switch-vllm20') or (extra == 'group-14-granite-switch-dev' and extra == 'group-14-granite-switch-vllm20') or (extra == 'group-14-granite-switch-test' and extra == 'group-14-granite-switch-vllm20') or (extra == 'group-14-granite-switch-vllm19' and extra == 'group-14-granite-switch-vllm20') or (extra != 'extra-14-granite-switch-tutorials' and extra != 'extra-14-granite-switch-vllm' and extra != 'group-14-granite-switch-dev' and extra != 'group-14-granite-switch-test' and extra != 'group-14-granite-switch-vllm19')" }, ] @@ -2542,7 +2543,8 @@ requires-dist = [ { name = "torch", specifier = ">=2.10.0" }, { name = "tqdm", marker = "extra == 'build'" }, { name = "tqdm", marker = "extra == 'compose'" }, - { name = "transformers", specifier = ">=5.5.1,<5.10.0" }, + { name = "transformers", specifier = ">=5.5.1,<5.17.0" }, + { name = "transformers", marker = "extra == 'audio'", specifier = ">=5.16.0" }, { name = "vllm", marker = "extra == 'vllm'", specifier = ">=0.19.1,<0.20.0" }, { name = "vllm", marker = "extra == 'vllm20'", specifier = ">=0.20.0,<0.21.0" }, { name = "vllm", extras = ["audio"], marker = "extra == 'audio'" }, @@ -6544,24 +6546,26 @@ wheels = [ [[package]] name = "safetensors" -version = "0.7.0" +version = "0.8.0" source = { registry = "https://pypi.org/simple" } -sdist = { url = "https://files.pythonhosted.org/packages/29/9c/6e74567782559a63bd040a236edca26fd71bc7ba88de2ef35d75df3bca5e/safetensors-0.7.0.tar.gz", hash = "sha256:07663963b67e8bd9f0b8ad15bb9163606cd27cc5a1b96235a50d8369803b96b0", size = 200878, upload-time = "2025-11-19T15:18:43.199Z" } +sdist = { url = "https://files.pythonhosted.org/packages/45/06/f955dbbb1859e3bd23c8ac6141af5106e7ad5fedec4a3a6e3d60f94b7001/safetensors-0.8.0.tar.gz", hash = "sha256:fabaf3e0f18a6618d9b36560682562157f77c2b71fcffc7b432be2baed9d753d", size = 325846, upload-time = "2026-06-09T07:52:25.563Z" } wheels = [ - { url = "https://files.pythonhosted.org/packages/fa/47/aef6c06649039accf914afef490268e1067ed82be62bcfa5b7e886ad15e8/safetensors-0.7.0-cp38-abi3-macosx_10_12_x86_64.whl", hash = "sha256:c82f4d474cf725255d9e6acf17252991c3c8aac038d6ef363a4bf8be2f6db517", size = 467781, upload-time = "2025-11-19T15:18:35.84Z" }, - { url = "https://files.pythonhosted.org/packages/e8/00/374c0c068e30cd31f1e1b46b4b5738168ec79e7689ca82ee93ddfea05109/safetensors-0.7.0-cp38-abi3-macosx_11_0_arm64.whl", hash = "sha256:94fd4858284736bb67a897a41608b5b0c2496c9bdb3bf2af1fa3409127f20d57", size = 447058, upload-time = "2025-11-19T15:18:34.416Z" }, - { url = "https://files.pythonhosted.org/packages/f1/06/578ffed52c2296f93d7fd2d844cabfa92be51a587c38c8afbb8ae449ca89/safetensors-0.7.0-cp38-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:e07d91d0c92a31200f25351f4acb2bc6aff7f48094e13ebb1d0fb995b54b6542", size = 491748, upload-time = "2025-11-19T15:18:09.79Z" }, - { url = "https://files.pythonhosted.org/packages/ae/33/1debbbb70e4791dde185edb9413d1fe01619255abb64b300157d7f15dddd/safetensors-0.7.0-cp38-abi3-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:8469155f4cb518bafb4acf4865e8bb9d6804110d2d9bdcaa78564b9fd841e104", size = 503881, upload-time = "2025-11-19T15:18:16.145Z" }, - { url = "https://files.pythonhosted.org/packages/8e/1c/40c2ca924d60792c3be509833df711b553c60effbd91da6f5284a83f7122/safetensors-0.7.0-cp38-abi3-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:54bef08bf00a2bff599982f6b08e8770e09cc012d7bba00783fc7ea38f1fb37d", size = 623463, upload-time = "2025-11-19T15:18:21.11Z" }, - { url = "https://files.pythonhosted.org/packages/9b/3a/13784a9364bd43b0d61eef4bea2845039bc2030458b16594a1bd787ae26e/safetensors-0.7.0-cp38-abi3-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:42cb091236206bb2016d245c377ed383aa7f78691748f3bb6ee1bfa51ae2ce6a", size = 532855, upload-time = "2025-11-19T15:18:25.719Z" }, - { url = "https://files.pythonhosted.org/packages/a0/60/429e9b1cb3fc651937727befe258ea24122d9663e4d5709a48c9cbfceecb/safetensors-0.7.0-cp38-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:dac7252938f0696ddea46f5e855dd3138444e82236e3be475f54929f0c510d48", size = 507152, upload-time = "2025-11-19T15:18:33.023Z" }, - { url = "https://files.pythonhosted.org/packages/3c/a8/4b45e4e059270d17af60359713ffd83f97900d45a6afa73aaa0d737d48b6/safetensors-0.7.0-cp38-abi3-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:1d060c70284127fa805085d8f10fbd0962792aed71879d00864acda69dbab981", size = 541856, upload-time = "2025-11-19T15:18:31.075Z" }, - { url = "https://files.pythonhosted.org/packages/06/87/d26d8407c44175d8ae164a95b5a62707fcc445f3c0c56108e37d98070a3d/safetensors-0.7.0-cp38-abi3-musllinux_1_2_aarch64.whl", hash = "sha256:cdab83a366799fa730f90a4ebb563e494f28e9e92c4819e556152ad55e43591b", size = 674060, upload-time = "2025-11-19T15:18:37.211Z" }, - { url = "https://files.pythonhosted.org/packages/11/f5/57644a2ff08dc6325816ba7217e5095f17269dada2554b658442c66aed51/safetensors-0.7.0-cp38-abi3-musllinux_1_2_armv7l.whl", hash = "sha256:672132907fcad9f2aedcb705b2d7b3b93354a2aec1b2f706c4db852abe338f85", size = 771715, upload-time = "2025-11-19T15:18:38.689Z" }, - { url = "https://files.pythonhosted.org/packages/86/31/17883e13a814bd278ae6e266b13282a01049b0c81341da7fd0e3e71a80a3/safetensors-0.7.0-cp38-abi3-musllinux_1_2_i686.whl", hash = "sha256:5d72abdb8a4d56d4020713724ba81dac065fedb7f3667151c4a637f1d3fb26c0", size = 714377, upload-time = "2025-11-19T15:18:40.162Z" }, - { url = "https://files.pythonhosted.org/packages/4a/d8/0c8a7dc9b41dcac53c4cbf9df2b9c83e0e0097203de8b37a712b345c0be5/safetensors-0.7.0-cp38-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:b0f6d66c1c538d5a94a73aa9ddca8ccc4227e6c9ff555322ea40bdd142391dd4", size = 677368, upload-time = "2025-11-19T15:18:41.627Z" }, - { url = "https://files.pythonhosted.org/packages/05/e5/cb4b713c8a93469e3c5be7c3f8d77d307e65fe89673e731f5c2bfd0a9237/safetensors-0.7.0-cp38-abi3-win32.whl", hash = "sha256:c74af94bf3ac15ac4d0f2a7c7b4663a15f8c2ab15ed0fc7531ca61d0835eccba", size = 326423, upload-time = "2025-11-19T15:18:45.74Z" }, - { url = "https://files.pythonhosted.org/packages/5d/e6/ec8471c8072382cb91233ba7267fd931219753bb43814cbc71757bfd4dab/safetensors-0.7.0-cp38-abi3-win_amd64.whl", hash = "sha256:d1239932053f56f3456f32eb9625590cc7582e905021f94636202a864d470755", size = 341380, upload-time = "2025-11-19T15:18:44.427Z" }, + { url = "https://files.pythonhosted.org/packages/39/a0/f718cda65b05407d228f97602cf60dca269c979867aa5beb25410de26cd3/safetensors-0.8.0-cp310-abi3-macosx_10_12_x86_64.whl", hash = "sha256:c554f85858e05226d3c2828e32395e677434685d6d94594a41643361c5e837f0", size = 473568, upload-time = "2026-06-09T07:52:18.829Z" }, + { url = "https://files.pythonhosted.org/packages/f5/b1/fa7c600e7dceae12e9606c7578cbc9ff1e1ed55844883ee5c92205e86226/safetensors-0.8.0-cp310-abi3-macosx_11_0_arm64.whl", hash = "sha256:c80201d22cbf405b80647a60ada77bba06c8fba2da2743ba1e89cdcc39a81f25", size = 484562, upload-time = "2026-06-09T07:52:17.518Z" }, + { url = "https://files.pythonhosted.org/packages/09/7d/65a7de0af421317bb36a067241e4235fff194eed60b961ed6d3f59a3fc60/safetensors-0.8.0-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:7a46e5ff292c356d6991e60942ba7f79817682d3a2cef0702136448cb9c4d235", size = 502844, upload-time = "2026-06-09T07:52:07.624Z" }, + { url = "https://files.pythonhosted.org/packages/91/4f/3175c9d75634e0e0dda0082794193521035edd7c70a6f212bf33ca06ddf4/safetensors-0.8.0-cp310-abi3-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:4124502b78f03534117c848f87a39b8f31e577b15eff423bf8bfb95f2a8c30d0", size = 511823, upload-time = "2026-06-09T07:52:09.565Z" }, + { url = "https://files.pythonhosted.org/packages/20/87/846c289e7aa2299eff406335717cf43ce8777194ece8aad75772e0411615/safetensors-0.8.0-cp310-abi3-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:7bc0a787ba8a35be368ee3574edfa2b1ad389eebd0a72e482ae275490e3f6c98", size = 633461, upload-time = "2026-06-09T07:52:11.128Z" }, + { url = "https://files.pythonhosted.org/packages/76/22/8d64d9df2c45d5ded401df889d0ad90882804ca172d79ec4f0df8f727fe0/safetensors-0.8.0-cp310-abi3-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:040070828e36dc8e122178bbbd5830ff9e97920affb84cbe0f46442497bed358", size = 545148, upload-time = "2026-06-09T07:52:13.603Z" }, + { url = "https://files.pythonhosted.org/packages/28/50/f203ff3a3ddfe19308efc83c5a3a29ed02bf786732ec35e68bf9162f3365/safetensors-0.8.0-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:fd6f3f93c9a0a7cc2788ee63fb763353d4bd2e89b0751bc78fcf7dda00bea774", size = 516040, upload-time = "2026-06-09T07:52:16.29Z" }, + { url = "https://files.pythonhosted.org/packages/46/fb/cdaed17ceb2948784fd9c36b6fd3e951b608547cea81a48e8ee6f8cfdfcb/safetensors-0.8.0-cp310-abi3-manylinux_2_31_riscv64.whl", hash = "sha256:fcdd41ec4628fee5799f807c73c353629130fbd942aa23d83c623dd6c9d52d78", size = 513832, upload-time = "2026-06-09T07:52:12.37Z" }, + { url = "https://files.pythonhosted.org/packages/0d/49/1e15de264dcc3b77943d2d0c56a95809956883b1c2d6d585c792523f180b/safetensors-0.8.0-cp310-abi3-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:8e9f537aa183a38ace122d27303dcd986b26bd2a7591f9181d7f0c396f4677ca", size = 559930, upload-time = "2026-06-09T07:52:14.743Z" }, + { url = "https://files.pythonhosted.org/packages/2a/43/bf38443278eab4b1be1fce2931e2b012ad9cb7df52ada751d0aab8f7659a/safetensors-0.8.0-cp310-abi3-musllinux_1_2_aarch64.whl", hash = "sha256:87eec7ffed2b809f05a398a8becb7d013f19f7837cd15d9748580d6cf30dbaf4", size = 678670, upload-time = "2026-06-09T07:52:20.032Z" }, + { url = "https://files.pythonhosted.org/packages/72/e3/68cd3fa5b48488e84add63e04cb12f3bc28ae4638c06d4508c6e88823d0e/safetensors-0.8.0-cp310-abi3-musllinux_1_2_armv7l.whl", hash = "sha256:4a95ae2b05d7726d751da4ebf626a2ca782b706e101bd894c95bc2450b1cffcc", size = 786679, upload-time = "2026-06-09T07:52:21.322Z" }, + { url = "https://files.pythonhosted.org/packages/29/4b/1c19c509d56e01f4fbb3d0a2e597450f6cc04d1d56cf52defb0a62dfd715/safetensors-0.8.0-cp310-abi3-musllinux_1_2_i686.whl", hash = "sha256:3ae091f16662658bdc019a4ff6cb4c085bb7d725eb5978b183ffd265863b6d2d", size = 765683, upload-time = "2026-06-09T07:52:22.594Z" }, + { url = "https://files.pythonhosted.org/packages/27/43/41c1621732edd934d868a00d1b891584c892a7b62a9aab82ea5a0a5623ee/safetensors-0.8.0-cp310-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:8e080062fcde23be189565e1c3305d16751a218ecf9412c8601e64204eb6f846", size = 722361, upload-time = "2026-06-09T07:52:23.924Z" }, + { url = "https://files.pythonhosted.org/packages/8e/3f/73ccf82579412b4a71c4ca673f10b5f1f888d7cf5af7fe24f27d30307be4/safetensors-0.8.0-cp310-abi3-win32.whl", hash = "sha256:2ddf52eac562eda224f99acfa7889d02968c1fd59a5b011ae7d8137c37e9c02d", size = 342401, upload-time = "2026-06-09T07:52:28.895Z" }, + { url = "https://files.pythonhosted.org/packages/1b/6d/3fba214c1e5e0f69991677ec3bc17023f0421776975e1de0c682dca475e2/safetensors-0.8.0-cp310-abi3-win_amd64.whl", hash = "sha256:096ec1a98435df7beb08853bb5aa9081a84f23d0adc67ed1a0a10550f608373f", size = 355540, upload-time = "2026-06-09T07:52:27.832Z" }, + { url = "https://files.pythonhosted.org/packages/8d/fc/7eedc3510d97878876e32774eebbeb61c43f148a96e915c84229a3e967aa/safetensors-0.8.0-cp310-abi3-win_arm64.whl", hash = "sha256:f7838e5135a406ad3e02efdcb8cf2e5397d368b0154537c4fec682dbc544d452", size = 340500, upload-time = "2026-06-09T07:52:26.745Z" }, ] [[package]] @@ -7010,28 +7014,29 @@ wheels = [ [[package]] name = "tokenizers" -version = "0.22.2" +version = "0.23.2" source = { registry = "https://pypi.org/simple" } dependencies = [ { name = "huggingface-hub" }, ] -sdist = { url = "https://files.pythonhosted.org/packages/73/6f/f80cfef4a312e1fb34baf7d85c72d4411afde10978d4657f8cdd811d3ccc/tokenizers-0.22.2.tar.gz", hash = "sha256:473b83b915e547aa366d1eee11806deaf419e17be16310ac0a14077f1e28f917", size = 372115, upload-time = "2026-01-05T10:45:15.988Z" } +sdist = { url = "https://files.pythonhosted.org/packages/18/1e/bc6587c5ab643b2e17776cace9070a2ae73549c86bffac9934a600bf3c31/tokenizers-0.23.2.tar.gz", hash = "sha256:7f0f085686b9de0d0079e6f874ae053600db64c5d13049e0bbc0119926d25aac", size = 385745, upload-time = "2026-09-03T08:55:42.89Z" } wheels = [ - { url = "https://files.pythonhosted.org/packages/92/97/5dbfabf04c7e348e655e907ed27913e03db0923abb5dfdd120d7b25630e1/tokenizers-0.22.2-cp39-abi3-macosx_10_12_x86_64.whl", hash = "sha256:544dd704ae7238755d790de45ba8da072e9af3eea688f698b137915ae959281c", size = 3100275, upload-time = "2026-01-05T10:41:02.158Z" }, - { url = "https://files.pythonhosted.org/packages/2e/47/174dca0502ef88b28f1c9e06b73ce33500eedfac7a7692108aec220464e7/tokenizers-0.22.2-cp39-abi3-macosx_11_0_arm64.whl", hash = "sha256:1e418a55456beedca4621dbab65a318981467a2b188e982a23e117f115ce5001", size = 2981472, upload-time = "2026-01-05T10:41:00.276Z" }, - { url = "https://files.pythonhosted.org/packages/d6/84/7990e799f1309a8b87af6b948f31edaa12a3ed22d11b352eaf4f4b2e5753/tokenizers-0.22.2-cp39-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:2249487018adec45d6e3554c71d46eb39fa8ea67156c640f7513eb26f318cec7", size = 3290736, upload-time = "2026-01-05T10:40:32.165Z" }, - { url = "https://files.pythonhosted.org/packages/78/59/09d0d9ba94dcd5f4f1368d4858d24546b4bdc0231c2354aa31d6199f0399/tokenizers-0.22.2-cp39-abi3-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:25b85325d0815e86e0bac263506dd114578953b7b53d7de09a6485e4a160a7dd", size = 3168835, upload-time = "2026-01-05T10:40:38.847Z" }, - { url = "https://files.pythonhosted.org/packages/47/50/b3ebb4243e7160bda8d34b731e54dd8ab8b133e50775872e7a434e524c28/tokenizers-0.22.2-cp39-abi3-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:bfb88f22a209ff7b40a576d5324bf8286b519d7358663db21d6246fb17eea2d5", size = 3521673, upload-time = "2026-01-05T10:40:56.614Z" }, - { url = "https://files.pythonhosted.org/packages/e0/fa/89f4cb9e08df770b57adb96f8cbb7e22695a4cb6c2bd5f0c4f0ebcf33b66/tokenizers-0.22.2-cp39-abi3-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:1c774b1276f71e1ef716e5486f21e76333464f47bece56bbd554485982a9e03e", size = 3724818, upload-time = "2026-01-05T10:40:44.507Z" }, - { url = "https://files.pythonhosted.org/packages/64/04/ca2363f0bfbe3b3d36e95bf67e56a4c88c8e3362b658e616d1ac185d47f2/tokenizers-0.22.2-cp39-abi3-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:df6c4265b289083bf710dff49bc51ef252f9d5be33a45ee2bed151114a56207b", size = 3379195, upload-time = "2026-01-05T10:40:51.139Z" }, - { url = "https://files.pythonhosted.org/packages/2e/76/932be4b50ef6ccedf9d3c6639b056a967a86258c6d9200643f01269211ca/tokenizers-0.22.2-cp39-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:369cc9fc8cc10cb24143873a0d95438bb8ee257bb80c71989e3ee290e8d72c67", size = 3274982, upload-time = "2026-01-05T10:40:58.331Z" }, - { url = "https://files.pythonhosted.org/packages/1d/28/5f9f5a4cc211b69e89420980e483831bcc29dade307955cc9dc858a40f01/tokenizers-0.22.2-cp39-abi3-musllinux_1_2_aarch64.whl", hash = "sha256:29c30b83d8dcd061078b05ae0cb94d3c710555fbb44861139f9f83dcca3dc3e4", size = 9478245, upload-time = "2026-01-05T10:41:04.053Z" }, - { url = "https://files.pythonhosted.org/packages/6c/fb/66e2da4704d6aadebf8cb39f1d6d1957df667ab24cff2326b77cda0dcb85/tokenizers-0.22.2-cp39-abi3-musllinux_1_2_armv7l.whl", hash = "sha256:37ae80a28c1d3265bb1f22464c856bd23c02a05bb211e56d0c5301a435be6c1a", size = 9560069, upload-time = "2026-01-05T10:45:10.673Z" }, - { url = "https://files.pythonhosted.org/packages/16/04/fed398b05caa87ce9b1a1bb5166645e38196081b225059a6edaff6440fac/tokenizers-0.22.2-cp39-abi3-musllinux_1_2_i686.whl", hash = "sha256:791135ee325f2336f498590eb2f11dc5c295232f288e75c99a36c5dbce63088a", size = 9899263, upload-time = "2026-01-05T10:45:12.559Z" }, - { url = "https://files.pythonhosted.org/packages/05/a1/d62dfe7376beaaf1394917e0f8e93ee5f67fea8fcf4107501db35996586b/tokenizers-0.22.2-cp39-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:38337540fbbddff8e999d59970f3c6f35a82de10053206a7562f1ea02d046fa5", size = 10033429, upload-time = "2026-01-05T10:45:14.333Z" }, - { url = "https://files.pythonhosted.org/packages/fd/18/a545c4ea42af3df6effd7d13d250ba77a0a86fb20393143bbb9a92e434d4/tokenizers-0.22.2-cp39-abi3-win32.whl", hash = "sha256:a6bf3f88c554a2b653af81f3204491c818ae2ac6fbc09e76ef4773351292bc92", size = 2502363, upload-time = "2026-01-05T10:45:20.593Z" }, - { url = "https://files.pythonhosted.org/packages/65/71/0670843133a43d43070abeb1949abfdef12a86d490bea9cd9e18e37c5ff7/tokenizers-0.22.2-cp39-abi3-win_amd64.whl", hash = "sha256:c9ea31edff2968b44a88f97d784c2f16dc0729b8b143ed004699ebca91f05c48", size = 2747786, upload-time = "2026-01-05T10:45:18.411Z" }, - { url = "https://files.pythonhosted.org/packages/72/f4/0de46cfa12cdcbcd464cc59fde36912af405696f687e53a091fb432f694c/tokenizers-0.22.2-cp39-abi3-win_arm64.whl", hash = "sha256:9ce725d22864a1e965217204946f830c37876eee3b2ba6fc6255e8e903d5fcbc", size = 2612133, upload-time = "2026-01-05T10:45:17.232Z" }, + { url = "https://files.pythonhosted.org/packages/4d/ed/8a443528baa6fac8dfe8c3b75b038c63ac92bb539bcabe311e227c718173/tokenizers-0.23.2-cp310-abi3-macosx_10_12_x86_64.whl", hash = "sha256:85a9a357a3764aecc904ee76bdaf8cf1ad8e5a67a1b929a487c4a39b49ed0e90", size = 3148852, upload-time = "2026-09-03T08:55:30.874Z" }, + { url = "https://files.pythonhosted.org/packages/67/49/22da045a91732384d3a3771816bf188dc5a1f702c32e635afa7c679c0bef/tokenizers-0.23.2-cp310-abi3-macosx_11_0_arm64.whl", hash = "sha256:986670e43691469dcee610ea0f846f91a8f84e91fc6f7a48d4c064414c0ec2bf", size = 3101593, upload-time = "2026-09-03T08:55:28.587Z" }, + { url = "https://files.pythonhosted.org/packages/2e/4d/8f569ed49372a3ed8e57099bd515055fd48d7c95912c4307cda6973c2168/tokenizers-0.23.2-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:a37039b5dfc4af84eb3ef0a92f4307e28936c8f9adccba2629d36f652e9bf7a2", size = 3516830, upload-time = "2026-09-03T08:55:14.741Z" }, + { url = "https://files.pythonhosted.org/packages/2a/de/e2f14c8919d5bf51874051d00d6c7b7e0e8bde6c6a2dbeddda7f642896ff/tokenizers-0.23.2-cp310-abi3-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:7b7e37ba198f24150f523e1242e83c4970de4a525480586be5dcc24d9add32c5", size = 3407975, upload-time = "2026-09-03T08:55:16.842Z" }, + { url = "https://files.pythonhosted.org/packages/c5/bd/93c69152d02ef06ce47aed8b2bf4952dcf733c935a62791873932b2934d9/tokenizers-0.23.2-cp310-abi3-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:43e4f2071e3cc8d5d86421c874aebc82659bb51a68bcdef5a0da75ee89511ccb", size = 3748165, upload-time = "2026-09-03T08:55:24.769Z" }, + { url = "https://files.pythonhosted.org/packages/2d/b7/56b84b80bc96942bba8eb23751a9e8a1fce4faaf4390425e7083f721c98c/tokenizers-0.23.2-cp310-abi3-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:325fee2e0418a9dc6c9ecf736a5f5f0db7875183ace9549ae339da76f7a1fbb7", size = 4024165, upload-time = "2026-09-03T08:55:18.806Z" }, + { url = "https://files.pythonhosted.org/packages/9b/8a/0175e216f005c2fe08238292663aa41e4c802b216e71047a69a0e9fc6fa3/tokenizers-0.23.2-cp310-abi3-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:950d7c9426fa72406a0ffeacdbc0bb9985f5db20eb8b263f29c79aaf83105703", size = 3591899, upload-time = "2026-09-03T08:55:22.752Z" }, + { url = "https://files.pythonhosted.org/packages/2c/ca/ca6b93c7820df123b2662a9469e8facc826ccc94e98fdd0d615f6431e73a/tokenizers-0.23.2-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:41c2f84d172449b4dadb9cdc508e3e364076613c35b16e76ecfe47a60d1e3305", size = 3386843, upload-time = "2026-09-03T08:55:26.584Z" }, + { url = "https://files.pythonhosted.org/packages/e9/a4/4f9106d317b14a80aefea9f0e3a8d07ef25f856a7607eb7f5ab894281fcb/tokenizers-0.23.2-cp310-abi3-manylinux_2_31_riscv64.whl", hash = "sha256:12f0835dc2ee694746a76adf7b1567d4346a4a502ebe93fb1f5f80ea49799b78", size = 3577314, upload-time = "2026-09-03T08:55:20.825Z" }, + { url = "https://files.pythonhosted.org/packages/8d/6a/1552b70fb0d9ab074fd3fc961435d01364e79c9058481822c3af6e8d402c/tokenizers-0.23.2-cp310-abi3-musllinux_1_2_aarch64.whl", hash = "sha256:eb2f9c8a24da020ea8c11a01a19c1c2547912d92121ae4a01cfbca46125dee40", size = 9967367, upload-time = "2026-09-03T08:55:33.188Z" }, + { url = "https://files.pythonhosted.org/packages/06/01/3ccb3a956c7528b2507b8a9714155c4baf86af593039db6ea375dd0c96c3/tokenizers-0.23.2-cp310-abi3-musllinux_1_2_armv7l.whl", hash = "sha256:f486f402f6f9abee5bb032553736813af0c710a86b2e0ca592634c55cea1f835", size = 9811886, upload-time = "2026-09-03T08:55:35.642Z" }, + { url = "https://files.pythonhosted.org/packages/fa/73/7038e612d48bda1599457f712f6bd3854eae1a9dc9c13aa47f835349db48/tokenizers-0.23.2-cp310-abi3-musllinux_1_2_i686.whl", hash = "sha256:bef235815a067b2648caf6dcc7a71091b0b0fff9ee8057f6451eb9335fae52ef", size = 10146224, upload-time = "2026-09-03T08:55:38.391Z" }, + { url = "https://files.pythonhosted.org/packages/b5/d8/8e9e4e0b287a338d8f88976729628c9d22e8a54cfaf9777018a7f7cb58a0/tokenizers-0.23.2-cp310-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:5c56bda1511921587789163e524d196ed8284174ac23abd7685d5ea8da6c4718", size = 10256304, upload-time = "2026-09-03T08:55:40.977Z" }, + { url = "https://files.pythonhosted.org/packages/f3/1f/c79a01f671a49728ebb0b61f7ff9ea45663b66cab40bc0858e9859b25c16/tokenizers-0.23.2-cp310-abi3-win32.whl", hash = "sha256:debf978920d93ba9c219bd67cc4bbfaf912c9039e41e7a28b91ec15e3728c95a", size = 2592809, upload-time = "2026-09-03T08:55:48.02Z" }, + { url = "https://files.pythonhosted.org/packages/db/f7/0a69ac6b82dbccf3f71add938a161c497952749294b8dd6dfe03a819dc40/tokenizers-0.23.2-cp310-abi3-win_amd64.whl", hash = "sha256:2e96f5699d5249c9c64aa8412e044f727aae3a4098cf830f9901ec1afc361cde", size = 2863236, upload-time = "2026-09-03T08:55:46.193Z" }, + { url = "https://files.pythonhosted.org/packages/d7/b0/dee84cb44175be1b4c35bd2f770727494e78f0bb38e571a623ade94dbebb/tokenizers-0.23.2-cp310-abi3-win_arm64.whl", hash = "sha256:e49c394456dd9985787fec76132438ba3fb8911f857b1bf3d40119f9292d41aa", size = 2729352, upload-time = "2026-09-03T08:55:44.345Z" }, ] [[package]] @@ -7915,7 +7920,7 @@ wheels = [ [[package]] name = "transformers" -version = "5.8.1" +version = "5.16.1" source = { registry = "https://pypi.org/simple" } dependencies = [ { name = "huggingface-hub" }, @@ -7930,9 +7935,9 @@ dependencies = [ { name = "tqdm" }, { name = "typer" }, ] -sdist = { url = "https://files.pythonhosted.org/packages/e7/e6/4134ea2fbea322cddc7ffc94a0d8ee47fe32ce8e876b320cd37d88edfc4d/transformers-5.8.1.tar.gz", hash = "sha256:4dd5b6de4105725104d84fd6abd74b305f4debfc251b38c648ee5dd087cf543b", size = 8532019, upload-time = "2026-05-13T03:21:57.234Z" } +sdist = { url = "https://files.pythonhosted.org/packages/ad/2e/ba418680ab901dae269360bb8642485eae04f1af91ee2ebb8bd6f3607305/transformers-5.16.1.tar.gz", hash = "sha256:17b0eac726ddc55e84ac58946063e0c6d37fd000c456b581f050ea0f4e822869", size = 9650542, upload-time = "2026-08-26T14:48:58.789Z" } wheels = [ - { url = "https://files.pythonhosted.org/packages/fc/b1/8be7e7ef0b5200491312201918b6125ef9c9df9dd0f0240ccef9ac824e6b/transformers-5.8.1-py3-none-any.whl", hash = "sha256:5340fb95962162cdfdae5cc91d7f8fedd92ed75216c1154c5e1f590fcf56dd0e", size = 10632882, upload-time = "2026-05-13T03:21:52.876Z" }, + { url = "https://files.pythonhosted.org/packages/0d/4d/ee3728674c0bbc637bb4af88ccf0be697f92e4e90b55f5dc110c44d61b61/transformers-5.16.1-py3-none-any.whl", hash = "sha256:2f2d5b98a5ad3718713653734298fa620754ed683702a635ebb587df3ed29c7e", size = 12080592, upload-time = "2026-08-26T14:48:55.083Z" }, ] [[package]] From 7b7faa647f5f1abb7fbdd7cdd8db5fba327a2ea8 Mon Sep 17 00:00:00 2001 From: aviv ron Date: Tue, 8 Sep 2026 19:37:21 +0300 Subject: [PATCH 2/5] fix(tests): stop _patched_pipeline leaking its mock onto transformers.pipeline Closes #121. _patched_pipeline patched transformers.pipelines.pipeline before transformers.pipeline. mock.patch.__enter__ records the current value so it can restore it, and transformers is a lazy module: resolving transformers.pipeline when it is not yet cached on the top-level module goes through transformers.pipelines. With the submodule patched first, the second patch read back the mock, recorded it as "the original", and faithfully restored it on exit. The mock then stayed installed for the rest of the process, and load() resolves `from transformers import pipeline` at call time, so every later real transcription picked it up. One of the mocks in this file raises the "does not recognize this architecture" ValueError, which _unsupported_architecture_error converts into ImportError: transformers 5.16.0 cannot load the ASR model 'ibm-granite/granite-speech-5.0-470m-turboctc': its architecture requires transformers>=5.16 -- naming the installed version as too old for itself, from a GPU test that had nothing to do with the unit test that leaked. That false trail is the reason this is worth more than a one-line diff of explanation. Swapping the two patches is the whole fix: transformers.pipeline is now read while transformers.pipelines is still real, so both record and restore the real function. Why it went unnoticed: the leak only occurs when transformers.pipeline is not already cached on the top-level module, which depends on what else ran first. tests/unit/test_asr.py alone restores correctly; the full tests/unit/ directory leaks. CI always runs the full suite, so CI always leaked. TestPatchedPipelineRestores guards both attributes, and forces the lazy-resolve precondition with transformers.__dict__.pop("pipeline", None) rather than relying on collection order. Verified to fail on the old ordering (assert is not ) and pass on the new one, so it is a real guard rather than a tautology. Verified on transformers 5.16.1 and 5.8.1: full tests/unit/ is 287 passed / 1448 skipped on both, and a probe asserting transformers.pipeline is the real function after the session passes on both. ruff check and format clean. Note this fixes the misleading diagnosis, not the GPU-tier failures it was masking: the leaked vLLM engine that starves later GPU tests is a separate teardown defect (#123) and is deliberately left for its own PR. Signed-off-by: aviv ron --- tests/unit/test_asr.py | 68 +++++++++++++++++++++++++++++++++++++++--- 1 file changed, 64 insertions(+), 4 deletions(-) diff --git a/tests/unit/test_asr.py b/tests/unit/test_asr.py index 24010c0a..38c45a57 100644 --- a/tests/unit/test_asr.py +++ b/tests/unit/test_asr.py @@ -401,16 +401,76 @@ def test_config_not_mutated(self): @contextlib.contextmanager def _patched_pipeline(factory): - """Patch both lookup paths: `from transformers import pipeline` re-resolves to - transformers.pipelines.pipeline, but transformers caches it on the top-level - module after the first access, so a later test would get the real one.""" + """Patch both lookup paths ``ASRTranscriber.load`` may resolve through. + + ``load()`` does ``from transformers import pipeline`` at call time, which + reads the top-level attribute; transformers is a lazy module, so that + attribute may not exist yet and gets resolved from ``transformers.pipelines`` + on first access. Both therefore need patching. + + **The order matters and is load-bearing.** ``transformers.pipeline`` must be + patched FIRST. ``mock.patch.__enter__`` records the current value so it can + restore it, and if ``transformers.pipelines.pipeline`` were replaced first, + resolving ``transformers.pipeline`` would return *that mock* and record it as + the original -- which the patch then faithfully restores on exit, leaving the + mock installed for the rest of the process. See + ``TestPatchedPipelineRestores``. + """ with ( - mock.patch("transformers.pipelines.pipeline", factory), mock.patch("transformers.pipeline", factory), + mock.patch("transformers.pipelines.pipeline", factory), ): yield +class TestPatchedPipelineRestores: + """``_patched_pipeline`` must leave ``transformers.pipeline`` as it found it. + + Regression guard for a leak that was expensive to diagnose. With the two + patches in the wrong order the helper restored its own mock instead of the + real function, so every later real ``pipeline()`` call in the session got it. + Because one of the mocks in this file raises the "does not recognize this + architecture" ValueError, ``_unsupported_architecture_error`` then reported a + bogus "requires transformers>=5.16" ImportError from an unrelated GPU test -- + naming the installed version as too old for itself. + + The leak only surfaces when ``transformers.pipeline`` is not already cached on + the top-level module, which is why a full-suite run reproduced it and this + file alone did not. The test forces that precondition instead of depending on + collection order. + """ + + def test_pipeline_attribute_is_restored(self): + import transformers + + real = transformers.pipeline # resolve once, to compare against + # Force the lazy-resolve path, which is what makes the ordering matter. + transformers.__dict__.pop("pipeline", None) + + sentinel = mock.Mock(side_effect=ValueError("this mock must not escape")) + with _patched_pipeline(sentinel): + from transformers import pipeline as inside + + assert inside is sentinel, "patch did not take effect" + + from transformers import pipeline as after + + assert after is not sentinel, ( + "_patched_pipeline leaked its mock onto transformers.pipeline; " + "check the patch order in the helper" + ) + assert after is real + + def test_pipelines_submodule_attribute_is_restored(self): + import transformers.pipelines + + real = transformers.pipelines.pipeline + sentinel = mock.Mock(side_effect=ValueError("this mock must not escape")) + with _patched_pipeline(sentinel): + assert transformers.pipelines.pipeline is sentinel + assert transformers.pipelines.pipeline is real + + class TestResolveTorchDtype: """asr_dtype resolution. bfloat16-on-CUDA is the default, but overridable.""" From 8a07b33ca3dc8d2b819442c09c3690d138e8e0e9 Mon Sep 17 00:00:00 2001 From: aviv ron Date: Wed, 9 Sep 2026 14:55:01 +0300 Subject: [PATCH 3/5] fix(vllm): alias full_attention so vLLM <=0.25 can load a 5.16-written config Closes #122. transformers 5.16 renamed the layer type "attention" to "full_attention" and rewrites the value inside PreTrainedConfig.__init__, so it is now the only spelling a written config can carry -- writing the old one back does not help. vLLM <=0.25 keys its Granite-hybrid layer table on the old name: # vllm/model_executor/models/granitemoehybrid.py:319 ALL_DECODER_LAYER_TYPES = {"attention": ..., "mamba": ...} layer_class = ALL_DECODER_LAYER_TYPES[config.layer_types[layer_idx]] KeyError: 'full_attention' register() now aliases the new name onto the same layer class. Reproduced on both 0.19.1 and 0.20.2; upstream fixed it in 0.26.0 by adding the key themselves, and setdefault makes this a no-op there, so the block can be deleted on a version bump without touching anything else. Two things this repairs, which is why it is not only a test fix: * The vLLM equivalence tier -- 14 failures plus 19 nested. Those tests build their upstream reference with transformers, save_pretrained it, and hand the directory to vllm.LLM; the saved config now says full_attention and its architectures field routes to vLLM's stale class, so the *control group* never starts. GraniteSwitchForCausalLM was never affected: it builds layers through a closure and does not consult that table. * Serving a stock Granite 4.0 hybrid checkpoint. In a venv built from this branch -- transformers>=5.16 comes from our own audio extra, vLLM is pinned <=0.25 -- `vllm serve ibm-granite/granite-4.0-micro` crashed. Composed Granite Switch checkpoints did not; the published 4.2-30b carries layer_types: ["full_attention", ...] and serves correctly. Why it lives in register() and not a test fixture: the lookup runs inside vLLM's *spawned* engine-core process, so nothing patched in the parent survives. vLLM loads its plugins in that process during init (v1/engine/core.py calls load_general_plugins()), reading the vllm.general_plugins entry point we already declare. It is the only hook we own that executes there. Aliasing adds a name, not behaviour: both keys resolve to the identical class object, so the decoder layer built is the one that was built before the rename. The broad except is deliberate -- if a future vLLM moves the module or the dict, skipping silently costs us this KeyError again, whereas raising would take down every engine start. A larger alternative was considered and rejected as too big for this change: retargeting the whole equivalence tier off granitemoehybrid onto the families we actually ship. That gap is real -- granite (4.1/4.2/30b) has no equivalence coverage in any tier and granitemoe has HF-only -- but it is separate work. tests/vllm/test_plugin_registration.py asserts both keys are present and resolve to the same object, and that register() stays re-entrant as its docstring promises. It runs in-process rather than through the usual subprocess wrapper because register() creates no engine and so opens no CUDA context; it skips cleanly where vLLM is absent. The assertions hold on 0.26+ too, so the test does not need editing when the alias becomes redundant. Verified: ruff check and format clean over 210 files; tests/unit/ is 286 passed / 1448 skipped on transformers 5.16.1, unchanged. The alias itself only demonstrates on GPU -- the proof is the 14 equivalence failures going green. Not addressed here: #123, the leaked vLLM engine that starves later GPU tests (8 failures + 8 errors), which is its own PR; and #127's tokenizer-fetch flake. Also untouched is the second stale comparison, is_hybrid in vllm/config/model.py, still broken in 0.28.0 -- we sidestep it by not declaring IsHybrid. Signed-off-by: aviv ron --- src/granite_switch/vllm/__init__.py | 25 +++++++++++ tests/vllm/test_plugin_registration.py | 62 ++++++++++++++++++++++++++ 2 files changed, 87 insertions(+) create mode 100644 tests/vllm/test_plugin_registration.py diff --git a/src/granite_switch/vllm/__init__.py b/src/granite_switch/vllm/__init__.py index c6549df8..741017a7 100644 --- a/src/granite_switch/vllm/__init__.py +++ b/src/granite_switch/vllm/__init__.py @@ -43,6 +43,31 @@ def register(): """ from vllm import ModelRegistry + # vLLM <=0.25 keys its Granite-hybrid layer table on the pre-5.16 spelling of + # a layer type, so a config written by transformers >=5.16 -- which renamed + # "attention" to "full_attention" and rewrites it inside + # PreTrainedConfig.__init__ -- raises KeyError at engine init: + # + # ALL_DECODER_LAYER_TYPES[config.layer_types[layer_idx]] + # KeyError: 'full_attention' + # + # Both names denote the same layer class, so aliasing is not a behaviour + # change: it teaches vLLM to accept the only spelling transformers can now + # produce. This must happen here rather than in a test fixture because the + # lookup runs in vLLM's spawned engine-core process, and this plugin hook is + # the one thing we own that executes there (v1/engine/core.py calls + # load_general_plugins() during init). setdefault makes it a no-op on 0.26+, + # where upstream added the key themselves, so the block can be deleted + # whenever we move off the 0.19/0.20 line. See issue #122. + try: + from vllm.model_executor.models import granitemoehybrid as _gmh + + _gmh.ALL_DECODER_LAYER_TYPES.setdefault( + "full_attention", _gmh.ALL_DECODER_LAYER_TYPES["attention"] + ) + except Exception: # pragma: no cover - must never block registration + pass + # Register config with transformers AutoConfig try: from transformers import AutoConfig diff --git a/tests/vllm/test_plugin_registration.py b/tests/vllm/test_plugin_registration.py new file mode 100644 index 00000000..3bd9432d --- /dev/null +++ b/tests/vllm/test_plugin_registration.py @@ -0,0 +1,62 @@ +# SPDX-License-Identifier: Apache-2.0 +"""What ``granite_switch.vllm.register()`` puts in place before any model loads. + +In-process on purpose, unlike most of ``tests/vllm/``: ``register()`` only +mutates registries and creates no engine, so it never opens a CUDA context and +needs no subprocess wrapper. +""" + +import importlib.util + +import pytest + +pytestmark = pytest.mark.skipif( + importlib.util.find_spec("vllm") is None, reason="requires vLLM installed" +) + + +class TestLegacyLayerTypeAlias: + """``register()`` must make vLLM accept the post-5.16 layer-type spelling. + + transformers 5.16 renamed the layer type ``attention`` to ``full_attention`` + and rewrites it inside ``PreTrainedConfig.__init__``, so it is now the only + spelling a written config can carry. vLLM <=0.25 keys + ``ALL_DECODER_LAYER_TYPES`` on the old name and dies with + ``KeyError: 'full_attention'`` while building decoder layers. + + The alias lives in ``register()`` because the lookup happens in vLLM's + spawned engine-core process, which loads plugins itself -- a fixture in the + test process would not reach it. Upstream added the key in 0.26.0, where this + becomes a no-op; the assertions hold either way, which is what lets the block + be deleted on a version bump without touching this test. + """ + + def test_full_attention_resolves_to_the_attention_layer(self): + from vllm.model_executor.models import granitemoehybrid as gmh + + from granite_switch.vllm import register + + register() + + table = gmh.ALL_DECODER_LAYER_TYPES + assert "attention" in table, "upstream renamed or removed the legacy key" + assert "full_attention" in table, ( + "register() did not alias the post-5.16 layer-type spelling; a config " + "written by transformers >=5.16 will KeyError at engine init" + ) + assert table["full_attention"] is table["attention"], ( + "full_attention must resolve to the very same layer class, not a copy " + "-- aliasing is meant to add a name, not a behaviour" + ) + + def test_register_is_re_entrant(self): + """``register()`` is called once per process by vLLM, but its docstring + promises re-entrancy and ``setdefault`` is what keeps that true here.""" + from vllm.model_executor.models import granitemoehybrid as gmh + + from granite_switch.vllm import register + + register() + first = gmh.ALL_DECODER_LAYER_TYPES["full_attention"] + register() + assert gmh.ALL_DECODER_LAYER_TYPES["full_attention"] is first From 3e45e3dcafd6d34868488839c83b44aef731af53 Mon Sep 17 00:00:00 2001 From: aviv ron Date: Wed, 9 Sep 2026 15:39:08 +0300 Subject: [PATCH 4/5] docs+tests(audio): fold in the MoE-audio documentation and its coverage Absorbs feature/moe-audio-support so it does not need a PR of its own: that branch carried no functional code, only documentation plus the tests that back it. Taken from staging/feature/moe-audio-support @ 8bf0bb3. What came across: * docs/AUDIO.md -- the audio cascade over a pure sparse MoE base. Hand-merged, not overwritten: this file had already diverged here (63+/25- from the TurboCTC edits, which touch the same sections). Verified afterwards that both sides survive -- the TurboCTC default, the transformers>=5.16 requirement and the 120s chunker window on one side, the granitemoe material on the other. * tests/composer/test_audio_marker_output_row.py -- the marker/reserved-row copy is now parametrized over MLP topology as well as embedding tying, taking it from 2 cases to 4 (untied/tied x dense/sparse-MoE). The fixup only ever touches embedding rows, so both topologies must behave identically; the sparse arm is what would notice if a shared-MLP-shaped assumption crept into the model construction it runs against. * tests/composer/test_granitemoe_audio_compose.py -- new, 9 tests over 4 classes: audio does not resurrect the shared MLP, does not widen the adapter surface and leaves no zero-width parameter; the control LUT goes stale on the marker and refresh fixes it idempotently and passes the validator; the marker output row; and survival across save/load. Deliberately NOT taken: a 5-line comment block in src/granite_switch/composer/tokenizer_setup.py noting that the <|unused_N|> convention also holds on granitemoe bases. True, but prose, and keeping src/ out of this commit makes the "no functional change" claim checkable rather than asserted -- `git show --stat` shows no src/ path at all. Verified: the two test files give 32 passed here; ruff check and format clean over 211 files. One thing a reviewer should not misread. The composer tier on this base reports pre-existing failures that predate this commit and are unrelated to it: TestGraniteMoeSR fails 6 of 8 even in isolation, in 0.37s, with "ValueError: not enough values to unpack (expected 5, got 3)". Public main is 15 commits behind staging and is missing #119, which changed composer return signatures and updated test_granitemoe_compose_e2e.py to match; the two are out of step here. It has gone unnoticed because the public repo's automatic CI runs only tests/unit/, so tests/composer/ is unchecked on every PR. #119 is expected to reach public main shortly, which resolves it. The test file imported above is based on #116, before that refactor, so it matches the signatures this base has. Signed-off-by: aviv ron --- docs/AUDIO.md | 34 ++- .../composer/test_audio_marker_output_row.py | 46 +++- .../composer/test_granitemoe_audio_compose.py | 241 ++++++++++++++++++ 3 files changed, 307 insertions(+), 14 deletions(-) create mode 100644 tests/composer/test_granitemoe_audio_compose.py diff --git a/docs/AUDIO.md b/docs/AUDIO.md index 7bf536c3..5e382ecd 100644 --- a/docs/AUDIO.md +++ b/docs/AUDIO.md @@ -225,10 +225,25 @@ in automatically — callers send standard chat messages, no manual marker neede Both Granite chat-template families are supported, detected at compose time: -| Family | Models | How the marker is emitted | +| Family | Role markers | How the marker is emitted | |---|---|---| -| `granite_format` | 4.0 / 4.1 | An `elif` added to the existing content-part loop | -| `chatml` | 4.2 | A flattening block, since ChatML has no content-part loop | +| `granite_format` | `<\|start_of_role\|>` | An `elif` added to the existing content-part loop | +| `chatml` | `<\|im_start\|>` | A flattening block, since ChatML has no content-part loop | + +**Nothing on the audio path is architecture-specific.** The compose-time gate is +`model_type.startswith("granite")` and the injection above keys off the *detected +template family*, never the architecture — so a dense base and a pure sparse MoE +base (`granitemoe`, no `shared_mlp`) go down identical code, and the marker's +output-row fixup only ever touches embedding rows. What does differ is the +adapter surface, not the audio: see *Audio + adapters* below. + +A base whose tokenizer carries **no chat template at all** is *not* refused: +`configure_audio_chat_template` warns and returns, and compose completes. The +checkpoint then carries `asr_enabled: true` while its template emits no +`<|audio|>` marker, so an audio content part on the chat path is dropped rather +than transcribed — offline `llm.generate` with a hand-written marker still works. +Compose from the instruct-tuned sibling, or supply a template first. (A template +that *is* present but whose family cannot be identified does raise.) The ChatML template consumes `message.content` as a string (`{%- set content = message.content | string %}`), so a multimodal parts *list* @@ -274,7 +289,7 @@ is rejected. Compose therefore copies a reserved `<|unused_N|>` row into the marker's row, so its logit is identical to a token the base model was trained not to emit, for -every hidden state. On the tied path (4.0/4.1) that row is shared with the input +every hidden state. On a tied-embedding base that row is shared with the input embedding, which is inert here: the marker is replaced by transcript ids before the decoder runs, and a marker without a matching audio item is rejected up-front, so the marker's input row is never read. @@ -289,7 +304,9 @@ rather than the basis. If a vocabulary has no reserved slots, compose warns and leaves the row as generated. Note the inventory is not stable across releases (4.1 has 69 unused ids, 4.2 has 72), so nothing should depend on a specific count or id range — -`find_reserved_never_emitted_token_id` looks them up each time. +`find_reserved_never_emitted_token_id` looks them up each time. The rows are +present on `granitemoe` bases too, so this policy needs no architecture-specific +fallback. ## Limitations (alpha) @@ -325,6 +342,13 @@ tokens as usual, and `embed_input_ids` applies the same token-exchange rewrite (control → substitute id) used for text — so an audio request that activates an adapter behaves identically to the text equivalent. +On a **pure sparse MoE** base the adapter surface is attention-only (`qkv_proj`, +`o_proj`), because there is no `shared_mlp` for the MLP-side groups to attach to — +see [SUPPORTED_MODELS.md](SUPPORTED_MODELS.md#pure-sparse-moe-granitemoe). Where +no adapter library targets such a base yet, compose an adapter-free audio skin +with `--built-in-adapters base --enable-audio`: the marker, its output row and the +control-LUT sizing are all independent of how many adapters are present. + ## Tests Everything on the audio path carries the `audio` marker, so the whole tier selects diff --git a/tests/composer/test_audio_marker_output_row.py b/tests/composer/test_audio_marker_output_row.py index f6c25955..81ee15a4 100644 --- a/tests/composer/test_audio_marker_output_row.py +++ b/tests/composer/test_audio_marker_output_row.py @@ -13,9 +13,15 @@ trained to keep low, into the marker's row. Covered here: * the reserved-token lookup (found, absent, highest-id-wins) -* the row copy on the untied path (Granite 4.2, distinct ``lm_head``) -* the row copy on the tied path (Granite 4.0/4.1, shared matrix) +* the row copy on the untied path (distinct ``lm_head``) +* the row copy on the tied path (shared matrix) * neighbouring rows are left alone + +The row copy is also parametrized over the MLP topology. The fixup only ever +touches embedding rows, so a dense base and a pure sparse MoE base +(``shared_intermediate_size == 0``, no ``shared_mlp`` module at all) must behave +identically -- and if a shared-MLP-shaped assumption ever creeps into the model +construction the fixup runs against, the sparse arm is what notices. """ import pytest @@ -36,8 +42,30 @@ _BYSTANDER_ID = 254 -def _tiny_config(tie: bool) -> GraniteSwitchConfig: +# (tie, sparse_moe) ids for the row-copy cases. +_TOPOLOGIES = [ + pytest.param(False, False, id="untied-dense"), + pytest.param(True, False, id="tied-dense"), + pytest.param(False, True, id="untied-sparse_moe"), + pytest.param(True, True, id="tied-sparse_moe"), +] + + +def _tiny_config(tie: bool, sparse_moe: bool = False) -> GraniteSwitchConfig: + # shared_intermediate_size == 0 is upstream's encoding for "no shared MLP", + # so it is a meaningful value, never a falsy one. A layer needs at least one + # MLP path, hence the expert bank. + moe_fields = ( + { + "shared_intermediate_size": 0, + "num_local_experts": 4, + "num_experts_per_tok": 2, + } + if sparse_moe + else {} + ) return GraniteSwitchConfig( + **moe_fields, vocab_size=300, hidden_size=64, intermediate_size=128, @@ -104,9 +132,9 @@ class TestInitializeAudioMarkerOutputRow: so it is worth pinning separately from the control tokens. """ - @pytest.mark.parametrize("tie", [False, True], ids=["untied", "tied"]) - def test_marker_row_matches_reserved_row(self, tie): - model = GraniteSwitchForCausalLM(_tiny_config(tie=tie)) + @pytest.mark.parametrize(("tie", "sparse_moe"), _TOPOLOGIES) + def test_marker_row_matches_reserved_row(self, tie, sparse_moe): + model = GraniteSwitchForCausalLM(_tiny_config(tie=tie, sparse_moe=sparse_moe)) head = model.get_output_embeddings().weight reserved_before = head[_RESERVED_ID].clone() @@ -121,9 +149,9 @@ def test_marker_row_matches_reserved_row(self, tie): # The source row is copied from, not moved. assert torch.equal(head[_RESERVED_ID], reserved_before) - @pytest.mark.parametrize("tie", [False, True], ids=["untied", "tied"]) - def test_other_rows_untouched(self, tie): - model = GraniteSwitchForCausalLM(_tiny_config(tie=tie)) + @pytest.mark.parametrize(("tie", "sparse_moe"), _TOPOLOGIES) + def test_other_rows_untouched(self, tie, sparse_moe): + model = GraniteSwitchForCausalLM(_tiny_config(tie=tie, sparse_moe=sparse_moe)) head = model.get_output_embeddings().weight bystander = head[_BYSTANDER_ID].clone() control_rows = head[[250, 251]].clone() diff --git a/tests/composer/test_granitemoe_audio_compose.py b/tests/composer/test_granitemoe_audio_compose.py new file mode 100644 index 00000000..78a635fa --- /dev/null +++ b/tests/composer/test_granitemoe_audio_compose.py @@ -0,0 +1,241 @@ +# SPDX-License-Identifier: Apache-2.0 +"""The audio cascade over a pure sparse MoE base, on both switch engines. + +Nothing on the audio path is architecture-specific by design: the compose-time +gate is ``model_type.startswith("granite")`` and the marker injection keys off +the *detected template family*, not the architecture. That is exactly why the +intersection had no coverage — each side is tested thoroughly on its own and +neither suite crosses into the other: + +* every ``tests/vllm/`` and composer MoE fixture sets ``num_local_experts=0`` + or a dense ``shared_mlp``; +* every audio test runs on a dense base. + +So a regression that only shows up when both hold — a shared-MLP-shaped +assumption in the marker fixup, or a control-LUT size derived from something the +expert bank changes — would pass both suites. These cases are CPU-only, +synthetic and ungated for the same reason +``tests/composer/test_control_lut_refresh.py`` is: gating is what hid the +``--switch-type multi --enable-audio`` crash. + +``shared_intermediate_size == 0`` is upstream's own encoding for "no shared +MLP", so it is a *meaningful* value and never a falsy one. +""" + +import pytest +import torch + +from granite_switch.composer.compose_granite_switch import ( + initialize_control_token_output_rows, + refresh_switch_control_lut, +) +from granite_switch.config import GraniteSwitchConfig +from granite_switch.hf import GraniteSwitchForCausalLM +from tests.shared.granitemoe_compose import ( + CONTROL_TOKEN_ID, + DEFAULT_GEOMETRY, + VOCAB_SIZE, + create_base_model, + create_lora_adapter, +) + +SWITCH_TYPES = ["single", "multi"] + +# The marker is added after the control tokens and is NOT one of them, so it +# pushes the vocabulary one past what the control table was sized for. That +# off-by-one is what makes a stale table specific to --enable-audio. +_AUDIO_MARKER_ID = VOCAB_SIZE +_VOCAB_WITH_AUDIO = VOCAB_SIZE + 1 +# Stand-in for a reserved <|unused_N|> row. The lookup that finds a real one is +# covered against real tokenizers in test_audio_marker_output_row.py; here the +# id only has to be a row the fixup can copy from. +_RESERVED_ID = 260 +_BYSTANDER_ID = 261 + + +@pytest.fixture(scope="module", params=SWITCH_TYPES) +def audio_moe_model(request, tmp_path_factory): + """A composed pure sparse MoE switch model with audio recorded in its config. + + Composed through ``GraniteSwitchComposer`` rather than the compose CLI: the + CLI needs a tokenizer in the base directory, and the synthetic base + deliberately ships none. The audio-specific steps the CLI would then run — + the marker's embedding row, its output-row fixup and the control-LUT + refresh — are applied by the tests themselves, against the same functions + ``build()`` calls. + """ + from granite_switch.composer import GraniteSwitchComposer + + root = tmp_path_factory.mktemp(f"moe_audio_{request.param}") + base_path = root / "base" + adapter_path = root / "adapter" + create_base_model(base_path, DEFAULT_GEOMETRY) + create_lora_adapter(adapter_path, DEFAULT_GEOMETRY) + + model = GraniteSwitchComposer.from_base_and_adapters( + base_model_name_or_path=str(base_path), + adapter_paths=[str(adapter_path)], + adapter_token_ids=[CONTROL_TOKEN_ID], + adapter_substitute_token_ids=[1], + adapter_names=["a"], + switch_type=request.param, + asr_enabled=True, + ).eval() + + expected = "MultiSwitch" if request.param == "multi" else "SingleSwitch" + assert type(model.model.switch).__name__ == expected + return model + + +def _fresh(audio_moe_model): + """A private copy, so a mutating test cannot leak into the next one.""" + config = GraniteSwitchConfig(**audio_moe_model.config.to_dict()) + return GraniteSwitchForCausalLM(config).eval() + + +@pytest.mark.audio +class TestComposedConfig: + def test_audio_does_not_resurrect_the_shared_mlp(self, audio_moe_model): + """``asr_enabled`` must not perturb the no-shared-MLP encoding.""" + config = audio_moe_model.config + assert config.asr_enabled is True + assert config.shared_intermediate_size == 0 + layer = audio_moe_model.model.layers[-1] + assert layer.shared_mlp is None + assert layer.has_shared_mlp is False + assert layer.has_experts is True + + def test_audio_does_not_widen_the_adapter_surface(self, audio_moe_model): + """The adapter surface stays attention-only. + + A pure sparse base has no MLP-side LoRA at all, and enabling audio is + not a reason for one to appear: the marker is a tokenizer/embedding + concern, not an adaptation site. + """ + assert set(audio_moe_model.config.lora_target_modules) == { + "qkv_proj", + "o_proj", + } + + def test_no_zero_width_parameter(self, audio_moe_model): + """Skipped, not sized to zero. + + ``nn.Linear(H, 0)`` still registers a ``[0, H]`` weight, which would then + be demanded of the base checkpoint. + """ + offenders = [ + name + for name, param in audio_moe_model.named_parameters() + if 0 in tuple(param.shape) + ] + assert offenders == [] + + +@pytest.mark.audio +class TestControlLutWithMarker: + """The audio off-by-one, on a pure sparse base, on both engines.""" + + def test_marker_makes_the_table_stale_and_refresh_fixes_it(self, audio_moe_model): + model = _fresh(audio_moe_model) + before = model.model.switch.control_to_substitute_lut.numel() + + model.resize_token_embeddings(_VOCAB_WITH_AUDIO) + + # Guard the guard: if the marker ever stops pushing vocab_size past the + # table, this test silently stops covering the bug it exists for. + assert model.config.vocab_size == _VOCAB_WITH_AUDIO + assert before < _VOCAB_WITH_AUDIO + assert model.model.switch.control_to_substitute_lut.numel() == before + + assert refresh_switch_control_lut(model) is True + + lut = model.model.switch.control_to_substitute_lut + assert lut.numel() == model.config.vocab_size + assert lut[CONTROL_TOKEN_ID].item() == 1 + assert int((lut >= 0).sum()) == 1 + # Must remain a buffer, or save_pretrained drops it from the checkpoint. + assert "control_to_substitute_lut" in dict(model.model.switch.named_buffers()) + + def test_refresh_is_idempotent(self, audio_moe_model): + model = _fresh(audio_moe_model) + model.resize_token_embeddings(_VOCAB_WITH_AUDIO) + assert refresh_switch_control_lut(model) is True + assert refresh_switch_control_lut(model) is False + + def test_validator_accepts_the_refreshed_table(self, audio_moe_model): + from granite_switch.composer.validator import validate_control_lut + + model = _fresh(audio_moe_model) + model.resize_token_embeddings(_VOCAB_WITH_AUDIO) + refresh_switch_control_lut(model) + validate_control_lut(model) # raises on a one-row-short table + + +@pytest.mark.audio +class TestMarkerOutputRow: + """The never-emitted row fixup is indifferent to the MLP topology.""" + + def test_marker_row_matches_reserved_row(self, audio_moe_model): + model = _fresh(audio_moe_model) + model.resize_token_embeddings(_VOCAB_WITH_AUDIO) + head = model.get_output_embeddings().weight + reserved_before = head[_RESERVED_ID].clone() + assert not torch.equal(head[_AUDIO_MARKER_ID], reserved_before) + + initialize_control_token_output_rows(model, [_AUDIO_MARKER_ID], _RESERVED_ID) + + head = model.get_output_embeddings().weight + assert torch.equal(head[_AUDIO_MARKER_ID], reserved_before) + assert torch.equal(head[_RESERVED_ID], reserved_before) + + def test_marker_and_control_rows_share_the_fixup(self, audio_moe_model): + """Compose passes control ids and the marker through one call.""" + model = _fresh(audio_moe_model) + model.resize_token_embeddings(_VOCAB_WITH_AUDIO) + head = model.get_output_embeddings().weight + reserved_before = head[_RESERVED_ID].clone() + bystander_before = head[_BYSTANDER_ID].clone() + + initialize_control_token_output_rows( + model, [CONTROL_TOKEN_ID, _AUDIO_MARKER_ID], _RESERVED_ID + ) + + head = model.get_output_embeddings().weight + assert torch.equal(head[CONTROL_TOKEN_ID], reserved_before) + assert torch.equal(head[_AUDIO_MARKER_ID], reserved_before) + assert torch.equal(head[_BYSTANDER_ID], bystander_before) + + +@pytest.mark.audio +class TestRoundtrip: + def test_audio_and_no_shared_mlp_survive_save_load(self, audio_moe_model, tmp_path): + """A strict reload: no ``ignore_mismatched_sizes`` crutch. + + The marker grows the vocabulary after construction, so this is also the + case where the persistent control-LUT buffer and the saved config must + agree — a shape disagreement is discarded on load and leaves the buffer + uninitialized rather than raising. + """ + model = _fresh(audio_moe_model) + model.resize_token_embeddings(_VOCAB_WITH_AUDIO) + initialize_control_token_output_rows(model, [_AUDIO_MARKER_ID], _RESERVED_ID) + refresh_switch_control_lut(model) + + save_dir = tmp_path / "composed" + model.save_pretrained(str(save_dir)) + reloaded = GraniteSwitchForCausalLM.from_pretrained(str(save_dir)).eval() + + assert reloaded.config.asr_enabled is True + assert reloaded.config.shared_intermediate_size == 0 + assert reloaded.config.vocab_size == _VOCAB_WITH_AUDIO + assert reloaded.model.layers[-1].shared_mlp is None + + lut = reloaded.model.switch.control_to_substitute_lut + assert lut.numel() == _VOCAB_WITH_AUDIO + assert lut[CONTROL_TOKEN_ID].item() == 1 + + head = reloaded.get_output_embeddings().weight + assert torch.equal( + head[_AUDIO_MARKER_ID], + model.get_output_embeddings().weight[_AUDIO_MARKER_ID], + ) From 4b5fac3fa4c83d17e1e584daa39aac045605a38b Mon Sep 17 00:00:00 2001 From: aviv ron Date: Thu, 10 Sep 2026 14:03:03 +0300 Subject: [PATCH 5/5] test(audio): drop the SingleSwitch arm from the MoE-audio compose fixture #131 removed SingleSwitch. test_granitemoe_audio_compose.py arrived from feature/moe-audio-support, which branched before that, so it still parametrized its fixture over ["single", "multi"] and asserted the switch class was SingleSwitch on the single arm -- 9 errors after the merge, every one a [single] case, while all 23 [multi] cases passed. Now SWITCH_TYPES = ["multi"] and the assertion is unconditional, which is what every other switch-parametrized file already does post-#131: test_control_lut_refresh.py:34, tests/hf/test_multi_switch.py:49, tests/vllm/test_multi_switch.py:52, tests/shared/gap_equivalence.py:15. So this converges on the established shape rather than inventing one. Nothing about the audio path changed -- the coverage that was single/multi parametrized is engine-independent (the marker's embedding and output rows, the control-LUT refresh, save/load survival), so dropping the removed engine loses no assertion. 23 passed. The wider audio tier is 128 passed / 53 skipped with one error that is this machine having no GPU ("Torch not compiled with CUDA enabled"), not a merge regression. Signed-off-by: aviv ron --- tests/composer/test_granitemoe_audio_compose.py | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/tests/composer/test_granitemoe_audio_compose.py b/tests/composer/test_granitemoe_audio_compose.py index 78a635fa..cfc79b93 100644 --- a/tests/composer/test_granitemoe_audio_compose.py +++ b/tests/composer/test_granitemoe_audio_compose.py @@ -39,7 +39,7 @@ create_lora_adapter, ) -SWITCH_TYPES = ["single", "multi"] +SWITCH_TYPES = ["multi"] # The marker is added after the control tokens and is NOT one of them, so it # pushes the vocabulary one past what the control table was sized for. That @@ -82,8 +82,7 @@ def audio_moe_model(request, tmp_path_factory): asr_enabled=True, ).eval() - expected = "MultiSwitch" if request.param == "multi" else "SingleSwitch" - assert type(model.model.switch).__name__ == expected + assert type(model.model.switch).__name__ == "MultiSwitch" return model