Skip to content

tests: invocation-tokenization tests depend on a live HF tokenizer download #127

Description

@aviv1ron1

Where it lives: main. Observed failing on feature/granite-turboctc-default (ef43e69), but the dependency is not branch-specific.

tests/composer/test_chat_template.py::TestInvocationFirstCharDropProperty fetches ibm-granite/granite-4.1-3b from the Hub at test time (_get_tokenizer). One CI run tokenized <context> as ['<context', '>'] instead of ['<', 'context', '>'], failing both cases:

AssertionError: invocation '<context>': first token decodes to '<context', expected '<'
AssertionError: invocation '<context>': dropping first char of the string produced
  tokens [2196, 29] but the tail of the full tokenization is [29]

Evidence it is environmental, not a code change:

  • passed in 22 of the 24 retained runs, including one ten hours earlier on the same line
  • locally, the same tokenizer gives the correct three tokens: ['<', 'context', '>'] for all four invocations

So a partial or mis-resolved tokenizer fetch on the runner. The property under test is real and worth keeping — it guards the Jinja invocation_text[1:] drop — but it should not be able to fail because of a network.

Fix: pin a tokenizer revision, vendor the tokenizer into tests/composer/fixtures/, or assert against a committed fixture. Vendoring also makes the test runnable offline, which the rest of the CPU tier already is.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    feature/granite-turboctc-defaultFound while landing the TurboCTC ASR default + transformers 5.16flaky-testNon-deterministic testprio:mediumReal bug, not blocking a mergetestsTest suite / CI harness

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions