You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Preserve an explicit prepend_bos value when BridgeCore.run_with_cache() eagerly tokenizes a scalar string input.
Consume the tokenization option before forwarding the resulting token tensor, rather than leaving an ineffective prepend_bos argument on the tensor forward.
Add offline native-Bridge regression coverage for direct-vs-cached parity with both prepend_bos=True and False.
Root cause and impact
run_with_cache() converts scalar string inputs with self.to_tokens(input) before calling forward(), but did not pass the caller's prepend_bos value. to_tokens() therefore fell back to cfg.default_prepend_bos, typically True. Although the original argument was subsequently forwarded, the input was already a token tensor and could no longer be retokenized.
As a result, bridge(text, prepend_bos=False) and bridge.run_with_cache(text, prepend_bos=False) could run on different token sequences. The cached path silently included an extra BOS position, shifting logits and position-indexed activations used for attention analysis, attribution, and activation patching.
The fix lives in BridgeCore, so it preserves the shared local/remote cache contract while leaving tensor/list inputs and driver-aware device placement unchanged. This PR intentionally does not broaden into padding_side, whose tokenization support differs between TransformerBridge and RemoteBridge and is not required to fix the demonstrated bug.
No documentation change is required because the existing prepend_bos contract is already documented; this change makes run_with_cache() conform to it.
Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to not work as expected)
This change requires a documentation update
Screenshots
N/A — code-only behavior fix.
Validation
Target regression: 2 passed.
Full affected files (test_boot_native.py and test_remote_bridge.py): 62 passed.
Related model-bridge unit surface selected by run_with_cache, tokenizer, and prepend_bos: 68 passed, 6 skipped.
Offline tiny native Bridge parity: direct logits, cached logits, and hook_embed all have shape (1, 1, 8) with prepend_bos=False; maximum direct-vs-cached logit difference is 0.0.
mypy .: no issues in 424 source files.
pycln, isort, Black, and git diff --check: passed for the changed files.
Checklist:
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation (no change required; existing docs already specify this behavior)
My changes generate no new warnings
I have added tests that prove my fix is effective or that my feature works
New and existing unit tests pass locally with my changes (the affected unit-test surface passed; the complete suite was not rerun locally)
I have not rewritten tests relating to key interfaces which would affect backward compatibility
The values and failure locations are identical in both runs. This PR only changes scalar-string tokenization in run_with_cache(), while both failing tests use direct tensor forward paths.
I have therefore kept this PR focused and have not changed the unrelated numerical tolerances. Please let me know whether you would prefer the coverage job to be retried, or these base-branch failures to be handled separately.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Summary
prepend_bosvalue whenBridgeCore.run_with_cache()eagerly tokenizes a scalar string input.prepend_bosargument on the tensor forward.prepend_bos=TrueandFalse.Root cause and impact
run_with_cache()converts scalar string inputs withself.to_tokens(input)before callingforward(), but did not pass the caller'sprepend_bosvalue.to_tokens()therefore fell back tocfg.default_prepend_bos, typicallyTrue. Although the original argument was subsequently forwarded, the input was already a token tensor and could no longer be retokenized.As a result,
bridge(text, prepend_bos=False)andbridge.run_with_cache(text, prepend_bos=False)could run on different token sequences. The cached path silently included an extra BOS position, shifting logits and position-indexed activations used for attention analysis, attribution, and activation patching.The fix lives in
BridgeCore, so it preserves the shared local/remote cache contract while leaving tensor/list inputs and driver-aware device placement unchanged. This PR intentionally does not broaden intopadding_side, whose tokenization support differs betweenTransformerBridgeandRemoteBridgeand is not required to fix the demonstrated bug.No documentation change is required because the existing
prepend_boscontract is already documented; this change makesrun_with_cache()conform to it.Fixes #1624
Type of change
Screenshots
N/A — code-only behavior fix.
Validation
2 passed.test_boot_native.pyandtest_remote_bridge.py):62 passed.run_with_cache, tokenizer, andprepend_bos:68 passed, 6 skipped.hook_embedall have shape(1, 1, 8)withprepend_bos=False; maximum direct-vs-cached logit difference is0.0.mypy .: no issues in 424 source files.git diff --check: passed for the changed files.Checklist: