fix(tokenizer): do not prepend a BOS token the tokenizer does not have - #1634

fix(tokenizer): do not prepend a BOS token the tokenizer does not have#1634

jlarson4 merged 1 commit into

TransformerLensOrg:dev-4.xTransformerLensOrg/TransformerLens:dev-4.xfrom

Chinmayrawat15:fix/tokenizer-bos-prependChinmayrawat15/TransformerLens:fix/tokenizer-bos-prependCopy head branch name to clipboard