New tokenizer.json doesn't match tokenizer.model.v3, there's no [SUFFIX], [MIDDLE] and [PREFIX] tokens.

#24

by mukel - opened Jun 1

mukel

Jun 1

•

It seems that tokenizer.json was generated using the older tokenizer.model.v3 without the FIM tokens.

Jun 1

•

Here are a few repos that have the fix:

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment