What is the meaning of </w> in vocab.json?

by skmda37 - opened Aug 21, 2023

Aug 21, 2023

Hi,

what is the meaning of in the tokens in vocab.json?

Thanks in advance!

Stefan

Dec 15, 2023

is end of the word token. compared to being in the middle or start of a word

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment