clean_up_tokenization_spaces in tokenizer_config.json?

by fahadh4ilyas - opened Jul 30, 2023

Jul 30, 2023

Why the value that should be for bos_token is moved to clean_up_tokenization_spaces while the value for bos_token becomes empty string?

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

Your need to confirm your account before you can post a new comment.

· Sign up or log in to comment