Text Generation
Transformers
PyTorch
Safetensors
longllama
text-generation-inference
custom_code

clean_up_tokenization_spaces in tokenizer_config.json?

#3
by fahadh4ilyas - opened

Why the value that should be for bos_token is moved to clean_up_tokenization_spaces while the value for bos_token becomes empty string?

Your need to confirm your account before you can post a new comment.

Sign up or log in to comment