Update tokenizer_config.json

by abhipn - opened Jun 28, 2023

base: refs/heads/main

←

from: refs/pr/7

Discussion Files changed

-4

abhipn

Jun 28, 2023

Configuring bos, eos, pad and unk tokens helps load the tokenizer successfully, otherwise getting a recursion error "RecursionError: maximum recursion depth exceeded while getting the str of an object."

Update tokenizer_config.json70ff3632

kbressem

medalpaca org Jun 28, 2023

Thanks!

kbressem changed pull request status to merged Jun 28, 2023

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

Your need to confirm your account before you can post a new comment.

· Sign up or log in to comment