LnL-AI
/

dbrx-base-tokenizer

Transformers.js

Inference Endpoints

Model card Files Files and versions Community

Qubitium commited on Apr 3

Commit

9c651bf

•

1 Parent(s): a115410

Update README.md

Files changed (1) hide show

README.md +1 -1

README.md CHANGED Viewed

@@ -15,7 +15,7 @@ Changes:
 1. Remove non-base model tokens
 2. Keep/Add `<|pad|>` special token to make sure padding can be differentiated from eos/bos.
- "100277": {
       "content": "<|pad|>",
       "lstrip": false,
       "normalized": false,

 1. Remove non-base model tokens
 2. Keep/Add `<|pad|>` special token to make sure padding can be differentiated from eos/bos.
+ "100261": {
       "content": "<|pad|>",
       "lstrip": false,
       "normalized": false,