Hugging Face
Models
Datasets
Spaces
Posts
Docs
Solutions
Pricing
Log In
Sign Up
baebee
/
nuph-tokenizer
like
0
Transformers
Transformers.js
tokenizers
experimental
auto tokenizer
llama
llama 2
custom code
Inference Endpoints
License:
mit
Model card
Files
Files and versions
Community
Train
Deploy
Use this model
baebee
commited on
Oct 11, 2023
Commit
698eb08
•
1 Parent(s):
9536ae0
Update README.md
Browse files
Files changed (1)
hide
show
README.md
+2
-1
README.md
CHANGED
Viewed
@@ -12,4 +12,5 @@ tags:
12
---
13
14
# Experimental tokenizer
15
-
- Derived from the LLaMa-2 tokenizer
12
---
13
14
# Experimental tokenizer
15
+
- Derived from the LLaMa-2 tokenizer
16
+
- Inspired by the ChatML & Zephyr tokenizers