nuph-tokenizer / README.md

baebee

Update README.md

3dd531b 12 months ago

preview code

raw

history blame contribute delete

No virus

256 Bytes

metadata

license: mit
tags:
  - tokenizers
  - transformers.js
  - experimental
  - auto tokenizer
  - llama
  - llama 2
  - custom code
library_name: transformers

Experimental tokenizer

Derived from the LLaMa-2 tokenizer
Inspired by the ChatML & Zephyr tokenizers