File size: 892 Bytes
cc01a5f 5d334ed cc01a5f 5d334ed 7f213ad cf5935b 7f213ad cc01a5f 7f213ad 7b35669 7f213ad 5d334ed |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 |
This is the tokenizer used by the Sabia-2 Small model.
Sabiá-2 Small is a proprietary LLM that can be used through an API endpoint, which we refer to as the "MariTalk API", or a downloadable version that can be used locally and is encrypted, known as "MariTalk Local".
The purpose of including this tokenizer is to allow you to estimate the number of tokens in your prompts and, therefore, the cost of using the model.
For example:
```python
import transformers
tokenizer = transformers.AutoTokenizer.from_pretrained("maritaca-ai/sabia-2-tokenizer-small")
prompt = "Com quantos paus se faz uma canoa?"
tokens = tokenizer.encode(prompt)
print(f'O prompt "{prompt}" contém {len(tokens)} tokens.') # It should print 11 tokens.
```
For more information on how to use the model, please refer to our documentation at [this link](https://maritaca-ai.github.io/maritalk-api/maritalk.html). |