File size: 1,003 Bytes
4f34e0f |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 |
---
title: Tiktoken Calculator
emoji: 🤗
colorFrom: yellow
colorTo: orange
sdk: gradio
python_version: 3.11
app_file: app.py
pinned: false
tags: ["gpt", "tiktoken", "tokenizer"]
---
# Tiktoken Calculator
Calculate the token count for GPT-4, GPT-3.5, GPT-3, and GPT-2.
> It uses [openai/tiktoken](https://github.com/openai/tiktoken) to calculate the token count.
## How to use
### HuggingFace Spaces
Go to <https://huggingface.co/spaces/JacobLinCool/tiktoken-calculator> and try it out!
### Docker
There is a pre-built Docker image available on Docker Hub: <https://hub.docker.com/r/jacoblincool/tiktoken-calculator>
```bash
docker run -p 7860:7860 jacoblincool/tiktoken-calculator
```
If you prefer to use Docker Compose, you can clone this repository and run:
```bash
docker compose up -d
```
### Local
I use Poetry to manage dependencies.
Setup the virtual environment after cloning this repository:
```bash
poetry install
```
Then run the app:
```bash
poetry run python app.py
```
|