metadata

title: Tiktoken Calculator
emoji: 🤗
colorFrom: yellow
colorTo: orange
sdk: gradio
python_version: 3.11
app_file: app.py
pinned: false
tags:
  - gpt
  - tiktoken
  - tokenizer

Tiktoken Calculator

Calculate the token count for GPT-4, GPT-3.5, GPT-3, and GPT-2.

It uses openai/tiktoken to calculate the token count.

How to use

HuggingFace Spaces

Go to https://huggingface.co/spaces/JacobLinCool/tiktoken-calculator and try it out!

Docker

There is a pre-built Docker image available on Docker Hub: https://hub.docker.com/r/jacoblincool/tiktoken-calculator

docker run -p 7860:7860 jacoblincool/tiktoken-calculator

If you prefer to use Docker Compose, you can clone this repository and run:

docker compose up -d

Local

I use Poetry to manage dependencies.

Setup the virtual environment after cloning this repository:

poetry install

Then run the app:

poetry run python app.py