tiktoken-calculator
#1
by
Nhamo
- opened
Thanks a lot for this useful tool. I just noticed that the response from openapi requests has an object called usage which contains the number of tokens in the request, response and the total. Why did you feel the need to write your own api for that?
I want a tool that can count the number of tokens in website content. This is because sometimes when I paste the content into ChatGPT and receive a "context too large" error, I'd like to know exactly how many tokens are in the content.
Also, it helps me to estimate the cost for fine-tuning gpt-3.5-turbo.
Thanks a lot for the response. I discovered the usage object after I had written a tiktoken based utility and thought: did I just waste my time.