Hugging Face partners with Wiz Research to Improve AI Security
•
12
Nicolas Patry
Narsil
AI & ML interests
None yet
Recent Activity
posted
an
update
19 days ago
Performance leap: TGI v3 is out. Processes 3x more tokens, 13x faster than vLLM on long prompts. Zero config !
3x more tokens.
By reducing our memory footprint, we’re able to ingest many more tokens and more dynamically than before. A single L4 (24GB) can handle 30k tokens on llama 3.1-8B, while vLLM gets barely 10k. A lot of work went into reducing the footprint of the runtime and its effect are best seen on smaller constrained environments.
13x faster
On long prompts (200k+ tokens) conversation replies take 27.5s in vLLM, while it takes only 2s in TGI. How so ? We keep the initial conversation around, so when a new reply comes in, we can answer almost instantly. The overhead of the lookup is ~5us. Thanks @Daniël de Kok for the beast data structure.
Zero config
That’s it. Remove all the flags your are using and you’re likely to get the best performance. By evaluating the hardware and model, TGI carefully selects automatic values to give best performance. In production, we don’t have any flags anymore in our deployments. We kept all existing flags around, they may come in handy in niche scenarios.
Read more: https://huggingface.co/docs/text-generation-inference/conceptual/chunking
new activity
21 days ago
huggingchat/chat-ui:Your feedback on HuggingChat
Articles
Organizations
Narsil's activity
upvoted
a
paper
2 months ago
upvoted
a
paper
6 months ago
upvoted
an
article
7 months ago
Article
🧨 Diffusers welcomes Stable Diffusion 3
•
92
upvoted
a
collection
8 months ago
upvoted
a
paper
8 months ago
upvoted
a
paper
about 1 year ago