hf-internal-testing/transformers_daily_ci
Updated
โข
167
โข
1
Hugging Face Internal Testing Organization
company
AI & ML interests
None defined yet.
Recent Activity
View all activity
hf-internal-testing's activity
hf-transformers-botย
updated
a
dataset
about 5 hours ago
ydshiehย
updated
a
dataset
2 days ago
lhoestqย
authored
a
paper
3 days ago
Post
1804
Introducing ๐๐
๐ข๐ง๐๐๐๐ญ๐ก: the best public math pre-training dataset with 50B+ tokens!
HuggingFaceTB/finemath
Math remains challenging for LLMs and by training on FineMath we see considerable gains over other math datasets, especially on GSM8K and MATH.
We build the dataset by:
๐ ๏ธ carefully extracting math data from Common Crawl;
๐ iteratively filtering and recalling high quality math pages using a classifier trained on synthetic annotations to identify math reasoning and deduction.
We conducted a series of ablations comparing the performance of Llama-3.2-3B-Base after continued pre-training on FineMath and observe notable gains compared to the baseline model and other public math datasets.
We hope this helps advance the performance of LLMs on math and reasoning! ๐
Weโre also releasing all the ablation models as well as the evaluation code.
HuggingFaceTB/finemath-6763fb8f71b6439b653482c2
HuggingFaceTB/finemath
Math remains challenging for LLMs and by training on FineMath we see considerable gains over other math datasets, especially on GSM8K and MATH.
We build the dataset by:
๐ ๏ธ carefully extracting math data from Common Crawl;
๐ iteratively filtering and recalling high quality math pages using a classifier trained on synthetic annotations to identify math reasoning and deduction.
We conducted a series of ablations comparing the performance of Llama-3.2-3B-Base after continued pre-training on FineMath and observe notable gains compared to the baseline model and other public math datasets.
We hope this helps advance the performance of LLMs on math and reasoning! ๐
Weโre also releasing all the ablation models as well as the evaluation code.
HuggingFaceTB/finemath-6763fb8f71b6439b653482c2
Post
1479
Introducing Moonshine Web: real-time speech recognition running 100% locally in your browser!
๐ Faster and more accurate than Whisper
๐ Privacy-focused (no data leaves your device)
โก๏ธ WebGPU accelerated (w/ WASM fallback)
๐ฅ Powered by ONNX Runtime Web and Transformers.js
Demo: webml-community/moonshine-web
Source code: https://github.com/huggingface/transformers.js-examples/tree/main/moonshine-web
๐ Faster and more accurate than Whisper
๐ Privacy-focused (no data leaves your device)
โก๏ธ WebGPU accelerated (w/ WASM fallback)
๐ฅ Powered by ONNX Runtime Web and Transformers.js
Demo: webml-community/moonshine-web
Source code: https://github.com/huggingface/transformers.js-examples/tree/main/moonshine-web
Post
1463
In the past seven days, the Diffusers team has shipped:
1. Two new video models
2. One new image model
3. Two new quantization backends
4. Three new fine-tuning scripts
5. Multiple fixes and library QoL improvements
Coffee on me if someone can guess 1 - 4 correctly.
1. Two new video models
2. One new image model
3. Two new quantization backends
4. Three new fine-tuning scripts
5. Multiple fixes and library QoL improvements
Coffee on me if someone can guess 1 - 4 correctly.
- 1 reply
Post
6320
We outperform Llama 70B with Llama 3B on hard math by scaling test-time compute ๐ฅ
How? By combining step-wise reward models with tree search algorithms :)
We show that smol models can match or exceed the performance of their much larger siblings when given enough "time to think"
We're open sourcing the full recipe and sharing a detailed blog post.
In our blog post we cover:
๐ Compute-optimal scaling: How we implemented DeepMind's recipe to boost the mathematical capabilities of open models at test-time.
๐ Diverse Verifier Tree Search (DVTS): An unpublished extension we developed to the verifier-guided tree search technique. This simple yet effective method improves diversity and delivers better performance, particularly at large test-time compute budgets.
๐งญ Search and Learn: A lightweight toolkit for implementing search strategies with LLMs and built for speed with vLLM
Here's the links:
- Blog post: HuggingFaceH4/blogpost-scaling-test-time-compute
- Code: https://github.com/huggingface/search-and-learn
Enjoy!
How? By combining step-wise reward models with tree search algorithms :)
We show that smol models can match or exceed the performance of their much larger siblings when given enough "time to think"
We're open sourcing the full recipe and sharing a detailed blog post.
In our blog post we cover:
๐ Compute-optimal scaling: How we implemented DeepMind's recipe to boost the mathematical capabilities of open models at test-time.
๐ Diverse Verifier Tree Search (DVTS): An unpublished extension we developed to the verifier-guided tree search technique. This simple yet effective method improves diversity and delivers better performance, particularly at large test-time compute budgets.
๐งญ Search and Learn: A lightweight toolkit for implementing search strategies with LLMs and built for speed with vLLM
Here's the links:
- Blog post: HuggingFaceH4/blogpost-scaling-test-time-compute
- Code: https://github.com/huggingface/search-and-learn
Enjoy!
- 2 replies
Post
1591
Made a HF Dataset editor a la gg sheets here:
lhoestq/dataset-spreadsheets
With Dataset Spreadsheets:
โ๏ธ Edit datasets in the UI
๐ Share link with collaborators
๐ Use locally in DuckDB or Python
Available for the 100,000+ parquet datasets on HF :)
With Dataset Spreadsheets:
โ๏ธ Edit datasets in the UI
๐ Share link with collaborators
๐ Use locally in DuckDB or Python
Available for the 100,000+ parquet datasets on HF :)
Post
900
Performance leap: TGI v3 is out. Processes 3x more tokens, 13x faster than vLLM on long prompts. Zero config !
3x more tokens.
By reducing our memory footprint, weโre able to ingest many more tokens and more dynamically than before. A single L4 (24GB) can handle 30k tokens on llama 3.1-8B, while vLLM gets barely 10k. A lot of work went into reducing the footprint of the runtime and its effect are best seen on smaller constrained environments.
13x faster
On long prompts (200k+ tokens) conversation replies take 27.5s in vLLM, while it takes only 2s in TGI. How so ? We keep the initial conversation around, so when a new reply comes in, we can answer almost instantly. The overhead of the lookup is ~5us. Thanks @Dani รซl de Kok for the beast data structure.
Zero config
Thatโs it. Remove all the flags your are using and youโre likely to get the best performance. By evaluating the hardware and model, TGI carefully selects automatic values to give best performance. In production, we donโt have any flags anymore in our deployments. We kept all existing flags around, they may come in handy in niche scenarios.
Read more: https://huggingface.co/docs/text-generation-inference/conceptual/chunking
3x more tokens.
By reducing our memory footprint, weโre able to ingest many more tokens and more dynamically than before. A single L4 (24GB) can handle 30k tokens on llama 3.1-8B, while vLLM gets barely 10k. A lot of work went into reducing the footprint of the runtime and its effect are best seen on smaller constrained environments.
13x faster
On long prompts (200k+ tokens) conversation replies take 27.5s in vLLM, while it takes only 2s in TGI. How so ? We keep the initial conversation around, so when a new reply comes in, we can answer almost instantly. The overhead of the lookup is ~5us. Thanks @Dani รซl de Kok for the beast data structure.
Zero config
Thatโs it. Remove all the flags your are using and youโre likely to get the best performance. By evaluating the hardware and model, TGI carefully selects automatic values to give best performance. In production, we donโt have any flags anymore in our deployments. We kept all existing flags around, they may come in handy in niche scenarios.
Read more: https://huggingface.co/docs/text-generation-inference/conceptual/chunking
Post
2017
Introducing a high-quality open-preference dataset to further this line of research for image generation.
Despite being such an inseparable component for modern image generation, open preference datasets are a rarity!
So, we decided to work on one with the community!
Check it out here:
https://huggingface.co/blog/image-preferences
Despite being such an inseparable component for modern image generation, open preference datasets are a rarity!
So, we decided to work on one with the community!
Check it out here:
https://huggingface.co/blog/image-preferences
- 7 replies
Post
2087
The Control family of Flux from
@black-forest-labs
should be discussed more!
It enables structural controls like ControlNets while being significantly less expensive to run!
So, we're working on a Control LoRA training script ๐ค
It's still WIP, so go easy:
https://github.com/huggingface/diffusers/pull/10130
It enables structural controls like ControlNets while being significantly less expensive to run!
So, we're working on a Control LoRA training script ๐ค
It's still WIP, so go easy:
https://github.com/huggingface/diffusers/pull/10130
Post
2268
Introducing TTS WebGPU: The first ever text-to-speech web app built with WebGPU acceleration! ๐ฅ High-quality and natural speech generation that runs 100% locally in your browser, powered by OuteTTS and Transformers.js. ๐ค Try it out yourself!
Demo: webml-community/text-to-speech-webgpu
Source code: https://github.com/huggingface/transformers.js-examples/tree/main/text-to-speech-webgpu
Model: onnx-community/OuteTTS-0.2-500M (ONNX), OuteAI/OuteTTS-0.2-500M (PyTorch)
Demo: webml-community/text-to-speech-webgpu
Source code: https://github.com/huggingface/transformers.js-examples/tree/main/text-to-speech-webgpu
Model: onnx-community/OuteTTS-0.2-500M (ONNX), OuteAI/OuteTTS-0.2-500M (PyTorch)
sayakpaulย
authored
a
paper
16 days ago
Post
1465
Let 2024 be the year of video model fine-tunes!
Check it out here:
https://github.com/a-r-r-o-w/cogvideox-factory/tree/main/training/mochi-1
Check it out here:
https://github.com/a-r-r-o-w/cogvideox-factory/tree/main/training/mochi-1
Post
3907
We just released Transformers.js v3.1 and you're not going to believe what's now possible in the browser w/ WebGPU! ๐คฏ Let's take a look:
๐ Janus from Deepseek for unified multimodal understanding and generation (Text-to-Image and Image-Text-to-Text)
๐๏ธ Qwen2-VL from Qwen for dynamic-resolution image understanding
๐ข JinaCLIP from Jina AI for general-purpose multilingual multimodal embeddings
๐ LLaVA-OneVision from ByteDance for Image-Text-to-Text generation
๐คธโโ๏ธ ViTPose for pose estimation
๐ MGP-STR for optical character recognition (OCR)
๐ PatchTST & PatchTSMixer for time series forecasting
That's right, everything running 100% locally in your browser (no data sent to a server)! ๐ฅ Huge for privacy!
Check out the release notes for more information. ๐
https://github.com/huggingface/transformers.js/releases/tag/3.1.0
Demo link (+ source code): webml-community/Janus-1.3B-WebGPU
๐ Janus from Deepseek for unified multimodal understanding and generation (Text-to-Image and Image-Text-to-Text)
๐๏ธ Qwen2-VL from Qwen for dynamic-resolution image understanding
๐ข JinaCLIP from Jina AI for general-purpose multilingual multimodal embeddings
๐ LLaVA-OneVision from ByteDance for Image-Text-to-Text generation
๐คธโโ๏ธ ViTPose for pose estimation
๐ MGP-STR for optical character recognition (OCR)
๐ PatchTST & PatchTSMixer for time series forecasting
That's right, everything running 100% locally in your browser (no data sent to a server)! ๐ฅ Huge for privacy!
Check out the release notes for more information. ๐
https://github.com/huggingface/transformers.js/releases/tag/3.1.0
Demo link (+ source code): webml-community/Janus-1.3B-WebGPU
albertvillanovaย
posted
an
update
about 1 month ago
Post
1361
๐จ How green is your model? ๐ฑ Introducing a new feature in the Comparator tool: Environmental Impact for responsible #LLM research!
๐ open-llm-leaderboard/comparator
Now, you can not only compare models by performance, but also by their environmental footprint!
๐ The Comparator calculates COโ emissions during evaluation and shows key model characteristics: evaluation score, number of parameters, architecture, precision, type... ๐ ๏ธ
Make informed decisions about your model's impact on the planet and join the movement towards greener AI!
๐ open-llm-leaderboard/comparator
Now, you can not only compare models by performance, but also by their environmental footprint!
๐ The Comparator calculates COโ emissions during evaluation and shows key model characteristics: evaluation score, number of parameters, architecture, precision, type... ๐ ๏ธ
Make informed decisions about your model's impact on the planet and join the movement towards greener AI!
Post
5502
Have you tried out ๐ค Transformers.js v3? Here are the new features:
โก WebGPU support (up to 100x faster than WASM)
๐ข New quantization formats (dtypes)
๐ 120 supported architectures in total
๐ 25 new example projects and templates
๐ค Over 1200 pre-converted models
๐ Node.js (ESM + CJS), Deno, and Bun compatibility
๐ก A new home on GitHub and NPM
Get started with
Learn more in our blog post: https://huggingface.co/blog/transformersjs-v3
โก WebGPU support (up to 100x faster than WASM)
๐ข New quantization formats (dtypes)
๐ 120 supported architectures in total
๐ 25 new example projects and templates
๐ค Over 1200 pre-converted models
๐ Node.js (ESM + CJS), Deno, and Bun compatibility
๐ก A new home on GitHub and NPM
Get started with
npm i @huggingface/transformers.Learn more in our blog post: https://huggingface.co/blog/transformersjs-v3
- 3 replies
Post
2632
Native tensor parallel has landed in transformers!!! https://github.com/huggingface/transformers/pull/34184 thanks a lot to the torch team for their support!
Contributions are welcome to support more models! ๐ฅ
Contributions are welcome to support more models! ๐ฅ
Post
2591
It's been a while we shipped native quantization support in
We currently support
This post is just a reminder of what's possible:
1. Loading a model with a quantization config
2. Saving a model with quantization config
3. Loading a pre-quantized model
4.
5. Training and loading LoRAs into quantized checkpoints
Docs:
https://huggingface.co/docs/diffusers/main/en/quantization/bitsandbytes
diffusers ๐งจWe currently support
bistandbytes as the official backend but using others like torchao is already very simple. This post is just a reminder of what's possible:
1. Loading a model with a quantization config
2. Saving a model with quantization config
3. Loading a pre-quantized model
4.
enable_model_cpu_offload()5. Training and loading LoRAs into quantized checkpoints
Docs:
https://huggingface.co/docs/diffusers/main/en/quantization/bitsandbytes
- 1 reply
albertvillanovaย
posted
an
update
about 2 months ago
Post
1454
๐ New feature of the Comparator of the ๐ค Open LLM Leaderboard: now compare models with their base versions & derivatives (finetunes, adapters, etc.). Perfect for tracking how adjustments affect performance & seeing innovations in action. Dive deeper into the leaderboard!
๐ ๏ธ Here's how to use it:
1. Select your model from the leaderboard.
2. Load its model tree.
3. Choose any base & derived models (adapters, finetunes, merges, quantizations) for comparison.
4. Press Load.
See side-by-side performance metrics instantly!
Ready to dive in? ๐ Try the ๐ค Open LLM Leaderboard Comparator now! See how models stack up against their base versions and derivatives to understand fine-tuning and other adjustments. Easier model analysis for better insights! Check it out here: open-llm-leaderboard/comparator ๐
๐ ๏ธ Here's how to use it:
1. Select your model from the leaderboard.
2. Load its model tree.
3. Choose any base & derived models (adapters, finetunes, merges, quantizations) for comparison.
4. Press Load.
See side-by-side performance metrics instantly!
Ready to dive in? ๐ Try the ๐ค Open LLM Leaderboard Comparator now! See how models stack up against their base versions and derivatives to understand fine-tuning and other adjustments. Easier model analysis for better insights! Check it out here: open-llm-leaderboard/comparator ๐