📚 LLM pretraining datasets - a HuggingFaceTB Collection

HuggingFaceTB 's Collections

📚 LLM pretraining datasets

🧩 SmolLM2 Intermdiate Checkpoints

SmolVLM2 📺 Smallest video LM ever 🤏🏻

The Ultimate Collection of Code Classifiers

SmolVLM 256M & 500M

SmolLM2

SmolVLM

💻 Local SmolLMs

Instruct datasets

🌌 Cosmopedia

Find textbooks in FineWeb with a classifier

FineWeb clustering & synthetic generations

Other: Stanford, OpenStax, khanAcademy, wikihow...

FW generation prompts

Wikipedia Science topics

Wikipedia textbooks

SFT Experiments

Decay mixture experiments

models

📚 LLM pretraining datasets

updated 3 days ago

A collection of datasets for LLM pretraining

HuggingFaceFW/fineweb

Viewer • Updated Jan 31 • 25B • 317k • 2.02k
HuggingFaceFW/fineweb-2

Viewer • Updated Jan 8 • 12.5B • 75.5k • 443

Note 🍷 Web datasets
HuggingFaceFW/fineweb-edu

Viewer • Updated Jan 31 • 3.3B • 499k • 648
mlfoundations/dclm-baseline-1.0

Preview • Updated Jul 22, 2024 • 302k • 208

Note 📚 Highly curated web datasets filtered using classifiers
HuggingFaceTB/finemath

Viewer • Updated Feb 6 • 48.3M • 11.5k • 292

Note 📐 Highly curated math pages from CommonCrawl
bigcode/the-stack

Viewer • Updated Apr 13, 2023 • 546M • 9.58k • 780
bigcode/the-stack-v2

Viewer • Updated Apr 23, 2024 • 5.45B • 4.98k • 341

Note 💻 Github code dataset
HuggingFaceTB/cosmopedia

Viewer • Updated Aug 12, 2024 • 31.1M • 22.8k • 591

Note Synthetic textbooks
HuggingFaceTB/smollm-corpus

Viewer • Updated Sep 6, 2024 • 237M • 12.4k • 309

Note Contains Cosmopedia v2 (synthetic textbooks) and Python-Edu (educational Python code)