Useful Pretrain-Datasets - a Leon-Leee Collection

Leon-Leee 's Collections

GPT-4 generated datasets

Useful Pretrain-Datasets

Code, Math, and Reasoning related Instruct datasets

High Quality Instruct Dataset

Code Benchmarks

Useful Pretrain-Datasets

updated 21 days ago

pretrain-datasets with (maybe) good quality

Zyphra/Zyda

Viewer • Updated 15 days ago • 4.35B • 56 • 60

Note English only
HuggingFaceFW/fineweb

Viewer • Updated 7 days ago • 46B • 35.1k • 1.53k

Note English only
HuggingFaceFW/fineweb-edu

Viewer • Updated 21 days ago • 3B • 242k • 363

Note English only
m-a-p/Matrix

Viewer • Updated about 1 month ago • 2.5M • 61 • 134

Note zh-en bilingual
bigcode/the-stack-v2

Viewer • Updated Apr 23 • 5.45B • 217 • 228

Note code
bigcode/the-stack-v2-train-full-ids

Viewer • Updated 27 days ago • 60.5M • 368 • 25

Note code
togethercomputer/RedPajama-Data-V2

Viewer • Updated Jan 18 • 2.25M • 8.69k • 325
cerebras/SlimPajama-627B

Viewer • Updated Jul 7, 2023 • 2.16M • 6.94k • 388
allenai/c4

Viewer • Updated Jan 9 • 10.4B • 231k • 210
EleutherAI/pile

Updated May 3, 2023 • 788 • 369

Note english
Skywork/SkyPile-150B

Viewer • Updated Dec 7, 2023 • 1.76M • 48 • 293

Note chinese
Tele-AI/TeleChat-PTD

Updated Mar 20 • 12 • 157

Note chinese
wenge-research/yayi2_pretrain_data

Viewer • Updated Dec 29, 2023 • 1.68M • 51 • 50

Note chinese and english
wikimedia/wikipedia

Viewer • Updated Jan 9 • 61.6M • 32.5k • 423

Note multi-lingual
liwu/MNBVC

Updated Feb 12, 2023 • 3

Note chinese
oscar-corpus/oscar

Updated Mar 21 • 3.68k • 167

Note multi-lingual
oscar-corpus/colossal-oscar-1.0

Preview • Updated Oct 26, 2023 • 31 • 29

Note multi-lingual
uonlp/CulturaX

Updated Feb 4 • 21.6k • 427

Note multi-lingual
JosephusCheung/GuanacoDataset

Viewer • Updated Apr 15 • 1.17M • 103 • 510
allenai/MADLAD-400

Updated Oct 31, 2023 • 30 • 110

Note multi-lingual