Aurélien-Morgan CLAUDON's picture

Aurélien-Morgan CLAUDON

Aurelien-Morgan

·

AI & ML interests

None yet

Recent Activity

liked a Space about 12 hours ago

akhaliq/anychat

replied to clem's post 1 day ago

Coming back to Paris Friday to open our new Hugging Face office! We're at capacity for the party but add your name in the waiting list as we're trying to privatize the passage du Caire for extra space for robots 🤖🦾🦿 https://t.co/enkFXjWndJ

reacted to m-ric's post with 👍 1 day ago

𝐇𝐮𝐠𝐠𝐢𝐧𝐠 𝐅𝐚𝐜𝐞 𝐫𝐞𝐥𝐞𝐚𝐬𝐞𝐬 𝐏𝐢𝐜𝐨𝐭𝐫𝐨𝐧, 𝐚 𝐦𝐢𝐜𝐫𝐨𝐬𝐜𝐨𝐩𝐢𝐜 𝐥𝐢𝐛 𝐭𝐡𝐚𝐭 𝐬𝐨𝐥𝐯𝐞𝐬 𝐋𝐋𝐌 𝐭𝐫𝐚𝐢𝐧𝐢𝐧𝐠 𝟒𝐃 𝐩𝐚𝐫𝐚𝐥𝐥𝐞𝐥𝐢𝐳𝐚𝐭𝐢𝐨𝐧 🥳 🕰️ Llama-3.1-405B took 39 million GPU-hours to train, i.e. about 4.5 thousand years. 👴🏻 If they had needed all this time, we would have GPU stories from the time of Pharaoh 𓂀: "Alas, Lord of Two Lands, the shipment of counting-stones arriving from Cathay was lost to pirates, this shall delay the building of your computing temple by many moons " 🛠️ But instead, they just parallelized the training on 24k H100s, which made it take just a few months. This required parallelizing across 4 dimensions: data, tensor, context, pipeline. And it is infamously hard to do, making for bloated code repos that hold together only by magic. 🤏 𝗕𝘂𝘁 𝗻𝗼𝘄 𝘄𝗲 𝗱𝗼𝗻'𝘁 𝗻𝗲𝗲𝗱 𝗵𝘂𝗴𝗲 𝗿𝗲𝗽𝗼𝘀 𝗮𝗻𝘆𝗺𝗼𝗿𝗲! Instead of building mega-training codes, Hugging Face colleagues cooked in the other direction, towards tiny 4D parallelism libs. A team has built Nanotron, already widely used in industry. And now a team releases Picotron, a radical approach to code 4D Parallelism in just a few hundred lines of code, a real engineering prowess, making it much easier to understand what's actually happening! ⚡ 𝗜𝘁'𝘀 𝘁𝗶𝗻𝘆, 𝘆𝗲𝘁 𝗽𝗼𝘄𝗲𝗿𝗳𝘂𝗹: Counting in MFU (Model FLOPs Utilization, how much the model actually uses all the compute potential), this lib reaches ~50% on SmolLM-1.7B model with 8 H100 GPUs, which is really close to what huge libs would reach. (Caution: the team is leading further benchmarks to verify this) Go take a look 👉 https://github.com/huggingface/picotron/tree/main/picotron

View all activity

Articles

Fancy Stateful Metaflow Service + UI on Google Colab ?

Organizations

Aurelien-Morgan's activity

upvoted a paper 16 days ago

Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving

Paper • 2407.00079 • Published Jun 24 • 5

upvoted a paper 17 days ago

GRAPE: Generalizing Robot Policy via Preference Alignment

Paper • 2411.19309 • Published 22 days ago • 42

upvoted an article 24 days ago

Article

Let’s make a generation of amazing image generation models

By

•

25 days ago

• 33

upvoted a collection about 1 month ago

🚀 Trending Demo

12 items • Updated 4 days ago • 9

upvoted 3 papers about 1 month ago

The Surprising Effectiveness of Test-Time Training for Abstract Reasoning

Paper • 2411.07279 • Published Nov 11 • 3

OpenCoder: The Open Cookbook for Top-Tier Code Large Language Models

Paper • 2411.04905 • Published Nov 7 • 111

SALSA: Soup-based Alignment Learning for Stronger Adaptation in RLHF

Paper • 2411.01798 • Published Nov 4 • 8

upvoted 3 collections about 2 months ago

SmolLM2

State-of-the-art compact LLMs for on-device applications: 1.7B, 360M, 135M • 15 items • Updated 18 days ago • 194

📑 Trending Papers - October 🔟

10 items • Updated 4 days ago • 6

MobileLLM

Optimizing Sub-billion Parameter Language Models for On-Device Use Cases (ICML 2024) https://arxiv.org/abs/2402.14905 • 9 items • Updated 24 days ago • 99

upvoted 3 articles 2 months ago

Article

Fancy Stateful Metaflow Service + UI on Google Colab ?

By

•

Oct 14

• 4

Article

Fixing Gradient Accumulation

Oct 16

• 43

Article

Model Card Generator Interface: Crafting Clear Insights into AI Models

By

•

Sep 27

• 4

upvoted a paper 2 months ago

Pixtral 12B

Paper • 2410.07073 • Published Oct 9 • 61

upvoted an article 2 months ago

Article

A Short Summary of Chinese AI Global Expansion

Oct 3

• 19

upvoted an article 3 months ago

Article

Llama can now see and run on your device - welcome Llama 3.2

Sep 25

• 180

upvoted a collection 3 months ago

Llama 3.2

This collection hosts the transformers and original repos of the Llama 3.2 and Llama Guard 3 • 15 items • Updated 14 days ago • 545

upvoted 3 articles 3 months ago

Article

Introducing Community Tools on HuggingChat

Sep 16

• 34

Article

Accelerate 1.0.0

Sep 13

• 51

Article

The Environmental Impacts of AI -- Primer

By

•

Sep 3

• 33