Aurélien-Morgan CLAUDON's picture

Aurélien-Morgan CLAUDON

Aurelien-Morgan

·

AI & ML interests

None yet

Recent Activity

liked a Space about 12 hours ago

akhaliq/anychat

replied to clem's post 1 day ago

Coming back to Paris Friday to open our new Hugging Face office! We're at capacity for the party but add your name in the waiting list as we're trying to privatize the passage du Caire for extra space for robots 🤖🦾🦿 https://t.co/enkFXjWndJ

reacted to m-ric's post with 👍 1 day ago

𝐇𝐮𝐠𝐠𝐢𝐧𝐠 𝐅𝐚𝐜𝐞 𝐫𝐞𝐥𝐞𝐚𝐬𝐞𝐬 𝐏𝐢𝐜𝐨𝐭𝐫𝐨𝐧, 𝐚 𝐦𝐢𝐜𝐫𝐨𝐬𝐜𝐨𝐩𝐢𝐜 𝐥𝐢𝐛 𝐭𝐡𝐚𝐭 𝐬𝐨𝐥𝐯𝐞𝐬 𝐋𝐋𝐌 𝐭𝐫𝐚𝐢𝐧𝐢𝐧𝐠 𝟒𝐃 𝐩𝐚𝐫𝐚𝐥𝐥𝐞𝐥𝐢𝐳𝐚𝐭𝐢𝐨𝐧 🥳 🕰️ Llama-3.1-405B took 39 million GPU-hours to train, i.e. about 4.5 thousand years. 👴🏻 If they had needed all this time, we would have GPU stories from the time of Pharaoh 𓂀: "Alas, Lord of Two Lands, the shipment of counting-stones arriving from Cathay was lost to pirates, this shall delay the building of your computing temple by many moons " 🛠️ But instead, they just parallelized the training on 24k H100s, which made it take just a few months. This required parallelizing across 4 dimensions: data, tensor, context, pipeline. And it is infamously hard to do, making for bloated code repos that hold together only by magic. 🤏 𝗕𝘂𝘁 𝗻𝗼𝘄 𝘄𝗲 𝗱𝗼𝗻'𝘁 𝗻𝗲𝗲𝗱 𝗵𝘂𝗴𝗲 𝗿𝗲𝗽𝗼𝘀 𝗮𝗻𝘆𝗺𝗼𝗿𝗲! Instead of building mega-training codes, Hugging Face colleagues cooked in the other direction, towards tiny 4D parallelism libs. A team has built Nanotron, already widely used in industry. And now a team releases Picotron, a radical approach to code 4D Parallelism in just a few hundred lines of code, a real engineering prowess, making it much easier to understand what's actually happening! ⚡ 𝗜𝘁'𝘀 𝘁𝗶𝗻𝘆, 𝘆𝗲𝘁 𝗽𝗼𝘄𝗲𝗿𝗳𝘂𝗹: Counting in MFU (Model FLOPs Utilization, how much the model actually uses all the compute potential), this lib reaches ~50% on SmolLM-1.7B model with 8 H100 GPUs, which is really close to what huge libs would reach. (Caution: the team is leading further benchmarks to verify this) Go take a look 👉 https://github.com/huggingface/picotron/tree/main/picotron

View all activity

Articles

Fancy Stateful Metaflow Service + UI on Google Colab ?

Organizations

Aurelien-Morgan's activity

New activity in paris-ai-running-club/README about 2 months ago

we're running a 10K: https://lagrandecourse.fr/

#4 opened 3 months ago by

New activity in open-llm-leaderboard/comparator 2 months ago

search selection list autocomplete

#1 opened 2 months ago by

Aurelien-Morgan

New activity in blog-explorers/README 2 months ago

[Support] Community Articles

#5 opened 9 months ago by

New activity in compmicro-czb/VirtualStaining 3 months ago

Impressive

#1 opened 3 months ago by

Aurelien-Morgan

New activity in paris-ai-running-club/README 5 months ago

next run wen?

#3 opened 5 months ago by

New activity in huggingface-projects/gemma-2-2b-it 5 months ago

Duplicate for private use

#1 opened 5 months ago by

Aurelien-Morgan

New activity in huggingface/HuggingDiscussions 7 months ago

[FEEDBACK] Notifications

#6 opened over 2 years ago by

[FEEDBACK] Notifications

#6 opened over 2 years ago by