Hugging Face's logo
Dokyoon's picture
57 258

Dokyoon

leeloolee
kaki-paper's profile picture ddobokki's profile picture victor's profile picture
Ā·

AI & ML interests

ai

Recent Activity

upvoted a paper 9 days ago
GUI Agents: A Survey
reacted to m-ric's post with šŸ‘ 15 days ago
š‡š®š š š¢š§š  š…šššœšž š«šžš„šžššš¬šžš¬ šš¢šœšØš­š«šØš§, šš š¦š¢šœš«šØš¬šœšØš©š¢šœ š„š¢š› š­š”ššš­ š¬šØš„šÆšžš¬ š‹š‹šŒ š­š«ššš¢š§š¢š§š  šŸ’šƒ š©ššš«ššš„š„šžš„š¢š³ššš­š¢šØš§ šŸ„³ šŸ•°ļø Llama-3.1-405B took 39 million GPU-hours to train, i.e. about 4.5 thousand years. šŸ‘“šŸ» If they had needed all this time, we would have GPU stories from the time of Pharaoh š“‚€: "Alas, Lord of Two Lands, the shipment of counting-stones arriving from Cathay was lost to pirates, this shall delay the building of your computing temple by many moons " šŸ› ļø But instead, they just parallelized the training on 24k H100s, which made it take just a few months. This required parallelizing across 4 dimensions: data, tensor, context, pipeline. And it is infamously hard to do, making for bloated code repos that hold together only by magic. šŸ¤ š—•š˜‚š˜ š—»š—¼š˜„ š˜„š—² š—±š—¼š—»'š˜ š—»š—²š—²š—± š—µš˜‚š—“š—² š—暝—²š—½š—¼š˜€ š—®š—»š˜†š—ŗš—¼š—暝—²! Instead of building mega-training codes, Hugging Face colleagues cooked in the other direction, towards tiny 4D parallelism libs. A team has built Nanotron, already widely used in industry. And now a team releases Picotron, a radical approach to code 4D Parallelism in just a few hundred lines of code, a real engineering prowess, making it much easier to understand what's actually happening! āš” š—œš˜'š˜€ š˜š—¶š—»š˜†, š˜†š—²š˜ š—½š—¼š˜„š—²š—暝—³š˜‚š—¹: Counting in MFU (Model FLOPs Utilization, how much the model actually uses all the compute potential), this lib reaches ~50% on SmolLM-1.7B model with 8 H100 GPUs, which is really close to what huge libs would reach. (Caution: the team is leading further benchmarks to verify this) Go take a look šŸ‘‰ https://github.com/huggingface/picotron/tree/main/picotron
reacted to alimotahharynia's post with šŸ”„ 15 days ago
Here's the space for our new article that leverages LLMs with reinforcement learning to design high-quality small molecules. Check it out at https://huggingface.co/spaces/alimotahharynia/GPT-2-Drug-Generator. You can also access the article here: https://arxiv.org/abs/2411.14157. I would be happy to receive your feedback.
View all activity

Organizations

sionic-ai's profile picture MultišŸ¤–Transformers's profile picture ģøģŠ¤ķŠøėŸ­ķŠø.ķ•œźµ­'s profile picture AI Safeguard's profile picture

leeloolee's activity

liked a dataset 16 days ago
liked a model 17 days ago
liked a model 18 days ago
liked a model 23 days ago
liked a Space 24 days ago
liked a dataset about 1 month ago
liked 2 Spaces about 1 month ago
liked a model about 1 month ago
liked 5 datasets 2 months ago
liked 3 datasets 3 months ago
liked a model 3 months ago
liked 2 datasets 3 months ago