Expert Support case study: Bolstering a RAG app with LLM-as-a-Judge
โข
24
Aymeric Roucher
m-ric
AI & ML interests
MLE at Hugging Face ๐ค
LLMs, Agents, RAG, Multimodal.
Recent Activity
liked
a Space
3 days ago
andrewrreed/phoenix-arize-observability-demo
posted
an
update
3 days ago
Made a new app to visualize the LLM race โ ๐ก๐ผ ๐๐๐ฟ๐ผ๐ฝ๐ฒ๐ฎ๐ป ๐ฐ๐ผ๐บ๐ฝ๐ฎ๐ป๐ ๐ถ๐ป ๐๐ต๐ฒ ๐๐ผ๐ฝ ๐ญ๐ฌ ๐ช๐บโ
See the app here ๐ https://huggingface.co/spaces/m-ric/llm-race-to-the-top
I've adapted an app by @andrewrreed that tracks progress of LLMs (https://huggingface.co/spaces/andrewrreed/closed-vs-open-arena-elo), on the Chatbot Arena leaderboard, to compare companies from different countries.
The outcome is quite sad, as a Frenchman and European.
The top 10 is exclusively US ๐บ๐ธ and Chinese ๐จ๐ณ companies (after great Chinese LLM releases recently, like the Qwen2.5 series), with the notable exception of Mistral AI ๐ซ๐ท.
American companies are making fast progress, Chinese ones even faster. Europe is at risk of being left behind. And the EU AI Act hasn't even come into force yet to slow down the EU market. We need to wake up ๐ฌ
โ ๏ธ Caution: This Chatbot Arena ELO ranking is not the most accurate, especially at high scores like this, because LLM makers can game it to some extent.
Articles
Organizations
m-ric's activity
posted
an
update
3 days ago
Post
1033
Made a new app to visualize the LLM race โ ๐ก๐ผ ๐๐๐ฟ๐ผ๐ฝ๐ฒ๐ฎ๐ป ๐ฐ๐ผ๐บ๐ฝ๐ฎ๐ป๐ ๐ถ๐ป ๐๐ต๐ฒ ๐๐ผ๐ฝ ๐ญ๐ฌ ๐ช๐บโ
See the app here ๐ m-ric/llm-race-to-the-top
I've adapted an app by @andrewrreed that tracks progress of LLMs ( andrewrreed/closed-vs-open-arena-elo), on the Chatbot Arena leaderboard, to compare companies from different countries.
The outcome is quite sad, as a Frenchman and European.
The top 10 is exclusively US ๐บ๐ธ and Chinese ๐จ๐ณ companies (after great Chinese LLM releases recently, like the Qwen2.5 series), with the notable exception of Mistral AI ๐ซ๐ท.
American companies are making fast progress, Chinese ones even faster. Europe is at risk of being left behind. And the EU AI Act hasn't even come into force yet to slow down the EU market. We need to wake up ๐ฌ
โ ๏ธ Caution: This Chatbot Arena ELO ranking is not the most accurate, especially at high scores like this, because LLM makers can game it to some extent.
See the app here ๐ m-ric/llm-race-to-the-top
I've adapted an app by @andrewrreed that tracks progress of LLMs ( andrewrreed/closed-vs-open-arena-elo), on the Chatbot Arena leaderboard, to compare companies from different countries.
The outcome is quite sad, as a Frenchman and European.
The top 10 is exclusively US ๐บ๐ธ and Chinese ๐จ๐ณ companies (after great Chinese LLM releases recently, like the Qwen2.5 series), with the notable exception of Mistral AI ๐ซ๐ท.
American companies are making fast progress, Chinese ones even faster. Europe is at risk of being left behind. And the EU AI Act hasn't even come into force yet to slow down the EU market. We need to wake up ๐ฌ
โ ๏ธ Caution: This Chatbot Arena ELO ranking is not the most accurate, especially at high scores like this, because LLM makers can game it to some extent.
posted
an
update
4 days ago
Post
753
๐ก๐ฒ๐ ๐น๐ฒ๐ฎ๐ฑ๐ฒ๐ฟ๐ฏ๐ผ๐ฎ๐ฟ๐ฑ ๐ฟ๐ฎ๐ป๐ธ๐ ๐๐๐ ๐ ๐ณ๐ผ๐ฟ ๐๐๐ -๐ฎ๐-๐ฎ-๐ท๐๐ฑ๐ด๐ฒ: ๐๐น๐ฎ๐บ๐ฎ-๐ฏ.๐ญ-๐ณ๐ฌ๐ ๐๐ผ๐ฝ๐ ๐๐ต๐ฒ ๐ฟ๐ฎ๐ป๐ธ๐ถ๐ป๐ด๐! ๐งโโ๏ธ
Evaluating systems is critical during prototyping and in production, and LLM-as-a-judge has become a standard technique to do it.
First, what is "LLM-as-a-judge"?
๐ It's a very useful technique for evaluating LLM outputs. If anything you're evaluating cannot be properly evaluated with deterministic criteria, like the "politeness" of an LLM output, or how faithful it is to an original source, you can use LLM-judge instead : prompt another LLM with "Here's an LLM output, please rate this on criterion {criterion} on a scale of 1 to 5", then parse the number from its output, and voilร , you get your score.
๐ง But who judges the judge?
How can you make sure your LLM-judge is reliable? You can have a specific dataset annotated with scores provided by human judges, and compare how LLM-judge scores correlate with human judge scores.
๐ Before even running that benchmark, to get you started, there's a new option to get you started: a leaderboard that measures how well different model perform as judges!
And the outcome is surprising, models come in quite different orders from what we're used to in general rankings: probably some have much better bias mitigation than others!
Take a deeper look here ๐ https://huggingface.co/blog/arena-atla
Evaluating systems is critical during prototyping and in production, and LLM-as-a-judge has become a standard technique to do it.
First, what is "LLM-as-a-judge"?
๐ It's a very useful technique for evaluating LLM outputs. If anything you're evaluating cannot be properly evaluated with deterministic criteria, like the "politeness" of an LLM output, or how faithful it is to an original source, you can use LLM-judge instead : prompt another LLM with "Here's an LLM output, please rate this on criterion {criterion} on a scale of 1 to 5", then parse the number from its output, and voilร , you get your score.
๐ง But who judges the judge?
How can you make sure your LLM-judge is reliable? You can have a specific dataset annotated with scores provided by human judges, and compare how LLM-judge scores correlate with human judge scores.
๐ Before even running that benchmark, to get you started, there's a new option to get you started: a leaderboard that measures how well different model perform as judges!
And the outcome is surprising, models come in quite different orders from what we're used to in general rankings: probably some have much better bias mitigation than others!
Take a deeper look here ๐ https://huggingface.co/blog/arena-atla
posted
an
update
4 days ago
Post
238
Lifehack of the day:
Adding "r.jina.ai/" before any url transforms it in Markdown using Jina AI's Reader! Here with @cyrilzakka 's blog post.
Adding "r.jina.ai/" before any url transforms it in Markdown using Jina AI's Reader! Here with @cyrilzakka 's blog post.
Reacted to
cfahlgren1's
post with โค๏ธ
5 days ago
Post
2866
You can clean and format datasets entirely in the browser with a few lines of SQL.
In this post, I replicate the process @mlabonne used to clean the new microsoft/orca-agentinstruct-1M-v1 dataset.
The cleaning process consists of:
- Joining the separate splits together / add split column
- Converting string messages into list of structs
- Removing empty system prompts
https://huggingface.co/blog/cfahlgren1/the-beginners-guide-to-cleaning-a-dataset
Here's his new cleaned dataset: mlabonne/orca-agentinstruct-1M-v1-cleaned
In this post, I replicate the process @mlabonne used to clean the new microsoft/orca-agentinstruct-1M-v1 dataset.
The cleaning process consists of:
- Joining the separate splits together / add split column
- Converting string messages into list of structs
- Removing empty system prompts
https://huggingface.co/blog/cfahlgren1/the-beginners-guide-to-cleaning-a-dataset
Here's his new cleaned dataset: mlabonne/orca-agentinstruct-1M-v1-cleaned
- 1 reply
posted
an
update
5 days ago
Post
763
๐ Meta teams use a fine-tuned Llama model to fix production issues in seconds
One of Meta's engineering teams shared how they use a fine-tuned small Llama (Llama-2-7B, so not even a very recent model) to identify the root cause of production issues with 42% accuracy.
๐ค 42%, is that not too low?
โก๏ธ Usually, whenever there's an issue in production, engineers dive into recent code changes to find the offending commit. At Meta's scale (thousands of daily changes), this is like finding a needle in a haystack.
๐ก So when the LLM-based suggestion is right, it cuts incident resolution time from hours to seconds!
How did they do it?
๐ Two-step approach:
โฃ Heuristics (code ownership, directory structure, runtime graphs) reduce thousands of potential changes to a manageable set
โฃ Fine-tuned Llama 2 7B ranks the most likely culprits
๐ Training pipeline:
โฃ Continued pre-training on Meta's internal docs and wikis
โฃ Supervised fine-tuning on past incident investigations
โฃ Training data mimicked real-world constraints (2-20 potential changes per incident)
๐ฎ Now future developments await:
โฃ Language models could handle more of the incident response workflow (runbooks, mitigation, post-mortems)
โฃ Improvements in model reasoning should boost accuracy further
Read it in full ๐ https://www.tryparity.com/blog/how-meta-uses-llms-to-improve-incident-response
One of Meta's engineering teams shared how they use a fine-tuned small Llama (Llama-2-7B, so not even a very recent model) to identify the root cause of production issues with 42% accuracy.
๐ค 42%, is that not too low?
โก๏ธ Usually, whenever there's an issue in production, engineers dive into recent code changes to find the offending commit. At Meta's scale (thousands of daily changes), this is like finding a needle in a haystack.
๐ก So when the LLM-based suggestion is right, it cuts incident resolution time from hours to seconds!
How did they do it?
๐ Two-step approach:
โฃ Heuristics (code ownership, directory structure, runtime graphs) reduce thousands of potential changes to a manageable set
โฃ Fine-tuned Llama 2 7B ranks the most likely culprits
๐ Training pipeline:
โฃ Continued pre-training on Meta's internal docs and wikis
โฃ Supervised fine-tuning on past incident investigations
โฃ Training data mimicked real-world constraints (2-20 potential changes per incident)
๐ฎ Now future developments await:
โฃ Language models could handle more of the incident response workflow (runbooks, mitigation, post-mortems)
โฃ Improvements in model reasoning should boost accuracy further
Read it in full ๐ https://www.tryparity.com/blog/how-meta-uses-llms-to-improve-incident-response
Reacted to
reach-vb's
post with ๐ฅ
6 days ago
Post
4017
What a brilliant week for Open Source AI!
Qwen 2.5 Coder by Alibaba - 0.5B / 1.5B / 3B / 7B / 14B/ 32B (Base + Instruct) Code generation LLMs, with 32B tackling giants like Gemnini 1.5 Pro, Claude Sonnet
Qwen/qwen25-coder-66eaa22e6f99801bf65b0c2f
LLM2CLIP from Microsoft - Leverage LLMs to train ultra-powerful CLIP models! Boosts performance over the previous SOTA by ~17%
microsoft/llm2clip-672323a266173cfa40b32d4c
Athene v2 Chat & Agent by NexusFlow - SoTA general LLM fine-tuned from Qwen 2.5 72B excels at Chat + Function Calling/ JSON/ Agents
Nexusflow/athene-v2-6735b85e505981a794fb02cc
Orca Agent Instruct by Microsoft - 1 million instruct pairs covering text editing, creative writing, coding, reading comprehension, etc - permissively licensed
microsoft/orca-agentinstruct-1M-v1
Ultravox by FixieAI - 70B/ 8B model approaching GPT4o level, pick any LLM, train an adapter with Whisper as Audio Encoder
reach-vb/ultravox-audio-language-model-release-67373b602af0a52b2a88ae71
JanusFlow 1.3 by DeepSeek - Next iteration of their Unified MultiModal LLM Janus with RectifiedFlow
deepseek-ai/JanusFlow-1.3B
Common Corpus by Pleais - 2,003,039,184,047 multilingual, commercially permissive and high quality tokens!
PleIAs/common_corpus
I'm sure I missed a lot, can't wait for the next week!
Put down in comments what I missed! ๐ค
Qwen 2.5 Coder by Alibaba - 0.5B / 1.5B / 3B / 7B / 14B/ 32B (Base + Instruct) Code generation LLMs, with 32B tackling giants like Gemnini 1.5 Pro, Claude Sonnet
Qwen/qwen25-coder-66eaa22e6f99801bf65b0c2f
LLM2CLIP from Microsoft - Leverage LLMs to train ultra-powerful CLIP models! Boosts performance over the previous SOTA by ~17%
microsoft/llm2clip-672323a266173cfa40b32d4c
Athene v2 Chat & Agent by NexusFlow - SoTA general LLM fine-tuned from Qwen 2.5 72B excels at Chat + Function Calling/ JSON/ Agents
Nexusflow/athene-v2-6735b85e505981a794fb02cc
Orca Agent Instruct by Microsoft - 1 million instruct pairs covering text editing, creative writing, coding, reading comprehension, etc - permissively licensed
microsoft/orca-agentinstruct-1M-v1
Ultravox by FixieAI - 70B/ 8B model approaching GPT4o level, pick any LLM, train an adapter with Whisper as Audio Encoder
reach-vb/ultravox-audio-language-model-release-67373b602af0a52b2a88ae71
JanusFlow 1.3 by DeepSeek - Next iteration of their Unified MultiModal LLM Janus with RectifiedFlow
deepseek-ai/JanusFlow-1.3B
Common Corpus by Pleais - 2,003,039,184,047 multilingual, commercially permissive and high quality tokens!
PleIAs/common_corpus
I'm sure I missed a lot, can't wait for the next week!
Put down in comments what I missed! ๐ค
posted
an
update
6 days ago
Post
1338
Great feature alert: ๐ฌ๐ผ๐ ๐ฐ๐ฎ๐ป ๐ป๐ผ๐ ๐๐๐ฒ ๐ฎ๐ป๐ ๐ฆ๐ฝ๐ฎ๐ฐ๐ฒ ๐ฎ๐ ๐ฎ ๐๐ผ๐ผ๐น ๐ณ๐ผ๐ฟ ๐๐ผ๐๐ฟ ๐๐ฟ๐ฎ๐ป๐๐ณ๐ผ๐ฟ๐บ๐ฒ๐ฟ๐.๐ฎ๐ด๐ฒ๐ป๐! ๐ ๏ธ๐ฅ๐ฅ
This lets you take the coolest spaces, like FLUX.1-dev, and use them in agentic workflows with a few lines of code! ๐งโ๐ป
On the video below, I set up my fake vacation pictures where I'm awesome at surfing (I'm really not) ๐
Head to the doc to learn this magic ๐ https://huggingface.co/docs/transformers/main/en/agents_advanced#import-a-space-as-a-tool-
This lets you take the coolest spaces, like FLUX.1-dev, and use them in agentic workflows with a few lines of code! ๐งโ๐ป
On the video below, I set up my fake vacation pictures where I'm awesome at surfing (I'm really not) ๐
Head to the doc to learn this magic ๐ https://huggingface.co/docs/transformers/main/en/agents_advanced#import-a-space-as-a-tool-
posted
an
update
10 days ago
Post
368
๐ ๐ฒ๐๐ฎ ๐๐ฒ๐ฎ๐บ ๐ท๐๐๐ ๐ฑ๐ฟ๐ผ๐ฝ๐ฝ๐ฒ๐ฑ ๐๐ต๐ฒ ๐ณ๐ถ๐ฟ๐๐ ๐ช๐ฎ๐๐ฒ๐ฟ๐บ๐ฎ๐ฟ๐ธ๐ถ๐ป๐ด ๐บ๐ผ๐ฑ๐ฒ๐น ๐๐ต๐ฎ๐ ๐ป๐ผ๐ ๐ฒ๐ฑ๐ถ๐ ๐ฐ๐ฎ๐ป ๐ฏ๐ฟ๐ฒ๐ฎ๐ธ!๐ก๏ธ
๐ค Ever heard of watermarking? It's a technique that allows you to mark in an image its original source. It's our best shield against AI-generated deepfakes, or content stolen from artists! ๐จ
๐ญ Watermarking systems are actually a pair of models: a watermark embedder that applies the watermark on the image, and its corresponding decoder that should detect the original watermark.
โ But current methods were very limited: they can only apply and detect the watermark on your image as a whole. So, if you're an attacker it's easy to break: just crop it! add text on top! or whatever, really, anything would work to break the watermark.
A team of researchers at Meta was not happy with this. ๐ค
So to withstand real-world attacks, they decided to make a watermarking model that would also work on any sub-part of the image. It's a real paradigm shift: they consider watermarking not as an image classification task, but as an image segmentation task!
๐๏ธ ๐๐ฟ๐ฐ๐ต๐ถ๐๐ฒ๐ฐ๐๐๐ฟ๐ฒ
โธ The "Embedder" (a variational autoencoder + embedder, 1.1M parameters in total) encodes a n-bit message into a watermark signal that is added to the original image
โธ [Only during training] The "Augmenter" randomly distorts the image: masks parts, crops, resizes, compresses. It's basically torture at this point.
โธ The "Extractor" (a vision transformer, or ViT, with 96M parameters) then re-extracts the message from the distorted image, by predicting a (1+n) vector per pixel to predict the watermarked parts and decode corresponding messages.
The performance blows existing models out of the water, they even created new tasks (segmentation-related) just to grok them!
Gerat work @pierrefdz and @tomsander1998 !
Paper here ๐ Watermark Anything with Localized Messages (2411.07231)
๐ค Ever heard of watermarking? It's a technique that allows you to mark in an image its original source. It's our best shield against AI-generated deepfakes, or content stolen from artists! ๐จ
๐ญ Watermarking systems are actually a pair of models: a watermark embedder that applies the watermark on the image, and its corresponding decoder that should detect the original watermark.
โ But current methods were very limited: they can only apply and detect the watermark on your image as a whole. So, if you're an attacker it's easy to break: just crop it! add text on top! or whatever, really, anything would work to break the watermark.
A team of researchers at Meta was not happy with this. ๐ค
So to withstand real-world attacks, they decided to make a watermarking model that would also work on any sub-part of the image. It's a real paradigm shift: they consider watermarking not as an image classification task, but as an image segmentation task!
๐๏ธ ๐๐ฟ๐ฐ๐ต๐ถ๐๐ฒ๐ฐ๐๐๐ฟ๐ฒ
โธ The "Embedder" (a variational autoencoder + embedder, 1.1M parameters in total) encodes a n-bit message into a watermark signal that is added to the original image
โธ [Only during training] The "Augmenter" randomly distorts the image: masks parts, crops, resizes, compresses. It's basically torture at this point.
โธ The "Extractor" (a vision transformer, or ViT, with 96M parameters) then re-extracts the message from the distorted image, by predicting a (1+n) vector per pixel to predict the watermarked parts and decode corresponding messages.
The performance blows existing models out of the water, they even created new tasks (segmentation-related) just to grok them!
Gerat work @pierrefdz and @tomsander1998 !
Paper here ๐ Watermark Anything with Localized Messages (2411.07231)
Reacted to
maxiw's
post with โค๏ธ๐๐ฅ
11 days ago
Post
4550
I was curious to see what people post here on HF so I created a dataset with all HF Posts:
maxiw/hf-posts
Some interesting stats:
Top 5 Authors by Total Impressions:
-----------------------------------
@merve : 171,783 impressions (68 posts)
@fdaudens : 135,253 impressions (81 posts)
@singhsidhukuldeep : 122,591 impressions (81 posts)
@akhaliq : 119,526 impressions (78 posts)
@MonsterMMORPG : 112,500 impressions (45 posts)
Top 5 Users by Number of Reactions Given:
----------------------------------------
@osanseviero : 1278 reactions
@clem : 910 reactions
@John6666 : 899 reactions
@victor : 674 reactions
@samusenps : 655 reactions
Top 5 Most Used Reactions:
-------------------------
โค๏ธ: 7048 times
๐ฅ: 5921 times
๐: 4856 times
๐: 2549 times
๐ค: 2065 times
Some interesting stats:
Top 5 Authors by Total Impressions:
-----------------------------------
@merve : 171,783 impressions (68 posts)
@fdaudens : 135,253 impressions (81 posts)
@singhsidhukuldeep : 122,591 impressions (81 posts)
@akhaliq : 119,526 impressions (78 posts)
@MonsterMMORPG : 112,500 impressions (45 posts)
Top 5 Users by Number of Reactions Given:
----------------------------------------
@osanseviero : 1278 reactions
@clem : 910 reactions
@John6666 : 899 reactions
@victor : 674 reactions
@samusenps : 655 reactions
Top 5 Most Used Reactions:
-------------------------
โค๏ธ: 7048 times
๐ฅ: 5921 times
๐: 4856 times
๐: 2549 times
๐ค: 2065 times
- 10 replies
posted
an
update
11 days ago
Post
3667
๐ง๐ต๐ฒ ๐ป๐ฒ๐
๐ ๐ฏ๐ถ๐ด ๐๐ผ๐ฐ๐ถ๐ฎ๐น ๐ป๐ฒ๐๐๐ผ๐ฟ๐ธ ๐ถ๐ ๐ป๐ผ๐ ๐ฆ, ๐ถ๐'๐ ๐๐๐ฏ ๐ฃ๐ผ๐๐๐! [INSERT STONKS MEME WITH LASER EYES]
See below: I got 105k impressions since regularly posting Hub Posts, coming close to my 275k on Twitter!
โ๏ธ Computed with the great dataset maxiw/hf-posts
โ๏ธ Thanks to Qwen2.5-Coder-32B for showing me how to access dict attributes in a SQL request!
cc @merve who's far in front of me
See below: I got 105k impressions since regularly posting Hub Posts, coming close to my 275k on Twitter!
โ๏ธ Computed with the great dataset maxiw/hf-posts
โ๏ธ Thanks to Qwen2.5-Coder-32B for showing me how to access dict attributes in a SQL request!
cc @merve who's far in front of me
-
- 9 replies
posted
an
update
13 days ago
Post
2344
A non-Instruct LLM assistant is mostly useless. ๐ง
Since it's mostly a model trained to complete text, when you ask it a question like "What to do during a stopover in Paris?", it can just go on and on adding more details to your question instead of answering, which would be valid to complete text from its training corpus, but not to answer questions.
โก๏ธ So the post-training stage includes an important Instruction tuning step where you teach your model how to be useful : answer questions, be concise, be polite... RLHF is a well known technique for this.
For people interested to understand how this step works, the folks at Adaptive ML have made a great guide!
Read it here ๐ https://www.adaptive-ml.com/post/from-zero-to-ppo
Since it's mostly a model trained to complete text, when you ask it a question like "What to do during a stopover in Paris?", it can just go on and on adding more details to your question instead of answering, which would be valid to complete text from its training corpus, but not to answer questions.
โก๏ธ So the post-training stage includes an important Instruction tuning step where you teach your model how to be useful : answer questions, be concise, be polite... RLHF is a well known technique for this.
For people interested to understand how this step works, the folks at Adaptive ML have made a great guide!
Read it here ๐ https://www.adaptive-ml.com/post/from-zero-to-ppo
posted
an
update
14 days ago
Post
3151
๐ค๐๐ฒ๐ป๐ฎ.๐ฑ-๐๐ผ๐ฑ๐ฒ๐ฟ-๐ฏ๐ฎ๐: ๐ป๐ฒ๐ ๐ฏ๐ฒ๐๐-๐ถ๐ป-๐ฐ๐น๐ฎ๐๐ ๐ผ๐ฝ๐ฒ๐ป ๐ฐ๐ผ๐ฑ๐ถ๐ป๐ด ๐บ๐ผ๐ฑ๐ฒ๐น, ๐ฏ๐ฒ๐ฎ๐๐ ๐๐ฃ๐ง-๐ฐ๐ผ ๐ผ๐ป ๐บ๐ผ๐๐ ๐ฐ๐ผ๐ฑ๐ถ๐ป๐ด ๐ฏ๐ฒ๐ป๐ฐ๐ต๐บ๐ฎ๐ฟ๐ธ๐!๐ฅ
๐ช It's the first time Open-Source coding model of this size class that clearly matches GPT-4o's coding capabilities!
โจ Completes the previous two Qwen 2.5 Coder release with 4 new size: 0.5B, 3B, 14B, 32B
๐ Support long context up to 128K (for the 14B and 32B models)
โ Drop-in replacement to GPT-4o as a coding assistant on Cursor or for Artifacts!
๐ค Models available right now on the Hub, under Apache 2.0 license!
They have setup a crazy Artifacts demo, you should go have a look!
๐ Qwen/Qwen2.5-Coder-Artifacts
๐ช It's the first time Open-Source coding model of this size class that clearly matches GPT-4o's coding capabilities!
โจ Completes the previous two Qwen 2.5 Coder release with 4 new size: 0.5B, 3B, 14B, 32B
๐ Support long context up to 128K (for the 14B and 32B models)
โ Drop-in replacement to GPT-4o as a coding assistant on Cursor or for Artifacts!
๐ค Models available right now on the Hub, under Apache 2.0 license!
They have setup a crazy Artifacts demo, you should go have a look!
๐ Qwen/Qwen2.5-Coder-Artifacts
posted
an
update
14 days ago
Post
777
๐๐ฟ๐ฒ ๐๐ฐ๐ฎ๐น๐ถ๐ป๐ด ๐น๐ฎ๐๐ ๐ผ๐๐ฒ๐ฟ? ๐ ๐ฟ๐ฒ๐ฝ๐ผ๐ฟ๐ ๐ณ๐ฟ๐ผ๐บ ๐๐ต๐ฒ ๐๐ป๐ณ๐ผ๐ฟ๐บ๐ฎ๐๐ถ๐ผ๐ป ๐ฎ๐ป๐ป๐ผ๐๐ป๐ฐ๐ฒ๐ฑ ๐๐ต๐ฎ๐ ๐ข๐ฝ๐ฒ๐ป๐๐ ๐ถ๐ ๐๐ฒ๐ฒ๐ถ๐ป๐ด ๐ฑ๐ถ๐บ๐ถ๐ป๐ถ๐๐ต๐ถ๐ป๐ด ๐ฟ๐ฒ๐๐๐ฟ๐ป๐ ๐ณ๐ฟ๐ผ๐บ ๐๐ฐ๐ฎ๐น๐ถ๐ป๐ด ๐๐ฝ ๐๐ต๐ฒ ๐ป๐ฒ๐
๐ ๐๐ฃ๐ง ๐บ๐ผ๐ฑ๐ฒ๐น๐.
๐ What are scaling laws? These are empiric laws that say "Every time you increase compute spent in training 10-fold, your LLM's performance will go up by a predictable tick". Of course, they apply only if you train your model with the right methods.
The image below illustrates it: they're from a paper by Google, "Scaling Autoregressive Models for Content-Rich Text-to-Image Generation", and they show how quality and instruction following of models improve when you scale the model up (which is equivalent to scaling up the compute spent in training).
โก๏ธ These scaling laws have immense impact: they triggered the largest gold rush ever, with companies pouring billions into scaling up theiur training. Microsoft and OpenAI spent 100B into their "Startgate" mega training cluster, due to start running in 2028.
๐ค So, what about these reports of scaling laws slowing down?
If they are true, they would mean a gigantic paradigm shift, as the hundreds of billions poured by AI companies into scaling could be a dead-end. โ๏ธ
But I doubt it: until the most recent publications, scaling laws showed no signs of weakness, and the researchers at the higher end of the scale-up seems to imply the scaling up continues.
Wait and see!
๐ What are scaling laws? These are empiric laws that say "Every time you increase compute spent in training 10-fold, your LLM's performance will go up by a predictable tick". Of course, they apply only if you train your model with the right methods.
The image below illustrates it: they're from a paper by Google, "Scaling Autoregressive Models for Content-Rich Text-to-Image Generation", and they show how quality and instruction following of models improve when you scale the model up (which is equivalent to scaling up the compute spent in training).
โก๏ธ These scaling laws have immense impact: they triggered the largest gold rush ever, with companies pouring billions into scaling up theiur training. Microsoft and OpenAI spent 100B into their "Startgate" mega training cluster, due to start running in 2028.
๐ค So, what about these reports of scaling laws slowing down?
If they are true, they would mean a gigantic paradigm shift, as the hundreds of billions poured by AI companies into scaling could be a dead-end. โ๏ธ
But I doubt it: until the most recent publications, scaling laws showed no signs of weakness, and the researchers at the higher end of the scale-up seems to imply the scaling up continues.
Wait and see!
- 1 reply
posted
an
update
17 days ago
Post
1618
๐๐ป๐ฑ๐ฟ๐ผ๐ถ๐ฑ๐๐ฎ๐ฏ: ๐๐ถ๐ฟ๐๐ ๐ฒ๐๐ฒ๐ฟ ๐๐๐๐๐ฒ๐บ๐ฎ๐๐ถ๐ฐ ๐ฏ๐ฒ๐ป๐ฐ๐ต๐บ๐ฎ๐ฟ๐ธ ๐ณ๐ผ๐ฟ ๐๐ป๐ฑ๐ฟ๐ผ๐ถ๐ฑ ๐บ๐ผ๐ฏ๐ถ๐น๐ฒ ๐ฎ๐ด๐ฒ๐ป๐๐ ๐๐ต๐ผ๐๐ ๐๐ต๐ฎ๐ ๐๐บ๐ฎ๐น๐น, ๐ณ๐ถ๐ป๐ฒ-๐๐๐ป๐ฒ๐ฑ ๐ผ๐ฝ๐ฒ๐ป ๐บ๐ผ๐ฑ๐ฒ๐น๐ ๐ฐ๐ฎ๐ป ๐ฝ๐ผ๐๐ฒ๐ฟ ๐ฎ ๐๐๐ฅ๐ฉ๐๐ฆ ๐๐๐๐๐ฒ๐บ ๐ผ๐ป ๐๐ผ๐๐ฟ ๐๐บ๐ฎ๐ฟ๐๐ฝ๐ต๐ผ๐ป๐ฒ ๐ฑ๐ฅ
A team from Tsinghua University just released AndroidLab, the first systematic framework to evaluate and train Android mobile agents that works with both text-only and multimodal models.
They show that fine-tuning small open-source models can significantly boost performance, matching that of much bigger closed models like GPT-4o.
The team built:
๐ย A reproducible benchmark with 138 tasks across 9 apps to evaluate mobile agents systematically
๐๐ฑย A framework supporting both text-only (via XML) and visual (via marked screenshots) interfaces
โ ย An instruction dataset of 10.5k operation traces for training mobile agents
Key insights:
- ๐ Fine-tuning improves performance BY A LOT: Open-source model Llama-3.1-8B improves from 2% to 24% success rate after training, nearly reaching GPT-4o performance although itโs much smaller
- โ๏ธ Text-only agents match multimodal ones: XML-based agents achieve similar performance to screenshot-based multimodal agents.
Read their paper here ๐ AndroidLab: Training and Systematic Benchmarking of Android Autonomous Agents (2410.24024)
A team from Tsinghua University just released AndroidLab, the first systematic framework to evaluate and train Android mobile agents that works with both text-only and multimodal models.
They show that fine-tuning small open-source models can significantly boost performance, matching that of much bigger closed models like GPT-4o.
The team built:
๐ย A reproducible benchmark with 138 tasks across 9 apps to evaluate mobile agents systematically
๐๐ฑย A framework supporting both text-only (via XML) and visual (via marked screenshots) interfaces
โ ย An instruction dataset of 10.5k operation traces for training mobile agents
Key insights:
- ๐ Fine-tuning improves performance BY A LOT: Open-source model Llama-3.1-8B improves from 2% to 24% success rate after training, nearly reaching GPT-4o performance although itโs much smaller
- โ๏ธ Text-only agents match multimodal ones: XML-based agents achieve similar performance to screenshot-based multimodal agents.
Read their paper here ๐ AndroidLab: Training and Systematic Benchmarking of Android Autonomous Agents (2410.24024)
posted
an
update
20 days ago
Post
2488
๐๐๐ป๐๐๐ฎ๐ป-๐๐ฎ๐ฟ๐ด๐ฒ ๐ท๐๐๐ ๐ฟ๐ฒ๐น๐ฒ๐ฎ๐๐ฒ๐ฑ ๐ฏ๐ ๐ง๐ฒ๐ป๐ฐ๐ฒ๐ป๐: ๐๐ฎ๐ฟ๐ด๐ฒ๐๐ ๐ฒ๐๐ฒ๐ฟ ๐ผ๐ฝ๐ฒ๐ป ๐ ๐ผ๐ ๐๐๐ , ๐ผ๐ป๐น๐ ๐ฑ๐ฎ๐ ๐ฎ๐ฐ๐๐ถ๐๐ฒ ๐ฝ๐ฎ๐ฟ๐ฎ๐บ๐ฒ๐๐ฒ๐ฟ๐ ๐ฏ๐๐ ๐ฏ๐ฒ๐ฎ๐๐ ๐๐๐ฎ๐ ๐ ๐ฏ.๐ญ-๐ฐ๐ฌ๐ฑ๐ ๐ผ๐ป ๐บ๐ผ๐๐ ๐ฎ๐ฐ๐ฎ๐ฑ๐ฒ๐บ๐ถ๐ฐ ๐ฏ๐ฒ๐ป๐ฐ๐ต๐บ๐ฎ๐ฟ๐ธ๐ ๐
โก Mixture of Experts (MoE) architecture: 389 B parameters in total, but only 52B are activated for any input
๐งช Trained on 7T tokens, including 1.5T tokens of synthetic data
๐๏ธ Architecture : Novel "recycle routing" prevents token dropping when experts are overrloaded
๐ Great benchmark results: Surpasses Llama-3-405B-Instruct in most benchmarks although it has 8x fewer active parameters
โฃ Impressive perf on MATH: 77.4
๐ย Large context length: up to 256K tokens
๐ License:
โฃ Commercial use allowed, except if your products have >100M monthly active users
โฃ No access in the EU
๐คย Model weights available on HF!
Read the full paper here ๐ย Hunyuan-Large: An Open-Source MoE Model with 52 Billion Activated Parameters by Tencent (2411.02265)
โก Mixture of Experts (MoE) architecture: 389 B parameters in total, but only 52B are activated for any input
๐งช Trained on 7T tokens, including 1.5T tokens of synthetic data
๐๏ธ Architecture : Novel "recycle routing" prevents token dropping when experts are overrloaded
๐ Great benchmark results: Surpasses Llama-3-405B-Instruct in most benchmarks although it has 8x fewer active parameters
โฃ Impressive perf on MATH: 77.4
๐ย Large context length: up to 256K tokens
๐ License:
โฃ Commercial use allowed, except if your products have >100M monthly active users
โฃ No access in the EU
๐คย Model weights available on HF!
Read the full paper here ๐ย Hunyuan-Large: An Open-Source MoE Model with 52 Billion Activated Parameters by Tencent (2411.02265)
posted
an
update
23 days ago
Post
1541
๐ง ย ๐๐๐๐๐ฅ: ๐ณ๐ถ๐ฟ๐๐ ๐บ๐๐น๐๐ถ๐บ๐ผ๐ฑ๐ฎ๐น ๐ฏ๐ฒ๐ป๐ฐ๐ต๐บ๐ฎ๐ฟ๐ธ ๐๐ผ ๐บ๐ฎ๐ธ๐ฒ ๐บ๐ผ๐ฑ๐ฒ๐น๐ ๐ณ๐ผ๐ฟ๐ด๐ฒ๐ ๐๐ต๐ฎ๐ ๐๐ฒ ๐๐ฎ๐ป๐ ๐๐ต๐ฒ๐บ ๐๐ผ ๐ณ๐ผ๐ฟ๐ด๐ฒ๐
With privacy concerns rising, we sometimes need our models to "forget" specific information - like a person's data - while keeping everything else intact. Researchers just released CLEAR, the first benchmark to test how well this works with both text and images.
โย Bad news: Current methods either fail to truly forget or end up forgetting way too much. It's like trying to remove a single ingredient from a baked cake!
โจย But there's hope: Adding simple mathematical constraints (L1 regularization) during the forgetting process significantly improves results.
๐ฏย Key insights:
โ ย The benchmark tests forgetting on 200 fictional personas
โฃ 3,770 visual Q&A pairs
โฃ 4,000 textual Q&A pairs
โฃ Additional real-world tests
๐ย Most current forgetting methods don't work well with both text and images
โฃ They either remember what they should forget
โฃ Or they forget too much unrelated information
โจย Simple mathematical constraints work surprisingly well
โฃ L1 regularization prevents excessive forgetting
โฃ Works especially well with the LLMU method
๐ย Read the full paper here: CLEAR: Character Unlearning in Textual and Visual Modalities (2410.18057)
With privacy concerns rising, we sometimes need our models to "forget" specific information - like a person's data - while keeping everything else intact. Researchers just released CLEAR, the first benchmark to test how well this works with both text and images.
โย Bad news: Current methods either fail to truly forget or end up forgetting way too much. It's like trying to remove a single ingredient from a baked cake!
โจย But there's hope: Adding simple mathematical constraints (L1 regularization) during the forgetting process significantly improves results.
๐ฏย Key insights:
โ ย The benchmark tests forgetting on 200 fictional personas
โฃ 3,770 visual Q&A pairs
โฃ 4,000 textual Q&A pairs
โฃ Additional real-world tests
๐ย Most current forgetting methods don't work well with both text and images
โฃ They either remember what they should forget
โฃ Or they forget too much unrelated information
โจย Simple mathematical constraints work surprisingly well
โฃ L1 regularization prevents excessive forgetting
โฃ Works especially well with the LLMU method
๐ย Read the full paper here: CLEAR: Character Unlearning in Textual and Visual Modalities (2410.18057)
posted
an
update
24 days ago
Post
2348
> Oasis: First Real-Time Video Game Without a Game Engine! ๐ฎ
DecartAI & Etched just released Oasis - a fully AI-generated video game running at 20 FPS (frames per second). The model takes keyboard inputs and generates everything - physics, rules, graphics - on the fly, without any game engine.
โก๏ธ What makes this special? Current text-to-video models (Mochi-1, Sora, Kling) generate about 1 frame every 10-20 seconds (that's the kind of device I had to play LoL back in the day, thus my low rankings). Oasis is 200 times faster, making it the first playable AI-generated game.
โ๏ธ Under the hood, it uses a vision transformer to encode space and a diffusion model to generate frames. The secret sauce is "dynamic noising" - a technique that keeps the video stable between frames.
Key insights:
โก๏ธ Generates 20 FPS, vs 0.2 FPS for other DIT-based video models
โฃ The specialized hardware Sohu developed by Etched allows to handle 10x more player than H100
๐ฎ Features real game mechanics
โฃ Movement, jumping, item management
โฃ Physics and lighting
โฃ Procedurally generated worlds
โ ๏ธ Current limitations
โฃ Blurry graphics at a distance
โฃ Objects sometimes change appearance
โฃ Memory issues in long sessions
Try it yourself, the playable demo is impressive! ๐ https://oasis.decart.ai/welcome
Code ๐ https://github.com/etched-ai/open-oasis
Read it in full ๐ https://oasis-model.github.io/
DecartAI & Etched just released Oasis - a fully AI-generated video game running at 20 FPS (frames per second). The model takes keyboard inputs and generates everything - physics, rules, graphics - on the fly, without any game engine.
โก๏ธ What makes this special? Current text-to-video models (Mochi-1, Sora, Kling) generate about 1 frame every 10-20 seconds (that's the kind of device I had to play LoL back in the day, thus my low rankings). Oasis is 200 times faster, making it the first playable AI-generated game.
โ๏ธ Under the hood, it uses a vision transformer to encode space and a diffusion model to generate frames. The secret sauce is "dynamic noising" - a technique that keeps the video stable between frames.
Key insights:
โก๏ธ Generates 20 FPS, vs 0.2 FPS for other DIT-based video models
โฃ The specialized hardware Sohu developed by Etched allows to handle 10x more player than H100
๐ฎ Features real game mechanics
โฃ Movement, jumping, item management
โฃ Physics and lighting
โฃ Procedurally generated worlds
โ ๏ธ Current limitations
โฃ Blurry graphics at a distance
โฃ Objects sometimes change appearance
โฃ Memory issues in long sessions
Try it yourself, the playable demo is impressive! ๐ https://oasis.decart.ai/welcome
Code ๐ https://github.com/etched-ai/open-oasis
Read it in full ๐ https://oasis-model.github.io/
posted
an
update
28 days ago
Post
1824
I'm very proud to have supported
@CGIAR
and
@Digigreen
in making http://Farmer.chat, an app that supports 20k smallholder farmers on a daily basis ๐พ
There are ~500 million smallholder farmers globally, playing a critical role in global food security. Having access to accurate information is essential for them.
๐ฌ An โagricultural extension serviceโ offers technical advice on agriculture, and also supplies farmers with the necessary inputs and services to support their agricultural production.
But agriculture extension agents are not in large enough numbers to cope with all the requests, especially in countries like Kenya, India, Ethiopia, and Nigeria.
๐ So the team set out to build an app called http://Farmer.Chat, to provide an agricultural extension service, by building on the immense knowledge accumulated by CGIAR.
โจ The app is technically impressive: behind the Whatsapp-type UX, an agent interprets the user's intent, and identifies which tool to call to best answer their request: weather API, RAG on a CGIAR-provided knowledge base, market data, etc. The RAG on the knowledge base is in itself a work of art.
๐ฏ A key part of building such a complex system is to be able to evaluate it properly. During our bi-weekly sessions with the team, I could support them in implementing the method called "LLM-as-a-judge" to tackle this problem.
It worked really well : thanks to the amazing work of the team, the app now successfully answered over 300 thousand requests, in 6 different languages, and it keeps growing!
โก๏ธ @Vinsingh , @rajgreen and I just wrote a blog post to describe how the app works, especially the LLM-as-a-judge system!
Read it here ๐ https://huggingface.co/blog/digital-green-llm-judge
There are ~500 million smallholder farmers globally, playing a critical role in global food security. Having access to accurate information is essential for them.
๐ฌ An โagricultural extension serviceโ offers technical advice on agriculture, and also supplies farmers with the necessary inputs and services to support their agricultural production.
But agriculture extension agents are not in large enough numbers to cope with all the requests, especially in countries like Kenya, India, Ethiopia, and Nigeria.
๐ So the team set out to build an app called http://Farmer.Chat, to provide an agricultural extension service, by building on the immense knowledge accumulated by CGIAR.
โจ The app is technically impressive: behind the Whatsapp-type UX, an agent interprets the user's intent, and identifies which tool to call to best answer their request: weather API, RAG on a CGIAR-provided knowledge base, market data, etc. The RAG on the knowledge base is in itself a work of art.
๐ฏ A key part of building such a complex system is to be able to evaluate it properly. During our bi-weekly sessions with the team, I could support them in implementing the method called "LLM-as-a-judge" to tackle this problem.
It worked really well : thanks to the amazing work of the team, the app now successfully answered over 300 thousand requests, in 6 different languages, and it keeps growing!
โก๏ธ @Vinsingh , @rajgreen and I just wrote a blog post to describe how the app works, especially the LLM-as-a-judge system!
Read it here ๐ https://huggingface.co/blog/digital-green-llm-judge