HuggingFaceFW/fineweb-2
Viewer
ā¢
Updated
ā¢
13.8B
ā¢
83.9k
ā¢
354
Eugene Siow
eugenesiow
AI & ML interests
None yet
Recent Activity
liked
a dataset
6 days ago
HuggingFaceFW/fineweb-2
liked
a Space
6 days ago
FunAudioLLM/CosyVoice2-0.5B
liked
a model
12 days ago
Datou1111/shou_xin
Organizations
eugenesiow's activity
liked
a
dataset
6 days ago
liked
a
Space
6 days ago
liked
a
model
12 days ago
upvoted
an
article
18 days ago
Article
šŗš¦āā¬ LLM Comparison/Test: 25 SOTA LLMs (including QwQ) through 59 MMLU-Pro CS benchmark runs
By
ā¢
ā¢
70
reacted to
m-ric's
post with š„
18 days ago
Post
1467
š¦šµš¼ššØš: š® ššŗš®š¹š¹ š²š»š±-šš¼-š²š»š± š®š“š²š»š ššµš®š š°š®š» š»š®šš¶š“š®šš² š®š»š šØš š®š»š± š¼ššš½š²šæš³š¼šæšŗš šŗšš°šµ šÆš¶š“š“š²šæ ššššš²šŗš! š²
A team from NUS and Microsoft just released an agent that can act on any UI (Desktop, Android, Web) without needing additional text information. It works extremely well : they applied their method on a tiny Qwen2-VL-2B, and they managed to beat methods that use either much more powerful vision models (like GPT-4V) without using any additional info (e.g. leveraging the DOM of a webpage) like previous methods did ! šš
They started from the idea that most existing methods rely heavily on text, which makes them less generalizable, while letting aside rich UI structure that user actually rely on when navigating this interfaces.
āļø They put several good ideas to work:
š” Simplify screenshots to the max:
They prune a lot the heavy visual content of UI screenshots, by removing cloned image patches (like any vast patch of the same color will be reduced to a small patch, while maintaining positional embeddings), then group patches from the same GUI elements together to simplify even further
š” Build a truly generalist dataset:
To train a general UI agent, you need trajectories from each possible UI, and express them in a common language. Authors merge datasets like OmniAct for Desktop, Mind2Web for websites, AMEX for Android trajectories to create a high-quality and diverse dataset.
ā”ļø Nice results ensued:
They fine-tune a tiny Qwen-2-VL-2B on their method, and it reaches SOTA on several task (element identification, web navigation), even beating methods that either use additional info from the DOM or use much bigger VLMS like GPT-4v! š
And performance could certainly jump with a slightly bigger vision model. Let's hope the community builds this soon! š
Paper added to my "Agents" collection š m-ric/agents-65ba776fbd9e29f771c07d4e
A team from NUS and Microsoft just released an agent that can act on any UI (Desktop, Android, Web) without needing additional text information. It works extremely well : they applied their method on a tiny Qwen2-VL-2B, and they managed to beat methods that use either much more powerful vision models (like GPT-4V) without using any additional info (e.g. leveraging the DOM of a webpage) like previous methods did ! šš
They started from the idea that most existing methods rely heavily on text, which makes them less generalizable, while letting aside rich UI structure that user actually rely on when navigating this interfaces.
āļø They put several good ideas to work:
š” Simplify screenshots to the max:
They prune a lot the heavy visual content of UI screenshots, by removing cloned image patches (like any vast patch of the same color will be reduced to a small patch, while maintaining positional embeddings), then group patches from the same GUI elements together to simplify even further
š” Build a truly generalist dataset:
To train a general UI agent, you need trajectories from each possible UI, and express them in a common language. Authors merge datasets like OmniAct for Desktop, Mind2Web for websites, AMEX for Android trajectories to create a high-quality and diverse dataset.
ā”ļø Nice results ensued:
They fine-tune a tiny Qwen-2-VL-2B on their method, and it reaches SOTA on several task (element identification, web navigation), even beating methods that either use additional info from the DOM or use much bigger VLMS like GPT-4v! š
And performance could certainly jump with a slightly bigger vision model. Let's hope the community builds this soon! š
Paper added to my "Agents" collection š m-ric/agents-65ba776fbd9e29f771c07d4e
upvoted
a
collection
18 days ago
liked
a
model
21 days ago
liked
a
model
29 days ago
liked
2
models
30 days ago
liked
2
Spaces
about 1 month ago
liked
a
dataset
about 1 month ago
upvoted
a
collection
about 1 month ago
liked
3
models
about 1 month ago
liked
3
datasets
about 1 month ago