Reinforcement Learning on Web Interfaces Using Workflow-Guided Exploration
Paper
•
1802.08802
•
Published
UI Agent
A collection of agents for user interfaces/interactions and UI program synthesis
Note Introducing the Workflow-Guided Exploration (WGE) framework, and a new synthetic dataset called MiniWoB++ (an extension of MiniWoB, a dataset proposed by "World of Bits: An Open-Domain Platform for Web-Based Agents"). Published at ICLR 2018
-
Mapping Natural Language Commands to Web Elements
Paper • 1808.09132 • Published -
Learning to Navigate the Web
Paper • 1812.09195 • Published -
Interactive Task and Concept Learning from Natural Language Instructions and GUI Demonstrations
Paper • 1909.00031 • Published -
Mapping Natural Language Instructions to Mobile UI Action Sequences
Paper • 2005.03776 • Published -
FLIN: A Flexible Natural Language Interface for Web Navigation
Paper • 2010.12844 • Published -
ActionBert: Leveraging User Actions for Semantic Understanding of User Interfaces
Paper • 2012.12350 • Published -
Grounding Open-Domain Instructions to Automate Web Support Tasks
Paper • 2103.16057 • Published -
Mobile App Tasks with Iterative Feedback (MoTIF): Addressing Task Feasibility in Interactive Visual Environments
Paper • 2104.08560 • Published -
AndroidEnv: A Reinforcement Learning Platform for Android
Paper • 2105.13231 • Published -
UIBert: Learning Generic Multimodal Representations for UI Understanding
Paper • 2107.13731 • Published -
Compositional Generalization for Natural Language Interfaces to Web APIs
Paper • 2112.05209 • Published -
VUT: Versatile UI Transformer for Multi-Modal Multi-Task User Interface Modeling
Paper • 2112.05692 • Published -
A Dataset for Interactive Vision-Language Navigation with Unknown Command Feasibility
Paper • 2202.02312 • Published -
META-GUI: Towards Multi-modal Conversational Agents on Mobile GUI
Paper • 2205.11029 • Published -
MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledge
Paper • 2206.08853 • Published • 1 -
WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents
Paper • 2207.01206 • Published • 1 -
Enabling Conversational Interaction with Mobile UI using Large Language Models
Paper • 2209.08655 • Published -
Spotlight: Mobile UI Understanding using Vision-Language Models with a Focus
Paper • 2209.14927 • Published -
ReAct: Synergizing Reasoning and Acting in Language Models
Paper • 2210.03629 • Published • 13 -
Understanding HTML with Large Language Models
Paper • 2210.03945 • Published • 1 -
Lexi: Self-Supervised Learning of the UI Language
Paper • 2301.10165 • Published -
Language Models can Solve Computer Tasks
Paper • 2303.17491 • Published -
Multimodal Web Navigation with Instruction-Finetuned Foundation Models
Paper • 2305.11854 • Published • 2 -
Voyager: An Open-Ended Embodied Agent with Large Language Models
Paper • 2305.16291 • Published • 8 -
From Pixels to UI Actions: Learning to Follow Instructions via Graphical User Interfaces
Paper • 2306.00245 • Published -
Mind2Web: Towards a Generalist Agent for the Web
Paper • 2306.06070 • Published • 18 -
Synapse: Trajectory-as-Exemplar Prompting with Memory for Computer Control
Paper • 2306.07863 • Published -
Android in the Wild: A Large-Scale Dataset for Android Device Control
Paper • 2307.10088 • Published • 9 -
A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesis
Paper • 2307.12856 • Published • 34 -
WebArena: A Realistic Web Environment for Building Autonomous Agents
Paper • 2307.13854 • Published • 20 -
AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation Framework
Paper • 2308.08155 • Published • 2 -
Never-ending Learning of User Interfaces
Paper • 2308.08726 • Published • 1 -
Empowering LLM to use Smartphone for Intelligent Task Automation
Paper • 2308.15272 • Published • 1 -
LASER: LLM Agent with State-Space Exploration for Web Navigation
Paper • 2309.08172 • Published • 10 -
You Only Look at Screens: Multimodal Chain-of-Action Agents
Paper • 2309.11436 • Published • 1 -
AXNav: Replaying Accessibility Tests from Natural Language
Paper • 2310.02424 • Published -
HeaP: Hierarchical Policies for Web Actions using LLMs
Paper • 2310.03720 • Published • 5 -
Automatic Macro Mining from Interaction Traces at Scale
Paper • 2310.07023 • Published -
A Zero-Shot Language Agent for Computer Control with Structured Reflection
Paper • 2310.08740 • Published • 14 -
GPT-4V in Wonderland: Large Multimodal Models for Zero-Shot Smartphone GUI Navigation
Paper • 2311.07562 • Published • 11 -
GAIA: a benchmark for General AI Assistants
Paper • 2311.12983 • Published • 175 -
Language Model Agents Suffer from Compositional Generalization in Web Automation
Paper • 2311.18751 • Published -
CogAgent: A Visual Language Model for GUI Agents
Paper • 2312.08914 • Published • 28 -
UINav: A Practical Approach to Train On-Device Automation Agents
Paper • 2312.10170 • Published -
ASSISTGUI: Task-Oriented Desktop Graphical User Interface Automation
Paper • 2312.13108 • Published • 3 -
AppAgent: Multimodal Agents as Smartphone Users
Paper • 2312.13771 • Published • 49 -
GPT-4V(ision) is a Generalist Web Agent, if Grounded
Paper • 2401.01614 • Published • 19 -
SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents
Paper • 2401.10935 • Published • 4 -
WebVoyager: Building an End-to-End Web Agent with Large Multimodal Models
Paper • 2401.13919 • Published • 22 -
VisualWebArena: Evaluating Multimodal Agents on Realistic Visual Web Tasks
Paper • 2401.13649 • Published • 1 -
Mobile-Agent: Autonomous Multi-Modal Mobile Device Agent with Visual Perception
Paper • 2401.16158 • Published • 16 -
Dual-View Visual Contextualization for Web Navigation
Paper • 2402.04476 • Published -
WebLINX: Real-World Website Navigation with Multi-Turn Dialogue
Paper • 2402.05930 • Published • 35 -
OS-Copilot: Towards Generalist Computer Agents with Self-Improvement
Paper • 2402.07456 • Published • 39 -
OmniACT: A Dataset and Benchmark for Enabling Multimodal Generalist Autonomous Agents for Desktop and Web
Paper • 2402.17553 • Published • 21 -
WorkArena: How Capable Are Web Agents at Solving Common Knowledge Work Tasks?
Paper • 2403.07718 • Published -
BAGEL: Bootstrapping Agents by Guiding Exploration with Language
Paper • 2403.08140 • Published -
AgentStudio: A Toolkit for Building General Virtual Agents
Paper • 2403.17918 • Published -
Octopus v2: On-device language model for super agent
Paper • 2404.01744 • Published • 55 -
AutoWebGLM: Bootstrap And Reinforce A Large Language Model-based Web Navigating Agent
Paper • 2404.03648 • Published • 22 -
Ferret-UI: Grounded Mobile UI Understanding with Multimodal LLMs
Paper • 2404.05719 • Published • 57 -
Autonomous Evaluation and Refinement of Digital Agents
Paper • 2404.06474 • Published • 1 -
OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments
Paper • 2404.07972 • Published • 41 -
MMInA: Benchmarking Multihop Multimodal Internet Agents
Paper • 2404.09992 • Published -
AgentKit: Flow Engineering with Graphs, not Coding
Paper • 2404.11483 • Published • 1 -
UIClip: A Data-driven Model for Assessing User Interface Design
Paper • 2404.12500 • Published -
Benchmarking Mobile Device Control Agents across Diverse Configurations
Paper • 2404.16660 • Published -
Latent State Estimation Helps UI Agents to Reason
Paper • 2405.11120 • Published -
AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agents
Paper • 2405.14573 • Published -
Devil's Advocate: Anticipatory Reflection for LLM Agents
Paper • 2405.16334 • Published -
Large Language Models Can Self-Improve At Web Agent Tasks
Paper • 2405.20309 • Published • 1 -
Mobile-Agent-v2: Mobile Device Operation Assistant with Effective Navigation via Multi-Agent Collaboration
Paper • 2406.01014 • Published • 29 -
On the Effects of Data Scale on Computer Control Agents
Paper • 2406.03679 • Published -
NATURAL PLAN: Benchmarking LLMs on Natural Language Planning
Paper • 2406.04520 • Published • 9 -
AgentGym: Evolving Large Language Model-based Agents across Diverse Environments
Paper • 2406.04151 • Published • 14 -
Synthetic Query Generation using Large Language Models for Virtual Assistants
Paper • 2406.06729 • Published -
UICoder: Finetuning Large Language Models to Generate User Interface Code through Automated Feedback
Paper • 2406.07739 • Published • 2 -
DigiRL: Training In-The-Wild Device-Control Agents with Autonomous Reinforcement Learning
Paper • 2406.11896 • Published • 17 -
VideoGUI: A Benchmark for GUI Automation from Instructional Videos
Paper • 2406.10227 • Published • 8 -
GUI-WORLD: A Dataset for GUI-oriented Multimodal LLM-based Agents
Paper • 2406.10819 • Published -
GUICourse: From General Vision Language Models to Versatile GUI Agents
Paper • 2406.11317 • Published -
Adversarial Attacks on Multimodal Agents
Paper • 2406.12814 • Published • 4 -
ICAL: Continual Learning of Multimodal Agents by Transforming Trajectories into Actionable Insights
Paper • 2406.14596 • Published • 4