Computer-use AI agents
Also known as Computer-use agents, Computer use, GUI agents, Browser agents
Computer-use AI agents are models that look at screenshots and control a mouse, keyboard or browser to carry out tasks in software built for people. In the 2024 OSWorld benchmark the best model tested completed 12.24% of tasks, against a human baseline of 72.36%, and prompt injection from web content remains far from a solved problem.
In plain language
A computer-use agent is an AI model that works with a computer the way a person does. It is given a goal, looks at an image of the screen, and then returns an action such as moving the cursor, clicking a button, typing text or scrolling. The software carries out that action, takes a new screenshot, and the loop continues until the task is finished or the agent stops.
Benchmark results show how far these systems are from reliable use. In the OSWorld benchmark, published as a preprint in 2024, the best model tested completed 12.24% of 369 tasks in real web and desktop applications, while the reported human baseline was 72.36%. In the WebArena benchmark, a 2023 preprint about websites, the best GPT-4-based agent completed 14.41% of tasks, against a human rate of 78.24%. Newer results are mostly reported by the developers themselves, and the sources reviewed here do not allow them to be checked independently. The security problem is also serious: a web page can contain hidden instructions that an agent may mistake for the user's own request.
Technical explanation
Many computer-use agents are built around a vision-language model that receives a screenshot and the task description, and outputs a structured action, such as a click at given coordinates or a keystroke. A separate control layer then carries out the action through operating-system input or browser automation. Some systems also read the page's underlying structure, such as an accessibility tree, alongside or instead of the image. A central difficulty is GUI grounding, which is the step of matching a described element, such as 'the Save button', to the correct position on the screen. The OSWorld authors identify GUI grounding and operational knowledge as the main weaknesses of the best model they tested.
Two benchmarks are most often cited. OSWorld (Xie and colleagues, 2024) provides 369 tasks in real web and desktop applications. WebArena (Zhou and colleagues, 2023) provides functional websites from four common domains. Scores are not directly comparable across papers and vendor reports, because the test setups differ. For example, Anthropic reported 14.9% on the screenshot-only OSWorld category for Claude 3.5 Sonnet in October 2024, and 22.0% when more steps were allowed. That vendor figure was reported by its developer and is not an independent measurement.
Security research highlights a specific risk: content the agent reads may contain instructions. Greshake and colleagues (2023) showed that attackers can plant instructions in data an LLM-integrated application retrieves, because such applications do not reliably separate data from instructions. Anthropic reported that, in red-teaming of browser use in August 2025, the attack success rate was 23.6% without mitigations and 11.2% in autonomous mode with mitigations, across 123 test cases in 29 scenarios. A November 2025 post reported 1.4% attack success for Claude Opus 4.5 with improved safeguards, against an internal adaptive Best-of-N attacker given 100 attempts per environment. Anthropic describes residual risk of this kind as meaningful. These vendor figures cover browser use rather than full computer use, and they use different test setups.
What the evidence says
Each statement is typed by the strength of its evidence and linked to the source that supports it.
Demonstrated Shown in a published experiment, test or shipping product.
The OSWorld benchmark contains 369 tasks in real web and desktop computer environments, and the best model tested in the 2024 preprint completed 12.24% of them, against a reported human baseline of 72.36%.
Sources: [1] · AbstractThe OSWorld authors state that the best model they tested struggles primarily with GUI grounding and operational knowledge.
Sources: [1] · Abstract · medium confidenceIn WebArena, the best GPT-4-based agent reached an end-to-end task success rate of 14.41%, against a human performance rate of 78.24%, on functional websites from four common domains.
Sources: [2] · AbstractGreshake and colleagues showed that attackers can remotely compromise LLM-integrated applications by planting malicious instructions in data the model is likely to retrieve, because these applications do not clearly separate data from instructions.
Sources: [4] · AbstractReported, unverified Stated by a company or team but not independently confirmed.
Anthropic reported that Claude 3.5 Sonnet scored 14.9% on the screenshot-only OSWorld category when it launched computer use in public beta in October 2024, and 22.0% with more steps allowed. The developer described this as notably better than the next-best system's 7.8%.
Sources: [3] · Benchmark results section · medium confidenceAnthropic described its computer use public beta as experimental and at times cumbersome and error-prone, and said that actions such as scrolling, dragging and zooming currently present challenges for Claude.
Sources: [3] · medium confidenceIn Anthropic's red-teaming of browser use for Claude for Chrome, the attack success rate was 23.6% without safety mitigations and 11.2% in autonomous mode with mitigations, across 123 test cases in 29 attack scenarios.
Sources: [5] · medium confidenceAnthropic reports 1.4% attack success for Claude Opus 4.5 with improved safeguards, against an internal adaptive Best-of-N attacker given 100 attempts per environment. It states that a 1% attack success rate still represents meaningful risk, and that prompt injection is far from a solved problem.
Sources: [6] · medium confidenceLimitations and open problems
- Benchmark success rates are far below human baselines: 12.24% against 72.36% in OSWorld (2024), and 14.41% against 78.24% in WebArena (2023), using the best systems tested in those papers.
- The OSWorld authors identify GUI grounding and operational knowledge as the main weaknesses of the best model they tested.
- Indirect prompt injection is a persistent threat, because the agent reads untrusted web content. Anthropic states that no browser agent is immune to prompt injection, and that a 1% attack success rate still represents meaningful risk.
- Vendor security figures use different test setups and mostly cover browser use, so they cannot be compared directly with each other or with independent results.
- Anthropic described its 2024 public beta of computer use as experimental and at times cumbersome and error-prone, and said that actions such as scrolling, dragging and zooming currently present challenges for the model.
- Speed, running cost and the effect on human work are not covered by the sources reviewed here, so they are not assessed in this profile.
Possible future uses
Ideas, not predictions. None of these has been demonstrated.
- Automating routine tasks in older business software that has no programming interface, using the same screens a staff member would use.
- Acting as a testing tool that uses a software interface the way an ordinary user would, to find faults before release.
- Helping people who find a mouse or keyboard hard to use by carrying out multi-step tasks from a spoken or written goal.
Timeline
- Feb 2023Greshake and colleagues publish their study of indirect prompt injection against LLM-integrated applications. Sources: [4]
- Jul 2023WebArena preprint reports that the best GPT-4-based agent completes 14.41% of website tasks, against a human rate of 78.24%. Sources: [2]
- 11 Apr 2024OSWorld preprint is submitted, reporting 369 tasks in real computer environments, a best-model success rate of 12.24% and a human baseline of 72.36%. Sources: [1]
- 22 Oct 2024Anthropic introduces computer use in public beta and reports 14.9% on the screenshot-only OSWorld category for Claude 3.5 Sonnet. Sources: [3]
- 25 Aug 2025Anthropic publishes its Claude for Chrome red-team figures, reporting attack success falling from 23.6% to 11.2% with mitigations in autonomous mode. Sources: [5]
- 24 Nov 2025Anthropic reports 1.4% attack success for Claude Opus 4.5 with improved safeguards, against an internal adaptive Best-of-N attacker with 100 attempts per environment. Sources: [6]
Futures this could help bring about
Glossary
- GUI
- Graphical user interface: the buttons, menus, windows and fields on a screen that a person clicks and types into.
- GUI grounding
- The step of working out which exact place on the screen matches a described element, such as a button or text field.
- Indirect prompt injection
- An attack in which instructions hidden in content the AI reads, such as a web page, try to override what the user asked for.
- Attack success rate
- The share of attack attempts that cause the AI system to do what the attacker wanted. Lower is better, and results depend on how the attacks were designed.
Sources
- preprint
OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments
Tianbao Xie, Danyang Zhang, Jixuan Chen et al.. arXiv, 11 Apr 2024. Accessed 21 Oct 2026. - preprint
WebArena: A Realistic Web Environment for Building Autonomous Agents
Shuyan Zhou, Frank F. Xu, Hao Zhu et al.. arXiv, Jul 2023. Accessed 21 Oct 2026. - company announcement
Introducing computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku
Anthropic, 22 Oct 2024. Accessed 21 Oct 2026. - preprint
Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection
Kai Greshake, Sahar Abdelnabi, Shailesh Mishra et al.. arXiv, Feb 2023. Accessed 21 Oct 2026. - company announcement
Claude for Chrome
Anthropic, 25 Aug 2025. Accessed 21 Oct 2026. - company announcement
Mitigating the risk of prompt injections in browser use
Anthropic, 24 Nov 2025. Accessed 21 Oct 2026.
Revision history
3 changes since this profile was created.