Epoch AI - OSWorld's 70% Success Rate Reflects AI Scripting and Automation Triumph, Not Generalized GUI Mastery

What does OSWorld tell us about AI’s ability to use computers?

We review OSWorld, a prominent computer use benchmark. Tasks are relatively simple, many don’t require GUIs, and success often hinges on interpreting ambiguous instructions. The benchmark is also not stable over time.

Epoch AIFlorian Brand

Gauging the True Capability of the "Digital Colleague": Insights and Limitations of the OSWorld Benchmark

In the current wave of Artificial Intelligence (AI) sweeping the globe, the industry harbors significant expectations for AI systems to evolve into genuine "Digital Colleagues." The OSWorld benchmark, a specialized tool designed to measure AI's ability to use computers, recently released data that seems to provide an exhilarating answer: the most advanced models have boosted their task success rate from an initial low of less than 10% to nearly 70%.

However, as emphasized by the analysis from Epoch AI, interpreting this seemingly impressive 70% score demands a high degree of criticality and prudence. This is because the progress represented by this figure may reflect AI's advancement in automation and scripting rather than a breakthrough in the generalizability of simulating human Graphical User Interface (GUI) interaction.

The Appearance of Progress: OSWorld's Testing Environment and Core Findings

The OSWorld benchmark comprises 361 computer usage tasks (excluding 8 samples involving Google Drive), aiming to evaluate AI's performance when executing simple, real-world computer operations.

Key Features of the Test Setup:

• Restricted Environment: Models operate within an internet-connected Ubuntu Virtual Machine (VM), typically with specific open-source applications like LibreOffice, Chrome, or GIMP pre-opened.
• Interaction Mechanism: AI interacts with the machine by writing Python code and using the pyautogui package to control the mouse and keyboard.
• Evaluation Criteria: Assessment is based solely on whether the final state of the machine achieves the specified goal; the intermediate steps and methods used by the AI are not evaluated.

The image below illustrates an example of the initial state of an OSWorld task, such as having a presentation file open in LibreOffice Impress.

OSWorld Task Initial State Example: LibreOffice Impress file is open

Source: Epoch AI

Quantitatively, the jump from a score of less than 10% to nearly 70% undeniably demonstrates significant progress in AI's ability to handle simple, Linux-based tasks. This suggests that AI can now perform routine tasks within a controlled open-source ecosystem, such as adding page numbers to a document or exporting a CSV file from a spreadsheet. Yet, when we delve into the complexity of these tasks, a crucial point emerges: most tasks are relatively straightforward. According to the analysis, 88% of the tasks require fewer than 20 atomic operations (clicks, text inputs, etc.) to complete. For instance, the median-length task (6 steps) is merely editing an image by filling the background layer with green in GIMP.

The Efficiency Trap: Scripting as a Proxy for GUI Interaction

To answer whether AI has truly mastered GUI interaction, we must focus on the core reason why the OSWorld results are difficult to interpret: a large portion of tasks allows AI to bypass the expected graphical interface operations.

The improvement in benchmark success rate likely reflects AI's enhanced capability in two areas, rather than solely pure GUI operation:

1. Terminal Efficiency: Approximately 15% of the tasks can be completed entirely through the command line (Terminal), such as force-quitting a frozen program. For professional users in a Linux environment, command-line operations are often far more efficient than GUI clicking.
2. Python Scripting as Substitution: More critically, an additional 30% of tasks can significantly reduce GUI usage through the Terminal and Python scripts. This means that in approximately 45% of the total tasks, the model can substitute error-prone mouse clicks and keyboard inputs with code execution.

For example, when handling spreadsheet tasks, a proficient AI agent won't click cell by cell like a human user in LibreOffice Calc. Instead, it will proactively install and invoke professional Python packages (such as openpyxl or pandas) to modify the data file directly.

This "code-over-click" strategy results in models with code execution tools scoring significantly higher on the official leaderboard than agents relying solely on the GUI. For instance, Claude Sonnet 4 was observed to install openpyxl and pandas to handle document extraction tasks, thereby circumventing cumbersome GUI operations.

Claude Sonnet 4 installs openpyxl and pandas via the terminal to handle document tasks, bypassing the GUI

Source: Epoch AI

Consequently, the 70% success rate is more a victory for AI's task automation and system programming capabilities than a comprehensive breakthrough in the simulation of human visual and interactive logic.

Inherent Limitations of the Benchmark: Ambiguity and Instability

Beyond the substitutive nature of the operations, OSWorld itself, as an evaluation tool, has multiple inherent limitations that challenge the generalizability of its results. This is where we must apply the spirit of EEAT (Expertise, Empirical, Authority, Trust) when assessing AI capabilities.

1. Deviation from Real-World Environments

OSWorld's environmental settings diverge significantly from mainstream commercial environments:

• OS and Application Choice: The benchmark utilizes the Ubuntu Linux system and open-source/freeware like LibreOffice. Considering the widespread use of Windows and Microsoft Office in corporate settings, OSWorld tasks do not directly represent the most common computer usage scenarios.
• Task Complexity: While the tasks are realistic, they are generally simple. Only 12% of tasks require more than 20 steps, and complex, multi-step processes involved in actual human work (e.g., one complex task involving 104 steps) represent only a tiny fraction.

2. Instruction Ambiguity and Model Interpretation

The OSWorld score measures not only AI's computer operation skills but also its ability to interpret and guess the intent of ambiguous instructions.

• Instruction Vagueness: Many task instructions exhibit moderate ambiguity, requiring the model to make "reasonable" guesses in the absence of clear information. This ability to navigate ambiguity is independent of computer usage skills. Even if a model possesses all necessary operational skills, it may fail due to a reasonable but incorrect guess about the instruction's intent.
• Ambiguity Cases: For instance, in a presentation editing task, the instruction requires "aligning" a text box. This can be ambiguous: does it mean aligning the text within the box, or moving the text box itself? The ambiguity is further compounded in complex slides involving grouped objects, where the reference to the "first text box" can be confusing.

Selected text box in the slide, where the instruction "align" may lead to multiple ambiguities

Source: Epoch AI

Selected grouped objects in the slide, further complicating the reference to "text box"

Source: Epoch AI

The human baseline score is approximately 72%, a fact that suggests instruction ambiguity affects nearly 28% of the tasks, underscoring the difficulty of "intent understanding" in real-world applications.

3. Benchmark Instability and Error Rate

The benchmark also faces inherent quality challenges that cannot be ignored:

• Challenges of Continuous Updates: Since a major update in July 2025, an additional 10% of task instructions have been modified. While continuous improvement is commendable, for a non-real-time benchmark, this practice is highly atypical as it reduces the significance of cross-time comparisons and may create the illusion of enhanced model capability.
• Task Error Rate: About 10% of the tasks contain severe errors rendering them invalid. These errors include "Gold Standard Errors" (where the correct answer in the evaluation function is wrong), "Overly Strict Evaluations" (requiring a date but checking if the time is set to 12PM), or "Vague References" (e.g., a mention of the "IMDB Top 30 list" with no clear reference point).

Assessment from an Investment Perspective

Returning to the initial question: Is OSWorld's 70% score AI's genuine mastery of the GUI, or a highly efficient script in disguise?

Our judgment is that this is currently more a triumph of efficient scripting and automation capabilities than a comprehensive mastery of complex, generalized GUI interaction.

• The Upside: AI's automation capabilities within the Linux ecosystem have reached a practical threshold. For tasks that can be clearly defined and scripted (e.g., data processing, file operations), AI's value as a "Digital Colleague" is rapidly materializing.
• The Limitations: OSWorld's success is primarily built on the model's ability to bypass the challenges of complex GUI visual perception and click precision through terminal commands and Python scripts. AI's generalization ability remains unproven when facing mainstream Windows/Microsoft Office environments and complex multi-step tasks that rely heavily on visual understanding, contextual reasoning, and interpreting ambiguous instructions.

Therefore, for investment analysts, evaluating the commercial value of Agentic AI must involve distinguishing between its capabilities in "automation" and "generalized interaction." The OSWorld results suggest that AI's automation capabilities are mature enough in specific, controlled environments; however, in the pursuit of a general agent capable of navigating complex, ambiguous, and unstructured computer environments like a human, we are still in a phase that requires patience and critical observation.