OSWorld-Pro

Properties

axis: agentic
verification: llm-judge
whatItTests: Process-based evaluation for computer-use agents: over 300 tasks decomposed into more than 2,800 sequentially dependent subgoals, grounded in over 67,000 human annotations, scored by human-aligned LLM judges at each subgoal to reveal how and where an agent's execution breaks down rather than only whether the final deliverable succeeded.
discriminates: [build-verify-reflect]
saturation: open
patternLabFit: Subgoal-level judged scoring is a finer-grained build-verify-reflect hook than OSWorld/OSWorld 2.0's end-state checker, useful for localizing exactly which step in a Pattern Lab computer-use pipeline fails (subgoal-irrelevant actions vs click-based mistakes) rather than only whether the run passed overall.
source: OSWorld-Pro: Process-based Evaluation for Computer Use Agents (arXiv:2609.24890)
notes: Submitted 2026-09-21, within the discovery window. Distinct from tracked osworld (end-state functional verifiers only) and osworld-2 (same end-state scoring stretched over much longer horizons): OSWorld-Pro's defining feature is procedural, subgoal-level judging across a dependent task chain, revealing process failure modes (subgoal-irrelevant actions, click-based mistakes) invisible to a final-deliverable check. The same model scores lower once intermediate steps are graded rather than just the end state.

axis: agentic
verification: llm-judge
whatItTests: Process-based evaluation for computer-use agents: over 300 tasks decomposed into more than 2,800 sequentially dependent subgoals, grounded in over 67,000 human annotations, scored by human-aligned LLM judges at each subgoal to reveal how and where an agent's execution breaks down rather than only whether the final deliverable succeeded.
discriminates: [build-verify-reflect]
saturation: open
patternLabFit: Subgoal-level judged scoring is a finer-grained build-verify-reflect hook than OSWorld/OSWorld 2.0's end-state checker, useful for localizing exactly which step in a Pattern Lab computer-use pipeline fails (subgoal-irrelevant actions vs click-based mistakes) rather than only whether the run passed overall.
source: OSWorld-Pro: Process-based Evaluation for Computer Use Agents (arXiv:2609.24890)
notes: Submitted 2026-09-21, within the discovery window. Distinct from tracked osworld (end-state functional verifiers only) and osworld-2 (same end-state scoring stretched over much longer horizons): OSWorld-Pro's defining feature is procedural, subgoal-level judging across a dependent task chain, revealing process failure modes (subgoal-irrelevant actions, click-based mistakes) invisible to a final-deliverable check. The same model scores lower once intermediate steps are graded rather than just the end state.