RecreationBench

Properties

axis: agentic
verification: mixed
whatItTests: A held-out evaluation suite of 250 diverse tasks across five platforms (Ubuntu, macOS, Windows, Android, Web) for hybrid computer-use agents that must both explore a graphical interface and write and run code: given a running reference application, an agent must reverse-engineer its behavior and rebuild a faithful implementation with no prescribed workflow, scored by reference-grounded programmatic and visual assertions validated by human reviewers before the suite is frozen.
discriminates: [build-verify-reflect]
saturation: open
patternLabFit: Its reference-grounded reverse-engineering task design (discover behavior, rebuild it, then run execution-grounded assertions against the rebuild) is a stricter build-verify-reflect loop than end-state-only computer-use benchmarks, and the finding that the best model passes all programmatic tests on only 2.8% of tasks despite 58.1% overall credit gives real headroom for testing whether a Pattern Lab pattern closes the gap between partial and full behavioral fidelity.
source: RecreationWorld: Scalable and Verifiable Environments for Hybrid Computer-Use Agents (arXiv:2609.22000)
notes: Submitted 2026-09-18 (v1), revised 2026-09-21 (v2), within the discovery window. Distinct from osworld / osworld-2 / osworld-pro (task specifications given upfront) and devicesworld (cross-device coordination, not reverse-engineering): RecreationBench's defining mechanism is recreation from a running reference with no prescribed workflow, spanning five platforms including Android, scored by both programmatic assertions and human-validated visual checks. The paper also names a training methodology (RecreationWorld) that improves transfer to five other out-of-distribution benchmarks, but the tracked entity here is the held-out RecreationBench evaluation suite specifically.

Provenance

Source: https://arxiv.org/abs/2609.22000

Properties

axis

agentic

verification

mixed

whatItTests

A held-out evaluation suite of 250 diverse tasks across five platforms (Ubuntu, macOS, Windows, Android, Web) for hybrid computer-use agents that must both explore a graphical interface and write and run code: given a running reference application, an agent must reverse-engineer its behavior and rebuild a faithful implementation with no prescribed workflow, scored by reference-grounded programmatic and visual assertions validated by human reviewers before the suite is frozen.

discriminates

[build-verify-reflect]

saturation

open

patternLabFit

Its reference-grounded reverse-engineering task design (discover behavior, rebuild it, then run execution-grounded assertions against the rebuild) is a stricter build-verify-reflect loop than end-state-only computer-use benchmarks, and the finding that the best model passes all programmatic tests on only 2.8% of tasks despite 58.1% overall credit gives real headroom for testing whether a Pattern Lab pattern closes the gap between partial and full behavioral fidelity.

source

RecreationWorld: Scalable and Verifiable Environments for Hybrid Computer-Use Agents (arXiv:2609.22000)

notes

Submitted 2026-09-18 (v1), revised 2026-09-21 (v2), within the discovery window. Distinct from osworld / osworld-2 / osworld-pro (task specifications given upfront) and devicesworld (cross-device coordination, not reverse-engineering): RecreationBench's defining mechanism is recreation from a running reference with no prescribed workflow, spanning five platforms including Android, scored by both programmatic assertions and human-validated visual checks. The paper also names a training methodology (RecreationWorld) that improves transfer to five other out-of-distribution benchmarks, but the tracked entity here is the held-out RecreationBench evaluation suite specifically.