CoArena

Properties

axis: methodology
verification: human
whatItTests: A live, continuously-refreshed evaluation methodology for computer-use and multi-agent systems: real users submit tasks that two systems (single-model or multi-agent pipelines behind the same tool interface) execute concurrently in identical sandboxed desktops, users blind-judge the two outcomes, and a public leaderboard is refit via a penalized Bradley-Terry model with a streaming update that recovers Elo; the paper formalizes five measurable properties of what makes an evaluation real-time (continuous arrival, live concurrent execution, online updates, freshness and contamination resistance, bounded feedback latency).
saturation: n-a
patternLabFit: Direct methodological analog to Chatbot Arena's pairwise Bradley-Terry/Elo ranking, but purpose-built for computer-use and multi-agent pipelines rather than general chat; its formal contamination-resistance and freshness properties are a template for keeping a Pattern Lab leaderboard from going stale the way a fixed task set does.
source: CoArena: Evaluating Computer-Use and Multi-Agent Systems in Real Time (arXiv:2609.14239)
notes: Submitted 2026-09-13, within the discovery window. Distinct from Chatbot Arena (general chat, not computer-use/multi-agent specific) and from the static OSWorld/OSWorld 2.0 family: CoArena's defining feature is continuous, live, concurrent task execution judged by the real user who submitted the task, plus a full formal account of what makes an arena real-time. The paper itself states every reported number is either derived from stated inputs or explicitly labeled illustrative, with no measurement of a deployed system yet, so no headline model ranking is asserted here and saturation is marked n-a like the tracked Chatbot Arena entry.

Provenance

Source: https://arxiv.org/abs/2609.14239

Properties

axis

methodology

verification

human

whatItTests

A live, continuously-refreshed evaluation methodology for computer-use and multi-agent systems: real users submit tasks that two systems (single-model or multi-agent pipelines behind the same tool interface) execute concurrently in identical sandboxed desktops, users blind-judge the two outcomes, and a public leaderboard is refit via a penalized Bradley-Terry model with a streaming update that recovers Elo; the paper formalizes five measurable properties of what makes an evaluation real-time (continuous arrival, live concurrent execution, online updates, freshness and contamination resistance, bounded feedback latency).

saturation

n-a

patternLabFit

Direct methodological analog to Chatbot Arena's pairwise Bradley-Terry/Elo ranking, but purpose-built for computer-use and multi-agent pipelines rather than general chat; its formal contamination-resistance and freshness properties are a template for keeping a Pattern Lab leaderboard from going stale the way a fixed task set does.

source

CoArena: Evaluating Computer-Use and Multi-Agent Systems in Real Time (arXiv:2609.14239)

notes

Submitted 2026-09-13, within the discovery window. Distinct from Chatbot Arena (general chat, not computer-use/multi-agent specific) and from the static OSWorld/OSWorld 2.0 family: CoArena's defining feature is continuous, live, concurrent task execution judged by the real user who submitted the task, plus a full formal account of what makes an arena real-time. The paper itself states every reported number is either derived from stated inputs or explicitly labeled illustrative, with no measurement of a deployed system yet, so no headline model ranking is asserted here and saturation is marked n-a like the tracked Chatbot Arena entry.