Terminal-Bench

Properties

axis: coding
verification: det-test
whatItTests: 89 hand-crafted, human-verified command-line tasks spanning scientific computing, software engineering, ML, security, sysadmin and data science, each requiring an agent to complete an end-to-end terminal workflow (compiling, training, configuring, debugging) rather than emit a single patch, graded with task-specific tests.
discriminates: [build-verify-reflect]
saturation: open
patternLabFit: Tasks require multi-step terminal workflows rather than a single patch, giving a longer-horizon build-verify-reflect loop than SWE-bench-style tasks. Straddles the coding and agentic axes (structurally closer to OSWorld/tau-bench); scoped here as a coding-agent benchmark since its tasks are software-engineering-workflow-centric, but flagged for the assembler's judgment on axis placement.
source: Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces (arXiv:2601.11868)
notes: Published January 2026, within the 'significant recent benchmark' window. Distinct from the SWE-bench family (single-repo patch against a test harness): tasks span full terminal workflows across arbitrary tools, not just a git diff against one repository. Axis straddle (coding vs agentic, cf. osworld) is called out explicitly for the assembler. Version churn (re-verified 2026-08-22, corrected 2026-09-22 against a live tbench.ai/news fetch): the official tbench.ai news feed lists Terminal-Bench 2.0 (2025-11-07), 'a harder, better verified version of Terminal-Bench and a new package evaluating and optimizing agents', and Terminal-Bench 2.1 (2026-05-06), 'a revision of Terminal-Bench 2.0 that fixes 28 tasks' (a separate 'Continuous Benchmarks' post, 2026-07-30, 'Benchmarks are software and should be maintained like software', is the actual source of the continuous-validation framing, not 2.1 itself); the same feed also lists further variants (Terminal-Bench 3.0 Call for Contributions, 2026-03-05; Terminal-Bench-Science Call for Contributions, 2026-03-08; Terminal-Bench Challenges, 2026-06-18; Harbor-Index, 2026-06-29), so the tracked 89-task citation describes the original registry, not the currently versioned task set. Further re-verification (2026-09-22): the tbench.ai news feed confirms Terminal-Bench 3.0 actually shipped 2026-07-30 ('Terminal-Bench 3.0 measures agent abilities at the frontier'), correcting the earlier 'call for tasks' framing; also new since the last check are Terminal-Bench-Science 0.1 (2026-08-27, 'Evaluating AI agents on scientific research workflows') and Terminal-Bench 4.0 (2026-08-28, 'Calibrating task resources, fixing tasks, and removing saturated tasks'). No named model, score, or explicit saturation verdict for the tracked 89-task registry could be confirmed: the tbench.ai leaderboard table and swebench.com's Verified tab are client-rendered and returned no row data via the fetch tool available this run. Saturation left at 'open' pending a reachable score; this is a notes-only correction/update, not a saturation-value change. Pruning saturated individual tasks out of the 89-task registry (Terminal-Bench 4.0) is a maintenance action, not a maintainer statement that the registry as a whole has saturated.

Provenance

Source: https://arxiv.org/abs/2601.11868

Properties

axis

coding

verification

det-test

whatItTests

89 hand-crafted, human-verified command-line tasks spanning scientific computing, software engineering, ML, security, sysadmin and data science, each requiring an agent to complete an end-to-end terminal workflow (compiling, training, configuring, debugging) rather than emit a single patch, graded with task-specific tests.

discriminates

[build-verify-reflect]

saturation

open

patternLabFit

Tasks require multi-step terminal workflows rather than a single patch, giving a longer-horizon build-verify-reflect loop than SWE-bench-style tasks. Straddles the coding and agentic axes (structurally closer to OSWorld/tau-bench); scoped here as a coding-agent benchmark since its tasks are software-engineering-workflow-centric, but flagged for the assembler's judgment on axis placement.

source

Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces (arXiv:2601.11868)

notes

Published January 2026, within the 'significant recent benchmark' window. Distinct from the SWE-bench family (single-repo patch against a test harness): tasks span full terminal workflows across arbitrary tools, not just a git diff against one repository. Axis straddle (coding vs agentic, cf. osworld) is called out explicitly for the assembler. Version churn (re-verified 2026-08-22, corrected 2026-09-22 against a live tbench.ai/news fetch): the official tbench.ai news feed lists Terminal-Bench 2.0 (2025-11-07), 'a harder, better verified version of Terminal-Bench and a new package evaluating and optimizing agents', and Terminal-Bench 2.1 (2026-05-06), 'a revision of Terminal-Bench 2.0 that fixes 28 tasks' (a separate 'Continuous Benchmarks' post, 2026-07-30, 'Benchmarks are software and should be maintained like software', is the actual source of the continuous-validation framing, not 2.1 itself); the same feed also lists further variants (Terminal-Bench 3.0 Call for Contributions, 2026-03-05; Terminal-Bench-Science Call for Contributions, 2026-03-08; Terminal-Bench Challenges, 2026-06-18; Harbor-Index, 2026-06-29), so the tracked 89-task citation describes the original registry, not the currently versioned task set. Further re-verification (2026-09-22): the tbench.ai news feed confirms Terminal-Bench 3.0 actually shipped 2026-07-30 ('Terminal-Bench 3.0 measures agent abilities at the frontier'), correcting the earlier 'call for tasks' framing; also new since the last check are Terminal-Bench-Science 0.1 (2026-08-27, 'Evaluating AI agents on scientific research workflows') and Terminal-Bench 4.0 (2026-08-28, 'Calibrating task resources, fixing tasks, and removing saturated tasks'). No named model, score, or explicit saturation verdict for the tracked 89-task registry could be confirmed: the tbench.ai leaderboard table and swebench.com's Verified tab are client-rendered and returned no row data via the fetch tool available this run. Saturation left at 'open' pending a reachable score; this is a notes-only correction/update, not a saturation-value change. Pruning saturated individual tasks out of the 89-task registry (Terminal-Bench 4.0) is a maintenance action, not a maintainer statement that the registry as a whole has saturated.