SRE-Bench

Properties

axis: agentic
verification: det-gold
whatItTests: Contamination-free binary reverse-engineering: 19 private, real-world-scale programs averaging 16.9K lines of code, combined with 44 in-house anti-analysis mechanisms to yield 262 binary instances and 1,572 deterministically graded tasks that an agent must solve by analyzing compiled binaries.
saturation: open
patternLabFit: A hard, deterministic-graded frontier with real headroom in the standardized setting (GPT-5.6-Sol and Claude-Fable-5.1 fully solve only 31.5% and 26.9% of graded instances), plus a clean capability-transfer finding (strong source-code security skill does not transfer to binary analysis) that cautions against assuming a pattern validated on one coding surface generalizes to adjacent ones. Headroom is shrinking at the very top: without a budget cap or safety guard, GPT-6-Astra reaches a near-perfect pass@4, though reliably selecting the correct candidate remains hard. Single-agent evaluation only; no collaboration discriminator asserted.
source: The Next Challenge for Agentic Cybersecurity: A Realistic, Contamination-Free Reverse Engineering Benchmark (SRE-Bench, arXiv:2608.11469)
notes: Submitted 2026-08-11; revised v2 2026-10-01. Benchmark name confirmed verbatim from the abstract: 'We introduce SRE-Bench, the first realistic, contamination-free RE benchmark.' v2 evaluates 13 agentic settings across 11 models: eight in public-facing settings and five in internal unconstrained settings with cyber safeguards disabled and no budget cap (the v1 five-frontier-LLM framing and the 61.4% best per-instance figure are no longer stated in the v2 abstract). The v2 paper body reports GPT-6-Astra at 99.2% pass@4 under the model-provider-internal evaluation. Distinct from every tracked entry: no tracked benchmark covers security/reverse-engineering of compiled binaries; the private-programs-plus-in-house-anti-analysis-mechanisms design is a contamination defense analogous to SWE-bench Pro's held-out repos but on a different surface.

Provenance

Source: https://arxiv.org/abs/2608.11469

Properties

axis

agentic

verification

det-gold

whatItTests

Contamination-free binary reverse-engineering: 19 private, real-world-scale programs averaging 16.9K lines of code, combined with 44 in-house anti-analysis mechanisms to yield 262 binary instances and 1,572 deterministically graded tasks that an agent must solve by analyzing compiled binaries.

saturation

open

patternLabFit

A hard, deterministic-graded frontier with real headroom in the standardized setting (GPT-5.6-Sol and Claude-Fable-5.1 fully solve only 31.5% and 26.9% of graded instances), plus a clean capability-transfer finding (strong source-code security skill does not transfer to binary analysis) that cautions against assuming a pattern validated on one coding surface generalizes to adjacent ones. Headroom is shrinking at the very top: without a budget cap or safety guard, GPT-6-Astra reaches a near-perfect pass@4, though reliably selecting the correct candidate remains hard. Single-agent evaluation only; no collaboration discriminator asserted.

source

The Next Challenge for Agentic Cybersecurity: A Realistic, Contamination-Free Reverse Engineering Benchmark (SRE-Bench, arXiv:2608.11469)

notes

Submitted 2026-08-11; revised v2 2026-10-01. Benchmark name confirmed verbatim from the abstract: 'We introduce SRE-Bench, the first realistic, contamination-free RE benchmark.' v2 evaluates 13 agentic settings across 11 models: eight in public-facing settings and five in internal unconstrained settings with cyber safeguards disabled and no budget cap (the v1 five-frontier-LLM framing and the 61.4% best per-instance figure are no longer stated in the v2 abstract). The v2 paper body reports GPT-6-Astra at 99.2% pass@4 under the model-provider-internal evaluation. Distinct from every tracked entry: no tracked benchmark covers security/reverse-engineering of compiled binaries; the private-programs-plus-in-house-anti-analysis-mechanisms design is a contamination defense analogous to SWE-bench Pro's held-out repos but on a different surface.