PetriBench

Properties

axis: math
verification: det-gold
whatItTests: A compact, self-contained, procedurally-scalable benchmark for LLM reasoning over dynamic state spaces modeled as Petri nets (a formalism for concurrent and distributed systems), with four task families and Easy/Medium/Hard difficulty tiers generated by increasing structural complexity, scored against exact ground truth.
saturation: open
patternLabFit: Its procedurally-generated difficulty ladder with smooth scaling by structural complexity is a candidate stress test for whether a Pattern Lab pattern's benefit holds as task complexity increases rather than only at one fixed difficulty, and its exact-ground-truth scoring on formal state-space reasoning gives a clean oracle distinct from natural-language math.
source: PetriBench: Benchmarking LLM Reasoning over Dynamic State Spaces (arXiv:2609.19883)
notes: Submitted 2026-09-17, within the discovery window. Axis placed as math (closest existing category: formal, symbolically-verifiable reasoning with exact ground truth) rather than a dedicated reasoning axis, which the current vocabulary does not have; flagged for the assembler's judgment, similar to the axis straddle already noted on the tracked terminal-bench entry (coding vs agentic). Distinct from math-500: PetriBench tests structured concurrent-system state-space reasoning rather than competition mathematics, with procedurally generated, difficulty-scalable instances rather than a fixed 500-item set.

axis: math
verification: det-gold
whatItTests: A compact, self-contained, procedurally-scalable benchmark for LLM reasoning over dynamic state spaces modeled as Petri nets (a formalism for concurrent and distributed systems), with four task families and Easy/Medium/Hard difficulty tiers generated by increasing structural complexity, scored against exact ground truth.
saturation: open
patternLabFit: Its procedurally-generated difficulty ladder with smooth scaling by structural complexity is a candidate stress test for whether a Pattern Lab pattern's benefit holds as task complexity increases rather than only at one fixed difficulty, and its exact-ground-truth scoring on formal state-space reasoning gives a clean oracle distinct from natural-language math.
source: PetriBench: Benchmarking LLM Reasoning over Dynamic State Spaces (arXiv:2609.19883)
notes: Submitted 2026-09-17, within the discovery window. Axis placed as math (closest existing category: formal, symbolically-verifiable reasoning with exact ground truth) rather than a dedicated reasoning axis, which the current vocabulary does not have; flagged for the assembler's judgment, similar to the axis straddle already noted on the tracked terminal-bench entry (coding vs agentic). Distinct from math-500: PetriBench tests structured concurrent-system state-space reasoning rather than competition mathematics, with procedurally generated, difficulty-scalable instances rather than a fixed 500-item set.