XYEval

Properties

axis: agentic
verification: mixed
whatItTests: A meta-evaluation framework that transforms an existing agent benchmark into an XY-problem test: mutates tasks so a user asks about their attempted (misleading) solution rather than the actual problem, then measures whether the agent resists the plausible-but-wrong suggestion and communicates its reasoning, evaluated across five models and six benchmark suites including tau2-bench.
saturation: open
patternLabFit: Its XY-mutation methodology is a ready-made stress test for whether a Pattern Lab producer agent actually pushes back on a misleading human or peer-agent suggestion rather than deferring to it; relevant wherever a pattern hands an agent authority to override a flawed instruction.
source: XYEval: Agents say yes to bad advice (arXiv:2609.23939)
notes: Submitted 2026-09-20, within the discovery window. Distinct from every tracked entry: no tracked benchmark specifically measures sycophancy toward a misleading user-proposed solution path. Verification marked mixed: scoring inherits whatever the underlying transformed benchmark uses across the six suites, not a single fixed method.

axis: agentic
verification: mixed
whatItTests: A meta-evaluation framework that transforms an existing agent benchmark into an XY-problem test: mutates tasks so a user asks about their attempted (misleading) solution rather than the actual problem, then measures whether the agent resists the plausible-but-wrong suggestion and communicates its reasoning, evaluated across five models and six benchmark suites including tau2-bench.
saturation: open
patternLabFit: Its XY-mutation methodology is a ready-made stress test for whether a Pattern Lab producer agent actually pushes back on a misleading human or peer-agent suggestion rather than deferring to it; relevant wherever a pattern hands an agent authority to override a flawed instruction.
source: XYEval: Agents say yes to bad advice (arXiv:2609.23939)
notes: Submitted 2026-09-20, within the discovery window. Distinct from every tracked entry: no tracked benchmark specifically measures sycophancy toward a misleading user-proposed solution path. Verification marked mixed: scoring inherits whatever the underlying transformed benchmark uses across the six suites, not a single fixed method.