GPQA-Diamond

Properties

axis

knowledge

verification

det-gold

whatItTests

198 expert-validated, Google-proof PhD-level multiple-choice questions in biology, chemistry and physics.

discriminates

[dynamic-role][debate][reputation-elo]

saturation

saturated

patternLabFit

Dynamic role assignment shows the largest lift here, the reputation/ELO pattern operationalized. Letter answers are deterministically scored.

source

GPQA: A Graduate-Level Google-Proof Q&A Benchmark (arXiv:2311.12022)