Arena-Hard-Auto

Properties

axis: methodology
verification: llm-judge
whatItTests: 500 challenging open-ended prompts curated automatically (BenchBuilder pipeline from Chatbot Arena/WildChat data) and scored via LLM-as-a-Judge; validated against human preference rankings and model-separation ability rather than gold correctness.
discriminates: [judge-calibration]
saturation: mid
patternLabFit: A methodology reference for validating an LLM-as-judge pipeline's alignment with human preference and its ability to separate close model performances (relevant wherever Pattern Lab uses judge-score to rank near-tied outputs).
source: From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipeline (arXiv:2406.11939)
notes: Distinct from JudgeBench: JudgeBench checks whether a judge picks the objectively-correct answer on verifiable tasks; Arena-Hard-Auto instead validates a judge pipeline's correlation with human preference and its power to separate models on open-ended prompts. Versioning: the official repo announced Arena-Hard-v2.0 on 2025-04-23 ('Better judges, new hard prompts, and additional eval for creative writing'); its v2.0-Preview set holds 500 fresh, challenging real-world user queries plus 250 creative writing queries sourced from Chatbot Arena, so whatItTests (500 prompts, v1 BenchBuilder pipeline) describes the original release.

axis: methodology
verification: llm-judge
whatItTests: 500 challenging open-ended prompts curated automatically (BenchBuilder pipeline from Chatbot Arena/WildChat data) and scored via LLM-as-a-Judge; validated against human preference rankings and model-separation ability rather than gold correctness.
discriminates: [judge-calibration]
saturation: mid
patternLabFit: A methodology reference for validating an LLM-as-judge pipeline's alignment with human preference and its ability to separate close model performances (relevant wherever Pattern Lab uses judge-score to rank near-tied outputs).
source: From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipeline (arXiv:2406.11939)
notes: Distinct from JudgeBench: JudgeBench checks whether a judge picks the objectively-correct answer on verifiable tasks; Arena-Hard-Auto instead validates a judge pipeline's correlation with human preference and its power to separate models on open-ended prompts. Versioning: the official repo announced Arena-Hard-v2.0 on 2025-04-23 ('Better judges, new hard prompts, and additional eval for creative writing'); its v2.0-Preview set holds 500 fresh, challenging real-world user queries plus 250 creative writing queries sourced from Chatbot Arena, so whatItTests (500 prompts, v1 BenchBuilder pipeline) describes the original release.