ChartJudgeBench

Properties

axis: methodology
verification: mixed
whatItTests: A diagnostic vision-language benchmark assessing large multimodal model (LMM) judges used as critics and reward models in chart-to-code agentic refinement and RL optimization: 1,003 Chart Perception Alignment instances for pairwise chart comparison and 650 Chart Reasoning Judgment instances for binary Accept/Reject verification across Chart Reproduction and Chart Editing tasks.
saturation: open
patternLabFit: Extends the JudgeBench-style judge-reliability check into a visual, agentic-refinement setting; its four named failure modes (positional bias, overpredicting Accept, weak aesthetic matching, and an unexpected leniency bias in RL-trained models) are a concrete pre-flight checklist before trusting any Pattern Lab judge that scores visual or generated-artifact refinement rounds.
source: ChartJudgeBench: Evaluating LMM Judges for Chart-to-Code Generation (arXiv:2609.24210)
notes: Submitted 2026-09-21, within the discovery window. Distinct from JudgeBench (text-only, objectively verifiable knowledge/math/coding tasks) and MobileJudgeBench (mobile GUI agent trajectories): ChartJudgeBench is vision-language and scoped narrowly to chart-to-code judging decisions, specifically for judges used as RL reward signals in agentic chart refinement. Verification marked mixed rather than det-gold: the abstract does not state whether CPA/CRJ ground truth is human-annotated.

axis: methodology
verification: mixed
whatItTests: A diagnostic vision-language benchmark assessing large multimodal model (LMM) judges used as critics and reward models in chart-to-code agentic refinement and RL optimization: 1,003 Chart Perception Alignment instances for pairwise chart comparison and 650 Chart Reasoning Judgment instances for binary Accept/Reject verification across Chart Reproduction and Chart Editing tasks.
saturation: open
patternLabFit: Extends the JudgeBench-style judge-reliability check into a visual, agentic-refinement setting; its four named failure modes (positional bias, overpredicting Accept, weak aesthetic matching, and an unexpected leniency bias in RL-trained models) are a concrete pre-flight checklist before trusting any Pattern Lab judge that scores visual or generated-artifact refinement rounds.
source: ChartJudgeBench: Evaluating LMM Judges for Chart-to-Code Generation (arXiv:2609.24210)
notes: Submitted 2026-09-21, within the discovery window. Distinct from JudgeBench (text-only, objectively verifiable knowledge/math/coding tasks) and MobileJudgeBench (mobile GUI agent trajectories): ChartJudgeBench is vision-language and scoped narrowly to chart-to-code judging decisions, specifically for judges used as RL reward signals in agentic chart refinement. Verification marked mixed rather than det-gold: the abstract does not state whether CPA/CRJ ground truth is human-annotated.