Judge-Panel Bias Amplification

Properties

family: amplification
severity: systemic
triggerError: An LLM-as-judge ensemble is deployed to produce a more reliable evaluation signal, with agents running a multi-round debate to reach a verdict.
mechanism: Existing biases (position, verbosity, bandwagon) are amplified sharply after the first debate round and remain elevated in all subsequent rounds, so the multi-agent ensemble produces a more biased verdict than a single judge would have -- the social dynamic of debate turns a calibration problem into an entrenched directional error.
blastRadius: Every downstream decision, ranking, or reward signal derived from the panel verdict inherits the amplified bias; systems that use the panel as a trusted oracle are systematically steered by the distorted signal.
mast: [Cat 2][Cat 3]
detection: Bias audit against held-out gold labels across multiple panel runs; variance tracking across rounds (rising bias with round count is the signal); comparison against a calibrated single-judge baseline.
mitigation: Use meta-judge architectures (which the paper shows exhibit greater bias resistance) rather than open debate; inject a calibrated bias-free agent as a standing panel member; apply PINE-style debiasing before aggregation.
source: Judging with Many Minds: Bias Amplification in Multi-Agent LLM-as-Judge (arXiv:2505.19477)

family: amplification
severity: systemic
triggerError: An LLM-as-judge ensemble is deployed to produce a more reliable evaluation signal, with agents running a multi-round debate to reach a verdict.
mechanism: Existing biases (position, verbosity, bandwagon) are amplified sharply after the first debate round and remain elevated in all subsequent rounds, so the multi-agent ensemble produces a more biased verdict than a single judge would have -- the social dynamic of debate turns a calibration problem into an entrenched directional error.
blastRadius: Every downstream decision, ranking, or reward signal derived from the panel verdict inherits the amplified bias; systems that use the panel as a trusted oracle are systematically steered by the distorted signal.
mast: [Cat 2][Cat 3]
detection: Bias audit against held-out gold labels across multiple panel runs; variance tracking across rounds (rising bias with round count is the signal); comparison against a calibrated single-judge baseline.
mitigation: Use meta-judge architectures (which the paper shows exhibit greater bias resistance) rather than open debate; inject a calibrated bias-free agent as a standing panel member; apply PINE-style debiasing before aggregation.
source: Judging with Many Minds: Bias Amplification in Multi-Agent LLM-as-Judge (arXiv:2505.19477)