MultiAgentBench

Properties

axis: multi-agent
verification: mixed
whatItTests: Evaluates LLM-based multi-agent systems across diverse interactive scenarios (negotiation, gaming, coordination) using milestone-based key performance indicators for collaboration/competition quality, not just task completion, across coordination topologies (star, chain, tree, graph).
discriminates: [dynamic-role]
saturation: open
patternLabFit: The milestone-based KPI methodology and topology comparison (star/chain/tree/graph) is a direct analog for evaluating which Pattern Lab coordination topology and strategy (e.g. group discussion, cognitive planning) most improves collaboration quality, not just final task success.
source: MultiAgentBench: Evaluating the Collaboration and Competition of LLM agents (arXiv:2503.01935)
notes: ACL 2025; code/datasets public at github.com/ulab-uiuc/MARBLE (also mirrored at github.com/MultiagentBench/MARBLE). Confirmed findings: graph topology performs best among coordination protocols in the research scenario, and cognitive planning improves milestone achievement rates by 3%. The 'producer-agreement' discriminator tag from the new-benchmarks lens submission is dropped: the paper's milestone-KPI framework does not report inter-rater/kappa agreement, so that tag is unsupported (matches the methodology-lens submission's own caveat: 'abstract does not specify inter-rater/kappa methodology, so producer-agreement/error-decorrelation tags are not asserted').

axis: multi-agent
verification: mixed
whatItTests: Evaluates LLM-based multi-agent systems across diverse interactive scenarios (negotiation, gaming, coordination) using milestone-based key performance indicators for collaboration/competition quality, not just task completion, across coordination topologies (star, chain, tree, graph).
discriminates: [dynamic-role]
saturation: open
patternLabFit: The milestone-based KPI methodology and topology comparison (star/chain/tree/graph) is a direct analog for evaluating which Pattern Lab coordination topology and strategy (e.g. group discussion, cognitive planning) most improves collaboration quality, not just final task success.
source: MultiAgentBench: Evaluating the Collaboration and Competition of LLM agents (arXiv:2503.01935)
notes: ACL 2025; code/datasets public at github.com/ulab-uiuc/MARBLE (also mirrored at github.com/MultiagentBench/MARBLE). Confirmed findings: graph topology performs best among coordination protocols in the research scenario, and cognitive planning improves milestone achievement rates by 3%. The 'producer-agreement' discriminator tag from the new-benchmarks lens submission is dropped: the paper's milestone-KPI framework does not report inter-rater/kappa agreement, so that tag is unsupported (matches the methodology-lens submission's own caveat: 'abstract does not specify inter-rater/kappa methodology, so producer-agreement/error-decorrelation tags are not asserted').