COMMA (Communicative Multimodal Multi-Agent Benchmark)

Properties

axis: multi-agent
verification: det-test
whatItTests: Multimodal puzzle benchmark where two agents hold complementary/unequal information and must communicate in language to solve tasks neither can complete alone; scored by partial success rate and an efficiency score, with documented failure modes for AI-AI interaction.
discriminates: [spawn-and-isolate]
saturation: open
patternLabFit: COMMA's complementary-information puzzles (agents can't succeed without exchanging what only they can see) map directly onto Pattern Lab's spawn-and-isolate pattern; its finding that some chain-of-thought models fall below a random baseline is a sharp discriminator for whether a collaboration pattern is actually functioning versus decorative.
source: COMMA: A Communicative Multimodal Multi-Agent Benchmark (arXiv:2410.07553, TMLR 2025)
notes: Distinct from BattleAgentBench and MultiAgentBench: COMMA is multimodal (vision+language) and specifically isolates inter-agent communication under complementary/unequal information via puzzles, not text-only negotiation or staged cooperation/competition tasks. Also reports 'many chain of thought reasoning models such as R1-Onevision and LLaVA-CoT struggle to outperform even a random baseline in agent-agent collaboration.'

axis: multi-agent
verification: det-test
whatItTests: Multimodal puzzle benchmark where two agents hold complementary/unequal information and must communicate in language to solve tasks neither can complete alone; scored by partial success rate and an efficiency score, with documented failure modes for AI-AI interaction.
discriminates: [spawn-and-isolate]
saturation: open
patternLabFit: COMMA's complementary-information puzzles (agents can't succeed without exchanging what only they can see) map directly onto Pattern Lab's spawn-and-isolate pattern; its finding that some chain-of-thought models fall below a random baseline is a sharp discriminator for whether a collaboration pattern is actually functioning versus decorative.
source: COMMA: A Communicative Multimodal Multi-Agent Benchmark (arXiv:2410.07553, TMLR 2025)
notes: Distinct from BattleAgentBench and MultiAgentBench: COMMA is multimodal (vision+language) and specifically isolates inter-agent communication under complementary/unequal information via puzzles, not text-only negotiation or staged cooperation/competition tasks. Also reports 'many chain of thought reasoning models such as R1-Onevision and LLaVA-CoT struggle to outperform even a random baseline in agent-agent collaboration.'