Hermosa Research·Experimental
Do Multi-Agent Patterns Beat a Single Model?
Multi-agent patterns are usually argued from architecture diagrams. We measured them instead: 13 interaction patterns from our interaction-patterns catalogue ran against two one-model controls on verifiable math and code tasks, with token accounting and injected faults. Most patterns cost more and scored less than sampling one model well. The place structure earned its cost was fault containment, and there the deciding factor was the verification style, not the number of agents.
- Published
- 12 July 2026
- Data as of
- 12 July 2026
- Research type
- [experimental]Tests whether an intervention changes an outcome. The rigor lives in controls, ablations, and threats to validity.[tentative]Reasonably supported, but parts are still unverified or liable to drift.
- Topics
- [multi-agent-systems][agent-evaluation][self-consistency][llm-as-a-judge][fault-injection][benchmarks]
Method: We ran each pattern over full task pools for 8 epochs with gemini-2.5-flash-lite, scored by deterministic verifiers, with item-clustered confidence intervals, paired deltas against a single-model control, per-item token accounting, and fault-injection arms operationalised from the cascade-failures dataset.
What we ran
Every arm solves the same task pools: math word problems checked numerically and coding tasks checked by hidden unit tests, each run for 8 epochs with gemini-2.5-flash-lite in every role. Two controls anchor the comparison: a single model call, and self-consistency (five samples of the same model, majority vote on the parsed answer). The 13 patterns are compiled from the same topology knobs the catalogue describes: parallel or sequential producers, vote or synthesis or debate or router aggregation, and reflect loops. Confidence intervals are clustered by task item; deltas against the control are paired by item.
Fault arms rerun a subset of patterns while injecting a known-wrong value: either one producer is replaced with a confident wrong answer, or every producer is told an authoritative-sounding falsehood. Both operationalise rows from our cascade-failures dataset, and containment measures whether the injected value became the final answer.
Findings
- Sampling one model well beat every multi-agent pattern on math. Self-consistency scored 99.5%against the single call's 95.5%; the best pattern arm (teacher-student, 94.0%) did not close that gap, and no pattern beat self-consistency.
- Parallel patterns lost about 30 points to a temperature choice, not to topology. Independent sampling with five agent producers scored 65.0% on math while self-consistency, which is the same mathematics with different sampling settings, scored 99.5%. Our parallel producers sample at temperature 0.9 to decorrelate their errors; that design decision, common in multi-agent implementations, dominated the outcome. We confirmed the voting mechanics were not the cause by fixing vote grouping to parsed answers and A/B testing the raw-string variant.
- Structured debate actively hurt code. Every other arm tied the 98.3% single-model control on unit-tested code within noise; debate scored 79.2% because the arbiter rewrites working implementations.
- Verification style, not agent count, decided fault containment. With one corrupted producer, build-verify-reflect (deterministic verifier plus a retry loop) contained the injected answer in 99.5% of runs. The llm-as-judge pattern contained 0.0%: the judge model approved the confident wrong answer in every single run. This reproduces the verification rubber-stamp cascade from the failure-mode literature under controlled conditions.
- Structure multiplies cost regardless of benefit. Pattern arms spent between roughly 2x and 15x the LLM calls of the single-model control per item; the cost column in the dataset prices every arm.
The results
The full measurement tables, including per-arm confidence intervals, paired deltas, cost multiples, and the fault-containment breakdown. The same data downloads as CSV or JSON on the dataset page.
Math word problems
25 tasks, 8 epochs per arm, model gemini-2.5-flash-lite. Confidence intervals are clustered by task item.
| Arm | Kind | Pass rate | 95% CI | vs single-model | Calls/item | Cost multiple | n |
|---|---|---|---|---|---|---|---|
| self-consistency | [Control] | 99.5% | 98.5 to 100.5 | +4.0 pp | - | - | 200 |
| single-model | [Control] | 95.5% | 90.0 to 101.0 | baseline | 1.0 | 1.00x | 200 |
| teacher-student | [Pattern] | 94.0% | 88.9 to 99.1 | -1.5 pp | 3.5 | 3.48x | 200 |
| build-verify-reflect | [Pattern] | 93.5% | 87.3 to 99.7 | -2.0 pp | 2.0 | 2.05x | 200 |
| dynamic-topology-routing | [Pattern] | 88.0% | 78.8 to 97.2 | -7.5 pp | 6.7 | 6.65x | 200 |
| adaptive-mixture-of-agents | [Pattern] | 87.0% | 77.7 to 96.3 | -8.5 pp | 6.6 | 6.63x | 200 |
| structured-debate+judges:3 | [Judge x3] | 69.0% | 52.1 to 85.9 | -26.5 pp | 6.0 | 6.00x | 200 |
| mixture-of-agents | [Pattern] | 68.5% | 50.8 to 86.2 | -27.0 pp | 4.0 | 4.00x | 200 |
| spawn-and-isolate | [Pattern] | 67.5% | 51.0 to 84.0 | -28.0 pp | 5.0 | 5.00x | 200 |
| structured-debate | [Pattern] | 66.5% | 48.4 to 84.6 | -29.0 pp | 4.0 | 4.00x | 200 |
| concurrent-shared-state | [Pattern] | 66.5% | 50.0 to 83.0 | -29.0 pp | 5.0 | 5.00x | 200 |
| independent-sampling | [Pattern] | 65.0% | 46.9 to 83.1 | -30.5 pp | 5.0 | 5.00x | 200 |
| group-of-agents | [Pattern] | 65.0% | 48.5 to 81.5 | -30.5 pp | 5.0 | 5.00x | 200 |
| topology-task-routing | [Pattern] | 64.0% | 46.2 to 81.8 | -31.5 pp | 4.0 | 4.00x | 200 |
| llm-as-judge | [Pattern] | 63.0% | 44.9 to 81.1 | -32.5 pp | 2.0 | 2.00x | 200 |
| role-specialization | [Pattern] | 63.0% | 45.2 to 80.8 | -32.5 pp | 4.0 | 4.00x | 200 |
| llm-as-judge+judges:3 | [Judge x3] | 61.5% | 43.7 to 79.3 | -34.0 pp | 4.0 | 4.00x | 200 |
| independent-sampling+vote:raw | [Pattern] | 59.5% | 41.9 to 77.1 | -36.0 pp | 5.0 | 5.00x | 200 |
Unit-tested code
12 tasks, 8 epochs per arm, model gemini-2.5-flash-lite. Confidence intervals are clustered by task item.
| Arm | Kind | Pass rate | 95% CI | vs single-model | Calls/item | Cost multiple | n |
|---|---|---|---|---|---|---|---|
| build-verify-reflect | [Pattern] | 99.4% | 98.2 to 100.6 | +1.0 pp | 1.2 | 1.19x | 96 |
| teacher-student | [Pattern] | 99.4% | 98.2 to 100.6 | +1.0 pp | 2.3 | 2.25x | 96 |
| dynamic-topology-routing | [Pattern] | 99.2% | 97.5 to 100.8 | +0.8 pp | 4.5 | 4.52x | 96 |
| adaptive-mixture-of-agents | [Pattern] | 99.0% | 96.9 to 101.0 | +0.6 pp | 4.7 | 4.73x | 96 |
| single-model | [Control] | 98.3% | 95.1 to 101.6 | baseline | 1.0 | 1.00x | 96 |
| self-consistency | [Control] | 98.3% | 95.1 to 101.6 | +0.0 pp | - | - | 96 |
| llm-as-judge | [Pattern] | 98.3% | 95.1 to 101.6 | +0.0 pp | 2.0 | 2.00x | 96 |
| concurrent-shared-state | [Pattern] | 98.3% | 95.1 to 101.6 | +0.0 pp | 5.0 | 5.00x | 96 |
| topology-task-routing | [Pattern] | 98.3% | 95.1 to 101.6 | +0.0 pp | 4.0 | 4.00x | 96 |
| llm-as-judge+judges:3 | [Judge x3] | 98.1% | 94.5 to 101.8 | -0.2 pp | 4.0 | 4.00x | 96 |
| independent-sampling | [Pattern] | 97.5% | 94.5 to 100.5 | -0.8 pp | 5.0 | 5.00x | 96 |
| spawn-and-isolate | [Pattern] | 97.5% | 93.7 to 101.3 | -0.8 pp | 5.0 | 5.00x | 96 |
| group-of-agents | [Pattern] | 97.5% | 92.6 to 102.4 | -0.8 pp | 5.0 | 5.00x | 96 |
| mixture-of-agents | [Pattern] | 97.3% | 92.0 to 102.6 | -1.0 pp | 4.0 | 4.00x | 96 |
| role-specialization | [Pattern] | 96.7% | 90.1 to 103.2 | -1.7 pp | 4.0 | 4.00x | 96 |
| structured-debate | [Pattern] | 79.2% | 63.4 to 94.9 | -19.2 pp | 4.0 | 4.00x | 96 |
| structured-debate+judges:3 | [Judge x3] | 72.9% | 50.8 to 95.0 | -25.4 pp | 6.0 | 6.00x | 96 |
Fault containment
A known-wrong answer is injected at the producer step (operationalising rows from the cascade-failures dataset); containment measures whether it reached the final answer. Math pool only.
| Pattern | Injected fault | Containment | Pass rate under fault | Caught at | n |
|---|---|---|---|---|---|
| build-verify-reflect | Consensus lock-in (all producers poisoned) | 99.5% | 97.0% | aggregation:118 ignored:65 reflection:14 verifier:2 none:1 | 200 |
| mixture-of-agents | Consensus lock-in (all producers poisoned) | 96.5% | 92.5% | aggregation:167 ignored:26 none:7 | 200 |
| spawn-and-isolate | Consensus lock-in (all producers poisoned) | 95.5% | 93.5% | aggregation:172 ignored:19 none:9 | 200 |
| structured-debate | Consensus lock-in (all producers poisoned) | 95.5% | 88.0% | aggregation:159 ignored:32 none:9 | 200 |
| independent-sampling | Consensus lock-in (all producers poisoned) | 94.0% | 74.0% | aggregation:171 ignored:17 none:12 | 200 |
| llm-as-judge | Consensus lock-in (all producers poisoned) | 91.5% | 67.5% | aggregation:142 ignored:41 none:17 | 200 |
| build-verify-reflect | Hallucination fan-out (one corrupted producer) | 99.5% | 85.5% | reflection:171 verifier:28 none:1 | 200 |
| spawn-and-isolate | Hallucination fan-out (one corrupted producer) | 94.0% | 66.5% | aggregation:188 none:12 | 200 |
| structured-debate | Hallucination fan-out (one corrupted producer) | 92.5% | 75.5% | aggregation:185 none:15 | 200 |
| independent-sampling | Hallucination fan-out (one corrupted producer) | 91.0% | 63.0% | aggregation:182 none:18 | 200 |
| mixture-of-agents | Hallucination fan-out (one corrupted producer) | 86.0% | 64.0% | aggregation:172 none:28 | 200 |
| llm-as-judge | Hallucination fan-out (one corrupted producer) | 0.0% | 0.0% | none:200 | 200 |
| llm-as-judge | Verification rubber-stamp (corrupted producer, judge verifier) | 0.0% | 0.0% | none:200 | 200 |
Limitations
- One small model family: every role runs gemini-2.5-flash-lite. Patterns may pay off differently with stronger models, heterogeneous model mixes, or tasks a single call cannot attempt at all; the harness supports per-slot model mixes for that follow-up.
- Small pools: the math pool has 25 items and the code pool 12, so code cells are directional. The frontier tier exists because the model solved classic benchmark shapes at ceiling; only novel rule-system tasks with long simulation horizons discriminated.
- The judge is the same model as the producers, so judge results measure self-agreement, not cross-model judging.
- Fault containment matches the injected value against the concluded answer; paraphrased propagation would undercount. An earlier run of this grid was discarded after we found a 1024-token output cap was truncating long derivations before the answer line, which is documented in the research record along with both experiments.