Loading...
Loading...
Dataset·Experiment
Measured results for multi-agent interaction patterns on verifiable tasks: pass rates with confidence intervals against single-model and self-consistency controls, token costs per item, and how well each topology contains an injected fault. Browse the tables below, or take the raw files.
25 tasks, 8 epochs per arm, model gemini-2.5-flash-lite. Confidence intervals are clustered by task item.
| Arm | Kind | Pass rate | 95% CI | vs single-model | Calls/item | Cost multiple | n |
|---|---|---|---|---|---|---|---|
| self-consistency | [Control] | 99.5% | 98.5 to 100.5 | +4.0 pp | - | - | 200 |
| single-model | [Control] | 95.5% | 90.0 to 101.0 | baseline | 1.0 | 1.00x | 200 |
| teacher-student | [Pattern] | 94.0% | 88.9 to 99.1 | -1.5 pp | 3.5 | 3.48x | 200 |
| build-verify-reflect | [Pattern] | 93.5% | 87.3 to 99.7 | -2.0 pp | 2.0 | 2.05x | 200 |
| dynamic-topology-routing | [Pattern] | 88.0% | 78.8 to 97.2 | -7.5 pp | 6.7 | 6.65x | 200 |
| adaptive-mixture-of-agents | [Pattern] | 87.0% | 77.7 to 96.3 | -8.5 pp | 6.6 | 6.63x | 200 |
| structured-debate+judges:3 | [Judge x3] | 69.0% | 52.1 to 85.9 | -26.5 pp | 6.0 | 6.00x | 200 |
| mixture-of-agents | [Pattern] | 68.5% | 50.8 to 86.2 | -27.0 pp | 4.0 | 4.00x | 200 |
| spawn-and-isolate | [Pattern] | 67.5% | 51.0 to 84.0 | -28.0 pp | 5.0 | 5.00x | 200 |
| structured-debate | [Pattern] | 66.5% | 48.4 to 84.6 | -29.0 pp | 4.0 | 4.00x | 200 |
| concurrent-shared-state | [Pattern] | 66.5% | 50.0 to 83.0 | -29.0 pp | 5.0 | 5.00x | 200 |
| independent-sampling | [Pattern] | 65.0% | 46.9 to 83.1 | -30.5 pp | 5.0 | 5.00x | 200 |
| group-of-agents | [Pattern] | 65.0% | 48.5 to 81.5 | -30.5 pp | 5.0 | 5.00x | 200 |
| topology-task-routing | [Pattern] | 64.0% | 46.2 to 81.8 | -31.5 pp | 4.0 | 4.00x | 200 |
| llm-as-judge | [Pattern] | 63.0% | 44.9 to 81.1 | -32.5 pp | 2.0 | 2.00x | 200 |
| role-specialization | [Pattern] | 63.0% | 45.2 to 80.8 | -32.5 pp | 4.0 | 4.00x | 200 |
| llm-as-judge+judges:3 | [Judge x3] | 61.5% | 43.7 to 79.3 | -34.0 pp | 4.0 | 4.00x | 200 |
| independent-sampling+vote:raw | [Pattern] | 59.5% | 41.9 to 77.1 | -36.0 pp | 5.0 | 5.00x | 200 |
12 tasks, 8 epochs per arm, model gemini-2.5-flash-lite. Confidence intervals are clustered by task item.
| Arm | Kind | Pass rate | 95% CI | vs single-model | Calls/item | Cost multiple | n |
|---|---|---|---|---|---|---|---|
| build-verify-reflect | [Pattern] | 99.4% | 98.2 to 100.6 | +1.0 pp | 1.2 | 1.19x | 96 |
| teacher-student | [Pattern] | 99.4% | 98.2 to 100.6 | +1.0 pp | 2.3 | 2.25x | 96 |
| dynamic-topology-routing | [Pattern] | 99.2% | 97.5 to 100.8 | +0.8 pp | 4.5 | 4.52x | 96 |
| adaptive-mixture-of-agents | [Pattern] | 99.0% | 96.9 to 101.0 | +0.6 pp | 4.7 | 4.73x | 96 |
| single-model | [Control] | 98.3% | 95.1 to 101.6 | baseline | 1.0 | 1.00x | 96 |
| self-consistency | [Control] | 98.3% | 95.1 to 101.6 | +0.0 pp | - | - | 96 |
| llm-as-judge | [Pattern] | 98.3% | 95.1 to 101.6 | +0.0 pp | 2.0 | 2.00x | 96 |
| concurrent-shared-state | [Pattern] | 98.3% | 95.1 to 101.6 | +0.0 pp | 5.0 | 5.00x | 96 |
| topology-task-routing | [Pattern] | 98.3% | 95.1 to 101.6 | +0.0 pp | 4.0 | 4.00x | 96 |
| llm-as-judge+judges:3 | [Judge x3] | 98.1% | 94.5 to 101.8 | -0.2 pp | 4.0 | 4.00x | 96 |
| independent-sampling | [Pattern] | 97.5% | 94.5 to 100.5 | -0.8 pp | 5.0 | 5.00x | 96 |
| spawn-and-isolate | [Pattern] | 97.5% | 93.7 to 101.3 | -0.8 pp | 5.0 | 5.00x | 96 |
| group-of-agents | [Pattern] | 97.5% | 92.6 to 102.4 | -0.8 pp | 5.0 | 5.00x | 96 |
| mixture-of-agents | [Pattern] | 97.3% | 92.0 to 102.6 | -1.0 pp | 4.0 | 4.00x | 96 |
| role-specialization | [Pattern] | 96.7% | 90.1 to 103.2 | -1.7 pp | 4.0 | 4.00x | 96 |
| structured-debate | [Pattern] | 79.2% | 63.4 to 94.9 | -19.2 pp | 4.0 | 4.00x | 96 |
| structured-debate+judges:3 | [Judge x3] | 72.9% | 50.8 to 95.0 | -25.4 pp | 6.0 | 6.00x | 96 |
A known-wrong answer is injected at the producer step (operationalising rows from the cascade-failures dataset); containment measures whether it reached the final answer. Math pool only.
| Pattern | Injected fault | Containment | Pass rate under fault | Caught at | n |
|---|---|---|---|---|---|
| build-verify-reflect | Consensus lock-in (all producers poisoned) | 99.5% | 97.0% | aggregation:118 ignored:65 reflection:14 verifier:2 none:1 | 200 |
| mixture-of-agents | Consensus lock-in (all producers poisoned) | 96.5% | 92.5% | aggregation:167 ignored:26 none:7 | 200 |
| spawn-and-isolate | Consensus lock-in (all producers poisoned) | 95.5% | 93.5% | aggregation:172 ignored:19 none:9 | 200 |
| structured-debate | Consensus lock-in (all producers poisoned) | 95.5% | 88.0% | aggregation:159 ignored:32 none:9 | 200 |
| independent-sampling | Consensus lock-in (all producers poisoned) | 94.0% | 74.0% | aggregation:171 ignored:17 none:12 | 200 |
| llm-as-judge | Consensus lock-in (all producers poisoned) | 91.5% | 67.5% | aggregation:142 ignored:41 none:17 | 200 |
| build-verify-reflect | Hallucination fan-out (one corrupted producer) | 99.5% | 85.5% | reflection:171 verifier:28 none:1 | 200 |
| spawn-and-isolate | Hallucination fan-out (one corrupted producer) | 94.0% | 66.5% | aggregation:188 none:12 | 200 |
| structured-debate | Hallucination fan-out (one corrupted producer) | 92.5% | 75.5% | aggregation:185 none:15 | 200 |
| independent-sampling | Hallucination fan-out (one corrupted producer) | 91.0% | 63.0% | aggregation:182 none:18 | 200 |
| mixture-of-agents | Hallucination fan-out (one corrupted producer) | 86.0% | 64.0% | aggregation:172 none:28 | 200 |
| llm-as-judge | Hallucination fan-out (one corrupted producer) | 0.0% | 0.0% | none:200 | 200 |
| llm-as-judge | Verification rubber-stamp (corrupted producer, judge verifier) | 0.0% | 0.0% | none:200 | 200 |