MCP-GRANITE

Properties

axis: agentic
verification: det-test
whatItTests: An open-source, extensible benchmark treating MCP tool-interface granularity as a controlled variable for edge- and IoT-deployed agents: 81 multi-step scenarios across 9 domains instantiated at 4 granularity levels (from fine-grained primitive tools to a single monolithic tool), evaluated on 9 locally-deployed models (268M-20.9B parameters) across 8,748 trials with task completion, tool-selection F1, argument accuracy, latency, and resource-usage metrics.
saturation: open
patternLabFit: Its finding that a 4-tool interface beats both fine-grained and monolithic extremes (task completion up 16.4% and 33.6% respectively, and argument accuracy nearly doubled) is a concrete design lever for how finely a Pattern Lab pipeline should decompose tools handed to a single agent role, independent of which model fills that role.
source: MCP-GRANITE Benchmark: GRANularity Interface TEsting for MCP-Based LLM Agents (arXiv:2609.24161)
notes: Submitted 2026-09-21, within the discovery window. Distinct from PlanBench-XL (tool-discovery at scale across 1,665 tools with induced failures) and tau-bench (dialogue-driven tool use): MCP-GRANITE's defining variable is interface granularity itself, held constant across 4 explicit levels, under edge/IoT resource constraints (locally-deployed models as small as 268M parameters). Verbatim finding: a 3.2B model at the optimal granularity outperforms a 20.9B model at a mismatched one.

axis: agentic
verification: det-test
whatItTests: An open-source, extensible benchmark treating MCP tool-interface granularity as a controlled variable for edge- and IoT-deployed agents: 81 multi-step scenarios across 9 domains instantiated at 4 granularity levels (from fine-grained primitive tools to a single monolithic tool), evaluated on 9 locally-deployed models (268M-20.9B parameters) across 8,748 trials with task completion, tool-selection F1, argument accuracy, latency, and resource-usage metrics.
saturation: open
patternLabFit: Its finding that a 4-tool interface beats both fine-grained and monolithic extremes (task completion up 16.4% and 33.6% respectively, and argument accuracy nearly doubled) is a concrete design lever for how finely a Pattern Lab pipeline should decompose tools handed to a single agent role, independent of which model fills that role.
source: MCP-GRANITE Benchmark: GRANularity Interface TEsting for MCP-Based LLM Agents (arXiv:2609.24161)
notes: Submitted 2026-09-21, within the discovery window. Distinct from PlanBench-XL (tool-discovery at scale across 1,665 tools with induced failures) and tau-bench (dialogue-driven tool use): MCP-GRANITE's defining variable is interface granularity itself, held constant across 4 explicit levels, under edge/IoT resource constraints (locally-deployed models as small as 268M parameters). Verbatim finding: a 3.2B model at the optimal granularity outperforms a 20.9B model at a mismatched one.