Tool scope binding findings

Tool scope binding is the idea that when you give an agent access to a specific set of tools, it only uses those tools. This seems like a basic guarantee. It is not.

We designed a test suite with 40 scenarios across four frameworks. Each scenario declares a tool set and a task that can be completed using only the declared tools. We then inject references to undeclared tools in the task context, in retrieved documents, and in previous conversation history. The question is whether the agent attempts to call undeclared tools.

Results by injection method:

Task context injection: 23% of runs across all frameworks included attempts to call undeclared tools. The agent reads a tool name in the task description and assumes it is available.

Retrieved document injection: 18% of runs. Lower than task context because the agent treats retrieved content as less authoritative, but still significant.

Conversation history injection: 31% of runs. This was the highest rate. If a tool appears in the conversation history as something that was previously used, the agent is more likely to attempt calling it even if it is not in the current tool set.

The frameworks that performed best had explicit tool validation at the execution layer. Not just at the planning layer. An agent can "plan" to use an undeclared tool all it wants. The question is whether the execution layer catches the invalid call before it reaches the actual tool.

We found that frameworks with runtime tool validation had a failure rate under 5%. Frameworks that only validate at the prompt level had failure rates above 25%.

This finding directly informed dimension 7 of the Fifteen Standard (scope adherence). We now test both planning-level and execution-level tool scope violations separately.