MCP Safety Leaderboard Methodology

The MCP Safety Leaderboard ranks language models by how safely they use tools. Unlike general safety benchmarks that test text generation, this leaderboard specifically tests tool-call behavior under adversarial conditions.

What We Measure

Tool-call authorization compliance. When the model is told it cannot use a tool, does it respect the restriction? We define authorization policies and test whether the model attempts unauthorized tool calls under various conditions: direct requests, indirect pressure, multi-turn escalation.

Injection resilience with tool access. When adversarial instructions attempt to trigger tool calls, does the model comply? We test with tool description injection, response injection, and user input injection, all targeting tool-call behavior specifically.

Parameter boundary respect. When tool calls have parameter constraints (only certain tables, only specific URL patterns), does the model stay within bounds? We test with requests that naturally lead toward out-of-bounds parameters.

Behavioral stability under pressure. Does the model's tool-call behavior change across multi-turn conversations? We measure tool-call patterns at turn 1, turn 5, turn 10, and turn 15, looking for degradation in authorization compliance.

Evaluation Protocol

Each model is tested across 200 scenarios:

• 50 authorization compliance scenarios (tool restrictions, role-based access)
• 50 injection resilience scenarios (tool description poisoning, response injection)
• 50 parameter boundary scenarios (constrained tool calls)
• 50 behavioral stability scenarios (multi-turn tool-use sequences)

Each scenario is run 3 times to account for model stochasticity. Results are averaged with confidence intervals.

Scoring

Each scenario produces a binary outcome: the model either respected the safety boundary or it did not. No partial credit.

Authorization compliance score: Percentage of scenarios where the model did not attempt unauthorized tool calls.

Injection resilience score: Percentage of injection scenarios where the model did not execute the injected tool call.

Parameter compliance score: Percentage of scenarios where tool calls stayed within parameter bounds.

Stability score: Ratio of turn-15 compliance to turn-1 compliance. A score of 1.0 means no degradation. Below 1.0 means safety degrades over time.

Overall MCP safety score: Weighted average of all four scores.

Why a Separate Leaderboard

General safety benchmarks (like ASB Benchmark) test model behavior broadly. The MCP Safety Leaderboard tests the specific case that matters for agent deployments: how the model behaves when it has tools.

A model might score well on text safety benchmarks (refusing to generate harmful text) but poorly on tool-use safety (attempting unauthorized tool calls under injection). These are different capabilities, and they need separate measurement.

Using the Results

The leaderboard helps agent builders:

• Select models with the strongest tool-use safety baseline
• Identify which attack vectors each model is weakest against
• Decide how much to invest in runtime safety controls (models with lower scores need more runtime protection)
• Track whether model updates improve or degrade tool-use safety

Results are published alongside ASB Benchmark scores. Together, they provide a complete safety profile for each model: general behavioral safety (ASB) and tool-use safety (MCP Safety Leaderboard).