MCPBench v0.1 — Methodology
This document describes how MCPBench tasks are created, how scores are computed, and how we defend against contamination. It is the citable reference for anyone interpreting leaderboard results.
Task design
Each task is a JSON document specifying a natural-language goal, an initial state for the MCP server, the subset of tools the agent may use, and a declarative success predicate. Tasks are assigned one of three categories:
Per server: 2 easy, 1 medium, 1 hard tasks in the public dev set. The private test set (16 per server) stratifies more finely across the three categories. Task parameters — filenames, row values, channel names, etc. — are sampled from bounded ranges so that tasks are reproducible but not trivially memorizeable.
Each task is human-reviewed for three properties: (1) the goal is unambiguous; (2) at least one correct solution path exists using only the declared available tools; (3) the success predicate correctly accepts all valid solutions and rejects all invalid ones.
Predicate grammar
Success predicates are evaluated against the final state of the MCP server's backing resource after the agent finishes (or hits a budget). Predicates are purely declarative — no code runs on the agent side.
Pass/fail is decided entirely server-side from that final state. The agent's self-report is never trusted: an agent that claims “completed” but leaves no verifiable state — or whose final state fails the predicate — is scored as a failure. That is what keeps the result neutral no matter who built the agent.
Primitive predicates are namespaced by server:
filesystem.fileExists,filesystem.fileContentMatches,filesystem.directoryHasFilesgithub.issueClosed,github.prMerged,github.repoHasFile,github.commitMatchesPatternpostgres.rowExists,postgres.tableExists,postgres.queryReturnsCount,postgres.tableHasColumn,postgres.columnEqualsslack.channelExists,slack.messagePosted,slack.channelHasMembers,slack.channelArchivedgmail.draftExists,gmail.messageLabelApplied,gmail.draftSent,gmail.messageIsReadbrowser.pageVisited,browser.urlIs,browser.formSubmittedcalendar.eventExists,calendar.eventHasAttendee,calendar.calendarExists,calendar.eventCountlinear.issueState,linear.issueHasLabel,linear.issueAssignee,linear.issuePrioritystripe.customerExists,stripe.invoiceExists,stripe.customerHasMetadata,stripe.refundIssuednotion.pageExists,notion.databaseHasRow,notion.pagePropertyEquals,notion.databaseExists
Compound predicates compose primitives: all([…]) requires every child to pass, any([…]) requires at least one, and not(…) inverts a child. This lets tasks specify multiple sufficient solution paths and assert what the agent must NOT have done (e.g. must not have deleted a channel it was only asked to archive).
Verifier implementations: TypeScript in packages/tasks/src/verifiers/ (used by the API to score hosted-endpoint submissions in-Worker), Python in packages/eval-runner/src/runner/verifier.py (used by the out-of-process runner). Both implementations are tested against a shared fixture corpus for conformance.
Scoring
Four metrics are reported per submission, per server, and aggregate. All metrics are computed server-side from the observed MCP trace — not from the agent's self-report.
tasks_passed / tasks_attemptedPass = predicate true, no budget exceeded, agent did not error out. This is the headline metric for leaderboard ranking.
mean(tool_calls_used / max_steps) over passed tasksLower is better. Only computed over passed tasks — a high-step failed task is not penalized twice. Expressed as a percentage.
calls_to_unlisted_tool / total_callsA call is hallucinated if the tool name is not in the task's available_tools list, regardless of whether the server accepts it.
recovery_tasks_passed_with_error / recovery_tasks_totalOnly on recovery-category tasks. A task that passes without any observed error is excluded — it gets credit in success rate but doesn't measure recovery.
A task is attempted iff the runner successfully dispatched it. Tasks that failed to dispatch due to infra errors on our side are excluded and counted separately as infra_errors.
Anti-contamination
Contamination — task content appearing in a model's training data — is the primary threat to benchmark validity. We layer five defences:
- Plaintext containment. The current month's private test set is never written to the open web in plaintext. Tasks exist in D1 (encrypted at rest), in the runner's memory during a run, and in the per-run MCP sandbox. Nowhere else.
- Sandboxed runs. Every agent run executes in an ephemeral Cloudflare Container with egress limited to the per-run MCP server URL. Agents cannot phone task content to an external store.
- Trace redaction. Run transcripts are stored with the goal text redacted by default. Submitters opt-in to publishing the full trace; opt-out is the default.
- Cryptographic attestation. Every task has a SHA-256 hash computed over its canonical JSON. When a month retires, we publish the
(task_id, hash, retired_on)list alongside the full plaintext. Anyone can verify that the published tasks match the hashes we attested during the active month. - Anomaly detection. Per-submitter score jumps> 2σ above the submitter's prior distribution flag for review. Per-task success-rate jumps across multiple submitters flag the task itself — that task may have leaked and will be retired early or perturbed.
The public dev set is explicitly assumed to be in training corpora. Scores against the dev set are not used for leaderboard ranking; they exist for development and as a baseline sanity check.
Private set rotation
The private test set (160 tasks, 16 per server) rotates monthly. Each month approximately 25% of tasks are replaced; ~75% carry over with predicate or initial-state perturbations to defeat trivial memorization. The retained tasks are perturbed at the parameter level — different filenames, row values, issue titles — so a model that memorized the exact prior task still faces a fresh challenge.
When a month retires, the full plaintext of that month's set is published with a Historical-NNNN-MM tag. Contemporary leaderboard scores reference the still-private current month; historical scores reference the now-public retired sets and are auditable.
Verifier conformance
Predicate evaluation is implemented twice: TypeScript (for API-side preview checks and task validation at CI time) and Python (for the production eval runner where the real MCP server resources live). The two implementations are tested against a shared fixture corpus to ensure they produce identical results for every predicate type across every server.
The conformance suite runs on every pull request. A task is not accepted into the dev set unless its predicate passes both implementations on its own initial state with a correct solution and fails both on an incorrect state.
Limitations
- End-state only. Predicates evaluate the final state of the MCP server, not the trajectory. An agent that achieved the goal by accident — deleting everything and recreating it — passes the same as one that took the elegant path (modulo efficiency).
- Single-server tasks. v0.1 tasks each involve exactly one MCP server. Cross-server composition (e.g. read a GitHub issue, write the result to Notion) is out of scope for this version.
- Mock servers, not production. All MCP servers run as mocks or sandboxed reference implementations. Behaviour may differ from production (rate limits, auth flows, edge cases in real APIs). Tasks are designed to avoid these edges, but coverage is not complete.
- No multimodal tasks. v0.1 is text-only. Screenshotting, image upload, PDF parsing are not covered.
- Public dev set in training data. We make no claim that dev-set scores are uncontaminated. Use private-set scores for any claim about uncontaminated performance.