Smoke mode

A refactor, a dependency bump or a new tool can break your MCP server without failing a single unit test. The server still starts. An agent that connects to it gets a tool whose result no longer matches the declared schema, or a handler that accepts arguments it should reject. Smoke mode catches that in CI: --smoke runs the normal audit, then calls your server’s tools over the same connection and checks the answers.

# One CI step: does my server still work?
mcpscore path/to/your/server.py --smoke

# Any-language servers and deployed servers work the same way
mcpscore --smoke --stdio ./my-go-server
mcpscore https://your-server.example/mcp --smoke --token $TOKEN

# Explicit consent to call every tool, not only read-only ones
mcpscore path/to/your/server.py --smoke --call-all

Running smoke checks — invoking tools/call, as requested with --smoke...

Audit finished. Final score: 75/105
Spec: 2025-11-25 negotiated (latest: 2026-07-28) · era: dual-era

Smoke checks (tools/call): 4 passed, 2 failed, 3 skipped — only readOnlyHint: true tools called (see --call-all). Never counted in the score.
  Tools called: 3 of 4 (a tool counts as called once the server answered a tools/call for it)
  pass smoke_structured_content [honest]: structuredContent conforms to the declared outputSchema
       returned 1 content block(s) [text]; structuredContent present; first text "{\"result\": \"ok\"}"
  FAIL smoke_structured_content [dishonest]: declared outputSchema is not honored: Invalid structured content returned by tool dishonest: 'result' is a required property
  skip smoke_structured_content [sloppy]: no outputSchema declared — the tool was not called for this check; declare an outputSchema to have its result verified
  skip smoke_structured_content [writer]: not called under the safety default (readOnlyHint is not true); pass --call-all to include it
  pass smoke_invalid_arguments [honest]: rejected with JSON-RPC error -32602
  pass smoke_invalid_arguments [dishonest]: rejected with JSON-RPC error -32602
  FAIL smoke_invalid_arguments [sloppy]: accepted schema-invalid arguments (returned a success result)
       returned 1 content block(s) [text]; first text "{}"
  skip smoke_invalid_arguments [writer]: not called under the safety default (readOnlyHint is not true); pass --call-all to include it
  pass smoke_unknown_tool: rejected with JSON-RPC error -32602
Gate failed — --smoke: 2 smoke check(s) failed

One run gives you the audit score and, under it, one line per smoke check. Each line reads verdict check_id [tool]: reason. This server has four tools: honest works, dishonest breaks its own output schema, sloppy accepts anything, and writer was never called because it is not annotated read-only.

--smoke invokes tools/call on the target. Run it against servers you operate. The ordinary audit never calls tools, so it is the right choice for anyone else’s server.

Smoke-test your own server

The steps below take a minimal server built with the MCP Python SDK from no coverage to a CI gate. Each step changes one thing and shows the run that proves it. The server has one tool:

server.py

from mcp.server import MCPServer

mcp = MCPServer("Basic Server")


@mcp.tool()
def add(a: int, b: int) -> int:
    """Add two numbers."""
    return a + b


if __name__ == "__main__":
    mcp.run()

Step 1: Run it and read the skips

Point mcpscore at the server and add --smoke. A server in its own project environment runs through uv run, the same way you start it yourself:

mcpscore --smoke --stdio uv run server.py

Smoke checks (tools/call): 0 passed, 1 failed, 2 skipped — only readOnlyHint: true tools called (see --call-all). Never counted in the score.
  Tools called: 0 of 1 (a tool counts as called once the server answered a tools/call for it)
  skip smoke_structured_content [add]: not called under the safety default (readOnlyHint is not true); pass --call-all to include it
  skip smoke_invalid_arguments [add]: not called under the safety default (readOnlyHint is not true); pass --call-all to include it
  FAIL smoke_unknown_tool: reported a tool execution error (isError) for a nonexistent tool — the spec's unknown-tool example is a protocol error, and an isError result claims the tool exists and ran
Gate failed — --smoke: 1 smoke check(s) failed

Tools called: 0 of 1. mcpscore calls only tools annotated readOnlyHint: true, and add has no annotations. The spec defaults a missing readOnlyHint to false, so add might write something, and smoke mode leaves it alone. Every skip says why, so the gap in coverage is visible. The failure is about the server, not about add. Step 3 explains it.

Step 2: Annotate the tool

Tell clients that add has no side effects. This is the one-line change:

server.py

from mcp.server import MCPServer
from mcp.types import ToolAnnotations

mcp = MCPServer("Basic Server")


@mcp.tool(annotations=ToolAnnotations(readOnlyHint=True))
def add(a: int, b: int) -> int:
    """Add two numbers."""
    return a + b

Run the same command again:

Audit finished. Final score: 116/135
...
Smoke checks (tools/call): 2 passed, 1 failed, 0 skipped — only readOnlyHint: true tools called (see --call-all). Never counted in the score.
  Tools called: 1 of 1 (a tool counts as called once the server answered a tools/call for it)
  pass smoke_structured_content [add]: structuredContent conforms to the declared outputSchema
       returned 1 content block(s) [text]; structuredContent present; first text "0"
  pass smoke_invalid_arguments [add]: rejected with an isError tool result
       returned 1 content block(s) [text]; isError: true; first text "Error executing tool add: 1 validation error for addArguments\na\n  Input should b" [truncated]
  FAIL smoke_unknown_tool: reported a tool execution error (isError) for a nonexistent tool — the spec's unknown-tool example is a protocol error, and an isError result claims the tool exists and ran

add is now called twice. With valid arguments it returned a result that matches the output schema the SDK derived from -> int. With a string where an integer belongs it refused, which is the rejection the check wants. A tool that answers gets a returned line: the number and types of content blocks, whether structuredContent was present, whether the result was flagged isError, and the first 80 characters of text. The annotation also raised the score from 114/135 to 116/135, because tools_annotations_present now passes. The same annotation tells every client that add does not modify its environment, and your smoke run now covers it.

Step 3: Read a failure

smoke_unknown_tool calls a tool name that cannot exist on your server and expects a JSON-RPC error, as Tools §Error Handling requires for unknown tools. This server answers with a tool result flagged isError, which tells the client the tool exists and ran. The MCP Python SDK answers this way by default, in both FastMCP (1.x) and MCPServer (2.x), so a server built on it fails this check until the SDK changes or the server rejects unknown names itself. A failure in your own tool reads the same way. The first example on this page shows the two most common ones:

  FAIL smoke_structured_content [dishonest]: declared outputSchema is not honored: Invalid structured content returned by tool dishonest: 'result' is a required property
  FAIL smoke_invalid_arguments [sloppy]: accepted schema-invalid arguments (returned a success result)
       returned 1 content block(s) [text]; first text "{}"

dishonest declares an output schema with a required result field and returns an object without it. sloppy returned a success result for arguments its input schema forbids. Both would reach an agent in production.

Step 4: Gate CI on it

Any smoke failure exits with code 4. Add the score gate the same way, and CI can tell a low score apart from broken tools:

# Fail the build on a low score or on broken tools, distinguishably
mcpscore --fail-under 80 --smoke --stdio uv run server.py

Exit code	Meaning
`0`	Score at or above 80, and every smoke check passed or skipped
`3`	Score below 80. Takes precedence when both gates fail
`4`	Score fine, but a smoke check failed

Raise the threshold to 90 against the 84% server from Step 1 and both gates fail. The score gate wins:

Audit finished. Final score: 114/135
Smoke checks (tools/call): 0 passed, 1 failed, 2 skipped — only readOnlyHint: true tools called (see --call-all). Never counted in the score.
Gate failed — --fail-under 90: score 114/135 (84%) is below the required 90%

Skips never gate. A server with no readOnlyHint: true tools and a correct unknown-tool rejection exits 0 and reports what it skipped and why. The full exit-code table is in the stability contract.

What each check catches

Each check reports pass, fail or skip. Skip means the check could not run. It never means the server failed it.

Check (`check_id`)	The regression it catches
`smoke_structured_content`	A handler no longer matches its declared `outputSchema`. Every callable tool that declares one is invoked, and the result must carry `structuredContent` that conforms to it, a spec MUST. An explicitly empty `{}` schema still requires the field to be present.
`smoke_invalid_arguments`	Input validation stopped working (Tools §Error Handling). Each callable tool is invoked with arguments that violate its schema and must reject them. A tool result flagged `isError` and a JSON-RPC error both count as rejection. Accepting the arguments, hanging, or crashing fails the check.
`smoke_unknown_tool`	Routing broke. A `tools/call` for a name that cannot exist on your server must be rejected with a JSON-RPC protocol error. A result, an `isError` result, and an internal error all fail.

For the two rejection checks, any JSON-RPC error code counts as a rejection except three: a request timeout, a closed connection, and -32603 internal error. Those three mean the server tried to run the call and failed. Each call has 30 seconds to answer.

Which tools get called

Only tools annotated readOnlyHint: true are called by default. An unannotated tool is skipped, and every skip is reported with its reason.

--call-all lifts that default. No other flag implies it, and it has no environment-variable form. On the unannotated server from Step 1, it calls add without the annotation:

mcpscore --smoke --call-all --stdio uv run server.py

Smoke checks (tools/call): 2 passed, 1 failed, 0 skipped — every tool called (--call-all). Never counted in the score.

The unknown-tool check uses a name derived from your server’s own catalog, so it cannot match a real tool. If the tool listing failed or is incomplete, the check skips instead of guessing a name.
Arguments come from each tool’s input schema, in this order: default, const, the first entry of examples, the first enum value, then a zero value for the type. No language model is involved, so the same server receives the same calls on every run.

Technical details: errors from upstream skip rather than fail

A tool that returns an error result for valid synthesized arguments reports as skip in smoke_structured_content. It may be rejecting the sample value, or a service behind it may be down, and an outage elsewhere should not fail your build. A JSON-RPC error to valid arguments skips for the same reason. Only behavior your server owns can fail a smoke check.Calls run one at a time, in catalog order: every structured-content call first, then every invalid-arguments call, then the unknown-tool call. A malformed schema fragment is tolerated rather than aborting the run.

Results stay out of the score

The score is deterministic: the same server gets the same score. Tool calls depend on upstream services and on the environment, so smoke results live in their own smoke section of the JSON report and never change score or max_score:

"smoke": {
  "executed": true,
  "reason": null,
  "call_all": false,
  "summary": { "passed": 4, "failed": 2, "skipped": 3 },
  "coverage": { "tools_called": 3, "tools_total": 4 },
  "checks": [
    {
      "check_id": "smoke_structured_content",
      "tool_name": "dishonest",
      "verdict": "fail",
      "message": "declared outputSchema is not honored: Invalid structured content returned by tool dishonest: 'result' is a required property",
      "details": {
        "basis": "MCP 2025-11-25 Tools §Output Schema (structured results MUST conform to the declared schema)",
        "called": true
      }
    },
    ...
  ]
}

When --smoke is requested but there is no session to call tools on, the section reports "executed": false with the reason instead of disappearing. That happens on a partial audit of an auth-gated server, on a modern-only probe audit, and when the tools catalog had to be recovered from the stateless lifecycle because the negotiated session could not list it.

Where it does not run

mcpscore.dev never runs smoke checks. Invoking tools on third-party URLs from our infrastructure is outside the website’s safety model, so smoke mode is CLI-only.
The GitHub Action accepts --smoke in args. A smoke failure fails the job after the report and PR comment are published. The comment does not list the smoke checks yet, and there is no dedicated smoke input.
Your test suite still has a job. Smoke checks prove the MCP contract: honest schemas, correct rejections, tools that answer. They do not test your business logic.

Recap

Run mcpscore --smoke against your own server and read the skip reasons.
Annotate read-only tools with readOnlyHint: true so smoke mode calls them.
Read each FAIL line as check, tool, and reason.
Gate CI with --fail-under and --smoke: exit 3 is the score, exit 4 is broken tools.

What’s next

GitHub Action

Run the same gate on every pull request

CI beyond GitHub

Exit codes and --json in any CI system

Stability Contract

Exit codes and the smoke report section

Authenticated Servers

Pass credentials so smoke can reach behind the gate

Get started

Guides

Use in CI

Scoring

Reference

Smoke-test your own server

Step 1: Run it and read the skips

Step 2: Annotate the tool

Step 3: Read a failure

Step 4: Gate CI on it

What each check catches

Which tools get called

Results stay out of the score

Where it does not run

Recap

What’s next

GitHub Action

CI beyond GitHub

Stability Contract

Authenticated Servers

​Smoke-test your own server

​Step 1: Run it and read the skips

​Step 2: Annotate the tool

​Step 3: Read a failure

​Step 4: Gate CI on it

​What each check catches

​Which tools get called

​Results stay out of the score

​Where it does not run

​Recap

​What’s next

GitHub Action

CI beyond GitHub

Stability Contract

Authenticated Servers

Smoke-test your own server

Step 1: Run it and read the skips

Step 2: Annotate the tool

Step 3: Read a failure

Step 4: Gate CI on it

What each check catches

Which tools get called

Results stay out of the score

Where it does not run

Recap

What’s next