Stability Contract

Build on rule_id, the report schema, the CLI shape, and the exit codes. Those change only with a major version. Do not build on messages, rule names, or the absolute score. Those move in minor releases, on purpose. This page is the contract, in force since 1.1.0.

​

Stable

​

Rule identifiers

A rule_id such as tools_description_present_in_all is never renamed and never reused for a different check. Rules are added in any release. A rule can be retired and stop running, but its identifier keeps its meaning forever. The rules reference lists retired ones too.

​

Report schema

The --json report carries schema_version, currently 1. Within a schema version, existing fields keep their names, types, and meanings. New fields appear in any release, so consumers must ignore unknown fields. A removal or type change bumps schema_version.

​

Package audits

package is null for every server audit. When it is an object, the report is a package audit and its score comes from the packaging rules alone, a rule set disjoint from server audits. Anything that compares or aggregates scores must branch on this field. For a package audit transport and spec.negotiated_version are both null.

​

Smoke results

smoke is absent unless the run used --smoke. It carries summary (pass/fail/skip counts), coverage (tools_called: distinct tools the server answered at least one tools/call for; tools_total: distinct tools the checks looked at), and checks. A check whose call completed sets details.called: true; a check that received a result also carries a bounded details.response (content_blocks, content_types, structured_content, is_error, and first_text capped at 80 code points). Consumers must not require it. Smoke results are never counted in score or max_score, and a smoke failure is reported through exit code 4, not through the report’s score fields. check_id values are stable in the same way rule_ids are.

​

Configuration

config is absent unless a mcpscore.toml or [tool.mcpscore] was applied (configure rules). When present, the score was computed under that policy: rules turned off did not run and appear in skipped_rules[] with reason disabled-by-config (a canonical skip reason is kept when the engine could not have judged the rule anyway), and re-ranked rules carry their configured severity in results[] with the default in details.severity_default. A configured score is comparable only across runs with the same config.sha256 and engine version. The badge and mcpscore.dev never apply a configuration.

​

CLI interface

The invocation shape is stable: mcpscore <target> with --json, --header, --token, and the MCPSCORE_TOKEN variable. New flags are added; existing flags keep their meaning. mcpscore --version prints mcpscore <version> to stdout, exits 0, and needs no target. --json writes exactly one JSON document to stdout with all logs on stderr. --sarif FILE writes SARIF 2.1.0 with one result per failed rule, keyed by ruleId = rule_id; the severity-to-level mapping and the primaryLocationLineHash fingerprint (a hash of the rule and target) keep their meaning, so code scanning alerts survive an upgrade.

​

Credentials

Header and token values are never logged and never written to the report. The report records only the boolean authenticated. This is a permanent commitment, not a default.

​

Evolving by design

We would rather grow the checks than freeze the number early. A calibrated scoring frame will revisit this once the ruleset settles.

​

Versioning

mcpscore follows SemVer. Breaking anything in the Stable section needs a major version. Rule additions and score movement are minor-version territory. Pre-releases (bN) can change anything.

​

Failure guidance

A failed rule may include suggested_fix: a nonblank plain-text repair hint of at most 255 Unicode code points. The field is omitted on passing results and on older findings without authored guidance; skipped rules remain separate. Consumers must tolerate its absence. In JavaScript, count code points with Array.from(text).length, rather than UTF-16 text.length. Supported findings include tool, resource, resource-template, prompt, server-metadata, capability, protocol, transport, auth/security, pagination, catalog-stability, readiness and package rules. The CLI prints their hint as Fix: on stderr. JSON stdout remains one complete report. SARIF appends sanitized repair advice to the finding message, using its stricter URL privacy policy; it does not invent source edits. Older findings still export without a repair hint. Optional metadata and list-change advice is labeled as a quality recommendation in the message; it is not a protocol mandate. Scores and applicability are unchanged. Schema failures retain existing detail keys and add details.issues, with a tool catalog index, a JSON Pointer field path, a reason and an expected constraint. These diagnostics explain the existing validator; they do not broaden it into complete JSON Schema conformance checking. A tool’s optional output schema is checked only when present. Arbitrary schema values are not copied into issues. Each tool contributes its first failing predicate; at most 20 issues are included. issues_total and issues_omitted describe the evidence limit. Schema and catalog validation paths longer than 255 Unicode code points are omitted with path_omitted: true; stored paths retain their exact JSON Pointer characters, including newlines, quotes and backslashes. Only CLI rendering escapes those characters. Indexes still identify the affected tool. Catalog metadata failures use the same bounded details.issues shape. Each issue identifies an entity_kind and a zero-based entity_index in the collected, concatenated catalog. Paths are relative to that item, using MCP wire field names: tool index 2 with /inputSchema/properties/query/description, for example. Prompt arguments and icons include their own indexes in the path. Server metadata paths start at /serverInfo, /instructions or /capabilities and have no catalog index. An empty-tools finding uses /tools relative to the catalog result. These new issues omit publisher names, URI values and image data. Indexes still identify entries with missing or duplicated names. Existing detail keys are retained for compatibility and are not covered by this new evidence bound. Human-readable messages can show server identity values on passes and up to three duplicate tool names on failures. Each quoted preview includes at most 60 source Unicode code points, with an explicit truncation marker; a value that is a single URL keeps up to 200, and a list renders item by item (at most 10 items, then a count). Printable Unicode stays readable; quotes, backslashes and nonprinting characters are escaped, so the rendered preview can be longer than the bound. The full values remain in legacy details. This is a display bound, not secret redaction. Website URL values are not echoed in passing messages. issues_total counts affected fields or occurrences, which can differ from a legacy count of distinct names or duplicate identifiers. At most 20 issues are included; overlong paths use path_omitted: true rather than a truncated pointer. Messages identify the first affected field; the complete bounded evidence remains in details. An optional field with an invalid declared value still needs repair. The guidance describes the current checks, not additional validation: server icons are checked for an https:// or data: prefix only, and a declared listChanged flag does not prove that notifications work. Header uniqueness still groups annotations by tool name; resolve duplicate tool names before treating cross-entry collisions as a confirmed error within one tool. Catalog collection failures are recorded in report listing_errors, and in details.collection_error when a declaration rule fails. Outcomes distinguish invalid_response, timeout, rpc_error and an unspecified collection_error. Validation locations and expected types are retained without raw response values or exception messages. Modern result objects with a missing or non-array tools field also receive sanitized field-shape diagnostics, while the probe outcome remains unsupported. These diagnostics do not change verdicts or score weights. Requirement strength is not inferred from severity or the presence of a hint. Use the rule’s source basis and applicability; a repair hint is advice for that finding, not a claim that every scored recommendation is mandatory.

​

Protocol and probe evidence

Failed protocol/probe checks retain their observations and can add details.expected for the particular fields checked. Human messages include observed error codes/statuses only when present; stdio messages omit HTTP status. Cursor probes distinguish a returned page from an unexpected response, rather than inferring success from an absent error code. Audit context is recorded once: transport at report level and spec.negotiated_version and spec.session_protocol_version in the specification block. The latter is null unless a legacy session was established. Existing target_version, spec and basis fields retain the rule’s revision and source. Unavailable probes still skip; a hint cannot turn missing evidence into failure. Readiness result-field issues use entity_kind: "response", the originating probe_id, and paths relative to the JSON-RPC result. missing_or_null means the collected observation cannot distinguish omission from explicit null; invalid_value means a value was observed but failed the existing predicate. Tool-schema issues use catalog indexes and paths relative to each tool, with the same 20-issue and 255-code-point path bounds as catalog findings. A network reference finding identifies the containing schema; it does not prove a fetch. Rule and probe evidence copies mask response_session_id as [redacted] (absence stays null), remove URL userinfo, and mask known credential query keys: access_token, refresh_token, id_token, token, api_key, apikey, client_secret, password, secret, authorization and code. Key matching is case-insensitive after URL decoding and treats hyphens like underscores. All scheme:// URLs (including postgres and HTTP) are matched case-insensitively, including dictionary keys and URLs embedded in challenge headers and exception text. Public query parameters, spelling, order, fragments, WWW-Authenticate challenges and diagnostic exception reasons remain available. If masked dictionary keys collide, retain every entry using deterministic [masked name N] suffixes. Literal keys are reserved before allocating aliases; summary maps with the same source names use the same aliases. These aliases are report labels, not original tool names; use catalog indexes to identify tools. Human messages restore bounded quoted previews of resource/issuer identifiers, version lists, Content-Type values and schema-validator reasons. They use the same bounded preview and control-character escaping described above, after credential masking. Indexed schema issues still omit raw values, and affected tools are counted by catalog index even when their names collide. For readiness schema failures, offending_tools retains its legacy name-keyed shape with a union of short, value-free problem labels. offending_tool_indexes maps each name to all affected catalog indexes, including duplicate names; unlike bounded issues, this summary index mapping is complete. Validator previews appear only in the human message, not in these structured summaries. These protections do not mutate observations used for scoring and are not a general secret detector. Arbitrary error text and public metadata can contain sensitive information outside these specific patterns; review a report before publishing it. Passing PKCE findings establish advertised S256 support, not an executed authorization flow. Error-hygiene passes mean no configured patterns were found in the sampled response, and catalog-stability passes compare two connections with the same authorization. Modern-protocol hints preserve valid legacy paths; readiness promotion into the main score is unchanged. Some existing predicates have narrower discovery coverage than the specifications. The challenge-origin finding describes mcpscore’s current comparison; it does not instruct relocating valid metadata. The dated MCP specification permits challenge or well-known discovery, and RFC 9728 section 3.3 requires resource identity validation. Likewise, issuer metadata failures describe the locations attempted, rather than proving no valid document exists. These limitations require separate correctness changes, not different repair wording.

​

What’s next

​

Package evidence

Package hints distinguish publisher metadata changes from consumer choices. npm deprecation and PyPI yanking retain their registry-specific names and publisher notices when available (details.withdrawal_reason). Human notice previews allow 200 Unicode code points so replacement URLs in prose remain useful; masking and control-character escaping happen before display. Longer notices are visibly truncated, with the full masked notice retained in details. Existing yanked detail values and verdicts remain unchanged. Public values use bounded quoted previews; URL credentials are masked in result messages and details. A missing license declaration or repository URL describes metadata, not legal permissions or whether the source exists elsewhere. Package audits never run the package and cannot confirm its MCP behavior.