Testing and Applying MCP Server Fixes With an Independent Conformance Suite

AgentRQ runs two MCP servers: a Core MCP that agents connect to directly, and a per-workspace MCP that fronts each workspace's OAuth, discovery, and tool surface. Both talk raw JSON-RPC over Streamable HTTP, negotiate a protocol revision, and have to get the lifecycle exactly right — initialize, then session, then tools. Our own test suite covers that behavior, but it's still the implementation grading its own homework. So we ran an independent, black-box conformance suite against the live servers: mcp-spec-test, built by HasMCP.

mcp-spec-test doesn't read our code at all. It drives a real running server through the actual protocol — handshakes, discovery, tool calls, error paths — the same way any MCP client would, and reports where the responses deviate from spec. That's a genuinely useful signal, because it catches the class of bug your own tests can't: the one where the implementation and the tests share the same wrong assumption.

Reproduce First, Fix Second

The suite doesn't hand you a diff — it hands you a failure report. Every item went through the same process before anything shipped: reproduce the exact request against our real workspace server, confirm the failure independently of the suite's own harness, then decide whether the report described an actual bug or a case where the suite itself had the spec wrong. Only after that did a fix — or a test — get written.

That second outcome turned out to matter as much as the first.

Bugs the Suite Actually Found

The protected-resource metadata pointed clients at a 404. Our OAuth discovery document emitted a non-standard singular authorization_server key holding the RFC 8414 metadata document URL. RFC 9728 §2 defines authorization_servers as an array of issuer identifiers, which the client derives the metadata URL from itself — so a strict client resolved a URL with the well-known path doubled and got nothing back. Both the Core MCP and per-workspace endpoints had this wrong, independently, because each derived its issuer separately, which is also how the two documents had drifted out of sync with each other in the first place. The fix collapsed both into one oauthIdentityFor() helper returning {issuer, resource, prmURL}, so the AS metadata and the protected-resource document can no longer disagree, and added the WWW-Authenticate: Bearer resource_metadata=... header RFC 9728 requires on every 401.

server/discover advertised a cache lifetime of zero. The result is a CacheableResult, and the SDK defaults ttlMs to 0 — which its own docs define as "immediately stale." Everything server/discover reports (advertised revisions, capabilities, the tool set) is fixed at server construction, so a zero TTL was telling every client to refetch static configuration on every single connection. We now set a one-hour TTL with cacheScope: "public", since the result carries no per-user state.

A newer protocol revision produced silence instead of an error. This one looked like three unrelated failures — tools/call missing content, tools/list returning something a client couldn't parse, initialize missing a protocolVersion — until we traced them to one cause. Our go-sdk version's supported revisions stopped at 2025-11-25, and its Streamable HTTP transport rejects any request carrying an MCP-Protocol-Version header outside that set with a plain-text HTTP 400, not a JSON-RPC error. The suite negotiated 2026-07-28 and sent that header on every subsequent request, so it parsed no JSON at all on any of them. The fix was a go-sdk upgrade to v1.7.0, which adds that revision and native server/discover support — the exact mechanism meant to prevent a client from guessing a revision the server can't serve.

We added a regression test that walks every revision server/discover advertises and asserts each one completes a full handshake, echoes its own version back, and yields a session that can actually list tools — the thing the suite couldn't get past.

When the Suite Was Wrong

Two reports described behavior that was already correct, and "fixing" either one would have broken the MCP lifecycle on purpose.

The suite flagged our server for refusing a second initialize call sent on a session that already completed one. That refusal is the spec working as intended — a session initializes exactly once, and the official MCP SDK's own test suite asserts this exact error ("second initialize error = %v, want duplicate initialize"). Accepting a repeat initialize would silently discard the negotiated state every later request on that session depends on, and would mask a real client bug: re-initializing an established session instead of starting a new one.

Separately, it flagged us for refusing a version-less tools/list sent before the handshake finished, on the theory that it "must be served, not refused." The reference SDK's own conformance fixture (lifecycle.txtar) asserts precisely the opposite: non-ping requests before initialized must be rejected. Serving that request to satisfy a third-party suite would mean failing the reference implementation's own tests.

Both got a test, not a fix — TestDuplicateInitializeOnSameSessionIsRefused and TestPreHandshakeRequestIsRefused — with the spec citations written directly into the test comments. The point of writing the reasoning down isn't the test coverage; it's making sure nobody reads a future conformance report and "fixes" a correct refusal back into a bug.

The Actual Workflow

Run the suite against a live Core MCP and workspace MCP instance. Take every failure and reproduce it by hand with a raw request against the real server, not the suite's abstraction of one. Check the failing behavior against the reference SDK's own conformance fixtures and tests before assuming the third-party suite is right — the two can and did disagree. Fix what's actually broken, and for anything the suite got wrong, write a test that pins the correct behavior down with the reasoning attached, so the next person — human or agent — doesn't "fix" it back into a bug six months later.

Both servers now run this suite as a standing regression gate alongside our own protocol tests. If you're running an MCP server of your own, mcp-spec-test from HasMCP is worth pointing at it — just don't take every failure it reports at face value. Reproduce it first.

---

AgentRQ is currently in public beta. Join our GitHub community to help shape the future of human-agent collaboration.