MCP Servers, End to End

Executive Summary

The Model Context Protocol (MCP) is an open, JSON-RPC 2.0-based standard that gives AI applications one uniform way to discover and invoke external tools, read external data, and reuse prompt templates. Anthropic introduced it in November 2024, and in December 2025 donated it to the Agentic AI Foundation under the Linux Foundation, making it vendor-neutral. OpenAI, Google DeepMind, and Microsoft have all adopted it.

The Python and TypeScript SDKs alone see roughly 97 million downloads per month, and public registries index close to 20,000 servers as of early 2026.

The problem MCP solves is integration combinatorics. Before MCP, every AI application that wanted to touch GitHub, Postgres, Slack, or an internal ticketing system needed a bespoke adapter per provider — M applications × N tools = M×N integration projects. MCP collapses that to M+N: an application implements the client side once, a service implements the server side once, and everything interoperates. It is deliberately modeled on the Language Server Protocol, which did the same for editors and programming languages.

An MCP server is a small program — often under 100 lines — that exposes three primitives: tools (functions the model can call), resources (data the application can load into context), and prompts (reusable interaction templates). It speaks JSON-RPC over stdio (local) or Streamable HTTP (remote). The current stable spec is 2025-11-25; a major 2026-07-28 revision makes the protocol fully stateless.

For enterprises the value is standardized governance: OAuth 2.1 authorization is defined at the protocol level, tool invocation is auditable at a single choke point, and the same server serves Claude, ChatGPT, Cursor, and any in-house agent. The cost is a new attack surface — tool poisoning and prompt injection through MCP are now among the most-cited AI security risks of 2026 — which this guide treats as a first-class engineering concern, not a footnote.

Why This Matters

Without a protocol, “connecting AI to your systems” means glue code that rots. A team wires GPT function-calling to Jira; six months later they adopt Claude Code and rewrite the integration; a second team builds the same Jira adapter for their LangChain agent with slightly different schemas; nobody owns auth consistently, so three copies of a Jira API token live in three config files.

Every one of those copies is an audit finding waiting to happen — a 2026 analysis found 53% of MCP-adjacent deployments still expose credentials as hard-coded config values, and that number was worse in the pre-protocol era because there was no standard place to put auth at all.

Velocity. One server, not one integration per surface. The GitHub MCP server works identically from Claude Desktop, Claude Code, Cursor, and VS Code.

Blast radius. Tools invoked by a model are arbitrary code execution, triggered by a probabilistic system. A protocol gives one place to enforce least privilege and one audit format.

Vendor mobility. The same servers work across Claude, OpenAI, and Google clients — switching model providers stops being an integration rewrite.

Cost. Cloudflare’s “Code Mode” pattern demonstrated 98%+ token savings by letting agents discover and call MCP tools dynamically instead of loading all definitions up front.

Core Concepts

Host, client, server

The host is the AI application the user interacts with — Claude Desktop, Claude Code, Cursor, an agent framework. The host embeds one or more MCP clients, each holding a 1:1 connection to one MCP server. A host running with five configured servers runs five client connections. This 1:1 pairing matters for security reasoning: each server sees only its own conversation with its own client, never the full model context.

The three primitives

Tools. Model-controlled executable functions. Each has a name, description, and JSON Schema for inputs. The model decides when to call them; the host mediates with user consent. The description is the API documentation the model reads — exactly why poisoned descriptions are the top client-side attack vector.

Resources. Application-controlled data, addressed by URI (file:///etc/hosts, db://users/42/profile). They behave like GET endpoints — read-only context the host can load. Resource templates parameterize them.

Prompts. User-controlled reusable templates — “triage this incident”, “review this PR” — surfaced as slash-commands in most hosts. They encode your team’s best interaction patterns into the server itself.

The protocol is also bidirectional: servers can request sampling (ask the host’s LLM to generate a completion), elicitation (ask the user a structured question mid-operation), and learn which directories the host has scoped it to via roots. Since 2025-11-25, an experimental tasks primitive supports call-now-fetch-later for long-running work, and the MCP Apps extension (SEP-1865, Jan 2026) lets tools return sandboxed interactive HTML UIs instead of plain text.

Spec lineage

Worth memorizing — half the confusion online is version drift.

Revision	What changed
2024-11-05	Initial release. stdio + HTTP+SSE transports. No auth framework.
2025-03-26	Streamable HTTP introduced; HTTP+SSE deprecated. OAuth 2.1 baseline.
2025-06-18	Auth cleanly split: MCP server = resource server only. RFC 9728 Protected Resource Metadata mandatory.
2025-11-25	Current stable. CIMD client registration, S256-only PKCE, RFC 8707 resource indicators, experimental Tasks, Extensions framework.
2026-07-28	RC at time of writing. Fully stateless: initialize handshake and Mcp-Session-Id removed; any request can hit any server instance. MCP Apps and Tasks become official extensions.

Architecture Overview

The end-to-end flow, from a user’s question to a tool result landing back in model context:

Walk the request path for “why is checkout-service crashing in prod?”:

1. Discovery. Each client fetched capabilities at startup (tools/list, resources/list, prompts/list). Under spec ≤2025-11-25 this begins with an initialize handshake; under 2026-07-28 that’s gone — version/capability info rides in _meta on every request.
2. Model decision. Claude matches the question against the list_unhealthy_pods description and emits a tool call with {"namespace":"prod"}.
3. Consent. The host intercepts — first-use approval, per-call approval, or allowlisted auto-approval. Human-in-the-loop is a protocol design principle, not an implementation detail.
4. Invocation. The client sends tools/call over the transport — a stdin line, or a POST with a Bearer token.
5. Execution. The server validates arguments, hits the kube-apiserver with its own scoped credentials — never the user’s raw token — and returns a result.
6. Context injection. The client hands the result to the host, which appends it to model context. Claude reasons and possibly chains further calls.

Two properties do the heavy lifting here. First, the server never sees the conversation — only the specific calls routed to it. Second, the model never holds credentials — the server owns downstream auth, the host owns transport auth, and the LLM only ever sees tool names and results.

Deep Technical Breakdown

The wire protocol

Everything is JSON-RPC 2.0. A tool invocation looks like this on the wire:

{"jsonrpc": "2.0", "id": 7, "method": "tools/call",
 "params": {"name": "list_unhealthy_pods", "arguments": {"namespace": "prod"}}}

And the response:

{"jsonrpc": "2.0", "id": 7,
 "result": {"content": [{"type": "text", "text": "[{\"pod\": \"checkout-7f9c...\", \"phase\": \"CrashLoopBackOff\"}]"}],
            "isError": false}}

Note the two-level error model: JSON-RPC error objects signal protocol failures, while isError: true inside a successful response signals a tool-level failure the model should read and reason about. Well-built servers put actionable text in tool errors because the consumer is an LLM that can self-correct — a bare stack trace wastes that opportunity.

Notifications (no id, no response expected) carry list-change events, progress updates, and cancellations. Progress plus the newer Tasks primitive is how a server runs a 10-minute Terraform plan without holding a connection hostage: return a task handle immediately, let the client poll working → completed.

stdio internals — the one rule that breaks everyone

With stdio, the client spawns your server and the process boundary is the security boundary. Messages are newline-delimited JSON; the server MUST write nothing to stdout except valid MCP messages. A stray print(), an uncaught traceback, or a library that logs to stdout corrupts the framing and the client either errors or hangs forever.

Warning: dotenv v17+ prints a banner to stdout, silently breaking Node-based stdio servers until DOTENV_CONFIG_QUIET=true is set. Log to stderr, always — the spec reserves stderr for logging.

stdio’s economics: zero network overhead (OS pipes push 10,000+ ops/sec), zero auth complexity, automatic lifecycle. Its hard limit: one client per process, same machine only. Load-testing published in 2026 showed a stdio server failing 20 of 22 requests at just 20 simulated concurrent clients — it was never designed for that shape.

Streamable HTTP internals

One endpoint, three interaction patterns:

• Plain request/response — client POSTs JSON-RPC, server answers application/json. Cheap, cacheable, stateless.
• Streamed response — server answers with text/event-stream, pushing progress and partial results for that request, then closes.
• Server push channel — client GETs the same endpoint for a long-lived SSE stream of unsolicited server→client messages. Broken streams resume via Last-Event-ID.

Under 2025-11-25, sessions are optional: a server may return Mcp-Session-Id at initialization and require it thereafter, which forces sticky routing or a shared session store when scaling horizontally. The 2026-07-28 revision deletes protocol-level sessions entirely — any request can land on any instance; if your application needs state, mint an explicit handle (a deployment_id) as a tool result and have the model pass it back. It also adds Mcp-Method/Mcp-Name headers for body-blind gateway routing, plus ttlMs/cacheScope metadata so list results become cacheable.

Security requirements specific to this transport: validate the Origin header on every connection (DNS-rebinding defense), bind local dev servers to 127.0.0.1 rather than 0.0.0.0, and authenticate everything non-local.

Authorization: OAuth 2.1, done the MCP way

The auth story matured through three painful revisions and is now genuinely solid. The 2025-11-25 model:

• The MCP server is purely an OAuth 2.1 resource server. It validates access tokens; it never issues them. Token issuance belongs to an authorization server — your enterprise IdP or a hosted AS.
• Discovery is standardized. A protected server answers unauthenticated requests with 401 plus a WWW-Authenticate header pointing at its Protected Resource Metadata document (RFC 9728), served at /.well-known/oauth-protected-resource.
• PKCE is mandatory, S256 only. The implicit grant and plain PKCE are banned.
• Tokens are audience-bound. Clients MUST send RFC 8707 resource indicators, so a token minted for one server is useless against any other. This kills cross-server token replay.
• Client registration moved to Client ID Metadata Documents (CIMD): the client’s identity is an HTTPS URL hosting its own metadata, anchoring trust in DNS+TLS instead of per-server registration databases.
• Step-up authorization (new in 2025-11-25): a server can reject an under-scoped call with 403 plus the required scopes, letting the client re-authorize incrementally.
• No token passthrough. When your server calls a downstream API, it must obtain its own token for that API — never forward the client’s token. Forwarding creates the classic confused-deputy vulnerability, and the spec explicitly prohibits it.

stdio servers sit outside this framework: the spawning environment is the trust boundary, and credentials arrive as environment variables. That’s fine for a laptop and unacceptable for anything shared — the single strongest argument for moving multi-user workloads to Streamable HTTP.

Step-by-Step: A Real Kubernetes Ops Server

Theory done. Let’s build k8s-ops, a read-only server that lets Claude triage a cluster — list unhealthy pods, pull logs, read events. We’ll use FastMCP (~70% of MCP servers across all languages, ~1M downloads/day), run it locally over stdio, wire it into Claude Code, then promote it to remote Streamable HTTP.

1 · Project setup

mkdir k8s-ops-mcp && cd k8s-ops-mcp
uv init --python 3.12
uv add fastmcp kubernetes

2 · The server

server.py — complete and runnable, no placeholders:

"""k8s-ops: read-only Kubernetes triage tools over MCP."""
import json, sys
from fastmcp import FastMCP
from kubernetes import client, config

mcp = FastMCP("k8s-ops")
HEALTHY_PHASES = {"Running", "Succeeded"}

def core_v1() -> client.CoreV1Api:
    try:
        config.load_incluster_config()
    except config.ConfigException:
        config.load_kube_config()
    return client.CoreV1Api()

@mcp.tool
def list_unhealthy_pods(namespace: str = "default") -> list[dict]:
    """List pods not Running/Succeeded, with restarts and last reason."""
    pods = core_v1().list_namespaced_pod(namespace=namespace)
    unhealthy = []
    for pod in pods.items:
        phase = pod.status.phase or "Unknown"
        restarts = sum(cs.restart_count for cs in (pod.status.container_statuses or []))
        reason = None
        for cs in pod.status.container_statuses or []:
            if cs.state.waiting: reason = cs.state.waiting.reason
            elif cs.last_state and cs.last_state.terminated: reason = cs.last_state.terminated.reason
        if phase not in HEALTHY_PHASES or restarts > 3 or reason:
            unhealthy.append({"pod": pod.metadata.name, "phase": phase,
                               "restarts": restarts, "reason": reason,
                               "node": pod.spec.node_name})
    return unhealthy

@mcp.tool
def get_pod_logs(pod: str, namespace: str = "default",
                 container: str | None = None, tail_lines: int = 100) -> str:
    """Last N log lines; auto-retries with previous=True for crash diagnosis."""
    api = core_v1()
    tail_lines = min(tail_lines, 500)  # protect the model's context window
    try:
        return api.read_namespaced_pod_log(name=pod, namespace=namespace,
                                            container=container, tail_lines=tail_lines)
    except client.ApiException:
        return api.read_namespaced_pod_log(name=pod, namespace=namespace,
                                            container=container, tail_lines=tail_lines,
                                            previous=True)

@mcp.tool
def recent_events(namespace: str = "default", limit: int = 25) -> list[dict]:
    """Warning-type events in a namespace, newest first."""
    events = core_v1().list_namespaced_event(namespace=namespace, field_selector="type=Warning")
    items = sorted(events.items, key=lambda e: e.last_timestamp or e.event_time or "", reverse=True)[:limit]
    return [{"object": f"{e.involved_object.kind}/{e.involved_object.name}",
             "reason": e.reason, "message": e.message, "count": e.count} for e in items]

@mcp.prompt
def triage(namespace: str) -> str:
    """Structured incident-triage workflow for a namespace."""
    return (f"Triage namespace '{namespace}': 1) list_unhealthy_pods, "
            f"2) pull get_pod_logs + recent_events per pod, 3) classify root "
            f"cause, 4) propose the single safest next action. No destructive commands.")

if __name__ == "__main__":
    print("k8s-ops starting", file=sys.stderr)   # stderr, never stdout
    mcp.run()

Design notes worth stealing: descriptions are written for the model; tail_lines is capped server-side so a careless call can’t flood the context window; the log tool auto-falls-back to previous=True because that’s the 2am behavior you actually want; and everything is read-only — writes belong in a separate server with separate consent and RBAC.

3 · Test before wiring anything

npx @modelcontextprotocol/inspector uv run server.py

MCP Inspector opens a web UI to list tools, run them with real arguments, and see raw JSON-RPC frames. Five minutes here saves an hour of “why doesn’t Claude see my tool”.

4 · Register with Claude Code (stdio)

claude mcp add k8s-ops -- uv run --directory /home/supraj/k8s-ops-mcp server.py

Everything after -- is the spawn command. Default scope is local; add --scope project to write a committable .mcp.json. Verify with claude mcp list. For Claude Desktop, the JSON equivalent:

{
  "mcpServers": {
    "k8s-ops": {
      "command": "/home/supraj/.local/bin/uv",
      "args": ["run", "--directory", "/home/supraj/k8s-ops-mcp", "server.py"]
    }
  }
}

Use absolute paths — desktop clients launch servers with a minimal PATH, and “works in my terminal, fails in Claude” is almost always this. Fully restart the app after editing.

5 · Promote to remote (Streamable HTTP)

The tool logic doesn’t change; the transport line does:

if __name__ == "__main__":
    mcp.run(transport="http", host="0.0.0.0", port=8000)  # serves /mcp

Containerize it, deploy into the cluster it observes with a least-privilege ServiceAccount instead of anyone’s kubeconfig — RBAC scoped to get/list on pods, pods/log, events, and namespaces — nothing more. Two replicas behind a Deployment, requesting 100m CPU / 128Mi memory each. Then register the remote endpoint:

claude mcp add k8s-ops --transport http https://mcp.internal.corp/k8s/mcp \
  --header "Authorization: Bearer ${MCP_TOKEN}"

config.load_incluster_config() picks up the ServiceAccount token, and the RBAC — not the MCP layer — is what physically prevents a prompt-injected “please delete the deployment” from ever succeeding. In production, front this with full OAuth 2.1 (FastMCP 3.x ships auth providers, granular per-component authorization, and OpenTelemetry instrumentation) — the Bearer-header form is the minimum viable gate, not the destination.

Production Architecture

How this looks when it’s no longer one engineer’s laptop.

Gateway-fronted, registry-governed. Mature deployments put an MCP gateway between clients and servers: one ingress that terminates OAuth, enforces tool-level RBAC, rate-limits, logs every tools/call, and multiplexes many backend servers behind one endpoint. Gartner projects 75% of API gateway vendors will ship MCP features during 2026. Pair the gateway with a private registry — an allowlist of vetted servers — so “an engineer added a random npm MCP server” becomes structurally impossible.

High availability. Under 2025-11-25, a server issuing Mcp-Session-Id needs sticky sessions or a shared Redis store. The 2026-07-28 stateless revision eliminates this tax: no handshake, no session header, plain round-robin, replicas are cattle. Mint application-level handles from tools instead of leaning on protocol sessions, and horizontal scaling becomes an HPA setting.

Multi-region. Standard patterns apply: regional deployments behind latency-based DNS, with the caveat that SSE push channels pin to a region — design servers so the push channel is an optimization, not a correctness requirement. Keep servers close to the data they access, not close to the model.

Multi-tenant. The protocol doesn’t define tenancy, so you build it: tenant identity comes from the validated OAuth token’s claims, every downstream query is scoped by tenant ID, and per-tenant rate limits/audit trails live in the gateway. Never derive tenancy from tool arguments — the model populates those, and models can be manipulated.

Lifecycle reality for local servers. Enterprises also run fleets of stdio servers on developer machines. Govern those with managed configuration (Claude Code supports system-deployed managed-mcp.json with allowlist/denylist policies) rather than hoping — a config file is a supply chain vector, as the 2026 Claude Code poisoned-config RCE disclosure demonstrated.

Security Considerations

MCP security in 2026 is not hypothetical. Treat every number below as table stakes.

Threat model — the big four

1. Tool poisoning.

Malicious instructions embedded in tool metadata that the model reads but the user never sees. 5 of 7 major MCP clients perform no static validation of server-supplied metadata. Mitigations: pin versions, hash and review descriptions at registry-ingestion time, alert on description changes, and treat annotations from non-first-party servers as untrusted.

2. Prompt injection via tool results.

A tool that fetches a webpage or ticket returns attacker-controlled text straight into model context. Defense is architectural: least-privilege tools, human confirmation on state-changing calls, and egress controls.

3. Confused deputy / token misuse.

Fixed by the spec if you follow it: RFC 8707 resource-bound tokens, no token passthrough, S256 PKCE. Audit for servers that “helpfully” forward the client’s bearer token — the most common auth anti-pattern in community servers.

4. Supply chain.

MCP servers are arbitrary code you run with real credentials. 1,184 malicious skills were found on one agent marketplace; poisoned repo configs have achieved RCE in coding agents. Vet like any dependency: pin versions, prefer official servers, scan, sandbox.

Hardening checklist for a remote server

OAuth 2.1 resource-server validation against your enterprise IdP; PRM document at the canonical well-known path; scopes mapped to individual tools; Origin validation; TLS 1.2+; an audit log entry per tools/call with caller identity, tool name, arguments hash, and result status; secrets from a manager, never env-baked into images; downstream credentials scoped as tightly as the k8s RBAC example above.

For compliance: SOC 2 and ISO 27001 map cleanly once the gateway gives centralized authn/authz and immutable audit logs; HIPAA/PCI workloads additionally want no-tenant-data-in-tool-descriptions and DLP on tool results. NIST’s AI Agent Standards Initiative (launched Feb 2026) is worth tracking.

Performance Optimization

The real bottleneck is tokens, not milliseconds. Every registered tool’s name, description, and schema occupies context on every model call. Cursor enforces a ~40-tool limit per client, and a dozen chatty servers can burn tens of thousands of tokens before the user types a word.

1. Ship fewer, better tools. Ten well-described tools beat forty granular ones. Never mix read and write in one tool — it breaks scoping.
2. Dynamic tool loading. Cloudflare’s Code Mode — agents discover and call tools programmatically instead of front-loading all definitions — demonstrated 98%+ token savings. Claude Code’s Tool Search does a lighter version.
3. Trim tool output. Cap list sizes and log tails server-side. Return structured summaries with drill-down handles, not full dumps.
4. Cache list results. The 2026-07-28 ttlMs/cacheScope metadata makes list/read results cacheable.

Latency mechanics. stdio is effectively free (microsecond pipe I/O). Streamable HTTP adds ~10ms transport overhead under load, plus a few ms for gateway token validation — noise against hundreds of ms to seconds of LLM inference. The dominant real-world latency is your tool’s downstream work; instrument that, not the protocol.

Throughput. One stdio process = one client (50 devs × 8 servers = 400 uncontrolled processes org-wide). One HTTP replica of a well-written async server comfortably handles hundreds of concurrent in-flight tool calls if the tools are I/O-bound; scale replicas for CPU-bound work.

Cost Optimization

Token spend is the dominant line item. A 30-tool config adding ~8–10K tokens of definitions to every one of a team’s thousands of daily model calls is real money at API prices — measure it (/context in Claude Code shows the breakdown). Definition trimming and dynamic loading routinely cut 50–90% of MCP-attributable input tokens.

Infrastructure is cheap if you keep it boring. An MCP server is a thin I/O proxy: 100m CPU / 128–256Mi per replica is a sane starting request. Hidden costs to watch: cross-region egress; NAT gateway processing for heavy outbound calls from private subnets (~$0.045/GB quietly dominates small-instance costs); idle SSE connections holding LB capacity; and per-request gateway pricing since agentic workloads are chatty — a single question can fan out to 10+ tool calls.

Sampling shifts cost, not removes it. A server using MCP sampling consumes the client’s model budget — handy for avoiding API-key sprawl, dangerous for surprise bills if a tool samples in a loop. Cost attribution per agent/tenant is an unsolved protocol gap; implement it yourself in the gateway with per-token accounting keyed on the OAuth subject.

Monitoring & Observability

Instrument the server like any production API, plus MCP-specific signals.

Golden signals, MCP-flavored

• Latency — p50/p95/p99 per tool, not per endpoint. Alert p95 > 2× a tool’s 7-day baseline for 10 min.
• Traffic — tools/call rate per tool, per caller. Alert at 5× baseline sustained 5 min, page at 20×.
• Errors — separate JSON-RPC protocol errors from isError:true tool failures from auth failures. Tool-error rate >5% over 15 min warrants a ticket; >25% pages.
• Saturation — in-flight tool calls per replica, downstream connection-pool utilization, and (pre-stateless) session-store size.

Tracing. Propagate W3C trace context from host through server into downstream calls, so one user question renders as a single trace. FastMCP 3.x ships OpenTelemetry instrumentation — wire it to your existing collector.

Logging. Structured JSON to stderr or your log pipeline: timestamp, caller subject, tool, arguments hash (not raw arguments), duration, outcome. Keep a separate immutable audit stream of every state-changing call — this is also your tool-poisoning forensics.

Dashboards. One overview, one per-tool drill-down, one security view (new client identities, description-hash changes, calls outside business hours). The security view is the one nobody builds and everybody needs after the first incident.

Troubleshooting Guide

Server “connects” then hangs or errors on first call

Symptoms: Client waits forever; malformed-message errors in host logs

Likely cause: Non-JSON bytes on stdout — a stray print(), traceback, or library banner (dotenv v17+ is a known offender) corrupting stdio framing

Fix: Log to stderr only; set DOTENV_CONFIG_QUIET=true; audit every dependency that writes to stdout

Server works in terminal, fails in Claude Desktop/Code

Symptoms: spawn npx ENOENT or silent failure to start

Likely cause: Desktop clients launch subprocesses with a minimal PATH; nvm-managed Node isn’t loaded in non-interactive shells

Fix: Use absolute paths to binaries in config; on Windows wrap with cmd /c

Edited config, nothing changed

Symptoms: New server never appears

Likely cause: Client only reads MCP config at startup; window-close ≠ restart

Fix: Fully quit and relaunch; run claude mcp list to confirm registration

claude mcp add registered garbage

Symptoms: claude mcp get shows wrong command

Likely cause: Missing — separator, so server args were parsed as CLI flags

Fix: Remove and re-add with — before the spawn command

Remote server returns 401 loops

Symptoms: Client keeps re-authing, never succeeds

Likely cause: Missing/misplaced PRM document at the canonical well-known path (RFC 9728)

Fix: Serve PRM at the canonical path; include resource_metadata in the 401’s WWW-Authenticate header

Token valid but calls rejected

Symptoms: 403 with scope errors despite successful login

Likely cause: Scopes may arrive in scp (space-separated string) or roles (array); server checks only one

Fix: Validate both claim shapes; implement step-up (403 + required scopes) rather than failing terminally

Tools missing from model context despite healthy server

Symptoms: /mcp shows connected; model never calls tools

Likely cause: Too many tools registered — hosts cap or truncate (Cursor ~40); or descriptions too vague

Fix: Trim/consolidate tools; disable unused servers per project; rewrite descriptions with when-to-use guidance

SSE / “transport not supported” errors against older tutorials

Symptoms: Connection refused or 405 on /sse

Likely cause: Following legacy HTTP+SSE (deprecated 2025-03-26) docs against a Streamable HTTP server, or vice versa

Fix: Target the single /mcp endpoint; wrap legacy stdio-only servers with mcp-proxy for Streamable HTTP

Replica scale-out breaks sessions

Symptoms: Intermittent “unknown session” errors behind an LB

Likely cause: 2025-11-25 Mcp-Session-Id requires sticky routing or shared session store; round-robin scatters sessions

Fix: Consistent-hash on the session header, move state to Redis, or adopt the stateless 2026-07-28 pattern

stdio server collapses under parallel agents

Symptoms: Timeouts and dropped requests at ~20 concurrent clients

Likely cause: stdio is one-client-per-process by design

Fix: Move multi-client workloads to Streamable HTTP; keep stdio for single-user local tooling

Cluster tools fail only in-cluster

Symptoms: Forbidden from kube-apiserver inside the pod, works on laptop

Likely cause: Laptop used your admin kubeconfig; pod uses the ServiceAccount, whose RBAC is missing a verb/resource (pods/log is separate!)

Fix: Grant explicit RBAC including subresources; test with kubectl auth can-i —as=system:serviceaccount:…

Common Mistakes

Building write tools first.

The production-correct order is read-only triage tools, then narrowly-scoped writes behind confirmation, in a separate server with separate credentials. Every prompt-injection scenario is bounded by what your credentials can do — the model’s judgment is not a security control.

Treating tool descriptions as comments.

Descriptions are the interface. Vague ones cause wrong-tool selection and hallucinated arguments; over-long ones burn context.

Passing the client’s token downstream.

The confused-deputy special. The spec prohibits it; half the community servers do it anyway. Your server is a resource server up-stream and an OAuth client down-stream, with its own token.

Protocol-version drift.

Copying a 2024-era HTTP+SSE tutorial in 2026, or assuming DCR when the ecosystem moved to CIMD. Check the spec revision a document targets before trusting a code block.

Returning firehoses.

A tool that returns 4,000 log lines “to be safe” destroys the context window, slows every later turn, and raises cost. Summarize, cap, paginate, hand back drill-down handles.

One mega-server for everything.

Bundling GitHub + database + filesystem + deploy tools means one credential set with union-of-everything permissions and no per-capability consent. Split by risk domain.

Skipping the Inspector.

Debugging tool schemas through a chat UI is archaeology. npx @modelcontextprotocol/inspector shows raw frames in minutes.

Best Practices Checklist

• Read-only and write capabilities split into separate servers/scopes
• Every tool description states purpose, when-to-use, and constraints; reviewed like code
• Input schemas constrain aggressively (enums, maxima, formats); output sizes capped server-side
• Tool errors return actionable text (isError:true), not stack traces
• All logging to stderr / log pipeline — nothing but JSON-RPC on stdout
• stdio for local single-user; Streamable HTTP for anything shared; no new HTTP+SSE
• Remote: OAuth 2.1 resource-server validation, PRM at the canonical well-known path, S256 PKCE, RFC 8707 resource-bound tokens
• No token passthrough; server holds its own least-privilege downstream credentials
• Origin validation on; local dev binds 127.0.0.1
• Server versions pinned; third-party tool descriptions hashed and change-alerted
• Audit log per tools/call: identity, tool, args hash, outcome; immutable stream for writes
• OpenTelemetry traces from host → server → downstream
• Per-tool latency/error/traffic dashboards + auth-failure alerting
• Load test at expected agent concurrency (agents parallelize harder than humans)
• Designed stateless (application-level handles, not protocol sessions) — 2026-07-28-ready
• Tested end-to-end with MCP Inspector before wiring any host

Alternative Approaches

The honest decision rule: MCP costs you a process and a protocol but buys portability, governance, and an ecosystem. Below ~3 tools in a single-provider app, native function calling is simpler. Above that, or the moment a second client appears, MCP amortizes immediately.

Approach	What it is	Strengths	Tradeoffs	Use when
MCP server	open protocol, portable across hosts	one integration serves every client; standardized auth/consent; huge ecosystem; vendor-neutral governance	new attack surface; token overhead of tool definitions; spec still evolving fast	capability should be reusable across agents/products, or third parties consume it
Native function calling	define tools directly in the model API call	zero extra processes; lowest latency; full control of execution	locked to one provider’s schema; you rebuild auth/audit/consent yourself; N clients = N integrations	single application, single model provider, handful of tools, tight loop
Framework tools	tools registered inside an agent framework (LangChain/ADK/Strands)	deep framework integration (memory, planning); fast prototyping	portability ends at the framework boundary; frameworks now ship MCP adapters anyway	tool only ever lives inside one agent app — and even then, MCP-wrapping keeps options open
OpenAPI + generated adapter	point the agent at an existing REST spec	reuses existing APIs and docs verbatim	REST semantics ≠ agent semantics; too many endpoints, no when-to-use guidance; token-heavy	large existing API estate; use as a generator starting point, then curate
A2A protocol	peer protocol for agent↔agent delegation	models collaboration between autonomous agents, not tool calls	different layer — complements rather than replaces MCP	orchestrating multiple long-lived agents; use MCP for their tools, A2A between them

Industry Use Cases

Startups. Ship one MCP server as their “AI-native API” — Stripe, Linear, Sentry-style servers make the product usable from every agent surface, turning agent ecosystems into a distribution channel.

Enterprises. Deploy gateway-fronted internal server fleets — database-query, Jira/ServiceNow, internal-docs servers — giving employees governed agent access to systems of record with SSO and audit intact.

Saas Vendors. Expose tenant-scoped MCP endpoints so customers’ agents operate their product; tenancy is enforced from OAuth claims, and the endpoint becomes part of the product surface.

Fintech. Runs read-heavy analysis servers (positions, transactions, risk) with hard human-in-the-loop on anything money-moving; resource-bound tokens map onto existing SOX/PCI control language.

Healthcare. Wires clinical-context resources (FHIR queries, formulary lookups) into assistant workflows where the server enforces minimum-necessary access and PHI never appears in tool descriptions or logs.

Ai / Devops. Expose cluster triage (this article’s example), CI/CD status, incident timelines, and cost data to agents — the server does exact, audited operations; the LLM supplies orchestration and judgment.

Future Trends

Statelessness lands. The 2026-07-28 revision (final spec expected July 28, 2026) removes the initialize handshake and protocol sessions. Expect hosted MCP to become as operationally boring as any HTTP microservice.

Extensions become the innovation lane. The formal Extensions framework means MCP Apps and Tasks evolve without bloating core — enterprise features (audit trails, SSO, gateway behavior) are slated to land as extensions.

Discovery matures. MCP Server Cards — .well-known metadata for crawlers and registries — are on the roadmap, pushing toward an “agent-navigable web”.

Security standardization. With NIST’s interoperability profile expected Q4 2026, expect tool-poisoning defenses (description signing, static validation) to move from research into client defaults.

Agent-to-agent convergence. The likely shape: MCP for agent↔tool, A2A-style protocols for agent↔agent, with shared identity and governance layers spanning both.

Key Takeaways

• MCP turns M×N AI-tool integrations into M+N. It won because it’s boring: JSON-RPC, two transports, three primitives, LSP-shaped — and vendor-neutral under the Linux Foundation since December 2025.
• The decision tree is short: same machine, one user → stdio; anything shared or remote → Streamable HTTP; HTTP+SSE → legacy, don’t build it.
• Descriptions are the API. Write them for the model; review third-party ones as executable content — tool poisoning is the ecosystem’s #1 client-side attack.
• Security is layered around the model, never delegated to it: least-privilege downstream credentials, OAuth 2.1 with resource-bound tokens, no token passthrough, human confirmation on writes, audit everything.
• The scarce resource is context tokens, not CPU. Fewer/better tools, capped outputs, and dynamic tool loading are worth 50–98% of MCP-attributable spend.
• Design stateless today (application handles, not protocol sessions) and the 2026-07-28 world — round-robin LBs, cattle replicas — is a free upgrade.
• Build read-only first. A triage server that can’t destroy anything ships in an afternoon and earns trust; write capabilities come later, separately, gated.

Frequently Asked Questions

When should I choose stdio over Streamable HTTP, precisely?

stdio when exactly one client on the same machine launches the server and the spawning environment is an acceptable trust boundary. Streamable HTTP the moment a second concurrent user, a network hop, per-user auth, or centralized audit appears. Migration is a transport-line change in every major SDK.

Is the SSE transport dead, and will my old server break?

The two-endpoint HTTP+SSE transport was deprecated in 2025-03-26 and exists only for backward compatibility. Existing servers still function, but new code should migrate to Streamable HTTP — which, confusingly, still uses SSE internally as an optional streaming mode.

How do I scale an MCP server horizontally under the current stable spec?

If you issue Mcp-Session-Id, either consistent-hash route on that header or externalize session state to Redis. Better: don’t issue sessions at all — most tool servers don’t need them — and you’re already compatible with the 2026-07-28 stateless model.

What actually changes in the 2026-07-28 revision?

The initialize handshake and protocol-level sessions are removed; version/capabilities travel in _meta on every request; server/discover provides up-front capability fetch; Mcp-Method/Mcp-Name headers enable body-blind routing; ttlMs/cacheScope make results cacheable; Tasks + MCP Apps move to the Extensions track.

Can one MCP server serve Claude, ChatGPT, and Cursor simultaneously?

Yes — that’s the point. All three speak the protocol. Watch client capability differences: not every host supports sampling, elicitation, or resources equally, so degrade gracefully when a capability wasn’t negotiated.

How do tools, resources, and prompts differ in who controls them?

Tools are model-controlled (the LLM decides to call, host mediates consent). Resources are application-controlled (the host decides what to load). Prompts are user-controlled (invoked explicitly, e.g. slash-commands).

What’s the correct way to handle a tool that takes ten minutes?

Pre-Tasks: return immediately with a job handle, expose check_status(handle), emit progress notifications. With the Tasks primitive: return a task; the client polls or subscribes through working → completed/failed. Never hold a synchronous call open that long.

How does authentication work for stdio servers?

It doesn’t, at the protocol level — the trust boundary is process spawning, and credentials arrive as environment variables. Acceptable for personal tooling, indefensible for shared credentials or machines.

What is tool poisoning and how do I actually defend against it?

Malicious instructions hidden in tool descriptions/annotations that the model reads but the UI doesn’t render, persisting across every session. Defenses: vetted private registries, version pinning, hash-and-alert on description changes, render full descriptions in review tooling, least-privilege credentials.

Should my MCP server validate the OAuth token itself or delegate to a gateway?

Both are spec-compliant. Gateway validation centralizes policy and keeps servers thin — right for fleets. Direct validation (JWKS + local JWT verification) suits standalone public servers. Either way the server must still enforce scope→tool authorization decisions.

How many tools is too many?

Hosts truncate or cap (Cursor at roughly 40 across all servers), and tool-selection accuracy degrades as the menu grows. Practical bar: 3–12 tools per server, under ~30 active per host session.

What does the resource parameter (RFC 8707) protect against, concretely?

Token replay across servers. Without audience binding, a token granted to a benign server could be presented to a different MCP server trusting the same authorization server. Resource indicators name the token’s intended server; anything else rejects it.

Why does my server work in python server.py but fail when the client launches it?

Environment. Clients spawn subprocesses with a minimal PATH and none of your shell profile. Use absolute binary paths in configs, pass secrets via the config’s env block, and test with env -i to simulate the client’s world.

Can an MCP server call an LLM without holding an API key?

Yes — sampling. The server sends a sampling/createMessage request; the client’s model executes it under the host’s key and consent flow. Costs land on the client’s budget, so never sample in unbounded loops.

How do I expose an existing REST API as an MCP server without hand-writing every tool?

Generators exist (FastMCP ships OpenAPI import), but treat the output as a draft: curate down to the 5–15 operations agents genuinely need, rewrite descriptions with when-to-use guidance, collapse pagination chains into single purposeful tools.

What’s the difference between MCP and A2A?

Layers. MCP standardizes agent↔tool/data. A2A-style protocols standardize agent↔agent delegation between autonomous peers. They compose: each agent uses MCP for its tools and an agent protocol to talk to other agents.

How should multi-tenancy be implemented when the protocol doesn’t define it?

Derive tenant identity exclusively from validated token claims, thread it through every downstream query, rate-limit and audit per tenant at the gateway — and never trust tool arguments for tenancy.

Is it safe to commit .mcp.json to a repo?

Server definitions, yes — that’s the intended team-sharing mechanism. Credentials, never: reference environment variables or use OAuth flows. Treat the file as a supply chain surface and review config diffs like code.

What should tool errors look like for best agent behavior?

Structured, actionable text with isError: true — what failed, why, and what a valid retry looks like. Models self-correct remarkably well on good errors and flail on “500 Internal Server Error”. Reserve JSON-RPC errors for protocol-level faults.

How do I test an MCP server in CI?

Three layers: unit-test tool functions as plain functions; protocol-test with an in-memory client (list tools, call them, assert on content/schemas); contract-test the deployed HTTP endpoint against staging, including auth failure paths.

What is an MCP server, in one sentence?

A program that exposes tools, resources, and prompts to AI applications over the Model Context Protocol — an open JSON-RPC 2.0 standard created by Anthropic in 2024 and now governed by the Linux Foundation’s Agentic AI Foundation.