TOPIC AI
AI Gateways a unified proxy layer between your app and multiple LLM providers
IN 10 SECONDS
An AI Gateway sits between your application and LLM providers (OpenAI, Anthropic, Google, etc.) like a reverse proxy. It handles provider failover, request/response logging, rate limiting, cost tracking, caching, and guardrails — all in one place. Instead of hardcoding provider-specific SDKs everywhere, your app talks to the gateway, and it routes to the right provider.
GOTCHA An AI Gateway is a single point of failure. If it goes down, every LLM call in your stack fails. Run it with high availability (multiple replicas, health checks) and always have a direct-to-provider fallback path for critical workflows.
HOW AN AI GATEWAY PROCESSES A REQUEST
01 Application sends a standardized request to the gateway endpoint — same format regardless of which LLM provider will serve it.
02 Gateway middleware applies rate limits, checks auth tokens, logs the request, and runs input guardrails (PII stripping, injection detection).
03 Provider selection picks the target provider based on routing rules — primary provider with automatic failover to a backup if the primary is down or slow.
04 Response pipeline logs tokens/cost, applies output guardrails, caches the result if configured, and returns the response to the application.
POKE IT YOURSELF
curl -X POST -H 'x-portkey-api-key: YOUR_KEY' -d '{"model":"gpt-4o","messages":[...]}' https://api.portkey.ai/v1/chat/completions — call GPT-4o through Portkey AI Gateway
npx -y @portkey-ai/gateway — run an open-source AI gateway locally
Drill this topic →
~80 sec read