AI / A CONCEPT NOTE
AI Gateways
a unified proxy layer between your app and multiple LLM providers
Overview · mechanism
pitfall · examples
01 / THE SHORT VERSION
The idea in a few sentences.
An AI Gateway sits between your application and LLM providers (OpenAI, Anthropic, Google, etc.) like a reverse proxy. It handles provider failover, request/response logging, rate limiting, cost tracking, caching, and guardrails — all in one place. Instead of hardcoding provider-specific SDKs everywhere, your app talks to the gateway, and it routes to the right provider.
02 / FOLLOW THE MECHANISM
How an ai gateway processes a request
Application
sends a standardized request to the gateway endpoint — same format regardless of which LLM provider will serve it.
Gateway middleware
applies rate limits, checks auth tokens, logs the request, and runs input guardrails (PII stripping, injection detection).
Provider selection
picks the target provider based on routing rules — primary provider with automatic failover to a backup if the primary is down or slow.
Response pipeline
logs tokens/cost, applies output guardrails, caches the result if configured, and returns the response to the application.
04 / COMMAND NOTES
Read the command, then the result.
Inspect the flags and arguments before trying an example. Snippets can need local setup, replacement values, or resources in your own environment.
call GPT-4o through Portkey AI Gateway
curl -X POST -H 'x-portkey-api-key: YOUR_KEY' -d '{"model":"gpt-4o","messages":[...]}' https://api.portkey.ai/v1/chat/completionsThe ellipsis omits required code or values. This sketch is not runnable as written.
run an open-source AI gateway locally
npx -y @portkey-ai/gateway05 / CHECK YOURSELF
Could you explain AI Gateways to a teammate?
Try it out loud in two sentences: what it is, and the one detail that changes the picture. If you stall, the gap is the part to reread.
Up next in AI engineeringExternal Integrationsconnecting AI agents to real-world APIs and SaaS platforms