How would you design an LLM gateway for a company?

A gateway sits between internal applications and model providers. Its first job is a single API, often OpenAI-compatible, that can reach several hosted providers and our own models. Teams integrate once, and the platform team can add or swap models without every application changing code.

Second is control. Each team gets its own credentials, quotas, rate limits and monthly budget, and provider keys never leave the gateway. Routing policies decide which models a team may use and where traffic goes: for example, a cheaper model by default, a stronger one for specific use cases, and automatic fallback to another provider when one returns errors or exceeds a latency threshold. Retries use backoff and respect provider rate limits.

Third is efficiency and governance. Exact-match caching and provider prompt caching cut cost on repeated requests. Streaming is passed through without buffering. Every request is logged with team, model, tokens, cost and latency, with personal data redacted, so we can attribute spend and debug. Policy hooks can block sensitive data from going to external providers and keep regulated data in approved regions. The gateway itself must be highly available, since it is now on the critical path for every AI feature.