RivenGet started
Back to blog
EngineeringAugust 1, 2026

AI Gateway Architecture: How to Route LLM Traffic at Scale

What is an AI gateway?

An AI gateway is a proxy layer that sits between your application and one or more LLM providers. It handles request routing, authentication, rate limiting, billing, and failover — so your application code stays clean and provider-agnostic.

Without a gateway, every provider change means updating application code, re-deploying, and hoping nothing breaks. With a gateway, you switch providers by changing a config file.

Core gateway capabilities

1. Multi-provider routing

The gateway accepts OpenAI-format requests and routes them to the appropriate provider based on the model name:

# All requests go to the same endpoint
POST https://api.rivenai.io/v1/chat/completions

# The gateway routes based on model:
# gpt-5.6 → OpenAI
# claude-opus-5 → Anthropic
# glm-5.2 → self-hosted or Z.ai
# deepseek-v3 → DeepSeek

2. Automatic failover

When a provider goes down, the gateway retries on another provider:

{
  "model": "gpt-5.6",
  "messages": [...],
  "fallback_models": ["claude-opus-5", "glm-5.2"]
}

If OpenAI returns 503, the gateway automatically retries with Anthropic. Your application gets a response, not an error.

3. Rate limiting and budgets

Per-key rate limits prevent runaway costs:

  • Requests per minute (RPM)
  • Tokens per minute (TPM)
  • Daily budget caps ($)
  • Per-model restrictions

4. Usage tracking and billing

Every request is logged with:

  • Input/output token counts
  • Cost (based on current pricing)
  • Charge (based on retail rates)
  • Model, provider, latency, status code

This enables per-customer billing, cost attribution, and margin analysis.

Architecture patterns

Pattern 1: Single gateway, multiple providers

App → Gateway → OpenAI
                → Anthropic
                → Self-hosted
                → DeepSeek

Simplest setup. The gateway handles all routing. Good for most teams.

Pattern 2: Gateway + caching layer

App → Cache → Gateway → Providers

Add a semantic cache (e.g., Redis with embeddings) to skip the gateway entirely for repeated queries. Caching can cut costs by 30-60% for common queries.

Pattern 3: Multi-region gateway

App (US) → Gateway (US) → OpenAI (US)
App (EU) → Gateway (EU) → OpenAI (EU)
                    → Self-hosted (EU)

Deploy gateway instances in multiple regions for lower latency and data sovereignty.

Pattern 4: Gateway + queue for batch

App → Queue → Gateway → Providers
         ↓
      Batch processor

For bulk processing (classification, summarization), queue requests and process in batches. Providers like OpenAI offer batch API at 50% discount.

Riven's gateway architecture

Riven uses a Bifrost-based gateway with these layers:

Layer 1: Ultra-router

The ultra-router is the entry point. It:

  • Validates the API key
  • Resolves model aliases (e.g., gpt-4 → gpt-5.6)
  • Routes to the appropriate backend (cloud, on-prem)
  • Returns 404 for unknown models before hitting the gateway

Layer 2: Gateway (Bifrost)

The gateway handles:

  • Provider selection (which key, which endpoint)
  • Request forwarding with proper auth headers
  • Retry with exponential backoff
  • Response streaming (SSE)
  • Timeout management

Layer 3: Workspace console

The management layer provides:

  • Provider configuration (keys, models, concurrency)
  • Virtual key governance (per-key model access)
  • Routing rules (CEL expressions for model routing)
  • Pricing sync (daily auto-pull from pricing-brain)

Layer 4: Billing

Every request is metered:

  • Token counts from the response
  • Cost calculation from model-rates.json
  • Charge calculation from pricing-brain
  • Usage logging for audit and analytics

Key design decisions

OpenAI-compatible API

The gateway uses the OpenAI API format as the universal interface. Every provider's API is translated to/from OpenAI format. This means:

  • Your application code uses one SDK
  • Switching providers requires zero code changes
  • OpenAI ecosystem tools (LangChain, etc.) work automatically

Model aliases

Models have multiple names. The alias system maps them all:

{
  "gpt-4": "gpt-5.6",
  "gpt-4-turbo": "gpt-5.6",
  "openai/gpt-4": "gpt-5.6"
}

This handles legacy model names, provider prefixes, and version migrations transparently.

Routing rules

CEL expressions route models to specific providers:

model == "glm-5.2" → self-hosted (primary), Z.ai (fallback)
model == "gpt-5.6" → OpenAI
request.size > 100000 → deepseek-v3 (cheaper for large contexts)

Concurrency and buffering

Each provider has configurable concurrency limits:

{
  "concurrency": 100,
  "buffer_size": 500
}

This prevents overwhelming a provider and enables graceful degradation under load.

Production considerations

Monitoring

Track these metrics per provider:

  • P50/P99 latency
  • Error rate (4xx vs 5xx)
  • Timeout rate
  • Cost per million tokens
  • Cache hit rate (if caching)

Security

  • API keys stored in Docker secrets and Key Vault
  • Per-key model restrictions
  • Rate limiting at the gateway level
  • No key exposure to client applications

Compliance

  • Full request/response logging for audit
  • Data residency controls (self-hosted option)
  • Per-key budget caps
  • SOC 2 / HIPAA-ready architecture

Getting started

Riven's gateway is available as a managed service — no infrastructure needed:

  1. Sign up for a free account
  2. Get your API key
  3. Point your OpenAI SDK to https://api.rivenai.io/v1
  4. Use any of 75+ models
from openai import OpenAI

client = OpenAI(
    api_key="rvn_...",
    base_url="https://api.rivenai.io/v1"
)

response = client.chat.completions.create(
    model="gpt-5.6",
    messages=[{"role": "user", "content": "Hello!"}]
)

That's it. One key, one endpoint, 75+ models.

Start free on Riven — no credit card required.