AI Cost Optimization: How to Cut Your LLM Bill in Half

The cost problem

LLM costs scale linearly with usage. If you're paying $100/day at 1,000 requests, you'll pay $1,000/day at 10,000. Without optimization, costs become unsustainable fast.

This guide covers seven strategies that can cut your LLM bill by 50% or more without sacrificing quality.

1. Model routing by complexity

Not every request needs a frontier model. Route based on task complexity:

def route_model(messages):
    total_chars = sum(len(m["content"]) for m in messages)
    
    # Simple, short requests → cheap model
    if total_chars < 200:
        return "glm-5.2"  # $0.50/$1.50 per 1M tokens
    
    # Code-related → specialized cheap model
    if any(kw in messages[-1]["content"].lower() for kw in ["code", "function", "debug", "python", "javascript"]):
        return "deepseek-v3"  # $0.27/$1.10 per 1M tokens
    
    # Long context → appropriate model
    if total_chars > 50000:
        return "claude-opus-5"  # 200K context
    
    # Default → general purpose
    return "gpt-5.6"  # $1.40/$4.40 per 1M tokens

Savings: 40-70% depending on traffic mix. Most requests are simple enough for cheaper models.

2. Prompt compression

Shorter prompts cost less. Compress without losing meaning:

# BEFORE (verbose)
prompt = """
You are a customer service assistant for a technology company called Riven AI.
Riven AI provides an API gateway that gives developers access to over 75
different AI models through a single API key. The company was founded to
make AI more accessible and affordable. Your job is to help customers with
their questions about the API, pricing, models, and technical issues.
Always be polite, professional, and concise. If you don't know the answer,
say so and direct them to [email protected].
"""

# AFTER (compressed)
prompt = """
Role: Riven AI customer service assistant.
Context: Riven AI = API gateway, 75+ models, single key, affordable AI access.
Rules: Be polite, concise. Unknown? Direct to [email protected].
"""

Savings: 50-80% reduction in input tokens per request.

System prompt caching

Many LLM providers cache system prompts across requests. Structure your prompts to maximize cache hits:

# Static system prompt (cached) + dynamic user input (not cached)
SYSTEM = "You are a helpful assistant. Format responses as JSON."

# This system prompt is identical across requests → cached by provider
response = client.chat.completions.create(
    model="gpt-5.6",
    messages=[
        {"role": "system", "content": SYSTEM},  # Cached
        {"role": "user", "content": user_input}  # Not cached
    ]
)

3. Response caching

Cache identical or similar requests:

import hashlib
import json
import time

class ResponseCache:
    def __init__(self, ttl=3600):
        self.cache = {}
        self.ttl = ttl
    
    def _key(self, model, messages, **kwargs):
        content = json.dumps({"m": model, "msg": messages, **kwargs}, sort_keys=True)
        return hashlib.sha256(content.encode()).hexdigest()
    
    def get(self, model, messages, **kwargs):
        key = self._key(model, messages, **kwargs)
        if key in self.cache:
            entry = self.cache[key]
            if time.time() - entry["time"] < self.ttl:
                return entry["response"]
        return None
    
    def set(self, model, messages, response, **kwargs):
        key = self._key(model, messages, **kwargs)
        self.cache[key] = {"response": response, "time": time.time()}

# Usage
cache = ResponseCache(ttl=3600)  # 1 hour cache

def cached_chat(model, messages, **kwargs):
    cached = cache.get(model, messages, **kwargs)
    if cached:
        return cached
    
    response = client.chat.completions.create(model=model, messages=messages, **kwargs)
    cache.set(model, messages, response, **kwargs)
    return response

Savings: 20-40% for applications with repeated queries (FAQ, classification, etc.)

4. Batch processing

Process multiple requests in a single API call:

# Instead of individual calls (expensive)
for question in questions:
    response = client.chat.completions.create(
        model="gpt-5.6",
        messages=[{"role": "user", "content": question}],
        max_tokens=100
    )
    # 100 requests × $0.001 each = $0.10

# Batch them (cheaper)
batch_prompt = "Answer each question. Format as JSON array.\n\n"
for i, q in enumerate(questions):
    batch_prompt += f"Q{i+1}: {q}\n"

response = client.chat.completions.create(
    model="gpt-5.6",
    messages=[{"role": "user", "content": batch_prompt}],
    max_tokens=2000
)
# 1 request × $0.02 = $0.02 (80% savings)

Savings: 70-90% for bulk processing tasks.

5. Token optimization

Max tokens

Always set max_tokens to prevent runaway completions:

# BAD: No max_tokens (can generate 4K+ tokens)
response = client.chat.completions.create(
    model="gpt-5.6",
    messages=[{"role": "user", "content": "Summarize this article"}]
)

# GOOD: Reasonable limit
response = client.chat.completions.create(
    model="gpt-5.6",
    messages=[{"role": "user", "content": "Summarize this article"}],
    max_tokens=500  # 500 tokens ≈ 375 words, plenty for a summary
)

Temperature

Lower temperature = more deterministic = fewer retries:

# For factual tasks: temperature=0 (deterministic)
response = client.chat.completions.create(
    model="gpt-5.6",
    messages=messages,
    temperature=0,  # Same input → same output (cacheable)
    max_tokens=200
)

# For creative tasks: temperature=0.7 (some variation)
response = client.chat.completions.create(
    model="gpt-5.6",
    messages=messages,
    temperature=0.7,
    max_tokens=500
)

Stop sequences

Use stop sequences to end generation early:

response = client.chat.completions.create(
    model="gpt-5.6",
    messages=[{"role": "user", "content": "Extract the email address from this text"}],
    stop=["\n\n", "Note:", "Disclaimer:"],  # Stop at unnecessary content
    max_tokens=100
)

6. Context window management

Don't send unnecessary context. Trim conversation history:

def trim_conversation(messages, max_tokens=4000):
    """Keep only the most recent messages within token budget."""
    total = 0
    trimmed = []
    
    # Always keep system message
    if messages and messages[0]["role"] == "system":
        trimmed.append(messages[0])
        total += count_tokens(messages[0]["content"])
        messages = messages[1:]
    
    # Add messages from most recent backwards
    for msg in reversed(messages):
        msg_tokens = count_tokens(msg["content"])
        if total + msg_tokens > max_tokens:
            break
        trimmed.insert(1, msg)  # Insert after system message
        total += msg_tokens
    
    return trimmed

Savings: 30-50% for conversational applications where history grows over time.

7. Monitoring and alerts

You can't optimize what you don't measure:

from dataclasses import dataclass
from datetime import datetime, timedelta
import time

@dataclass
class CostTracker:
    daily_budget: float = 50.0  # $50/day default
    alert_threshold: float = 0.8  # Alert at 80%
    
    def __post_init__(self):
        self.spent = 0.0
        self.date = datetime.now().date()
        self.calls = []
    
    def record(self, model, input_tokens, output_tokens):
        # Reset daily
        if datetime.now().date() != self.date:
            self.date = datetime.now().date()
            self.spent = 0.0
            self.calls = []
        
        # Calculate cost
        rates = {
            "gpt-5.6": (1.4e-6, 4.4e-6),
            "glm-5.2": (0.5e-6, 1.5e-6),
            "deepseek-v3": (0.27e-6, 1.1e-6),
        }
        in_rate, out_rate = rates.get(model, (1e-6, 3e-6))
        cost = (input_tokens * in_rate) + (output_tokens * out_rate)
        
        self.spent += cost
        self.calls.append({"model": model, "cost": cost, "time": time.time()})
        
        # Alert if approaching budget
        if self.spent > self.daily_budget * self.alert_threshold:
            self._alert()
    
    def _alert(self):
        pct = (self.spent / self.daily_budget) * 100
        print(f"COST ALERT: ${self.spent:.2f}/${self.daily_budget:.2f} ({pct:.0f}%)")
    
    def summary(self):
        by_model = {}
        for call in self.calls:
            model = call["model"]
            by_model[model] = by_model.get(model, 0) + call["cost"]
        
        return {
            "total": self.spent,
            "budget": self.daily_budget,
            "remaining": self.daily_budget - self.spent,
            "by_model": by_model,
            "calls": len(self.calls),
        }

Putting it all together

Here's a cost-optimized LLM client that implements all strategies:

class OptimizedLLMClient:
    def __init__(self, api_key, daily_budget=50.0):
        self.client = OpenAI(api_key=api_key, base_url="https://api.rivenai.io/v1")
        self.cache = ResponseCache(ttl=3600)
        self.tracker = CostTracker(daily_budget=daily_budget)
    
    def chat(self, messages, max_tokens=500, temperature=0.3):
        # 1. Route to appropriate model
        model = route_model(messages)
        
        # 2. Trim context
        messages = trim_conversation(messages)
        
        # 3. Check cache
        cached = self.cache.get(model, messages, max_tokens=max_tokens)
        if cached:
            return cached
        
        # 4. Make request
        response = self.client.chat.completions.create(
            model=model,
            messages=messages,
            max_tokens=max_tokens,
            temperature=temperature
        )
        
        # 5. Track cost
        self.tracker.record(
            model=model,
            input_tokens=response.usage.prompt_tokens,
            output_tokens=response.usage.completion_tokens
        )
        
        # 6. Cache response
        self.cache.set(model, messages, response, max_tokens=max_tokens)
        
        return response
    
    def batch(self, prompts, max_tokens=100):
        """Process multiple prompts in batches."""
        batch_prompt = "\n\n".join(f"Q{i+1}: {p}" for i, p in enumerate(prompts))
        response = self.chat(
            messages=[{"role": "user", "content": batch_prompt}],
            max_tokens=max_tokens * len(prompts)
        )
        return response

Expected savings

| Strategy | Savings | Implementation effort | |---|---|---| | Model routing | 40-70% | Low | | Prompt compression | 50-80% (input tokens) | Low | | Response caching | 20-40% | Medium | | Batch processing | 70-90% (bulk tasks) | Medium | | Token limits | 10-30% | Low | | Context trimming | 30-50% (conversational) | Medium | | Monitoring | Prevents overruns | Low |

Combined, these strategies typically reduce LLM costs by 50-70% without noticeable quality degradation.

Start optimizing

All these strategies work with Riven's gateway. Sign up free at rivenai.io/sign-up and start building cost-optimized AI applications.