On-Prem AI Fleet

Live

Riven-owned GPUs running Riven-owned weights. Three models serve the sovereign lane, and they are tried before any cloud provider sees a token.

The on-prem fleet is the reason Riven can describe itself as sovereign. Inference runs on hardware Riven operates, using weights Riven holds, on the same Docker Swarm cluster as the rest of the platform. Riven Ultra Router tries this fleet first for every request; cloud is the fallback, not the default.

The three on-prem models#

Model	Role	Notes
Qwen 3.6-35B	Primary	Carries the majority of on-prem traffic — general reasoning, chat, tool use, and code
Nemotron-3-Puzzle-75B	Heavy reasoning	Larger capacity for hard multi-step problems; fewer concurrent slots
Bonsai-27B	Efficient / ternary	Ternary-weight model — very low memory footprint, high throughput for classification, triage, and short-form work

These are pools, not single processes. Each model runs across multiple replicas, and a replica going unhealthy removes it from rotation without failing the request — see the failover sequence in Riven Ultra Router.

Why ternary matters#

Bonsai-27B uses ternary weights, which trades a small amount of headroom for a large reduction in memory per replica. In practice that means many more concurrent slots on the same hardware, which is what makes it viable to run high-volume background work — mail triage, labelling, extraction — entirely on-prem instead of paying cloud rates for it.

Reaching the fleet#

You do not address a node directly. Request a tier alias and the router places it. The response tells you where it landed.

curl

curl https://api.rivenai.io/v1/chat/completions \
  -H "Authorization: Bearer $RIVEN_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model": "riven-best", "messages": [{"role": "user", "content": "Hi"}]}' \
  -i | grep x-riven-upstream

response header

x-riven-upstream: onprem-qwen36-a

An onprem- prefix on x-riven-upstream means the request never left Riven infrastructure. A provider name means it fell back — see Cloud Fallback for when and why that happens.

Capacity and scheduling#

Riven Fleet Scheduler allocates GPU slots between interactive and batch work, prioritising interactive.
Riven Spot Broker adds elastic capacity when on-prem is saturated and the workload tolerates it.
Riven Relay delivers weights to nodes, pinned by digest so a node cannot silently change models.
Utilisation, queue depth, and per-node health are visible in Riven Telemetry and summarised on Riven Status.

Data handling#

On-prem inference does not retain prompts or completions as training data. Requests are traced for operations and billing — token counts, latency, routing decision — not archived as content. The full posture is documented in Riven Trust.

Workloads that must never fall back to a third party can pin routing to the on-prem lane. That is an organisation-level policy on Enterprise and Group tiers — see Riven Business Suite.

Search docs