Riven Fleet Scheduler

Places inference work on GPUs: which model is loaded where, how many replicas, and which request goes to which worker.

The on-prem fleet is finite hardware serving many models. Fleet Scheduler is what makes that tractable: it decides what is resident on which GPU, and which worker each request lands on.

What it decides#

• Placement — which model weights occupy which GPUs, given VRAM limits and observed demand.
• Replica count — how many workers serve a model, scaled against queue depth rather than a fixed guess.
• Request assignment — which worker takes a request, favouring workers whose KV cache already holds the conversation prefix.
• Batching — grouping concurrent requests to raise throughput without pushing time-to-first-token past its budget.
• Admission — queue, shed, or spill to cloud when demand exceeds capacity, instead of accepting work that will time out.

Why prefix affinity matters#

Long conversations re-send their history on every turn. Routing a follow-up to the worker that already holds that prefix in its KV cache avoids recomputing it — which is a large latency and throughput win on exactly the traffic that is most common. The scheduler therefore treats conversation continuity as a placement input, not an afterthought.

Interaction with the router#

Division of labour: Ultra Router decides *which model and which pool* should serve a request; Fleet Scheduler decides *which worker inside that pool* takes it, and reports capacity back so the router knows when a pool is full. When capacity is genuinely exhausted, the Spot Broker is what widens it.

Degradation#

Under saturation the scheduler prefers a slower answer to a failed one, then failover to a failed one. Requests queue briefly, then spill to cloud fallback. 503 UPSTREAMS_EXHAUSTED means both paths were unavailable.