Riven Evals

Measure whether a change actually helped. Build scored test suites over prompts, models, and agents, then run them on every change instead of guessing.

Service

surfaces_evals

Prompt and model changes are easy to make and hard to judge. Evals gives that judgment a number: a suite of cases, a grader per case, and a score you can compare across runs.

Anatomy of a suite#

• Cases — an input plus what a good answer looks like. Import from production traffic or write by hand.
• Graders — exact match, structural checks against a schema, or a model-graded rubric for open-ended answers.
• Targets — the prompt, model, or agent under test. One suite can run against several targets to compare them.
• Runs — a scored execution of the suite, retained so two runs can be diffed case by case.

What it is for#

• Regression gates — block a prompt change that drops a score below its floor.
• Model migration — quantify what moving from one model to another does to your workload before you move it.
• Router validation — confirm that an alias change in Ultra Router did not degrade quality.
• Agent reliability — score whether a Computer task completes, not just whether it returns.

Relationship to Studio#

Studio is for exploring one prompt interactively; Evals is for proving a change across many cases repeatably. The usual path is Studio to find a candidate, Evals to justify shipping it.