Evals

Subspace grades its own LLM seams two ways. The recorded-run harness pins the prompt-to-behavior contract in CI by replaying recorded model responses through the real machinery and asserting on the resulting database. The eval sweep compares two versions of one agent definition head to head, graded by judges, under a hard budget ceiling. Both grade prompts, never the open network.

The recorded-run harness

The harness covers the three LLM seams where a wrong model output would silently corrupt state:

Seam	What it grades
Event triage	Severity routing through the classifier and triage.
Mail-rule labeling	Label decisions through the rule classifier and mail rules.
CRM enrichment	A full agent run through the real runner and toolkit, enriching CRM.

A case is {name, input, recorded, expected}, checked into a per-seam fixture JSON and validated on load. The recorded field is the model’s canned response for that input.

Note

recorded: null means the model must never be consulted for this case (a rule-path triage decision, the already-sent loop guard). The replay stub rejects if it is reached, so a machinery regression that starts calling the model surfaces as a test failure rather than a quiet cost.

Replay mode (CI, default)

Replay runs as part of pnpm test. For each case, the recorded response is injected at the seam (a classifier stub, or a scripted AgentModel transcript) while the real producer, routing, labeling, and run machinery executes against a mock-world database clone. The expected assertions are then evaluated on the resulting DB state, so the harness pins not just the prompt but everything around it.

Assertions are per-seam and every field is optional:

triage

severity, route, decidedBy (rule / llm / fallback), provisional, taskCreated, notified.

mail-rules

matched, labels, notLabels, skippedInbox, stamped.

enrichment

runStatus, pageHasBullet / pageLacksBullet ({page, contains}), memoryItem ({kind, contains}), toolOutputContains ({tool, contains}).

A failing case prints its name and a per-assertion expected-versus-actual, so a regression points straight at the broken behavior.

Live mode

Live mode runs the same cases and assertions against the real model:

EVALS_LIVE=1 ANTHROPIC_API_KEY=… corepack pnpm --filter @subspace/server evals:live

It grades the production providers (SUBSPACE_MODEL, default claude-sonnet-5), prints a per-case scorecard, and exits 1 on any failure. Tools still run under PROVIDERS=fake (so web.fetch serves only registered fixtures): live evals grade prompts, never the open network. Cases marked live: false are synthetic failures (a classifier error, a hallucinated level, wrong-shape tool args) and are replay-only, skipped live.

Tip

Re-record a drifted fixture with EVALS_RECORD=1, which writes each live response back into its fixture. Per-case record: true records just that case; record: false protects synthetic recordings from a blanket re-record. Add a case by appending to the seam’s fixture JSON, then pnpm test replays it.

The eval sweep

The eval-sweep workflow is a judge-scored A/B comparison of two immutable versions of one agents/ definition. It is registered and manual-only. Invoke the seeded agents/eval-sweep definition through the ordinary agent command, or call agents.evalSweep {agentPageId, evalSet, armA, armB, judges}.

1. Cases from an evals/ page

   Cases come from a versioned page under evals/: each top-level node is one case, using its text as the default prompt or props.eval.prompt for an explicit or multiline prompt.

2. Plan and budget

   The plan step resolves both historical definition snapshots, captures the exact eval-set version, and resolves the selected judges/ pages. It rejects the entire sweep before its first child if cases × 2 arms × judges exceeds EVAL_SWEEP_MAX_GRADES (default 24). Paid execution also requires EVALS_LIVE=1, the same explicit posture as live evals. The eight-child fan-out cap bounds one sweep to four cases.

3. Run both arms

   It spawns each arm from its stored definition hash, gathers children in durable batches of at most six, disables their global auto-judge path, and explicitly grades only the budgeted judges (irrespective of ordinary judge gates).

4. Report

   The final page under evals/ is advisory: per-judge pass rate for A and B, verdict counts, case and child-run links, the two definition hashes, the eval-set snapshot, and the budget provenance. It never promotes or edits the live definition.

Caution

A sweep spends real model budget on every arm and judge, which is why the budget ceiling is checked before the first child and paid execution requires EVALS_LIVE=1. Choose your judge set and eval-set size with the cases × 2 × judges product in mind.

Related

Run introspectionThe judges an eval sweep scores with, and the observation layer.

Agent definitionsThe definition versions a sweep compares, pinned by content hash.