Skip to content
Subspace

Run introspection

Every run leaves a durable transcript, and Subspace can read it back. Observations summarize each run through five fixed lenses; judges grade runs against rubrics you author; a weekly pattern sweep clusters observations into named patterns on an agents/health page; and eval sweeps compare two definition versions head to head. All of it is opt-in telemetry, cascade-deleted with the run and age-pruned, so introspection never becomes silent cost or clutter.

Observations

Section titled “Observations”

When a run finishes, the run-introspection consumer (subscribed to the (agent-runs, run-finished) outbox notice) makes one cheap grader call and writes 0 to 5 rows into ops.run_observations, keyed (run_id, lens) over a fixed five-lens vocabulary:

Lens What it captures
user-intent What the user was actually trying to accomplish.
task-resolution Whether the run resolved the task, and how completely.
user-sentiment Signals of user satisfaction or frustration in the transcript.
agent-struggle Where the agent thrashed, looped, or hedged.
environment-issue External failures: a dead tool, a missing secret, a flaky server.

These observe agents running in Subspace (the run transcript), never any model or training work. Trivial runs (one assistant message and no tool calls) are skipped, and child runs observe individually. Observations render as green lens chips on the run tree’s anchor row (the lens on the chip, the note in its tooltip). They are telemetry, not knowledge: ops rows, cascade-deleted with the run and pruned at 90 days on a daily boot sweep.

Judges

Section titled “Judges”

A judge is an LLM-as-judge rubric page in the judges/ namespace (recognized by namespace, like agents/ and skills/). The page body is the rubric; props.judge gates which runs it grades:

---
judge:
  on:
    agent: "research*"          # glob over definition names; absent = all
    usedTools: [web.search]     # only runs that called ALL of these
  model: claude-haiku-4-5       # default SUBSPACE_GRADER_MODEL
---


# grounded-citations


Pass if every factual claim links to a source page it actually supports.
Fail if any claim is uncited or contradicted by its cited page.

Under RUN_INTROSPECTION=full, grading rides the same consumer as observations: per completed non-trivial run, each matching judge gets one grader call and writes one verdict from a fixed vocabulary into ops.run_judgements (unique per run and judge, redelivery-idempotent):

  • pass (green badge)
  • fail (red badge)
  • not_applicable (semantic abstention: the run was out of the rubric’s scope)
  • error (amber badge: the grader itself threw or replied off-vocabulary)

Verdict badges appear on the run tree’s anchor row (judge name and reasoning in the badge and its tooltip). A per-judge pass-rate trend (pass / (pass + fail), overall and last 7 days) is rewritten into the agents/health page after each graded batch.

Pattern sweeps

Section titled “Pattern sweeps”

The pattern-sweep registered workflow runs on a schedule (the mock-world seed configures agents/pattern-sweep weekly on Monday at 06:00) and clusters the observations recorded since the last sweep into named patterns:

  1. Cluster and merge

    New observations merge into existing clusters or form new ones, keeping a count and up to three example run ids per cluster.

  2. Retire and drop

    A cluster idle for 30 days is retired; a retired cluster is dropped 90 days after it was last seen.

  3. Rewrite the health page

    The sweep rewrites the agents/health page: a patterns section plus the judge trends. Patterns are knowledge, not telemetry: a normal, linkable, versioned kb page. Cluster state lives in the page’s metadata.patterns.

The whole sweep is a single ungated tool call (ops.patternSweep, auto-safe because it is telemetry-derived), single-flighted by an advisory transaction lock and watermarked by the maximum observation timestamp it has seen, so overlapping schedules never double-count.

Eval sweeps

Section titled “Eval sweeps”

The eval-sweep workflow judge-scores an A/B comparison of two immutable versions of one definition against a versioned eval set, under a hard budget ceiling. It shares the judge and observation machinery described here and is covered in full on the evals page.

The trust posture

Section titled “The trust posture”

Introspection assumes a run is untrusted input. That posture is enforced structurally by the plane split and the capability broker.

--role=all: one process, one memory space. This is hygiene plus a seam, not a security boundary. Credentials still flow only through the broker, and gated tools still pause for approval.

Sandboxed custom-function tools run in QuickJS with a curated host, a third plane below the worker. See security and trust for the full model.

Section titled “Related”
EvalsThe recorded-run harness and the judge-scored eval sweep.
Tools & approvalsThe capability broker and the approval gates introspection assumes.