The agent engine

Every agent in Subspace runs on one durable engine: the Workflow DevKit (WDK) hosted in-process over a self-hosted Postgres World. This is the only engine. Because the World is the same Postgres that holds everything else, an agent’s checkpoint, its output nodes, and its approval card all commit in one transaction, so a paused run and its task card cannot split-brain. This chapter is the conceptual picture; for authoring and operating agents, start with the agents overview.

Workflows over a Postgres World

The engine drives registered workflows: any exported "use workflow" function is one entry in the registry. The default entry, agent-loop, is the familiar tool-calling loop (prompt, model call, tool calls, repeat). Other entries are arbitrary compiled topologies: follow-up-nudge is a durable timer workflow, and sub-agents are child runs spawned with agents.spawn.

Running over the Postgres World buys three things the hand-rolled loop could not offer cheaply:

Durable waits

A workflow can sleep('3 days') at zero suspended cost. The wait is a row and a timer in the World, not a process held open.

Sub-agents

A workflow spawns child runs and the parent/child edges are recorded, so an orchestration is a real run tree. See orchestration.

Idempotency claims

Steps are at-least-once, and side effects claim an idempotency key so a replay never sends twice (below).

The bundle is built, not run from source

Workflows are not executed from their TypeScript source. They compile to a standalone bundle that the engine imports as a library.

pnpm build:workflows     # rebuild the WDK agent bundle

Caution

After editing anything under the workflows directory you must rebuild the bundle, or the server keeps running the old code. The e2e suites rebuild automatically; a long-lived dev server does not, so restart it after a rebuild.

Boot ordering is strict and load-bearing: the World is set before any bundle is imported, and all bundles are imported before world.start. Getting that order right is what lets in-flight runs (including runs pinned to older or plugin-shipped workflow versions) resume cleanly after a restart. Installed plugins can ship their own compiled workflow bundles, and every version directory on disk is imported at boot so its in-flight runs can still resume.

Steps delegate to the runtime

Workflow bodies stay deliberately thin. The real business logic (model calls, tool execution, persistence, approval gating) lives in a shared runtime, and steps reach it through a Symbol handshake (Symbol.for('subspace.agents.runtime')). Keeping logic in the runtime rather than the workflow body matters for two reasons:

• Replay safety. Workflow bodies must stay replay-compatible across deploys, so they are kept minimal and all the real work happens in steps.
• In-flight enforcement. Because the seam lives in the runtime, changes to permissions or gating take effect for runs already in flight on older bundles, with no rebuild required.

ops.run_steps is the shared journal

Every meaningful thing a run does, a model call, a tool call, an output, an approval, appends a row to ops.run_steps. This one journal is the product surface: it powers the run tree UI (virtual subnode rows under the invoking bullet), the per-step logs, and the transcript. Run introspection, judges, and evals all read from it too. See run introspection.

The run’s checkpoint (its message history) lives in ops.agent_runs. Because the checkpoint, the run_steps row, the output nodes (ordinary kb commands), and, on pause, the approval row and its task card all commit in one transaction, a run’s durable state is always internally consistent.

Approvals and questions resume as hooks

When a run hits an outbound or irreversible tool, or when it asks the user a question, it does not busy-wait. It suspends on a hook and becomes a durable row:

1. Pause is a row

   The run pauses, an approvals row (or a question node) opens, and a task-queue card appears, all in one commit. The process is free to do other work or restart entirely.

2. A decision rides the outbox

   When you approve, reject, or answer, the decision commits and emits a run-resumed or question-answered outbox notice. The engine consumes that notice and resumes the workflow exactly where it suspended.

3. Boot sweeps catch the gap

   If the server crashes in the window between a decision and its delivery, a boot sweep re-delivers decided approvals on startup. A separate sweep fails any run left with no workflow id (the crash window inside run start), so a run is never silently stuck.

Tip

A run stuck in running almost always means a resume notice or a boot sweep did not fire, not that the model is thinking. The durable truth is in ops.outbox and the World’s run table, not in memory.

Failure handling

A step that fails is retried up to the engine’s attempt budget. If it still fails, the failure propagates to the workflow’s catch, which calls markRunFailed, flipping the run’s product status to failed. Nothing is left half-committed, because each step’s effects were themselves transactional.

Idempotency claims

Steps are at-least-once: a retry or a replay can run the same step body twice. For pure computation that is harmless, but a tool that sends mail or writes to a calendar must not fire twice. Those side-effecting tools take an effect claim in ops.effect_claims, keyed <tool>:<callId> (the durable tool-call id checkpointed with the assistant message). The first execution claims the key; a replay finds the claim already taken and returns the recorded result instead of re-running the effect. This closes the at-least-once window that a naive loop would leave open. See tools and approvals.

Related

Agents overviewDefinitions, the run tree, and how you actually build and run agents.

OrchestrationSub-agents, spawns, and durable waits as run trees.

Run introspectionObservations, judges, and evals read from the step journal.

Security & trustThe capability broker and approval gates that wrap every tool.