Backup & restore

Because every durable subsystem in Subspace (pages, mail, calendar, tasks, agent runs, search) shares one PostgreSQL cluster, backup is one story instead of four. Subspace bundles WAL-G for continuous archiving, replicates blobs out of band, snapshots the filesystem mirror nightly, and drills a real restore every month. Hosted tenant cells use an R2 prefix per tenant for both WAL-G and blobs; credentials and prefixes are never shared between cells.

Tenant data keys and KMS escrow

ops.secrets values are encrypted under a per-tenant data-encryption key (DEK). The plaintext DEK stays inside the tenant boundary. Local/self-hosted mode keeps the existing 0600 $SUBSPACE_HOME/secret.key file; it is deliberately not an environment variable, so run inspection and worker environment capture cannot disclose it.

Hosted mode sends only a KMS-wrapped envelope (dataKeyId, wrapping-key id, algorithm, and ciphertext) to the control plane. The envelope is durably escrowed before rotation activates the new generation; the control plane retains active and previous envelopes. Rotation then writes an on-volume keyring containing the new and previous generations before re-encrypting rows. Ciphertexts identify their key generation, while legacy v1 rows try retained generations, so an interrupted rotation is dual-decrypt safe. Lost-key recovery asks KMS to unwrap the escrow envelope inside the tenant cell and atomically recreates secret.key; neither the control-plane database nor ordinary environment variables ever contain the plaintext key.

The production adapter uses AWS KMS symmetric encryption (Encrypt/Decrypt with SYMMETRIC_DEFAULT). It binds every operation to the encryption context subspace:tenant-id=<tenant> and subspace:purpose=subspace-tenant-data-key, so moving an envelope to another tenant does not make it decryptable. Cells obtain AWS credentials from the SDK’s workload-identity chain; do not put a DEK in an environment variable. The cell IAM role should be limited to kms:Encrypt and kms:Decrypt on the configured key, with an IAM condition requiring those two encryption-context values.

The authenticated cell protocol is POST /v1/key-envelopes, followed by POST /v1/key-envelopes/activate, plus tenant-scoped lookup and recovery reads under GET /v1/key-envelopes/*. Escrow is idempotent on (tenantId, dataKeyId). Activation atomically moves the previous active generation to retained; recovery returns the active envelope first and every retained generation newest-first. A mismatched tenant is forbidden, a reused key id with different envelope bytes conflicts, and failure to escrow or activate leaves the cell’s local active key and database ciphertext unchanged. Bootstrap follows the same ordering: both remote steps succeed before secret.key is created.

The production image includes a separate key-escrow-control-plane executable. It connects only to external PostgreSQL, migrates the dedicated control_plane schema, and stores the wrapped envelope, tenantId, generation/status, credential digests, and bounded audit metadata. Per-tenant row locks make generation allocation/idempotency and activation monotonic under concurrent requests; an operation and its audit record commit in one transaction. It never writes to a tenant’s embedded ops.* tables. The in-memory adapter remains a protocol test double, not a production persistence choice. See the environment reference for the deployment contract and credential rotation/revocation procedure.

Continuous WAL archiving

Setting a standard WAL-G repository variable (WALG_S3_PREFIX, WALG_FILE_PREFIX, and so on) turns on PostgreSQL’s archive_command with archive_timeout=60s. That timeout sets the PostgreSQL-only recovery point objective: even a quiet cluster ships a WAL segment at least once a minute. A 02:10 nightly schedule runs wal-g backup-push to take a full base backup and retains 14 dailies. A tenant is only declared recoverable through a later composed recovery point, described below. SUBSPACE_WALG_BIN overrides the bundled binary for local development.

Note

WAL-G ships as a single static Go binary, bundled per platform the same way the Postgres binaries are. There’s nothing to install separately.

Blob replication

Content-addressed blobs (blobs/ab/cd/<sha256>) are immutable once written, which makes them easy to replicate independently of the database. Setting SUBSPACE_BLOB_REPLICA_URL turns on the blob-replicator outbox consumer: every referenced blob uploads to the replica target within seconds of being written. Deletes become seven-day delayed tombstones, so a restore never ends up referencing a blob that was deleted out from under it. An optional SUBSPACE_BLOB_REPLICA_TOKEN supplies bearer auth for the replica endpoint.

Composed recovery point

WAL and blob RPOs do not compose by themselves. After a base backup, the nightly job drains blob replication, waits for existing blob-reference writers, and briefly takes SHARE locks on every table used by the blob keep-set. It checks that every hash is in the uploaded ledger and, when the repository supports HEAD, exists remotely. Only then does it record ops.recovery_points with the current WAL LSN, blob count, and digest.

New blob-reference commits wait behind the locks and therefore land after that LSN. A PITR restore targeted at the recorded LSN cannot expose a database row whose body was absent from the verified replica snapshot. If any object is pending or missing, no recovery point is issued and the backup attempt is marked failed in ops.backup_runs.

Restic snapshots of the OKF dump

With RESTIC_REPOSITORY configured, the same nightly job snapshots okf/, the filesystem mirror of the knowledge graph, and retains 14 daily snapshots (SUBSPACE_RESTIC_BIN overrides the restic binary). The OKF dump is derived, it can always be rebuilt from the database, but it’s also the surface coding agents and external tools read against directly, so it gets its own lightweight, independently restorable snapshot line.

Retention

A nightly retention pass keeps every store bounded:

Store	Retention
outbox	At least 30 days, and never pruned past any consumer’s cursor. A consumer that lags past the 30-day floor is reset and rebuilt from base tables rather than left to block pruning forever.
events	12 months, then payloads drop.
kb_commands	30 days hot (undo, audit), then batched into compressed, replicated gzip JSONL archive blobs.
Agent run transcripts	90 days, then truncated to run_steps metadata.

The pass finishes with VACUUM (ANALYZE) on pruned tables, so WAL volume and base backup size stay bounded over the life of an instance.

The monthly restore drill

A scheduled monthly job, and the equivalent on-demand command pnpm doctor -- --restore-drill --recovery-lsn 0/ABCDEF, exercises the full restore path end to end. The scheduled drill selects the latest composed point in ops.recovery_points; the manual drill requires its LSN explicitly. Both fail before fetching WAL-G if no recovery point is available, so a drill never silently promotes unbounded LATEST WAL.

1. Fetch the latest backup

   wal-g backup-fetch pulls the most recent base backup into a temporary directory.

2. Replay WAL

   Archived WAL segments replay forward from the base backup, exercising the same point-in-time recovery path a real disaster recovery would use.

3. Boot read-only

   A read-only Postgres cluster boots on a scratch port against the restored data directory, proving the backup actually starts a working server, not just that files exist in a bucket.

4. Verify every blob

   Every row in files is checked against the blob replica: hash and download each referenced blob, so a restore can’t silently succeed while missing attachments.

5. OKF-parse every page

   Every live page is exported and parsed as OKF, catching corruption that a raw row-count check would miss.

6. Prove WDK checkpoints are resumable

   Every live ops.agent_runs row must still join to its durable workflow.workflow_runs checkpoint. The drill records the live-run count and a sample run id, proving an in-flight run survived the selected WAL point rather than silently degrading into an orphaned product row.

Every scheduled attempt, success or failure, is recorded in ops.backup_runs, including the recovery point and resumable-run proof, so drift in drill health is visible over time, not only at the moment you actually need a restore.

Caution

Run the manual drill (pnpm doctor -- --restore-drill --recovery-lsn 0/ABCDEF) after any change to backup configuration, target storage, or Postgres version. Select the LSN from a composed ops.recovery_points row whose blob barrier completed. Don’t wait for the next scheduled monthly run to find out the new configuration doesn’t actually restore.

Verified export and import

A recovery export is a directory package, never a live PGDATA tarball. While blob reference tables remain locked, Subspace exports a repeatable-read PostgreSQL snapshot with pg_dump --snapshot, copies exactly the referenced content-addressed blobs, and writes a machine-readable manifest.json. The manifest pins:

• the database dump checksum and format;
• every blob path, hash, and size;
• installed plugin versions and manifest hashes;
• agent definition/skill/MCP/fn artifact hashes observed by runs;
• the latest composed recovery point; and
• the opaque KMS-wrapped tenant DEK (never secret.key).

Creation uses a pending directory, verifies every checksum, then atomically renames it. Import verifies the complete package before writing a blob or invoking pg_restore, and rehashes every blob through the normal BlobStore. With the app connected to the source database, an operator/control-plane job can run:

pnpm doctor -- --export-package /safe/tenant-export --wrapped-key /run/sealed/dek-envelope.json
pnpm doctor -- --verify-package /safe/tenant-export

On a stopped, fresh self-hosted PostgreSQL home, start only the embedded database and run pnpm doctor -- --import-package /safe/tenant-export. Finally, unwrap the manifest’s DEK through the named KMS key inside the destination and start Subspace. This same path is the offboarding feature, hosted migration path, and deterministic DR fixture.

Manual PITR restore

To restore an instance from scratch: fetch the base backup associated with the last good ops.backup_runs record and replay WAL into a fresh $SUBSPACE_HOME/pg/, stopping at its recoveryPoint.walLsn; sync blobs/ from the corresponding tenant replica prefix; recover the tenant DEK from its KMS envelope; then re-pair devices. The drill accepts the same target explicitly as pnpm doctor -- --restore-drill --recovery-lsn 0/ABCDEF.

Do not blindly promote LATEST after a partial outage. WAL newer than the last composed point may legitimately reference a blob that had not reached R2 when the cell failed.

Volume sizing and blob retention

Start tenant volumes at 5–10 GB and alert at 70%/85% utilization. The built-in sizing report splits PostgreSQL, blobs, plugins, OKF, and other bytes; attachments and captured artifacts normally dominate. Railway volumes grow but never shrink, so expansion is the only capacity action—VACUUM reuses PostgreSQL space but does not return provisioned volume capacity. Run pnpm doctor -- --volume-usage to emit the machine-readable byte breakdown consumed by capacity alerts.

Blob retention is reference based, not an arbitrary age cutoff: an attachment remains while any live file, mail body/header, code output, or command archive refers to its hash. After the final reference is deleted, weekly GC removes the local orphan (after its in-flight grace window) and queues a seven-day remote tombstone. This prevents attachment- heavy tenants growing forever without making a valid historical reference unrestorable. The database/audit retention windows above remain independent.

pg_upgrade orchestration

Major PostgreSQL version bumps ship both majors’ binaries in the same release. On boot, the server compares the data directory’s PG_VERSION against the bundled major:

1. If they match, it starts normally (a minor version bump is just a binary swap against the same catalog).
2. If the data directory is on the older major, it forces a fresh base backup and a final WAL archive cycle first, so there’s a recovery point that predates the upgrade.
3. It runs pg_upgrade --link into a new data directory, which takes minutes even at tens of gigabytes because it hard-links rather than copies data files.
4. It starts the upgraded cluster, health-checks it, and completes one full archive cycle before deleting the old data directory.
5. If pg_upgrade fails, it falls back to a logical pg_dumpall | psql restore. Because both majors’ binaries are present, this fallback is always available.

This whole sequence is exposed directly as subspace doctor --pg-upgrade, and it runs in CI against a seeded previous-major cluster before any release that bumps the bundled Postgres major ships.

Self-hostingThe home directory layout, the single-process model, and moving a home directory to an always-on box.

Environment variablesEvery backup, retention, and Postgres-upgrade related variable.