Backup & restore

Because every durable subsystem in Subspace (pages, mail, calendar, tasks, agent runs, search) shares one PostgreSQL cluster, backup is one story instead of four. Subspace bundles WAL-G for continuous archiving, replicates blobs out of band, snapshots the filesystem mirror nightly, and drills a real restore every month so “we have backups” is a tested claim rather than a hope.

Continuous WAL archiving

Setting a standard WAL-G repository variable (WALG_S3_PREFIX, WALG_FILE_PREFIX, and so on) turns on PostgreSQL’s archive_command with archive_timeout=60s. That timeout is the worst-case recovery point objective: even a quiet cluster ships a WAL segment at least once a minute, so the worst-case data loss window is roughly 90 seconds. A 02:10 nightly schedule runs wal-g backup-push to take a full base backup and retains 14 dailies. SUBSPACE_WALG_BIN overrides the bundled binary for local development.

Note

WAL-G ships as a single static Go binary, bundled per platform the same way the Postgres binaries are. There’s nothing to install separately.

Blob replication

Content-addressed blobs (blobs/ab/cd/<sha256>) are immutable once written, which makes them easy to replicate independently of the database. Setting SUBSPACE_BLOB_REPLICA_URL turns on the blob-replicator outbox consumer: every referenced blob uploads to the replica target within seconds of being written. Deletes don’t replicate immediately either, they become seven-day delayed tombstones, so a restore never ends up referencing a blob that was deleted out from under it before the delete had a chance to propagate. An optional SUBSPACE_BLOB_REPLICA_TOKEN supplies bearer auth for the replica endpoint.

Restic snapshots of the OKF dump

With RESTIC_REPOSITORY configured, the same nightly job snapshots okf/, the filesystem mirror of the knowledge graph, and retains 14 daily snapshots (SUBSPACE_RESTIC_BIN overrides the restic binary). The OKF dump is derived, it can always be rebuilt from the database, but it’s also the surface coding agents and external tools read against directly, so it gets its own lightweight, independently restorable snapshot line.

Retention

An append-forever store is a leak, not a durability guarantee. A nightly retention pass keeps every store bounded:

Store	Retention
outbox	At least 30 days, and never pruned past any consumer’s cursor. A consumer that lags past the 30-day floor is reset and rebuilt from base tables rather than left to block pruning forever.
events	12 months, then payloads drop.
kb_commands	30 days hot (undo, audit), then batched into compressed, replicated gzip JSONL archive blobs.
Agent run transcripts	90 days, then truncated to run_steps metadata.

The pass finishes with VACUUM (ANALYZE) on pruned tables, so WAL volume and base backup size stay bounded over the life of an instance rather than growing unbounded.

The monthly restore drill

Untested backups are not backups. A scheduled monthly job, and the equivalent on-demand command pnpm doctor -- --restore-drill, exercises the full restore path end to end:

1. Fetch the latest backup

   wal-g backup-fetch pulls the most recent base backup into a temporary directory.

2. Replay WAL

   Archived WAL segments replay forward from the base backup, exercising the same point-in-time recovery path a real disaster recovery would use.

3. Boot read-only

   A read-only Postgres cluster boots on a scratch port against the restored data directory, so the drill proves the data directory actually starts a working server, not just that files exist in a bucket.

4. Verify every blob

   Every row in files is checked against the blob replica: hash and download each referenced blob, so a restore can’t silently succeed while missing attachments.

5. OKF-parse every page

   Every live page is exported and parsed as OKF, catching corruption that a raw row-count check would miss.

Every attempt, success or failure, is recorded in ops.backup_runs, so drift in drill health is visible over time rather than only at the moment you actually need a restore.

Caution

Run the manual drill (pnpm doctor -- --restore-drill) after any change to backup configuration, target storage, or Postgres version. Don’t wait for the next scheduled monthly run to find out the new configuration doesn’t actually restore.

Manual restore

To restore an instance from scratch: fetch the latest base backup and replay WAL into a fresh $SUBSPACE_HOME/pg/ (wal-g backup-fetch plus PITR replay to the desired point in time), sync blobs/ from the blob replica target, and re-pair your devices against the restored instance. This is the same sequence the monthly drill runs automatically, just pointed at a real target directory instead of a scratch one.

pg_upgrade orchestration

Major PostgreSQL version bumps ship both majors’ binaries in the same release. On boot, the server compares the data directory’s PG_VERSION against the bundled major:

1. If they match, it starts normally (a minor version bump is just a binary swap against the same catalog).
2. If the data directory is on the older major, it forces a fresh base backup and a final WAL archive cycle first, so there’s a recovery point that predates the upgrade.
3. It runs pg_upgrade --link into a new data directory, which takes minutes even at tens of gigabytes because it hard-links rather than copies data files.
4. It starts the upgraded cluster, health-checks it, and completes one full archive cycle before deleting the old data directory.
5. If pg_upgrade fails, it falls back to a logical pg_dumpall | psql restore. Because both majors’ binaries are present, this fallback is always available, not a best-effort escape hatch.

This whole sequence is exposed directly as subspace doctor --pg-upgrade, and it runs in CI against a seeded previous-major cluster before any release that bumps the bundled Postgres major ships.

Self-hostingThe home directory layout, the single-process model, and moving a home directory to an always-on box.

Environment variablesEvery backup, retention, and Postgres-upgrade related variable.