Files & blobs

Every file Subspace stores, an upload, an email attachment, a code cell’s image output, a mirrored Google Doc, lands in the same content-addressed blob store. A file element in a page is a thin pointer into that store plus a viewer, and a file can even own a page, turning a PDF into an annotatable outline.

The blob store

Blobs live on disk as blobs/ab/cd/<sha256>, sharded two levels deep by the first four hex characters of the content hash so no directory ever holds an unreasonable number of files. Writes are crash-safe by construction: a new blob is written to a temp file, fsynced, renamed into place, and only then does the referencing database row (a kb.files row, a mail attachment row, a code-cell output) commit. If the process crashes between the rename and the row commit, the result is an orphaned blob, never a row pointing at missing bytes.

Orphans are swept by a weekly blob GC: a blob-gc row in ops.schedules, seeded idempotently on boot for Sunday 03:15, fires a { systemJob: 'blob-gc' } target through the ordinary scheduler, the same single-flight, watermark-tracked, scheduled-task machinery any other scheduled job uses. The sweep diffs the blob directory against every table that can reference a blob, kb.files, mail body blobs, headers.attachments references, and code-cell image outputs, and deletes orphans older than a 24-hour grace window, so a blob mid-upload is never mistaken for garbage.

Uploading a file

attach file in the page header uploads into the blob store and drops a file node in place:

images preview inline;
PDFs render through pdf.js with page navigation;
everything else gets a plain download row.

A file can own a page

The frame action open as page ↗ mints a page owned by the file (metadata.file.fileId points back at it): the viewer sits on top, an ordinary outline sits below. This is the annotation surface, you can write bullets alongside a PDF the same way you’d write bullets anywhere else, and those bullets are ordinary nodes that link, tag, and search like any other page content.

Text extraction feeds search

On upload, Subspace extracts text where it can: PDFs through pdf-parse, .docx through mammoth, and plain text/* files verbatim. Extracted text feeds search.file_text, so file contents, not just filenames, are findable through search’s ⌘K as file hits.

Connected provider mirrors

Google Drive, Dropbox, and SharePoint (via Microsoft Graph) connect as read-only mirrors behind one shared connector contract: list / download / upload / changes.

Change tracking is provider-native: opaque Drive page tokens, Dropbox cursors (with long-poll), and Graph delta links persist in ops.provider_sync_state, and a cursor only advances once every file it named has actually landed, so a crash mid-sync re-processes rather than skips.
Materialization: mirrored blobs and their kb.files rows appear as file-owned pages under connected-files/. Each carries provider_meta (the provider’s file id, revision, and mirror timestamp). Google Docs export as Markdown and are flagged lossy if you ever write them back. An upstream deletion marks the mirror but keeps the local blob and page history intact.
Read-only by default, with an explicit write-back. Provider pages reject ordinary edits in both the outliner and the command path. The replace upstream contents action sends the mirrored revision as a CAS precondition (If-Match for Drive/Graph, Dropbox’s update rev). If the upstream revision has moved since the mirror was taken, Subspace refreshes the mirror, preserves your edited candidate as a second file rather than dropping it, and files a provider-merge card in Needs confirmation for a human to reconcile.
Credentials: Drive reuses the Google grant already established for mail and calendar (new connections request the Drive scope; connections made before Drive support was added need to reconnect once). Dropbox reads DROPBOX_ACCESS_TOKEN; SharePoint reads MICROSOFT_GRAPH_ACCESS_TOKEN and MICROSOFT_GRAPH_DRIVE_ID. On boot these import into encrypted ops.secrets, and every connector call reacquires its access token through the capability broker, so a rotated token doesn’t require a restart.
Status and manual sync: GET /provider-sync reports authenticated status for every connected provider; POST /provider-sync/:provider[?account=…] lets an admin trigger one connector’s sync on demand. Tests run against a deterministic Drive fixture by setting PROVIDERS=fake together with SUBSPACE_PROVIDER_SYNC_FAKE=1.