Knowledge ingestion

Knowledge ingestion is the pipeline that turns everything you capture, extension snapshots, scraped pages, newsletter digests, into knowledge base content someone can actually read. Raw material lands under ingestion/raw first, then a chain of scheduled agents compiles it into per-source summaries, concept pages, and index pages, entirely through audited kb.edit commands. A separate triage stage watches every new source and routes it to the specific project pages that opted in to hear about it.

From capture to raw page

Every source, whatever produced it, lands through the same tool: ingest.capture {url, title, markdown, screenshotFileId?, source?}. It files the extracted content as a page under ingestion/raw and emits a capture-received event. Nothing downstream ever sees live HTML, only the extracted markdown.

Extension captures

The Chrome extension popup extracts an article with Readability, converts it to markdown, and calls ingest.capture with a screenshot attached.

Scrapers

Agent-driven fetches that call ingest.capture directly for a URL, sharing the same raw-page-plus-stored-file lane as the extension.

Newsletters

The ingest-newsletter agent runs on a Mon/Thu schedule over the pages linked from your [[ingestion sources]] list and captures each one.

The compiler agents

Boot ensures four editable agent definitions live under agents/. Each is an ordinary agent definition page, so you read and tune its prompt, tools, and schedule like any other.

ingest-newsletter

Runs Mon/Thu over the [[ingestion sources]] list, calling ingest.capture for each linked source. A stale watermark on boot triggers one catch-up run, the same rule the scheduler applies to every missed fire.

wiki-compiler

Bound to the capture-received event (see triggers), so it runs immediately after any raw page lands. It maintains per-source summaries, interlinked concept pages, and index pages. Its tool allowlist is scoped to kb.edit, so every diff it makes, a new concept page, an added [[link]], an updated index, is one audited kb_commands entry, reviewable and revertible like a human edit.

wiki-linter

Runs weekly. It reads the corpus, optionally validates questionable facts with web.fetch, and files what it finds, contradictions, missing-data gaps, connection or new-article suggestions, as tasks through tasks.create. It never edits content silently: everything it produces lands in the task queue for you to act on.

wiki-qa

Invoke with /wiki-qa <question> from any page. It files its answer as ordinary child nodes under the invoking bullet, and can also use kb.edit to persist the answer to a destination page or as an OKF code-cell artifact.

Tip

Because the compiler chain writes only through kb.edit, every page it touches carries full version history: you can diff a wiki-compiler edit against the prior version exactly like a manual change.

Project-relevance ingest triage

Compiling a general wiki is only half the problem, the other half is getting a freshly landed source in front of the specific project it matters to, without flooding every page that vaguely relates. Any page opts in with:

metadata:
  triage:
    onIngest: true
    brief: "Segmentation project. Cares about clustering methods, warm-start tricks, and eval protocols for imbalanced cohorts."
    context: [] # optional page refs pinned as required reading for the judge

Each newly captured source runs through a bounded pipeline:

1. Injection screening

   The source passes prompt-injection screening before anything else touches it.

2. Embedding shortlist

   An embedding pass shortlists which opted-in pages the source is even plausibly related to. Most sources stop here, most pages never see most sources.

3. Cheap-model judge

   For each shortlisted candidate, a cheap-model agent reads the page’s brief plus the source summary, and may drill into bounded, read-only project context: the page itself, any pages named in context, its linked and backlink pages, its page-scoped open tasks, and its page inbox.

4. Verdict

   A useful verdict emits an ordinary ingest-source-relevant event carrying the source link, a one-line rationale, the target page, and a confidence score. Trusted event triage then decides whether that becomes a page-inbox item, a queue task, or a notification, by severity, same as any other event. An irrelevant source leaves no residue: no inbox item, no task, nothing.

Caution

The judge stage runs at untrusted severity: it can write a page-inbox item and a link, nothing more. It cannot create queue tasks, dispatch further work, or ask you a question, only the trusted triage funnel downstream escalates to those. Sources flagged by injection screening are excluded and separately marked, never silently dropped.

Autoresearch goal pages skip this generic path entirely: they carry a specialized hypothesis-relevance triage instead, so a goal never receives duplicate generic verdicts alongside its research-specific ones.

Keeping project context fresh: profile pages

Relevance triage is only as good as its picture of what a project actually does, and a hand-maintained description goes stale the moment the underlying code or hyperparameters change. A shared research/profile page (a model card, in effect) solves this for autoresearch goals: it holds a human-authored zone (intent, constraints) alongside a machine-derived “current recipe” zone that updates itself.

Every ingested run for a goal that references a profile through metadata.research.profile refreshes that recipe before anything else happens: a run on the main branch replaces the baseline outright; an off-main run clusters into a variant keyed by its config signature (the branch is provenance, not identity), and variants expire once they go runless or their hypothesis is refuted or parked. A baseline change material enough to matter emits research-profile-drift, which rides trusted triage into the profile’s page inbox. A goal that opts in automatically pins its profile in triage.context, so both the relevance judge above and heartbeat/literature triage always read the current recipe, not a stale description someone wrote three months ago.

Related

RAGThe compiled wiki this pipeline produces is exactly what /rag agents retrieve over.

Events & triageHow severity, the classifier, and routing decide inbox vs. queue vs. notification.

SchedulerCron rows, single-flight locking, and catch-up runs for ingest-newsletter and wiki-linter.

Autoresearch domain modelGoals, hypotheses, and the profile pages that ingest triage keeps current.