Contracts & conformance

Every MLOps provider, whether it wraps MLflow, Weights & Biases, Dagster, or nothing more than a directory of JSON files, speaks the same narrow contract. @subspace/contract-mlops defines that contract once; a conformance kit proves any given provider actually implements it. This page covers both, plus the filesystem tracker that ships as the credential-free reference implementation.

The tracker and launcher contracts

A contract is a mandatory core plus optional capability groups a provider declares or doesn’t; everything else stays in the provider’s own native tool namespace rather than being forced into the shared shape. This keeps the waist narrow: a provider that only ever adds capabilities it actually has, and consumers (dashboards, the metric chart viewer, Autoresearch) can check capabilities before calling a group’s tool.

mlops.tracker core tool slots: runs.list, runs.get, metrics.final. Capability groups: metrics.history (tool slot metrics.series), artifacts (artifacts.list, artifacts.fetch), sweeps (sweeps.get), registry (models.list, models.get), alerts (alerts.subscribe).

mlops.launcher core tool slots: submit, status, cancel. Capability groups: capacity (capacity), sweeps (submitSweep), logs (logs).

The normalized shapes both contracts pass through:

Shape	Carries
runSummary	provider, id, status, url, name, startedAt, finishedAt (list-sized)
runDetail	runSummary plus gitRef, params, final metrics, groupId, capped native payload
metricPoint	key, step, value, optional ts (one sampled point of one series)
launchSpec	gitRef, config, hypothesis, provider-keyed providerOptions pass-through
launchStatus	jobId, status, optional trackerRunId, detail
capacity	integer slots, optional human-readable detail
artifactInfo	path, isDir, optional size
registeredModel	provider, name, description, versions[]
runPageMeta	the metadata.mlops shape on an mlops/run page: provider, id, status, url, timestamps

launchSpec.providerOptions is how a provider-specific knob (a W&B queue name, a Dagster partition key) rides through a generic submit call without polluting the shared schema: only the bound provider ever reads its own key.

Note

Run status is one closed enum everywhere in the contract: queued, running, finished, failed, killed. A provider’s native status strings get mapped onto this set at the tracker boundary, so every dashboard, chart, and workflow condition can branch on it without knowing which provider produced the run.

The ingest loop shape

Every tracker provider implements the same four-stage loop, whether it is polling a REST API or re-reading a directory:

1. Watermark poll

   List runs that changed since a per-binding monotonic watermark, stored in ops.plugin_state. Active (non-terminal) runs are deliberately relisted on every sweep so in-flight metrics and status changes converge.

2. Materialize

   Parse each listing into runSummary/runDetail and create or update the corresponding mlops/run page. Page identity is deterministic per (source page, run id), so repeated sweeps never duplicate a page; changes are content-diffed so an unchanged run is a no-op.

3. Group

   Fold sweep, experiment, or job membership (groupId) into provider-scoped pages alongside the run pages, so an MLflow experiment or a W&B sweep is itself a page you can open.

4. Emit

   Each created or updated run page fires ('mlops', 'mlops-run-ingested', pageId) on the outbox. Consumers, the mlops-core dashboard’s live update, or a subscribing Autoresearch workflow, react to that single event type regardless of which provider produced the run.

mlops.ingest {page?} runs this sweep on demand (read-only reach grant, no approval needed): omit page to sweep every binding, or pass one to scope to a single linked-directory binding. The mlops.ingest-runs registered workflow wraps the same call so you can put it on a cron schedule: per binding, or an Autoresearch loop can start it programmatically.

The conformance kit

@subspace/contract-mlops/conformance is the acceptance gate for any provider plugin, not just documentation of intent. It ships as reusable test modules a provider’s own test suite imports and runs against its live (or fixture-backed) implementation:

• A tracker-core module verifies stable relists (the same run doesn’t reappear as a duplicate), watermark monotonicity, and detail/final parity between runs.list and runs.get.
• A launcher-core module verifies the submit → status → cancel lifecycle behaves as the contract expects.
• One capability module per declared group: metrics.history, artifacts, and registry each get their own behavioral checks, run only for providers that declare that group.

Installation itself is a gate too: validateProviderDeclaration(contract, provides) runs at install time and rejects a manifest that leaves a core tool slot unmapped, declares a capability the contract doesn’t recognize, declares a capability without mapping that group’s tool slots, or is missing/invalid its configSchema JSON. A provider plugin cannot activate with a broken contract mapping.

Declaring a new provider

A provider’s subspace-plugin.json maps contract slots to its own tool names under provides, and lists which capability groups it advertises:

{
  "name": "mlflow",
  "dependencies": { "mlops-core": ">=0.2.0" },
  "provides": {
    "mlops.tracker": {
      "tools": {
        "runs.list": "fn.mlflow.runs.list",
        "runs.get": "fn.mlflow.runs.get",
        "metrics.final": "fn.mlflow.metrics.final",
        "metrics.series": "fn.mlflow.metrics.series",
        "artifacts.list": "fn.mlflow.artifacts.list",
        "artifacts.fetch": "fn.mlflow.artifacts.fetch",
        "models.list": "fn.mlflow.models.list",
        "models.get": "fn.mlflow.models.get"
      },
      "capabilities": ["metrics.history", "artifacts", "registry"],
      "configSchema": "schemas/mlflow-binding.schema.json"
    }
  }
}

Only the tool slots for the core plus whichever capabilities you list need mapping; a provider that skips sweeps and alerts (as MLflow does) simply omits them, and downstream consumers check capabilities before ever calling sweeps.get or alerts.subscribe against that provider.

The filesystem tracker

The filesystem tracker is the credential-free reference read side, and it ships inside mlops-core rather than as a separate plugin. Every two minutes, and once at boot, the server sweeps every page carrying metadata.fs.path. A runs/<id>/ directory beneath that path materializes an mlops/run page:

• metrics.json holds final scalar values; its presence marks the run finished.
• metrics.jsonl is append-per-step; the tracker treats the run as running and the last line wins for current values, while the full series feeds mlops.metrics.history.

The page’s metadata.mlops parses against runPageMeta: provider local, numeric scalars in metrics, a sibling params.json file’s contents in params, the raw payload in native, and the source directory recorded under metadata.mlops.local. Because there is no API, no token, and no network call involved, the filesystem tracker is the fastest way to see the whole ingest → page → dashboard → chart loop end to end, and it is what the local-runs launcher pairs with for a fully local workflow.

See trackers for the platform providers (MLflow, W&B, Dagster) that build on this same contract, and launchers for the submit side.