Quality & release plane

The quality plane (M13) is how Jentrix judges the agents it runs. The control plane (M1–M11) decides what agents may do; the execution plane (M12) runs them durably; the quality plane records why a run behaved the way it did, verifies an agent or playbook is good before it gains authority, versions what is released, and stops a bad release's traffic automatically — but only in the safe direction.

The governing rule for this whole family: automatic only in the safe direction. The platform may revoke authority automatically (a guardrail rollback, disabling a failing playbook); it never grants it automatically (promotion, trust raises, and release activation always pass a human/policy gate). This is the M11 "agents never self-approve" invariant extended to releases.

Two owners, one join key:

• Jentrix/Postgres owns quality verdicts — eval cases, datasets, run results, releases, experiments, and guardrail decisions are domain rows with the usual authz, activity, and MCP treatment.
• The trace backend owns raw evidence — OTel spans are exported, sampled, and retained outside Postgres. Domain rows carry traceId stamps and verdicts, never span payloads. Traces are evidence for graders and humans; they are never authorization-relevant state.

Trace plane (M13.1)

Every managed run leaves exactly one queryable trace. A W3C trace context is created at the entry point (request_run, an MCP tool call, a cron dispatch, a webhook) and propagated through the workflow, Activities, sandbox, broker, and provider adapters. Spans follow the OTel GenAI conventions (model, token usage, tool calls, stop reason). The traceId is stamped centrally in the shared writers onto both activity ledgers (Activity, WorkspaceActivity) plus PolicyEvaluation, CredentialGrant, SandboxRun, AgentRun, WebhookDelivery, and Artifact — so call sites cannot forget it.

Redaction is a security boundary, not a filter. Redaction runs at the collector, unconditionally, before export (src/lib/otel/collector.ts → src/lib/otel/redaction.ts); the exporter is private, so there is structurally no path to the backend that skips it. Prompt/output content is scrubbed to a length-only summary; secrets and PII match a conservative superset. A seeded secret provably never reaches the backend.

What is live today (honesty box). Span export is provider-optional: the OTLP/HTTP transport (src/lib/otel/otlp-exporter.ts) ships spans only when OTEL_EXPORTER_OTLP_ENDPOINT is configured — without it the collector still redacts and then drops, so this deployment has no external trace backend until the endpoint is set (OTEL_EXPORTER_OTLP_HEADERS / OTEL_SERVICE_NAME are honored). Trace contexts open at the MCP boundary (safe()), every cron route, and the inbound GitHub/Telegram/Vercel/CI receivers; UI server-action writes are not yet stamped — their traceId columns stay null until an action-side boundary lands.

SLOs are a product contract. TimingSpan aggregates get p50/p95 targets for MCP initialize, warm tool call, worker Activity, and end-to-end run classes (src/server/telemetry/slo.ts), surfaced on the Control Tower quality panel and enforceable as experiment guardrails and CI checks. The panel's percentiles are computed in SQL over the full window (readSloLatency in src/server/telemetry/span-store.ts — percentile_cont p50/p95, counts, and error counts grouped by span name, no row cap), so n is the true window count even at production tool-call volume; the capped in-memory aggregation remains only on the debug/trace read paths. mcp.initialize is emitted at the MCP mount itself (withInitializeTiming wrapping the /api/[transport] handler, inside auth). worker.activity and run.summary only have producers when the managed worker / managed runs are live — the panel labels those classes accordingly while they are NO_DATA.

Eval registry (M13.2)

EvalCase / EvalDataset / EvalDatasetVersion / EvalRun / EvalResult are real models. Graders come in three kinds:

• deterministic — schema/equality/predicate assertions over structured output;
• model-graded — a pinned grader model + prompt rubric (the judgment is made once at capture and sealed as evidence — never re-rolled at grade time);
• trace — assertions over the M13.1 spans (egress budget, tool-call budget, error-code cleanliness).

Each case drives the real managed-run lifecycle (a durable SandboxRun begin → claim → run → finalize, budget-enforced, with the genuine managed-run id reconciled back onto the eval). The sandbox-backed managed ExecutionBackend has landed (M16): ManagedExecutionBackend runs a real release workload on the local-docker driver through the provider proxy (below), and the managed-eval gate is derived from the live worker registry, not a constant — it is green only while an enabled, live worker advertising an approved production driver class (local-docker; in-process/host-process are excluded permanently by a code allowlist and can never satisfy managed gating). When such a worker backs a run, a gating verdict measures the released agent's real behavior. With no live approved worker enrolled the gate is closed and the in-process simulator (the PR/test lane — it returns the fixture's declared subject behavior with constant provider/model and synthetic token accounting) is used; a managed-mode gating run in that state is refused, never silently demoted. Enrolling a runner node on the Runner Nodes page and enabling a profile are operator actions (the gate flips green from the registry the moment an approved worker is live; see the enablement runbook in prds/reference-software-company-acceptance.md). Failure handling is honest either way: a workload that fails (resource limit, crash) ERRORs its case and the run — it is never graded into a pass — and a run that overruns its budget ERRORs even when the overrunning case is the last one; any engine exception lands the run ERRORED, never stranded RUNNING.

Operator surface (settings/evals)

The plane is operable from the app, not only over MCP. /[ws]/settings/evals is the registry management page: datasets (each row drills down in place to its immutable versions — v#, pinned-case count, note, short content hash, head marked — and an empty registry offers ADMIN members a one-click Seed starter datasets action, a thin shim over the same mcpSeedStarterEvalDataset core the seed_starter_eval_dataset tool calls, so Role.ADMIN is re-checked server-side), cases by grader kind, and the runs list with search + facets (status, gating, dataset) + sort (newest default, score) over the COMPLETE run set — the read holds the whole list and the client narrows it whole-list in memory, so a run older than the newest hundred still surfaces when it matches an applied filter. Each run row deep-links to the run-detail page (/[ws]/settings/evals/runs/[evalRunId]): the summary (status; score always beside its per-case pass fraction; dataset + version; executing worker, its driver class, and the R8 per-run cost), a per-case results table where each row links to its trace, and — for a refused gating run — the terminal reason shown verbatim. Both surfaces read through src/server/queries/evals.ts; seeding is the one mutation the page exposes.

Managed provider proxy + budget contract (M16.3a — R7/R8)

When the managed backend runs a real release workload (not the in-process simulator), the sandbox never holds a raw provider key. Each case run gets a short-lived, HMAC-signed workload token (agents/lib/provider-proxy.ts) addressed to a host-side provider proxy that runs next to the LocalDockerDriver. The token is bound to the SandboxRun id (the token↔run audit identity), the release's pinned provider/model, the whole-EvalRun token/spend ceilings, and an expiry. The proxy is the single egress + spend choke point and enforces, per provider request:

• egress destination allowlist — only the pinned provider's API host (PROVIDER_EGRESS_HOSTS); the proxy refuses any other destination (403);
• provider/model pins — an off-pin request is refused (403);
• expiry — a request past the token's short lifetime is refused (401);
• per-run token/spend hard-stop, mid-flight — cumulative spend for the EvalRun is re-derived from the durable ledger before every forward, so the check survives a proxy restart. It is NOT post-hoc. The pricing meter (createPricingMeter) derives real per-request spendUsd from token counts × the pinned model's list price, so a run bounded only by maxSpendUsd still trips.

Admits only the proxy — enforced at the network layer, not by env convention. The managed-eval sandbox joins the worker's per-worker EvalEgressGateway: an --internal Docker network with NO NAT route to the internet, whose only reachable peer is a dual-homed forwarder that relays to the host provider proxy. HTTP(S)_PROXY/empty NO_PROXY are still injected for well-behaved clients, but they are belt-and-suspenders: a process that ignores them and opens a raw socket to api.anthropic.com has nowhere to route — the kernel drops the packet, so every byte the sandbox emits lands at the proxy. The worker's LocalDockerDriver is wired with egressNetwork (the internal net) + egressProxyUrl (the in-network gateway); a gateway that cannot be provisioned fails the run closed (never a bypass-prone host-gateway NAT bridge).

Budget fail-closed bounds (documented). Eval budgets (maxCases/maxTokens/maxSpendUsd) apply to the whole EvalRun, are reserved against the proxy before dispatch (the ceiling rides in the workload token), decremented per case, and each overrun dimension stops the run within a bounded overshoot:

Every managed case run carries mandatory wallMs/memory/CPU/PID/disk limits (DEFAULT_MANAGED_LIMITS) even when the eval budget bounds only tokens/spend — a run is never launched limit-free.

Crash-safe cost accounting. The proxy appends one durable ProviderProxyLedger row per metered (and per blocked) request — the authoritative token/spend truth, never process memory. Budget enforcement, usage reconciliation (src/server/evals/provider-ledger.ts — the adapter reads the ledger, not the sandbox's self-reported transcript), and daily cost accounting rolled up per workspace/day (rollupDailyProviderCost, a date_trunc('day') GROUP BY) all read these rows, so all three survive a worker/proxy restart.

Datasets have append-only versions: a version pins its member cases and each case's grader; adding a case cuts a new version and advances a mutable head pointer, so two runs of the same version are comparable and advancing the head never changes what an existing release was gated on.

Gating evidence is captured, not queried — downstream of redaction. A gating run exports its spans unsampled and captures the grader-relevant evidence (span set, inputs, outputs, post-redaction) as a content-addressed artifact at completion (src/server/evals/evidence.ts, evidence-store.ts). Graders execute over the captured bundle, never over live trace-backend queries. Missing, expired, or unreachable evidence is a distinct fail-closed state (EVIDENCE_UNAVAILABLE / EVIDENCE_EXPIRED) — never a cached pass (src/server/evals/regrade.ts). Evidence artifacts capture as EXPIRING and a cron sweep step (src/server/evals/retention.ts) deletes them after a TTL (default 90 days, STACKS_EVIDENCE_TTL_DAYS) unless a live citation holds them — a non-retired release's gating run, a pointer transition whose activated release is live, a running experiment, or a live playbook's promotion snapshot. Deleting uncited evidence degrades any later regrade to the fail-closed state, never a cached pass.

Verified promotion gates (M13.3)

Promotion-class operations resolve references instead of counting array elements:

• PolicyRevision is append-only and content-hashed; promotion resolves requiredPolicyIds to the current revisions and snapshots the revision ids + hashes into the promoted object.
• Playbook promotion requires ONE terminal, complete, passing gating EvalRun over a specific dataset version whose subject digest matches the playbook configuration, with every referenced case a member of that version that PASSED in that one run — promotion never assembles per-case passes across runs, and failures are structured envelopes that name the missing/failing reference.
• An incident's eval gap becomes a real regression EvalCase and advances the regression dataset's head; a release or playbook that reintroduces the behavior fails its gating run.

Immutable releases (M13.4)

AgentProfile stays the stable identity; a new immutable, content-addressed AgentRelease pins the full executable configuration — model + provider adapter, worker/sandbox image digests, code snapshot, prompt/tool-profile snapshots by content hash, skillRefs, policy revisions, gating dataset version, budgets — all hashed into a canonical configurationDigest.

Lifecycle is one-way and the pointer alone defines "active":

• Creation is ungated → CANDIDATE (this is what breaks the create/evaluate circularity; iterate by cutting cheap candidates).
• A green gating run for the exact digest marks it ELIGIBLE — over MCP, mark_release_eligible (admin scope; the server re-verifies the run is a PASSED, digest-equal, gating AGENT_RELEASE run before transitioning).
• RETIRED is terminal.
• There is deliberately no ACTIVE or ROLLED_BACK status: "active" is derived solely from AgentProfile.activeReleaseId, so status and pointer can never disagree.

Activation is where the new-authority gate lives — a compare-and-swapped ReleasePointerTransition (per-profile pointerRevision) requiring an ELIGIBLE release whose gating run is digest-equal, whose pinned dataset version equals the regression head, and whose policy pins equal the current required revisions at transition time. A stale head returns a CONFLICT naming the new head — re-cut and re-evaluate, or take a governed waiver (M4 exact-payload approval: single-use, expiring, tuple-bound, invalidated by any further head or configuration change). Rollback re-points to any non-retired ELIGIBLE release without re-running the new-authority gate — the safe direction — still as a reason-bearing transition.

Governance is resolved server-side (2026-07-16 hardening): a workspace-governed release must pin its full runtime set and a gating dataset version (resolved and workspace-checked from the version row), and its policy pins are derived from the workspace's current required set (enabled system/workspace-layer policies plus agent-scoped ones) — caller-supplied pins cannot narrow the gate, and a policy added after a cut makes the release stale at activation. Workspace bring-up (AGE-248): a fresh workspace has no eval dataset, so no governed release could be cut and no managed claim could ever pass — seed_starter_eval_dataset (admin scope + ADMIN role, idempotent) seeds the M13.2 grader smoke sets plus platform-smoke-managed-release, whose explicit JSON contract is graded from the real managed-provider output. The returned gating head is this managed set; fixture-substituted golden output can never authorize a release. Claim/launch resolves the pointed-to release and the preflight compares the release's pins against the claiming worker fail-closed: a governed claim that declares no capabilities (or no acceptedModels) is refused; the sdk/image/sandbox/snapshot stamp dimensions verify when declared and remain advisory until the managed fleet stamps them (AGE-188).

Experiments: shadow, canary, guardrails, rollback (M13.5)

An Experiment routes a fraction of eligible claims to a candidate (canary) or duplicates inputs through it side-effect-free (shadow). Starting a canary carries the full activation bar (a green gating run for the candidate's exact digest at the current head + current policy revisions, an explicit admin gate, and candidate preflight). During a canary the pointer never leaves the baseline; the router overrides the release for the canary fraction by a stable hash of the claim id (reproducible, auditable — never random sampling).

Shadow mode does not execute yet (honesty box). The canary path is live end-to-end (routing, guardrail sweep, rollback, promotion); a SHADOW experiment today is a governance record only — it is swept for drift and window expiry but no capture-only backend exists, so nothing re-executes the baseline's inputs and no comparison report is produced. The zero-side-effect guard and the report builder (src/server/experiments/shadow.ts) are shipped, tested cores awaiting the staging tier (AGE-188 / AGE-236). Do not read a RUNNING shadow experiment as evidence the candidate was exercised.

Guardrails (spend, latency SLO, quality score, incident count) are swept continuously; a breach stops candidate traffic by atomically terminating the experiment ROLLED_BACK (no pointer write — the pointer already names the baseline) and opens an incident with the comparison report attached. Governance drift mid-window terminates STALE (head advance / policy edit) or SUPERSEDED (an admin moved the pointer). The claim-assignment write is the linearization point: an assignment is strictly before or after the terminal CAS, never concurrent — so no candidate work starts after a rollback. Only promotion moves the pointer, and only by an explicit human action.

Quality gates in CI + acceptance (M13.6)

The gap matrix is the regression suite. The M13.6 exit-criteria are enforced as CI gates and a nightly ledger.

Acceptance status (as of 2026-07-16): NOT met. The conformance suite and the acceptance walk are executable, fixture-seeded proofs of the decision cores — they are not the live acceptance itself. The three-consecutive-real-green-nights exit counts from the ledger reset on 2026-07-16 (the originally committed ledger was pre-seeded and has been discarded — see "Nightly ledger" below), and the two-week Agentic-OS dogfooding window has not elapsed. Until both complete — and eval execution runs on a real backend (see the eval-registry honesty box) — M13 should be described as machinery shipped, acceptance pending.

CI conformance suite

tests/quality/ is the consolidated plane-3 conformance gate — one check per exit-criterion invariant, each exercising the production decision core the invariant lives in (src/server/agent-releases/currency.ts, src/server/experiments/{drift,guardrails,routing}.ts, src/server/evals/regrade.ts, the collector + redaction pipeline, src/server/telemetry/slo.ts). The invariants:

• activation requires a digest-equal green gating run on the pinned dataset version (equal to the current head) with policy pins current at transition time;
• the pointer alone defines "active" (no status column ever disagrees with it);
• waivers are single-use, expiring, tuple-bound, and invalidated by any further head or configuration change;
• playbook promotion fails on a missing/failing case and never assembles passes across runs;
• canary start is refused without its gate;
• governance drift mid-window terminates STALE;
• the claim-vs-rollback race never starts candidate work after the CAS;
• evidence fails closed (trace backend offline and evidence-deleted both yield EVIDENCE_UNAVAILABLE, never a cached pass);
• pointer CAS under concurrent transitions;
• a dataset head-advance leaves pointer-holding releases untouched;
• redaction conformance across BOTH stores — the seeded secret is searched for in the trace backend AND in captured evidence artifacts;
• SLO regression checks over TimingSpan aggregates on the golden paths.

Two invariants whose essence is a Postgres CAS / cross-run assembly — pointer CAS under concurrency and playbook promotion never assembling across runs — are additionally proven against the real ops (tests/quality/*.conformance.test.ts) with the same in-memory control plane the module tests use. Run it locally with pnpm quality:conformance; it also rides the ci job's pnpm test, and a dedicated fast quality CI job runs it with no database.

Nightly ledger

The plane-3 conformance suite + the dogfooding-acceptance walk append one dated entry to a nightly-green ledger (tests/quality/record-nightly.ts, the tests/chaos/record-nightly pattern); the exit opened once the consecutive-green streak reached three and stayed green. The scheduled quality-nightly.yml job was retired 2026-08-14 (STA-37) — the M13.6 exit it existed to open had been satisfied for weeks, and every gate it ran already runs on every PR (the quality job's pnpm quality:conformance plus a throwaway-ledger quality:record, and the suite rides pnpm test), so the scheduled ledger accrual was its only unique output and nothing read it. pnpm quality:record still appends a night on demand; tests/quality/nightly-ledger.test.ts proves the accumulation mechanics.

Nights are earned, never vendored: the committed ledger file ships empty (the CI cache carries the real history), and the recorder refuses any date other than today unless the test-only STACKS_QUALITY_ALLOW_DATE_OVERRIDE=1 flag is set — the nightly workflow never sets it. The streak was reset on 2026-07-16 after the post-merge audit found the original ledger pre-seeded; the three-real-nights exit counts from the first scheduled run after that date.

Dogfooding acceptance walk

The acceptance walk (tests/quality/nightly/acceptance.ts) is the executable, fixture-seeded version of the two-week Agentic-OS dogfooding: every managed run has a queryable trace envelope (an unsampled root run-summary span that survives sampling, with full step spans present when sampled in); at least one incident → regression case → blocked-release loop is exercised end-to-end (an incident advances the regression head and blocks a release that would reintroduce the behavior, while the re-cut release activates); and each experiment terminal path is exercised (one guardrail rollback, one promotion). The live two-week procedure runs the workspace regression + platform datasets on the managed plane; this walk is its deterministic regression proof.

Where things live