Simulation, comms, playbooks & incidents

The Agentic Operating System's final layer adds the safety-and-leverage capabilities that close the loop: dry-run any risky plan before it runs, bring stakeholder conversations into context under source governance, turn repeated successful runs into reusable playbooks, and escalate failures into structured postmortems. It also lights up the deferred MCP write tools now that the UI and policy semantics are stable.

Simulation / dry-run mode

A simulation replays a staged plan against the ops core inside a never-commit boundary and reports what would happen — with zero external side effects.

• Propose. From a task's detail panel, "Dry-run a move to…" creates a SimulationRun. Agents call create_simulation with a plan of staged ops (move_task, bulk_move_tasks, update_task, bulk_update_tasks, set_task_labels, set_task_assignees, archive_task, send_outbound).
• Review. The preview shows per-op diffs and a projected side-effect table: notifications, activities, webhooks, automations that would fire, estimated cost, credential grants, required approvals, and lease conflicts — all computed before anything happens. Unsupported ops are surfaced as warnings (they would run unprojected at execute time).
• Approve → execute once. Approval binds an idempotency key and freezes a plan checksum. Execute runs each op through the real ops core exactly once; a second execute is an idempotent no-op. If the stored plan no longer matches its checksum, execute returns CONFLICT (409) and re-approval is required.

How it stays side-effect-free: a dry-run runs under an AsyncLocalStorage boundary (runWithSimulation) so the event fan-out (emitBoardEvent / emitWorkspaceEvent — Pusher + webhook enqueue), post-response work (scheduleAfterResponse: automation dispatch, GitHub push, notification email), and the DB write itself are all recorded, not performed — the transaction is always rolled back. A rollback alone is insufficient because those effects fire after commit; the boundary intercepts every one. There is no execute_simulation MCP tool in v1 — execution requires the human approval surface.

Communication integrations

A communication source is a governed Slack/email/calendar/doc connector. It is an M5 source record (1:1), so it inherits trust/freshness/privacy/owner/ project-scope — no duplicate governance. No secrets are stored on the connector; connector secret refs live in the credential broker.

• Ingest. Messages/meetings land as IngestedMessage rows (deduped by (source, externalId)), carrying the source's privacy boundary denormalized for fast filtering. Slack and document adapters are live when their optional provider configuration is present; connector OAuth authorization uses MCP elicitation and stores only a credential-broker reference.
• Link. Messages link to tasks/contacts/projects/decisions/Work Orders and appear in the linked-conversation panel. A project Work Order surfaces its linked conversations + meeting commitments. Governance is enforced: a client-private conversation scoped to one project cannot be linked into a different project's context.
• Draft or governed send. draftReply writes a draft (a comment on the linked task). sendOutbound fails closed unless outbound is enabled, the Policy Engine allows it, and the provenance floor is satisfied by a human action, exact-payload approval, or matching send route. Authorized sends use the live provider adapter and transactional delivery engine. Agents use the admin-scoped send_communication MCP tool; privilege alone never bypasses the provenance floor. See Communication connectors for the full contract.

Playbook mining

Repeated successful runs become reusable playbooks.

• A miner clusters successful delivery and harness runs by their workflow signature into a PlaybookDraft (trigger, context manifest, tools, policies, evidence runs, eval candidates). The daily outcomes job mines both live ledgers and refreshes only pristine drafts.
• Owners approve / reject / edit drafts, then promote an approved draft to a Playbook. Promotion is gated: a playbook cannot be enabled for broad use without non-empty required policy IDs AND eval case IDs (admin-only). A promoted playbook can be frozen into an immutable, content-addressed SkillVersion through the UI or convert_playbook_to_skill.

Incident / postmortem flow

A failed run or bad action escalates into an incident with severity, owner, status, and review date.

• The postmortem records an append-only timeline, root cause, affected tasks/contacts/repos, a rollback action log, and explicit policy gaps and eval gaps. It links to runs/agents/policies/credentials/artifacts/ decisions.
• Follow-up tasks are linked to both the incident and (via the task's board) its project, so the corrective work lives on the board while the incident stays a record.
• createRegressionEvalFromIncident creates a real incident-origin EvalCase, cuts a new immutable version of the workspace regression dataset, and records the resulting case id on the incident.

Resolution requirements

Every incident resolves only with a root cause, an affected scope, a timeline, a mitigation/result, and either ≥1 follow-up task or an explicit "no follow-up" rationale.

A HIGH or CRITICAL incident additionally requires the full disposition (operator control plane R11 / M18-AC16). Every one of these is a durable fact already on the record — the resolve call itself takes nothing but the R6 "no follow-up" rationale, so a resolution is auditable a second later, a sweep later, and in a review:

Three refusals are worth naming. Claiming CASE_CREATED without a case id is rejected — the claim is not the decision. Claiming it with an id that does not resolve is rejected the same way: Incident.evalGaps is loose Json with no foreign key, so the id is proven by lookup against EvalCase — it must be a live (unarchived) case in this workspace authored for this incident (EvalCase.incidentId), which is what "Create regression eval" writes. A fabricated, cross-workspace, archived, or someone-else's-incident id is refused by name at the write boundary and re-verified at resolution (a case archived after it was recorded, or a row written before this check existed, blocks the resolve), and no EvalCase is ever created to satisfy the claim. And declining a regression guard (NOT_NEEDED) is governed exactly like making one: owner-or-ADMIN, rationale required. The gate itself is pure (src/server/incidents/resolution.ts) and takes the proven ids as a fact (verifiedEvalCaseIds, fail-closed when empty); resolveIncident is the shell that looks them up and enforces the gate, and the reconciler never resolves incidents.

A dead decision is recoverable, and that is load-bearing. evalGaps is append-only — both write paths (createRegressionEvalFromIncident, updateIncident({ evalGap })) can only push, and neither can edit or delete an earlier entry. So the gate reads the whole log, the shell proves every EvalCase id on it in one lookup, and the operative decision is the newest one that holds up. An incident whose first entry names a dead case is unblocked by doing exactly what the refusal says — create a regression eval, or record a reasoned NOT_NEEDED — and the refusal names the dead ids so nobody re-records them. Reading only the first entry would let one bad row shadow every later fix and strand a high-severity incident permanently; the audit row on INCIDENT_RESOLVED therefore snapshots the decision that was proven (regressionEvalDecision / regressionEvalCaseId), which after a recovery is a different entry from the one first written. Note the ordering rule that keeps this honest: a case id is read ahead of the decision label, so an id smuggled in beside NOT_NEEDED is still a CASE_CREATED claim that must resolve — recovery is appending a clean NOT_NEEDED, never relabelling a dead id.

The EXECUTOR-run requirement lives here, at resolution, and not on the auto-open path: a production deploy can legitimately have no Jentrix execution behind it (a human push, a manual redeploy, a rollback), and fabricating a run to satisfy a link is the one outcome this platform must never produce. At resolution a human is present to supply what the graph could not.

Follow-ups are commitments

An IncidentTask marked required is a corrective commitment, not a note. It is attached that way by the R11 auto-escalation (which always opens one) or by a human ticking "required", and it ends in exactly one of two recorded dispositions:

• COMPLETED — a MEMBER-level statement that the corrective work is done.
• WAIVED — an ADMIN-only override that requires a reason. This is the platform's only waiver channel for the coupling below: tuple-bound to the row, audited, and permanent.

An archived task is not a disposition. Closing the loop is a decision somebody records, not a side effect of tidying a board.

Completion coupling

An execution cannot reach a clean acceptance verdict while a correlated incident's REQUIRED follow-ups are still open. The coupling rides on the existing harness acceptanceState ladder — HarnessStatus.COMPLETED keeps its meaning ("all stages committed"), so nothing can wedge behind an incident:

• Undisposed → the run's acceptanceState stays PENDING, exactly like an open blocking finding.
• WAIVED → CONDITIONAL. A waiver is a visible conditional outcome, never a clean ACCEPTED, and because the disposition is permanent the verdict cannot lapse back once a TTL expires.
• COMPLETED → the coupling clears.

Which incidents belong to an execution is decided by the shared ownership rule (src/server/control-tower/lifecycle-correlation.ts) — the same functions Control Tower's lifecycle view uses, so the two can never disagree. Both shipped channels bind: a deployment owned through its PR's head branch (even when Deployment.taskId is null), and an incident owned through Incident.runId → AgentRun → AgentJob. The read is fail-closed: if it fails, the acceptance state is left unchanged rather than reading as "nothing outstanding". get_harness reports unresolvedRequiredCount and waivedRequiredCount beside the finding counts. There is deliberately no MCP tool that waives an execution's own corrective obligations.

Archive and restore (R13)

"Remove" means archive everywhere in the operator control plane — the one exception is a security credential, where remove means revoke (a credential grant, a credential record, a provider secret, an API token, a worker claim credential: the material stops working immediately and there is nothing to restore). That sentence is not left implicit: REMOVAL_SEMANTICS in src/server/archive/dependency-blockers.ts is a TOTAL map over both families, and a subject with no answer is a type error.

Every archive reports its active dependencies and cannot silently break running work (M18-AC19). One pure rule table decides, per subject:

Three things make this hold rather than merely exist:

• The rules live in one pure table, not in six call sites. Blocking relations, restore-safety, the effect line, and the replacement workflow are each a TOTAL Record over the subject union, so a new archivable subject cannot compile until all four have explicit answers (dependency-blockers.test.ts pins the totality).
• Only an ACTIVE dependency blocks. A historical one — a terminal run that used the configuration — is reported and never blocks, which is exactly what makes archiving safe: R12 requires archived configuration to stay visible in the runs that used it.
• A refusal offers the way through. The 409 names the blockers and the replacement workflow ("point the listed worker releases at another verified connection, then archive this connection"), on every surface — the hint the MCP envelope carries and the message a UI toast renders.

The shell (src/server/archive/operations.ts) loads the facts and each archive op calls assertArchiveAllowed first; the dependency report lands in the audit payload of the archive that did happen, and the management surfaces preview the same preflight before offering the action (previewArchiveDependenciesAction), so the preview and the refusal can never disagree.

Every subject in the table above has a shipped archive path that consults the preflight — a rule table nothing calls is not a rule, so dependency-blockers.test.ts asserts the caller per subject: archiveProviderConnection, archiveModelProfile, the worker's archive report, archivePermissionTemplate / restorePermissionTemplate (src/server/permissions/operations.ts), archiveHarnessTemplate, and archiveHarness. The permission template's operator path is the one R13 names explicitly, so it is wired end to end and pinned link by link: the worker page (/[ws]/agents/[agentId]) loads loadPermissionTemplateArchiveState and mounts PermissionTemplateArchiveControl, which renders R13's five columns from the server-computed preflight and calls the archivePermissionTemplateAction / restorePermissionTemplateAction session shims over the ops core. The one exception is the Work Item, which R13 explicitly delegates to the product's existing task archival — and which is safe precisely because nothing can block it (an empty blocking set).

Archiving a permission template is the fail-closed direction: an archived template reads as absent to loadWorkerPermissionState, so the launch/claim gate refuses NEW managed work for that worker while an attempt already executing keeps running under the immutable revision it pinned. Restore is offered because RESTORE_SAFE says so — the flag is read from the shared table, never re-decided at the call site, and the published revisions are immutable, so a restored worker resumes under exactly the revision it had.

"Reads as absent to the gate" is not the same as "gone from the operator's view", and the worker page keeps the two apart: because the effective-permission preview resolves to nothing once the template is archived, the page passes an archived-specific reason into the permissions section rather than letting it fall through to its default "this worker has no permission template yet" — an archived record reported as one that never existed is precisely the failure R12 forbids. The archive row itself stays mounted with its state badge, dependency count and Restore action, so the archive is visible and reversible from the same place it was performed.

State-honest mutation UX (R14)

R14 asks every operator mutation for six things: a loading state, a success confirmation, a serialized domain refusal, an optimistic-concurrency conflict, retry guidance, and an aria-live error announcement. The first two come from the shared async-feedback contract already described in CLAUDE.md (Button's loading/loadingText, Pending, the success toast). The last four are one shared primitive, split the way the shared list controls are — pure model in src/lib/mutation-refusal.ts, client half in src/components/sync/mutation-error.tsx — and mounted on every M18.4 mutation surface. Which surfaces those are is DERIVED, never listed: the population is every client component that mounts MutationError, read off the tree by tests/unit/ux/state-honest.test.tsx, so a surface added later cannot escape the contract by not being remembered. The converse scan is what makes that honest — a component that drives one of the M18.4 mutation actions and mounts no MutationError fails the suite, which is how the H10.8 harness-gate decision panel was found destructuring isPending and dropping error on the floor. Five surfaces satisfy it today: the R13 archive control, the R12 integrity-queue drawer, the §10.6 disposition panel, the R11 incident resolution, and the H10.8 gate decision.

Four things about it are load-bearing.

A toast is not an announcement. Before this, a refused mutation surfaced as a sonner toast: it auto-dismissed, it was not bound to the control that was refused, and it was not announced. The surface then looked exactly as it did before the operator pressed the button — the "state-honest" failure R14 names. MutationError renders next to the control, persists, and carries role="alert" + aria-live="assertive" (progress and success stay polite via Pending, so the two never talk over each other).

The classification is a MESSAGE classifier, and that is forced. A Server Action rejects across the network as a plain Error — the AuthzError subclass, its status, and its current payload are erased by serialization, so the client holds only err.message. The MCP surface keeps the structure (it has an error envelope with a code); the UI surface does not, and a second transport for it would be a parallel path. So the classifier matches the wordings the ops cores actually throw, each named at its source, and is deliberately CONSERVATIVE: an unrecognized message is refused with generic guidance, never upgraded into a conflict. The gotcha that bit during authoring is pinned by a test — a message that merely MENTIONS expectedUpdatedAt is not a conflict ("expectedUpdatedAt is required" is a validation failure, and "reload and reapply" is the wrong instruction for it).

The server's words are carried verbatim. R14 asks for a serialized domain refusal; only the guidance is ours. A surface that paraphrases the platform's reason is how an operator ends up debugging the paraphrase.

disabled is not the double-submit boundary. The async contract's "a disabled control by itself is not sufficient feedback" has a converse the R14 work made concrete: disabled is a DOM property, so it only stops a real pointer event — a keyboard repeat landing mid-transition, a render-prop element the primitive maps disabled onto as aria-disabled, or any programmatic invocation still reaches the handler and fires the mutation again. So Button (inertHandler, src/components/ui/button.tsx) WITHHOLDS onClick while loading || disabled, in the one module that owns loading. Every mutation surface inherits it, and "cannot submit twice" becomes a property of the component rather than of the environment it renders in. Nothing an operator can observe in a browser changes.

Empty states are the other half. Each carries ONE primary action, and which one depends on WHY the list is empty: a narrowed list is repaired by clearing the narrowing (ListNoMatches), a genuinely empty workspace by creating the first record. Never a pointer to MCP or the CLI — those are additional surfaces over the same ops core, not the way an operator is expected to get unstuck.

tests/unit/ux/state-honest.test.tsx drives every derived surface through four states — each from its own render, with useSyncedTransition parameterized so the state is one the suite produced rather than one it inferred — plus the two behaviors a static render cannot reach, driven through the control's own handler as the SHIPPED Button forwards it: an invocation while pending cannot call the action a second time, and an action rejecting with a stale write routes to the CONFLICT branch instead of the generic one. It runs in the repo's node environment; no DOM environment was added, and the full-async-PERIOD property is proven where it lives (useSyncedTransition + Button, with sync-store.test.ts covering the store) while each surface owes only the ROUTING obligation — take pending/error from useSyncedTransition().run, pass loading to Button.

End-to-end closure proof (AC24)

The milestone's closing criterion is that the whole flow works end to end — Work Item → PRD → Plan → Code → Test → Deploy → Monitor → Improve — with one correlated execution visible in the Control Tower. Per-phase unit tests cannot make that claim: each is free to invent a fresh execution per phase and still go green. So tests/unit/lifecycle/end-to-end.sim.test.ts is a sim in the sense src/server/harness/liveness.sim.test.ts is: one world of authoritative records, advanced by eight named durable transitions, re-read after every hop, with the correlation invariant asserted against every phase and every evidence row rather than a sample.

Two things about its construction are what make it evidence.

It reads through the PRODUCTION shell, not the pure projector. Each hop writes Prisma-shaped rows into an in-memory substrate (tests/unit/lifecycle/prisma-substrate.ts — a generic where/orderBy matcher, deliberately not canned per-call answers, so the double can disagree when the shell asks for the wrong thing), and every assertion reads them back through listExecutionLifecycles / getExecutionLifecycle. That shell is where the authorized set is read, where the R9 correlation rules run, where filter-first/paginate-last lives, and where the facet menus are collected before narrowing — none of which a fixture-fed projector exercises. requireWorkspaceRole runs for real against a modelled membership row; only Postgres is modelled. An earlier version of this file built LifecycleFacts by hand and called projectExecutionLifecycle directly, which proved the projector and not the closure.

It proves closure at DEPTH. The finished execution is buried under 320 newer rows — past the scan bound this milestone retired — and is then reached by both of R10's identifiers (Work Item id and human key), by free-text search on either correlation id, by the completed view, and by the template and environment dimensions, with facet choices still collected from before the narrowing. Closure that only holds on page one is not closure, and a bound reintroduced on the run query turns thirteen of these tests red rather than one.

The ordering assertion is the shell's, not the sim's, in one place worth knowing: Monitor's first signal is DERIVED from the Deployment row, so the Deploy hop also opens the incident the promotion raised — otherwise a successful promotion would complete Monitor a hop early, and the Monitor hop is the observation that resolves it.

Authorization parity across UI / MCP / script (AC23)

AC23's claim is NEGATIVE — for six authority families (provider, credential, permission, release, deploy, waiver) a member must not gain authority through a surface difference. Per-surface suites cannot make that claim: each is green while another surface quietly skips a check, because none of them compares the three. tests/unit/authz/parity.test.ts is the comparison.

The matrix is derived, and asymmetry is the finding, not a defect. A universal six × three table is not reachable on this checkout, and forcing one would mean GROWING the surface — a new tool, a new script command — which is precisely the escalation path AC23 forbids. So every cell is classified from shipped artifacts at test time: ops cells bind by DIRECT IMPORT of the family's mutation entrypoint (a rename fails pnpm typecheck rather than silently emptying a cell); MCP cells come from the REAL registered server (connectMcp → listTools) intersected with a source scan of the tool-registration module, so a cell names the registered tool that reaches the entrypoint rather than one whose name merely looks related; script cells come from HARNESS_TEMPLATE_COMMANDS + planCall, planned for real. Today: provider, credential and release are MCP-PRESENT; permission and deploy are ABSENT; a waiver is PRESENT_BY_PARAMETER (set_active_release's waiverId CITES a waiver an admin already granted — no tool GRANTS one); the script reaches none of them.

Absence is a maintained property. For every non-PRESENT cell the suite proves no registration block reaches the family's mutation entrypoints and that no planned script call does either — symbol-level, because module-level would be both false and too weak (tools.ts legitimately imports openIncident for create_incident while never reaching setIncidentFollowUpDisposition). The RATCHET is asserted explicitly on both axes: a synthetic registration block and a synthetic script command are fed through the SAME derivation and must flip their cell, so the day someone registers a deploy or waiver tool the suite is red until the matrix drives it.

Every PRESENT cell is driven, and the measurement is guarded against vacuity. Each family runs owner/admin/member/non-member through every surface that carries it, against the real ops core and the real requireWorkspaceRole. The member refusal must be identical across surfaces — same class, and for an authority refusal the same SENTENCE, since the ops core is the single authority and the MCP boundary carries its refusal through rather than composing one. The trap this design has to avoid is a family whose subject fixture is missing: every role then gets an identical 404 and the parity assertions pass while measuring nothing. So each family seeds exactly the rows its entrypoint reads on the way to its authorization check and nothing past it, the NON-MEMBER outcome must be the MEMBERSHIP refusal specifically (which proves the check was reached), and the observed role floors are recorded as an assertion — ADMIN for five families, MEMBER for the delivery entrypoint whose DEPLOY authority is the always-human gate above it.

Proving absence is necessary and not sufficient, so one row is driven on all three surfaces. The six families establish that the script cannot be the permissive surface for authority it cannot reach — but a suite that only ever proves absence never measures the script against the other two, and a scripted path that skipped a role check would sit in exactly that blind spot. So the table carries a SEVENTH row, templateLaunch: the one mutation the shipped command vocabulary does carry (launch → launch_harness_template), driven for owner/admin/member through the ops core, the registered MCP tool, and the real main(argv, deps) — the same entrypoint agents/scripts/harness-template.ts calls with process.argv.slice(2), with only connect injected so the parse → plan → call → exit-code path runs against a real safe(), a real scope check and the real ops core. All three invoke ONE argv definition, so "the script was called differently" can never explain a disagreement. The same anti-vacuity rule binds it and bites harder here: the launch runs against a REAL template created and published through the ops core, an OWNER and an ADMIN must SUCCEED on all three surfaces before the MEMBER refusal counts as the role gate answering, and the refused scripted launch must leave no run, stage or assignment row behind while the byte-identical ADMIN invocation writes exactly one run. Parity is about effect, not only about exit codes. The launch is a dry run — real rows, real authorization, provably inert — so the probe cannot start work. The script prints the server's structured envelope verbatim, which is what lets the sentence be compared across all three rather than just the outcome class; the read side (preview) is checked on the same footing, since a resolution a member may not launch is one they may not read either.

MCP/API write-tool expansion

With the UI and policy model stable, the deferred write tools graduate. Phase 2 adds create/version tools (create_project, create_decision, create_work_order, version_work_order, request_run); Phase 3 adds the policy-aware tools (create_simulation, create_incident, resolve_approval, request_credential_grant). All conform to the MCP quality contract: correct scope class, spec annotations, Zod output schemas, idempotency keys on creates, expectedUpdatedAt on updates, and structured errors with policy-aware hints.

Retention policy

Open question L904, resolved:

• Unexecuted simulations expire 15 minutes after creation (a sweep flips proposed/approved → expired); the executed/approved record is retained for audit.
• Run traces, artifacts, and Work Orders are retained per a workspace-configurable window; every entity carries archivedAt from day one, so retention is a query/cron policy, not a schema change. Redaction leaves an audit tombstone — it never hard-deletes.
• Credential audit logs (grants, evaluations, approval bindings) are immutable and never auto-deleted.