Skip to main content

​
PF-125 Agent Runs — Operator Runbook

Last updated: 2026-06-25 Owner: Jeremy Bloom Audience: Anyone supervising agent runs, reviewing the PF-45 go-live gate, or pulling the emergency stop. Spec: PF-125 — Multi-Agent Orchestration & Durable Runs · Epic: #1779 (Agentic Workspace) · ADR: ADR-026
Phase-1 posture (flag-off foundation). Everything PF-125 ships in Phase 1 is additive and inert. The interactive-tools capability is behind the PF-45 flag pf.agents.interactive_tools_enabled (default false), and the guardrail spine is unsatisfiable today (redaction + injection screening seams are absent), so tools are withheld fail-closed on every path. There is no run-creation surface yet — the Run Timeline panel renders empty. Flipping the flag in production is a human gate, not a deploy step.

​
1. What PF-125 is

PF-125 wraps the in-stack agentic loop (runAgenticLoop, supabase/functions/_shared/ai/agentic-loop.ts) in a durable, resumable, checkpointed run-state machine. A “run” is one agent task with a tracked lifecycle, an append-only audit trail, periodic checkpoints, and a hard token/step budget. Runs act under a scoped AgentPrincipal (PF-30-EN-01 pf_agent_can) — never service_role, never auth.uid() impersonation. Authoritative state lives in four tables (all RLS FORCE, org-scoped):

​
Lifecycle transition graph

Terminal states (completed, failed, cancelled, budget_exceeded) have no outgoing transitions. A BEFORE UPDATE trigger rejects any direct write to lifecycle_state — even by service_role. State changes go only through the governed RPCs.

​
2. The guardrail spine (why tools are withheld)

An agent’s tools reach a model only when the full spine holds and the PF-45 flag is on — on every path, identically (FR-9/FR-10):
  1. a scoped AgentPrincipal is bound to the run,
  2. redaction is active (PF-27-EN-01),
  3. a budget check is available (FR-12),
  4. injection screening is enabled (PF-126).
In Phase 1, (2) and (4) are not built, so the spine is unsatisfiable → tools are withheld fail-closed. A write-capable run that reaches the boundary off-spine is refused outright (SpineNotSatisfiedError, run advanced to failed) — never silently downgraded to a “running but toothless” state. The decision logic is pure and unit-tested (supabase/functions/_shared/ai/agent-spine.ts; red-team suite tests/red-team/pf/pf-125-spine-bypass.red-team.test.ts).

​
3. The PF-45 flag — activation & rollback (HUMAN GATE)

pf.agents.interactive_tools_enabled (seeded false) is the master switch for interactive tool use. Do not flip it in production until all spine prerequisites are live: PF-27-EN-01 redaction, PF-126 injection screening, and the C-01..C-08 controls + PF-111 budget / PHI-audit integration deferred from Phase 1. Confirm each, then enable per-org (org-scoped feature flag) — never globally as a first step.
  • Activate (per org): set the flag on for one pilot org, drive a single read-only run, watch the Run Timeline audit trail end-to-end, then widen.
  • Rollback / kill switch: set the flag back to false. Tools stop being offered on the next request immediately (the gate is evaluated per request). In-flight runs continue under the state machine; cancel them explicitly (§5) if you need them stopped.
The flag being off is not a substitute for the spine — both must hold. Turning the flag on while the spine is unsatisfiable still yields zero tools (by design).

​
4. Monitoring runs (Run Timeline panel)

Route: Settings → Platform → Agents → Runs (/settings/platform/agents/runs).
  • View requires pf.agents.runs.view; Cancel/Resume require pf.agents.runs.manage.
  • The panel is read-only except for the governed Cancel/Resume actions, which route through the RPCs — the UI never writes lifecycle_state.
  • Each row shows the skill, lifecycle badge, a Write-capable badge when applicable, the bound principal (last 4 only — HITECH attribution, never the full UUID), token usage vs budget, and an expandable lifecycle audit trail.
  • Phase 1: with no run-creation path and the flag off, the panel normally shows the empty state. It exists so the audit trail is observable the moment the durable path is enabled.
The lifecycle event types are registered in the workflow-events catalog (pf_agent_run_started, pf_agent_run_checkpointed, pf_agent_run_completed, pf_agent_run_failed, pf_agent_run_budget_exceeded; owning_core: pf, category: platform) so downstream automation can subscribe — the runtime emitters land with the flag-on rollout, not in Phase 1.

​
5. Operating a run

All actions go through the governed RPCs (also surfaced as Cancel/Resume in the panel):
  • Cancel a run: Cancel in the panel, or pf_agent_run_advance(<id>, 'cancelled', '<reason>'). cancelled is terminal — the run cannot be resumed afterward.
  • Resume a paused run: Resume in the panel, or pf_agent_run_resume(<id>).
  • Budget exhaustion: a run that hits its token/step budget advances itself to budget_exceeded (terminal) and the loop is not re-dispatched.

​
6. Emergency stop

  1. Stop new tool use immediately: set pf.agents.interactive_tools_enabled → false for the affected org (or all orgs). New requests get zero tools on the next call.
  2. Halt in-flight runs: cancel each non-terminal run via the panel or pf_agent_run_advance(<id>, 'cancelled', 'operator emergency stop').
  3. Investigate: read the pf_agent_run_events audit trail for the affected runs (expandable in the panel). Principals are attributed; correlate with pf_agent_can grants if a scope looks wrong.
  4. Never bypass the state machine with a direct SQL UPDATE — the trigger rejects it, and doing so would break the audit trail. There is no service-role write path by design.

​
7. Invariants (do not violate)

  • Agents act under a scoped principal via pf_agent_can — never service_role or auth.uid() impersonation.
  • No PHI in prompts, logs, code, or tests. Redaction must be live before the flag is enabled.
  • Tenant isolation: every run/task/checkpoint/event is organization_id-scoped under FORCE RLS.
  • Lifecycle state changes go only through the governed RPCs; the BEFORE UPDATE trigger is the backstop.
  • Flipping PF-45 on in production is a human gate — it requires the full spine, not just a green CI.

​
References

  • Spec: specs/pf/PF-125-* · Epic: #1779 · ADR: docs/architecture/decisions/ADR-026-*
  • State machine: supabase/functions/_shared/ai/agent-run-state-machine.ts
  • Guardrail spine (pure): supabase/functions/_shared/ai/agent-spine.ts
  • Red-team suite: tests/red-team/pf/pf-125-spine-bypass.red-team.test.ts
  • Panel: src/platform/agents/ui/RunTimelinePage.tsx
  • Migrations: supabase/migrations/20260630000001_pf125_multi_agent_orchestration.sql (+ permission/flag seeds)