> ## Documentation Index
> Fetch the complete documentation index at: https://docs.encoreos.io/llms.txt
> Use this file to discover all available pages before exploring further.

# PF-125 Agent Runs — Operator Runbook

> Operator runbook for the PF-125 durable multi-agent run-state machine: the PF-45 flag, the guardrail spine, monitoring runs, and emergency stop.

# PF-125 Agent Runs — Operator Runbook

**Last updated:** 2026-06-25
**Owner:** Jeremy Bloom
**Audience:** Anyone supervising agent runs, reviewing the PF-45 go-live gate, or pulling the emergency stop.
**Spec:** PF-125 — Multi-Agent Orchestration & Durable Runs · **Epic:** #1779 (Agentic Workspace) · **ADR:** ADR-026

> **Phase-1 posture (flag-off foundation).** Everything PF-125 ships in Phase 1 is additive and inert.
> The interactive-tools capability is behind the PF-45 flag `pf.agents.interactive_tools_enabled`
> (**default `false`**), and the guardrail spine is **unsatisfiable** today (redaction + injection
> screening seams are absent), so tools are withheld fail-closed on every path. There is no
> run-creation surface yet — the Run Timeline panel renders empty. Flipping the flag in production is a
> **human gate**, not a deploy step.

***

## 1. What PF-125 is

PF-125 wraps the in-stack agentic loop (`runAgenticLoop`, `supabase/functions/_shared/ai/agentic-loop.ts`)
in a **durable, resumable, checkpointed run-state machine**. A "run" is one agent task with a tracked
lifecycle, an append-only audit trail, periodic checkpoints, and a hard token/step budget. Runs act under
a **scoped AgentPrincipal** (PF-30-EN-01 `pf_agent_can`) — **never** `service_role`, never
`auth.uid()` impersonation.

Authoritative state lives in four tables (all RLS FORCE, org-scoped):

| Table                      | Holds                                                                                    |
| -------------------------- | ---------------------------------------------------------------------------------------- |
| `pf_agent_runs`            | one row per run; `lifecycle_state`, principal, budgets, `write_capable`, terminal reason |
| `pf_agent_tasks`           | the unit(s) of work a run executes                                                       |
| `pf_agent_run_checkpoints` | periodic snapshots + cursor for resume                                                   |
| `pf_agent_run_events`      | append-only lifecycle audit trail (from→to transitions)                                  |

### Lifecycle transition graph

```
queued ──────────▶ running ─────────▶ completed        (terminal)
   │                  │   │ │ │ │
   │                  │   │ │ │ └────▶ failed           (terminal)
   ▼                  │   │ │ └──────▶ budget_exceeded  (terminal)
cancelled (terminal)  │   │ └────────▶ cancelled        (terminal)
                      │   └──────────▶ waiting_approval ─▶ {running, cancelled, failed}
                      └──────────────▶ paused ──▶ resuming ──▶ {running, failed, cancelled}
```

Terminal states (`completed`, `failed`, `cancelled`, `budget_exceeded`) have no outgoing transitions.
A **BEFORE UPDATE trigger** rejects *any* direct write to `lifecycle_state` — even by `service_role`.
State changes go **only** through the governed RPCs.

***

## 2. The guardrail spine (why tools are withheld)

An agent's tools reach a model **only** when the full spine holds **and** the PF-45 flag is on — on
every path, identically (FR-9/FR-10):

1. a **scoped AgentPrincipal** is bound to the run,
2. **redaction is active** (PF-27-EN-01),
3. a **budget check is available** (FR-12),
4. **injection screening** is enabled (PF-126).

In Phase 1, (2) and (4) are not built, so the spine is unsatisfiable → **tools are withheld
fail-closed**. A write-capable run that reaches the boundary off-spine is **refused outright**
(`SpineNotSatisfiedError`, run advanced to `failed`) — never silently downgraded to a "running but
toothless" state. The decision logic is pure and unit-tested
(`supabase/functions/_shared/ai/agent-spine.ts`; red-team suite
`tests/red-team/pf/pf-125-spine-bypass.red-team.test.ts`).

***

## 3. The PF-45 flag — activation & rollback (HUMAN GATE)

`pf.agents.interactive_tools_enabled` (seeded `false`) is the master switch for interactive tool use.

**Do not flip it in production until** all spine prerequisites are live: PF-27-EN-01 redaction, PF-126
injection screening, and the C-01..C-08 controls + PF-111 budget / PHI-audit integration deferred from
Phase 1. Confirm each, then enable per-org (org-scoped feature flag) — never globally as a first step.

* **Activate (per org):** set the flag on for one pilot org, drive a single read-only run, watch the
  Run Timeline audit trail end-to-end, then widen.
* **Rollback / kill switch:** set the flag back to `false`. Tools stop being offered on the next request
  immediately (the gate is evaluated per request). In-flight runs continue under the state machine;
  cancel them explicitly (§5) if you need them stopped.

The flag being off is **not** a substitute for the spine — both must hold. Turning the flag on while the
spine is unsatisfiable still yields zero tools (by design).

***

## 4. Monitoring runs (Run Timeline panel)

Route: **Settings → Platform → Agents → Runs** (`/settings/platform/agents/runs`).

* **View** requires `pf.agents.runs.view`; **Cancel/Resume** require `pf.agents.runs.manage`.
* The panel is **read-only** except for the governed Cancel/Resume actions, which route through the RPCs
  — the UI never writes `lifecycle_state`.
* Each row shows the skill, lifecycle badge, a **Write-capable** badge when applicable, the bound
  principal (**last 4 only** — HITECH attribution, never the full UUID), token usage vs budget, and an
  expandable lifecycle audit trail.
* Phase 1: with no run-creation path and the flag off, the panel normally shows the empty state. It
  exists so the audit trail is observable the moment the durable path is enabled.

The lifecycle event types are **registered** in the workflow-events catalog
(`pf_agent_run_started`, `pf_agent_run_checkpointed`, `pf_agent_run_completed`,
`pf_agent_run_failed`, `pf_agent_run_budget_exceeded`; `owning_core: pf`, `category: platform`)
so downstream automation can subscribe — the runtime emitters land with the flag-on rollout, not in Phase 1.

***

## 5. Operating a run

All actions go through the governed RPCs (also surfaced as Cancel/Resume in the panel):

| Need              | RPC                                                                                      | Notes                                                 |
| ----------------- | ---------------------------------------------------------------------------------------- | ----------------------------------------------------- |
| Advance lifecycle | `pf_agent_run_advance(p_run_id, p_to_state, p_reason)`                                   | rejects illegal transitions and direct-write attempts |
| Checkpoint        | `pf_agent_run_checkpoint(p_run_id, p_seq, p_snapshot, p_cursor, p_retention_expires_at)` | snapshot + cursor for resume; honors retention expiry |
| Resume            | `pf_agent_run_resume(p_run_id)`                                                          | resumes a `paused` run from its latest checkpoint     |

* **Cancel a run:** Cancel in the panel, or `pf_agent_run_advance(<id>, 'cancelled', '<reason>')`.
  `cancelled` is terminal — the run cannot be resumed afterward.
* **Resume a paused run:** Resume in the panel, or `pf_agent_run_resume(<id>)`.
* **Budget exhaustion:** a run that hits its token/step budget advances itself to `budget_exceeded`
  (terminal) and the loop is not re-dispatched.

***

## 6. Emergency stop

1. **Stop new tool use immediately:** set `pf.agents.interactive_tools_enabled` → `false` for the
   affected org (or all orgs). New requests get zero tools on the next call.
2. **Halt in-flight runs:** cancel each non-terminal run via the panel or
   `pf_agent_run_advance(<id>, 'cancelled', 'operator emergency stop')`.
3. **Investigate:** read the `pf_agent_run_events` audit trail for the affected runs (expandable in the
   panel). Principals are attributed; correlate with `pf_agent_can` grants if a scope looks wrong.
4. **Never** bypass the state machine with a direct SQL `UPDATE` — the trigger rejects it, and doing so
   would break the audit trail. There is no service-role write path by design.

***

## 7. Invariants (do not violate)

* Agents act under a scoped principal via `pf_agent_can` — **never** `service_role` or `auth.uid()`
  impersonation.
* No PHI in prompts, logs, code, or tests. Redaction must be live before the flag is enabled.
* Tenant isolation: every run/task/checkpoint/event is `organization_id`-scoped under FORCE RLS.
* Lifecycle state changes go **only** through the governed RPCs; the BEFORE UPDATE trigger is the
  backstop.
* Flipping PF-45 on in production is a **human gate** — it requires the full spine, not just a green CI.

***

## References

* Spec: `specs/pf/PF-125-*` · Epic: #1779 · ADR: `docs/architecture/decisions/ADR-026-*`
* State machine: `supabase/functions/_shared/ai/agent-run-state-machine.ts`
* Guardrail spine (pure): `supabase/functions/_shared/ai/agent-spine.ts`
* Red-team suite: `tests/red-team/pf/pf-125-spine-bypass.red-team.test.ts`
* Panel: `src/platform/agents/ui/RunTimelinePage.tsx`
* Migrations: `supabase/migrations/20260630000001_pf125_multi_agent_orchestration.sql` (+ permission/flag seeds)