> ## Documentation Index
> Fetch the complete documentation index at: https://docs.encoreos.io/llms.txt
> Use this file to discover all available pages before exploring further.
# FW Operating Runbook
> On-call runbook for workflow automation: observability dashboards, dead-letter queue recovery, replay, and incident response.
**Companion docs:**
* [Automation Observability (admin)](/fw/automations)
* [Dead Letter Queue (admin)](/fw/dead-letter-queue)
* [Workflow Execution Replay (admin)](/fw/forms-workflows)
* [Audit & Compliance Reporting (admin)](/fw/audit-compliance)
* [FW Developer Reference](/fw/durable-execution-worker-api-reference)
***
## Quick triage table
| Symptom | First page to open | Likely cause | First action |
| ------------------------------------------------------------- | ---------------------------------------------------------------------- | -------------------------------------- | ------------------------------------------------- |
| Workflow executions stuck "in flight" for hours | [Automation Observability](/fw/automations) | Worker crashed or saturated | Confirm worker healthy; if not, restart |
| DLQ counter rising fast | [DLQ admin guide](/fw/dead-letter-queue) | External dependency outage | Triage by error class; throttle source |
| `fw_workflow_audit_events` hash mismatch (FW-62 once shipped) | [Audit Trail (FW-43)](/fw/audit-compliance) | DBA-level tamper or migration drift | Pause writers; security review |
| Inbound webhook 5xx flood | [External Webhooks (Inbound)](/fw/inbound-webhooks) | Source mis-configured | Lower endpoint rate limit; coordinate with source |
| KPI snapshot lag (no new snapshots) | [KPI Dashboards & Snapshots](/fw/workflow-kpi-dashboard) | Snapshot cron stuck (deferred edge fn) | Run ad-hoc; track FW-58 deferred items |
| Rate-limit saturation across many workflows | [Rate Limiting & Throttling](/fw/rate-limits) | Org-wide spike or runaway workflow | Identify workflow; pause / throttle |
| Approval chain stuck without escalation | [Approval Chains](/fw/approval-chains) | Misconfigured SLA or bad assignee | Reassign; fix chain |
| Execution timeout cascade | [Retry & Circuit Breakers](/fw/durable-execution-worker-api-reference) | External dependency timeout | Open circuit; address dependency |
| Prefill engine returning wrong values | [Form Prefill & Smart Defaults](/fw/all-forms) | Allowlist / mapping bug | Disable rule; rebuild from sample data |
***
## Runbook 1 — Worker stuck (FW-46)
### Symptoms
* Queue depth growing in [Automation Observability](/fw/automations).
* New executions don't progress past "queued".
* p95 wait time spiking.
### Diagnosis
1. Open Automation Observability → switch to **Current** view.
2. Confirm queue depth + zero throughput.
3. Check the platform's edge-function dashboard (Supabase) for `workflow-executor-worker` — is it running? Crashing? Out-of-memory?
4. Check `fw_module_settings.fw_worker_concurrency_ceiling` per org. Is it 0? (Misconfig.)
5. If FW-61 has shipped (planned), open the worker capacity dashboard for saturation telemetry.
### Recovery
1. If the worker has crashed: re-deploy the edge function. Pre-FW-46-EN-02 (planned) the redeploy will lose any in-flight pgmq leases — the watchdog (FW-49) will time them out and re-enqueue.
2. If it's a config issue: raise `fw_worker_concurrency_ceiling` for the affected org via `fw_module_settings`.
3. If it's an external-dependency timeout: see Runbook 5.
4. Confirm queue drains in Automation Observability.
### Postmortem actions
* Capture timeline + what you changed.
* File an enhancement entry under [`FW-46-ENHANCEMENTS.md`](https://github.com/Encore-OS/encoreos/blob/development/specs/fw/specs/FW-46-ENHANCEMENTS.md) if the issue is structural.
* Update this runbook with the specific fix.
***
## Runbook 2 — DLQ growing (FW-47)
### Symptoms
* DLQ counter rising in observability.
* Workflow owners receiving DLQ notifications.
### Diagnosis
1. Open the [DLQ admin page](/fw/dead-letter-queue).
2. Filter by error class (`transient` / `policy` / `data` / `external`).
3. For `external` failures, identify the dependency and its status.
### Recovery
1. **Confirm affected organization\_id** and ensure all commands include an `organization_id` filter so changes apply to that org only.
2. **External dependency restored** → bulk retry the affected entries (scoped to the confirmed organization\_id).
3. **Bad data at source** → ask the source owner to fix; bulk discard impossible-to-fix entries (scoped to the confirmed organization\_id).
4. **Misconfigured workflow** → disable the workflow until fixed (scoped to the confirmed organization\_id).
5. **Permission policy changed** → grant the needed permission OR discard entries (scoped to the confirmed organization\_id).
### Postmortem
* Tighten DLQ retention if the queue is overflowing the dashboard.
* Set up a saved filter for the error class to speed future triage.
* Consider adding [Rate Limiting](/fw/rate-limits) on the source.
***
## Runbook 3 — Audit chain break (FW-43; FW-62 once shipped)
### Symptoms (FW-62 once shipped)
* Audit integrity verifier reports a hash mismatch.
* Compliance officer can't reconcile a date range.
### Symptoms (today, pre-FW-62)
* Suspected tampering / accidental privileged write.
* Audit row count anomaly.
### Diagnosis
1. Confirm the suspected range in [Audit & Compliance Reporting](/fw/audit-compliance).
2. Pre-FW-62: cross-reference application logs vs DB row count for the range.
3. Post-FW-62: open Audit & Compliance > Integrity tab, validate the range, identify the broken hash chain link.
### Recovery
1. **Confirm affected organization\_id** and ensure all pause/restore operations are scoped to that org only.
2. **Pause writers** to the affected tables (scoped to the confirmed organization\_id).
3. **Engage security review.** Determine: privilege misuse, migration error, or true tampering?
4. **Restore** from PF-11 cold-storage archive if available for the range (scoped to the confirmed organization\_id).
5. **Document** in incident ledger; report per HIPAA breach-notification rules if applicable.
### Postmortem
* Accelerate FW-62 implementation if not already in progress.
* Audit who has `service_role` access; rotate.
* File security-auditor pre-flight against `fw_workflow_audit_events`.
***
## Runbook 4 — Webhook flood (FW-59)
### Symptoms
* A specific webhook endpoint receiving 100s+ requests / minute.
* 429 responses in the endpoint logs (good — rate limit working).
* Or downstream workflow saturated (rate limit too loose).
### Diagnosis
1. Open the endpoint's **Logs** tab.
2. Confirm source IP / API key.
3. Confirm rate-limit state.
### Recovery
1. **Confirm affected organization\_id** and ensure all throttle/disable operations are scoped to that org only.
2. **Lower endpoint rate limit** ([Rate Limiting & Throttling](/fw/rate-limits)) to throttle (scoped to the confirmed organization\_id).
3. **Coordinate with source** — they should implement exponential backoff on 429.
4. **Disable the endpoint** if the source is uncoordinated and PHI-touching (scoped to the confirmed organization\_id).
5. **Rotate the secret** if the flood looks malicious (scoped to the confirmed organization\_id).
### Postmortem
* Add IP allow-list if the source has stable IPs.
* Document the partner's accepted rate in `docs/architecture/integrations/`.
* Consider FW-53 budget changes platform-wide.
***
## Runbook 5 — Execution timeout cascade (FW-49)
### Symptoms
* Many executions hitting `timed_out` status.
* Watchdog auto-cancellation events flooding audit log.
### Diagnosis
1. Open [Automation Observability](/fw/automations), filter status = `timed_out`.
2. Identify common workflow / step.
3. Inspect external dependencies the step calls.
### Recovery
1. **Confirm affected organization\_id** and ensure all circuit breaker/retry operations are scoped to that org only.
2. **Open circuit breaker manually** ([Retry & Circuit Breakers](/fw/durable-execution-worker-api-reference)) on the affected workflow node (scoped to the confirmed organization\_id) — fast-fails subsequent calls so they go to DLQ instead of timing out.
3. **Address the dependency** (restart, scale, etc.).
4. **Reset circuits** when dependency is healthy (scoped to the confirmed organization\_id).
5. **Bulk retry** from DLQ (scoped to the confirmed organization\_id).
### Postmortem
* Tune workflow timeout per node based on observed dependency p95.
* Consider compensation actions for partial-state recovery.
***
## Runbook 6 — KPI snapshot lag (FW-58)
### Symptoms
* KPI dashboard shows stale values.
* Trend charts have gaps.
### Diagnosis
1. The scheduled-PDF edge function and snapshot cron are **deferred** in the current FW-58 release. Manual ad-hoc render is the only dispatch path today.
2. Confirm `fw_kpi_snapshots` rows for the period via DB.
3. If snapshot cron has shipped per FW-58 follow-up, check edge function health.
### Recovery
1. Run ad-hoc snapshot from KPI > [KPI page](/fw/workflow-kpi-dashboard) > Refresh.
2. For PDF dispatch: render ad-hoc and email out-of-band until FW-58 deferred items ship.
***
## Runbook 7 — Rate-limit saturation across the org
### Symptoms
* Multiple workflows hitting rate limits.
* Stats page shows widespread throttling.
### Diagnosis
1. Open [Rate Limiting > Stats](/fw/rate-limits).
2. Identify the top-throttled workflow (single bad actor likely).
3. Cross-check with [Automation Observability](/fw/automations) — is one workflow consuming > 50% of throughput?
### Recovery
1. **Confirm affected organization\_id** and ensure all disable/tune operations are scoped to that org only.
2. **Disable the runaway workflow** until investigated (scoped to the confirmed organization\_id).
3. **Tune the per-org limit** if legitimate growth (raise carefully; scoped to the confirmed organization\_id).
4. **Add a per-workflow limit** on the noisy workflow even after it's restored (scoped to the confirmed organization\_id).
***
## Runbook 8 — Approval queue overload (FW-34)
### Symptoms
* Approval inbox showing 100s of pending items for a single approver.
* SLA breach alerts piling up.
### Diagnosis
1. Confirm the approver is real and authenticated.
2. Confirm the routing rule isn't broken (FW-54 → all routes lead to one user).
3. Confirm the approver hasn't [delegated](/fw/delegations) (delegation should redistribute).
### Recovery
1. **Bulk reassign** to peers (forms\_admin can do this).
2. **Set up role-based assignment** in the chain so workload distributes.
3. **Configure escalation** so SLA breaches auto-route.
***
## Runbook 9 — Prefill engine failure (FW-60)
### Symptoms
* Forms loading with no prefilled values when they should.
* Or prefilled with wrong values.
### Diagnosis
1. Open the form's **Prefill** tab.
2. For each rule, confirm the entity record exists / URL parameter present.
3. Check rule priority (first-win).
4. Test in Preview with the same context.
### Recovery
1. **Disable the bad rule** to stop blast radius.
2. **Fix the mapping** (entity field, JSONPath, context key).
3. **Re-enable**.
4. If the issue is in the allowlist: file a request with platform team to extend / fix.
***
## When to escalate
* Suspected tampering / breach → security review (Runbook 3).
* Worker outage that affects > 5 orgs simultaneously → platform team.
* Persistent regression after a FW PR → revert + escalate to FW core lead.
* Compliance / regulatory deadline at risk → compliance officer.
## Updating this runbook
* After every significant incident, add a new runbook section or update an existing one.
* Cross-link to the admin / developer / compliance docs in the Mintlify `docs/fw/` surface.
* Bump the version.
* Reviewed quarterly.