> ## Documentation Index
> Fetch the complete documentation index at: https://docs.encoreos.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Eval & Observability

> Internal PHI-free eval harness: L1 JSONL floor, L2 self-hosted Langfuse tracing/datasets/experiments, L3 Promptfoo gate — how to run them and the PHI guardrail.

Tooling for measuring the spec/automation workflow's output quality. **Scope: the
internal, PHI-free eval harness** — synthetic fixtures and skill/spec markdown, NOT
in-product PHI AI. (In-product LLM tracing is the separate PF-111 edge seam at
`supabase/functions/_shared/ai/`, which is metadata-only against cloud Langfuse —
do not confuse the two.)

## Three-layer ownership

| Layer                | Owns                                                                     | Where                                                                                                                    |
| -------------------- | ------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------ |
| **L1 — JSONL floor** | Dependency-free CLI telemetry (model, tokens, cost, latency per command) | `automation/_platform/ai-call.ts` appends to gitignored `.claude/tmp/eos-spec-events.jsonl`; `eos-spec metrics` reads it |
| **L2 — Langfuse**    | LLM-call traces + datasets/experiments/scores                            | self-hosted (`infra/langfuse/`), fed by OTEL on the `ai` v6 gateway calls                                                |
| **L3 — Promptfoo**   | Offline pre-merge rubric gate (informational)                            | `evals/promptfoo/` + `npm run eval:promptfoo` (landed separately)                                                        |

L1 always runs. L2 is opt-in (`LANGFUSE_TRACING_ENABLED=1` + keys). L3 is on-demand.

## L2 — self-hosted Langfuse

Stack: `langfuse-web` (host `127.0.0.1:3001`) + worker + postgres + clickhouse +
redis + minio, all on the internal compose network (no port collision with the
Supabase `543xx` stack). Distinct compose project `langfuse`.

```bash theme={null}
cp infra/langfuse/env.example infra/langfuse/.env   # then fill in real secrets
#   openssl rand -hex 32  → ENCRYPTION_KEY   |   openssl rand -hex 16 → passwords
npm run langfuse:up                                  # bring the stack up
curl -fsS http://localhost:3001/api/public/health    # {"status":"OK", ...}
npm run langfuse:down                                # stop it
```

The compose uses **headless init** (`LANGFUSE_INIT_*` in `.env`): on first boot it
provisions the org/project and sets the project API keys deterministically, so no
UI click-through is needed. Log into the UI at [http://localhost:3001](http://localhost:3001) with
`LANGFUSE_INIT_USER_EMAIL` / `LANGFUSE_INIT_USER_PASSWORD`.

### Tracing the automation AI calls

`callAiGateway` passes `experimental_telemetry`; `automation/_platform/otel.ts`
bootstraps the OTEL SDK with the `LangfuseSpanProcessor` when enabled. To trace a
run (e.g. the skills judge):

```bash theme={null}
export LANGFUSE_TRACING_ENABLED=1
export LANGFUSE_BASE_URL=http://localhost:3001   # see gotcha below
export LANGFUSE_PUBLIC_KEY=...   # = LANGFUSE_INIT_PROJECT_PUBLIC_KEY
export LANGFUSE_SECRET_KEY=...   # = LANGFUSE_INIT_PROJECT_SECRET_KEY
export AI_GATEWAY_API_KEY=...    # from the main checkout .env.local
npm run skills:judge
```

Traces appear as `judge:ai.generateText` with token usage in the Langfuse UI.

> **Gotcha:** the SDK reads **`LANGFUSE_BASE_URL`** (default `cloud.langfuse.com`).
> The main checkout's `.env.local` sets that to the *cloud* PF-111 instance; if you
> `source` it wholesale, the cloud value wins and exports your local keys to the
> wrong host (401 "Invalid credentials. Confirm the correct host"). Set
> `LANGFUSE_BASE_URL` explicitly to `http://localhost:3001`, and only pull
> `AI_GATEWAY_API_KEY` out of `.env.local`.

## L2 — datasets & experiments (C3 scorer)

`npm run eval:experiment` (`evals/langfuse/run-experiment.ts`) loads the two
reviewable spec fixtures as a Langfuse dataset (`spec-workflow-fixtures`), runs an
experiment whose **task** is the `claude -p` spec-review actor, and attaches an
**LLM-as-judge evaluator** graded through the AI Gateway. It prints a scorecard and
a dataset-run URL. Same env as above (plus the run needs the actor's local Claude
Code auth). Each actor call is \~\$0.35–0.50 / \~2 min.

## Skill-regression evals (Tier B)

`npm run eval:skills` (`evals/skills/run-skill-evals.ts`) regression-tests Agent
Skills. For each skill that ships `automation/skills/<name>/evals/evals.json`, it
sends every case's `prompt` to a fresh `claude -p` and grades (via the AI Gateway)
whether the skill's behaviour was applied. Cases are tagged `should_trigger:
true|false` so the gate catches BOTH **under-triggering** (skill should fire but
didn't) and **over-triggering** (fires on unrelated prompts). Informational /
non-blocking; `--strict` exits non-zero on any failed case. CI: `eval-skills.yml`.

`evals.json` schema:

```json theme={null}
{ "skill": "dead-code-audit",
  "cases": [
    { "id": "dc-trigger-process", "prompt": "…", "should_trigger": true,  "criteria": "applies the Fallow dead-code workflow…" },
    { "id": "dc-notrigger-feature", "prompt": "…", "should_trigger": false, "criteria": "applies the dead-code-audit workflow" }
  ] }
```

> **Finding (2026-06-04) — skills DO load in headless `claude -p`.** A stream-json
> spike confirmed `system:init` registers all skills and the response pulled
> **body-only** content (repo-specific `npm run deadcode` / Fallow / report paths
> that exist only in the SKILL.md body, not its \~100-token description). This
> reverses the earlier assumption (from the 2026-05-26 actor spike) that skill
> evals needed the Claude Agent SDK — they run on plain `claude -p`, no SDK. First
> dogfood: **4/4 cases** (code-review + dead-code-audit, trigger 0.90 / no-trigger 0.00).

## Verified \[2026-06]

Dogfooded against the self-hosted stack (Langfuse v3.178.0):

* **Spike B:** a single `generateText` call traced cleanly (`spikeB:ai.generateText`,
  model + token usage).
* **L2 wiring:** the real `callAiGateway` path emitted a `judge:ai.generateText`
  trace AND the L1 JSONL line (cost computed) — both layers from one call.
* **C3:** `eval:experiment` produced a dataset run with an average rubric score
  (0.80 across the two fixtures), viewable in the UI.

## AI integration evals (synthetic harness)

The following four suites live under `evals/datasets/` and extend the harness with
regression coverage for the AI integration expansion work
([design doc §4](../superpowers/specs/2026-06-20-ai-integration-expansion-design.md)).

### Suite inventory

| Suite file                             | Purpose                                                                                                                    | Gated?                                                                                   |
| -------------------------------------- | -------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------- |
| `predict-denial-risk-contract.test.ts` | Contract tests for the denial-risk inference function — asserts output range, determinism, and level-enum                  | `gated:true` (deterministic; CI-blocking)                                                |
| `pf-ai-skill-contract.test.ts`         | Golden rubric tests for governed AI skills (GR-06 compliance advisor + others); runs `invokeSkill` → asserts pass criteria | `gated:false` (report-only)                                                              |
| `pf-ai-skill-drift.test.ts`            | Snapshot drift guard — detects silent system-prompt changes to governed skills                                             | Vitest-native (not registered in `run-scorecard.ts`; skipped unless `RUN_SKILL_DRIFT=1`) |
| `ai-skill-injection.test.ts`           | Prompt-injection safety regression cases — verifies the AI gateway rejects injection payloads                              | `gated:false` (report-only)                                                              |

> **Note:** `pf-ai-skill-drift.test.ts` is a Vitest-native test (skipped unless `RUN_SKILL_DRIFT=1` is set), NOT a suite registered in `evals/run-scorecard.ts`. The gated/promotion path below does not apply to it.

### Promotion path

New suites start `gated:false` (advisory / report-only). To promote to `gated:true`
(CI-blocking):

1. **Run nightly** via the DGX workflow (`.github/workflows/eval-ai-skills-regression.yml`) for at
   least one week without flapping.
2. **Confirm stable** — pass rate at or above the suite's declared `passThreshold`
   across nightly runs; no environment-dependent failures.
3. **Flip** `gated: true` in the suite's entry in the `SUITES` array in
   `evals/run-scorecard.ts`.
4. Update this doc's table above.

Only `predict-denial-risk-contract` starts `gated:true` because it tests a
deterministic function with no LLM calls. All LLM-dependent suites start
`gated:false`.

### How to add a golden case

1. **Read an exemplar** — study `evals/datasets/gr-06-compliance-advisor.ts` for the
   dataset fixture pattern and `evals/datasets/pf-ai-skill-contract.ts` for how to
   wire a scorer.

2. **Author a synthetic case** — write a scenario using only synthetic data (no real
   org UUIDs, no PHI). Add it to the relevant `*-contract.ts` or `*-injection.ts`
   dataset file.

3. **PHI check** — run `npm run eval:fixtures:phi:check` to confirm your addition is
   clean.

4. **Run locally** — execute the suite directly:

   ```bash theme={null}
   npx vitest run evals/datasets/<suite>.test.ts
   ```

   On the GB10 box with the full AI stack, set `LOCAL_AI_BASE_URL` to point at the
   local OpenRouter-compatible gateway:

   ```bash theme={null}
   LOCAL_AI_BASE_URL=http://localhost:11434 npx vitest run evals/datasets/<suite>.test.ts
   ```

5. **Open a PR** — include test output showing the new case passes; reviewer confirms
   it is synthetic and PHI-free.

### METADATA-ONLY caveat and candidate sourcing

The PF-111 edge seam (`supabase/functions/_shared/ai/`) logs **metadata only** to
`pf_ai_usage_logs` — no prompt or response content is captured. This is by design for
PHI safety. As a result, **you cannot mine production traces for golden-case content**.

Instead, use the candidate finder to identify *which* skills have the highest failure
rates and latency, then **hand-author** synthetic golden cases that cover those failure
modes:

```bash theme={null}
# Requires: SUPABASE_URL and SUPABASE_EVAL_SERVICE_ROLE_KEY (service-role key)
SUPABASE_URL=<url> SUPABASE_EVAL_SERVICE_ROLE_KEY=<key> \
  npx tsx scripts/evals/find-eval-candidates.ts
```

This queries `pf_ai_usage_logs` for the past 30 days and prints a ranked table
(`skillCode`, `failures`, `total`, `p95ms`) so you can prioritize which skills need
new golden cases most urgently.

Real-trace content curation (capturing anonymized prompt/response pairs for eval
authoring) is the deferred content-capture item described in
[design doc §4](../superpowers/specs/2026-06-20-ai-integration-expansion-design.md).
It requires an explicit PHI-review before enabling content capture on the PF-111 lane.

## PHI guardrail

These tools are for **synthetic, PHI-free fixtures only**. Unlike the PF-111
in-product lane (metadata-only), this harness **captures prompt/completion content**
in traces — which is fine for skill/spec markdown and synthetic fixtures, and is
why **Langfuse trace storage is self-hosted** (traces stay on the box). Note this is
*storage* residency only: the model traffic itself (the `claude -p` actor and the
gateway LLM-judge) **still leaves the host** to the LLM provider, so the inputs must
remain synthetic and PHI-free regardless. Never pipe real specs containing PHI, real
org UUIDs, or patient data through these tools without a scrub (see
`.claude/rules/security.md`). If a PHI-bearing eval is ever needed, disable content
capture and treat it like the PF-111 metadata-only lane.