Skip to main content
Tooling for measuring the spec/automation workflow’s output quality. Scope: the internal, PHI-free eval harness — synthetic fixtures and skill/spec markdown, NOT in-product PHI AI. (In-product LLM tracing is the separate PF-111 edge seam at supabase/functions/_shared/ai/, which is metadata-only against cloud Langfuse — do not confuse the two.)

​
Three-layer ownership

L1 always runs. L2 is opt-in (LANGFUSE_TRACING_ENABLED=1 + keys). L3 is on-demand.

​
L2 — self-hosted Langfuse

Stack: langfuse-web (host 127.0.0.1:3001) + worker + postgres + clickhouse + redis + minio, all on the internal compose network (no port collision with the Supabase 543xx stack). Distinct compose project langfuse.
The compose uses headless init (LANGFUSE_INIT_* in .env): on first boot it provisions the org/project and sets the project API keys deterministically, so no UI click-through is needed. Log into the UI at http://localhost:3001 with LANGFUSE_INIT_USER_EMAIL / LANGFUSE_INIT_USER_PASSWORD.

​
Tracing the automation AI calls

callAiGateway passes experimental_telemetry; automation/_platform/otel.ts bootstraps the OTEL SDK with the LangfuseSpanProcessor when enabled. To trace a run (e.g. the skills judge):
Traces appear as judge:ai.generateText with token usage in the Langfuse UI.
Gotcha: the SDK reads LANGFUSE_BASE_URL (default cloud.langfuse.com). The main checkout’s .env.local sets that to the cloud PF-111 instance; if you source it wholesale, the cloud value wins and exports your local keys to the wrong host (401 “Invalid credentials. Confirm the correct host”). Set LANGFUSE_BASE_URL explicitly to http://localhost:3001, and only pull AI_GATEWAY_API_KEY out of .env.local.

​
L2 — datasets & experiments (C3 scorer)

npm run eval:experiment (evals/langfuse/run-experiment.ts) loads the two reviewable spec fixtures as a Langfuse dataset (spec-workflow-fixtures), runs an experiment whose task is the claude -p spec-review actor, and attaches an LLM-as-judge evaluator graded through the AI Gateway. It prints a scorecard and a dataset-run URL. Same env as above (plus the run needs the actor’s local Claude Code auth). Each actor call is ~$0.35–0.50 / ~2 min.

​
Skill-regression evals (Tier B)

npm run eval:skills (evals/skills/run-skill-evals.ts) regression-tests Agent Skills. For each skill that ships automation/skills/<name>/evals/evals.json, it sends every case’s prompt to a fresh claude -p and grades (via the AI Gateway) whether the skill’s behaviour was applied. Cases are tagged should_trigger: true|false so the gate catches BOTH under-triggering (skill should fire but didn’t) and over-triggering (fires on unrelated prompts). Informational / non-blocking; --strict exits non-zero on any failed case. CI: eval-skills.yml. evals.json schema:
Finding (2026-06-04) — skills DO load in headless claude -p. A stream-json spike confirmed system:init registers all skills and the response pulled body-only content (repo-specific npm run deadcode / Fallow / report paths that exist only in the SKILL.md body, not its ~100-token description). This reverses the earlier assumption (from the 2026-05-26 actor spike) that skill evals needed the Claude Agent SDK — they run on plain claude -p, no SDK. First dogfood: 4/4 cases (code-review + dead-code-audit, trigger 0.90 / no-trigger 0.00).

​
Verified [2026-06]

Dogfooded against the self-hosted stack (Langfuse v3.178.0):
  • Spike B: a single generateText call traced cleanly (spikeB:ai.generateText, model + token usage).
  • L2 wiring: the real callAiGateway path emitted a judge:ai.generateText trace AND the L1 JSONL line (cost computed) — both layers from one call.
  • C3: eval:experiment produced a dataset run with an average rubric score (0.80 across the two fixtures), viewable in the UI.

​
AI integration evals (synthetic harness)

The following four suites live under evals/datasets/ and extend the harness with regression coverage for the AI integration expansion work (design doc §4).

​
Suite inventory

Note: pf-ai-skill-drift.test.ts is a Vitest-native test (skipped unless RUN_SKILL_DRIFT=1 is set), NOT a suite registered in evals/run-scorecard.ts. The gated/promotion path below does not apply to it.

​
Promotion path

New suites start gated:false (advisory / report-only). To promote to gated:true (CI-blocking):
  1. Run nightly via the DGX workflow (.github/workflows/eval-ai-skills-regression.yml) for at least one week without flapping.
  2. Confirm stable — pass rate at or above the suite’s declared passThreshold across nightly runs; no environment-dependent failures.
  3. Flip gated: true in the suite’s entry in the SUITES array in evals/run-scorecard.ts.
  4. Update this doc’s table above.
Only predict-denial-risk-contract starts gated:true because it tests a deterministic function with no LLM calls. All LLM-dependent suites start gated:false.

​
How to add a golden case

  1. Read an exemplar — study evals/datasets/gr-06-compliance-advisor.ts for the dataset fixture pattern and evals/datasets/pf-ai-skill-contract.ts for how to wire a scorer.
  2. Author a synthetic case — write a scenario using only synthetic data (no real org UUIDs, no PHI). Add it to the relevant *-contract.ts or *-injection.ts dataset file.
  3. PHI check — run npm run eval:fixtures:phi:check to confirm your addition is clean.
  4. Run locally — execute the suite directly:
    On the GB10 box with the full AI stack, set LOCAL_AI_BASE_URL to point at the local OpenRouter-compatible gateway:
  5. Open a PR — include test output showing the new case passes; reviewer confirms it is synthetic and PHI-free.

​
METADATA-ONLY caveat and candidate sourcing

The PF-111 edge seam (supabase/functions/_shared/ai/) logs metadata only to pf_ai_usage_logs — no prompt or response content is captured. This is by design for PHI safety. As a result, you cannot mine production traces for golden-case content. Instead, use the candidate finder to identify which skills have the highest failure rates and latency, then hand-author synthetic golden cases that cover those failure modes:
This queries pf_ai_usage_logs for the past 30 days and prints a ranked table (skillCode, failures, total, p95ms) so you can prioritize which skills need new golden cases most urgently. Real-trace content curation (capturing anonymized prompt/response pairs for eval authoring) is the deferred content-capture item described in design doc §4. It requires an explicit PHI-review before enabling content capture on the PF-111 lane.

​
PHI guardrail

These tools are for synthetic, PHI-free fixtures only. Unlike the PF-111 in-product lane (metadata-only), this harness captures prompt/completion content in traces — which is fine for skill/spec markdown and synthetic fixtures, and is why Langfuse trace storage is self-hosted (traces stay on the box). Note this is storage residency only: the model traffic itself (the claude -p actor and the gateway LLM-judge) still leaves the host to the LLM provider, so the inputs must remain synthetic and PHI-free regardless. Never pipe real specs containing PHI, real org UUIDs, or patient data through these tools without a scrub (see .claude/rules/security.md). If a PHI-bearing eval is ever needed, disable content capture and treat it like the PF-111 metadata-only lane.