> ## Documentation Index > Fetch the complete documentation index at: https://docs.encoreos.io/llms.txt > Use this file to discover all available pages before exploring further. # ADR-023: Agent Harness, Skills & Eval-Tooling Direction > An external research report ("Agent Harness, Skills/Commands/CLI, and Evaluation Framework for Encore — A Recommendation", 2026-06) recommended: **Status:** Accepted (directional) **Date:** 2026-06-04 **Decision Makers:** Jeremy Bloom (project owner) > **See also [ADR-026](/architecture/decisions/ADR-026-langchain-ecosystem-and-agent-os)** — > extends this disposition to the LangChain ecosystem (LangChain / LangGraph / LangSmith) > and Agent OS by name. *** ## Context An external research report ("Agent Harness, Skills/Commands/CLI, and Evaluation Framework for Encore — A Recommendation", 2026-06) recommended: 1. **Mastra** as the standard production agent-orchestration harness (TS-native, on the Vercel AI SDK), with the **Claude Agent SDK** for headless/CI automation and the **Vercel AI SDK** as the streaming layer. 2. Consolidating the Claude Code dev workflow around the unified skill model; a **thin CLAUDE.md that `@AGENTS.md`-imports** a canonical AGENTS.md. 3. A **two-tier, CI-integrated eval setup**: product-agent evals (Evalite / vitest-evals + autoevals) and **skill-regression evals** (Anthropic `skill-creator` + the community Skill Eval GitHub Action). The report is well-sourced and its core framing — that the *production harness* question and the *dev-workflow* question are distinct problems — is sound. However, it was written with an imperfect picture of our stack, which changes several conclusions. This ADR records our grounded position so the research is actionable for the team. ### Stack corrections (the report assumed otherwise) | Report assumed | Encore reality | | ------------------------------------------ | ---------------------------------------------------------------------------------------------------- | | "Bun/Turbo/Vitest" runtime | **Node 24 + npm** + Turbo + Vitest (no Bun; its Bun caveats are moot) | | About to ship product agents | **No in-product agent exists** — zero `mastra` / `@anthropic-ai/claude-agent-sdk` / `langgraph` deps | | Recommend `@AGENTS.md` import in CLAUDE.md | We **deliberately rejected** this (see Decision 3) | | Eval tooling greenfield | **Already shipped** Promptfoo C4 + self-hosted Langfuse C3 (2026-06-04) | ## Decision ### 1. Production agent harness (Mastra / Claude Agent SDK) — DEFERRED We do **not** adopt Mastra (or any production agent-orchestration framework) now. A harness orchestrates *product agents*, and Encore ships none today. Adopting one with no agent to host is pure carrying cost. Additionally, Mastra holds **no SOC 2 certification** as of early 2026 — a non-starter for a healthcare-grade platform without further diligence. Revisit **only** when a concrete, funded product-agent feature exists. At that point the report's analysis is a good starting shortlist (Mastra for TS-native orchestration on our Vercel/Supabase stack; Claude Agent SDK for headless CI automation that reuses our skills/CLAUDE.md). The **Vercel AI SDK (`ai@6`)** is already our model/gateway/streaming layer (PF-111), so that part of the recommendation is effectively already in place. ### 2. Eval tooling — PARTIALLY DONE; close the skill-regression gap The report's eval recommendations map onto work we **already shipped** on 2026-06-04 (see the internal eval-observability tooling plan and [EVAL\_OBSERVABILITY.md](/development/EVAL_OBSERVABILITY)): * **L3 / Promptfoo C4** — non-blocking spec-workflow rubric gate (PR #814). * **L2 / self-hosted Langfuse C3** — traces + datasets + experiments (PR #815). * **L1 / JSONL floor** — `eos-spec metrics` (pre-existing). The genuine **gap** the report surfaces is **skill-regression evals** (its "Tier B"): we have `automation/_platform/judge.ts` (a rubric *scorer* for skills) but nothing that tests *trigger behaviour* (does a skill fire when it should?) or *skill-vs-no-skill* lift. **Accepted as a follow-up:** author `skill-creator` `evals/evals.json` for our 3–5 highest-value skills and gate them in CI. This complements — does not overlap — the harness already shipped. **Promptfoo acquisition watch:** OpenAI announced acquisition of Promptfoo (2026-03-09 per the report). Our usage is structurally insulated — OSS CLI only, grader routed through our own AI Gateway, non-blocking, and we already self-host Langfuse and emit OTEL. No action required; keep the portable-instrumentation hedge. ### 3. Context files (CLAUDE.md / AGENTS.md) — KEEP CURRENT POSTURE We **reject** the report's `@AGENTS.md`-import recommendation. CLAUDE.md already documents the deliberate reason: auto-importing AGENTS.md would push CLAUDE.md to \~3× its 200-line target and *reduce* adherence; Claude reads AGENTS.md on demand instead. Our current posture already matches the report's *intent* (lean always-loaded context): CLAUDE.md is \~150 lines, AGENTS.md \~311, with 15 per-core `AGENTS.md` (nearest-wins) and 7 path-scoped lazy-loading `rules/*.md`. The report's strongest caution — the **ETH Zurich study** (arXiv:2602.11988, Feb 2026: bloated/redundant context files *reduce* task success and raise cost >20%) — is noted. `constitution.md` is \~2,222 lines; it is on-demand (not always-loaded), so less acute, but a redundancy audit against code/linters/rules is a reasonable future hygiene task. **No structural change now.** ### 4. Command → skill consolidation — BACKLOG (triage, not big-bang) We have **85 commands vs 40 skills**. The report's guidance (migrate logic-bearing commands into progressively-disclosed skills; keep prompt-template commands as commands) is sound. **Accepted as a triage backlog item**, not a big-bang migration. ### 5. Claude Code OTEL telemetry → self-hosted Langfuse — ACCEPTED (do next) Highest ROI / lowest effort item the report surfaces, and not yet configured. Enable `CLAUDE_CODE_ENABLE_TELEMETRY` routed to the Langfuse instance we just stood up, to capture dev-workflow token/cost/tool-decision telemetry per `agent.name`/`skill.name` — the report's "data flywheel". Dogfoods Phase 2 directly. ## Consequences **Do next (accepted):** (a) Claude Code OTEL → self-hosted Langfuse; (b) skill-regression evals for the top 3–5 skills. **Backlog:** command→skill consolidation triage; context-file redundancy audit (constitution.md). **Explicitly deferred:** Mastra and any product agent-orchestration harness; Claude Agent SDK headless automation; Evalite/vitest-evals product-agent eval tier — all gated on a real product-agent feature existing. **Rejected:** `@AGENTS.md` import into CLAUDE.md (conflicts with a documented local decision). ### Recommendation disposition (at a glance) | Report recommendation | Disposition | | ----------------------------------- | -------------------------------------------------- | | Mastra as production harness | **Deferred** (no product agent; no SOC 2) | | Claude Agent SDK (headless CI) | **Deferred** (revisit with product agents) | | Vercel AI SDK as model/stream layer | **Already adopted** (`ai@6`, PF-111) | | Thin CLAUDE.md `@AGENTS.md` import | **Rejected** (documented local decision) | | Lean context files | **Already in place**; constitution audit = backlog | | Promptfoo + Langfuse evals | **Already shipped** (#814 / #815) | | Skill-regression evals (Tier B) | **Accepted** (follow-up) | | Claude Code OTEL flywheel | **Accepted** (do next) | | Command→skill consolidation | **Backlog** (triage) | ## Follow-through findings (2026-06-04, dogfood-grounded) Before implementing the two "accepted next" items we tried them. The results revise the dispositions above — and surface one shared root cause. **Root cause across #1 and #2: the report's dev-workflow recommendations assume Claude Code's *full runtime* (interactive session or the Claude Agent SDK), but our *verifiable automation* runs headless `claude -p`, where neither feature fires.** * **#1 Claude Code OTEL → Langfuse — DOES NOT DELIVER (headless). Status → Deferred.** With `CLAUDE_CODE_ENABLE_TELEMETRY=1` + the documented OTLP env vars (`OTEL_TRACES_EXPORTER=otlp`, endpoint = our Langfuse `…/api/public/otel`, Basic auth), `claude -p` (v2.1.162) exported **nothing**: zero OTLP POST reached Langfuse, and `OTEL_LOG_LEVEL=debug` produced no telemetry output. We did **not** ship a config that looks right but produces nothing (constitution §3.1 — prefer dogfood over artifact-existence). Note Langfuse is a GenAI *trace* store anyway; Claude Code telemetry is metrics/events (a metrics backend like SigNoz/Prometheus is the right target). Our **LLM-call** observability is already covered by the gateway seam (`automation/_platform/otel.ts` + PF-111). Revisit only via an interactive session or the Agent SDK, against a metrics backend. * **#2 Skill-regression evals — BLOCKED on headless skill dispatch. Status → Spike first (via Agent SDK), do not build on `claude -p`.** The Anthropic `skill-creator` framework needs the Skill tool to actually dispatch. Our prior dogfood (`evals/SPIKE_FINDINGS.md`, 2026-05-26) established that headless `claude -p "use the X skill"` reads the SKILL.md as a *document* — the Skill tool is not dispatched — so trigger-rate / skill-vs-no-skill testing is not faithful in headless CI. The report's own bridge (Claude Agent SDK with `settingSources:['project']`) is the right substrate; **spike that it dispatches skills before building the eval harness.** This ties #2 to the (deferred) Agent SDK item rather than to the promptfoo/headless lane. * **#3 Command→skill consolidation — DO NOT mass-migrate. One real defect found.** A deterministic classifier (lines + numbered-steps + headers + script-refs) scored **43 of 85** commands as "logic-heavy", but **size ≠ skill**: the real axis is *triggering* (explicit user-invocation = command; auto-apply-on-context = skill). A long, deliberately-invoked `/validate-spec` is correctly a command. So mass migration is rejected (churn + loss of explicit-invocation semantics). The genuine defect: **`pre-commit-check` exists as BOTH a command (133 lines) and a skill (60 lines)** — the exact "if both exist, the skill wins" collision the report warns of, so the command is shadowed/dead (and the skill's `related-skills` self-references `pre-commit-check`). **Action item:** resolve via the registry deprecation flow (as #810 did for agents) — not a blind delete. Otherwise: keep current command/skill split. * **#4 Context-file bloat — audited; no change now.** `constitution.md` is **2,222 lines** but **on-demand** (referenced, not `@`-imported — the ETH Zurich always-loaded-bloat risk is mitigated). Concentration: **§5 Database/Naming/Env (\~783 lines) + §6 PWA (\~483 lines) = \~57%** of the file; §5 overlaps `.claude/rules/database.md`. **Extraction candidate** (move §5/§6 detail into path-scoped `rules/*.md`, leave authoritative summaries) — backlog, owner-gated, not done here (trimming a 2,222-line authority doc needs careful review). ### Revised disposition | Item | Was | Now | | --------------------------- | -------------------- | ---------------------------------------------------------------------------------- | | Claude Code OTEL → Langfuse | Accepted (do next) | **Deferred** — no headless export; wrong backend for metrics | | Skill-regression evals | Accepted (follow-up) | **Spike-first** — needs Agent SDK skill dispatch, not headless `claude -p` | | Command→skill consolidation | Backlog (triage) | **Triaged: do NOT mass-migrate**; fix `pre-commit-check` collision (registry flow) | | Context-file de-bloat | Backlog | **Audited**; §5/§6 extraction candidate, owner-gated | ## References * Eval-observability plan: `docs/superpowers/plans/2026-06-02-eval-observability-tooling.md` * Operator guide: `docs/development/EVAL_OBSERVABILITY.md` * Headless actor limits: `evals/SPIKE_FINDINGS.md` (2026-05-26) * ETH Zurich, "Evaluating AGENTS.md…", arXiv:2602.11988 (Feb 2026)