Skip to main content
Status: Accepted (directional) Date: 2026-06-04 Decision Makers: Jeremy Bloom (project owner)
See also ADR-026 — extends this disposition to the LangChain ecosystem (LangChain / LangGraph / LangSmith) and Agent OS by name.

​
Context

An external research report (“Agent Harness, Skills/Commands/CLI, and Evaluation Framework for Encore — A Recommendation”, 2026-06) recommended:
  1. Mastra as the standard production agent-orchestration harness (TS-native, on the Vercel AI SDK), with the Claude Agent SDK for headless/CI automation and the Vercel AI SDK as the streaming layer.
  2. Consolidating the Claude Code dev workflow around the unified skill model; a thin CLAUDE.md that @AGENTS.md-imports a canonical AGENTS.md.
  3. A two-tier, CI-integrated eval setup: product-agent evals (Evalite / vitest-evals + autoevals) and skill-regression evals (Anthropic skill-creator + the community Skill Eval GitHub Action).
The report is well-sourced and its core framing — that the production harness question and the dev-workflow question are distinct problems — is sound. However, it was written with an imperfect picture of our stack, which changes several conclusions. This ADR records our grounded position so the research is actionable for the team.

​
Stack corrections (the report assumed otherwise)

​
Decision

​
1. Production agent harness (Mastra / Claude Agent SDK) — DEFERRED

We do not adopt Mastra (or any production agent-orchestration framework) now. A harness orchestrates product agents, and Encore ships none today. Adopting one with no agent to host is pure carrying cost. Additionally, Mastra holds no SOC 2 certification as of early 2026 — a non-starter for a healthcare-grade platform without further diligence. Revisit only when a concrete, funded product-agent feature exists. At that point the report’s analysis is a good starting shortlist (Mastra for TS-native orchestration on our Vercel/Supabase stack; Claude Agent SDK for headless CI automation that reuses our skills/CLAUDE.md). The Vercel AI SDK (ai@6) is already our model/gateway/streaming layer (PF-111), so that part of the recommendation is effectively already in place.

​
2. Eval tooling — PARTIALLY DONE; close the skill-regression gap

The report’s eval recommendations map onto work we already shipped on 2026-06-04 (see the internal eval-observability tooling plan and EVAL_OBSERVABILITY.md):
  • L3 / Promptfoo C4 — non-blocking spec-workflow rubric gate (PR #814).
  • L2 / self-hosted Langfuse C3 — traces + datasets + experiments (PR #815).
  • L1 / JSONL floor — eos-spec metrics (pre-existing).
The genuine gap the report surfaces is skill-regression evals (its “Tier B”): we have automation/_platform/judge.ts (a rubric scorer for skills) but nothing that tests trigger behaviour (does a skill fire when it should?) or skill-vs-no-skill lift. Accepted as a follow-up: author skill-creator evals/evals.json for our 3–5 highest-value skills and gate them in CI. This complements — does not overlap — the harness already shipped. Promptfoo acquisition watch: OpenAI announced acquisition of Promptfoo (2026-03-09 per the report). Our usage is structurally insulated — OSS CLI only, grader routed through our own AI Gateway, non-blocking, and we already self-host Langfuse and emit OTEL. No action required; keep the portable-instrumentation hedge.

​
3. Context files (CLAUDE.md / AGENTS.md) — KEEP CURRENT POSTURE

We reject the report’s @AGENTS.md-import recommendation. CLAUDE.md already documents the deliberate reason: auto-importing AGENTS.md would push CLAUDE.md to ~3× its 200-line target and reduce adherence; Claude reads AGENTS.md on demand instead. Our current posture already matches the report’s intent (lean always-loaded context): CLAUDE.md is ~150 lines, AGENTS.md ~311, with 15 per-core AGENTS.md (nearest-wins) and 7 path-scoped lazy-loading rules/*.md. The report’s strongest caution — the ETH Zurich study (arXiv:2602.11988, Feb 2026: bloated/redundant context files reduce task success and raise cost >20%) — is noted. constitution.md is ~2,222 lines; it is on-demand (not always-loaded), so less acute, but a redundancy audit against code/linters/rules is a reasonable future hygiene task. No structural change now.

​
4. Command → skill consolidation — BACKLOG (triage, not big-bang)

We have 85 commands vs 40 skills. The report’s guidance (migrate logic-bearing commands into progressively-disclosed skills; keep prompt-template commands as commands) is sound. Accepted as a triage backlog item, not a big-bang migration.

​
5. Claude Code OTEL telemetry → self-hosted Langfuse — ACCEPTED (do next)

Highest ROI / lowest effort item the report surfaces, and not yet configured. Enable CLAUDE_CODE_ENABLE_TELEMETRY routed to the Langfuse instance we just stood up, to capture dev-workflow token/cost/tool-decision telemetry per agent.name/skill.name — the report’s “data flywheel”. Dogfoods Phase 2 directly.

​
Consequences

Do next (accepted): (a) Claude Code OTEL → self-hosted Langfuse; (b) skill-regression evals for the top 3–5 skills. Backlog: command→skill consolidation triage; context-file redundancy audit (constitution.md). Explicitly deferred: Mastra and any product agent-orchestration harness; Claude Agent SDK headless automation; Evalite/vitest-evals product-agent eval tier — all gated on a real product-agent feature existing. Rejected: @AGENTS.md import into CLAUDE.md (conflicts with a documented local decision).

​
Recommendation disposition (at a glance)

​
Follow-through findings (2026-06-04, dogfood-grounded)

Before implementing the two “accepted next” items we tried them. The results revise the dispositions above — and surface one shared root cause. Root cause across #1 and #2: the report’s dev-workflow recommendations assume Claude Code’s full runtime (interactive session or the Claude Agent SDK), but our verifiable automation runs headless claude -p, where neither feature fires.
  • #1 Claude Code OTEL → Langfuse — DOES NOT DELIVER (headless). Status → Deferred. With CLAUDE_CODE_ENABLE_TELEMETRY=1 + the documented OTLP env vars (OTEL_TRACES_EXPORTER=otlp, endpoint = our Langfuse …/api/public/otel, Basic auth), claude -p (v2.1.162) exported nothing: zero OTLP POST reached Langfuse, and OTEL_LOG_LEVEL=debug produced no telemetry output. We did not ship a config that looks right but produces nothing (constitution §3.1 — prefer dogfood over artifact-existence). Note Langfuse is a GenAI trace store anyway; Claude Code telemetry is metrics/events (a metrics backend like SigNoz/Prometheus is the right target). Our LLM-call observability is already covered by the gateway seam (automation/_platform/otel.ts + PF-111). Revisit only via an interactive session or the Agent SDK, against a metrics backend.
  • #2 Skill-regression evals — BLOCKED on headless skill dispatch. Status → Spike first (via Agent SDK), do not build on claude -p. The Anthropic skill-creator framework needs the Skill tool to actually dispatch. Our prior dogfood (evals/SPIKE_FINDINGS.md, 2026-05-26) established that headless claude -p "use the X skill" reads the SKILL.md as a document — the Skill tool is not dispatched — so trigger-rate / skill-vs-no-skill testing is not faithful in headless CI. The report’s own bridge (Claude Agent SDK with settingSources:['project']) is the right substrate; spike that it dispatches skills before building the eval harness. This ties #2 to the (deferred) Agent SDK item rather than to the promptfoo/headless lane.
  • #3 Command→skill consolidation — DO NOT mass-migrate. One real defect found. A deterministic classifier (lines + numbered-steps + headers + script-refs) scored 43 of 85 commands as “logic-heavy”, but size ≠ skill: the real axis is triggering (explicit user-invocation = command; auto-apply-on-context = skill). A long, deliberately-invoked /validate-spec is correctly a command. So mass migration is rejected (churn + loss of explicit-invocation semantics). The genuine defect: pre-commit-check exists as BOTH a command (133 lines) and a skill (60 lines) — the exact “if both exist, the skill wins” collision the report warns of, so the command is shadowed/dead (and the skill’s related-skills self-references pre-commit-check). Action item: resolve via the registry deprecation flow (as #810 did for agents) — not a blind delete. Otherwise: keep current command/skill split.
  • #4 Context-file bloat — audited; no change now. constitution.md is 2,222 lines but on-demand (referenced, not @-imported — the ETH Zurich always-loaded-bloat risk is mitigated). Concentration: §5 Database/Naming/Env (~783 lines) + §6 PWA (~483 lines) = ~57% of the file; §5 overlaps .claude/rules/database.md. Extraction candidate (move §5/§6 detail into path-scoped rules/*.md, leave authoritative summaries) — backlog, owner-gated, not done here (trimming a 2,222-line authority doc needs careful review).

​
Revised disposition

​
References

  • Eval-observability plan: docs/superpowers/plans/2026-06-02-eval-observability-tooling.md
  • Operator guide: docs/development/EVAL_OBSERVABILITY.md
  • Headless actor limits: evals/SPIKE_FINDINGS.md (2026-05-26)
  • ETH Zurich, “Evaluating AGENTS.md…”, arXiv:2602.11988 (Feb 2026)