Caught Regressions
An eval harness earns its keep by what it catches. These are three real,
documented catches from this repo’s history — what broke, how the harness caught it, and
the evidence. Nothing on this page is invented; every claim links to the issue, PR, or CI
run that recorded it.
M3 · BASELINE COMPARE
Deliberate scorer sabotage turned CI red
- What broke
- For the M3 acceptance test (PR #39), the outcome_success scorer was deliberately sabotaged on a branch — a change that would have silently shipped a broken gate.
- How the harness caught it
- The artifact-based baseline compare diffed the branch scorecard against the last green main run and failed the job with a readable per-metric breakdown.
book-happy-path: passed → failed
outcome_success: 1 → 0
aggregate 6/6 → 5/6
M4 · SCRIPTED VS REAL
First real-LLM runs made zero tool calls
- What broke
- The first real Claude-driven runs completed “successfully” while making ZERO tool calls: empty tool schemas, no call context in the prompt, and the seam design’s outcome-mode scoring had never been implemented (issue #43).
- How the harness caught it
- The M4 dual-adapter comparison published RED on the live dashboard instead of silence — every scenario regressed scripted → real, visibly. Fixed in PR #44: real per-tool JSON schemas, a scenario-derived call-context block, and order-independent outcome scoring; real runs then scored outcome_success 6/6.
M4/M5 · LATENCY GATE
Real runs green on outcome, permanently red on the 800ms gate
- What broke
- After #43 was fixed, real runs booked correctly (outcome_success 6/6) but stayed red forever: real Claude turns take 2–7s against the 800ms Tier-1 latency budget — a mis-calibrated gate, not a mis-behaving agent (issue #45).
- How the harness caught it
- The harness surfaced the miscalibration instead of hiding it: the replay timing overlay redlined every real turn against the 800ms budget, making the gate itself the visible defect. Redesigned in PR #46 to a real-run budget, with the 800ms redline retained as the reference line.