Trial by Trace Paired Audit: Separating Model Intelligence from Harness Reliability#

A 190-Container Paired A/B Evaluation of nomos-claudecode:4.1.0 Against Stock Claude Code v2.1.220 Across All 95 Tasks of TraceTrial-Bench (arXiv:2610.32285).

[!NOTE] Benchmark Provenance & Paired Control Invariants
Date Evaluated: November 2, 2026 • Benchmark: TraceTrial-Bench (Task/final.manifest, 95 tasks across 7 reliability pillars, 0 execution errors)
Control Arm: Stock Claude Code v2.1.220 CLI inside hermetic Docker containers (n = 95)
Treatment Arm: harnessbench/nomos-claudecode:4.1.0 (Claude Code v2.1.220 + compiled nomos hook Go binary, n = 95)
Fixed Backing Model: gemini-2.5-flash-lite (in-container LiteLLM gateway) • Fixed Judge Model: gemini-3.5-flash-lite

1. Industry Leaderboard Context: The Hidden Frontier Retry Tax#

Most coding benchmarks measure only whether a final diff passes unit tests, conflating model capability with harness reliability. Published in October 2026, Trial by Trace (TraceTrial-Bench, arXiv:2610.32285) decouples those two layers by capturing native execution trajectories (Trace IR) and independently scoring Agent Deliverables (0 - 100) and Harness Reliability (0 - 100) across 3,420 container runs (4 harnesses x 9 models x 95 tasks):

Examining individual model-by-harness slices in the published 3,420-run dataset exposes a critical systems insight:

1. Frontier Models Burn Dollars Absorbing Broken Harnesses (46.3% Defect Absorption):
   On Claude Code + Opus 5 ($0.660 / run), the model absorbs 46.3% of underlying harness defects by burning extra reasoning turns to recover from silent truncation, unexpanded path errors, or ambiguous tool exit channels. On a weaker harness (Codex + Opus 5), per-run token cost spikes to $2.121 / run while Overall score falls from 77.6 to 65.4.
2. Cost-Efficient & Local Models Collapse Without Deterministic Guardrails (57.9% Harness-Caused Failure):
   As parameter scale drops from Opus 5 to smaller models (HY3 at 30.5% harness-caused failures, and Gemini 2.5 Flash-Lite at 57.9% in our stock Claude Code control run), defect absorption drops from 46.3% to 8.4%. Small and local models cannot brute-force broken tool loops.
3. Testing Against the #1 Baseline (Claude Code):
   Rather than benchmarking against a weak scaffold, we locked #1-ranked Claude Code v2.1.220 and fixed Gemini 2.5 Flash-Lite across both arms, testing whether Nomos's compiled Go nomos hook substrate improves the strongest commercial CLI harness without changing model weights.

flowchart LR
    subgraph Manifest["TraceTrial-Bench (95 Tasks / 7 Pillars)"]
        T["Context (15) • Execution (27)<br/>Governance (9) • Lifecycle (19)<br/>Observability (7) • Tools (17) • Verification (1)"]
    end

    subgraph Arms["190 Paired Docker Containers (Fixed Gemini 2.5 Flash-Lite)"]
        Ctrl["Control Arm (n = 95)<br/>Stock Claude Code v2.1.220<br/>Overall: 44.89 | Harness-Caused: 55"]
        Treat["Treatment Arm (n = 95)<br/>Claude Code + Nomos v4.1.0 Go Hooks<br/>Overall: 48.78 | Harness-Caused: 45"]
    end

    subgraph Judge["Two-Layer Trace IR Judge"]
        Out["Agent Deliverables: +15.2% Rel.<br/>Context Continuity: +42.8%<br/>Head-to-Head: 36W / 30T / 29L"]
    end

    Manifest --> Ctrl
    Manifest --> Treat
    Ctrl --> Judge
    Treat --> Judge

2. Full 95-Task Paired Scorecard (Control vs. v4.0.0 vs. v4.1.0)#

We ran two complete engineering iterations across all 95 containerized tasks (0 execution errors across all 285 total container runs):

• v4.0.0 (Python Overlay Prototype): Lifted Agent deliverables (+4.98), but regressed on Harness reliability (-1.80) because passing --settings on the CLI shadowed task-local .claude/settings.json hooks in Pillars O (-6.23) and T (-6.51).
• v4.1.0 (Compiled Go nomos hook Engine): Replaced the prototype with a 28 MB static Go binary (~4ms invocation), additive $CLAUDE_CONFIG_DIR/settings.json hook chaining, Claude Code v2.1.220 hookSpecificOutput wire parity, pre-flight $VAR/~ read/write path guards, and a 3-strike identical-failure circuit-breaker (with a 12-strike exemption for legitimate background status polling).

Causal Failure Attribution & Head-to-Head Summary (n = 95)#

• Causal Failure Reduction: Cut harness_caused failure attributions from 55 (Control) -> 52 (v4.0.0) -> 45 (v4.1.0, -18.2% relative reduction).
• Winning Head-to-Head Task Record: Flipped 28 wins / 37 ties / 30 losses (v4.0.0) into 36 Nomos wins / 30 ties / 29 Control wins overall (24 wins / 53 ties / 18 losses on the Harness axis).
• Largest Paired Single-Task Lifts (v4.1.0 vs. Stock Control):
  1. G3-ratefeed-001: Overall 10.0 -> 70.0 (+60.0) (Agent 0.0 -> 100.0, Harness 20.0 -> 40.0)
  2. T7-opsplugin-compliance-002: Overall 12.2 -> 63.6 (+51.4) (Agent 9.1 -> 27.3, Harness 15.4 -> 100.0)
  3. O2-harness-refusal-notice-001: Overall 37.5 -> 87.5 (+50.0, +81.2 vs v4.0.0) (Agent 0.0 -> 75.0, Harness 75.0 -> 100.0)
  4. C4-resume-fidelity-001: Overall 35.0 -> 83.8 (+48.8) (Agent 20.0 -> 80.0, Harness 50.0 -> 87.5)
  5. C3-certretention-002: Overall 22.6 -> 70.2 (+47.6) (Agent 16.7 -> 83.3, Harness 28.6 -> 57.1)
  6. C4-artifact-sign-handoff-002: Overall 42.9 -> 85.7 (+42.9) (Agent 42.9 -> 85.7, Harness 42.9 -> 85.7)
  7. E4-deploy-002: Overall 5.6 -> 47.8 (+42.2) (Agent 0.0 -> 40.0, Harness 11.1 -> 55.6)
  8. C3-benchbaseline-001: Overall 36.9 -> 78.6 (+41.7) (Agent 16.7 -> 100.0, Harness 57.1 -> 57.1)

3. Four Systems Engineering Takeaways#

1. Additive Hook Chaining Is Mandatory (+10.39 Observability & +5.92 Tooling Recovery)#

Passing --settings flags on the CLI overrides workspace-local .claude/settings.json hooks and compliance plugins, silently breaking user audit pipelines (-6.23 on O and -6.51 on T in v4.0.0). Merging compiled /usr/local/bin/nomos hook pre-tool and post-tool entries additively into $CLAUDE_CONFIG_DIR/settings.json kept /workspace 100% clean for strict workspace_scope verifiers while preserving every workspace-local hook.

2. Wire-Schema Precision ({} on Allow) Preserves Manual Approval Boundaries#

Inspecting Claude Code v2.1.220 validation schemas revealed that returning "permissionDecision": "allow" on PreToolUse auto-approves tool execution even when a runner launches in --permission-mode manual (ask_always tasks). Emitting {} on allow and {"hookSpecificOutput":{"hookEventName":"PreToolUse","permissionDecision":"deny","permissionDecisionReason":"..."}} on deny lifted O2-harness-refusal-notice-001 from 6.2 (v4.0.0) and 37.5 (Control) to 87.5 (100.0 Harness).

3. Silence on Allow Beats Verbose Prompt Coaching (-39.3% Ablation Proof)#

Ablation testing proved that injecting synthetic coaching prose into PostToolUse outputs polluted native error channels and masked POSIX EACCES and ENOENT diagnostics, degrading guardrail attribution by -39.3%. Keeping PostToolUse 100% silent on non-edit channels and firing only on deterministic AST, JSON, or JSONL syntax errors recovered full diagnostic fidelity.

4. Pre-Flight Path Hygiene & Poll-Safe Loop Breakers Multiply Continuity (+42.8% on C)#

Budget models frequently pass unexpanded $WORKSPACE or ~ literals directly to Read, Glob, Grep, and Write tools, and repeat identical failing calls (215 to 711 consecutive turns in unguarded runs). Rejecting unexpanded shell tokens pre-flight and enforcing a 3-strike identical-failure circuit-breaker (keyed by (tool_name, args_sha256, workspace_mtime_digest), with a 12-strike allowance for status/poll/wait background supervision commands) produced a +78.3% relative lift on Context & Continuity deliverables.

4. Open-Core Reproducibility#

Every nomos hook guard evaluated in this dispatch ships in the Apache 2.0 nomos binary (v4.1.0+) with zero cloud or telemetry dependencies:

• Canonical Specification & Vectors (VEC-TTB-01..07): `tracetrial_harness_reliability_spec.md`
• Cleanroom Adversarial Test Suite: go test ./cli -run TestTraceTrialBench (7/7 fail-closed boundary vectors verified in 1.98s)
• Upstream Benchmark Suite: `TraceLite-AI/TraceTrial-Bench` (arXiv:2610.32285)