# Trial by Trace Paired Audit: Cutting Harness-Caused Agent Failures by 18.2% on Claude Code

> Across 190 isolated Docker containers on TraceTrial-Bench (arXiv:2610.32285), adding compiled 4ms Go hooks to stock Claude Code v2.1.220 lifted Agent deliverables +15.2% (+78.3% in Context & Continuity) without changing a single model weight.

- **Canonical URL:** https://sophialabs.ca/dispatches/tracetrial-bench-paired-harness-audit
- **Published:** 2026-11-02
- **Author:** Mark Gantlett (Founder & Principal Systems Architect)
- **Hardware / Substrate:** 190 Hermetic Docker Containers • Claude Code v2.1.220 • Nomos v4.1.0 Static Go Binary (CGO_ENABLED=0) (Open-Core v4.1.0)

## Key Empirical Metrics

- **Agent Deliverable Lift:** +15.2% Rel.
- **Context & Continuity:** +42.8% (+78.3% Agent)
- **Harness-Caused Failures:** -18.2% (55 -> 45)
- **Compiled Hook Latency:** ~4ms Static Go


---

# Trial by Trace Paired Audit: Separating Model Intelligence from Harness Reliability **A 190-Container Paired A/B Evaluation of `nomos-claudecode:4.1.0` Against Stock `Claude Code v2.1.220` Across All 95 Tasks of `TraceTrial-Bench` (`arXiv:2610.32285`).** > [!NOTE]
> **Benchmark Provenance & Paired Control Invariants** > **Official Benchmark Listing**: [TraceLite-AI/TraceTrial-Bench #community-runs (GitHub)](https://github.com/TraceLite-AI/TraceTrial-Bench#community-runs) • [TraceLite-AI/TraceTrial-Bench #community-runs (Hugging Face)](https://huggingface.co/datasets/TraceLite-AI/TraceTrial-Bench#community-runs) > **Date Evaluated**: November 2, 2026 • **Benchmark**: `TraceTrial-Bench` (`Task/final.manifest`, 95 tasks across 7 reliability pillars, 0 execution errors) > **Control Arm**: Stock `Claude Code v2.1.220` CLI inside hermetic Docker containers (`n = 95`) > **Treatment Arm**: `harnessbench/nomos-claudecode:4.1.0` (`Claude Code v2.1.220` + compiled `nomos hook` Go binary, `n = 95`) > **Fixed Backing Model**: `gemini-2.5-flash-lite` (in-container LiteLLM gateway) • **Fixed Judge Model**: `gemini-3.5-flash-lite` --- ## 1. Industry Leaderboard Context: The Hidden Frontier Retry Tax Most coding benchmarks measure only whether a final diff passes unit tests, conflating **model capability** with **harness reliability**. Published in October 2026, *Trial by Trace* (`TraceTrial-Bench`, `arXiv:2610.32285`) decouples those two layers by capturing native execution trajectories (`Trace IR`) and independently scoring **Agent Deliverables** (`0 - 100`) and **Harness Reliability** (`0 - 100`). Below is the benchmark authors' published public leaderboard ([tracelite.ai/leaderboard.html](https://tracelite.ai/leaderboard.html)) across their 3,420 container runs (`4` harnesses x `9` models x `95` tasks), which motivated selecting `#1` `Claude Code` for our own 190-container paired audit in §2: | TraceLite Authors' Public Leaderboard (`855` runs each across 9 models) | Overall Score | Task Pass | Agent Score | Harness Macro | Clean (`100%` Both) | Harness-Caused Failures | Defect Absorbed by Model | Mean Cost / Run |
| :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
| **1. `Claude Code`** | **73.5** | **44.0%** | **75.9** | 71.2 | 9.1% | 22.7% | 34.9% | $0.436 |
| **2. `OpenClaw`** | 71.8 | 38.2% | 71.7 | **72.0** | **11.7%** | 26.0% | 26.5% | $0.633 |
| **3. `DeepSeek Harness`** | 71.0 | 40.0% | 73.3 | 68.8 | 9.8% | **21.8%** | 30.2% | $0.446 |
| **4. `Codex` (OpenAI CLI)** | 65.2 | 35.8% | 70.7 | 59.8 | 2.8% | 28.1% | 33.0% | $0.595 | Examining individual model-by-harness slices in the published 3,420-run dataset exposes a critical systems insight: 1. **Frontier Models Burn Dollars Absorbing Broken Harnesses (`46.3%` Defect Absorption)**: On `Claude Code + Opus 5` (`$0.660 / run`), the model absorbs **46.3%** of underlying harness defects by burning extra reasoning turns to recover from silent truncation, unexpanded path errors, or ambiguous tool exit channels. On a weaker harness (`Codex + Opus 5`), per-run token cost spikes to **$2.121 / run** while Overall score falls from `77.6` to `65.4`.
2. **Cost-Efficient & Local Models Collapse Without Deterministic Guardrails (`57.9%` Harness-Caused Failure)**: As parameter scale drops from `Opus 5` to smaller models (`HY3` at `30.5%` harness-caused failures, and `Gemini 2.5 Flash-Lite` at **57.9%** in our stock `Claude Code` control run), defect absorption drops from `46.3%` to `8.4%`. Small and local models cannot brute-force broken tool loops.
3. **Testing Against the #1 Baseline (`Claude Code`)**: Rather than benchmarking against a weak scaffold, we locked **#1-ranked `Claude Code v2.1.220`** and **fixed `Gemini 2.5 Flash-Lite`** across both arms, testing whether Nomos's compiled Go `nomos hook` substrate improves the strongest commercial CLI harness without changing model weights. ```mermaid
flowchart LR subgraph Manifest["TraceTrial-Bench (95 Tasks / 7 Pillars)"] T["Context (15) • Execution (27)<br/>Governance (9) • Lifecycle (19)<br/>Observability (7) • Tools (17) • Verification (1)"] end subgraph Arms["190 Paired Docker Containers (Fixed Gemini 2.5 Flash-Lite)"] Ctrl["Control Arm (n = 95)<br/>Stock Claude Code v2.1.220<br/>Overall: 44.89 | Harness-Caused: 55"] Treat["Treatment Arm (n = 95)<br/>Claude Code + Nomos v4.1.0 Go Hooks<br/>Overall: 48.78 | Harness-Caused: 45"] end subgraph Judge["Two-Layer Trace IR Judge"] Out["Agent Deliverables: +15.2% Rel.<br/>Context Continuity: +42.8%<br/>Head-to-Head: 36W / 30T / 29L"] end Manifest --> Ctrl Manifest --> Treat Ctrl --> Judge Treat --> Judge
``` --- ## 2. Full 95-Task Paired Scorecard (`Control` vs. `v4.0.0` vs. `v4.1.0`) We ran two complete engineering iterations across all 95 containerized tasks (`0` execution errors across all 285 total container runs):
- **`v4.0.0` (Python Overlay Prototype)**: Lifted Agent deliverables (`+4.98`), but regressed on Harness reliability (`-1.80`) because passing `--settings` on the CLI shadowed task-local `.claude/settings.json` hooks in Pillars `O` (`-6.23`) and `T` (`-6.51`).
- **`v4.1.0` (Compiled Go `nomos hook` Engine)**: Replaced the prototype with a `28 MB` static Go binary (`~4ms` invocation), additive `$CLAUDE_CONFIG_DIR/settings.json` hook chaining, Claude Code v2.1.220 `hookSpecificOutput` wire parity, pre-flight `$VAR`/`~` read/write path guards, and a 3-strike identical-failure circuit-breaker (with a 12-strike exemption for legitimate background status polling). | Pillar / Cohort (`n = 95`, `0` errors) | Control Overall | `v4.0.0` Overall | **`v4.1.0` Overall** | **`v4.1.0` vs Control** | Agent Axis (`Control -> v4.1.0`) | Harness Axis (`Control -> v4.0.0 -> v4.1.0`) |
| :--- | :---: | :---: | :---: | :---: | :---: | :---: |
| **Full 95-Task Suite** | `44.89` | `46.48` | **`48.78`** | **`+3.89` (`+8.7%` rel.)** | `34.78 ->` **`40.05` (`+5.27` / `+15.2%` rel.)** | `55.01 -> 53.20 ->` **`57.52` (`+2.51`)** |
| **`C` · Context & Continuity (`n = 15`)** | `43.55` | `49.81` | **`62.16`** | **`+18.62` (`+42.8%` rel.)** | `32.59 ->` **`58.12` (`+25.52` / `+78.3%` rel.)** | `54.51 -> 53.91 ->` **`66.21` (`+11.71`)** |
| **`V` · Verification & Batch (`n = 1`)** | `7.14` | `7.14` | **`42.86`** | **`+35.71`** | `0.00 -> 0.00` (`0.00`) | `14.29 -> 14.29 ->` **`85.71` (`+71.43`)** |
| **`G` · Guardrails & Safety (`n = 9`)** | `44.96` | `52.46` | **`50.18`** | **`+5.22` (`+11.6%` rel.)** | `35.46 ->` **`46.73` (`+11.27` / `+31.8%` rel.)** | `54.45 -> 59.67 -> 53.63` (`-0.82`) |
| **`O` · Observability & Audit (`n = 7`)** | `50.48` | `45.77` | **`52.16`** | **`+1.69` (`+6.39` vs `v4.0.0`)** | `35.71 -> 34.92` (`-0.79`) | `65.24 -> 59.01 ->` **`69.40` (`+4.17`)** |
| **`E` · Execution & Recovery (`n = 27`)** | `46.57` | `47.10` | **`47.58`** | **`+1.02`** | `42.92 ->` **`45.75` (`+2.83`)** | `50.22 -> 47.58 -> 49.42` (`-0.80`) |
| **`T` · Tooling & Protocol (`n = 17`)** | `41.76` | `40.46` | `41.61` | `-0.15` (`+1.14` vs `v4.0.0`) | `20.19 ->` **`20.47` (`+0.29`)** | `63.33 -> 56.82 -> 62.74` (`+5.92` vs `v4.0.0`) |
| **`L` · Long-Horizon State (`n = 19`)** | `46.27` | `47.84` | `44.75` | `-1.53` | `39.15 -> 36.02` (`-3.13`) | `53.39 -> 54.23 ->` **`53.47` (`+0.08`)** | ### Causal Failure Attribution & Head-to-Head Summary (`n = 95`)
- **Causal Failure Reduction**: Cut `harness_caused` failure attributions from `55` (Control) -> `52` (`v4.0.0`) -> **`45` (`v4.1.0`, `-18.2%` relative reduction)**.
- **Winning Head-to-Head Task Record**: Flipped `28` wins / `37` ties / `30` losses (`v4.0.0`) into **`36` Nomos wins / `30` ties / `29` Control wins** overall (**`24` wins / `53` ties / `18` losses** on the Harness axis).
- **Largest Paired Single-Task Lifts (`v4.1.0` vs. Stock Control)**: 1. `G3-ratefeed-001`: Overall `10.0 ->` **`70.0` (`+60.0`)** (`Agent 0.0 -> 100.0`, `Harness 20.0 -> 40.0`) 2. `T7-opsplugin-compliance-002`: Overall `12.2 ->` **`63.6` (`+51.4`)** (`Agent 9.1 -> 27.3`, `Harness 15.4 -> 100.0`) 3. `O2-harness-refusal-notice-001`: Overall `37.5 ->` **`87.5` (`+50.0`, `+81.2` vs `v4.0.0`)** (`Agent 0.0 -> 75.0`, `Harness 75.0 -> 100.0`) 4. `C4-resume-fidelity-001`: Overall `35.0 ->` **`83.8` (`+48.8`)** (`Agent 20.0 -> 80.0`, `Harness 50.0 -> 87.5`) 5. `C3-certretention-002`: Overall `22.6 ->` **`70.2` (`+47.6`)** (`Agent 16.7 -> 83.3`, `Harness 28.6 -> 57.1`) 6. `C4-artifact-sign-handoff-002`: Overall `42.9 ->` **`85.7` (`+42.9`)** (`Agent 42.9 -> 85.7`, `Harness 42.9 -> 85.7`) 7. `E4-deploy-002`: Overall `5.6 ->` **`47.8` (`+42.2`)** (`Agent 0.0 -> 40.0`, `Harness 11.1 -> 55.6`) 8. `C3-benchbaseline-001`: Overall `36.9 ->` **`78.6` (`+41.7`)** (`Agent 16.7 -> 100.0`, `Harness 57.1 -> 57.1`) --- ## 3. Four Systems Engineering Takeaways ### 1. Additive Hook Chaining Is Mandatory (`+10.39` Observability & `+5.92` Tooling Recovery)
Passing `--settings` flags on the CLI overrides workspace-local `.claude/settings.json` hooks and compliance plugins, silently breaking user audit pipelines (`-6.23` on `O` and `-6.51` on `T` in `v4.0.0`). Merging compiled `/usr/local/bin/nomos hook pre-tool` and `post-tool` entries additively into `$CLAUDE_CONFIG_DIR/settings.json` kept `/workspace` 100% clean for strict `workspace_scope` verifiers while preserving every workspace-local hook. ### 2. Wire-Schema Precision (`{}` on Allow) Preserves Manual Approval Boundaries
Inspecting `Claude Code v2.1.220` validation schemas revealed that returning `"permissionDecision": "allow"` on `PreToolUse` auto-approves tool execution even when a runner launches in `--permission-mode manual` (`ask_always` tasks). Emitting `{}` on allow and `{"hookSpecificOutput":{"hookEventName":"PreToolUse","permissionDecision":"deny","permissionDecisionReason":"..."}}` on deny lifted `O2-harness-refusal-notice-001` from `6.2` (`v4.0.0`) and `37.5` (`Control`) to **`87.5` (`100.0` Harness)**. ### 3. Silence on Allow Beats Verbose Prompt Coaching (`-39.3%` Ablation Proof)
Ablation testing proved that injecting synthetic coaching prose into `PostToolUse` outputs polluted native error channels and masked POSIX `EACCES` and `ENOENT` diagnostics, degrading guardrail attribution by **-39.3%**. Keeping `PostToolUse` 100% silent on non-edit channels and firing only on deterministic AST, JSON, or JSONL syntax errors recovered full diagnostic fidelity. ### 4. Pre-Flight Path Hygiene & Poll-Safe Loop Breakers Multiply Continuity (`+42.8%` on `C`)
Budget models frequently pass unexpanded `$WORKSPACE` or `~` literals directly to `Read`, `Glob`, `Grep`, and `Write` tools, and repeat identical failing calls (`215` to `711` consecutive turns in unguarded runs). Rejecting unexpanded shell tokens pre-flight and enforcing a 3-strike identical-failure circuit-breaker (keyed by `(tool_name, args_sha256, workspace_mtime_digest)`, with a 12-strike allowance for `status`/`poll`/`wait` background supervision commands) produced a **+78.3% relative lift** on Context & Continuity deliverables. --- ## 4. Open-Core Reproducibility Every `nomos hook` guard evaluated in this dispatch ships in the Apache 2.0 `nomos` binary (`v4.1.0+`) with zero cloud or telemetry dependencies: - **Canonical Specification & Vectors (`VEC-TTB-01..07`)**: [`tracetrial_harness_reliability_spec.md`](https://github.com/mgantlett/nomos/blob/main/intent/architecture/specs/tracetrial_harness_reliability_spec.md)
- **Cleanroom Adversarial Test Suite**: `go test ./cli -run TestTraceTrialBench` (`7/7` fail-closed boundary vectors verified in `1.98s`)
- **Upstream Benchmark Suite**: [`TraceLite-AI/TraceTrial-Bench`](https://github.com/TraceLite-AI/TraceTrial-Bench) (`arXiv:2610.32285`)
