76.40% Pass@1 and 92.13% Pass@5 on Terminal-Bench 2.0 Using a Flash-Class Model Under Compiled Nomos Governance#

Executive Briefing to Full Trial Forensics: How Deterministic In-Container Harness Control Turned a Lightweight Flash Model Into a 76.40% Pass@1 / 92.13% Pass@5 Terminal Engineer (0 API Errors Across 445 Live Containers).

Audit Provenance & Verification Receipt
Benchmark Suite: [email protected] via `harbor` (89 containerized tasks x n_attempts = 5 = 445 trials)
Public Harbor Hub Job: `62c5ee9e-fbe5-4695-b7bf-675b5e690fd9` (ready_for_leaderboard: true, completed_trials: 445, api_errors: 0)
Agent & Substrate: `nomos v4.17.0` + nomos_harbor:NomosGeminiCli (`nomos-harbor` 480fc94)
Model Endpoint: vertex_ai/gemini-3.5-flash (Vertex endpoint executed as gemini-3.8-flash in 442/442 active trajectories)
Companion Proofs: Field Dispatch D-12 (`AgentDojo`: `0.00%` ASR / `85.04%` UA) · EB-06 Executive Brief (`gantlett.io`)

Executive Summary: Scaffolds Beat Raw Model Weight#

For engineering leaders deploying autonomous coding agents, raw model spend usually rises faster than reliability. Teams reach for expensive flagship reasoning models to keep terminal agents from spiraling on compiler errors, missing dependencies, or broken test loops, yet published single-model baselines still stall well below production reliability.

On the full, official Terminal-Bench 2.0 evaluation (89 tasks x 5 independent container attempts = 445 runs), Nomos v4.17.0 (nomos-harbor) demonstrated that deterministic substrate control unlocks frontier-grade execution from a lightweight Flash-class model:

1. 76.40% Single-Shot Pass@1 (340 / 445 trials passed): Outperforms published lightweight Gemini 3 Flash + Gemini CLI (47.4%) by +29.0 pp, stock Gemini CLI 0.67.0 on our D-11 paired baseline (65.17%) by +11.2 pp, and stock Claude Opus 4.6 + Claude Code (58.0%) by +18.4 pp while also improving +5.6 pp over our v4.17 single-shot k=1 run (70.79%).
2. 92.13% Pass@5 (82 / 89 tasks solved): Across five attempts, only 7 of the 89 hard systems, kernel, ML, and compiler benchmarks never cleared their pytest/oracle verifier. Almost half the entire suite (48 / 89, 53.9%) passed 5 / 5 deterministically.
3. Zero Security-Utility Paradox (D-12 + D-13 Combined): Paired with our parallel AgentDojo v1.2.2 evaluation (Dispatch D-12), the same nomos v4.17.0 binary simultaneously achieved 0.00% Targeted Attack Success Rate (0/949 breaches) at 85.04% Utility Under Attack (-0.53 pp utility cost) and 76.40% Pass@1 / 92.13% Pass@5 on real Linux terminal engineering.
4. High-Throughput Cache Economics (91.20% Cached Prompt Tokens): Prefix stability inside Nomos-governed sessions kept 1.442B of 1.581B prompt tokens on Vertex prompt cache, holding API transport/rate errors at 0 / 445 over 20h 13m of continuous multi-container execution.

Part 1: Full k=5 Trajectory Progression & Suite Distribution#

Evaluators on tbench.ai measure both mean resolution (Pass@1) and multi-sample solvability (Pass@k) across all 89 heterogeneous Docker environments:

Per-Task Reliability Distribution (89 Unique Tasks x 5 Trials)#

Part 2: Architecture Drill-Down (NomosGeminiCli Inside Ephemeral Docker)#

Unlike prompt-wrapper benchmarks that only prepend system instructions, nomos_harbor:NomosGeminiCli compiles the deterministic Go nomos binary directly into the agent execution path inside each ephemeral container (gemini_agent.py, leaderboard_kit.py):

Harbor Host Orchestrator
   │  Spawns isolated Docker task environment (1/89) per attempt (k=1..5)
   ▼
[1] VertexGeminiCli Base Setup (`vertex_ai/gemini-3.5-flash`)
   │  Clean Vertex AI auth; zero API-key leakage into container image
   ▼
[2] In-Container Nomos Governance Injection (`nomos v4.17.0`)
   │  Wires deterministic pre-tool / post-tool loop circuit breakers +
   │  CSPRNG untrusted output fencing + telemetry hooks inside container
   ▼
[3] Autonomous Solve & Verifier Oracle (`pytest` / ground-truth script)
   │  Produces `gemini-cli.trajectory.jsonl` + ATIF `trajectory.json` + `reward`
   ▼
[4] `populate_context_post_run` FinOps Funnel (`telemetry_funnel.py`)
      Extracts `promptTokens`, `cachedTokens`, `completionTokens` AFTER
      Gemini CLI flushes session JSONL -> posts to Cockpit (`benchmark-harbor`)

Key Engineering Fixes Behind 445/445 Stability#

1. Zero-Error Multi-Attempt Harness Hygiene (SUB-2333..2338): Ordering populate_context_post_run() after trajectory harvest solved an earlier ordering race where Harbor trials reported 0 tokens to Cockpit because context had not yet parsed gemini-cli.trajectory.jsonl. Across this run, 442 / 442 token-producing trials flushed cleanly to benchmark-harbor (444 total records including 2 preflight runs).
2. Vertex Endpoint Model Provenance: Harbor invoked vertex_ai/gemini-3.5-flash via GeminiCli, and Vertex served gemini-3.8-flash (successor_model per LiteLLM's Vertex registry), logged consistently across all 442 active model_usage payloads. Calling this out explicitly keeps external reproduction unambiguous.
3. Parallel Substrate Isolation: Host-side trunk development on nomos open-core remained physically isolated from container runs via static per-container artifacts and zero shared host task state, yielding 0 nomos-related harness failures across 20h 13m.

Part 3: Forensic Audit of Timeouts, Timing, and All 7 Unsolved (0/5) Tasks#

We audit all 445 outcomes verbatim without cherry-picking:

Execution Outcome Buckets (n = 445)#

Notice that 53 / 105 (50.5%) of all missed trials were AgentTimeoutError wall-clock cutoffs rather than logical dead-ends, explaining why Pass@5 climbs from 76.40% (Pass@1) to 92.13% (82 / 89 tasks) as compile/search heavy runs land within budget.

Root-Cause Breakdown of Every 0/5 Task#

Part 4: Token Telemetry & Reconciled FinOps Receipt#

Every non-zero trial was tracked locally in Harbor (result.json) and streamed to Cockpit (benchmark-harbor):

Note on the 3-trial gap (442 vs 445 telemetry records): the 3 break-filter-js-from-html trials that hit Vertex's empty Turn-0 stream error (0 tokens) are skipped intentionally by emit_harbor_trial_spend when prompt_tokens == 0, so 442 active main trials + 2 preflight trials (gpt2-codegolf__XG4SbKn, llm-inference-batching-scheduler__ySFLsG6) = 444 telemetry events.

Part 5: Exact Provenance & Reproducibility#

pip install nomos-harbor harbor
harbor run \
  --dataset [email protected] \
  --agent nomos_harbor:NomosGeminiCli \
  --model vertex_ai/gemini-3.5-flash \
  --n-attempts 5 \
  --job-name nomos-v417-gemini-35-flash-k5

• Public Harbor Hub Job: `62c5ee9e-fbe5-4695-b7bf-675b5e690fd9` (ready_for_leaderboard: true)
• Adapter Source: `mgantlett/nomos-harbor` (480fc94)
• Paired Security Companion: Field Dispatch D-12 (`AgentDojo`: `0.00%` ASR / `85.04%` UA)