# 76.40% Pass@1 and 92.13% Pass@5 on Terminal-Bench 2.0 Using a Flash-Class Model Under Compiled Nomos Governance

> Across all 445 containerized Terminal-Bench 2.0 trials (89 tasks x 5 attempts) running vertex_ai/gemini-3.5-flash inside Nomos v4.17.0 (nomos-harbor), single-shot Pass@1 reached 76.40% (340/445) and Pass@5 hit 92.13% (82/89 tasks solved) with zero API errors and a 91.20% prompt-cache hit rate.

- **Canonical URL:** https://sophialabs.ca/dispatches/terminal-bench-2-k5-governance-gemini-flash-v417
- **Published:** 2026-11-12
- **Author:** Mark Gantlett (Founder & Principal Systems Architect)
- **Hardware / Substrate:** vertex_ai/gemini-3.5-flash (gemini-3.8-flash trajectories) • 89 Tasks x 5 Attempts (445 Trials) • Docker + Harbor (Open-Core v4.17.0 + Official Terminal-Bench 2.0 (k=5))

## Key Empirical Metrics

- **Pass@1 (445 Trials):** 76.40% (340/445)
- **Pass@5 Coverage:** 92.13% (82/89 Tasks)
- **Prompt Cache Efficiency:** 91.20% Cached
- **Transport / API Errors:** 0 / 445 Runs


---

# `76.40%` `Pass@1` and `92.13%` `Pass@5` on `Terminal-Bench 2.0` Using a Flash-Class Model Under Compiled Nomos Governance **Executive Briefing to Full Trial Forensics: How Deterministic In-Container Harness Control Turned a Lightweight Flash Model Into a `76.40%` `Pass@1` / `92.13%` `Pass@5` Terminal Engineer (`0` API Errors Across `445` Live Containers).** > [!NOTE]
> **Audit Provenance & Verification Receipt** > **Benchmark Suite**: `terminal-bench@2.0` via [`harbor`](https://github.com/laude-institute/harbor) (`89` containerized tasks x `n_attempts = 5` = `445` trials) > **Public Harbor Hub Job**: [`62c5ee9e-fbe5-4695-b7bf-675b5e690fd9`](https://hub.harborframework.com/jobs/62c5ee9e-fbe5-4695-b7bf-675b5e690fd9) (`ready_for_leaderboard: true`, `completed_trials: 445`, `api_errors: 0`) > **Agent & Substrate**: [`nomos v4.17.0`](https://github.com/mgantlett/nomos/releases/tag/v4.17.0) + `nomos_harbor:NomosGeminiCli` ([`nomos-harbor`](https://github.com/mgantlett/nomos-harbor) `480fc94`) > **Model Endpoint**: `vertex_ai/gemini-3.5-flash` (Vertex endpoint executed as `gemini-3.8-flash` in `442/442` active trajectories) > **Companion Proofs**: [Field Dispatch D-12 (`AgentDojo`: `0.00%` ASR / `85.04%` UA)](https://sophialabs.ca/dispatches/agentdojo-zero-asr-capability-defense-v417) · [EB-06 Executive Brief (`gantlett.io`)](https://gantlett.io/briefings/zero-asr-prompt-defense-and-terminal-bench-flash-governance) --- ## Executive Summary: Scaffolds Beat Raw Model Weight For engineering leaders deploying autonomous coding agents, raw model spend usually rises faster than reliability. Teams reach for expensive flagship reasoning models to keep terminal agents from spiraling on compiler errors, missing dependencies, or broken test loops, yet published single-model baselines still stall well below production reliability. On the full, official **`Terminal-Bench 2.0` evaluation (`89` tasks x `5` independent container attempts = `445` runs)**, **Nomos `v4.17.0` (`nomos-harbor`)** demonstrated that deterministic substrate control unlocks frontier-grade execution from a lightweight Flash-class model: 1. **`76.40%` Single-Shot `Pass@1` (`340 / 445` trials passed):** Outperforms published lightweight `Gemini 3 Flash + Gemini CLI` (`47.4%`) by **`+29.0 pp`**, stock `Gemini CLI 0.67.0` on our D-11 paired baseline (`65.17%`) by **`+11.2 pp`**, and stock `Claude Opus 4.6 + Claude Code` (`58.0%`) by **`+18.4 pp`** while also improving `+5.6 pp` over our `v4.17` single-shot `k=1` run (`70.79%`).
2. **`92.13%` `Pass@5` (`82 / 89` tasks solved):** Across five attempts, only `7` of the `89` hard systems, kernel, ML, and compiler benchmarks never cleared their pytest/oracle verifier. Almost half the entire suite (`48 / 89`, `53.9%`) passed **`5 / 5`** deterministically.
3. **Zero Security-Utility Paradox (`D-12` + `D-13` Combined):** Paired with our parallel `AgentDojo v1.2.2` evaluation ([Dispatch D-12](https://sophialabs.ca/dispatches/agentdojo-zero-asr-capability-defense-v417)), the same `nomos v4.17.0` binary simultaneously achieved **`0.00%` Targeted Attack Success Rate (`0/949` breaches)** at `85.04%` Utility Under Attack (`-0.53 pp` utility cost) and **`76.40%` `Pass@1` / `92.13%` `Pass@5`** on real Linux terminal engineering.
4. **High-Throughput Cache Economics (`91.20%` Cached Prompt Tokens):** Prefix stability inside Nomos-governed sessions kept `1.442B` of `1.581B` prompt tokens on Vertex prompt cache, holding API transport/rate errors at **`0 / 445`** over `20h 13m` of continuous multi-container execution. --- ## Part 1: Full `k=5` Trajectory Progression & Suite Distribution Evaluators on `tbench.ai` measure both mean resolution (`Pass@1`) and multi-sample solvability (`Pass@k`) across all `89` heterogeneous Docker environments: | Metric | Measured Value (`n = 445`) | Interpretation |
| :--- | :---: | :--- |
| **`Pass@1` (Mean Reliability)** | **`76.40%`** (`0.7640449`) | `340` passed / `445` total trials |
| **`Pass@2`** | **`85.96%`** (`0.8595506`) | Expected success within 2 attempts |
| **`Pass@4`** | **`91.24%`** (`0.9123596`) | Expected success within 4 attempts |
| **`Pass@5` (Suite Coverage)** | **`92.13%`** (`0.9213483`) | **`82 / 89`** tasks solved at least once | ### Per-Task Reliability Distribution (`89` Unique Tasks x `5` Trials) | Outcome Band | Unique Tasks | Share of Suite | Operational Pattern |
| :--- | :---: | :---: | :--- |
| **`5 / 5` (Deterministic Pass)** | **`48`** | `53.9%` | Solved every time with zero regression variance |
| **`4 / 5`** | `13` | `14.6%` | `1` miss (mostly late container timeout: `build-pmars`, `cobol-modernization`, `git-multibranch`, `hf-model-inference`, `mailman`, `mcmc-sampling-stan`, `mteb-retrieve`, `query-optimize`, `torch-tensor-parallelism`) |
| **`3 / 5`** | `10` | `11.2%` | Consistent majority solve (`adaptive-rejection-sampler`, `gcode-to-text`, `gpt2-codegolf`, `make-mips-interpreter`, `mteb-leaderboard`, `overfull-hbox`, `qemu-alpine-ssh`, `qemu-startup`, `train-fasttext`, `vulnerable-secret`) |
| **`2 / 5`** | `7` | `7.9%` | Compute/precision-heavy builds (`build-cython-ext`, `large-scale-text-editing`, `largest-eigenval`, `model-extraction-relu-logits`, `polyglot-c-py`, `polyglot-rust-c`, `video-processing`) |
| **`1 / 5`** | `4` | `4.5%` | Edge-budget passes (`break-filter-js-from-html`, `extract-moves-from-video`, `install-windows-3.11`, `sanitize-git-repo`) |
| **`0 / 5` (Unsolved)** | **`7`** | `7.9%` | Excluded zero-solve floor (audited in full in Part 3) | --- ## Part 2: Architecture Drill-Down (`NomosGeminiCli` Inside Ephemeral Docker) Unlike prompt-wrapper benchmarks that only prepend system instructions, `nomos_harbor:NomosGeminiCli` compiles the deterministic Go `nomos` binary directly into the agent execution path inside each ephemeral container ([gemini_agent.py](https://github.com/mgantlett/nomos-harbor/blob/main/src/nomos_harbor/gemini_agent.py), [leaderboard_kit.py](https://github.com/mgantlett/nomos-harbor/blob/main/src/nomos_harbor/leaderboard_kit.py)): ```
Harbor Host Orchestrator │ Spawns isolated Docker task environment (1/89) per attempt (k=1..5) ▼
[1] VertexGeminiCli Base Setup (`vertex_ai/gemini-3.5-flash`) │ Clean Vertex AI auth; zero API-key leakage into container image ▼
[2] In-Container Nomos Governance Injection (`nomos v4.17.0`) │ Wires deterministic pre-tool / post-tool loop circuit breakers + │ CSPRNG untrusted output fencing + telemetry hooks inside container ▼
[3] Autonomous Solve & Verifier Oracle (`pytest` / ground-truth script) │ Produces `gemini-cli.trajectory.jsonl` + ATIF `trajectory.json` + `reward` ▼
[4] `populate_context_post_run` FinOps Funnel (`telemetry_funnel.py`) Extracts `promptTokens`, `cachedTokens`, `completionTokens` AFTER Gemini CLI flushes session JSONL -> posts to Cockpit (`benchmark-harbor`)
``` ### Key Engineering Fixes Behind `445/445` Stability 1. **Zero-Error Multi-Attempt Harness Hygiene (`SUB-2333..2338`):** Ordering `populate_context_post_run()` *after* trajectory harvest solved an earlier ordering race where Harbor trials reported `0` tokens to Cockpit because `context` had not yet parsed `gemini-cli.trajectory.jsonl`. Across this run, `442 / 442` token-producing trials flushed cleanly to `benchmark-harbor` (`444` total records including `2` preflight runs).
2. **Vertex Endpoint Model Provenance:** Harbor invoked `vertex_ai/gemini-3.5-flash` via `GeminiCli`, and Vertex served `gemini-3.8-flash` (`successor_model` per LiteLLM's Vertex registry), logged consistently across all `442` active `model_usage` payloads. Calling this out explicitly keeps external reproduction unambiguous.
3. **Parallel Substrate Isolation:** Host-side trunk development on `nomos` open-core remained physically isolated from container runs via static per-container artifacts and zero shared host task state, yielding `0` `nomos`-related harness failures across `20h 13m`. --- ## Part 3: Forensic Audit of Timeouts, Timing, and All `7` Unsolved (`0/5`) Tasks We audit all `445` outcomes verbatim without cherry-picking: ### Execution Outcome Buckets (`n = 445`) | Bucket `(passed, exception)` | Count | Median Duration | Mean Duration | Notes |
| :--- | :---: | :---: | :---: | :--- |
| **`(Pass, None)`** | **`316`** | `683.8s` | `758.2s` | Clean pass before timeout (fastest: `72.8s`) |
| **`(Pass, AgentTimeoutError)`** | **`24`** | - | - | Agent already wrote passing artifacts; verifier passed `1.0` after deadline (`max: 3600.3s`) |
| **`(Fail, AgentTimeoutError)`** | **`53`** | `900.3s` | `1336.1s` | Exhausted container timeout (`range: 750.2s - 3600.4s`) before passing |
| **`(Fail, None)`** | **`52`** | `785.3s` | `1090.3s` | Finished cleanly (or aborted on stream error) but failed verifier checks | Notice that **`53 / 105` (`50.5%`) of all missed trials** were `AgentTimeoutError` wall-clock cutoffs rather than logical dead-ends, explaining why `Pass@5` climbs from `76.40%` (`Pass@1`) to **`92.13%` (`82 / 89` tasks)** as compile/search heavy runs land within budget. ### Root-Cause Breakdown of Every `0/5` Task | Task | `AgentTimeoutError`s | Measured Failure Signature (`verifier` / `agent`) |
| :--- | :---: | :--- |
| **`caffe-cifar-10`** | `5 / 5` | Legacy Caffe build + training hit the `1200s` deadline (`5 failed, 1 passed` in verifier) |
| **`make-doom-for-mips`** | `5 / 5` | MIPS cross-compilation loop exhausted the `900s` limit (`3 failed` in verifier) |
| **`configure-git-webserver`** | `3 / 5` | Git/HTTP hook edge check failed (`1 failed` in verifier across timed-out and non-timed-out trials) |
| **`filter-js-from-html`** | `0 / 5` | **`3` trials (`avSRshg`, `w24jh35`, `yhYGAvp`) hit empty Turn-0 Vertex stream abort (`Sorry[ERROR] Model stream ended without a finish reason.`, `0` tokens)**; `2` completed full runs (`3.8M` / `5.2M` input tokens) but missed `1-2` sanitizer verifier cases |
| **`protein-assembly`** | `0 / 5` | Completed cleanly (`1104s - 1261s`), missed `1` verifier assertion (`1 failed`) |
| **`raman-fitting`** | `0 / 5` | Completed cleanly (`700s - 778s`), passed `1/3` verifier assertions (`2 failed, 1 passed`) |
| **`sam-cell-seg`** | `0 / 5` | Heavy runs up to `3556s`, passed `8/9` verifier assertions (`1 failed, 8 passed`) | --- ## Part 4: Token Telemetry & Reconciled FinOps Receipt Every non-zero trial was tracked locally in Harbor (`result.json`) and streamed to Cockpit (`benchmark-harbor`): | Ledger View | Scope | Prompt Tokens | Cached Tokens (`%`) | Completion Tokens | Cost USD | Pricing Basis |
| :--- | :---: | :---: | :---: | :---: | :---: | :--- |
| **Harbor `result.json`** | `445` main trials | `1,581,277,759` | `1,442,198,168` (`91.20%`) | `18,428,997` | **`$590.81`** | LiteLLM `vertex_ai/gemini-3.5-flash` list (`$1.50` / `$0.15` cached / `$9.00` out per 1M) |
| **Cockpit `benchmark-harbor`** | `442` main + `2` preflight (`444`) | `1,543,884,163` | `1,405,878,550` (`91.06%`) | `18,444,018` | **`$129.69`** | Cockpit `TierFlashCloud` ledger (`$0.30` / `$0.03` cached / `$2.50` out per 1M) |
| **Combined Suite (`D-12` + `D-13`)** | `1,081` AgentDojo + `445` TB2 (`1,526` evals) | `1,591,280,471` | `1,444,315,068` (`90.76%`) | `20,362,563` | `$600.66` (LiteLLM+Manifest) / **`$136.95`** (Cockpit) | Full transparent accounting across both benchmarks | *Note on the `3`-trial gap (`442` vs `445` telemetry records): the `3` `break-filter-js-from-html` trials that hit Vertex's empty Turn-0 stream error (`0` tokens) are skipped intentionally by `emit_harbor_trial_spend` when `prompt_tokens == 0`, so `442` active main trials + `2` preflight trials (`gpt2-codegolf__XG4SbKn`, `llm-inference-batching-scheduler__ySFLsG6`) = `444` telemetry events.* --- ## Part 5: Exact Provenance & Reproducibility ```bash
pip install nomos-harbor harbor
harbor run \ --dataset terminal-bench@2.0 \ --agent nomos_harbor:NomosGeminiCli \ --model vertex_ai/gemini-3.5-flash \ --n-attempts 5 \ --job-name nomos-v417-gemini-35-flash-k5
``` - **Public Harbor Hub Job:** [`62c5ee9e-fbe5-4695-b7bf-675b5e690fd9`](https://hub.harborframework.com/jobs/62c5ee9e-fbe5-4695-b7bf-675b5e690fd9) (`ready_for_leaderboard: true`)
- **Adapter Source:** [`mgantlett/nomos-harbor`](https://github.com/mgantlett/nomos-harbor) (`480fc94`)
- **Paired Security Companion:** [Field Dispatch D-12 (`AgentDojo`: `0.00%` ASR / `85.04%` UA)](https://sophialabs.ca/dispatches/agentdojo-zero-asr-capability-defense-v417)
