Polyglot Brownfield Bench: From Safety Brake to Pass-Rate Multiplier#

A 24-Trajectory Paired Audit Across Local RTX 4080 Silicon (Qwen3.8-27B-IQ3_M) and Frontier Cloud (gemini-3.8-flash) Comparing Stock ReAct Control Against Compiled Open-Core `nomos v4.11.0` on Mature Python, Go, and JavaScript Repositories.

[!NOTE] Benchmark Provenance & Two-Tier Reproducibility Contract
Date Evaluated: November 8, 2026 • Suite: RealRepo-BrownfieldBench (24 paired trajectories across 6 pinned production OSS repositories)
Tier 1 (Local RTX 4080, Subsystem-Hinted Single-File Fixes, 233,313 LOC): Qwen3.8-27B-IQ3_M (temp = 0.0, 24 turns, 0 cloud API calls) across sqlglot, urfave/cli, commander.js, spf13/cobra, and go-chi/chi (Nomos 5/6 vs. Control 4/6)
Tier 2 (Frontier Cloud, Zero-File-Hint Multi-File PRs, 196,896 LOC): gemini-3.8-flash (temp = 0.0, max_tokens = 16,384, 28 turns, zero file path hints) across sqlglot, commander.js, go-chi/chi, and BurntSushi/toml (Nomos v2 3/6 vs. Control 2/6)
Evaluation Oracle: All Nomos scaffolding stripped prior to applying the exact historical upstream fix_sha FAIL_TO_PASS + PASS_TO_PASS test patch and monotonic AST merge-base audit

1. Full 11-Language Polyglot Parity & Switchable Brownfield Controls#

Mature repositories contain battle-tested 1,000+ to 10,913 LOC files where naive linters fail on Day 1 and unmanaged agents compound technical debt on every commit. As of nomos v4.11.0, all 11 supported languages execute under unified AST complexity/LOC ratcheting, anti-shim detection, unchecked-error hygiene, and statement coverage verification:

Every enforcement dimension is declaratively switchable inside `nomos.yaml` when onboarding legacy repositories (strict_mode: false):

governance:
  quality_gates:
    strict_mode: false          # Monotonic AST baseline ratchet against trunk
    verify_scope: "diff"        # "diff" (worktree changes only) | "repo" (full tree)
    tdd_mode: "ratchet"         # "off" | "ratchet" (new modules) | "diff" | "strict"
    coverage_mode: "live"       # "off" | "artifact" (LCOV/JaCoCo) | "live"
    tdd_exclude:                # Optional glob exclusions for generated/glue paths
      - "cmd/**"
      - "generated/**"

2. Tier 1 Baseline: Local RTX 4080 Qwen3.8-27B (5/6 vs. 4/6 Stock)#

In Tier 1, each prompt named the general subsystem where a single-module upstream bug lived (769 to 10,913 target-file LOC across 233,313 repository LOC), running 100% locally on a single 16GB RTX 4080:

On spf13/cobra, stock Control repeatedly edited 269-LOC getCompletions without AST feedback until Go syntax broke on Turn 24. Under Nomos, nomos verify intercepted Turn 8's inline growth ([BASELINE main: 1022 LOC, 1192 CDS | delta]), steering Qwen3.8-27B to compress its fix into a zero-growth 1-line slice clone (append([]string{}, args[:len(args)-1]...)) and ship on Turn 12.

3. Caught by Our Own Telemetry: Why Zero-Hint v1 Stalled (1/6 vs. 3/6)#

For Tier 2 (RealRepo-BrownfieldBench v2), we stripped 100% of file path hints and moved to multi-file cross-cutting PRs on frontier gemini-3.8-flash (28 turns across sqlglot, commander.js, go-chi/chi, and BurntSushi/toml). On our initial v1 run (SUB-1931), stock Control beat Nomos 3/6 to 1/6. Trajectory forensics exposed three harness defects:

1. Upfront Prompt Scolding Induced Analysis Paralysis: Our v1 SessionStart banner injected an upfront warning: "Ratchet Invariant: Do not increase function V(G)/LOC on hotspots (extract helpers); never swallow errors." On commander-help-group-conflict, gemini-3.8-flash became so cautious about touching 736-LOC lib/help.js that it executed 27 consecutive read/search calls and 0 edits, timing out without writing a line of code.
2. Topology Map Noise (_examples/ and typings/): SynthesizeBrownfieldTopology ranked _examples/graphql/main.go (391 LOC) and typings/index.test-d.ts (1,100+ LOC) as top hotspots, steering cold-start turns into demo and type-test folders while hiding 150-LOC core entrypoints (chi.go, lib/argument.js).
3. Unchecked Read Spirals: PostToolUse hooks only fired on Write mutations, leaving Read and Grep loops completely unmonitored during passive over-reading.

4. The Control-Loop Fix & Zero-Hint gemini-3.8-flash Scorecard (1/6 -> 3/6)#

Rather than weakening a single AST invariant at commit time, we upgraded the orientation and pacing physics (SUB-1933, SUB-1934):

flowchart LR
    subgraph Boot["1. Navigation-First Handshake (SessionStart)"]
        Map["Core Modules (>=120 LOC) + Shared Modules"]
        Scope["BROWNFIELD (scope=diff, tdd=ratchet, cov=live)"]
        Clean["Noise Filter: Skip _examples/, typings/, *-test"]
    end

    subgraph Loop["2. Active Worktree Loop (PostToolUse)"]
        Read["12 / 20 Consecutive Reads -> Pacing Nudge"]
        Edit["Source Edit -> Reset Pacing + Run Fast AST Radar"]
    end

    subgraph Release["3. Atomic 2PC Release (nomos task sync)"]
        Verify["Diff-Scoped Monotonic AST Ratchet + Upstream Tests"]
        Trunk["Protected main (12/12 Clean Across Both Tiers)"]
    end

    Boot --> Loop
    Loop --> Release

• Navigation-First Handshake: Removed the upfront Ratchet Invariant scold, renamed Ratchet Hotspots to Core Modules (>= 120 LOC), surfaced scope=diff explicitly, and filtered _-prefixed directories (_examples/), typings/, .test-d.ts, and *-test harnesses out of the cold-start map.
• Worktree Read-Spiral Pacing Gyroscope (cmd_hook_pacing.go): Emitted a concise 1-line PostToolUse nudge at 12 and 20 consecutive Read/Grep calls inside active task worktrees, resetting immediately on any source edit.

Re-running all 12 zero-hint trajectories (SUB-1935) tripled Nomos pass rate from 1/6 (16.7%) to 3/6 (50.0%), flipping Nomos ahead of Control (2/6, 33.3%):

5. Three Engineering Takeaways for Agent Harness Design#

1. Separate Orientation From Enforcement: Telling a reasoning model upfront "do not increase cyclomatic complexity on legacy hotspots" before it has even located the bug biases its action prior toward endless reading. Give the agent a clean map at SessionStart, let it draft a working hypothesis, and enforce structural invariants reactively on real AST diffs.
2. Pacing Nudges Beat Hard Read Caps: Hard-blocking read tools breaks legitimate investigation on 150k-LOC repositories, whereas a non-blocking PostToolUse nudge at 12 consecutive reads broke the commander-help-group-conflict stall (27 reads -> edited on turn 24 and passed) without hurting deep search.
3. Keep Honest Frontier Ceilings in Public Benchmarks: Both Control and Nomos hit the 28-turn ceiling on sqlglot's 6,717-LOC parser when given zero file hints (with Nomos drafting its patch on turns 24-27 of sqlglot-snowflake-grants right as the turn budget expired). Retaining unsolved 150k-LOC tasks alongside wins makes harness improvements measurable as turn budgets and models scale.