Nomos-BrownfieldBench: Why Greenfield AI Demos Collapse Inside 5-Year-Old Codebases#

A 16-Trajectory Paired Local-Silicon Evaluation of Qwen3.8-27B-IQ3_M (Control vs. Nomos AST Sensor Probes & 2PC Receipts) Across Legacy Monolith Traps and Wiped-Context Cold Handovers.

[!NOTE] Benchmark Provenance & Zero-Cloud Control Invariants
Date Evaluated: November 7, 2026 • Suite: Nomos-BrownfieldBench (bf-01..bf-04 Harbor tasks + seq-01..seq-04 4-sprint sequential monolith chain)
Local Silicon & Quantization: Single NVIDIA RTX 4080 (16GB VRAM, 15,510 MiB allocated, 99/99 GPU layers offloaded, 40.0 tok/s eval, 0 cloud API calls)
Fixed Backing Model: Qwen3.8-27B-IQ3_M.gguf via llama-server (temperature = 0.0, --reasoning-budget 0)
Control Arm: Stock ReAct coding loop with full read/write/pytest access (n = 8 trajectories)
Treatment Arm: Identical Qwen3.8-27B-IQ3_M weights governed by sub-5ms Nomos AST sensor probes and verified .nomos_releases.json 2PC release receipts (n = 8 trajectories)

1. The Greenfield Illusion vs. Six-Month Repository Comprehension#

Almost every viral AI coding demo starts from git init. In an empty repository, there are zero legacy conventions to violate, zero hidden downstream callers sharing mutable dictionaries, zero stale README.md files contradicting production contracts, and zero obligation for another engineer (or a fresh agent session) to understand the codebase 180 days later.

Inside a real five-year-old enterprise codebase, unmanaged coding agents fail in four predictable ways even when local unit tests turn green:

1. Duplicate Helper Sprawl (Law 6 Violation): Instead of discovering the canonical retry or HTTP client module, the agent writes an inline while loop or invents a fifth wrapper function.
2. God-File Entropy Ratcheting: Facing a 17-branch legacy routing function, the fastest path to a green test is jamming an 18th elif branch directly into the monolith.
3. Stale Documentation Poisoning (Law 4 Violation): Aged repositories accumulate 2023 README.md lore (SHA-1 X-Sig headers) that actively contradicts newer schema contracts (HMAC-SHA256).
4. Day-180 Cold-Handover Amnesia: Once the original developer closes their chat tab, architectural rationale evaporates. Six months later, incoming sessions burn thousands of tokens performing blind file archaeology across dead legacy modules (fx_table_2022.py).

flowchart LR
    subgraph Suite["Nomos-BrownfieldBench (16 Paired Trajectories)"]
        Traps["Part 1: Legacy Traps (bf-01..04)"]
        Seq["Part 2: 4-Sprint Wiped-Context Chain"]
    end

    subgraph Silicon["Fixed Local Silicon (RTX 4080 16GB)"]
        Ctrl["Control Arm: Stock Qwen3.8-27B (temp=0.0)"]
        Gov["Nomos Arm: Qwen3.8-27B + Sub-5ms AST Probes + 2PC Receipts"]
    end

    subgraph Outcomes["Deterministic Verification & Entropy Delta"]
        Res1["Traps: 2/4 (50%) -> 4/4 (100%) Survival"]
        Res2["God-File V(G): 17 -> 6 (-11 McCabe)"]
        Res3["Day-180 Handover: -22.2% Reads / -10.6% Tokens"]
    end

    Traps --> Ctrl
    Traps --> Gov
    Seq --> Ctrl
    Seq --> Gov
    Ctrl --> Outcomes
    Gov --> Outcomes

2. Part 1 Scorecard: Harbor Brownfield Trap Suite (bf-01..bf-04)#

Each bf-* task packages a messy multi-file legacy Python workspace with realistic historical technical debt and an external Harbor-compliant tests/test.sh oracle that checks both functional edge cases and structural invariants (untampered local test SHA-256 digests, SSoT module reuse, McCabe cyclomatic complexity caps, and downstream seam integrity).

Forensic Breakdown of the Two Stock Failures#

1. bf-02-god-file-ratchet (V(G) = 17 Legacy Order Router):
   Tasked with blocking suspended accounts before VIP fast-path promotion, the unmanaged stock agent edited order_router.py in 4 turns, passed the local unit test, and declared victory while leaving the monolithic route_order function at McCabe V(G) = 16 (FAIL against the V(G) <= 12 brownfield ratchet). Under Nomos, the sub-5ms pre-write AST sensor probe rejected that initial God-function patch on Turn 4 (NOMOS COMPLEXITY RATCHET DENY), prompting Qwen3.8-27B to extract clean helper functions and drop maximum file complexity from 17 down to 6 (-11) with zero human review intervention.
2. bf-03-stale-readme-trap (2023 README.md SHA-1 Lore vs. 2PC Contract):
   The workspace README.md documented a legacy 2023 X-Sig: sha1=<hex> header, whereas contracts/webhook_spec.py and the verified .nomos_releases.json 2PC receipt specified X-Nomos-Sig-256: sha256=<hmac> with a 300-second replay window. The stock agent anchored on README.md, hit contradictions across 12 turns (23,460 tokens), and timed out at turn budget without a valid fix. Grounded by verified 2PC release provenance, the Nomos arm ignored the stale README prose, implemented the exact HMAC-SHA256 + replay-window contract in 9 turns, and cut token consumption by 39.8% (14,122 vs. 23,460).

3. Part 2 Scorecard: 4-Sprint Wiped-Context Chain (Day 1 -> Day 180 Cold Handover)#

To test whether a repository remains comprehensible six months after initial AI edits, we executed four sequential feature sprints on the same persistent financial settlement monolith (settlement.py, batch_close.py, audit_trail.py, contracts/fx_rates.py, and dead legacy fx_table_2022.py). Between every sprint, 100% of agent chat history was wiped - simulating a brand-new engineer or cold-start coding session arriving months later:

• Control Arm: Each sprint inherits only the mutated files left on disk by prior sprints.
• Nomos Arm: Each sprint that passes verification automatically appends a cryptographic .nomos_releases.json 2PC release receipt summarizing the verified architectural contract (SPRINT-1 FX SSoT, SPRINT-2 ROUND_HALF_UP tax basis points, SPRINT-3 deterministic SHA-256 batch seal) before context is wiped for seq-04-day180-cold-handover (multi-currency net refund reconciliation).

The Compounding Crossover at Day 90 and Day 180#

Notice the exact economic curve across the four sprints:

• Sprints 1 & 2 (Initial Investment): Recording and reading verified 2PC receipts adds modest upfront prompt overhead while the codebase is still small (16,272 vs. 13,272 tokens on Sprint 1).
• Sprints 3 & 4 (Late-Stage Cold Handover Payoff): As cross-module invariants accumulate across settlement.py, contracts/fx_rates.py, and audit_trail.py, the stock cold-start agent has to re-open dead decoy files (fx_table_2022.py and README.md) to reverse-engineer why ROUND_HALF_UP and canonical FX rates were chosen. By Sprint 4 (Day-180 Cold Handover), the Nomos-governed session resolves the multi-currency refund specification in 10 turns instead of 12 (-13.0% across Sprints 3-4), burns 10.6% fewer tokens (17,243 vs. 19,291), performs 22.2% fewer archaeology file reads (7 vs. 9), and maintains tighter AST structural entropy (+0.1664 vs. +0.1986).

4. Onboarding Existing Brownfield Repositories Without a Day-1 Rewrite#

In open-core `nomos`, teams do not have to clean up 4,000 legacy lint warnings before adopting deterministic governance:

1. Diff-Scoped Brownfield Mode (strict_mode: false): Running nomos init in an existing monorepo scopes TDD and AST complexity checks strictly to modified packages and touched files, holding every new AI diff to clean structural bounds while leaving untouched legacy modules undisturbed.
2. Hermetic Worktree Isolation (Law 3): Protected main stays physically read-only; coding agents operate inside disposable worktrees/<repo>-<task> sandboxes until all 7 Laws pass.
3. Cryptographic Git Notes & Verified Memory (Law 2 & Law 4): nomos task sync attaches tamper-evident walkthrough receipts directly onto refs/notes/nomos/releases and promotes pitfalls to mneme.db strictly after compile and AST verification pass - giving both human engineers and cold-start agents durable six-month repository memory.

Reproducing Nomos-BrownfieldBench on Local Silicon#

# 1. Validate all 10 Harbor adversarial & brownfield task manifests
python3 shimbench/validate_suite.py

# 2. Run all 16 paired trajectories (bf-01..04 + seq-01..04) against local llama-server
python3 -u ./brownfield_bench.py --suite all \
  -o briefings/brownfield_qwen27b_audit.md \
  --json-output briefings/brownfield_qwen27b_audit.json