BENCHMARK Open-Core v4.24.0 + SlopCodeBench 2026-11-18 • 10 min read

SlopCodeBench 41-Checkpoint Paired Audit: Cutting Multi-Release Code Erosion by 99.6% Across Nine Applications

Across 9 multi-checkpoint greenfield-to-production Python specifications (41 checkpoints per arm) in SprocketLab/slop-code-bench on Claude Code v2.1.252 + Claude Sonnet 5.5, compiled Nomos v4.24.0 governance cut final structural erosion by 99.6% (0.0034 vs 0.8798 Control), held 0.0000 erosion across all 6 unseen applications (including a frozen zero-edit 3-app blind holdout), reduced scb-check findings by 64.7% (399 vs 1,131), capped peak complexity at 9.8 vs 61.9 (up to 123 CC in Control monoliths), and lifted overall test pass rate from 89.7% to 90.0%.

🖥️ 82 Paired Docker Trajectories (9 Problems / 41 Checkpoints per Arm) • Claude Code 2.1.252 • Anthropic Claude Sonnet 5.5 • Static Go Binary (CGO_ENABLED=0)
Final Erosion (↓ lower) 0.0034 (-99.6% vs 0.8798)
Unseen 6-App Erosion (↓ lower) 0.0000 (27/27 ckpts clean)
scb-check Debt (↓ lower) 399 (-64.7% vs 1,131)
Pass Rate / Core (↑ higher) 90.0% (+0.3pp) / 80.5%
Mark Gantlett
Mark Gantlett
Founder & Principal Systems Architect
Sophia AI
⚡ AI AUGMENTED
Tier 1 Frontier Reasoning + On-Premise RTX 4080 Silicon
FIG D-15 // SLOPCODEBENCH 41-CHECKPOINT AUDIT - Decomposing growing multi-release Python monoliths into 9.2 clean modules per app (0.0000 unseen erosion, CC <= 10, -64.7% scb-check debt) on Claude Code v2.1.252 + Claude Sonnet 5.5.

SlopCodeBench 41-Checkpoint Paired Audit: Cutting Multi-Release Code Erosion by 99.6% Across Nine Applications#

A Three-Split Paired Evaluation (9 Multi-Release Problems / 41 Checkpoints per Arm) of Apache 2.0 `nomos v4.24.0` Against Ungoverned Control Claude Code 2.1.252 (Anthropic Claude Sonnet 5.5) on `SprocketLab/slop-code-bench`.

Evaluation Protocol & Split Discipline
Harness & Model: [email protected] on anthropic/claude-sonnet-5.5 via OpenRouter (thinking=low, just-solve.jinja, pass_policy=any-case)
Nomos Release Binary: `mgantlett/nomos@9293f3c` (v4.24.0, Apache 2.0, CGO_ENABLED=0 static Linux amd64)
Metric Directionality: Test pass & solved rates: Higher is better (↑). Structural erosion, peak cyclomatic complexity (cc_max), verbosity ratio, scb-check findings, and spend: Lower is better (↓).
Paired Cost (82 Primary Endpoint Runs): `$14.3284` Control vs. `$31.9377` Nomos v4.24.0 across all 9 problems (41 checkpoints per arm)
Related Dispatches: Field Dispatch D-14 (`HarnessSecurity-Bench`) · Field Dispatch D-13 (`Terminal-Bench 2.0`)


1. Why Single-Shot Coding Benchmarks Miss Software Rot#

Standard coding benchmarks run a single prompt against one issue and discard the repository once tests pass. Production teams live inside Checkpoint 2 through Checkpoint 5: the releases added next week and next quarter on top of yesterday's AI diff.

SprocketLab/slop-code-bench directly measures that compounding rot across sequential specifications (checkpoint_1 through checkpoint_5), grading both functional test correctness and structural decay (erosion, verbosity, peak cyclomatic complexity cc_max, module count, and scb-check==0.1.3 static debt findings).

To avoid overfitting Nomos rules to any single suite, we enforced strict three-split separation:

  1. Split A - Dev Calibration (3 Problems, 14 Checkpoints: file_backup, etl_pipeline, log_query): Iterative polyglot AST sensor calibration (r4 final calibration arm).
  2. Split B - Held-Out Hardening (3 Difficulty-Stratified Problems, 13 Checkpoints: env_manager, database_migration, dag_execution): Unseen until v4.24.0 was frozen; surfaced two container PreToolUse bugs (.venv symlink walking, compound SQL heredoc parsing, and .tmp/tests/ scratch script filtering) fixed in SUB-3233 + SUB-3235 (r6 hardened arm).
  3. Split C - Untouched Blind n=1 Holdout (3 Fresh Unseen Problems, 14 Checkpoints: migrate_configs, file_merger, metric_transform_lang): Executed one single frozen pass with zero changes to Nomos (SHA-256: 327f178e4bbb75dbc5c8cbee10e36f39d86250d4fcdd34be18510dbf9ca06e5f) and zero problem or test peeking.

2. Combined 9-Problem (41-Checkpoint) Empirical Scorecard#

SlopCodeBench 41-Checkpoint Empirical Scoreboard

Metric Across 9 Problems (41 Checkpoints / Arm) Direction Control (Ungoverned) Nomos v4.24.0 (r4 + r6 + Blind n=1) Delta vs. Control
Mean Test Pass Rate (41 ckpts) Higher is better ↑ 89.7% 90.0% +0.3 pp
Core Checkpoints Solved (41 ckpts) Higher is better ↑ 80.5% 80.5% Parity (33/41)
Strict 100% Checkpoints Solved Higher is better ↑ 4.9% 4.9% Parity (2/41)
Mean Structural Erosion (41 ckpts) Lower is better ↓ 0.8406 0.0046 -99.4% cut
Mean Final Structural Erosion (9 apps) Lower is better ↓ 0.8798 0.0034 (0.0000 on 6/6 unseen) -99.6% cut
Mean / Final cc_max Lower is better ↓ 51.0 / 61.9 9.3 / 9.8 -81.8% / -84.2%
Mean / Final Verbosity Lower is better ↓ 0.4007 / 0.3818 0.1157 / 0.1155 -71.1% / -69.7%
Combined Final scb-check==0.1.3 Findings Lower is better ↓ 1,131 (175 + 426 + 530) 399 (60 + 157 + 182) -64.7% cut
Mean Final Module Count Modular split 1.0 files (9/9 monoliths) 9.2 files +8.2 files / app
Total API Spend (OpenRouter) Lower is better ↓ `$14.3284` `$31.9377` Bounded multi-turn AST refactor budget

Under unguarded Control across all 9 applications (41 checkpoints), Claude Sonnet 5.5 placed 100% of production code into a single monolithic Python file every single time (1.0 files mean), letting cyclomatic complexity spike to CC 101 (file_merger) and CC 123 (metric_transform_lang). Under Nomos v4.24.0, agents decomposed every application cleanly (9.2 files mean, CC <= 10 ceiling enforced), holding 0.0000 structural erosion across every single one of the 27 held-out and blind checkpoints.


3. Split-by-Split Results & Untouched Blind n=1 Holdout#

Split C: Untouched Blind n=1 Holdout (14 Checkpoints, Zero Nomos Edits)#

Metric Directionality: Mean Pass is Higher is better ↑. Mean/Final Erosion, Final Verbosity, Final cc_max, Final scb-check, and Spend are Lower is better ↓.

Problem Arm Mean Pass (↑ higher) Mean Erosion (↓ lower) Final Erosion (↓ lower) Final Verbosity (↓ lower) Final cc_max (↓ lower) Final Files / LOC Final scb-check (↓ lower) Spend (↓ lower)
migrate_configs (5 ckpts) Control 86.1% 0.6602 0.8047 0.4564 49 1 / 573 102 `$1.4736`
migrate_configs (5 ckpts) Nomos Frozen n=1 89.0% (+2.9 pp) 0.0000 0.0000 0.1890 10 7 / 983 49 (-52.0%) `$2.6073`
file_merger (4 ckpts) Control 94.9% 0.8636 0.8764 0.4891 101 1 / 1021 131 `$1.3886`
file_merger (4 ckpts) Nomos Frozen n=1 85.4% 0.0000 0.0000 0.1091 10 8 / 1280 48 (-63.4%) `$2.8654`
metric_transform_lang (5 ckpts) Control 94.0% 0.8750 0.8911 0.4377 77 (123 peak) 1 / 2641 297 `$4.2637`
metric_transform_lang (5 ckpts) Nomos Frozen n=1 94.0% (+0.0 pp) 0.0000 0.0000 0.0728 10 27 / 3681 85 (-71.4%) `$8.0040`
Blind Split Total (14 ckpts) Control vs. Frozen n=1 91.4% vs 89.7% 0.7950 vs 0.0000 0.8574 vs 0.0000 0.4611 vs 0.1236 75.7 vs 10.0 1.0 vs 14.0 530 vs 182 (-65.7%) `$7.1259` vs `$13.4766`

On metric_transform_lang (2,641-LOC Control monolith, 123 peak cyclomatic complexity, 297 scb-check warnings), frozen Nomos v4.24.0 matched Control's 94.0% test pass rate across all 5 checkpoints (97.8%, 99.0%, 98.8%, 78.3%, 95.9%) while organizing the implementation cleanly into 27 modules with 0.0000 erosion, cc_max <= 10, -71.4% fewer scb-check warnings, and zero false-positive pre-execution hook denials across all 14 blind checkpoints.

Splits A & B: Dev Calibration (14 ckpts) & Held-Out Hardening (13 ckpts)#

Split & Problem Control Pass (↑ higher) Nomos Pass (↑ higher) Control Final Ero (↓ lower) Nomos Final Ero (↓ lower) Control scb-check (↓ lower) Nomos scb-check (↓ lower)
Split A - Dev (14 ckpts, r4) 89.8% 90.0% 0.9162 (CC 60.3) 0.0102 (CC 10.0) 175 60 (-65.7%)
file_backup (4 ckpts) 68.6% 68.6% 0.9656 0.0000 15 8
etl_pipeline (5 ckpts) 97.2% 97.6% 0.9220 0.0000 57 29
log_query (5 ckpts) 99.4% 99.4% 0.8609 0.0307 103 23
Split B - Held-Out (13 ckpts, r6) 87.7% 90.3% (+2.6 pp) 0.8659 (CC 49.7) 0.0000 (CC 9.3) 426 157 (-63.1%)
env_manager (5 ckpts) 94.7% 94.3% 0.9383 0.0000 85 49
database_migration (5 ckpts) 93.8% 93.8% 0.8675 0.0000 154 36 (-76.6%)
dag_execution (3 ckpts) 66.0% 78.0% (+12.0 pp) 0.7918 0.0000 187 72 (-61.5%)

Notice why Split B (database_migration + dag_execution) paid huge real-world product dividends before tagging v4.24.0: single-shot benchmarks (SWE-bench, Terminal-Bench 2.0, HarnessSecurity-Bench) never exercised greenfield Python builds across 5 releases inside a strict container (nomos init --strict). Stress-testing SlopCodeBench uncovered and fixed five genuine substrate bugs:

  1. Worktree .venv Symlink Exclusion (SUB-3233): walkExplicitRelativeFiles now skips .venv, venv, node_modules, and .tmp segments just like WalkWithExcludes, preventing false-positive alias failures on virtualenv symlinks while keeping repo source symlinks blocked.
  2. Compound Heredoc & Identifier-Quoted DDL Precision (SUB-3233, SUB-3235): PreToolUse now strips static cat/tee source-write heredoc bodies per unquoted shell segment (even after python3 -m venv venv && ...), allowing migration generators containing DROP TABLE strings to be authored while still blocking live sqlite3/psql DROP TABLE commands, and CWE-89 distinguishes identifier-quoted DDL queries from interpolated values.
  3. Aligned Scratch Suite Guard (SUB-3235): Container PreToolUse now explicitly allows .tmp/tests/ verification suites recommended at SessionStart, while keeping untrusted hidden-dot shell scripts denied.
  4. Headless Container Stop Refactoring Mandate (SUB-3233): Container Stop explicitly states no human operator is present to relax nomos.yaml, driving agents to decompose complex >400-LOC functions up front rather than stalling.
  5. Polyglot Wrapper, Clone & Guard-Sprawl Sensors (SUB-3223, SUB-3229, SUB-3233): Enforced across Python, Ruby, TypeScript/JS, Go, Rust, Java, and C#: 0/1-caller micro-wrapper bans, >=5-LOC sibling clone detection, isinstance/typeof validator-return sprawl checks (>=4), Go-style (val, err) tuple bans outside Go, and chained </<= diff boundary-mutant spot-checks.

4. Open-Core Reproducibility#

Every gate, sensor, and container hook evaluated in this 41-checkpoint audit ships in `mgantlett/nomos v4.24.0` under the Apache 2.0 license with zero telemetry:

EXECUTIVE 3-MINUTE SUMMARY // GANTLETT SYSTEMS INC. (EB-08)
Stop Checkpoint-5 Rot: Eliminating Multi-Release AI Code Erosion Across 41 Sequential Releases
Read Boardroom Memo ↗
Ask Sophia Docs & Specs Sophia AI · Grounded Nomos Architecture & Docs Concierge