Tier 2 Local Hardware & SLM Benchmarks#

Empirical Inference Metrics for Autonomous AI Execution on NVIDIA RTX 4080 (16 GB VRAM).

1. Executive Summary#

As part of the Nomos Tier 2 Sovereign Engine rollout, we benchmarked and deployed Qwen3.8-27B (Dense, Q3_K_M) and Qwen2.5-Coder-14B (Q4_K_M) on a dedicated workstation (gaming-dev-pc) equipped with an Intel Core i5-12600KF and an NVIDIA GeForce RTX 4080 (16,376 MiB VRAM).

Through targeted memory optimization, 100% of the 27-billion-parameter model layers are offloaded into GPU VRAM with zero PCIe memory spillover, delivering native chain-of-thought reasoning and autonomous tool dispatching at ~38.8 tokens/second.

2. Empirical Performance Metrics#

Live inference benchmark against local llama-server (CUDA 13.x / FlashAttention enabled):

Metric	Measured Value	Operational Impact
Model	`Qwen3.8-27B-Q3_K_M.gguf`	27 Billion dense parameters in 13.6 GB weights
Generation Speed	38.80 – 38.84 tokens/sec	Sub-3-second code refactoring cycles
Prompt Ingestion	263.1 – 277.6 tokens/sec	Fast ingestion of AST contexts and test logs
Time to First Token (TTFT)	~277 ms	Near-instantaneous response for autonomous ReAct loop
GPU Offload	100% (99/99 layers in VRAM)	Zero layers spilled to system RAM
Total VRAM Footprint	15,388 MiB / 16,376 MiB	Fills physical VRAM with 700 MiB safety buffer
Parallel Execution Slots	2 Slots (`-np 2`)	Concurrent multi-worker support
Context Window per Slot	16,384 tokens (`-c 32768`)	16k tokens per worker for complex codebase analysis
KV Cache Quantization	`q8_0` (Key & Value)	High numerical precision without attention drift
Batch Optimization	`-b 2048 -ub 1024`	High-throughput AST batch ingestion

3. Hardware Memory Topology & Headless NixOS Mode#

On a 16 GB VRAM GPU, desktop display servers (KDE Plasma 6, KWin Wayland, browser hardware acceleration) consume approximately 1,100 to 1,200 MiB of VRAM at idle:

flowchart TD
    subgraph S1["🖥️ Desktop Session (KDE Plasma 6)"]
        D1["Display Server (Wayland/KWin): ~1.2 GB"]
        V1["Usable Inference VRAM: ~14.8 GB (90%)"]
    end
    subgraph S2["⚡ Headless NixOS (headless-ai)"]
        D2["Display Server: 0 MB"]
        V2["100% Usable VRAM: 16.0 GB (Zero Spillover)"]
    end

Boot Profile	Display Server Allocation	Usable VRAM for Nomos Code Inference	Model Offload Capacity
Desktop Session (KDE Plasma 6)	~1.2 GB (Wayland / Compositor)	~14.8 GB (90.4%)	1x Qwen3.8-27B (Q3_K_M) + 8k context
Headless NixOS (`headless-ai`)	0 MB (Console Mode)	~16.0 GB (100.0%)	2x Qwen3.8-27B (Q3_K_M) + 16k context slots

NixOS Specialisation Configuration#

To enable zero-overhead autonomous batch execution, a dedicated headless-ai profile is declared in the host NixOS configuration:

specialisation."headless-ai".configuration = {
  # Reclaim ~1.2 GB VRAM by disabling GUI display layers
  services.displayManager.sddm.enable = lib.mkForce false;
  services.desktopManager.plasma6.enable = lib.mkForce false;
  services.xserver.enable = lib.mkForce false;

  # Boot directly to text multi-user target
  systemd.defaultUnit = "multi-user.target";
};

This configuration exposes a dedicated boot entry in systemd-boot:

Default: Full graphical KDE Plasma 6 desktop for daily interactive engineering.
headless-ai: Pure console environment with 100% VRAM dedicated to the Tier 2 Nomos Code Engine.

4. Nomos Code Integration & Concurrency Guardrails#

The local engine integrates directly into the Nomos Sovereign Tier 2 ecosystem:

Transactional Mailbox Guard: The hardware concurrency governor (nomos-code/src/engine/governor.go) strictly caps active worker leases to 2 concurrent workers, matching the 2 physical llama-server slots (-np 2).
Reasoning Effort Dispatch: Native support for chain-of-thought <think> traces, with tunable reasoning_effort (low, medium, high, none) based on task burden.
Automated Zombie Reaper: 5-minute sub-second heartbeat leases ensure that crashed workers automatically release their slot back to the mailbox queue.