# Benchmarking local models: what 600+ graded runs actually showed

Results from a homegrown graded-task harness run against ~25 locally served models on a single 121.7 GB unified-memory box. Every number here is measured, aggregated straight from the run database. No vendor claims, no published leaderboard scores.

## Method, briefly

Suites of coding and reasoning tasks, each with test cases. Model-generated code is **executed** in a separate process with a hard timeout, and graded on whether the cases pass. Two metrics come out:

- **case_pass_rate** — fraction of individual test cases passed. The headline number below.
- **perfect_rate** — fraction of *runs* where every case passed. Harsher, and more honest about whether you would actually ship the output.

Each model is run over a suite multiple times, so mean-vs-best spread is visible. `tok/s` is mean decode throughput on the same machine, so it is comparable across rows.

## coding-v3 — 47 tasks, 285 runs, 23 models

| model | runs | mean pass | best | perfect | tok/s |
|---|---|---|---|---|---|
| qwen3.8:27b | 2 | **0.966** | 0.973 | 0.947 | 32.4 |
| gemma4:26b | 14 | 0.915 | 0.997 | 0.876 | 51.0 |
| DeepSeek-V4-Flash (IQ2_XXS) | 37 | 0.900 | 0.903 | 0.883 | 15.3 |
| ornith-nvfp4 | 12 | 0.899 | 0.928 | 0.822 | 57.3 |
| qwen3.6:35b-a3b | 12 | 0.876 | 0.876 | 0.787 | **74.7** |
| qwen3-coder-next | 4 | 0.875 | 0.889 | 0.832 | 52.4 |
| qwen3-coder:30b | 12 | 0.860 | 0.867 | 0.684 | 63.2 |
| devstral-small-2:24b | 12 | 0.854 | 0.864 | 0.729 | 12.7 |
| devstral-2:123b | 12 | 0.843 | 0.849 | 0.755 | 2.5 |
| qwen3-coder:tuned | 12 | 0.826 | 0.826 | 0.723 | 60.6 |
| laguna-xs-2.1 | 12 | 0.820 | 0.862 | 0.702 | 69.1 |
| Qwen-AgentWorld-35B-A3B (Q4_K_M) | 12 | 0.812 | 0.812 | 0.745 | 63.3 |
| qwen3.6:27b | 12 | 0.799 | 0.799 | 0.787 | 12.1 |

## full-v2 — 119 tasks, the widest suite

| model | runs | mean pass | best | perfect | tok/s |
|---|---|---|---|---|---|
| gemma4 (vLLM) | 1 | **0.960** | 0.960 | 0.896 | 28.4 |
| qwen3.8:27b | 1 | 0.949 | 0.949 | 0.882 | 29.5 |
| muse-glimmer:30b | 1 | 0.949 | 0.949 | 0.891 | 65.2 |
| gemma4:26b | 3 | 0.929 | 0.936 | 0.845 | 56.7 |
| qwen3-coder-next | 3 | 0.880 | 0.891 | 0.815 | 58.2 |
| qwen3.6:35b-a3b | 2 | 0.863 | 0.863 | 0.714 | 66.4 |
| Qwen-AgentWorld-35B-A3B | 1 | 0.849 | 0.849 | 0.697 | 59.3 |
| nemotron (vLLM) | 4 | 0.828 | 0.872 | 0.715 | 70.9 |
| nemotron3-nano | 1 | 0.730 | 0.730 | 0.563 | 52.2 |
| nemotron-3.5-lightning:30b-a3b | 5 | 0.727 | 0.861 | 0.640 | 75.5 |

## Five things the data says

### 1. A benchmark everything passes measures nothing

coding-v1 (18 tasks) has **four models tied at exactly 1.000**, with several more above 0.92. It is saturated and cannot rank anything.

| coding-v1 | mean pass |
|---|---|
| gemma4:26b, llama4, qwen3-coder-next, qwen3-coder:tuned | 1.000 |
| llama3:70b-instruct | 0.949 |
| nemotron-3.5-lightning:30b-a3b | 0.947 |

Widening to 29 tasks (coding-v2) and then 47 (coding-v3) reopens the spread — top to bottom goes from a 0.05 band to a 0.17 band. **If your eval saturates, the honest response is harder tasks, not a better-sounding average.**

### 2. Slower is not better — the correlation is roughly zero

`devstral-2:123b` scores 0.843 at **2.5 tok/s**. `qwen3.6:35b-a3b` scores **0.876 at 74.7 tok/s** — 30× the throughput *and* a better score, on the same tasks and the same machine. `qwen3.6:27b` scores 0.799 at 12.1 tok/s while `laguna-xs-2.1` scores 0.820 at 69.1.

Whatever the big dense model is buying, it is not visible here.

### 3. A 2-bit quant of a large model beat 4-bit quants of smaller ones

`DeepSeek-V4-Flash` at **IQ2_XXS** — a ~2-bit GGUF — placed third on coding-v3 at 0.900 mean over 37 runs, ahead of several Q4 and NVFP4 models. Its `perfect_rate` of 0.883 is the second-highest in the table, beating `gemma4:26b` (0.876) which outscores it on case pass rate.

For a fixed memory budget, squeezing a bigger model harder appears to beat running a smaller model gently. One data point, one architecture — but a strong one, and it runs against the usual "never go below 4-bit" advice.

### 4. Rankings do not survive a change of suite

| model | coding-v3 | full-v2 | langpref-v1 | agent-v1 |
|---|---|---|---|---|
| qwen3.6:35b-a3b | 0.876 | 0.863 | 0.917 | **0.722** (last) |
| nemotron-3.5-lightning:30b-a3b | — | **0.727** (last) | **1.000** (first) | — |
| gemma4:26b | 0.915 | 0.929 | 0.944 | 0.938 |
| qwen3.8:27b | **0.966** (first) | 0.949 | **0.750** (last) | — |

`nemotron-3.5-lightning` is first on language-preference and last on the broad suite. `qwen3.8:27b` is first on coding and last on language-preference. Only `gemma4:26b` is consistently near the top everywhere.

A single-number leaderboard would have picked a different winner for four different reasons. Pick the suite that matches your workload, or accept you are measuring someone else's.

### 5. Run-to-run variance is large enough to invert rankings

`gemma4:26b` on coding-v3 across 14 runs: mean **0.915**, best **0.997**. `nemotron-3.5-lightning` on full-v2 across 5 runs: mean **0.727**, best **0.861** — a 13-point swing on identical inputs.

Anything reported from a single run, anywhere, is inside the noise. Several rows in the tables above are single runs and should be read that way — `qwen3.8:27b`'s 0.966 is two runs, not a settled result.

## Two harness lessons worth stealing

**Cap output generously.** Grading with `max_tokens` at 512 truncates long answers mid-function, and truncation grades as failure. Every score from that era was depressed by an artifact of the harness, not the model. 1024 minimum for graded coding work.

**Serving engine is a variable, not a constant.** The same model appears twice in the vision suite under two serving configurations at **65.6 vs 138.0 tok/s** — a 2.1× throughput difference with no change of weights. Record the engine alongside the model or the numbers are not comparable, including with your own earlier numbers.

## What is not here

No long-context axis. Every suite above uses short prompts, so none of these numbers say anything about behaviour at 100k+ tokens, which is where several of these models are actually deployed. That gap is the next thing worth measuring, and it is the one most likely to reorder the tables.