Benchmarking local models: what 600+ graded runs actually showed
Results from a homegrown graded-task harness run against ~25 locally served models on a single 121.7 GB unified-memory box. Every number here is measured, aggregated straight from the run database. No vendor claims, no published leaderboard scores.
Method, briefly
Suites of coding and reasoning tasks, each with test cases. Model-generated code is executed in a separate process with a hard timeout, and graded on whether the cases pass. Two metrics come out:
- case_pass_rate — fraction of individual test cases passed. The headline number below.
- perfect_rate — fraction of runs where every case passed. Harsher, and more honest about whether you would actually ship the output.
Each model is run over a suite multiple times, so mean-vs-best spread is visible. tok/s is mean decode throughput on the same machine, so it is comparable across rows.
coding-v3 — 47 tasks, 285 runs, 23 models
| model | runs | mean pass | best | perfect | tok/s |
|---|---|---|---|---|---|
| qwen3.8:27b | 2 | 0.966 | 0.973 | 0.947 | 32.4 |
| gemma4:26b | 14 | 0.915 | 0.997 | 0.876 | 51.0 |
| DeepSeek-V4-Flash (IQ2_XXS) | 37 | 0.900 | 0.903 | 0.883 | 15.3 |
| ornith-nvfp4 | 12 | 0.899 | 0.928 | 0.822 | 57.3 |
| qwen3.6:35b-a3b | 12 | 0.876 | 0.876 | 0.787 | 74.7 |
| qwen3-coder-next | 4 | 0.875 | 0.889 | 0.832 | 52.4 |
| qwen3-coder:30b | 12 | 0.860 | 0.867 | 0.684 | 63.2 |
| devstral-small-2:24b | 12 | 0.854 | 0.864 | 0.729 | 12.7 |
| devstral-2:123b | 12 | 0.843 | 0.849 | 0.755 | 2.5 |
| qwen3-coder:tuned | 12 | 0.826 | 0.826 | 0.723 | 60.6 |
| laguna-xs-2.1 | 12 | 0.820 | 0.862 | 0.702 | 69.1 |
| Qwen-AgentWorld-35B-A3B (Q4_K_M) | 12 | 0.812 | 0.812 | 0.745 | 63.3 |
| qwen3.6:27b | 12 | 0.799 | 0.799 | 0.787 | 12.1 |
full-v2 — 119 tasks, the widest suite
| model | runs | mean pass | best | perfect | tok/s |
|---|---|---|---|---|---|
| gemma4 (vLLM) | 1 | 0.960 | 0.960 | 0.896 | 28.4 |
| qwen3.8:27b | 1 | 0.949 | 0.949 | 0.882 | 29.5 |
| muse-glimmer:30b | 1 | 0.949 | 0.949 | 0.891 | 65.2 |
| gemma4:26b | 3 | 0.929 | 0.936 | 0.845 | 56.7 |
| qwen3-coder-next | 3 | 0.880 | 0.891 | 0.815 | 58.2 |
| qwen3.6:35b-a3b | 2 | 0.863 | 0.863 | 0.714 | 66.4 |
| Qwen-AgentWorld-35B-A3B | 1 | 0.849 | 0.849 | 0.697 | 59.3 |
| nemotron (vLLM) | 4 | 0.828 | 0.872 | 0.715 | 70.9 |
| nemotron3-nano | 1 | 0.730 | 0.730 | 0.563 | 52.2 |
| nemotron-3.5-lightning:30b-a3b | 5 | 0.727 | 0.861 | 0.640 | 75.5 |
Five things the data says
1. A benchmark everything passes measures nothing
coding-v1 (18 tasks) has four models tied at exactly 1.000, with several more above 0.92. It is saturated and cannot rank anything.
| coding-v1 | mean pass |
|---|---|
| gemma4:26b, llama4, qwen3-coder-next, qwen3-coder:tuned | 1.000 |
| llama3:70b-instruct | 0.949 |
| nemotron-3.5-lightning:30b-a3b | 0.947 |
Widening to 29 tasks (coding-v2) and then 47 (coding-v3) reopens the spread — top to bottom goes from a 0.05 band to a 0.17 band. If your eval saturates, the honest response is harder tasks, not a better-sounding average.
2. Slower is not better — the correlation is roughly zero
devstral-2:123b scores 0.843 at 2.5 tok/s. qwen3.6:35b-a3b scores 0.876 at 74.7 tok/s — 30× the throughput and a better score, on the same tasks and the same machine. qwen3.6:27b scores 0.799 at 12.1 tok/s while laguna-xs-2.1 scores 0.820 at 69.1.
Whatever the big dense model is buying, it is not visible here.
3. A 2-bit quant of a large model beat 4-bit quants of smaller ones
DeepSeek-V4-Flash at IQ2_XXS — a ~2-bit GGUF — placed third on coding-v3 at 0.900 mean over 37 runs, ahead of several Q4 and NVFP4 models. Its perfect_rate of 0.883 is the second-highest in the table, beating gemma4:26b (0.876) which outscores it on case pass rate.
For a fixed memory budget, squeezing a bigger model harder appears to beat running a smaller model gently. One data point, one architecture — but a strong one, and it runs against the usual "never go below 4-bit" advice.
4. Rankings do not survive a change of suite
| model | coding-v3 | full-v2 | langpref-v1 | agent-v1 |
|---|---|---|---|---|
| qwen3.6:35b-a3b | 0.876 | 0.863 | 0.917 | 0.722 (last) |
| nemotron-3.5-lightning:30b-a3b | — | 0.727 (last) | 1.000 (first) | — |
| gemma4:26b | 0.915 | 0.929 | 0.944 | 0.938 |
| qwen3.8:27b | 0.966 (first) | 0.949 | 0.750 (last) | — |
nemotron-3.5-lightning is first on language-preference and last on the broad suite. qwen3.8:27b is first on coding and last on language-preference. Only gemma4:26b is consistently near the top everywhere.
A single-number leaderboard would have picked a different winner for four different reasons. Pick the suite that matches your workload, or accept you are measuring someone else's.
5. Run-to-run variance is large enough to invert rankings
gemma4:26b on coding-v3 across 14 runs: mean 0.915, best 0.997. nemotron-3.5-lightning on full-v2 across 5 runs: mean 0.727, best 0.861 — a 13-point swing on identical inputs.
Anything reported from a single run, anywhere, is inside the noise. Several rows in the tables above are single runs and should be read that way — qwen3.8:27b's 0.966 is two runs, not a settled result.
Two harness lessons worth stealing
Cap output generously. Grading with max_tokens at 512 truncates long answers mid-function, and truncation grades as failure. Every score from that era was depressed by an artifact of the harness, not the model. 1024 minimum for graded coding work.
Serving engine is a variable, not a constant. The same model appears twice in the vision suite under two serving configurations at 65.6 vs 138.0 tok/s — a 2.1× throughput difference with no change of weights. Record the engine alongside the model or the numbers are not comparable, including with your own earlier numbers.
What is not here
No long-context axis. Every suite above uses short prompts, so none of these numbers say anything about behaviour at 100k+ tokens, which is where several of these models are actually deployed. That gap is the next thing worth measuring, and it is the one most likely to reorder the tables.