What the atlas has measured

Every figure in this repository comes from bench/portable_bench.py — one harness, written here, run by the people who wrote the recipes. That is a conflict of interest, so both recipes have also been run through the pinned workloads of inference-atlas: a different harness, a different prompt set, fixed workload definitions, and results that live as one JSON file per measurement with the hardware captured rather than typed.

This page is the index. It says what has been measured there, what it changed here, and what is still missing.

Why it is worth having

The atlas is not a leaderboard and does not rank anything. It records evidence: a cell is a configuration on a piece of hardware, and its colour says how well attested that cell is, not how fast it is. For this repository the value is narrower and more useful — it is a second opinion on our own numbers, taken with a harness we did not write.

Three things came out of it that this repository did not have before, and one that it had wrong.

1. Page-cache residency is a hidden variable, and it moves

A quarter of the GGUF checkpoint is served from NVMe by design. How much of the 26.82 GiB n-gram table happens to be in the page cache is therefore an independent variable in every measurement — and nothing in the recipe was reporting it. Every atlas cell for the editing recipe now carries a mincore(2) reading taken immediately before the workload.

What that produced:

The last row is the important one: a long workload warms itself, so it spends most of its own duration warm no matter where it started. And the third row is why run.sh defers its warmer until after the server is serving — loading the model streams the whole GGUF through a 121 GiB box and evicts the table region on the way. That ordering fix came from @rumi-ali and the atlas runs are what confirmed it.

Decode at 25.88% residency measured 34.12 tok/s, which sits inside the 34.99–37.43 spread of two otherwise identical cold runs. So residency is real and worth recording, and it is not visible above the noise on this workload — both statements at once.

The noise floor is 6.5% on serve-single, established by repeating the same cell three times. Nothing smaller than that should be read as a difference.

2. The long-context recipe was capped by a scheduler default

The upstream container ships --max-num-seqs 2. The atlas ran both arms of the A/B — same workloads, same box, one variable — and the result changed what this repository ships:

• All four concurrency-1 cells are identical between the arms. The cap costs nothing when the server is quiet.
• Every loaded cell is 1.2–2.7× better with it lifted, and time-to-first-token is where it shows: eight callers with 2k prompts wait 150.8 s for a first token at two slots, 2.4 s at sixty-four.
• The 64-caller cell at SEQS=2 lost 127 of 640 requests to the workload's own timeout — the only cell in either arm that failed requests.

Full table and the reasoning: ../recipes/vllm-longctx/README.md. The recipe now ships SEQS=16.

3. Vision, where the docs were simply wrong

This repository said images were a reason to choose the long-context recipe, because the GGUF has no vision tensors. The tensors part is true; the conclusion was not. With the projector Unsloth publishes alongside the quants, the editing recipe scores 0.967 on eval-vision-v1 — the same 58/60 as the NVFP4 checkpoint, with identical per-category splits.

vision.md has the comparison, including the two items each engine misses.

4. Two slots cost nothing when nobody is waiting

The editing recipe forced --parallel 1 until 2026-08-30. Running the same workloads at two slots, with the vision projector loaded, gives a like-for-like pair at concurrency 1:

The two-slot decode figure lands in the middle of the three one-slot repeats — inside the 6.5% noise floor, so the second slot and the 0.9 GiB projector cost nothing measurable when only one caller is present. prefill-8k is 6.7% higher, but there is only one run on each side and no repeat to bound prefill noise, so that one is suggestive, not established.

They do batch under load, which the paired loaded workloads then showed: 1.24× on serve-short-c16 and 1.30× on serve-chat-c8, all requests completing on both sides. The default moved to 2 on the strength of it. Going further does not work — 64 slots is worse than 8 at every concurrency and loses 63 of 64 requests at 16 offered, because llama.cpp partitions the context rather than pooling it. parallel.md has the curve.

5. A default we could not justify, changed by an outside report

serve.sh disabled prefix caching and explained it with an unsourced claim about a GB10 kernel bug. @faparicior reported it working; measuring it settled the question. Same box, same SEQS=64, only the flag differing, on serve-prefix-c16-v1 — the pinned workload that groups its requests by shared prefix:

Correctness was checked before throughput: three identical requests at temperature 0 returned byte-identical answers with a real cache hit behind them. The flag now defaults on, and PREFIX_CACHE=0 reproduces every prefill figure published here — all of which were cache-free.

Where the numbers live

Each file records the engine build, the exact flags, the captured hardware fingerprint, the workload definition, and a list of gotchas — including the ones that say a given cell is not comparable with another.

Where the two harnesses disagree

They mostly do not, but the differences are honest and worth stating rather than averaging away.

Flat in both, which is the claim that matters. The atlas runs 5–10% lower throughout; the prompt content, the warmup handling and max-num-seqs all differ. The honest range across both is ~2,030–2,460.

Still missing

• Editing-recipe cells above 32k context, and its decode-at-depth curve.
• Any cell for a quant other than UD-Q4_K_XL.
• Anything measured on llama.cpp master. Every editing-recipe cell was taken on the pre-merge commit 035e227 plus the patches in patches/; the PR merged on 2026-08-27 and nothing has been re-measured since.
• A vision throughput number for either recipe. eval-vision-v1 gives a score and a wall clock, not tok/s.