Editing recipe — llama.cpp + context-copying speculation

For coding agents that rewrite files you hand them.

This is the specialist. On the task it is built for it is more than twice as fast as anything else measured on this box; on writing new text it is the slower of the two recipes. If you mostly chat with the model or feed it long documents, use ../vllm-longctx instead.

Measured

Cold page cache — the n-gram table only 17.7% resident, i.e. the state you actually boot into.

That is a 3.2× spread across four tasks with one model and one server. The number depends almost entirely on how much of the answer already exists in the question.

Why

--spec-type ngram-mod drafts candidate spans by looking for repetition in your prompt. Hand it a file and ask for one change, and nearly every token of the answer is already in front of it — it lifts 60-token spans in a single verified step. Ask for something new and it has nothing to copy, so you get the model's honest unaccelerated rate of about 28.

Speculation is exact: the model verifies every drafted token, so output is byte-identical to running without it. You are not trading quality for speed.

Install

./run.sh setup    # builds llama.cpp, fetches ~105 GB
./run.sh serve    # starts on http://localhost:8000/v1
./run.sh bench    # measures the four tasks above

From the repository root you can also use ../../run.sh edit setup|serve|bench|all.

Settings

Two things that are not worth doing

Do not bother warming the table. The recipe used to recommend it. Re-measured on the current build with the canreuse-qwen4exp patch applied, warming buys nothing: prose is 27.8 tok/s at both 17.7% and 58.1% residency, identical to the decimal, and the medians moved in both directions by less than the run-to-run spread. tools/warm_table.py is still here for A/B work, but the boot ritual is unnecessary.

Do not drop to a lower-bit K-quant for speed. UD-Q3_K_XL moves 19% fewer bytes per token and is 14% slower on prose (24.0 vs 27.8). K-quant dequantisation costs more than the memory traffic it saves, because this configuration is not bandwidth-bound — see ../../docs/ruled-out.md. Choose a smaller quant for disk space or quality reasons, not for speed.

Vision

setup fetches mmproj-F16.gguf (~0.9 GiB) from the same Unsloth repo as the quants, and serve passes --mmproj when it finds it. With it, this recipe scores 0.967 on the atlas image eval — the same score as the NVFP4 long-context recipe, with identical per-category splits, and about 2.6× slower over the 60 items. MMPROJ=none turns it off; every number in this repository that predates 2026-08-30 was measured without it. See ../../docs/vision.md.

Known issues

• Two requests at a time, since 2026-08-30. --parallel 2 — measured at 1.24× on 16 concurrent short requests and 1.30× on 8 concurrent chat requests, and free when only one caller is present. The cost is context: llama.cpp divides --ctx-size across slots, so two slots means 131,072 tokens per request rather than 262,144 — and a prompt above that is refused with a 400, not truncated: the atlas prefill-128k workload (161,064 tokens on this tokenizer) fails all ten requests at two slots and fits at one. PARALLEL=1 restores the full window. Do not raise it far — at 64 slots the server thrashes its prompt cache and loses requests. ../../docs/parallel.md has the whole curve.
• Quantized KV aborts on this architecture. Keep it at f16; at ~24 KB/token it is cheap.
• No MTP. The GGUF converter drops the model's trained draft head — we confirmed zero MTP tensors across all four shards. If you want that, use the other recipe.
• The default build is a pinned pre-merge commit. PR #27742 merged upstream on 2026-08-27, so qwen4exp is in master now. run.sh still defaults to the PR commit 035e227 plus the patches, because that is the build every number in this repo was measured on. REF=master ./run.sh setup builds the merged code instead, with no patches — see ../../docs/sources.md.

Patches

Two, in ../../patches/. Both are upstream as of master — they apply to the pinned pre-merge commit and are skipped automatically under REF=master.

• canreuse-qwen4exp.patch — implements can_reuse() for the qwen4exp graph inputs so CUDA graph capture engages at all. Without it the graph is rebuilt every token. This is why our prose figure (27.8) is well above the 19–25 tok/s others report for the same quant on the same box. Master implements can_reuse() on both graph inputs itself.
• rowband-ple-quant.patch — only needed if you quantize the model yourself. Staging the 51.2B table through an f32 buffer needs 204.8 GB; this dequantizes in ≤2 GiB row bands. Master processes rows in slabs bounded by max_buf_size, which does the same thing for every tensor rather than for this one.