How many requests at once

The editing recipe shipped --parallel 1 from the start. It now ships 2. This page is why, and why the answer is not the same as the long-context recipe's, even though the knob looks the same.

The thing that makes llama.cpp different

--ctx-size is divided by --parallel. Each slot gets a fixed, private share of the context, decided at startup:

Read off /slots on a running server, and the products come out exactly at 262,144 every time.

vLLM does not work this way. It keeps one shared KV pool — 654,635 tokens at GPU_MEM=0.85 — and hands it out on demand, which is why raising --max-num-seqs there costs nothing when the server is quiet and why that recipe ships 16. Here a slot count is a standing claim on the context, whether or not anyone is using it.

So the trade is real and it runs in one direction: concurrency is bought with the maximum prompt any single request can hold.

What two slots cost, and what they buy

Nothing, and 1.2–1.3×.

Cost, at one caller — three one-slot repeats bound the noise floor at 6.5%, and the two-slot figure lands inside it:

Buy, under load — paired runs, same box, same projector, slot count the only variable:

One measurement subtlety worth carrying: the first serve-short-c16 one-slot run started at 0.06% page-cache residency, straight after a server restart, against its two-slot twin's 72.22%. That confound was the same size as the effect, so the one-slot run was repeated warm (75.26%) and came back at 36.40 against the cold run's 36.98 — 1.6% apart. Residency did not distort it, and the repeat also bounds the noise on that workload.

131,072 tokens is still four times the largest context this recipe has ever been measured at, so two slots give away a ceiling nobody was reaching.

But the ceiling is real, and here is what hitting it looks like. The atlas's prefill-128k workload sends prompts that come to 161,064 tokens on this tokenizer. At two slots every one of the ten requests was refused outright:

{"error":{"code":400,
  "message":"request (161064 tokens) exceeds the available context size (131072 tokens), try increasing it",
  "type":"exceed_context_size_error","n_prompt_tokens":161064,"n_ctx":131072}}

0 of 10 completed. The same workload fits at PARALLEL=1, where the single slot holds all 262,144. So the rule is not "two slots are free" — it is two slots are free up to a 131,072-token prompt, and a hard 400 above it. If you edit files that large, set PARALLEL=1 and take the single-stream configuration; nothing else in the recipe changes.

The cell is published as a failure rather than dropped, because a configuration that cannot run a workload is a fact about the configuration.

What more slots cost

More than they buy, and past a point the server stops working.

sweep-parallel-1-64-i1k-o256-v1 offers 1→64 concurrent requests against a fixed server. Run at 8 slots and at 64:

Harness headline for the whole sweep: 43.62 tok/s at 8 slots, 27.35 at 64.

64 slots is worse than 8 at every single level, and it collapsed at 16 offered requests: 63 of 64 failed with RemoteProtocolError: peer closed connection without sending complete message body. It was not a crash — no assert, no CUDA error, no out-of-memory in the server log. The log is wall-to-wall:

srv alloc: - making room for prompt cache entry, removing oldest entry (size = 400.407 MiB)
srv alloc: - making room for prompt cache entry, removing oldest entry (size = 405.919 MiB)

Prompt-cache thrashing, with generation falling to 1.3–1.9 tok/s and three-second windows at 0.30 t/s, so the workload's own 300-second budget expired underneath it. (The server process was also gone when the box was checked half an hour later, with the log ending mid-request and no shutdown line. We could not confirm what removed it, so we are not claiming it crashed.)

Eight slots is the last setting on this list that serves its own concurrency well — up to 8 callers at ~3 s to first token — but it costs 87.5% of the context ceiling to get there, and first-token latency past its slot count is a cliff, not a slope: 3.12 s at 8 offered, 50.96 s at 16.

The reading

• 2 is the default. Free when idle, 1.2–1.3× when two people or two tools are talking to it, and 131k of context left over.
• Raise it only if you know your prompts are short. 8 slots serves 8 concurrent callers properly, at 32k each. That is a coding-assistant configuration, not a long-document one.
• Do not go looking for vLLM's numbers here. The long-context recipe serves 16 concurrent at 96–109 tok/s aggregate with first tokens under 2.7 s. This engine cannot do that at any slot count, because the context is partitioned rather than pooled. If you need real concurrency, that is the recipe for it.
• --parallel 64 is not a configuration, on this model, on this box.

One thing this page cannot tell you: whether the 1.24–1.30× holds on llama.cpp master. Every figure here was measured on the pinned pre-merge commit 035e227 plus the patches in patches/. See sources.md.

Cell data: inference-atlas.