Workloads

A workload is a pinned answer to "what exactly was run". One JSON file per workload, schema in schemas/workload.schema.json, shape summarised in SPEC §4.

A result is only comparable to another result when both name the same workload_id. That is the whole reason this directory exists: the model, the quantization, the engine and the flags are recorded in the result, and the question that was asked of them is recorded here.

Immutability

A published workload is never edited. Not the concurrency, not the number of requests, not the dataset it points at, not a typo in a number. Every existing result that references it was produced under the old definition, and silently changing it turns those numbers into a lie.

immutable: true and the mandatory -vN suffix say so in the data. The only edits ever allowed on a published file are the ones that cannot change what runs: name, description, notes.

Proposing a v2

1. Copy the file to <same-name>-v2.json and change id to match the filename.
2. Set "supersedes": "<the v1 id>".
3. Change what you meant to change, set created to today.
4. Leave the v1 file exactly as it is. It keeps its results; the site can show them side by side and mark the v1 cells stale rather than wrong.
5. If the change is really "the dataset changed", the dataset needs a -v2 too (see datasets/README.md) — a workload pointing at different bytes is a different workload.

Naming: <kind>-<shape>-v<N>, lower-case kebab, with the shape readable at a glance (c8 = concurrency 8, i1k = ~1k input tokens, o256 = 256 output tokens).

Kinds

image is the one kind whose shape lives entirely in params: width, height, steps, seed, images_per_request and ref_images are all required there, because the same prompts at another size or step count are a different measurement rather than the same one run differently. Its dataset supplies prompts and, for the edit workloads, the reference images they condition on. Scored image suites are not this kind: they are kind: eval with an image scorer (ocr, clip, rgba, fidelity), they produce a scores block like every other eval, and their render spec is frozen per case in the dataset row.

metrics_required lists the metric paths a result must carry non-null. Dotted paths address into a distribution (ttft_ms.p50). For kind: eval the three required names — accuracy, success_rate, avg_latency_ms — live in the result's scores block rather than in metrics; success_rate there means "the share of requests that completed at all", independently of whether the answers were right.

All workloads

Small-model tier

Three workloads (2026-09-22) for 1-3B models, chosen from what the existing small-model results show rather than from a wish list. The evidence is google/gemma-4-E2B-it on two engines (vLLM bf16 on a GB10, LM Studio MLX 4-bit on an M2 Max):

• Saturated for small models already: eval-tools-v1 1.00 / 1.00, eval-format-v1 1.00 / 0.87, eval-json-v1 0.93 / 0.99 (its extraction category 40 of 40), eval-multilingual-v1 0.96 / 0.96, eval-knowledge-v1 0.94 / 0.98. Short extraction, classification and JSON-from-a-sentence suites would land in the same place, so there are none here.
• Still spreading, so no "easy tier" is needed: eval-math-v1 0.62 / 0.98 and eval-reasoning-v1 0.18 / 0.73. The v1 suites are the easy tier for this size. No small model has run a -v2 suite yet; run them before concluding they sit at the floor.
• Batch-1 latency is covered: serve-single-i256-o256-v1 already records TTFT and TPOT for one short request at a time (45 ms and 25.5 ms p50 for the vLLM run). A tiny-prompt variant would differ by a few milliseconds of prefill.
• Not covered: throughput past 64 streams, reading a provided passage without inventing (closed-book eval-hallucination-v1 is 0.07 for the MLX run, and nothing measures the grounded case), and tool use that is harder than copying arguments.

The eval runtimes are wide because thinking decides them: without it the answers are a few dozen tokens each; with it a small model can spend a few hundred to a thousand tokens per item at 4 streams. The sweep sends 2,112 requests of 256 output tokens.

Running them: the sweep only measures a point if the engine admits that many sequences (vLLM --max-num-seqs, SGLang --max-running-requests, llama.cpp --parallel), and prompts repeat within a point from 64 streams up, so run it with prefix caching off or say that it was on. The tools suite needs the engine's tool-call parser enabled, and the parser is part of the configuration.

Conventions a runner must honour

• params is the contract, description is prose. Anything a runner needs is a key in params, never only in the text.
• input_tokens is nominal. It says which prompts to pick, not what the tokenizer will produce. Record the real count in metrics.input_tokens_total.
• dataset_buckets (serving/sweep) filters prompts-* rows by length bucket; dataset_categories and dataset_target_tokens filter eval rows.
• Sweep points. For the two parallelism sweeps, requests per point are max(params.num_requests, concurrency * params.requests_per_concurrency), so a low-concurrency point is not dominated by warmup. params.concurrency is the first point of the axis and is otherwise ignored.
• serve-prefix-c16-v1 must send all rows of one prefix_id back to back (params.group_by), must not shuffle, and excludes warmup_per_group requests per group from the aggregate. Run it twice on the same engine, with and without the prefix-cache flag, and compare ttft_ms.
• longctx-depth-sweep-v1 checks the needle at every point. A wrong needle answer is a failed request — it lowers success_rate and adds a failures[] entry with category malformed-output. A point that exceeds the served max-model-len is recorded as a failure with category context-overflow, never omitted, because "did not fit" is a result.
• Eval concurrency does not change the score, only the wall clock. It is 4 everywhere except eval-longctx-v1, which runs at 1 to keep KV cache pressure honest, and eval-longgen-integrity-v1, where concurrency 1 is part of the configuration under test: batching changes the scheduler, the KV-cache layout and the attention path, so a run at another concurrency measures something else.
• eval-longgen-integrity-v1 scores token integrity, not correctness. accuracy is the share of 150-to-290-line generations that came back with no spliced token. A clean run of 36 is not proof of absence: at a one-in-five per-generation rate a clean pass is a plausible outcome for an affected build, so report the run rather than the absence.
• Image workloads run one at a time. concurrency is 1 and the runner does not parallelise: a transformer over thousands of latent tokens saturates a single GPU on its own, so a second request in flight measures the queue. Each prompt is rendered repeat times and s_per_image is the distribution over every measured render; warmup_requests are rendered first and excluded. Where a lane reports its own inference time (timing.total_ms) that is used instead of the wall clock, and which of the two applied is in raw.payload per render.
• guidance: null means the lane's own default. Guidance above 1 runs a second transformer forward per denoising step, so two lanes whose defaults differ are doing different amounts of work; record the effective value in the result's args before comparing seconds per image.
• No generated image is ever stored. The runners render into a temporary directory and delete it; results carry numbers, and for the fidelity suite a 64-bit perceptual hash. scores.items[].predicted is null on image suites — an OCR transcription is the generated content read back, and it does not belong in the repository either.
• The fidelity suite needs a local bf16 reference. atlas-bench t2i-reference --spec bf16.json --out DIR renders it once on the same box; every quantized run then passes --reference-bundle DIR. The scorer compares the render digest first and refuses to score a case whose reference came from a different spec.
• params.reasoning: "default" means "leave the engine's own default alone". Whatever it was, record it in the result's args — reasoning settings move eval scores and latency more than most flags.
• Scorer names. eval.scorer uses the kebab-case vocabulary of workload.schema.json (code-exec); dataset rows use the snake_case scorer module name from SPEC §8 (code_exec). They mean the same thing. Always prefer the row's own scorer when a dataset mixes them.

Adding a workload

A new workload is a PR that adds one file. Before writing it, check the table above: a workload that differs from an existing one only in num_requests splits the coverage map for no gain. Good reasons to add one: a new axis (a concurrency the sweeps do not reach), a new dataset, or a shape that models a real deployment nobody has captured yet.

Checklist:

• id ends in -v1, filename equals id + ".json";
• dataset_id exists under datasets/;
• metrics_required only names metrics the runner can actually fill for that kind;
• immutable: true, created today, supersedes null;
• pnpm validate passes.