Datasets

Every byte of test data in Inference Atlas is authored in this repository — synthetic, generated by the scripts in _gen/, and MIT licensed (SPEC §0.6). No third-party corpus is mirrored here, nothing was scraped, and no model output was used as data. That is what lets anyone fork the project, run the same workloads and compare numbers without a licence question.

A dataset is pinned to the workloads that reference it. Published datasets are immutable: changing one means a new -v2 id, because two results are only comparable when they saw the same bytes.

Index

eval-longgen-integrity-v1 (2026-09-02) is a different axis again. Every suite above asks a short question, so a serving build that corrupts roughly one token in several thousand scores 100 % on all of them: the answer ends before the defect has room to appear. This one asks for 36 code generations of 150 to 290 lines each and scores them mechanically for spliced tokens — an identifier or number welded together from two fragments — so accuracy reads as "share of long generations that came back intact".

The -v2 rows are the second-generation eval wave (2026-09-01): the v1 capability evals saturated — several models score 98–100 % on knowledge, math, reasoning, tools and format — so v2 raises the difficulty ceiling. eval-math-v2, eval-reasoning-v2 and eval-knowledge-v2 supersede their v1 ids (which stay published and immutable); eval-science-v2 (applied physics/chemistry with every constant pinned in the prompt) and eval-commonsense-v2 (goal-tracking traps, altered classic riddles, false premises, literal-text questions) and eval-security-v2 (defensive cyber security: toy crypto, CIDR arithmetic, vulnerability classification, log forensics, incident-response traps) are new suites, versioned -v2 to mark the wave they belong to.

Licence for all of the above: MIT. Total, including the generator scripts: about 11.5 MB against a 25 MB budget, which _gen/check.py prints and enforces.

Validate the whole corpus:

uv run datasets/_gen/check.py

Token counts

Every dataset states the same heuristic in its dataset.json:

approx_tokens = ceil(sum(len(m.content) for m in messages) / 4)

It is tokenizer-independent on purpose, so a prompt's bucket does not change when you swap models. Real counts differ: for Latin-script prose the heuristic is usually within ±15 %, for source code it under-counts a little, and for Chinese and Japanese the real count runs roughly 2–4× higher. Workloads name a nominal input_tokens; results should record what the engine actually reported in metrics.input_tokens_total.

Row schemas

kind: prompts

{
  "id": "mix-0001",
  "topic": "science", // one of the dataset's topics
  "bucket": "m", // xs | s | m | l | xl | xxl, see below
  "lang": "en", // natural language of the prompt
  "approx_tokens": 597, // chars/4, includes shared_prefix when set
  "messages": [{ "role": "user", "content": "..." }],
  "shared_prefix": null, // string | null
}

Length buckets are inclusive ranges on approx_tokens:

shared_prefix is a contract: when it is non-null the harness must send it as a leading system message, [{role: "system", content: shared_prefix}, *messages]. approx_tokens already includes it. Only prompts-shared-prefix-v1 uses it.

prompts-code-v1 adds code_lang (python | typescript | go | rust | sql) and task (write | fix | explain | refactor), and its rows have a system message naming the language.

kind: eval

{
  "id": "math-0001",
  "category": "arithmetic",
  "difficulty": "easy", // easy | medium | hard
  "prompt": "...", // or "messages": [...] when a system turn matters
  "answer": "42", // string | number | object, per scorer
  "scorer": "numeric",
  "choices": ["...", "..."], // mc rows only; answer is the letter label
  "tests": "assert f(1) == 2", // code_exec rows only
  "image": "images/vis-0001.png", // vision rows only, relative to the dataset dir
  "meta": {}, // per-scorer extras, see below
}

Always use the row's own scorer. A workload's eval.scorer is only the dataset default; eval-reasoning-v1 mixes mc and exact, eval-multilingual-v1 mixes contains, mc and numeric, and the -v2 evals mix mc, exact and numeric per row.

kind: haystack

{
  "id": "hay-8k-d50",
  "kind": "single", // single | multi
  "target_tokens": 8192,
  "seed": 7008292,
  "algorithm": "haystack-v1",
  "needles": [{ "depth": 0.5, "text": "...", "answer": "48213", "line_number": 143 }],
  "question": "...",
  "answer": "48213",
  "chars": 32771,
  "lines": 331,
  "approx_tokens": 8193,
  "sha256": "…", // digest of the materialised document
  "static_file": "static/hay-8k-d50.txt", // null above 32k tokens
}

Answer extraction and scorers

Applied in this order to the raw model output, before any scorer except instruction (which needs the raw text) runs:

1. drop everything inside <think>…</think>, including an unterminated leading <think> block;
2. drop markdown code fences, keeping the fenced content;
3. if any line matches /^\s*(?:final answer|answer)\s*[:\-]\s*(.+)$/i, take the capture of the last such line and use only that;
4. strip surrounding whitespace, matching quotes, and a single trailing . or !.

One item is correct or it is not; there is no partial credit. accuracy = correct / total. A request that failed (timeout, 5xx, context overflow) counts as incorrect and appears in scores.failures and metrics.requests_failed. Output that is merely malformed — invalid JSON, a refusal, a preamble where one word was asked for — is a wrong answer, not a failed request.

eval-tools-v1

meta.tools holds OpenAI function schemas; send them as the request's tools with tool_choice: "auto" and score tool_calls[0]:

• answer.tool_call = {name, arguments} → the name must match and the parsed arguments must match, subset by default (extra optional arguments tolerated) unless meta.arguments_match is "exact". Strings compare case-insensitively after stripping; numbers compare numerically.
• answer.tool_call = null → correct only when the response contains no tool call at all. The text of the reply is not scored.

eval-longgen-integrity-v1

Rows carry a 20–25k-character synthetic JavaScript ES-module project as the prompt plus a task: write one complete new module of the stated line range, importing at least six named exports from the project, and reply with the file contents and nothing else. answer is "clean" — the expected observation is "no spliced token", not a string the model types — and difficulty orders the items by requested output length (easy ≈ 1200 output tokens, medium ≈ 1800, hard ≈ 2400, recorded exactly in meta.target_output_tokens).

meta.context_identifiers is every identifier the generated project defines or exports, computed from the generated source. The scorer needs it: without a definition set it cannot tell a spliced name from one the project supplied. The generic Answer:-line extraction step is not applied here — the output is a source file, and a line beginning answer: inside it is code.

An item is correct when the generation contains no splice. That is deliberately not a correctness check: wrong-but-intact code scores 1.0, and the other eval suites are where correctness is measured. A clean run of 36 is also not proof of absence — at an observed rate of roughly one affected generation in five, a clean pass is a plausible outcome for an affected build.

eval-vision-v1

meta.ground_truth records what was drawn (shape list, bar heights, clock hands, dice values) for debugging a disagreement; it is not used for scoring. A text-only model scores near zero — record that as a real result, because some engines silently drop image parts instead of failing.

The instruction rule DSL

eval-instruction-v1 rows carry answer = {"all": [rule, …]}. The item is correct when every rule passes against the raw output. The normative implementation is eval-instruction-v1/rules.py — evaluate(answer, text) -> (bool, failed_rules) — and python datasets/eval-instruction-v1/rules.py --self-test must pass. The harness instruction scorer must agree with it; check.py runs the self-test.

Shared definitions, all normative:

• word — a match of [A-Za-z0-9]+(?:['’-][A-Za-z0-9]+)*; don't is one word, 3.5 is two
• lines — text.strip().split("\n") with blank lines dropped
• paragraphs — re.split(r"\n\s*\n", text.strip()), empties dropped
• sentences — split on [.!?]+ followed by whitespace or end of text
• bullets — lines whose lstrip() starts with a marker followed by a space

Each row also carries meta.example_pass, a compliant answer written by the generator and verified against the rules — proof that the rule set is satisfiable. It is never sent to the model.

The haystack algorithm

haystack-v1 stores recipes, not text: a 256k-token document is a megabyte, and there are eight sizes. The harness rebuilds the exact document from the recipe using haystack-v1/build.py, which is standard-library only so bench/ can import it directly. Given seed, target_tokens and needles:

1. budget = target_tokens * 4 (the same chars/4 heuristic);
2. rng = random.Random(seed);
3. needle_chars = Σ(len(text) + 13 + 1) over the needles, where 13 is len("Line 000000: ");
4. generate filler sentences one at a time — rng.choice(TEMPLATES), then fill its slots left to right, one rng call each — stopping as soon as filler_chars + needle_chars >= budget, counting each line as 13 + len(sentence) + 1;
5. insert the needles sorted by depth ascending: needle j (0-based) goes to index round(depth_j * F) + j, where F is the number of filler lines, clamped to [0, len(lines)];
6. number every line from 1: f"Line {i:06d}: {sentence}";
7. join with "\n", no trailing newline.

Steps 4 and 5 must stay in that order or the numbering changes. Two implementations agree exactly when they produce the same SHA-256, which every row records in sha256 and check.py re-verifies. build_prompt(item) returns PREAMBLE + "\n\n" + document + "\n\n" + question, which is what a longctx eval must send.

Documents at or below 32k tokens are also materialised as haystack-v1/static/<id>.txt for tooling that wants a file; those carry one trailing newline that is not part of the hashed text.

eval-longctx-v1 embeds its own recipe per row in meta.haystack, so it is self-contained; its prompt is the question only, and sending it without the document is a bug that reads as a catastrophic accuracy drop.

Adding a dataset

1. Write one generator in _gen/ (PEP 723 header, fixed seed, stdlib or Pillow), and make it prove its own answers — compute them, brute-force them, or execute them. Do not hand-type a key.
2. Emit dataset.json via _lib.base_dataset_json / _lib.eval_dataset_json so the licence, provenance, token heuristic and scoring block stay identical across datasets.
3. Run uv run datasets/_gen/check.py until it is green, and run the generator twice to confirm the output is byte-identical.
4. Add a row to the index above and to _gen/README.md, and add the workload that references it (workloads/README.md).
5. Keep the whole of datasets/ under 25 MB. If a dataset would be large, store a recipe and a reference builder, the way haystack-v1 does.

Content rules: written or generated here, never copied; unique text rather than repeated text, so prefix caching and tokenizer merges cannot flatter a prefill number; facts that cannot go stale; and no item whose answer depends on taste, sentiment or the current date.