DGX Spark, v0.1.0 image, shipped config.yml, idle server, bench/bench.py --runs 3 --max-tokens 800.
Client tok/s = completion tokens ÷ (last chunk − first chunk). Server = TabbyAPI's completion_tokens_per_sec.
Acceptance from the server log (Draft: accepted / proposed).
Story — "Write a short story about a lighthouse keeper (about 500 words)."
temperature 1.0 (server default)
| run | prompt tok | TTFT | tokens | time | client tok/s | server tok/s | chunks | draft accepted |
|---|---|---|---|---|---|---|---|---|
| 1 | 18 | 0.60 s | 794 | 26.7 s | 29.8 | 29.3 | 156 | 311 / 533 (58.4 %) |
| 2 | 18 | 0.59 s | 794 | 26.3 s | 30.2 | 29.7 | 160 | 310 / 476 (65.1 %) |
| 3 | 18 | 0.60 s | 794 | 25.7 s | 30.9 | 30.3 | 160 | 333 / 466 (71.5 %) |
median 30.2 tok/s
temperature 0
| run | prompt tok | TTFT | tokens | time | client tok/s | server tok/s | chunks | draft accepted |
|---|---|---|---|---|---|---|---|---|
| 1 | 18 | 0.60 s | 794 | 24.6 s | 32.3 | 31.7 | 153 | 321 / 448 (71.7 %) |
| 2 | 18 | 0.60 s | 794 | 25.0 s | 31.7 | 31.2 | 157 | 321 / 441 (72.8 %) |
| 3 | 18 | 0.60 s | 794 | 25.0 s | 31.8 | 31.2 | 157 | — |
median 31.8 tok/s
the same prompt through Open WebUI (temperature unset → 1.0)
Browser on a laptop, Open WebUI on a separate host, server on the LAN. Measured at Open
WebUI's /api/chat/completions with stream_options.include_usage.
| run | prompt tok | TTFT | tokens | client tok/s | server tok/s | chunks |
|---|---|---|---|---|---|---|
| 1 | 18 | 0.64 s | 735 | 31.4 | 30.8 | 145 |
| 2 | 18 | 0.63 s | 780 | 30.6 | 30.1 | 158 |
| 3 | 18 | 0.63 s | 794 | 30.4 | 29.9 | 161 |
Code — "Write a Python module implementing a thread-safe LRU cache class with get, put, capacity eviction, and a small pytest test file. Code only, no explanation."
temperature 0
| run | prompt tok | TTFT | tokens | time | client tok/s | server tok/s | chunks | draft accepted |
|---|---|---|---|---|---|---|---|---|
| 1 | 38 | 1.47 s | 794 | 13.8 s | 57.7 | 54.3 | 48 | 648 / 714 (90.8 %) |
| 2 | 38 | 1.08 s | 794 | 13.9 s | 57.1 | 53.7 | 49 | 645 / 718 (89.8 %) |
| 3 | 38 | 1.07 s | 794 | 13.9 s | 57.3 | 53.9 | 49 | 645 / 714 (90.3 %) |
median 57.3 tok/s
temperature 1.0
| run | prompt tok | TTFT | tokens | time | client tok/s | server tok/s | chunks | draft accepted |
|---|---|---|---|---|---|---|---|---|
| 1 | 38 | 1.11 s | 659 | 12.5 s | 52.7 | 49.2 | 44 | 524 / 648 (80.9 %) |
| 2 | 38 | 1.11 s | 794 | 14.0 s | 56.7 | 53.3 | 49 | 644 / 720 (89.4 %) |
| 3 | 38 | 1.13 s | 794 | 14.9 s | 53.3 | 50.3 | 54 | 630 / 746 (84.5 %) |
median 53.3 tok/s
Long context (384k configuration)
| prompt | result | prefill | rate | memory |
|---|---|---|---|---|
| 100,000 tokens, needle | found | 125 s | ~800 tok/s | — |
| 300,000 tokens, needle | found | 374 s | ~800 tok/s | 106 GB used / 15 GB avail |
1M configuration (max_seq_len: 1048576): loads at 108 GB used / 12 GB available; 600,000-token
needle found after a 938 s prefill (~640 tok/s) at 113 GB used / 8 GB available.
Settings sweep at 8.8k context, structured output, temperature 0, 1.5k tokens
Every variant tried — draft length, draft confidence threshold, 8-bit cache modes (ignored for the sparse layers), dynamic draft length, prefill graph capture — landed in the same band:
| configuration | decode tok/s | draft accepted |
|---|---|---|
| shipped and all variants | 45 – 53 | 85–99 % |
| no draft model | 29 | — |