Performance benchmarks

GPU Sharing Impact

This is the benchmark no other inference tool can show, because no other inference tool shares the GPU with a game renderer. The measurements below come from the Atelico Compute Graphics Scheduler (CGS), which schedules inference and rendering on a single shared hardware context instead of letting the OS context-switch between them. The chart compares two scenes (Sponza and San Miguel at 4K) rendered five ways: game alone, the three CGS scheduling modes (PrioritizeGraphics, Balance, PrioritizeCompute), and an Other baseline where our engine runs in a separate process and contends with the game for the GPU, simulating what any other inference solution looks like without CGS.

About the scenes. Both are canonical computer-graphics test scenes from Morgan McGuire's archive: Sponza is the ~280K-triangle Renaissance atrium that's been the standard GPU benchmark since the early 2000s, light enough to render at hundreds of FPS so any inference-induced frame-time impact is easy to spot. San Miguel is a much heavier ~10M-triangle Mexican courtyard restaurant scene (Guillermo M. Leal Llaguno) that runs at ~120 FPS on an RTX 5080, stressing the GPU enough that any extra scheduling overhead shows up immediately.

Same hardware (RTX 5080), same scenes, same Llama 3.2 3B inference workload during render. Atelico's three CGS scheduling modes (PrioritizeGraphics, Balance, PrioritizeCompute) keep the game at native FPS; without CGS (right-most column), the WDDM driver context-switches between graphics and inference and the game collapses to single-digit FPS on the heavy scene.

LLM Decode Throughput vs Other Inference Engines

Methodology. Decode tokens per second on the same hardware and prompt, using the industry-standard pure-decode protocol so every column on the table measures the same quantity.

• Protocol: BOS-only prompt, 128 generated tokens, median of 5 runs, deterministic sampling. Every column reports decode-loop time only (forward passes plus sampling, no prefill, no HTTP). Time-to-first-token and prefill rate live in separate sections of this page.
• Tool equivalence: Atelico's bench_minimal, llama.cpp's llama-bench -p 0 -n 128 (tg128), MLX-LM's CLI, vLLM's tg128, and Ollama's server-side eval_count / eval_duration all measure the same quantity, so the numbers compare like-for-like.
• Table layout: one section per architecture; rows are model variants, columns are serving engines.
• Default format: all numbers are GGUF Q4_K_M unless a different format is noted next to the model name (e.g. Bonsai 1.7B (1-bit MLX), Qwen3 0.6B (Q8_K_M), where all engines were benched on the same weights for apples-to-apples).
• Missing cells: engines that don't run on a backend are dropped from that table entirely (vLLM is CUDA-only).
• Variance, ties, and winners: run-to-run variance on M1 Max is ±2–3% due to thermals, so deltas under 5% should be read as ties. In each row, the cell with the highest rounded tok/s (and any other cell that displays the same rounded value) is rendered in bold to mark the winner.
• vLLM at batch=1: the single-user real-time scenario this page targets is vLLM's weakest mode by design. vLLM is built for high-throughput batched server deployments, whereas in-game local inference is one player, one request at a time, which is exactly the regime Atelico is built for and where vLLM gives up most of its throughput.

Prefix Cache Speed-up

Multi-turn conversations reuse the prompt prefix from one turn to the next. The engine's automatic token-replay prefix cache turns time-to-first-token from a function of the full conversation length into a function of just the new user message: when the beginning of the prompt is byte-identical to a previous request, the engine replays the cached KV state for that prefix instead of recomputing it.

Methodology. Mean time-to-first-token across 5 turns of a character-dialogue RAG pipeline (Qwen 3 0.6B Q8_K_M) at three prefix sizes (~100, ~1,000, ~2,000 tokens). Cold = no cache, the engine re-prefills the persona every turn. Warm = the persona is reused from the previous turn, so only the changing tail is prefilled.

How this scales. The win grows with the cached prefix length: marginal at ~100 tokens (constant per-request overhead from sampling, schema compilation and JSON-mode setup dominates both bars), multi-x from ~1,000 tokens onward, and saturates as the changing tail becomes a smaller and smaller fraction of total prefill time. NVIDIA ratios are smaller than Apple Silicon's at the same size because the RTX 3090 prefills so quickly in absolute terms that the constant per-request overhead is a bigger fraction of warm TTFT.

Text-to-Speech

Two engines: Kokoro (high-quality neural, 54 voices) and Atelico TTS (lightweight, instant voice cloning). Both run entirely on device. Realtime factor: how much faster than wall-clock the audio is synthesized; 18× means one second of speech in ~56 ms, comfortably under a single 60 FPS frame.

Speech-to-Text (Whisper)

Realtime factor (higher is better) across the small English-only variants (tiny.en / base.en / small.en) and the three full multilingual top-of-line models (distil-large-v3, large-v3-turbo, large-v3). Pick the size that fits your accuracy and latency budget; every size runs locally with no cloud hop.

What this means for a player. Realtime factor is the inverse of how long the engine takes to transcribe one second of audio, so the rough latency between the player finishing an utterance and the transcribed text being available in the game is ~1,000 / RTF ms per second of speech. A 1-second voice command transcribed at 30× resolves in ~33 ms (under one frame at 30 FPS); the same command at 100× resolves in ~10 ms. The big multilingual models still run locally in real time on consumer hardware: even Whisper large-v3 on M1 Max comes in under half a second per second of audio.

Audio-to-Face

From a usage perspective: Audio-to-Face turns each chunk of spoken audio (TTS output, recorded VO, or live mic) into the 52 ARKit blendshape weights that drive a character's face. The cold first-frame latency is the delay from the first audio sample arriving to the first set of blendshape weights leaving the engine on a fresh load. Warm first-frame latency is the same delay after the model is hot in memory (what the player actually experiences for every line beyond the first). The steady-state per-frame cost is what the rest of the dialogue then pays per frame for as long as the character keeps talking.

Image Generation

Atelico IG generates a 512×512 image in about a second on an M1 Max and well under half a second on an RTX 3090, at output quality close to FLUX.2 [pro]. It is small enough to fit in the unified memory budget of every consumer device we ship to, including phones; the cloud-served FLUX.2 [pro] is a 32B-parameter model that does not.

For context, the chart also shows two cloud baselines from Replicate measured end-to-end (queue + inference + network): FLUX schnell at ~0.8 s and FLUX.2 [pro] at ~10.8 s, both at 1024×1024.

Model Sizes

Different sizes hit different deployment targets. Mobile devices (iOS / Android) typically have 4–8 GB of usable RAM after the OS and the rest of the game; current-generation consoles share ~12 GB across game and AI; PCs span the full range from 8 GB integrated to 24 GB+ discrete VRAM. Pick a model that leaves room for everything else your game is doing. All sizes are GGUF Q4_K_M unless a different format is noted next to the model name (e.g. Bonsai 1.7B (1-bit MLX)). Memory consumption below is an estimate for greedy decode at the default 4 K-token context; longer contexts grow the KV cache roughly linearly.

Rough deployment guide: the top of the table (under ~2 GB) fits mobile and integrated-graphics laptops; the middle (~2–6 GB) is the sweet spot for current consoles and mainstream PCs; the bottom (~9 GB and up) targets discrete-GPU PCs with 12 GB+ VRAM. The 1-bit Bonsai family is unusually memory-light for its parameter count and is the cheapest way to ship a multi-billion-param LLM in a mobile build.