LLM Inference | Atelico

Streaming Inference

Token-by-token streaming for responsive UIs. Display partial responses as they generate — dialogue feels alive instead of waiting for a wall of text.

Qwen3 0.6B, Qwen3.5 9B and LLaMA 3.2 1B streaming locally on an M1 Max, with tokens per second and time to first token on screen.

// Stream tokens as they generate
await Atelico.StreamChat(
    "Tell me about this dungeon.",
    "You are a mysterious guide.",
    (string token) => {
        dialogueLabel.text += token;
    }
);

Supported Models

Load any GGUF from Hugging Face or bundle a model with your game. GGUF Q2–Q8 for compatibility, SafeTensors for full precision, Bonsai's 1-bit MLX for extreme compression. The engine can support any open-source model.

Smallest footprint

Bonsai 1-bit MLX

1.7B (~250 MB) · 8B (~1.15 GB)

An 8B-class model in ~1.15 GB. Small enough to ship inside the game on Steam Deck, Switch, or mobile.

Industry baseline

LLaMA 3.x

3.2 1B · 3.2 3B · 3.1 8B

Float (BF16) and GGUF Q2–Q8. The default reference family for chat workloads.

Broad size range

Qwen 3 / 3.5

0.6B · 0.8B · 2B · 4B · 9B

Float, Q4_K_M, and Qwen-native Q8_K_M. Strong dense models from sub-1B up to 9B.

MoE option

Gemma 4

E2B · 26B-A4B (MoE)

Google's open family. The 26B-A4B MoE activates ~4B parameters per token for quality at lower compute.

Lean & mid-tier

SmolLM 3

3B · Float & Q4_K_M

High quality per parameter at 3B. Solid default for tighter VRAM budgets that still want capable dialogue.

NVIDIA hybrid

Nemotron-H · Nemotron-3 Elastic

4B · 12B · Float & Q4_K_M

NVIDIA's hybrid-attention 4B and the 12B "Elastic" quality tier for richer narrative generation.

Size in memory, by parameter class

Same parameter count, very different footprint. Bonsai's 1-bit MLX wins in every bucket: an 8B-class model fits in ~1.15 GB, ~4× smaller than the equivalent GGUF Q4.

Decode Throughput

Same one-line call, faster than llama.cpp on every measured model on RTX 3090.

// Sample once. Same call works on every supported model.
let reply = engine.llm()
    .chat(prompt, system_prompt)
    .await?;

LLaMA 3.2 1B · Q4_K_M

614 tok/s

vs llama.cpp 539 (+14%)

Qwen 3.5 0.8B · Q4_K_M

435 tok/s

vs llama.cpp 393 (+11%)

Gemma 4 E2B · Q4_K_M

263 tok/s

vs llama.cpp 186 (+42%)

Pure-decode, median of 5 runs. vLLM is CUDA-only and runs at its weakest mode by design at batch=1 — the single-user real-time regime this engine targets. Full per-model and per-backend tables (vs llama.cpp, Ollama, vLLM) live on the benchmarks page.

GGUF Support

Point the engine at any .gguf file. Compatible with the thousands of GGUF-quantized models on Hugging Face. Q2, Q3, Q4, Q5, Q6, Q8, and F16 all supported — pick the quality/size tradeoff that fits your target hardware.

Atelico.Init();
Atelico.LoadModel("Assets/Models/my-custom-model-q4.gguf");

string response = await Atelico.Chat(
    "Hello!",
    "You are a friendly shopkeeper."
);

Structured Output

Constrain output at the token level. Four ways to lock the shape, all enforced during generation: every token the model emits is masked against the constraint, so invalid output is impossible (not unlikely — impossible).

JSON Schema — full Draft-7 (nested objects, arrays, enums, ranges, required fields).
Choice / enum — pick from a fixed list of strings.
Regex — any regular expression. IDs, timestamps, dice notation, version strings.
Lark grammar — context-free grammars for shapes JSON Schema can't express (DSLs, command languages).

let schema = json!({
    "type": "object",
    "properties": {
        "action":     { "type": "string",
                        "enum": ["attack", "defend", "flee", "talk"] },
        "target":     { "type": "string" },
        "confidence": { "type": "number", "minimum": 0, "maximum": 1 }
    },
    "required": ["action", "target", "confidence"]
});

let reply = engine.llm()
    .generate_structured(prompt, &schema)
    .await?;
// reply: { "action": "defend", "target": "dragon", "confidence": 0.85 }

Constraint trees are cached across requests with the same shape, so the per-call overhead drops to near-zero on the second call. Full reference →

Prefix Cache

When requests share a prompt prefix (system prompt, world state, character backstory) the engine reuses the cached KV state instead of re-prefilling. Up to ~4× faster time-to-first-token on a ~1,000-token persona prompt — the win scales with the cached length.

Answer Caching

Identical prompts return cached results instantly. In-memory cache with configurable TTL and LRU eviction, isolated per adapter, namespace, and temperature. Useful for repetitive queries like NPC barks, tooltip descriptions, or any prompt that recurs frequently with the same parameters.

Multi-Backend Routing

Same API, different backends. Route per-request: local on-device models for gameplay, cloud providers for development, testing, or heavier capability calls. All through the same engine.chat().

LLM Chat & Streaming