Text-to-speech and speech-to-text

Text-to-Speech

Two TTS engines, each built for different needs. Both run entirely on-device with no cloud dependency.

Kokoro TTS

High-quality neural TTS with 54 built-in voices across 9 languages (English, Italian, Spanish, French, German, Portuguese, Japanese, Chinese, Hindi). Natural prosody and expression suitable for cinematic dialogue and narrative. Supports evolutionary voice cloning: provide a reference audio clip and Kokoro adapts to match the voice over multiple iterations.

Atelico TTS

Lightweight, fast TTS designed for high-throughput scenarios. Supports instant voice cloning from a single audio sample with no training step. Ideal for ambient NPC barks, item descriptions, and any situation where you need many different voices quickly.

TTS Performance

Synthesis speed is measured as realtime factor (RTF): how many seconds of audio are produced per wall-clock second of compute. 5× means 1 s of audio in 200 ms; 18× means 1 s in 55 ms. Higher is better.

Fast enough for real-time dialogue without pre-rendering, and well below the per-frame budget at 30 FPS (33 ms) once combined with per-sentence streaming.

Per-Sentence Streaming

For longer text, the engine streams audio sentence by sentence. The first sentence starts playing while subsequent sentences are still being synthesized. This keeps latency low for narrative sequences, quest log readouts, or any multi-sentence output.

// Synthesize speech from text
byte[] audio = await AtelicoAudio.Synthesize(
    "Welcome to the tavern, traveler.",
    voice: "kokoro-en-heart"
);

// Stream sentence by sentence
AtelicoAudio.SynthesizeStream(
    "A long tale of adventure. With many sentences. Each streamed as audio.",
    voice: "kokoro-en-heart",
    onChunk: (byte[] chunk) => { PlayAudioChunk(chunk); },
    onComplete: () => { Debug.Log("Done"); }
);

Audio2Face: Lip-sync & Facial Animation

Feed any audio (TTS output, recorded VO, live mic) and Audio2Face emits a stream of ARKit-52 blendshape weights at 60 fps. The same blendshape stream drives both 3D rigs (MetaHuman, Apple ARKit avatars, Live Link face capture, any morph-target rig named after the ARKit standard) and 2D animation (sprite mouth/eye atlases, Live2D / Spine rigs mapped to the ARKit channels) — same data, two rendering paths.

Animation streams alongside synthesis: the first frames are available within tens of milliseconds, so the face can start moving the instant TTS produces its first sentence (no pre-rendering, no waiting for the full audio buffer).

Audio2Face Performance

Streamed wall-clock latency at 60 fps. The per-frame budget for 60 fps is 16.7 ms; both backends sit comfortably under it in steady state.

Steady-state cost is small enough to run alongside LLM inference, TTS, and the renderer in the same frame budget: the whole voice-to-face pipeline (mic → STT → LLM → TTS → face) fits in a single device, no cloud round-trips.

Speech-to-Text

Whisper-based speech recognition runs on-device. Supports multiple model sizes (tiny, base, small, medium, large) so you can choose the tradeoff between accuracy and speed for your target hardware. Useful for voice commands, player speech input to NPCs, and accessibility features.

STT Performance

How fast each Whisper size transcribes audio on the two reference platforms. Realtime factor is seconds of audio transcribed per wall-clock second of compute (a 22× RTF means 10 s of speech is transcribed in 450 ms). Time per minute of audio is the same number expressed as wall-clock: how long the model takes to process one minute of speech.

For most game-engine use cases (short voice commands and NPC dialogue prompts), the .en small variants run well above realtime on both backends. When the player can speak any language, large-v3-turbo and distil-large-v3 are the sweet spot: full multilingual quality at 6–7× realtime on a laptop GPU and 33× on a desktop NVIDIA card.

// Transcribe audio to text
string text = await AtelicoAudio.Transcribe(audioBytes);

Voice-to-Voice NPC Pipeline

Combine STT, LLM, and TTS into a single voice-to-voice NPC interaction: the player speaks, Whisper transcribes, the LLM generates a response, and Kokoro or Atelico TTS speaks it back. The entire pipeline runs on-device with no network round-trip.