Search pictures, sounds, and 3D assets by describing them

One Space for Text, Pictures, and Sound

Ordinary embeddings turn text into vectors. A cross-modal model goes further: it embeds text, images, and audio into the same space, so a picture of a harbor, the sentence "ships moored at the docks", and a recording of creaking ropes and gulls all land close together.

Search becomes a nearest-neighbour lookup. Embed your library once, embed the query, and rank by cosine similarity. The model reads the pixels and listens to the audio, so nothing has to be tagged or named in advance.

In our demo, 100 fantasy locations each come with a picture, a description, and an ambient sound. "A sailing ship at sea" returns the ship's deck, the captain's cabin, and the hold in the storm. "Sawing wood" returns the carpenter's workshop, by its sound alone.

What It's For

• In-game search: a codex, photo mode, or crafting catalogue players can query in plain language.
• Matching authored assets to a scene: when a scene is procedurally generated and changes with what the player does, a fixed list of assets per level no longer works. Describe the scene as it unfolds (a flooded crypt the player just broke into, a market emptying as night falls) and retrieve the sounds, background images, and 3D assets from your authored library that fit it.
• Generated asset cache: when a player asks for an asset whose description is close to one generated before, reuse the cached result instead of generating it again. Embed each request, look up its nearest neighbour, and only generate on a miss.
• Asset library search: artists and sound designers describe the texture, concept art, or sound effect they need instead of guessing filenames.
• Finding near-duplicates: query with an image or a clip to surface similar assets across a large project.

Querying at Runtime

Cross-modal vectors are ordinary float vectors, so they go straight into the engine's vector store, next to the Semantic KV Store entries you already author, with the same facet filters. Embed both sides with the same model: a CLIP image vector is only comparable to a CLIP text vector.

// Index once: CLIP puts pictures and text in one space.
string imageVector = engine.Embeddings.EmbedImage(
    "{\"model\":\"clip\",\"image\":\"" + base64Png + "\"}");
engine.KvStore.Insert("locations", entriesWithImageVectors);

// At runtime: embed a sentence with the same model and search.
string queryVector = engine.Embeddings.EmbedText(
    "{\"model\":\"clip\",\"input\":[\"a sailing ship at sea\"]}");
string matches = engine.KvStore.Query("locations", queryWithVector);

Retrieval Speed

How long one query takes on the two reference platforms, end to end: embedding the query and searching a 10,000-entry index for its top 24 matches. Each row shows the fastest model for that kind of query input (median of 20 warm runs). Lower is better.

Text, image, and audio queries complete within one or two frames at 60 FPS. Video and 3D are only embedded by EBind, the largest model, so they suit background work such as building or refreshing an index rather than per-frame lookups.

Supported Models

Each model covers a different set of modalities. Pick the smallest one that spans the modalities you need to search across.

CLIP, SigLIP, and CLAP are available from every SDK. ImageBind and EBind live in the engine's embedding library, and they search in every direction within their modalities: a picture can find a sound, and a sound can find a picture.