Voice and image search
Query the index with a recording or a picture — POST /voice and POST /image, both turning the input into a text query first.
Both endpoints solve the same problem: the index contains text, the user has something else. The platform converts the input into a query, then runs an ordinary tenant-scoped search, so filters, facets and quotas behave exactly as they do for typed queries.
Voice
curl -sS -X POST "$AAC_API/indexes/documentation/voice" \
-H "Authorization: Bearer $AAC_KEY" \
-H "Content-Type: application/json" \
-d "$(jq -n --rawfile audio clip.webm \
'{ audio: ($audio | @base64), mime_type: "webm", language: "en" }')"{
"index": "documentation",
"mode": "ai",
"transcript": "how much does the pro plan cost",
"found": 3,
"hits": [
{
"id": "pricing",
"score": 2180,
"document": { "title": "Pricing", "url": "https://example.com/pricing" },
"highlights": []
}
],
"latency_ms": 940,
"charged_cents": 0,
"request_id": "req_1a3c5e7b9d0f2a4c6e8b0d1f"
}| Field | Type | Notes |
|---|---|---|
audio | string | Base64 audio, 16 characters to 8 MB |
mime_type | string | wav (default), mp3, webm, ogg, m4a |
language | string ≤ 8 | ISO code hint for the transcriber |
model_id | string ≤ 128 | Engine voice model; forces the search service path |
query_by | string | Fields for the resulting query |
Two execution modes
- AI transcription (
mode: "ai"). When configured, the platform transcribes audio and searches the resulting text. The response containstranscriptand normal search hits. - Managed voice transcription (
mode: "native"). A configured voice model transcribes the audio. The transcript then runs through the same search pipeline as a typed query, including tenant scope, document access, private-field protection and quotas. Results use the application search contract:
{
"index": "documentation",
"mode": "native",
"transcript": "example query",
"found": 2,
"hits": [],
"result": { "transcription": "example query", "results": [{ "found": 2, "hits": [] }] },
"request_id": "req_7c9e1b3d5f7a9c1e3b5d7f9a"
}Failures
| Status | Code | Cause |
|---|---|---|
422 | EMPTY_TRANSCRIPTION | Audio decoded but contained no speech |
422 | INVALID_AUDIO | Payload could not be decoded as audio |
413 | AUDIO_TOO_LARGE | Clip over the accepted size |
503 | VOICE_MODEL_NOT_CONFIGURED | Voice search has not been configured for this deployment |
Voice searches consume one search unit, and the transcript is written to the audit log — do not send audio that contains secrets you would not want logged.
Image
curl -sS -X POST "$AAC_API/indexes/documentation/image" \
-H "Authorization: Bearer $AAC_KEY" \
-H "Content-Type: application/json" \
-d '{
"image": "https://example.com/uploads/price-table.png",
"hint": "pricing table screenshot",
"query_by": "title,content",
"per_page": 5
}'{
"index": "documentation",
"mode": "ai",
"description": "pricing comparison table with monthly plans",
"model": "gpt-4o-mini",
"found": 2,
"hits": [
{
"id": "pricing",
"score": 1540,
"document": { "title": "Pricing", "url": "https://example.com/pricing" },
"highlights": []
}
],
"latency_ms": 1180,
"charged_cents": 0,
"request_id": "req_3e5b7d9f1a3c5e7b9d1f3a5c"
}| Field | Type | Notes |
|---|---|---|
image | string | Base64 image or an http(s) URL, 16 characters to 12 MB |
mime_type | string | Optional; inferred for URLs |
hint | string ≤ 200 | Extra context for the vision model ("which plan is highlighted?") |
query_by | string | Fields for the generated query |
filter_by | string ≤ 1000 | Restrict the candidate set before ranking |
per_page | int 1–50 | Default 10, hard-capped at 50 |
description is the intermediate query the vision model produced — show it in a debug panel, log it, or return it to the user as "searching for: …", which makes an odd result set explicable.
Failures
Image search needs a provider; there is no engine-only fallback:
{
"error": {
"code": "AI_NOT_CONFIGURED",
"message": "Set OPENAI_API_KEY to use image search",
"request_id": "req_9f1b3d5c7a9e1b3d5f7c9a1b"
}
}That is a 503. An oversized or malformed image fails validation as 422 VALIDATION_ERROR before any provider call.
What this is and is not
- It is text retrieval with a nicer input. The image is described in words, then matched against text. A screenshot of a chart will not find "the document that contains a similar chart".
- It is not CLIP or a multi-modal index. No image embeddings are stored. If you need visual similarity, keep your own vector pipeline and use
vector_queryin/search. - Costs differ. Voice and image calls meter a search unit plus AI tokens for the description or transcript;
charged_centsreports any wallet overage.
Wiring it into a UI
The widget already does the browser part for voice: enable Voice search in the console's embed panel and the InstantSearch voiceSearch widget appears in the search box, using the browser's Speech Recognition API — no audio ever leaves the client for that path. For your own UI:
const stream = await navigator.mediaDevices.getUserMedia({ audio: true });
const recorder = new MediaRecorder(stream, { mimeType: "audio/webm" });
const chunks: Blob[] = [];
recorder.ondataavailable = (event) => chunks.push(event.data);
recorder.onstop = async () => {
const base64 = await new Promise<string>((resolve) => {
const reader = new FileReader();
reader.onload = () => resolve(String(reader.result).split(",")[1] ?? "");
reader.readAsDataURL(new Blob(chunks, { type: "audio/webm" }));
});
const response = await fetch("/api/v1/indexes/documentation/voice", {
method: "POST",
headers: { Authorization: `Bearer ${sessionKey}`, "Content-Type": "application/json" },
body: JSON.stringify({ audio: base64, mime_type: "webm" }),
});
const { transcript, hits } = await response.json();
render(transcript, hits);
};
recorder.start();Never call this endpoint from a browser with a aac_live_… key: mint a scoped key server-side (Embed session) or proxy the call through your backend, which is the better option for voice and image because the payloads are large and the calls are expensive.