A/B experiments

Relevance work is guesswork until it is measured. An A/B experiment sends each API key consistently to one of two search profiles and lets the query log answer the question: which configuration produces fewer zero-result searches and higher match scores?

Search experimentsCopy Anchor Link

A search experiment belongs to one index and compares two arms, control and treatment, each pointing at a saved search profile (or at the index default when the profile is empty). While an experiment is running, every REST search and every embed widget search on that index is served one arm's profile.

curl -sS -X POST "$AAC_API/experiments?index=documentation" \
  -H "Authorization: Bearer $AAC_KEY" \
  -H "Content-Type: application/json" \
  -d '{"name":"weights v2","controlProfileId":"prof_control","treatmentProfileId":"prof_weights_v2","treatmentWeight":50,"status":"running"}'

All /experiments routes take ?index=<slug> and need a key with settings:write. The console has the same controls on the index's Experiments page.

Status changes emit the experiment.started and experiment.finished webhooks.

Who lands in which armCopy Anchor Link

Assignment is a hash of the unit key, so the same unit always gets the same arm, across instances and deploys. The unit key is the first of:

1. session_id — your durable anonymous visitor id (client_id in the widget),
2. user_id,
3. visit_id,
4. otherwise the API key (REST) or the embed session's issuer (widget) — all such traffic shares one arm.

Send a stable session_id from your frontend; without it, a whole server-side key is one unit. The first assignment of a unit is stored and never changed, so a visitor keeps the same arm for the whole experiment.

Only one experiment per index is served at a time: if several are running, one of them is picked. An explicit profile= in the request always wins over the experiment, which is how the console previews a variant without disturbing it.

Reading the reportCopy Anchor Link

Every search an experiment decides is logged with its arm, and every event carries the query_id of the search it came from, so attribution is exact for every unit kind (key, visitor, user, visit). Each variant reports assignments (units exposed to the arm), searches served by the arm, and the clicks, conversions and revenue (revenue_cents, the sum of conversion value_cents) on those searches. Send query_id with your events — events without it are not attributed. ctr and conversion_rate are per search; uplift is the relative CTR difference of treatment over control, or null while either side has no data. Searches logged before this attribution existed are not counted.

Legacy: settings.abTestCopy Anchor Link

Indexes can still carry the older per-key split in their settings. It applies only when no experiment is running and the request names no profile. Prefer search experiments for new work; the rest of this page describes the legacy mechanism.

How the split worksCopy Anchor Link

The experiment is a property of the index, stored in its settings:

{
	"abTest": {
		"enabled": true,
		"profiles": ["control", "weights-v2"],
		"split": 50
	}
}

Assignment is deterministic, not random per request:

const bucket = createHash("sha256").update(`${apiKey.keyId}:${index.id}`).digest()[0] % 100;
const profile = bucket < (abTest.split ?? 50) ? abTest.profiles[0] : abTest.profiles[1];

Two consequences follow, and both matter:

• A given key always lands in the same bucket. Users do not flip between variants mid-session, and a cached page cannot disagree with the last result.
• The unit of randomisation is the key, not the visitor. One server-side key means one bucket for all its traffic — fine for comparing configurations across tenants, useless for a per-user split. For a per-user experiment, issue one key per cohort or pass profile explicitly.

An explicit profile in the request always wins over the A/B bucket, which is how you preview a variant in the console without disturbing the experiment.

Reading the resultsCopy Anchor Link

curl -sS "$AAC_API/analytics/experiments?days=30&index=documentation" \
  -H "Authorization: Bearer $AAC_KEY"

{
	"days": 30,
	"index": "documentation",
	"buckets": [
		{
			"profile": "control",
			"searches": 8240,
			"zero_rate": 0.11,
			"avg_latency_ms": 18,
			"avg_top_score": 1720.4
		},
		{
			"profile": "weights-v2",
			"searches": 8106,
			"zero_rate": 0.07,
			"avg_latency_ms": 19,
			"avg_top_score": 2145.8
		}
	],
	"request_id": "req_3b5d7f9a1c3e5b7d9f1a3c5e"
}

days is clamped to 1–90. Without index, the buckets aggregate across every index in the organisation — useful when the same profile pair is applied everywhere.

Only queries that ran under a profile appear, so the totals will be smaller than your overall search count: organic traffic to keys outside the experiment, semantic calls with an explicit profile, console previews and suggestion requests are all excluded.

A workable routineCopy Anchor Link

1. Create the candidate profile. Copy the control profile, change one thing — usually query_by_weights or num_typos, occasionally facet_by.
2. Preview it. Send ?profile=weights-v2 against real queries in the console; GET /indexes/{slug}/widget-config lists the profiles and their configs if you want to diff them programmatically.
3. Enable abTest with both slugs and a 50/50 split. Leave it running for at least a week — a day of traffic cannot separate 11% from 7% zero-results.
4. Read /analytics/experiments, and cross-check /analytics/overview to make sure overall traffic did not change shape during the window.
5. Decide, then clean up. Promote the winner to the index default profile, disable abTest, and keep the loser as a documented rollback.

PitfallsCopy Anchor Link

• Two profiles, not more. The schema is a pair plus a split percentage; a three-way test needs a second index or a different mechanism.
• Unequal traffic is normal. Hashing is uniform over keys, not over requests: with three active keys a 50/50 split can land 2:1.
• avg_top_score is not comparable across queries. It compares two profiles over the same traffic mix; comparing it to last month means comparing it to different queries.
• Latency is not the metric. A hybrid profile will be slower and better; decide on relevance, then check the latency cost is acceptable.
• Changing the profile mid-experiment invalidates it. The buckets are computed from the stored config at query time, so an edit applies to in-flight traffic and muddies the comparison.

Where it is configuredCopy Anchor Link

abTest lives in the index settings object. It is not exposed by the public REST API — the console owns index settings, and the public surface is read-only with respect to them. Set it through the console's index settings or, for automation, through the console's oRPC procedures from an authenticated session.