Delta sync
How a connector decides what changed, why a 40 000-item feed usually writes zero documents, and how deletions are handled.
A connector run is not an import. It is a sync: it reads the source, compares each document against what is already indexed, and writes only the difference.
On a stable catalogue this means most runs are cheap. A 40 000-item feed where nothing changed performs one read and zero writes.
How a change is detected
Each stored document keeps a payloadHash — a hash of its normalised content, not
of the raw feed record.
feed record → normalise (map fields, strip HTML, defaults)
→ hash the result
→ compare with the stored hash for that external_idNormalising before hashing matters. If the source adds a fetched_at timestamp or
reorders keys, the raw bytes differ but the document does not — hashing raw
records would rewrite your whole catalogue on every run.
The comparison outcome per document:
| Stored hash | New hash | Action |
|---|---|---|
| absent | any | insert |
| present | different | update |
| present | same | skip (counted, not written) |
present, deletedAt set | any | unhide and update |
Every run reports skipped alongside indexed, so you can see the delta working:
{ "loaded": 40000, "indexed": 12, "skipped": 39988, "pruned": 0 }Deletions
This is where a naive connector loses money: it upserts, so a product deleted in the CMS stays in the index forever.
We handle it in two ways depending on the source.
Feeds without an explicit deletion signal
A sitemap or a JSON export does not say "this product is gone" — it simply stops
listing it. After a complete run, the connector compares the set of
external_ids it loaded with the set stored for that connector, and marks the
difference:
-- conceptually
UPDATE search_document
SET deleted_at = now(), status = 'deleted'
WHERE connector_id = ? AND external_id NOT IN (loaded set)Rules that keep this safe:
| Rule | Why |
|---|---|
| Only on a complete run | A truncated feed would otherwise mark the rest of the catalogue as deleted |
Only when pruneDeleted is on | Deletions are opt-out, not opt-in, but still controllable |
deletedAt is not overwritten | Keeps the instant the deletion was first noticed, not the last run |
Empty feed + mode: full | Clears the index — that is what a full run of an empty source means |
An incomplete run (pagination stopped, request failed partway, maxDocuments
cap hit) never prunes. It also does not advance the cursor.
Sources with an explicit signal
CMS adapters — Bitrix, WordPress, Shopify — receive deletions from the platform
itself (a webhook, an event, an unpublished state). Those are passed through
directly rather than inferred by set difference, which is more accurate: a product
that is temporarily hidden is not the same as one deleted.
Soft delete, not hard delete
A pruned document is marked, not erased:
- it disappears from search results immediately;
- its payload and revision history stay in the database;
- it can be restored through the soft-delete + restore API.
Hard deletion exists for GDPR erasure requests and is explicit:
curl -sS -X DELETE "$AAC_API/indexes/products/documents/SKU-123?hard=true" \
-H "Authorization: Bearer $AAC_KEY"Watermarks
Sources that support a cursor — a feed with updated_at, an API listing changed
items — advance a watermark per run.
| Situation | Cursor |
|---|---|
| Complete run, no failures | Advances to the new position |
| Complete run, some documents failed | Stays put — a partial import must not lose the retry window |
| Incomplete run | Stays put |
| Connector disabled | Not advanced; resumed from the stored value |
That is why a failed run is retried from the same position instead of skipping the documents it failed to write.
What delta sync does not do
| Assumption | Reality |
|---|---|
| "It detects a schema change" | No — that is reindexing |
| "It removes documents missing from one page" | Only after a complete run |
| "It runs in the background for hours" | It runs within the request; very large sources need pagination and maxDocuments |
| "Changed source field order = changed document" | No — the hash is over the normalised document |
Verifying a run
The console shows the last 20 jobs (Search → Connectors → Recent jobs) with
loaded, indexed, skipped, pruned and failed. A run where skipped is
zero on an unchanged catalogue is a signal that the hash was invalidated — usually
because a field mapping was changed, which legitimately rewrites everything once.