Connectors

A connector is a saved source plus a mapping. It reads documents from somewhere outside AACSearch and writes them into one index.

There are two families, and they fail in different ways, so it is worth knowing which one you are configuring.

Choosing a typeCopy Anchor Link

CMS types exist because a feed cannot tell you that a product was deleted. The module sits inside the platform and forwards what happened. Supported CMS platforms are Bitrix, WordPress and Shopify; they are all configured as connectors, as described on this page.

Where connectors are managedCopy Anchor Link

Note: connectors are managed in the console (Search → Connectors). There is no public REST endpoint for creating or running a connector yet; the REST surface covers documents, search, and index settings. Automation should call the console's job history or drive a feed that a connector reads.

Console operations, all on the connector row:

A deleted connector leaves its documents in the index on purpose. Uninstalling a plugin should not empty your search results — to remove the documents, run a full-mode prune or delete by source_type.

Configuration shared by all typesCopy Anchor Link

Crawler identity and robotsCopy Anchor Link

Connectors that fetch HTML (sitemap, url, docsearch) identify themselves as AACSearchBot/1.0 (+https://aacsearch.dev/bot) on every request, including robots.txt. Machine feeds (json, csv, xml, yaml) do not use this bot or read robots rules — you point them at an explicit feed URL.

For each site origin, the crawler loads /robots.txt once per run (10 s timeout, 500 KB cap). Rules follow RFC 9309: the most specific User-agent group applies (AACSearchBot, then *), longest path match wins, and Allow beats an equally long Disallow. If robots.txt returns 404, everything is allowed; if it returns 5xx or times out, the run stops with ROBOTS_UNAVAILABLE and complete: false so pruning does not delete documents that were not re-checked.

Disallowed URLs are skipped (counted as robotsDisallowed), not errors. Crawl-delay is honoured as a pause between requests to the same origin (capped at 10 s); without it the default gap is 250 ms. Page-level X-Robots-Tag, <meta name="robots">, and <meta name="aacsearchbot"> are respected: noindex drops the page, nofollow keeps the page but does not enqueue links (docsearch). Run history shows robotsDisallowed, noindex, and nofollow when they explain a smaller index than expected.

idField and why it mattersCopy Anchor Link

The identifier determines whether a run inserts a document or updates it. Getting it wrong is the most expensive mistake in a connector: every run creates a second copy of every document, and the original is then pruned as "missing from the source".

The resolution order is:

1. config.idField — the field you name;
2. external_id — the convention every CMS adapter uses;
3. id — often present in JSON exports;
4. a hash of the content — last resort, and it changes when the content changes.

Only the fourth produces duplicates. If your document count doubles after a run, this is the setting to check first.

What a run doesCopy Anchor Link

Every run follows the same five steps:

read source  →  normalise  →  compare payloadHash  →  write changed  →  prune vanished

The comparison is why a stable catalogue costs almost nothing: 40 000 items with three changes writes three documents. See delta sync.

SchedulingCopy Anchor Link

Set cronExpr and timezone, or use a fixed interval. Deletions are only pruned after a complete run, which is what makes a nightly full sync meaningful for a documentation site. See scheduling.

Verifying a connectorCopy Anchor Link

The console lists the last 20 jobs per organisation with loaded, indexed, skipped, pruned and failed. Reading them:

LimitsCopy Anchor Link

CMS push module operationsCopy Anchor Link

Bitrix, PrestaShop and DIAFAN modules run inside your CMS and bundle the generated PHP SDK. Install the module ZIP from the connection wizard, paste its connection string, review discovered fields, apply the schema and start a full sync. Keep the module scheduler enabled: full runs resume from acknowledged batch checkpoints; commands, deltas and deletes also depend on scheduler ticks.

Use an index-bound CMS key with documents:write, indexes:write, search:read. Keys remain on the CMS server. Switching the API endpoint or index clears the old sync cursors. Storefront search checks current CMS visibility and falls back to native search for stale or restricted hits or backend timeouts. Autocomplete returns no restricted hits. Check the organization's document limit before importing a large catalog; push modules use the same quota as other ingestion.