Connectors and reindexing

Two operations keep an index current: connectors bring documents in from outside, reindexing rebuilds the collection when the schema — not the content — has to change.

ConnectorsCopy Anchor Link

A connector is a saved source plus a mapping. It is configured in the console (Search → Connectors) and runs manually or on an interval. Four types exist:

ConfigurationCopy Anchor Link

Every imported document gets source_type set to the connector type, so a bad run is easy to undo:

curl -sS -X DELETE "$AAC_API/indexes/documentation/documents" \
  -H "Authorization: Bearer $AAC_KEY" \
  -G --data-urlencode "filter_by=source_type:=sitemap"

Safety railsCopy Anchor Link

• SSRF protection. Connector URLs are resolved before fetching; loopback, link-local, private and metadata ranges are refused, including IPv6 forms that embed such an IPv4 address, and the connection goes to the address that was checked (no DNS rebinding). Every redirect is checked again; credentials (Authorization, cookies, the connector's own secret headers) are never sent to another origin after a redirect. SEARCH_ALLOW_PRIVATE_URLS=true disables the address check for local development; production ignores it.
• Credentials follow the endpoint. Stored secrets are shown as *** and stay encrypted. Changing a connector's URL, domain or host requires typing its secrets again (CREDENTIALS_REQUIRED), so a saved password is never sent to a new host. An encrypted enc:… value pasted as a secret is refused (INVALID_SECRET_VALUE).
• Checksums. Each document's content is hashed and compared with the stored hash, so a nightly run re-imports only what changed — the difference between 40,000 writes and 40.
• Per-document failures. A document the engine rejects lands in the dead-letter queue rather than aborting the run.
• Caps. maxDocuments bounds a run; the 5 MB response limit bounds memory.

Dead lettersCopy Anchor Link

Rejected documents are stored with the source payload and the engine's message. After fixing the mapping or the schema, replay them from the console; the queue is also visible in the admin cockpit across tenants. A growing queue is the earliest signal that a feed changed shape — before the zero-result rate moves.

Running jobsCopy Anchor Link

The scheduler runs connectors, the analytics rollup, retention and snapshots. To run a cycle immediately — from CI, or from an external worker in a deployment that disables the in-process timer — use the admin endpoint. It is a platform-operator call: besides an API key it needs x-platform-admin-token set to the deployment's SEARCH_PLATFORM_ADMIN_TOKEN (403 otherwise):

curl -sS -X POST "$AAC_API/admin/jobs/run?job=connectors" \
  -H "Authorization: Bearer $AAC_KEY" \
  -H "x-platform-admin-token: $AAC_PLATFORM_ADMIN_TOKEN"

{
	"job": "connectors",
	"result": { "connectors": "ok" },
	"request_id": "req_5b7d9f1a3c5e7b9d1f3a5c7e"
}

job accepts connectors, rollup, retention, snapshot, graph or all. Anything else is 422 INVALID_QUERY with the message job must be connectors|rollup|retention|snapshot|graph|all. The run is written to the audit log. result is keyed by the work that ran: connectors also expires TTL documents in the same cycle, rollup reports the rows it aggregated, retention the rows it cleaned, snapshot the status of the engine backup.

ReindexingCopy Anchor Link

the search service cannot drop or retype a field in place, and some settings are only accepted at creation time. Those changes need a rebuild. POST /indexes/{slug}/reindex does it without downtime:

1. A shadow collection <alias>_v<n+1> is created with the current schema and settings.
2. Documents are streamed from the live collection into the shadow.
3. Synonyms and curations are replayed onto the shadow.
4. The alias is repointed atomically.
5. The old collection is kept as the job's rollback collection (dropped by the scheduler after SEARCH_ROLLBACK_RETENTION_DAYS, default 7) and a SearchSchemaMigration row records what happened.

Because searches read through the alias, there is no window where a query sees a half-filled collection. In-flight results are simply served by the old collection until the swap.

curl -sS -X POST "$AAC_API/indexes/documentation/reindex" \
  -H "Authorization: Bearer $AAC_KEY"

{
	"index": "documentation",
	"job_id": "job_01HZYC4M",
	"status": "queued",
	"status_url": "/api/v1/jobs/job_01HZYC4M",
	"request_id": "req_2f4a6c8e0b2d4f6a8c0e2b4d"
}

The answer is 202 Accepted: the rebuild runs as a background job. Poll GET /jobs/{jobId} (see Jobs) for phase, progress and, when it is completed, result.documents and result.duration_ms. A second reindex of the same index while one is active is 409 JOB_CONFLICT.

Requirements and limitsCopy Anchor Link

• Scope settings:write.
• The reindex plan feature — Free does not have it, Pro and Enterprise do.
• It runs as a scheduler job, never inside the API request. If the worker dies, the job is picked up again after its lease expires. ?sync=true blocks until the rebuild finishes and is meant for test suites only.
• Documents come from the live engine collection, not from Postgres, so anything indexed outside the platform is preserved.

When a reindex is the answerCopy Anchor Link

Checking the resultCopy Anchor Link

curl -sS "$AAC_API/indexes/documentation/stats" -H "Authorization: Bearer $AAC_KEY"

Compare documents with the pre-reindex count and watch queries_24h and avg_latency_24h_ms for a regression. A field-level diff is available in the console's schema view, which shows the live collection's fields as the engine reports them.