File uploads

The file connector turns uploaded documents into searchable records. It accepts PDF, DOCX, plain text and Markdown. CSV is not part of it — CSV has its own connector type.

How it worksCopy Anchor Link

1. In the console, open Index → Connectors → Upload files and pick the files. The console asks the API for a signed upload URL per file. The type and the plan's size limit are checked before a URL is signed.
2. The browser uploads straight to object storage (the private connector-uploads bucket). File bytes never pass through the API.
3. The console creates a file connector that lists the uploaded objects.
4. Running the connector queues a file_import job. The scheduler claims it, downloads each object, checks its magic bytes, extracts the text and imports one parent document per file.

The declared content type is never trusted on its own: a .pdf must start with a PDF signature and a .docx must be a real Word archive. A file whose bytes do not match its extension is refused.

LimitsCopy Anchor Link

A DOCX is a zip archive; its directory is checked before anything is unpacked, so an archive that would expand past 50 MB (or an entry more than 100× its compressed size) is dead-lettered as ARCHIVE_TOO_LARGE without being inflated. A file that takes longer than the time budget is dead-lettered as EXTRACT_TIMEOUT.

The size limit is enforced twice: when the upload URL is signed and again when the worker downloads the object, so a client that misreports the size cannot make the worker read more than the plan allows.

Quota and chunkingCopy Anchor Link

Each file becomes one parent document. Long files are chunked later in the normal ingest path, and quota still counts one document per file — the same rule as any other long document.

Progress and errorsCopy Anchor Link

The job is visible in Sync and through GET /jobs/{jobId}. A completed file_import job reports:

{
	"loaded": 12,
	"indexed": 11,
	"failed": 1,
	"skipped": 0,
	"files": [
		{
			"key": "connector-uploads/…/handbook.pdf",
			"filename": "handbook.pdf",
			"documentId": "…",
			"status": "indexed",
			"pages": 42,
			"chars": 81233
		},
		{
			"key": "connector-uploads/…/scan.pdf",
			"filename": "scan.pdf",
			"documentId": "…",
			"status": "failed",
			"error": "…"
		}
	]
}

A file that cannot be read (encrypted PDF, corrupt archive, no text) becomes a row in the index's dead letters with the reason. The run goes on with the remaining files; one bad file never fails the whole import.

RetentionCopy Anchor Link

Uploaded objects expire 7 days after upload (bucket lifecycle rule). The indexed documents stay. Running the connector again after the objects expired dead-letters the missing files — their documents are not deleted.

Known limitsCopy Anchor Link

• OCR is opt-in. A scanned PDF has no text layer; without OCR it is dead-lettered as "no text". Enable OCR per index in Search settings → OCR (settings.search.ocr): pages without a text layer that contain an image are sent to an OCR service you run (an HTTP endpoint, or a self-hosted tesseract-server), at most 100 pages per file. A failed OCR call dead-letters the file as OCR_FAILED; a file that also has text pages is indexed with that text. The job result reports ocr_pages.
• The file list is fixed after creation. To add or replace files, create a new file connector.
• Extraction errors are recorded as dead letters but cannot be retried from there; upload the fixed file in a new connector.