XML, RSS, YML and YAML feeds
Index RSS/Atom feeds, Yandex Market YML and Google Merchant product feeds, any XML catalogue, or a YAML list of records.
Two feed connector types read structured files from a public URL:
| Type | Reads | Typical source |
|---|---|---|
xml | RSS 2.0 / RSS 1.0, Atom, Yandex Market YML, Google Merchant, any XML with an item path | blog feed, marketplace export, ERP XML |
yaml | a YAML list of records, a map with a list under a key, a map of records | Jekyll _data, static-site catalogues, config-style exports |
Both use the same field mapping as json and csv (idField, titleField,
contentField, urlField, tagsField), the same full / delta modes with
checksum skip, schedules, maxDocuments and pruneDeleted.
XML: format detection
The format is detected from the root element; set dialect only to override it.
| Root | Dialect | Records | Fields produced |
|---|---|---|---|
<rss> / <rdf:RDF> | rss | channel/item | id (guid or link), title, content (content:encoded or description, HTML stripped), url, published_at, author, category, tags from <category> |
<feed> | atom | entry | id, title, content / summary, url (alternate link), published_at, updated_at, author, tags from category@term |
<yml_catalog> | yml | shop/offers/offer | id attribute, title (name, or typePrefix vendor model), content (description + vendor + params), price, oldprice, currency, category (name resolved via shop/categories), category_path, vendor, vendor_code, available, image, barcode, params, one param_<name> field per <param name> |
<rss> with g: items | google | channel/item | g:id, g:title, g:description, g:link, price + currency parsed from g:price ("159.00 USD", "1.234,50 EUR"), sale_price, availability, available, brand, image, category (g:product_type), google_product_category, condition, gtin, mpn |
| anything else | generic | itemPath | the element's children and attributes (attributes lose @; an element with attributes reads as its text, price.currency reads the attribute) |
Dialect fields are copied onto each document automatically. Declare the ones you
want to filter or facet on (for example price as float, category,
vendor, brand, currency as faceted string) in the index schema; the rest
are stored and returned with hits.
{
"type": "xml",
"config": {
"url": "https://shop.example.com/export/yml.xml",
"maxDocuments": 5000
}
}A generic catalogue:
{
"type": "xml",
"config": {
"url": "https://erp.example.com/catalog.xml",
"itemPath": "catalog.products.product",
"idField": "sku",
"titleField": "name",
"contentField": "summary",
"urlField": "page",
"fields": ["price", "price.currency", "brand"]
}
}YAML
Records are found in this order: itemPath if set; a top-level list; the one
key whose value is a list of records; a map of records (the key becomes id, and
title when the record has no title/name); several --- documents.
{
"type": "yaml",
"config": {
"url": "https://raw.githubusercontent.com/org/site/main/_data/products.yml",
"idField": "sku",
"fields": ["price", "category", "dimensions.width"]
}
}fields copies extra values; dotted paths read nested ones and are stored with
_ (dimensions.width → dimensions_width).
Mapping fallbacks
| Document field | Order |
|---|---|
external_id | idField → id → external_id → guid → hash of title + url |
title | titleField → title → name |
content | contentField → content → description → body → summary → text → title |
url | urlField → url → link |
HTML in content is stripped. A record repeated in one feed is written once (the last occurrence wins).
Safety and limits
- Only public
http(s)URLs; every redirect hop is checked (no private ranges). - Responses above 5 MB are refused (
TOO_LARGE). - XML with
<!ENTITY>declarations is refused (INVALID_XML) — no entity expansion, no external entities. A plain<!DOCTYPE yml_catalog SYSTEM "shops.dtd">is fine; nothing is fetched. - YAML uses the core schema only (no custom tags) and caps alias expansion
(
INVALID_YAML). - Errors name the cause: an HTML page instead of a feed, JSON instead of XML, a
wrong item path (
INVALID_CONFIG), malformed markup with the line number.