AX surfaces as a machine-readable feed
Four root surfaces plus a per-page Markdown projection make a site machine-readable; the root four are cached per site, and all five are stamped `X-Kywi-AX`.
Six URLs answer for a Kywi site without an API key, a session or a schema: four at the root, plus a Markdown twin of any page and a status probe. They are generated from published content on demand, so they are a feed you poll, not an export you schedule.
The six surfaces
Every AX surface is public and unauthenticated, and every successful
response from the five content surfaces (not the status probe) carries an
X-Kywi-AX header naming which surface produced it — that
header is how a consumer tells a generated feed apart from an ordinary page that happens
to return text.
| Name | Type | Default | Description |
|---|---|---|---|
/llms.txt | text/plain | — | Site outline for a model, depth-limited by |
/llms-full.txt | text/plain | — | Every included page as |
/sitemap.xml | application/xml | — | Indexable pages with priority and change frequency. |
/robots.txt | text/plain | — | Crawl policy, optionally referencing the sitemap and llms.txt. |
/<path>.md | text/markdown | — | One page, projected to Markdown. Off by default. |
/api/v1/ax/status | application/json | — | Which surfaces are on, and how many pages each is drawing from. |
curl -sD- -o /dev/null https://example.test/llms.txt
# cache-control: public, max-age=3600
# content-type: text/plain; charset=utf-8
# x-kywi-ax: llmsCaching, and why a publish shows up immediately
Those four bodies are held server-side under a per-site key for ax.cacheTtl seconds, 3600 by default, which is also the max-age your client is told. The TTL is not what governs freshness after an edit: every content mutation drops the site's entire AX cache namespace, so a publish is reflected on the very next request. The hour is the ceiling for a site nobody is editing, not the delay after an edit.
The Markdown projection and the status probe are the exceptions — neither is cached and neither sends a Cache-Control, so a crawler pulling thousands of .md URLs hits the database each time.
The Markdown twin of a page
/some/path.md is not a route — middleware rewrites it onto the AX handler, which resolves the node by its URL Override or its full tree path, the override winning. On a path-prefixed child site, a path under the child's prefix resolves in that child site. No locale prefix is stripped: /fr/about.md looks for a node literally at /fr/about and honestly 404s if there isn't one. A page flagged to be excluded from search 404s here too.
The body is deterministic, not generated: a three-line header, then an HTML to Markdown conversion of the page's stored body — or of its layout, serialized, when that column is empty, which is the normal case for a layout-composed page. This surface reads no cookie and resolves no audience, so it serves the Default arm (or a concluded experiment's locked winner). A personalized page has exactly one Markdown twin.
# Slugs, paths, and moving a page
URL: https://www.example.test/authoring/slugs-paths-and-moving-a-page
Type: docPage
A slug is derived from the title once, at creation; a moved page’s old URL
redirects to the new one, and URL Override gives a page its own address.
A page’s URL is the chain of slugs from the site root down to it: …
## What renaming a page does
…When the body opens with its own h1, as every page on this docs site does, that heading leads the document and no separate # title line is printed, so the document has one H1. The URL: line names the page's URL Override when it has one, else its tree path.
Keeping one page out of the sitemap
Two page settings decide what a page contributes to these surfaces. Exclude from search & AX is the broad one: the page is left out of /sitemap.xml, both llms files, its .md twin and site search. Exclude from Sitemap is the narrow one: the page leaves /sitemap.xml and nothing else, so an assistant reading /llms.txt still finds it. A page is in the sitemap only when both are off, and the Navigation tab shows Exclude from Sitemap as checked and locked while the broad one is on. Over the API the flag is excludeFromSitemap on a public content read, so a headless consumer building its own sitemap can honor it.
Neither flag is needed for a page outside its display window: it is in none of the AX surfaces, its .md twin included.
Absolute URLs, and the host they are built from
Every <loc> and URL: line is absolute, and the origin is resolved once per generated body: the site's own baseUrl (on its sites[] entry) wins, then the request host when it matched a configured domain or alias, then the global ax.baseUrl, then https:// plus the site's domain. An unmatched host never counts — these bodies are cached with no origin in the key, so a client-supplied Host would pin itself into everyone else's feed for the rest of the TTL. The scheme is inferred rather than read from the request, so behind a TLS-terminating proxy, or wherever a site's public origin is not its domain, set that site's baseUrl. ax.baseUrl is one value for every site, so on a multi-site install it is only a fallback.
Checking what is on
/api/v1/ax/statusNo authAnswers for the site the request host resolves to. counts is
computed live from published content — the quickest way to tell a misconfigured surface
from an empty one. What each surface includes is configured in
the
ax block.
{
"data": {
"enabled": true,
"surfaces": {
"llmsTxt": true, "llmsFull": true, "sitemap": true,
"robots": true, "jsonLd": true, "markdown": true
},
"urls": {
"llmsTxt": "/llms.txt", "llmsFull": "/llms-full.txt",
"sitemap": "/sitemap.xml", "robots": "/robots.txt"
},
"counts": { "published": 73, "indexable": 73, "excluded": 0, "sitemapExcluded": 0 }
}
}indexable counts the pages actually in the sitemap, so it drops for pages flagged Exclude from Sitemap. sitemapExcluded counts every page carrying that flag, including ones already out for another reason, and excluded counts the broad search-and-AX flag.
The caching contract these headers sit inside is Caching semantics for an integrator.