What the file is
llms.txt is a proposed convention for providing a concise overview and links to useful documentation. Its presence does not prove that an engine reads or prioritizes it. Support must be established for a particular consumer rather than assumed for all AI systems.
Google's documented position
Google says that it does not use llms.txt files for Search and that maintaining such a file neither helps nor harms visibility or rankings there. This includes its generative search features. Do not describe the file as a Google ranking factor.
What it does not replace
| Mechanism | Actual role |
|---|---|
| robots.txt | Communicates crawler access rules |
| Public HTML and links | Make content accessible and discoverable |
| Canonical and status handling | Describe preferred and available pages |
| Sitemap | Lists URLs for discovery |
| llms.txt | Optional overview for consumers that choose to use it |
Search and training are separate
OAI-SearchBot and GPTBot have distinct documented purposes. A file listing pages does not override access rules or establish whether material can appear in search. Check the crawler's actual documentation.
When maintaining it can be reasonable
A documentation publisher may find a concise index useful for a known consumer or internal workflow. Keep links current, avoid publishing private material, and treat the file as an additional representation of existing information. Do not promise citation lift or claim its link order controls a search engine.
What to fix first
If an important page returns an error, lacks usable HTML, or is isolated from navigation, address that problem. An optional text overview is not a substitute for a useful, technically accessible page.