What is llms.txt? The New AI Crawler Standard Explained
The web is undergoing a massive transformation. Rather than just human users navigating pages with traditional browsers, millions of artificial intelligence (AI) models, Large Language Models (LLMs), and autonomous crawlers (such as GPTBot, ClaudeBot, and Perplexity) scan the internet daily to extract information, answer developer queries, and synthesize answers.
The Problem with Traditional Sitemaps and Robots.txt
Traditional XML sitemaps and robots.txt files were engineered decades ago for classic search engines like Google and Bing. While they excel at link discovery and crawl-budget permissions, they do not provide AI models with clean, condensed markdown summaries, semantic priority, or full text context.
Enter the llms.txt Standard
The llms.txt format is a community-driven open standard hosted directly at the root of any domain (https://yourdomain.com/llms.txt). Formatted in standard Markdown, it provides a high-signal index map of a website's primary documentation, guides, APIs, and key pages designed specifically for consumption by LLMs.
What is llms-full.txt?
For deep technical documentation and API specifications, the standard includes a companion file called llms-full.txt. While llms.txt acts as a table of contents with URLs and short summaries, llms-full.txt aggregates the full text of all documentation pages into a single file. This allows AI assistants to ingest an entire product manual in a single HTTP request without having to perform dozens of round-trip web requests.
Real-World llms.txt Syntax Example
> Complete API documentation and integration guides for developer workflows.
## Getting Started
- [Quickstart Guide](https://example.com/docs/quickstart/): Step-by-step setup in under 5 minutes.
- [Authentication](https://example.com/docs/auth/): API key configuration and security best practices.
## Core APIs
- [REST API Endpoints](https://example.com/api/v1/): Complete reference for endpoints and payloads.
- [Webhooks](https://example.com/docs/webhooks/): Event subscriptions and payload signatures.
## Optional
- [Full Documentation Export](https://example.com/llms-full.txt) [full]
How to Create and Optimize Your llms.txt
1. Use Curated High-Signal Pages: List only valuable documentation, getting started guides, and pricing/contact information. Avoid listing privacy policies or pagination links.
2. Always Use Absolute URLs: AI crawlers index links globally; relative paths can result in broken connections.
3. Validate Your Syntax: Use our free LLMs.txt Validator and LLMs.txt Generator to build and audit your files in seconds.