LLMs.txt and AI Crawlers

AI Crawlers Are Not Search Engine Crawlers

When Googlebot crawls your site, it indexes your pages and potentially shows them in search results, sending you traffic. When GPTBot or ClaudeBot crawls your site, the purpose is different: they collect content for AI model training or to generate real-time AI search answers.

This distinction matters because the value exchange is different. Google sends you traffic in return for crawling. AI crawlers may use your content to generate answers without linking back. Originality.ai found that only 38% of AI-generated answers include a citation to the source they drew from (Originality.ai, 2025). Understanding this helps you make informed decisions about what to allow.

Cloudflare reported that AI bot traffic increased by 4,000% between January 2024 and January 2026 across their network (Cloudflare, 2026). AI crawling is no longer marginal; it is a significant portion of total bot traffic for many sites.

Known AI Crawler User Agents

Each AI company uses a distinct user agent string. Knowing these is essential for writing accurate robots.txt rules:

A Dark Visitors analysis found that only 5% of the top 1,000 websites explicitly address AI crawlers in their robots.txt (Dark Visitors, 2025). Most site owners are unaware these bots exist. Meanwhile, Ahrefs reported that GPTBot alone requests data from over 600 million unique URLs per month (Ahrefs, 2025).

Controlling AI Crawlers with robots.txt

You manage AI crawler access the same way you manage traditional bots: through robots.txt directives. The key difference is that you may want different policies for different AI bots.

# Block all AI crawlers from training data
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

# Allow Perplexity (sends traffic via citations)
User-agent: PerplexityBot
Allow: /

# Allow ChatGPT browsing (user-initiated, cites sources)
User-agent: ChatGPT-User
Allow: /

Notice the strategic distinction: you might block training-data crawlers (they use your content without attribution) while allowing real-time search bots (they cite you and send traffic). This is the nuanced approach most SEO professionals recommend.

What Is llms.txt?

The llms.txt specification was proposed by Jeremy Howard (co-founder of fast.ai and Answer.AI) in late 2024 as a standard way for websites to communicate with large language models. While robots.txt tells crawlers what they can and cannot access, llms.txt tells AI systems what your site is about and which content is most important.

The file is placed at your domain root: https://example.com/llms.txt. It uses a simple Markdown-based format that both humans and machines can read. The specification is still evolving, but early adoption is growing. As of early 2026, thousands of sites have published llms.txt files, including documentation sites, developer tools, and content publishers.

llms.txt Structure

# Site Name

> Brief description of what this site is about.

## Docs

- [Getting Started](https://example.com/docs/start): Introduction to the platform
- [API Reference](https://example.com/docs/api): Complete API documentation
- [Tutorials](https://example.com/docs/tutorials): Step-by-step guides

## Optional

- [Blog](https://example.com/blog): Latest articles and updates
- [Changelog](https://example.com/changelog): Version history

The format uses a Markdown heading for the site name, a blockquote for the description, and organized link sections. The "Docs" section contains your most important content. The "Optional" section contains supplementary material. AI systems can prioritize content based on which section it appears in.

Should You Block or Allow AI Crawlers?

This is the most debated question in SEO right now. There is no single right answer. The decision depends on your business model and traffic sources:

Advanced: The Full AI Crawler Stack

A comprehensive AI visibility strategy uses three files together:

1. robots.txt: Controls which bots can access which pages. This is the access control layer.
2. llms.txt: Provides AI systems with a structured guide to your site. This is the context layer.
3. Structured data (JSON-LD): Marks up individual pages with entities, relationships, and attributes. This is the semantic layer.

Together, these three files give AI systems a complete picture: what they can access, what matters most, and what each page means. Sites that implement all three layers are positioned for maximum visibility in AI-generated answers.

Monitoring AI Crawler Activity

To make informed decisions, you need to know which AI bots are visiting your site and how often. Here is how to monitor:

• Server access logs: Filter for known AI user agent strings. Most hosting providers give you access to raw logs.
• Cloudflare / CDN dashboards: Cloudflare identifies AI bot traffic separately from regular bot traffic. Their Bot Analytics dashboard shows crawl frequency by bot type.
• Dark Visitors: A free tool that monitors which AI bots are crawling your site and generates robots.txt rules based on your preferences.
• Screaming Frog log analyzer: Import server logs and filter by user agent to see crawl patterns, frequency, and which pages AI bots focus on.