Which AI crawlers should I allow in robots.txt?

Allow the retrieval and search bots: OAI-SearchBot (ChatGPT search), PerplexityBot, Claude-SearchBot, and Googlebot. Those index you for answers. The training bots (GPTBot, ClaudeBot, CCBot) are optional to block; blocking them doesn't remove you from AI answers.

Does llms.txt improve AI search visibility?

Not currently. Major engines don't read llms.txt. Google confirmed it doesn't use it and its crawlers don't even request it. llms.txt is useful for dev tools like Cursor and Copilot, but it doesn't affect whether ChatGPT or Perplexity cite you.

Why isn't my site showing up in ChatGPT or Perplexity?

Usually because crawler access alone isn't enough. If the bots aren't blocked, the gap is content and authority: extractable, structured, factual pages plus presence on sources engines trust, like Reddit and Wikipedia. A young site with few inbound links won't get cited even with a perfect robots.txt.

how to make your website visible to ai crawlers

To get cited by AI, allow the retrieval bots — OAI-SearchBot, PerplexityBot, Claude-SearchBot, and plain Googlebot — in robots.txt, and serve them clean, extractable content. That’s the part that matters. The part every guide oversells, llms.txt, does close to nothing right now. Letting crawlers in makes you eligible to be quoted, not quoted.

which ai crawlers actually matter?

Two kinds of bot show up at your site, and only one gets you cited. Training bots hoover text to train future models — blocking them changes nothing about whether ChatGPT quotes you today. Search and retrieval bots index you for answers and fetch you mid-response. Those are the ones you want in.

bot	who	what it does	let it in
GPTBot	OpenAI	trains models	optional
OAI-SearchBot	OpenAI	indexes for ChatGPT search	yes
ChatGPT-User	OpenAI	live fetch during an answer	yes (skips robots.txt anyway)
ClaudeBot	Anthropic	trains models	optional
Claude-SearchBot	Anthropic	indexes for Claude search	yes
PerplexityBot	Perplexity	indexes for Perplexity	yes
Googlebot	Google	Search + AI Overviews	yes
Google-Extended	Google	not a bot — a training opt-out flag	leave it

The one that trips people: blocking Google-Extended does not pull you out of AI Overviews. Those run off the normal Search index through Googlebot. Google-Extended only controls whether your text trains Gemini — same deal as Applebot-Extended for Apple.

how do i set up robots.txt for ai crawlers?

Don’t block the search bots. That’s most of the job. If you want your content kept out of model training but still cited, split it:

User-agent: OAI-SearchBot
User-agent: PerplexityBot
User-agent: Claude-SearchBot
User-agent: Googlebot
Allow: /

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Google-Extended
Disallow: /

Before adding anything, check what’s already there. The most common reason a site is invisible to AI isn’t a missing file — it’s a leftover Disallow: / from a staging config, or Cloudflare’s bot-fight mode quietly 403-ing every crawler. And two bots ignore robots.txt no matter what you write: ChatGPT-User and Perplexity-User fetch on a user’s behalf and skip it. Stopping those is a firewall job, not a robots.txt line.

does llms.txt do anything?

Barely, and not for search. llms.txt is a proposed standard — a markdown file at /llms.txt listing your key pages for a model to read. Fine idea on paper. In practice the big engines don’t read it. Google’s John Mueller said no AI service has claimed to use it and the logs show they don’t even request it — his comparison was to the keywords meta tag. One study of ~900 sites found zero llms.txt requests from verified frontier-lab crawlers; adoption across the top 1000 sites is 0%.

Where it earns its keep: dev tools like Cursor and Copilot, when you or an agent points them at a project’s docs. That’s real, and it’s not search visibility. Ship the file if you want — ten minutes — just don’t file it under “now AI will find me.”

what about serving markdown to ai?

This one has receipts. Serving a clean markdown version of your pages — a /page.md alternate the crawler can grab — actually gets fetched. Drupal’s Dries Buytaert logged a month of his own traffic: GPTBot pulled the markdown 34.8% of the time, OAI-SearchBot 22.7%. Markdown drops the nav, scripts, and div soup that make HTML cost roughly 4x the tokens, so the model gets your point instead of your markup.

This is exactly Time’s play. Since August 5 it serves AI crawlers a markdown version of its articles — with ads baked in that human readers never see — while Googlebot gets normal HTML and the training-only GPTBot gets a flat 406. Retrieval bots in, training bot out, humans none the wiser. Whatever you make of ads only a machine can read, the plumbing is a correct read of who’s worth serving.

why am i still not in any ai answers?

Because none of the above makes you cited — it makes you eligible. Getting quoted is decided by the content, not the config: extractable, structured, specific writing — definitions, numbers, comparisons, step lists — sitting where the engine looks. Perplexity leans on Reddit hard; ChatGPT favors Wikipedia-shaped facts. About 90% of ChatGPT’s citations come from outside Google’s top 20, so pure ranking isn’t the lever either.

Case in point, this site. neuralshit’s robots.txt is wide open, the sitemap’s clean, there’s no noindex anywhere — textbook. Google has indexed zero of its pages. The crawlers aren’t the problem; a young domain nobody links to that never got verified in Search Console is. The checklist here is table stakes. Discovery and something worth quoting are the actual job.

So: making your site “visible to AI” is an afternoon of robots.txt and a clean markdown route. Staying invisible after that is a content problem wearing a technical costume.