If people cannot find your brand in ChatGPT, Perplexity or Google's AI answers, the cause is often not your content. It is that the AI crawlers are being blocked before they ever read the page. This happens constantly on sites behind Cloudflare, and most owners have no idea it is on. The good news: it is a two-minute fix.
The short answer
Cloudflare ships a feature that adds Disallow: / rules for AI crawlers to your robots.txt, and it is often enabled by default. When it is on, bots like GPTBot (ChatGPT), ClaudeBot (Claude), PerplexityBot and Google-Extended (Google's AI features) are told not to read your site. Classic Google Search still works, so nothing looks broken, but you have quietly opted out of AI search. I found this exact problem on my own site, where the engines had literally never been able to read me.
How to check in 30 seconds
Open https://yourdomain.com/robots.txt in a browser and look for blocks like this:
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
If you see those, the AI engines you care about are being turned away. You may also see a Content-Signal: ai-train=no line, which is Cloudflare's managed signal.
I built a free tool that does this check for you and shows which AI crawlers can and cannot reach your site: the AI Crawler Checker. Paste a domain, and it reads the live robots.txt and reports each bot's status.
How to fix it (Cloudflare)
There are two separate settings, and both have to be permissive:
- In the Cloudflare dashboard for your domain, find Manage AI bot access and set "Block AI training bots" to Do not block (allow crawlers). This is the firewall rule.
- Right below it, open Manage your robots.txt and choose Disable robots.txt configuration (or the equivalent non-blocking option). This is the setting that actually writes those
Disallow: /lines into your file.
Flipping only the first one is the common mistake. The second dropdown is what rewrites robots.txt, so if it stays on "block training," the bots are still turned away. After changing both, re-check robots.txt and confirm the AI-bot Disallow: / lines are gone.
Blocking is only half the story
Letting crawlers in gets you to the starting line. It does not, on its own, get you cited. Two more things matter:
- AI crawlers mostly do not run JavaScript. Googlebot can eventually render JS, but GPTBot, ClaudeBot and PerplexityBot generally read the raw HTML. If your content is injected client-side after hydration, it can be invisible to them even when it is allowed. Make sure the important content and your JSON-LD are in the initial HTML (server-side rendering or static generation).
- Structure makes you extractable. Clean Organization or Person schema with a complete
sameAs, FAQ markup, answer-first content, and anllms.txtall help an engine understand and quote you. This is the difference between "allowed to read" and "actually cited."
Allow AI search without feeding training
A fair concern is: "I want to show up in AI answers, but I do not want my content used to train models." Those are different bots, and you can separate them. Allow the search and answer crawlers (OAI-SearchBot, ChatGPT-User, PerplexityBot, Google-Extended) while disallowing the training crawlers (GPTBot, CCBot, Bytespider). It is a legitimate middle ground, and it is exactly the kind of policy a good tool should let you set with one click rather than hand-editing robots.txt.
Doing this on WordPress
If you run WordPress, managing crawler access, llms.txt and schema by hand across a whole site is tedious and easy to get wrong. That is why I built AI Search Optimizer, a free plugin that handles all of it: AI-crawler presets, an llms.txt manager, a schema and entity builder, and an audit that scores how AI-search-ready your site is. It is on GitHub (opens in new tab) and heading into the WordPress.org directory.
The takeaway
AI search is a new front door, and a surprising number of sites have it locked without realizing. Check your robots.txt today. If the AI crawlers are blocked, unblock them, make sure your content is in the raw HTML, and give the engines the structure they need to cite you. The sites that fix this early are the ones that will show up as the habit of asking an AI instead of searching keeps growing.