Is your website blocking AI? How to check and fix AI crawler access in 2026

How to check if your website is blocking AI crawlers: five places a block can hide

Each AI company runs several bots, and each layer of your stack can treat them differently. A training crawler, a search crawler and a fetch triggered by someone typing a question into ChatGPT can all get different answers from the same website.

1. robots.txt as the public receives it, including anything your CDN adds on the way out.Check 1
2. Your CDN, host or firewall. A CDN (content delivery network) such as Cloudflare sits in front of your site and can return a 403, a 429 or a challenge page whatever robots.txt says.Check 2
3. The page itself: login walls, redirect chains, JavaScript-only content, noindex and snippet controls.Check 3
4. Google's Search Console setting for AI Overviews and AI Mode, which has nothing to do with robots.txt.Check 4
5. What actually reached your server, in your logs or bot reports.Check 5

For Google, a page has to be indexed and eligible to show a snippet in ordinary Search before it can appear as a supporting link in AI Overviews or AI Mode. Google's guidance also names CDN and hosting blocks specifically as something to rule out.¹ Eligible just means eligible to be chosen as a link, though. Getting chosen as a supporting link is a separate question.

How many websites block AI crawlers?

Fewer than you'd think. And even fewer deliberately.

The ai-visibility.org.uk quarterly robots.txt study, run by Mark McNeece at 365i, crawled 1,744 websites. In that sample:²

GPTBot also gets blocked roughly twice as often as ChatGPT-User. That one is interesting. Plenty of site owners blocking OpenAI have shut out the training crawler and left the user-triggered fetcher alone, which may or may not be what they intended.

These figures count declared rules in robots.txt. They can't show whether a firewall let the unnamed bots through, or whether any crawler obeyed. The number I'd pay most attention to is the 84.2%. Most sites are running on whatever defaults their platform, plugin or CDN shipped with, and have never checked.

Which AI crawler do you need to allow for ChatGPT, Claude, Perplexity and Google?

Be specific here. Almost every bad AI crawler decision I come across starts with blocking the wrong bot.

OpenAI runs three main agents. GPTBot collects content that may be used to train OpenAI's models. OAI-SearchBot surfaces sites in ChatGPT search, and OpenAI recommends allowing it if you want to appear there. Sites that block it can still turn up as plain navigational links. ChatGPT-User fetches a page when someone asks ChatGPT to look at it, and OpenAI says robots.txt rules may not apply because a person started the request.^3,4 A robots.txt change takes about 24 hours to reach ChatGPT search.³

So blocking GPTBot keeps you out of training data. ChatGPT search is governed by OAI-SearchBot.

Check 1: Is your robots.txt blocking GPTBot, OAI-SearchBot or other AI crawlers?

Fetch https://yourdomain.com/robots.txt from a browser or the command line, from outside your CMS, and test the exact page URL you care about against each agent. The rules for which group a crawler follows and how paths match are set out in RFC 9309.⁹ Three mistakes come up again and again.

The blanket disallow

User-agent: *
Disallow: /

Every compliant crawler, Googlebot and bingbot included, is told to stay out. If your file opens like this and you want to be found in search at all, fix it before thinking about AI-specific rules.

Named groups ignore the wildcard

User-agent: *
Disallow: /private/

User-agent: GPTBot
Allow: /

GPTBot finds its own group and follows that group only. The Disallow: /private/ line under * doesn't apply to it, so /private/ is open. To keep it closed, repeat the rule inside the GPTBot group:

User-agent: GPTBot
Disallow: /private/

robots.txt was never security, either. Anything private needs a login or a server-level block.

The file doesn't load properly

curl -i https://yourdomain.com/robots.txt
curl -I https://yourdomain.com/important-page/

Check the status code, any redirects and the body that comes back. A 403 or 5xx on robots.txt is a problem in its own right, and crawlers handle it in different ways. One more trap: if robots.txt blocks a page, crawlers never get to read that page's noindex tag, so they can't act on it.¹⁰

Check 2: Is Cloudflare or your web host blocking AI bots?

This is where most accidental blocks live.

Cloudflare's managed robots.txt setting, available on every plan, adds AI crawler rules to the file your visitors receive. If you already have a robots.txt, Cloudflare puts its rules in front of yours and serves the two as one file. If you don't have one, Cloudflare serves its own. Its documented example disallows GPTBot, ClaudeBot, CCBot, Google-Extended, Applebot-Extended, Amazonbot, Bytespider and meta-externalagent.¹¹ Cloudflare also says that on the Free plan, a domain with no robots.txt of its own and managed robots.txt switched off will show crawlers its Content Signals Policy instead.¹¹

In practice that means WordPress or your SEO plugin can show you one robots.txt while the public gets another. Always check the live URL.

Where to look on each platform:

• Cloudflare: the served robots.txt, the managed robots.txt toggle, AI Crawl Control and Security Events. AI Crawl Control is the enforcement side and can block a named AI crawler outright, whatever robots.txt says.¹¹
• Squarespace: Settings → Crawlers. The AI crawler option edits robots.txt for you, and you can't edit the file directly.¹²
• WordPress: the live file, your SEO plugin, any security plugin, and whatever the host or CDN adds on top.
• Wix, Shopify or Webflow: compare the platform's robots settings with the file actually served, then check any app firewall in front of it.

What if your host is blocking AI crawlers and robots.txt says allow?

I found this one the hard way. The robots.txt on this site allowed CCBot, and CCBot was still being turned away. The block sat at server level on HostGator, somewhere I couldn't see or edit, and it took a fair amount of back-and-forth with their support before it was lifted. It's the reason the checker on this site runs a live CCBot request next to the robots.txt read.

So if your robots.txt says allow and a live request gets a 403, ask your host directly about server-level crawler rules. Be specific about the genuine bot. A checker sends its request from its own IP address, and a host can reasonably say it blocks that IP and leave it there. What you need to know is how real Common Crawl, OpenAI or Anthropic traffic gets treated.

Check 3: Can AI crawlers actually read your page once they get in?

A 200 response is a good start. Now look at what came back with it.

Google says its generative AI features use publicly accessible, crawlable content,¹³ and the same holds for every other AI search product. Pages that return 200 but hand a crawler a JavaScript shell with no text in it, a cookie or login wall, a bot challenge dressed up as a normal page, or a redirect that dumps everything on the home page all fail quietly. Nothing in your analytics will flag it. Snippet controls matter here too. nosnippet keeps a page out of AI Overviews, but it strips your ordinary search snippet at the same time.¹⁰

View the source of the page, or fetch it with curl, and check the words a buyer would need are in the HTML. Test a representative article and a service page. The home page is usually the best-built page on the site, so it tells you the least.

Check 4: Why isn't your site showing in Google AI Overviews or AI Mode?

Blocking or allowing Google-Extended changes nothing here, whatever you've read.

AI Overviews and AI Mode run on Googlebot access and ordinary Search eligibility. On top of that, Search Console now has its own control under Settings → Search generative AI, with three options: include (the default), exclude, or inherit from a parent property. Excluding a site removes its links and content from AI Overviews, AI Mode and generative AI features in Discover. Google rolled the control out to all websites worldwide as of 31 August 2026. It doesn't affect AI training, and Google points to Google-Extended for that. Changes generally take a few days to apply.¹⁴ Google's optimisation guide now states plainly that a site must be set to include to be eligible at all.¹³

The setting exists because of the UK. On 3 June 2026 the Competition and Markets Authority imposed a conduct requirement on Google under the Digital Markets, Competition and Consumers Act, giving publishers a way to opt out of AI search features without losing their place in ordinary results.¹⁵ The requirement also covers opting out of AI fine-tuning, clearer attribution with direct links, and directory and page-level controls that haven't arrived yet.^15,16 The current control works at property level.

Note: If you run subdomains or country folders as separate Search Console properties, check each of them, because an exclude on the parent flows down to any child set to inherit.

Check 5: How to see if GPTBot, ClaudeBot and other AI bots are visiting your site

Server logs show the requests that reached your server and the response it sent. Don't expect Google Analytics to help here. It only counts visits that run its JavaScript tag, and crawlers don't, so GA4 will show no AI bots even when they're visiting every day. You need something that records requests on the server side, and what's available depends on your setup:

• Behind a CDN such as Cloudflare: anything the CDN stopped never reaches your server, so check its security events as well as your server logs.
• Shared hosting with cPanel: raw access logs, plus AWStats, which lists named bots in its robots and spiders section. No command line needed.
• Self-hosted WordPress: a bot-logging plugin records crawler visits on the server side, so you get a bot log even if your host doesn't provide one.
• Wix: built-in bot reports under Analytics & Reports → Reports → Marketing & SEO, including bot traffic by page and response status over time.¹⁸
• Shopify or Squarespace: you won't get bot logs, so rely on the live checks and your platform's crawler settings.

Whatever you're on, referral visits from chatgpt.com or perplexity.ai in your analytics are a useful indirect sign. They can't show whether every crawler gets in, but they do prove some of your pages are being fetched and cited.

If you do have logs, remember that user-agent strings are trivial to fake. Match any suspicious IP against the operator's published list before you draw conclusions. OpenAI and Perplexity both publish JSON IP lists from their crawler pages.^3,6

# Adjust the log path and format for your server
grep -Ei 'GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|PerplexityBot|CCBot|bingbot' /path/to/access.log | tail -50

What the response codes mean for the page you're testing

A missing entry isn't proof of a block. Neither is one 403 from a tool pretending to be GPTBot.

Free AI crawler checker vs other tools: which should you use?

Each tool answers a different question. A robots tester reads your stated rules. A live checker sees how your server responds to its own network. Only your logs and the operators' own tools get close to genuine bot traffic. Google makes the point itself: no third-party tool has access to its internal ranking or AI systems,¹³ and that includes ours.

Does llms.txt help AI crawlers access your site?

No. Not for access.

Google says AI Overviews and AI Mode need no new AI text file or special markup.¹³ An llms.txt file can't undo a 403, an exclude setting in Search Console or a login wall. If you already keep one for an agent workflow, keep it accurate. Otherwise spend the time on access first, then on pages that answer your buyers' questions directly. The guide to getting cited by ChatGPT covers that second part.

What to do when AI bots ignore robots.txt

robots.txt is a request. Cloudflare's own documentation says compliance is voluntary and some operators ignore it.¹¹ If you need a block that holds, set a server or WAF rule tied to a verified identity or IP list. Then test it, because an over-broad rule is exactly how sites end up blocking the crawlers they wanted.

The Perplexity episode shows why evidence matters. In August 2025 Cloudflare published an analysis attributing undeclared crawling to Perplexity on sites that had blocked its declared bots.¹⁹ Perplexity rejected Cloudflare's account.²⁰ A user-agent string couldn't settle that argument, and it won't settle one on your server either. Compare request patterns, source IPs and response logs before blaming a named company for a strange fetch.

How to block AI training crawlers but stay visible in ChatGPT and AI search

This is the setup many businesses actually want: keep content out of model training, stay findable in AI search. A narrow robots.txt policy does most of it. Treat this as a starting point. It doesn't cover every AI operator.

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Googlebot
Allow: /

User-agent: bingbot
Allow: /

Keep private paths protected at the server. Decide on Google-Extended separately, since it covers Google's training uses and leaves AI Overviews alone. And a ChatGPT-User line won't reliably stop user-initiated fetches, given OpenAI's own caveat.

Newer signals are worth knowing about. Cloudflare's Content Signals let a robots.txt state separate preferences for search, real-time AI input and model training.²¹ The Really Simple Licensing (RSL) standard and the IETF's AI Preferences working group are both working on clearer ways to state permissions.^22,23 These describe what you'd like to happen. Before relying on one, check whether the operators you care about honour it.

The 10-minute AI crawler access check

1. Open your public /robots.txt and confirm it loads with a 200.
2. Test one high-value URL against Googlebot, bingbot, OAI-SearchBot, GPTBot, ClaudeBot and PerplexityBot.
3. Compare the public file with your CMS, host and CDN settings. Look for rules you didn't write.
4. Run the AI crawler checker and compare the browser, GPTBot and CCBot responses.
5. View the page source and confirm the key content is in the HTML.
6. In Search Console, check Settings → Search generative AI for the property, and for any parent it inherits from.
7. Use URL Inspection to confirm indexing and snippet eligibility.
8. Look for 403, 429 and challenges in CDN events, server logs or your platform's bot reports, and verify crawler IPs where you can.
9. Fix the layer that failed, re-run the check on the same URL, and note the date and result.