Is your site ready for
AI search?

Why AI readiness is its own thing

Ranking on Google and being usable by an AI assistant overlap, but they aren't the same. AI models crawl, parse and summarise your site to answer questions, and they can only recommend a page they were able to reach and understand. If your robots.txt blocks their crawler, or your content is buried in JavaScript with no structure, you're invisible to them no matter how you rank.

AI readiness is the check on that plumbing: can the models get in, and can they make sense of what they find?

Is my website AI ready?

Your site is AI ready when four things are true. AI crawlers are allowed in. The HTML your server sends already contains your content, not an empty shell that fills in after JavaScript runs. The page says clearly what it is, through a title, a description, headings and structured data. And crawlers can find the rest of your pages through a sitemap.

A site can pass one of these and fail the rest. A perfect robots.txt doesn't help if the page arrives with no text in it, which is why the test scores them together.

Every check in your report, explained

The report groups nine checks into three categories: AI access, AI understanding and Technical. Each one passes, gets a warning or fails, and every gap comes with a fix.

AI crawlers can access your site

Reads your robots.txt for GPTBot, ClaudeBot, CCBot, Google-Extended, PerplexityBot and the older anthropic-ai token, and flags any that are blocked from the whole site. Blocking some is a warning; blocking all the major ones is a fail. No robots.txt counts as open.

llms.txt file present

Looks for a file at /llms.txt. A missing one is only ever a warning, because the standard is still a proposal.

XML sitemap available

Passes when robots.txt lists a sitemap or one answers at /sitemap.xml, so crawlers can discover pages your links don't reach.

Structured data (JSON-LD)

Checks your homepage for a JSON-LD block, the schema.org markup that states what the page is: an organization, a product, an article.

Title and meta description

Both present passes, one missing is a warning, neither is a fail. They're often the first summary any system reads.

Clear heading structure

Exactly one H1 and at least one H2 passes. More than one H1, or an H1 with no H2s under it, is a warning. No H1 at all is a fail.

Readable content

An approximate Flesch reading ease score of your homepage text: 50 or higher passes, 30 to 49 is a warning, below 30 fails. Pages with fewer than 120 words are marked not scored and left out of your total.

Served over HTTPS

Passes when your homepage answers over a secure connection.

Mobile-friendly viewport

Checks for a viewport meta tag, the line that tells browsers to fit the page to a phone screen.

Training crawlers vs AI search crawlers

Most AI companies run more than one crawler, and they do different jobs. Training crawlers collect pages to train future models. Search crawlers fetch pages to answer a question right now, which is how a page gets cited with a link. Blocking one doesn't block the other, so you can opt out of training and still appear in AI answers.

OpenAI

GPTBot collects training data. OAI-SearchBot powers ChatGPT search, and ChatGPT-User fetches a page when a user asks ChatGPT to open it.

Anthropic

ClaudeBot collects training data. Claude-SearchBot and Claude-User fetch pages for Claude's search and for users' requests. You may still see the retired anthropic-ai token in older robots.txt files.

Perplexity

PerplexityBot indexes pages for Perplexity's answers, and Perplexity-User fetches a page when a user asks about it.

Google

Google-Extended is a robots.txt token, not a crawler. It controls whether your content is used for Gemini models. AI Overviews and AI Mode come from Google Search and use Googlebot, so blocking Google-Extended doesn't remove you from them.

Common Crawl

CCBot builds a public archive of the web that many AI models have been trained on.

When robots.txt isn't the problem

A clean robots.txt doesn't guarantee AI can read you. These are the other common reasons a site stays invisible:

Firewall or CDN bot protection

Some CDNs and security plugins block AI crawlers by default, answering them with an error page. This test fetches your site as SEOLadders, not as each AI crawler, so it can't see a rule that only targets AI bots. Check your CDN's bot settings.

Content that needs JavaScript

Most AI crawlers read the HTML your server sends and don't run JavaScript. If your text and headings only appear after scripts load, they may see an empty page. This test reads raw HTML too, so missing headings or a not-scored readability check can be a sign of this.

A noindex tag

A noindex meta tag or header keeps a page out of Google, which also keeps it out of AI Overviews and AI Mode.

Thin or vague copy

A homepage that never says plainly what you do, for whom, gives a model nothing to quote.

The signals that matter most

Start with access: make sure robots.txt doesn't block the AI crawlers you want to be cited by, and that you expose a sitemap. Then help them understand: add JSON-LD structured data, a clear title and description, one H1 with a sensible heading hierarchy, and readable copy. An llms.txt file is a cheap extra on top.

None of these guarantees a citation, because models still weigh quality and relevance, but missing them can rule you out before you're even considered.

Does llms.txt matter?

llms.txt is a proposed standard: a plain-text file at your site root that lists your most important pages with a short description of each. It's cheap to publish and some AI agents and developer tools read it.

Whether the big AI search engines use it is still unclear. None has committed to it publicly, and Google has said you don't need it, or any other special file, to appear in its AI features. Publish one because it costs you ten minutes, not because it will move your citations on its own. That's why a missing llms.txt only ever earns a warning here.

How to close the gaps

Work top-down: fix anything blocking access first, then add the understanding signals, then polish the technical basics. Re-run the test after each change to confirm the score moves.

1. 1Unblock the AI search crawlers you want citing you in robots.txt
2. 2Check your CDN or firewall isn't blocking AI bots
3. 3Publish a sitemap.xml and an llms.txt
4. 4Add JSON-LD schema for your key pages
5. 5Fix titles, descriptions and heading structure
6. 6Tighten copy so it's clear and easy to parse

Readiness is step one, not the finish line

A ready site is one AI can read. Whether AI actually recommends it depends on your content, how well it answers the questions people ask, and how often other trusted sites mention you. Once the score is in good shape, check whether AI tools name you today with the free AI visibility checker, and re-run this test after any change to robots.txt, your templates or your CDN settings.