What an AI crawler actually sees on your website

An AI crawler often receives less than your browser renders. Fetch your page's raw HTML and check whether three answers are present in plain text: who you serve, what you offer and what it costs. If any is missing, or buried beneath several paragraphs, assume an AI system cannot retrieve and quote it.

Buyers now ask ChatGPT and AI search tools those questions directly. In our customer conversations, marketers described buyers learning "everything they need to know" without visiting the company website.

This 30-minute audit compares your rendered page with its raw HTML, then scores five checks against your specific buyer.

An AI crawler may receive less than your browser shows

Your browser assembles a page from HTML, CSS, JavaScript and external resources. The finished page can look complete while the original HTML contains very little usable content.

AI systems do not all collect website information in the same way. Some fetch pages directly. Others use search indexes or retrieval partners. Their ability to run JavaScript also varies.

That makes raw HTML a useful crawler-side approximation. It shows what remains when browser rendering is removed from the equation.

The stakes are visible in our own market scan: T2D3 is cited in 4% of buyer queries, and 2 of 48 query cells. In the same scan, HubSpot picked up 20 mentions and Clay 15. Broad recognition, not page design, is doing most of that work.

Start with one commercially important page. Your homepage, pricing page or primary service page is usually enough for the first test.

Open three versions:

1. The rendered page: Load the page normally in your browser.
2. The original source: Add view-source: before the URL in Chrome.
3. The fetched HTML: Run curl -L https://example.com/page in a terminal.

Search each version for the same phrases. Try your industry, main offer, pricing language and one differentiator.

If those phrases appear in the browser but not the source or fetched HTML, JavaScript may be carrying too much of the message.

Audit the questions your buyer asks before auditing the technology

A generic crawler test can confirm that text exists. It cannot confirm that the right text exists.

Write down three questions tied to your ideal customer profile (ICP), meaning the buyer your go-to-market plan is built around. Use questions that affect a shortlist or purchase decision.

For a B2B SaaS company, one set might be:

• Which industries does this company serve?
• What problem does the product solve?
• How does pricing work?

For a fractional CMO practice, the questions could change:

• Does this approach support several B2B SaaS clients?
• Can it preserve each client's ICP, positioning and brand voice?
• Does it cover strategy and execution?

Now search the raw HTML for complete answers. A navigation label containing "Pricing" does not count. The buyer needs an answer, not evidence that another page might have one.

This is where ICP-aware grading matters. A technically crawlable page can still fail if it answers questions for the wrong buyer. Our market scan found 30 distinct vendors recommended by buyer LLMs in this category, so answering the wrong buyer's question is enough to leave you out of a named shortlist.

Answer-first pages give retrieval systems a clean passage to quote

Finding the right words is only half the test. Their shape matters too.

One structure that worked for us uses four parts:

1. A heading that states the buyer's question in natural language
2. A direct answer in the first one or two sentences
3. Evidence such as a mechanism, named example or customer result
4. Supporting detail for readers who need to evaluate the claim

Here is a reusable template:

How does [product or service] help [specific buyer] achieve [outcome]?

[Product or service] helps [buyer] achieve [outcome] by [specific mechanism]. It is designed for [relevant situation], including [two concrete examples].

[evidence, limitations and implementation detail]

The tradeoff is editorial. Answer-first writing removes some suspense. In return, the buyer and retrieval system can identify the page's claim without reconstructing it from five sections.

Do not hide the answer behind "The future of…" language. If the section explains pricing, use a heading such as "How our pricing works."

In our scan, four buyer queries returned no mention of us at all, including one that almost restates our purpose: how to run several fractional CMO clients without re-typing context into five AI tools. No page of ours answered that question in those words.

A five-point score turns the crawler comparison into an action list

Score the page with one point for each check it passes:

• Access: The URL returns a successful response and is not blocked by robots.txt or a noindex directive.
• Raw HTML parity: The primary offer and buyer language appear in the fetched HTML.
• Buyer-question coverage: The page answers all three ICP-specific questions.
• Answer-first structure: Each answer follows its heading within one or two sentences.
• Evidence: Important claims include a mechanism, named example or measurable result.

A score of five means the page passes this basic test. A score of three or four gives you a specific revision queue. Below three, fix access and raw HTML before rewriting supporting copy.

Run the same audit across five commercial pages. That creates a 25-point baseline you can repeat after changes.

What makes the score useful is that it grades the page the way a buyer reads it. Andrea Nicholas, a management consultant who uses the T2D3 approach with clients, described the appeal of a framework in plain terms: "It's soup to nuts. It's like plug and play." A five-point score does the same job for a page audit: you know what passed, what failed and what to fix next.

AI visibility belongs beside keyword rankings in the report

A Semrush ranking report no longer describes the whole discovery path. Customer conversations made the reporting gap clear: buyers can see a brand in an AI Overview without clicking its website.

Add four items to the monthly report:

• The three buyer questions tested
• Whether each answer exists in raw HTML
• Whether AI results mention or cite the company
• Which pages or outside sources support those answers

An AI mention does not guarantee a first-page ranking or a click. It is still evidence that the brand entered the buyer's research path.

AI-search readiness is one dimension of a buyer-scored website audit

In the T2D3 OS website audit, AI-search readiness is dimension nine of ten. We compare raw HTML with the rendered page, then test answer-first structure against the company's specific buyer.

That context prevents a common false positive: awarding a high score because a page is crawlable while ignoring whether it answers purchase questions.

Start this week with one page, three buyer questions and the five-point score. Fix the largest raw-versus-rendered gap first, then rewrite each commercial answer so its claim appears directly beneath the relevant heading.