How AI agents actually read your website

Imagine hiring a brilliant assistant who is blind to visual design, never executes JavaScript unless forced to, reads at a million words a minute, and abandons any task that gets confusing. That's a reasonable model of the AI agents that increasingly stand between your business and its customers. Here's what a typical agent visit looks like.

Step 1: the machine files

Before touching your homepage, a well-behaved agent checks robots.txt (am I allowed here?), sitemap.xml (what exists?), and increasingly llms.txt (what does the owner want me to know?). Missing files aren't fatal, but they remove the guardrails that keep the agent on your intended path.

Step 2: the raw HTML

The agent fetches your page and reads the source — not the rendered result. If your content only appears after a JavaScript framework boots, the agent sees a nearly empty shell. This is the single most common catastrophic failure we find: beautiful sites that are blank documents to a machine.

Step 3: building a model of the page

From the HTML, the agent extracts structure: the title, the H1, the navigation landmarks, headings, links, forms. Semantic HTML is the difference between a labelled map and a pile of rectangles. One clear H1 tells the agent what the page is; six competing H1s tell it nothing.

Step 4: the trust check

Who operates this site? The agent looks for Organization schema, visible contact details, policy pages, and HTTPS. Sites that fail the trust check can still be read — but they won't be cited, recommended, or transacted with.

Step 5: the action decision

If the user's goal is a task — book, buy, enquire — the agent hunts for the path: a form with labelled fields, a clear primary CTA, an unambiguous next step. Unlabelled inputs and five competing buttons read as noise, and noisy sites lose the task to cleaner competitors.

The seven give-up points

• Blocked by an over-aggressive WAF or bot wall
• Empty HTML that requires JavaScript to render
• No parseable title or headline
• No structured data to verify the business
• No visible contact or trust signals
• Forms without labels
• No clear next action

Every one of these is checkable, and every one is fixable. That's precisely what our readiness scan measures — the same walk-through, automated, with a fix plan at the end.