Schema and technical access for AI search

What does a technically readable page look like?

You are reading one. This page is built the way we build every page on attensira.com, and each choice exists because an AI engine reads differently from a person.

• The text is in the HTML the server sends. Fetch this URL with a plain HTTP client and the answer at the top, every heading and every paragraph is in the response. Nothing on it needs JavaScript to appear.
• It has a Markdown twin. Every page on the site is also served as plain Markdown, and the site lists its pages for agents in an llms.txt file.
• It carries structured data that matches what you see. The page declares itself an article in a guide, with its breadcrumb trail, its author and dates, and the questions and answers in the FAQ below, in JSON-LD. Every claim in the source list at the bottom is also marked up, and each one is visible on the page.
• It answers first. The 40 to 60 word answer sits directly under the title, before anything else.
• Its robots.txt names each AI agent and allows the search and user agents of every engine we track, along with most training agents. It disallows three crawlers outright: CCBot, AhrefsBot and SemrushBot.

What this produces is reach. In the 30 days to 4 October 2026, PerplexityBot fetched 331 different pages on our site and OpenAI's agents 419, counted at the edge by user agent. Access is not our bottleneck.

What it does not produce on its own is citations. As the earlier chapters show, our category answers lean on other people's pages. Technical access is the entry ticket. This chapter is about getting the ticket right, and about not paying extra for things that do not help.

Which technical work matters most?

In order of how often it is the problem.

1. Robots.txt that lets the right agents in. The search agents (OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot, bingbot) and the user agents (ChatGPT-User, Claude-User, Perplexity-User) need access to the pages you want cited. A single wildcard disallow written to stop training crawlers blocks all of them. The crawler guide's robots.txt chapter has the syntax.

2. Content in the initial HTML. AI crawlers do not run JavaScript; that is why they never show up in browser analytics. If the paragraph that answers a question is fetched and inserted by a script after the page loads, an agent reading the HTML may never see it. Server-render or pre-render the text that matters. Attensira's own page reader falls back to a plain HTTP fetch when needed, and its documentation warns that a page built entirely in JavaScript may then come back empty.

3. Real status codes and short redirects. A missing page should return 404, not a "not found" message with a 200. An old URL should reach its new home in one hop. A crawler that hits a chain of redirects or a soft error wastes its visit and may index the wrong thing.

4. Snippets allowed. Google says a page must be eligible to show in Search with a snippet to be a supporting link in AI Overviews or AI Mode, and lists nosnippet, data-nosnippet, max-snippet and noindex as the controls that limit what is shown. A blanket nosnippet on a template, often left over from an old legal or paywall decision, removes every page using it from that pool.

5. Internal links and a sitemap. Google recommends making content findable through internal links. Pages nobody links to are found late or not at all.

6. Reliability under load. A page that times out for a crawler is not read, and a robots.txt that errors can be treated by a compliant crawler as a full disallow until it recovers. Keep both fast and boring.

Does schema markup help a page get cited?

It helps engines understand a page. It is not a citation switch.

Google is explicit for its own AI features: there is no special schema.org structured data you need to add, and you do not need new machine-readable files, AI text files or markup to appear. Structured data still powers various Google search features, and Google asks that it match the visible text. None of the other operators' crawler documentation we reviewed describes schema as a factor in citation.

So treat schema as hygiene, with three rules.

1. Mark up what the page actually is. An article is an Article, a product page a Product or SoftwareApplication, a question-and-answer block a FAQPage, a trail of parent pages a BreadcrumbList.
2. Match the visible text exactly. If the FAQ markup lists a question, the question is on the page. If the product markup states a price, the page shows that price. Markup that says more than the page is the fastest way to lose trust in the markup.
3. Do not expect it to fix a weak page. An engine that has a vague paragraph and perfect markup still has nothing to quote.

On our own pages, the FAQ block, breadcrumbs and article data are generated from the same content as the visible text, from one source, so they cannot drift apart. That is the safest way to do it at any scale.

Which schema types are worth adding?

The ones that describe what the page already is. For most sites, five types cover nearly every page.

• Any page below the home page

  Type

  BreadcrumbList

  What it states

  Where the page sits in the site

• Article, guide or post

  Type

  Article

  What it states

  Headline, author, publish and update dates

• Questions and answers shown on the page

  Type

  FAQPage

  What it states

  Each question and its visible answer

• Product or pricing page

  Type

  Product or SoftwareApplication

  What it states

  Name, category, and the offer shown on the page

• Home or about page

  Type

  Organization

  What it states

  Name, logo, official profiles

Two habits matter more than the type list. Generate markup from the same data as the visible text, so the two cannot disagree. And update the dates in Article markup only when the content changes; a "modified" date that moves every night tells engines nothing.

What about llms.txt and Markdown versions of pages?

Cheap, harmless, and not documented as a ranking or citation factor by Google. llms.txt is a proposal for a Markdown file at a site's root that summarises the site for agents. Google says its AI features need no AI text files.

We publish one anyway, along with Markdown twins of our pages, because it costs little and some agents look for it. Perplexity's answers to "what is Attensira" in early October 2026 cited our llms-full.txt file directly, so at least one engine reads it. Whether it changed what Perplexity said about us, we cannot tell from the answers alone. The crawler guide's llms.txt chapter covers what to put in it.

Should you block crawlers by IP or rate-limit them?

Only with care. Two documented pitfalls:

• IP blocks can break the opt-out you meant. Anthropic warns that blocking its IP addresses may stop it reading your robots.txt, so the block may not work as a reliable opt-out. robots.txt per agent is the documented control.
• Crawl-delay is honoured unevenly. Anthropic says its bots honour the Crawl-delay extension. Amazon says Amazonbot does not support it. If load is the problem, a server-side rate limit per agent is more predictable than a directive some agents ignore.

If one agent is genuinely hammering your server, and on our site Meta's training agent made about nine in ten AI crawler requests in a month, decide by job. Blocking a training agent costs you nothing in today's answers. Rate-limiting a search agent can.

How do you audit a page in ten minutes?

1. Robots: read your robots.txt for each search and user agent. Are the page's path and its parent directory allowed?
2. HTML: fetch the page with curl or any plain HTTP client. Is the answer paragraph in the response?
3. Status: does the URL return 200 directly, without a redirect chain?
4. Snippet controls: is there a nosnippet, max-snippet or noindex in the head or on the paragraph that matters?
5. Canonical: does the page's canonical point at itself, not at a different URL?
6. Markup: does the structured data validate, and does every value in it appear on the page?
7. Links: is the page linked from at least a few related pages on your site?
8. Logs: in your server or edge logs, has any search agent fetched it in the last month?

The page token inspector shows what text a crawler receives from a URL, and the crawler URL inspector shows which agents' rules allow it.

What should you not spend time on?

A short list, expanded in the what does not work chapter:

• Special "AI schema" or custom markup types. No engine documents them.
• Separate pages written only for bots. Content that differs from what people see is a risk with every search engine and has no documented benefit.
• Chasing crawler volume. More requests from a training agent is not more visibility.

Once a page passes the ten-minute audit, technical work stops being the constraint. What decides the answer from there is what the page says and who else says it.