How to Get a Research-Supply Website Read and Cited by AI Search

How AI search finds and cites a page

Google AI Overviews, ChatGPT search, Perplexity, Gemini and Copilot all work in roughly the same order. A crawler fetches pages and stores them in an index, either the operator's own or a search engine's. When someone asks a question, the system retrieves passages that look relevant, writes an answer from them, and links some of the pages it drew on. A page that is never fetched, or fetched but unreadable, never reaches the last step.

Each system has its own crawler and its own index, and none of them publishes how it chooses which sources to cite. What they have in common is that they prefer passages that answer a question directly, come from a source they can identify, and agree with other sources. That is why the work below is mostly about access, clarity and evidence rather than about tricks.

Research supply adds one more filter. Assistants are cautious with anything that sounds like health advice, and a page that makes usage or outcome claims is both a compliance problem and a weak source. Catalog facts, documentation and plain business information are what a research-supply site can be cited for, and they are also what buyers are searching for.

Crawler access, bot by bot

robots.txt is read per user agent, and AI operators now run several agents each, with different jobs. Blocking one does not block the others, and allowing a training crawler is a separate decision from allowing a search crawler. Decide per bot, and write the rule down.

Names and purposes change, so check each operator's current documentation before you edit the file. As published at the time of writing:

• Googlebot. Crawls for Google Search, and AI Overviews draw on the same index. There is no separate AI Overviews crawler to allow.
• Google-Extended. Not a crawler but a robots.txt token. It controls whether content Google has already crawled may be used for Gemini models and grounding; it does not change Google Search ranking or inclusion.
• Bingbot. Crawls for Bing, and Copilot answers draw on the Bing index, so blocking Bingbot also removes the site from Copilot's search results.
• OAI-SearchBot and ChatGPT-User. OpenAI's search crawler and its agent for pages a user asks ChatGPT to open. GPTBot is the separate training crawler.
• PerplexityBot and Perplexity-User. Perplexity's index crawler and its user-triggered fetcher.
• ClaudeBot, Claude-SearchBot and Claude-User. Anthropic's training crawler, search crawler and user-triggered fetcher.
• Applebot-Extended. A token, like Google-Extended, that controls use of crawled content for Apple's models.

What a sensible robots.txt looks like

For a business that wants to be found, the simplest rule is to allow everything public and close only what should never be indexed: admin, account, cart and API paths. Name the AI crawlers explicitly as well, so that a later, narrower rule for all user agents does not shut them out by accident.

This site does exactly that. Its robots.txt is generated by the app: one rule for all user agents that allows the site and disallows /api/, and a second rule naming GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, Bingbot and several others with the same allow and disallow. Preview deployments disallow everything and send a noindex header, so a test address never competes with the real one.

• Keep the sitemap line in robots.txt, pointing at a sitemap that lists only canonical, indexable pages.
• Do not block the framework's script and style paths. Crawlers that do render pages need them.
• Check robots.txt after every deployment. A staging rule that reaches production can remove a site from every index at once.
• Remember that robots.txt is a request, not access control. Anything private needs a login, not a disallow line.
• If a firewall or bot-protection service sits in front of the site, check that it is not challenging the crawlers you have allowed. A bot that receives a challenge page reads nothing.

Rendering: what the crawler actually receives

Googlebot runs JavaScript, though rendering can happen later than the first fetch. Many AI crawlers are reported to read only the HTML the server returns. If product names, lot numbers or answers appear only after a script runs in the browser, those crawlers may see an empty shell.

The fix is to send the important text in the first response: server rendering or static generation, which frameworks such as Next.js do by default for pages that do not opt out. A quick check is to fetch a page with a command-line tool or view the page source, not the inspector, and search for a sentence you expect to be quoted. If it is not in the source, assume some crawlers cannot see it.

• Keep headings, answers, product facts and FAQ answers in the HTML, not inside tabs that load on click.
• Give report and COA data a text entry on the page. A certificate that exists only as an image or a scanned PDF tells a crawler very little.
• Avoid age gates or pop-ups that replace the page content for visitors without cookies, since crawlers arrive without cookies.
• Keep pages fast and stable. A slow or failing response is a page the crawler may skip; Core Web Vitals explained covers the page-speed side.

llms.txt: useful, not magic

llms.txt is a proposed convention (llmstxt.org): a plain-text file at the root of a site that summarises what the site is and links its most important pages, written for language models rather than for people. It is a proposal, not a standard, and the major search and AI operators have not committed to reading it. Treat it as a low-cost summary that may help some tools, never as a substitute for crawlable pages.

If you publish one, generate it from the same data as the pages, so it cannot drift. This site serves /llms.txt and a longer /llms-full.txt from routes that read the same service, product and guide data the pages use, including each guide's short answer and the research-use-only notice. When a guide changes, both files change with it.

• State what the business is, who it serves, where stock ships from and how pricing and contact work, and link the key pages.
• Never put anything in llms.txt that the visible site does not also say.

Entity consistency

An assistant has to decide what your business is before it can cite it. It builds that picture from your site, your structured data and whatever else on the web mentions you. When the business name, address, contact details and description differ from page to page, the picture blurs, and a blurred entity is a risky source.

Pick one canonical version of each fact and use it everywhere: the footer, the contact page, the Organization markup, the llms.txt file, invoices, marketplace profiles and social accounts. In research supply, also keep the description of what you sell identical: research materials, research use only, the same compound names. A site that calls the same product by a code on one page and a compound name on another makes the match harder; if codes exist, list them as synonyms on the product page, as this site's catalog does.

• One legal or trading name, spelled the same way everywhere.
• One support email and one phone or messaging number, shown as text, not only as images.
• An about page that names the operator and, where true, the parent company.
• Organization markup with the same name, URL, logo and contact details as the page.

Answer-first page structure

Retrieval works on passages, not whole pages. A paragraph that answers the question in its first two or three sentences, and still makes sense when lifted out on its own, is easier to quote than an answer spread across five paragraphs of introduction.

Write each important page around the question it answers. Put the question in the title or first heading, the answer directly under it, then the detail that earns the answer. Every guide on this site follows that shape: the short answer at the top is the passage most likely to be quoted, and it is marked as the speakable part of the page. Answer engine optimization applies the same idea to whole sites.

• One topic per page, with a heading that says what the page answers.
• The answer in the opening sentences, with the subject named, not "it" or "this".
• Specific nouns: the compound name, the report number, the city stock ships from.
• FAQ answers that stand alone, without "as mentioned above".

Sources, citations and markup that matches the page

An assistant is more comfortable quoting a claim it can check. For a research-supply site, the checkable facts are already there: the laboratory name, report number and date on a certificate of analysis, the lot number on the label, the publication record behind a summary of research. Put them on the page as text and link to the source.

This site's research pages show the pattern: publication statistics with the queries that produced them, and summaries in which every finding names the model studied and links the cited paper. The testing page lists published reports by product and size. Neither page makes a claim the source does not support, and neither tells anyone how to use a compound.

The opposite also holds. A blanket purity claim, an uncited statistic or a certification the business cannot document is a reason for a careful system to look elsewhere, and a reason for a payment provider or ad reviewer to look closer.

Schema.org markup in JSON-LD tells machines what a page is: an organization, a product, an article, a dataset, a set of questions and answers. It does not by itself earn a citation, and no operator promises that it does. What it does is remove ambiguity, which helps any system that reads it describe the page accurately.

The rule that matters most is that markup must describe what the visible page says. A product with no price on the page gets no price in the markup; an FAQ in the markup appears on the page word for word. Structured data for peptide websites covers each type a research-supply site needs, with examples.

Freshness and dates

Assistants answering questions about suppliers, availability or documentation tend to prefer current sources, and a page with no date gives them nothing to go on. Show an updated date on pages that change, put the same date in the Article markup as dateModified, and give the sitemap a matching lastModified.

Only change the date when the content changes. Bumping dates without edits is easy to spot, because the text is the same as last time, and it teaches crawlers that your dates mean nothing. On this site the guide date, the Article markup and the sitemap entry all come from one updated field.

• Review key pages on a schedule and record what changed.
• Add new lots and reports before the stock goes on sale, so documentation is never behind the catalog.
• Retire pages with a redirect to the nearest current page, not a dead end.

Measuring AI referrals

Most visits from AI tools arrive as referrals from the assistant's domain, such as chatgpt.com, perplexity.ai, gemini.google.com or copilot.microsoft.com, and some assistants add a utm_source tag to the links they show. Create a channel or segment in your analytics for those referrers so the traffic is counted on its own instead of disappearing into general referrals. Check the hostnames that actually arrive in your own reports, because they change.

Two limits are worth knowing. Clicks from Google AI Overviews are reported as ordinary Google search traffic, so they cannot be separated from other organic visits. And many people read an AI answer and never click, so referral counts understate how often a site is used as a source.

To see which sources assistants use, ask them the questions your buyers ask, on a fixed list and on a schedule, and record which sites they cite. Then judge the work by what matters to a wholesale business: inquiries. This site keeps the landing page, referrer and utm tags for the session in the browser and attaches them to an inquiry only when the visitor sends one, so an inquiry that started in an assistant can be counted as such. B2B lead generation for research suppliers covers measuring inquiries through to orders.

What not to do

• Fake reviews or ratings. Invented testimonials and review markup without real reviews break search guidelines and the compliance rules for this category. They also teach any careful system that the site's other claims may be invented too.
• Uncited or overstated claims. No blanket purity figures, invented statistics or undocumented certifications. If a fact has no source, leave it out.
• Usage or outcome language. No dosing, usage or results framing anywhere, including FAQs and markup. See research-use-only storefront copy.
• Cloaking. Showing crawlers different text from what visitors see, including hidden text written for AI. Search engines treat it as spam, and an assistant that quotes text a visitor cannot find has been misled.
• Prompt text aimed at AI. Hidden instructions telling an assistant what to say are a form of cloaking, and they put the whole site's credibility at risk.
• Mass-produced thin pages. Hundreds of near-identical pages add nothing to the index. Pages at scale work when each one states something true and specific to that page.
• Guarantees. Anyone promising a citation, a ranking or a placement in an AI answer is promising something no operator offers.

Checklist

Run through this list before launch and after any major change to templates, hosting or copy. We build research-supply sites to it as part of SEO and AI search and generative engine optimization; send your site address and what you sell from the website development page, and we will tell you what we would fix first.

• robots.txt allows the search crawlers you want, by name, and closes only private paths.
• No firewall rule challenges the crawlers you allowed.
• Key text is in the server HTML: headings, answers, product facts, report data, FAQs.
• A sitemap lists canonical, indexable pages with honest lastModified dates.
• llms.txt, if used, is generated from the same data as the pages.
• Business name, contact details and description match on every page and profile.
• Each important page answers its question in the opening sentences.
• Facts link to their sources: reports, lot numbers, cited papers.
• Structured data matches the visible page, with no invented prices, reviews or claims.
• Updated dates change only when the content does.
• Analytics has a channel for AI referrers, and inquiries carry their source.
• Copy is research-use-only throughout, with no usage or outcome claims.