Peptide companies cannot buy their way in front of customers, so being cited by AI assistants is not a novelty channel for this category, it is distribution. Which makes one file decisive: robots.txt now carries the switches that decide whether AI systems may read and use your content at all. Most operators have never looked at theirs, and a surprising number are blocking the very systems they hope will recommend them.
The controls, from the documentation
Every major AI system honors robots.txt user-agent tokens, and the subtlest one is Google's. Per Google's own crawler documentation, Google-Extended "doesn't have a separate HTTP request user agent string. Crawling is done with existing Google user agent strings; the robots.txt user-agent token is used in a control capacity", and the token "has no effect on Google Search". In plain terms: blocking Google-Extended does not stop Google crawling you, and does not touch your rankings. It controls whether that content may feed Google's AI training-related uses. OpenAI's GPTBot, Anthropic's ClaudeBot and the other named agents work as conventional crawlers with their own tokens.
Alongside robots.txt sits a newer, informal proposal: llms.txt, a published spec, now at version 2, for a file that gives language models a curated map of what a site is and where its authoritative content lives. It is a proposal rather than a standard the platforms have committed to, and honest positioning says so. But it costs a page of markdown, and serious publishers in adjacent categories already ship it: the largest cannabis SEO agency serves a 36-kilobyte one, which we verified this week while writing our comparison pages.
The failure mode we found on our own site
The reason this article exists is that we lived the failure. On 11 August 2026 we found our own production robots.txt carrying nine Disallow rules for GPTBot, ClaudeBot, Google-Extended, CCBot and others that we never wrote. A CDN-level managed feature was injecting them on top of our explicit Allow rules, producing a self-contradictory file, on the site of a company that sells AI-search visibility. The fix was one setting, and the lesson generalizes: your robots.txt is what the edge serves, not what your repository says, and the only way to know is to fetch the live file and read it.
What a peptide site should actually do
- Fetch your live robots.txt today and read every user-agent block. If an AI agent is disallowed and you do not know why, someone or something decided your policy for you.
- Decide the policy deliberately. For a business that depends on being found and cited, allowing the major AI agents is the coherent default; blocking them is a decision to be invisible in answers.
- Remember what Google-Extended does and does not control, per the documentation quoted above, before anyone tells you blocking it protects your rankings. It does not touch Search.
- Ship an llms.txt if you have authoritative content worth mapping, and describe it honestly as the informal proposal it is.
- Give the crawlers something worth citing. Policy grants access; only sourced, dated, verifiable content earns the citation. That is the other nine articles in this library.
The strategic point is the same one that runs through this whole library: in a category with no paid channel, answer-engine visibility is not garnish. The technical half costs an afternoon. The content half is the actual moat, and it is the half most of the category still refuses to build.
Frequently asked questions
Does blocking Google-Extended hurt Google rankings?
No, per Google's own documentation: the Google-Extended token has no effect on Google Search. It also has no separate crawler; existing Google user agents do the crawling and the token works in a control capacity over AI training-related use of the content. Blocking or allowing it is a policy choice about AI use, not an SEO lever.
Should a peptide company allow AI crawlers?
For a category barred from paid advertising, being citable by AI assistants is one of the few distribution channels available, so allowing the major agents is the coherent default. The decision should be deliberate either way, and the live robots.txt should be checked rather than assumed, since CDN-level features can inject rules silently.
What is llms.txt and is it a standard?
It is a published proposal, currently at version 2 at llmstxt.org, for a file that gives language models a curated map of a site's authoritative content. It is not a standard the platforms have committed to honoring, and honest positioning says so. It is cheap to ship and serious publishers in adjacent categories already do.
How do we know if something is blocking AI crawlers on our site?
Fetch the live file at yourdomain.com/robots.txt and read every user-agent block, because what the edge serves can differ from what your repository contains. We found nine injected Disallow rules on our own site this month, added by a CDN feature rather than by anyone on the team, and the only way to catch that is to read the served file.
