How it works
An AI crawler is a bot. It requests pages from your site much like a browser does, reads the HTML, and sends what it finds back to the company that runs it.
Different crawlers do different jobs:
- Training crawlers collect pages to help train future models. GPTBot and ClaudeBot mostly work this way.
- Live retrieval fetchers grab a page in real time when a user asks a question. OAI-SearchBot, ChatGPT-User, and PerplexityBot fall into this group.
- Index builders store a copy of your page so an answer engine can pull from it later without visiting again.
Each crawler identifies itself with a user agent string, such as GPTBot or ClaudeBot. Your robots.txt file, which sits at yourdomain.com/robots.txt, is where you tell each one what it may read.
Most AI crawlers handle JavaScript poorly, or not at all. If your treatment prices, FAQs, or service descriptions only appear after a script loads, the crawler may see an empty page. Text that sits in the HTML is text an AI crawler can actually use.
Blocking is also not the same as removing. If you block a crawler today, content it already collected may still exist inside a model. And blocking one company's crawler has no effect on the others. Each one is a separate rule.
Why it matters for aesthetic clinics
Patients now ask ChatGPT, Perplexity, and Google's AI Overviews things like "best place for lip filler near me" or "how much does CoolSculpting cost." The answer they get is assembled from pages an AI crawler was allowed to read. If your site is closed to those crawlers, or your content is invisible to them, you are not in the running.
This matters more for clinics than for most local businesses. Aesthetic treatments are high-consideration and high-price. Patients research for weeks before they book. That research increasingly starts inside an answer engine rather than a results page, and a growing share of those sessions end without a click to any website at all. If the AI does not name you, the patient never learns you exist.
There is a real tension here, and clinic owners feel it. Your before-and-after galleries, your pricing, and your treatment write-ups are assets you paid for. Letting AI companies read them for free can feel wrong. But the practical trade is simple. Blocking the crawlers does not stop your competitors from being cited. It only guarantees that you are not.
AI crawler vs search crawler
| AI crawler | Search crawler | |
|---|---|---|
| Purpose | Feed model training or answer generation | Build an index of ranked links |
| Examples | GPTBot, ClaudeBot, PerplexityBot | Googlebot, Bingbot |
| What you get back | A mention or citation inside an answer | A link on a results page |
| JavaScript | Usually limited or none | Googlebot renders most JavaScript |
| Controlled by | robots.txt, per user agent | robots.txt, meta robots, Search Console |
One overlap is worth knowing. Google uses Googlebot for both regular search and AI Overviews. The separate Google-Extended control affects Gemini and model training, not whether you appear in AI Overviews. So blocking Google-Extended will not remove you from AI Overviews, and blocking Googlebot removes you from everything.
The Ownerized take
We treat AI crawlers as a distribution channel, not a threat. Our first step is checking what each crawler can actually reach on your site, then fixing the pages that decide bookings: treatments, pricing, locations, and the questions patients really ask. Open access is worth little if the content behind it is thin or buried in JavaScript, because access plus substance is what gets a clinic cited. Crawler access is one of the nine surfaces we check in the AI Growth System.
Common mistakes
- Copying a robots.txt from another site without reading it. Plenty of templates block GPTBot by default.
- Blocking AI crawlers to protect content, then wondering why competitors get named in ChatGPT and you do not.
- Assuming
Google-Extendedcontrols AI Overviews. It does not. - Putting prices, FAQs, and service details behind JavaScript or inside images. Crawlers read text.
- Blocking a crawler and expecting content already in a model to disappear. It will not.
- Placing robots.txt in a subfolder. It only works at the root of each domain.
- Setting the rules once and never revisiting them. New crawlers appear regularly.
Frequently asked questions
Should my clinic block AI crawlers?
For most clinics, no. Blocking stops AI answer engines from citing your treatment pages, but it does nothing to stop competitors from being cited instead. The realistic choice is not privacy versus exposure. It is whether patients hear your clinic named or someone else's when they ask about your treatments.
How do I know which AI crawlers are reading my site?
Check your server logs or CDN analytics for user agent strings like GPTBot, ClaudeBot, PerplexityBot, and OAI-SearchBot. Cloudflare and similar providers show AI bot traffic in a dashboard. Then open yourdomain.com/robots.txt in a browser to see which crawlers you currently allow.
Will blocking GPTBot remove my clinic from ChatGPT?
No. Blocking GPTBot stops future collection, but content already used in training stays inside the model. It also does not block ChatGPT's live browsing agents, which use different user agent names. Removing a clinic from a model that has already learned it is not something robots.txt can do.
Is robots.txt legally binding?
No. Robots.txt is a voluntary convention, not a lock. Major AI companies including OpenAI, Anthropic, and Google publish their crawler names and honour it. Less reputable scrapers ignore it entirely. If you need enforcement rather than a polite request, that is a job for your CDN or firewall rules.
Do AI crawlers read our before-and-after photos?
Crawlers read text far more reliably than images. An AI answer engine is unlikely to interpret a gallery photo, but it will read the alt text, captions, and treatment description around it. Write those as real sentences, because that is the part a crawler can quote.