Free tool
robots.txt generator
Decide which crawlers may read your site: search engines, AI training crawlers, AI search and assistants that fetch pages on a user's behalf. The file is built in your browser — nothing is sent to us.
In short
robots.txt is a request, not a lock. The large search engines and AI providers publish the names of their crawlers and state that they follow the file; scrapers and spam bots do not have to. Use it to tell well-behaved crawlers what they may read, and check everything else on the server.
Starting point
Which crawlers should stay out?
Ticked crawlers get Disallow: /. All others follow the rules for all crawlers, including the paths below.
Each line starts with /, for example /admin/ or /cart/. An * stands for any characters.
The full address of your XML sitemap, starting with https://.
How to put the file live
- 1Choose the crawlers and paths above and copy the result.
- 2Save it as a plain text file named robots.txt (UTF-8).
- 3Upload it to the root of your domain so that it opens at https://your-domain/robots.txt. Every subdomain needs its own file.
- 4Open that address in the browser and check that exactly this text appears. If your host or CDN adds rules of its own, you will see them there too.
What robots.txt cannot do
- It protects nothing. Anyone can still open a blocked path, and the file is public: it even lists the paths you would rather hide.
- Bots that ignore it are not stopped by it — and form spam and automated sign-ups come from exactly those.
- A blocked page can still appear in search results if other sites link to it. To keep a page out of the index, allow crawling and use noindex.
Frequently asked questions
Does robots.txt block bots?
No. It is a request that well-behaved crawlers follow. Scrapers, spam bots and many automated tools ignore it. Stopping those takes a check on the server, for example rate limits or bot protection on your forms.
If I block GPTBot, does my site disappear from ChatGPT?
Not necessarily. OpenAI uses separate names: GPTBot for training, OAI-SearchBot for search in ChatGPT and ChatGPT-User when a user asks about a page. Blocking one leaves the others untouched. Anthropic and Perplexity also use separate names for these purposes.
Does blocking Google-Extended affect my Google ranking?
According to Google, no. Google-Extended only controls whether content crawled by Googlebot may be used for Gemini models. Crawling and ranking in Google Search stay the same.
Where does the file have to be?
In the root of the host, at /robots.txt — for example https://example.com/robots.txt. A file in a subfolder is ignored, and every subdomain needs its own.
What is the Content-Signal line?
A proposal published at contentsignals.org for stating how content may be used once it has been crawled: for search results, as input for AI answers or for AI training. It expresses your wishes; whether a crawler respects them is up to its operator.
Is my input sent anywhere?
No. The generator runs entirely in your browser. Nothing you enter is transmitted or stored.
When bots ignore your robots.txt
SilentShield assesses submissions to your forms on the server by the behaviour that led to them — by default without a task for your visitors.