New customer offer−20 %20% off the first 3 months when paid monthlyView plans
SilentShield

Free tool

robots.txt generator

Decide which crawlers may read your site: search engines, AI training crawlers, AI search and assistants that fetch pages on a user's behalf. The file is built in your browser — nothing is sent to us.

In short

robots.txt is a request, not a lock. The large search engines and AI providers publish the names of their crawlers and state that they follow the file; scrapers and spam bots do not have to. Use it to tell well-behaved crawlers what they may read, and check everything else on the server.

Starting point

Which crawlers should stay out?

Ticked crawlers get Disallow: /. All others follow the rules for all crawlers, including the paths below.

Search engines

Index your pages for classic search results. Blocking them removes your site from their search.

AI training and data collection

Collect content for training AI models. Google-Extended and Applebot-Extended are not separate crawlers: they only control whether Google and Apple may use crawled content for their AI models. Search is not affected.

AI search

Index pages so that AI search can show them and cite them as a source.

Assistants acting for a user

Fetch a page when someone asks an assistant about it. Some providers state that robots.txt rules may not apply to these requests.

Each line starts with /, for example /admin/ or /cart/. An * stands for any characters.

The full address of your XML sitemap, starting with https://.

Content signals (optional)

A Content-Signal line (contentsignals.org) states how content may be used after crawling. Not every crawler reads it.

How to put the file live

  1. 1Choose the crawlers and paths above and copy the result.
  2. 2Save it as a plain text file named robots.txt (UTF-8).
  3. 3Upload it to the root of your domain so that it opens at https://your-domain/robots.txt. Every subdomain needs its own file.
  4. 4Open that address in the browser and check that exactly this text appears. If your host or CDN adds rules of its own, you will see them there too.

What robots.txt cannot do

  • It protects nothing. Anyone can still open a blocked path, and the file is public: it even lists the paths you would rather hide.
  • Bots that ignore it are not stopped by it — and form spam and automated sign-ups come from exactly those.
  • A blocked page can still appear in search results if other sites link to it. To keep a page out of the index, allow crawling and use noindex.

Frequently asked questions

Does robots.txt block bots?

No. It is a request that well-behaved crawlers follow. Scrapers, spam bots and many automated tools ignore it. Stopping those takes a check on the server, for example rate limits or bot protection on your forms.

If I block GPTBot, does my site disappear from ChatGPT?

Not necessarily. OpenAI uses separate names: GPTBot for training, OAI-SearchBot for search in ChatGPT and ChatGPT-User when a user asks about a page. Blocking one leaves the others untouched. Anthropic and Perplexity also use separate names for these purposes.

Does blocking Google-Extended affect my Google ranking?

According to Google, no. Google-Extended only controls whether content crawled by Googlebot may be used for Gemini models. Crawling and ranking in Google Search stay the same.

Where does the file have to be?

In the root of the host, at /robots.txt — for example https://example.com/robots.txt. A file in a subfolder is ignored, and every subdomain needs its own.

What is the Content-Signal line?

A proposal published at contentsignals.org for stating how content may be used once it has been crawled: for search results, as input for AI answers or for AI training. It expresses your wishes; whether a crawler respects them is up to its operator.

Is my input sent anywhere?

No. The generator runs entirely in your browser. Nothing you enter is transmitted or stored.

When bots ignore your robots.txt

SilentShield assesses submissions to your forms on the server by the behaviour that led to them — by default without a task for your visitors.