Home / Free tools / Technical SEO / Robots.txt Generator

Robots.txt generator.

Pick a preset, add your rules, test any URL and download a valid robots.txt. It warns you before you block your whole site by mistake.

Use the tool ↓Free website audit
Technical SEO

About this tool.

A robots.txt file sits at the root of your website and tells search engine crawlers which parts they should not crawl, such as admin areas, cart pages or endless filter URLs. It also points crawlers to your sitemap. One wrong line can hide your whole site from Google, so this tool checks your rules as you build them and lets you test real URLs before you upload anything.

How to use it
  1. Choose a starting preset (allow all, WordPress, Next.js, online shop) or paste your existing file to edit it.
  2. Add or change rules for each crawler, tick any AI crawlers you want to block, and add your sitemap address.
  3. Read the checks, test a few real URLs, then copy the file or download robots.txt and upload it to your site's root folder.
Start from:
Crawler group 1
Block AI crawlers (optional)
Edit an existing robots.txt
Your robots.txt
User-agent: *
Allow: /

2 lines. Upload it to your site's root folder so it opens at yourdomain.com/robots.txt.

Checks on your rules
  • No problems found in your rules.
  • No sitemap line yet. Adding one helps crawlers find all your pages.
  • robots.txt is public. Anyone can read it, so do not use it to hide private pages.
Test a URL before you upload
Try:

Type a page address or path to see whether the crawler may visit it.

Good to know: robots.txt controls crawling, not indexing. A blocked page can still appear in Google if other sites link to it. To keep a page out of search, use a noindex tag on a page Google is allowed to open, or put it behind a login. The file is public too, so never list secret URLs in it.

Questions

Good to know.

Where do I put the robots.txt file?+
In the root folder of your domain, so it opens at https://yourdomain.com/robots.txt. A subdomain such as shop.yourdomain.com needs its own file. A file placed in a sub-folder is ignored.
Does robots.txt remove a page from Google?+
No. It only asks crawlers not to visit a page. If the page is already indexed or linked from elsewhere it can still show in results, usually without a description. Use a noindex tag or a password to keep a page out of search.
Should I block AI crawlers?+
That is your choice. Crawlers such as GPTBot, ClaudeBot and CCBot are used to collect content for AI products, and well-behaved ones follow robots.txt. Blocking them does not change your Google Search ranking. Google-Extended is a separate control for how Google's AI products use your content, not for Search.
Does Google follow Crawl-delay?+
No. Google ignores the Crawl-delay line. Bing and some other crawlers respect it. If your server struggles with crawl traffic, fix it at the server or hosting level instead.
How accurate is the URL tester?+
It applies the same matching rules Google documents: the most specific (longest) matching rule wins, and Allow beats Disallow when they are equally specific. Treat it as a strong guide, and confirm important URLs with the robots.txt report in Google Search Console after you upload.
More free tools

Try these next.

Technical SEO

XML Sitemap Generator

Turn a list of URLs into a valid sitemap.xml, ready for Search Console.

Open tool
Technical SEO

Page Structure Finder

Paste a page's HTML and see its heading outline, with checks for a missing h1 and skipped levels.

Open tool
Technical SEO

RSS Feed Validator

Paste an RSS 2.0 or Atom feed to check required fields, dates and guids, and preview the first items.

Open tool

See all free tools →

Got a website in mind? Let's build it.

Send your idea or your current site. We reply within one working day with next steps and a fixed quote.

Start a project →Book a free callChat on WhatsApp
WhatsApp us
Get a fixed quote