robots.txt Generator

What a robots.txt file is and what it does

A robots.txt file is a plain text file that lives at the root of your domain and tells search engine crawlers which parts of your site they are allowed to request. When a well-behaved bot like Googlebot arrives, it checks this file first and follows the rules it finds there. Our free robots.txt generator builds a clean, valid file for you in seconds, so you can create a robots.txt without memorizing the syntax or risking a typo that takes pages offline.

It is important to understand what robots.txt does not do. It controls crawling, not indexing. Blocking a URL with Disallow stops compliant crawlers from fetching the page, but Google can still index that URL and show it in search results without a snippet if other pages link to it. If your goal is to keep a page out of the index entirely, the robots.txt file is the wrong tool. Use a noindex directive instead, delivered through a meta robots tag or the X-Robots-Tag HTTP header, and leave the page crawlable so the directive can actually be read. Our team covers this trade-off in depth as part of our SEO and organic growth services.

The syntax: User-agent, Allow, Disallow, and Sitemap

A robots.txt file is made of one or more groups of rules. Each group starts with a User-agent line that names the crawler the rules apply to, then lists the paths to block or permit. The four directives you will use most are:

• User-agent — names the bot the rules target. An asterisk (*) matches all crawlers, though AdsBot crawlers must be named explicitly.
• Disallow — specifies a path that the matched crawler must not request.
• Allow — permits a path, typically to override a broader Disallow for a specific subfolder or file.
• Sitemap — points crawlers to the full URL of your XML sitemap. This is supported by Google, Bing, and other major engines.

Common rules you can generate

Most sites only need a handful of patterns. Here are the ones our generator produces:

• Allow all crawlers — a User-agent: * line with an empty Disallow: permits full access.
• Block a folder — Disallow: /admin/ stops crawlers from requesting anything under that directory.
• Block a specific bot — name it on the User-agent line, for example User-agent: AhrefsBot followed by Disallow: /.
• Add your sitemap — include Sitemap: https://www.yoursite.com/sitemap.xml as an absolute URL.

Where the robots.txt file must live

The file must sit at the root of your domain and be reachable at https://www.yoursite.com/robots.txt. Crawlers only look in that one location, so a file placed in a subfolder or named anything other than robots.txt in lowercase will be ignored. Each subdomain needs its own file, and the rules apply only to the protocol, host, and port where the file is served.

How to test your robots.txt

Before you rely on a new file, confirm it behaves the way you expect. Open the file directly in a browser to verify it loads and reads correctly. Then use the robots.txt report and the URL Inspection tool inside Google Search Console to check whether a specific URL is allowed or blocked for Googlebot. Testing catches problems before they cost you traffic, and it takes only a minute.

Common mistakes to avoid

A small slip in a robots.txt file can have an outsized impact. Watch for these:

• Blocking your whole site — a stray Disallow: / under User-agent: * hides every page from search engines. This often survives from a staging environment by accident.
• Blocking CSS and JavaScript — if Google cannot fetch your stylesheets and scripts, it cannot render pages properly, which can hurt rankings. Keep those assets crawlable.
• Expecting it to remove pages from the index — Disallow does not equal noindex. Use a meta robots noindex tag for that.
• Wrong location or filename — anything other than a lowercase robots.txt at the domain root is invisible to crawlers.

If you manage a multilingual or multi-region site, your crawl directives work alongside other technical signals. Pair this tool with our hreflang generator to keep international targeting clean.

Frequently asked questions

Does robots.txt keep a page out of Google? No. It stops crawling, but a blocked URL can still appear in results without a snippet. Use a noindex directive to remove a page from the index.

Do I even need a robots.txt file? Not strictly. If you want every page crawled, you can skip it. But a file is useful for pointing crawlers to your sitemap and managing crawl access to low-value sections.

Is robots.txt case sensitive? The filename must be lowercase, and the paths inside are case sensitive, so /Folder/ and /folder/ are treated as different.

Will every bot obey it? Reputable crawlers respect robots.txt, but malicious bots and scrapers can ignore it. It is not a security mechanism, so protect sensitive content with authentication.

For the full specification, see Google Search Central's robots.txt documentation. Want a technical SEO audit that goes beyond the file itself? Get in touch with our team and we will review your crawl strategy end to end.