Robots.txt Generator

A robots.txt file tells search engine crawlers and AI bots which parts of your site they may visit. One wrong line can hide your whole site from Google, so this generator builds a safe file for you: start from a WordPress preset, add your sitemap, block private paths and choose which AI crawlers to allow.

What robots.txt is

robots.txt is a plain text file that lives at the root of your domain, for example example.com/robots.txt. Well behaved crawlers read it before they crawl your site. It contains simple rules: which crawler the rule is for, and which paths it should or should not visit.

Two important facts:

• robots.txt controls crawling, not indexing. A blocked page can still appear in Google if other sites link to it, usually without a description. To keep a page out of results, use a noindex tag instead and let Google crawl it to see that tag.
• It is a request, not a lock. Good bots follow it. Bad bots ignore it. Never use robots.txt to hide private data; protect it with a password instead.

How to use the generator

1. Choose a starting point. WordPress site is right for most people. Allow everything is for simple static sites. Block everything is only for staging or private test sites.

2. Add your sitemap URL. This helps every search engine find your pages. On WordPress it is often /wp-sitemap.xml, or /sitemap_index.xml with Yoast or Rank Math.

3. Add extra paths to block, one per line, such as /cart/ or /checkout/ on a store.

4. Tick any AI crawlers you want to block from your content.

5. Copy or download the file and upload it to the root of your site.

The generator runs in your browser and warns you if you choose a setting that could hide your site from search.

Reading the rules

The WordPress preset explained

The recommended WordPress file is short on purpose:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Sitemap: https://example.com/sitemap_index.xml

It keeps crawlers out of the admin area, allows the AJAX file that front-end features need, and points to your sitemap. Old advice to block /wp-includes/, /wp-content/ or /wp-content/plugins/ is outdated. Google needs to load your CSS and JavaScript to see your pages properly. Blocking them can hurt how your site is understood and ranked.

What you should and should not block

Good candidates to block

• Cart, checkout and account pages on a store, which are personal and useless in search.
• Internal search results like /?s= on large sites, which can create endless thin pages.
• Admin and login areas.
• Staging copies of your site, blocked completely.

Never block

• Your CSS, JavaScript and image folders.
• Pages you want in search results.
• Pages you want removed from search. Use noindex for those, so Google can crawl them and see the tag.
• Your sitemap.

AI crawlers: allow or block?

Many AI companies now crawl the web, either to train models or to answer questions live. robots.txt lets you decide, per crawler, whether they may read your content. The main ones in the generator:

There is a trade-off. Blocking AI crawlers keeps your content out of some AI training, but it can also mean AI assistants mention your site less often when people ask questions. For tool sites, blogs and businesses that want to be found, allowing them is usually better. For original creative work, photography or paid content, blocking training crawlers can make sense. You can also allow live answer bots like ChatGPT-User while blocking training bots like GPTBot.

How to upload robots.txt

WordPress with Yoast or Rank Math

Both plugins have a file editor for robots.txt in their tools section. Paste the generated text there and save.

WordPress without a plugin

WordPress shows a virtual robots.txt by default. To use your own, upload a real robots.txt file to the main folder of your site (the one with wp-config.php) using your host’s file manager. A real file replaces the virtual one.

Any other site

Upload the downloaded file to the root folder so it opens at yourdomain.com/robots.txt. It must be named exactly robots.txt in lowercase.

Test after uploading

1. Open yourdomain.com/robots.txt in a browser and check that the text matches.

2. In Google Search Console, open the robots.txt report to see the version Google fetched and any errors.

3. Use the URL Inspection tool on an important page to confirm it is not blocked.

4. Check that your sitemap line works with the Sitemap Checker.

Example files for common situations

A normal blog or business site

Use the WordPress preset with your sitemap line. Nothing else is needed. Most sites do not benefit from long robots.txt files; every extra rule is one more chance to block something by mistake.

An online store

Start from the WordPress preset and add /cart/, /checkout/ and /my-account/. These pages are different for every visitor and should never appear in search. Product filter URLs with many parameters can also be blocked on very large stores, but test carefully so product pages stay crawlable.

A staging or development copy

Use Block everything. Better still, protect staging with a password so nobody, including bots that ignore robots.txt, can reach it. Remember to replace the file before the site goes live.

A site that wants AI visibility

Allow everything for all crawlers, add your sitemap, and publish an llms.txt file with the llms.txt Generator so AI tools get a clear summary of your important pages.

How crawlers read the file

Each crawler looks for the group of rules that names it most specifically. If there is a group for GPTBot, GPTBot follows that group and ignores the general User-agent: * group. That is why the generator puts AI crawler rules in their own groups. Within a group, the most specific matching path wins, so Allow: /wp-admin/admin-ajax.php beats Disallow: /wp-admin/ for that one file. Paths are case sensitive, and a rule that starts with a slash matches everything that begins with that path.

The mistake that hides whole websites

The most damaging robots.txt error is a leftover Disallow: / under User-agent: *. It often comes from a staging site that was copied to the live domain, or from WordPress’s “Discourage search engines” setting being left on after launch. Traffic then drops to almost nothing within weeks. If your site suddenly disappears from Google, check robots.txt first, and check Settings, Reading in WordPress.

robots.txt vs noindex vs password

Choosing the right tool for each goal avoids the classic mix-up where a page is blocked in robots.txt and also has noindex, so Google never sees the noindex and keeps the page listed.

Frequently asked questions