Robots.txt: The Complete Guide for 2026 (Syntax, Examples, and AI Crawlers)

A robots.txt file is a plain text file that tells web crawlers which parts of your site they can and cannot access. It sits at the root of your domain, it uses a handful of simple rules, and it has quietly shaped how search engines read the web since 1994.

Most people meet robots.txt for one of two reasons. Maybe a tool flagged a page as "blocked by robots.txt" and they want to know why. Or they read that AI crawlers like GPTBot and ClaudeBot are scraping their content, and they want to push back. This guide covers both, plus everything in between.

I have kept the theory short and the examples plentiful. You can copy most of the code blocks straight into your own file.

What is robots.txt?

Robots.txt is a text file that lives in your site's root directory. Its job is to give instructions to bots before they crawl your pages. When a well behaved crawler arrives at a site, it looks for the file at the root address first, reads the rules, and decides where it is allowed to go.

The technical name for this system is the Robots Exclusion Protocol, sometimes shortened to REP. The file itself is the most common way that protocol gets used, which is why "robots.txt" and "Robots Exclusion Protocol" often get treated as the same thing.

Here is the part that trips people up. Robots.txt is advisory, not enforcement. It is closer to a posted sign than a locked door. Googlebot, Bingbot, and other reputable crawlers read the file and follow it. A malicious scraper can read the same file and ignore every line, or worse, treat your disallowed folders as a map of where the interesting stuff is. So robots.txt is a traffic manager, never a security tool.

What does robots.txt actually do?

It manages crawler traffic. That covers three main jobs:

• Keep crawlers away from pages that waste their time, like internal search results, faceted filter URLs, or admin areas.
• Reduce load on your server by steering bots away from heavy or duplicated sections.
• Point crawlers to your sitemap so they can find your important pages faster.

What it does not do is remove a page from Google. That distinction matters enough that it gets its own section further down.

Where robots.txt lives and how to find it

A robots.txt file only works from the root of a host. For the site www.example.com, the file must live at www.example.com/robots.txt. You cannot tuck it inside a subfolder and expect it to work.

Two rules follow from this:

1. The filename must be lowercase and exact. robots.txt works. Robots.TXT does not.
2. Each subdomain and each protocol needs its own file. Rules on example.com do not apply to blog.example.com, and rules on https:// do not carry over to a different port.

Want to see any site's file? Type the homepage address, add /robots.txt, and load it. Try https://www.google.com/robots.txt or https://www.cloudflare.com/robots.txt. This is also the fastest way to find your own file and confirm it is live.

How robots.txt works

A crawler that respects the protocol follows a simple sequence. It requests your robots.txt file. It scans for a group of rules that match its own name. It obeys the most relevant rule in that group. Then it crawls whatever is left open.

The file is built from groups. Each group starts with a User-agent line that names a bot, followed by the rules that apply to that bot. A crawler picks the single group that best matches its name and ignores the rest.

When two rules inside a group conflict, matching behaviour differs by engine. Google and Bing follow the most specific rule, measured by the number of characters in the path. Many other crawlers follow the first matching rule instead. If you write clean, non overlapping rules, you rarely need to think about this.

Robots.txt syntax and format

The syntax is small, which is both the good news and the bad news. There are only a few directives to learn, but a single misplaced slash can hide an entire website.

Save the file as UTF-8 encoded plain text. Put one directive per line. Blank lines separate groups, so do not drop them randomly inside a group.

Here are the directives you will actually use.

User-agent

This names the bot a group of rules applies to. An asterisk is a wildcard that means "every bot."

User-agent: *

To target one crawler, use its exact name:

User-agent: Googlebot

Disallow

The Disallow directive tells the named bot not to crawl a path. The path is everything after your domain.

User-agent: *
Disallow: /admin/

An empty Disallow value means nothing is blocked, so the bot may crawl the whole site.

Allow

The Allow directive carves out an exception inside a disallowed area. It is mainly honoured by Google and other modern crawlers, and it is handy for opening one file inside a blocked folder.

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Sitemap

The Sitemap directive points crawlers to your XML sitemap. It uses an absolute URL and it can sit anywhere in the file. Adding a sitemap to robots.txt is one of the easiest wins in technical SEO, and plenty of sites forget it.

Sitemap: https://www.example.com/sitemap.xml

Wildcards and pattern matching

Two special characters help you match groups of URLs. The asterisk matches any sequence of characters. The dollar sign anchors a rule to the end of a URL.

User-agent: *
Disallow: /*?# block any URL containing a question mark
Disallow: /*.pdf$# block anything ending in .pdf

Crawl-delay

Crawl-delay asks a bot to wait a set number of seconds between requests. It is unofficial. Bing and a few others respect it. Googlebot ignores it completely, so setting a crawl delay for Google does nothing. If a crawler is overloading your server, better hosting or a fix in Search Console will serve you far better than this line.

Crawling is not indexing (the mistake that costs traffic)

This is the single most important idea in this guide, so read it twice.

Disallow blocks crawling. It does not block indexing.

If you disallow a page, Google will not read its contents. But if another site links to that page, Google can still index the URL. You then see the dreaded result in search with no description, showing the message that a description is not available because of the site's robots.txt file. The page sits in the index looking broken, which is the worst of both worlds.

To keep a page out of search results, you need a different tool. Use a noindex meta tag in the page's HTML head, or a noindex HTTP header, or password protection for anything truly private. And here is the catch that catches everyone: for Google to see your noindex tag, it has to crawl the page. So do not Disallow a page in robots.txt and add noindex at the same time. The block stops Google from ever seeing the noindex instruction.

Short version. Robots.txt controls crawling. Meta robots controls indexing. Keep the two jobs separate.

Robots.txt examples you can copy

Real files beat theory. Here are the patterns you will reach for most often.

Allow all crawlers full access. An empty file does the same thing, but being explicit is clearer.

User-agent: *
Disallow:

Block all crawlers from the entire site. Useful on a staging server. Dangerous on a live one.

User-agent: *
Disallow: /

Block one directory for every bot.

User-agent: *
Disallow: /private/

Block one bot while allowing the rest.

User-agent: AhrefsBot
Disallow: /

User-agent: *
Disallow:

A practical file for a small business site.

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /cart/
Disallow: /checkout/
Disallow: /*?s=

Sitemap: https://www.example.com/sitemap.xml

That last one blocks the admin area, the cart and checkout pages, and internal search result URLs, while keeping the AJAX file open and handing over the sitemap. It is a sensible starting point for most content and ecommerce sites.

Controlling AI crawlers in 2026

This is where robots.txt has changed the most in recent years. In the 2020s, site owners started using the file to deny bots that collect content for generative AI, and that shift is now the main reason people edit their file at all.

AI crawlers announce themselves with their own user agent names, so you can allow or block them the same way you would any search bot. The big ones to know:

• GPTBot is OpenAI's crawler for training data.
• ClaudeBot is Anthropic's crawler.
• PerplexityBot is Perplexity's crawler.
• CCBot belongs to Common Crawl, whose open dataset feeds many AI models.
• Google-Extended is a control token for Google's AI products. It does not affect normal Google Search crawling.
• Applebot-Extended governs whether Apple can use your content for AI training.

To ask the common AI crawlers to stay away from your whole site:

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: PerplexityBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Google-Extended
Disallow: /

Keep in mind that AI crawler names change and new ones appear often. Check your server logs now and then to spot user agents you have not accounted for, and remember that compliance is voluntary. A block is a request, not a wall.

The RFC 9309 standard

For most of its life, robots.txt was a de facto standard that nobody officially owned. That changed in September 2022, when the Internet Engineering Task Force published RFC 9309, which formalised the Robots Exclusion Protocol. RFC 9309 codifies the parsing rules that Google and other major crawlers already followed, so it did not reinvent the file. It gave the old honour system a proper specification.

Cloudflare's Content Signals Policy

Traditional robots.txt answers one question: can a bot crawl this path? It says nothing about what the bot may do with the content afterward. That gap is exactly what publishers worry about in the age of AI answers.

In September 2025, Cloudflare launched its Content Signals Policy to fill that gap. It adds a machine readable line to robots.txt that expresses how your content may be used after it is fetched. The policy defines three signals:

• search, for building a search index and showing links or snippets.
• ai-input, for using content in AI answers such as retrieval augmented generation.
• ai-train, for training AI models.

Each signal takes a value of yes or no, or gets left blank to mean no stated preference. A publisher who wants classic search but no AI training would write something like this:

User-agent: *
Content-Signal: search=yes, ai-train=no
Allow: /

Cloudflare rolled the policy out across more than 3.8 million domains that use its managed robots.txt feature, with a default of search=yes and ai-train=no. It released the policy under a CC0 licence so any platform can adopt it, and it is working with standards bodies to push it forward. The IETF's AIPREF working group is now developing a formal standard for AI usage preferences, so expect this area to keep moving.

One honest caveat. Content signals are preferences, not enforcement. Some crawlers will honour them and some will not. If you need to actually stop a bot, pair the signals with a firewall rule or bot management, not robots.txt alone.

How to create a robots.txt file

You do not need special software. Any plain text editor works, including Notepad, TextEdit, or vi.

The steps are short:

1. Create a file and name it robots.txt, all lowercase.
2. Add your groups of rules, one directive per line.
3. Save it as UTF-8 plain text. Do not use a word processor, since programs like Word can insert curly quotes and hidden characters that break the file.
4. Upload it to the root directory of your site, often called public_html or www, using FTP or your host's file manager.
5. Load yourdomain.com/robots.txt in a browser to confirm it is live.

If you prefer not to write the file by hand, a robots.txt generator can build one from a few form inputs. Generators are fine for simple sites. For anything with custom rules, writing it yourself gives you far more control and fewer surprises.

WordPress robots.txt

WordPress creates a virtual robots.txt file automatically, so you may already have one even if you never made it. The catch is that a virtual file does not exist on disk, which makes it harder to edit directly.

Most people manage the WordPress robots.txt through an SEO plugin. Yoast SEO and Rank Math both include a file editor under their tools, where you can type rules and save without touching FTP.

There is also one setting that quietly controls the most powerful rule in the file. Under Settings, then Reading, there is a checkbox labelled "Discourage search engines from indexing this site." Tick it and WordPress adds User-agent: * and Disallow: / to your file, which hides the whole site. This is meant for sites under construction. Forgetting to untick it after launch is one of the most common reasons a new WordPress site vanishes from Google.

How to test and validate your robots.txt

Writing the file is half the job. Testing it is the other half, and skipping the test is how a stray slash takes a site offline for weeks before anyone notices.

A few ways to check your work:

• Open Google Search Console, go to Settings, then Crawling, and use the robots.txt report. It shows the file Google fetched and flags parsing errors.
• Use a robots.txt tester or validator to enter a specific URL and see whether it is allowed or blocked for a given user agent.
• Run a robots.txt checker after any site migration or redesign, since those are the moments when leftover Disallow: / rules escape into production.

Test before you push changes live, not after. It takes a minute and saves a lot of grief.

Common robots.txt mistakes

I have audited a lot of these files, and the same errors show up again and again.

• Leaving `Disallow: /` on a live site after a launch or migration. It blocks everything.
• Blocking CSS and JavaScript files. Crawlers need these to render your pages. Block them and Google may see a broken layout.
• Trailing slash slip ups. Disallow: /blog/tags also blocks /blog/tags-explained/. Add the slash, Disallow: /blog/tags/, to limit the rule to that folder.
• Using robots.txt to hide private data. It does the opposite. Disallowed paths are visible to anyone who reads the file. Use passwords for anything sensitive.
• Forgetting the sitemap line. A large share of sites never add it, and it is free discovery.
• Trying to noindex through robots.txt. As covered above, a blocked page cannot be crawled, so its noindex tag never gets read.

Robots.txt best practices checklist

Keep this list handy when you write or review a file.

• Place the file at the root, named robots.txt in lowercase.
• Give every subdomain its own file.
• Write one directive per line and separate groups with blank lines.
• Block only what you truly want kept out of crawling, such as admin areas, internal search, and duplicate parameter URLs.
• Never block resources like CSS and JS that crawlers need to render pages.
• Add your Sitemap line.
• Use noindex, not Disallow, when you want a page out of search results.
• Decide your stance on AI crawlers and set the matching user agent rules.
• Test in Search Console and with a validator before every deploy.
• Review the file after any migration, redesign, or CMS change.