Robots.txt Best Practices in 2026

Q: Does `Crawl-delay` work for Googlebot?

No. Googlebot does not support `Crawl-delay` in `robots.txt`. If crawl rate is a problem, use Google Search Console settings where available and fix the server-side issue that makes crawling expensive.

Conceptual digital illustration of web crawlers and robots.txt code directives in a high-tech blue environment

The robots.txt file controls crawler access, not visibility by itself. Its job is to tell compliant bots which paths they may or may not crawl. That makes it useful for crawl management, sitemap discovery, and AI bot policy, but weak for security and insufficient for deindexing content on its own.

That distinction matters because robots.txt is one of the easiest files to misuse. Teams often block pages they actually want indexed, assume a Disallow rule removes URLs from search, or treat robots.txt like a privacy wall when it is really just a public set of crawl instructions.

This guide covers robots.txt best practices for modern SEO and AI-crawler management, with clear boundaries on what the protocol can and cannot do.

Quick takeaways

Use robots.txt to manage crawling, not to hide sensitive content.
Do not rely on robots.txt alone if your actual goal is non-indexing.
Be explicit about AI bot policy instead of assuming one wildcard rule will cover every use case safely.

Understanding the Scope: What Robots.txt Should and Should Not Do

Before implementing specific rules, you must understand the functional boundaries of the Robots Exclusion Protocol (REP). A common misconception is that robots.txt is a security tool or a way to remove content from search results. It is neither.

The Primary Role of Robots.txt

The main purpose of this file is to manage crawler traffic. By preventing bots from accessing low-value pages, you ensure they spend their limited time on your high-priority content. This is particularly important for large websites with thousands of URLs.

Manage Crawl Budget: Direct bots away from infinite scroll pages, search result filters, and duplicate content.
Prevent Server Overload: Stop aggressive crawlers from hitting resource-heavy scripts or applications.
Specify Sitemap Locations: Provide a clear path for crawlers to find your XML sitemaps.
Validate Format Correctness: Cross-check sitemap syntax and node patterns with this XML sitemap example and format guide.
Control AI Access: Explicitly allow or deny permission for LLM (Large Language Model) training bots to ingest your content.

What Robots.txt Cannot Accomplish

You must not rely on robots.txt for tasks it was never designed to handle. Misusing the file can create a false sense of security or lead to indexing errors.

It does not guarantee non-indexing: If a page is blocked in robots.txt but has external links pointing to it, Google may still index the URL without crawling the content. To keep a page out of the index, use a noindex meta tag or X-Robots-Tag.
It does not provide security: The robots.txt file is public. Anyone can view it by appending /robots.txt to your domain. Never list sensitive directories or "hidden" admin paths here, as you are essentially providing a map to potential attackers.
It is not a legal barrier: While reputable bots follow these rules, malicious scrapers and some aggressive AI bots may ignore your directives entirely.

Foundational Robots Txt Best Practices for Modern SEO

To build a robust robots.txt file, you must adhere to specific syntax and structural requirements. Errors in formatting can lead to crawlers ignoring your instructions or, in the worst-case scenario, treating the entire site as disallowed.

1. Proper File Placement and Naming

The file must be named exactly robots.txt in lowercase. It must reside in the root directory of your website.

Correct: https://example.com/robots.txt
Incorrect: https://example.com/scripts/robots.txt
Incorrect: https://example.com/Robots.txt

If you have multiple subdomains, each one requires its own robots.txt file. A file located on example.com will not govern the behavior of bots on blog.example.com. The same applies to CDNs: you cannot use yourwebsite.com/robots.txt to block content served from yourcdn.com. Each domain needs its own file.

2. Syntax and Directive Structure

The file consists of groups of directives. Each group starts with a User-agent line, followed by Allow or Disallow instructions.

User-agent: Identifies the specific bot you are addressing (e.g., Googlebot, Bingbot, GPTBot).
Disallow: Tells the bot not to visit a specific path or pattern.
Allow: Overrides a disallow rule for a specific sub-path.
Sitemap: Provides the full URL to your XML sitemap.

3. Using Wildcards Effectively

The Robots Exclusion Protocol supports two main wildcards: the asterisk (*) and the dollar sign ($).

The Asterisk (*): Represents any sequence of characters. For example, Disallow: /search?* blocks all URLs that start with /search?.
The Dollar Sign ($): Indicates the end of a URL. For example, Disallow: /*.php$ blocks any URL ending in .php, but would allow index.php?id=1.

4. Case Sensitivity and Path Matching

Paths in robots.txt are case-sensitive. If you disallow /Admin/, a crawler might still visit /admin/. Always match the exact casing used in your URL structure.

When a bot evaluates rules, it follows the most specific match. If you have: Disallow: /user/ Allow: /user/profile The crawler will be allowed to access the profile page because the Allow directive is more specific than the general Disallow on the parent folder.

5. Handling the 500 KB Limit

Google and other major search engines generally ignore robots.txt files that exceed 500 KB. Keep your file lean and focused. If you find your file growing too large, it is likely a sign that your URL structure is disorganized or that you are trying to use robots.txt for tasks better suited for noindex tags.

Managing AI Crawlers and LLM Training Bots

The rise of generative AI has introduced a new class of crawlers. These bots do not crawl to index your site for search results; they crawl to ingest data for training models. Managing these requires specific robots txt best practices to protect your intellectual property while maintaining search visibility.

Identifying Key AI Agents

Several major AI companies have released specific user-agent strings for their crawlers. You should address these individually if you wish to opt out of AI training. For implementation patterns, see AI bot controls in robots.txt and practical llms.txt examples for SaaS.

GPTBot: The main crawler for OpenAI.
ChatGPT-User: Used by ChatGPT plugins and features to interact with the web in real-time.
Google-Extended: Used by Google to improve its Gemini and Vertex AI models. Note that blocking this does not affect Googlebot's ability to index you for Search.
CCBot: The Common Crawl bot, which provides data used by many different AI models, including those from Anthropic and Meta.
Claude-Web: The crawler for Anthropic’s Claude model.

Strategic Blocking of AI Bots

You may choose to block AI bots while allowing search engines. This prevents your content from being used to generate AI answers that might reduce your click-through rate.

User-agent: GPTBot
Disallow: /
 
User-agent: Google-Extended
Disallow: /
 
User-agent: CCBot
Disallow: /

The "All Bots" Dilemma

Using User-agent: * applies to every bot that doesn't have a specific block dedicated to it. However, if you define a specific block for Googlebot, Googlebot will ignore the User-agent: * block entirely. It only follows the most specific group of directives that matches its name.

If you want to block all AI bots but allow search engines, you must be explicit. Many AI companies are now honoring the User-agent: * directive for training, but search engines also follow this. A better approach is to allow the search engines you want and block the specific AI agents you don't.

Real Case Study: The Impact of Blocking CSS and JS

In 2015, Google updated its technical guidelines to emphasize that "Googlebot needs to see your site like an average user." A significant observation was made by the SEO community when large-scale sites began seeing "Partial" rendering errors in Google Search Console.

The Case: A major e-commerce platform had a legacy robots.txt file that disallowed /assets/ and /plugins/ to save crawl budget. These folders contained the site's CSS and JavaScript files.

The Observation: When Googlebot was blocked from these files, it could not render the page layout. It saw a broken, unstyled version of the site. Because the mobile-friendly elements were defined in the CSS, Google concluded the site was not mobile-friendly. This led to a measurable drop in mobile search rankings.

The Fix: The team updated the robots.txt to explicitly allow access to these resources: Allow: /*.js Allow: /*.css

The Result: Within two weeks of the change, Google Search Console reported "Page is mobile-friendly," and the site’s mobile rankings recovered to their previous levels.

Lesson Learned: Never block search engines from accessing the resources required to render your page. This includes CSS, JavaScript, and image files that contribute to the user experience.

Common Mistakes to Avoid

Even experienced webmasters make errors that can jeopardize their site’s SEO. Avoid these common pitfalls to ensure your robots.txt remains effective.

1. Blocking the Entire Site on Production

It is common to use Disallow: / on staging environments to prevent them from appearing in search results. However, this rule is frequently left in place during a "go-live" migration. Always verify that your production robots.txt is not blocking the root directory.

This happens more often than you would expect. One e-commerce team launched a redesigned site without removing the staging robots.txt. Within days, organic traffic collapsed. Google Search Console's Coverage report showed a spike in "Disallowed by robots.txt" errors across the entire site. The fix was straightforward — update the file and request recrawling of key pages — but recovery still took several days. The lesson: always include a robots.txt check in your deployment checklist.

2. Conflicting Directives

If you have conflicting rules within the same user-agent group, the behavior can be unpredictable. Disallow: /catalog/ Allow: /catalog/ In this scenario, Google usually defaults to the Allow because it is more permissive or specific, but other crawlers might choose the Disallow. Be clear and avoid redundancy.

3. Using Robots.txt to Hide Sensitive Data

As mentioned, robots.txt is a public file. If you have a directory like /tmp/internal-admin-login/, putting it in robots.txt tells every hacker exactly where your admin login is. Use server-side authentication (like .htaccess password protection) or IP whitelisting instead.

4. Improper Use of Trailing Slashes

The presence or absence of a trailing slash changes the meaning of a directive.

Disallow: /folder blocks /folder, /folder/, and /folder-name-123.
Disallow: /folder/ only blocks the directory /folder/ and its contents. Be precise with your paths to avoid over-blocking or under-blocking content.

5. Ignoring 4xx and 5xx Errors

If your robots.txt file returns a 5xx (Server Error), Google will assume there is a temporary issue and will stop crawling the site altogether to avoid adding more stress to your server. If it returns a 404 (Not Found), Google assumes you have no restrictions and will crawl everything. Ensure your robots.txt file is consistently accessible and returns a 200 OK status code.

Practical Robots Rules for SEO and Performance

Tailoring your robots.txt to your specific CMS or site structure is essential. Here are practical implementation strategies for common scenarios.

Selective Parameter Blocking

When blocking query parameters, you often need to keep some while blocking others. A blanket Disallow: /*? blocks all URLs with any query string, which can break pagination or tracking if you are not careful.

Use selective Allow directives to override:

User-agent: *
Disallow: /*?
Allow: /*?page=*
Allow: /*?ref=*

This blocks most query strings but preserves access to pagination and referral parameters.

Faceted navigation (filters for size, color, price) can create millions of duplicate URLs. This is the biggest drain on crawl budget for online stores.

Strategy: Identify the URL parameters used for filtering and block them. Disallow: /*?color= Disallow: /*?price= Disallow: /*?sort_by=

Internal Search Results

Search engines do not want to index your internal search results. This is considered "search results in search results," which provides a poor user experience. Disallow: /search/ Disallow: /query/

Staging and Development Environments

If you have a development site, use a catch-all block.

User-agent: *
Disallow: /

Note: It is safer to also use password protection (Basic Auth) for staging sites, as some bots ignore robots.txt.

WordPress Specifics

WordPress creates several virtual paths that don't need to be indexed.

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
 
Sitemap: https://example.com/sitemap_index.xml

Note: You must allow admin-ajax.php because many themes and plugins use it to load content dynamically.

Advanced AI Crawler Rules: A Deep Dive

As AI technology evolves, the way we interact with their crawlers must also evolve. Simply blocking everything may not be the best long-term strategy.

The Trade-off: Visibility vs. Protection

If you block GPTBot, your content will not be used to train future versions of GPT models. However, this also means that when users ask ChatGPT for information related to your niche, the AI may not have the most up-to-date or accurate information about your brand.

Some publishers are choosing to allow ChatGPT-User while blocking GPTBot. This allows the AI to "browse" the web for specific user queries (providing citations and links) while preventing the bulk "ingestion" of the site for model training.

Handling "Common Crawl" (CCBot)

CCBot is one of the most important bots to manage. Its data is used by a vast array of AI companies. If you want a broad "No AI" policy, CCBot should be your first target.

User-agent: CCBot
Disallow: /

Google-Extended and the Future of Search

Google has been very clear that Google-Extended is the toggle for their AI training (Gemini). If you are a news organization or a content creator concerned about AI-generated summaries replacing your traffic, blocking Google-Extended is a standard precaution. It does not negatively impact your performance in standard Google Search or Google News.

Robots.txt vs noindex vs X-Robots-Tag

These three tools look similar but solve different problems. Confusing them is one of the most common causes of indexing issues.

Robots.txt prevents crawling. It tells bots "don't go here." It does not guarantee prevention of indexing if the URL is linked elsewhere.
noindex meta tag (<meta name="robots" content="noindex">) allows crawling but prevents indexing. The bot must crawl the page to see this tag.
X-Robots-Tag HTTP header works like the noindex meta tag but is delivered in the HTTP header. Useful for non-HTML files (PDFs, images) or for applying noindex across many pages without modifying HTML.

The critical interaction: if you Disallow a page in robots.txt AND apply a noindex tag to it, the noindex tag will never be seen by the crawler because it is blocked from accessing the page. The page can then remain indexed — often without a snippet, which is worse than not appearing at all.

Rule of thumb:

Use robots.txt to save crawl budget on unimportant pages that you genuinely don't want crawlers to spend time on.
Use noindex for pages you do not want indexed but need crawlers to access so they can discover the directive.

Troubleshooting: "Indexed, though blocked by robots.txt"

You disallowed a page in robots.txt, but it still appears in Google's search results with a generic description. This happens because:

Robots.txt told Googlebot not to crawl the page.
Googlebot could not see the noindex tag because it never crawled the content.
Other websites or internal links pointed to the URL.
Google indexed the URL based on link signals alone, without content.

To fix it:

Remove the Disallow rule from robots.txt for that specific page.
Add a noindex meta tag to the page's HTML <head>.
Wait for Google to crawl the page and see the noindex tag.
Once the page drops from search results, re-add the Disallow rule if you want to save crawl budget.

Validation Workflow: How to Test Your Rules

Never upload a robots.txt file without testing it first. A single typo can de-index your entire site.

Step 1: Use the Google Search Console Robots.txt Tester

Google provides a legacy tool within Search Console that allows you to test your file against specific URLs.

Navigate to the Robots.txt Tester tool.
Paste your new code into the editor.
Enter various URLs from your site (e.g., your homepage, a product page, a blocked admin page).
Click "Test" to see if the URL is "Allowed" or "Blocked."

Step 2: Check for Logic Errors

Ensure that your Allow and Disallow rules don't cancel each other out. Check that your Sitemap URL is the absolute path (including https://) and that it is reachable.

Step 3: Monitor "Crawl Stats"

After deploying a change, monitor the "Crawl Stats" report in Google Search Console. Look for a sudden drop in "Total crawl requests." If you intended to block a large section of the site, this drop is expected. If you didn't, it indicates you've over-blocked.

Step 4: Verify via "Inspect URL"

Use the "URL Inspection" tool in Google Search Console for key pages. If the tool says "Crawl allowed? No: blocked by robots.txt," you have confirmed the rule is active.

Robots.txt for International and Multi-regional Sites

If you manage a site with multiple languages or regions, your robots.txt strategy needs to account for how search engines discover these versions.

Sitemaps for Each Region

If you have separate sitemaps for different languages, list them all at the bottom of your robots.txt file.

Sitemap: https://example.com/en/sitemap.xml
Sitemap: https://example.com/es/sitemap.xml
Sitemap: https://example.com/fr/sitemap.xml

Don't Block Hreflang Resources

Search engines need to crawl the different versions of your pages to understand the hreflang relationships. If you block the Spanish version of your site in robots.txt, Google won't be able to see the hreflang="es" tag, which can break your international targeting.

Security Considerations and the "Robots.txt Leak"

While we have established that robots.txt is not a security tool, it is important to understand how it can be used against you. Security researchers and malicious actors often scan robots.txt files to find "interesting" directories.

The Better Way to Hide Folders

Instead of: Disallow: /private-api-v2/

Use:

Noindex via Header: Send an X-Robots-Tag: noindex HTTP header for that directory. This tells search engines not to index it without revealing the path in a public text file.
Authentication: Require a login to access the directory.
IP Restriction: Only allow access from your office or VPN IP addresses.

Handling "Leaked" URLs

If a sensitive URL has already been indexed because it was blocked in robots.txt (preventing Google from seeing a noindex tag), follow this sequence:

Remove the Disallow rule in robots.txt.
Add a noindex meta tag to the page.
Wait for Google to crawl the page and see the noindex tag.
Once the page is removed from search results, re-add the Disallow rule if you want to save crawl budget.

The Role of Robots.txt in Crawl Budget Management

For enterprise-level sites with millions of pages, crawl budget is a critical SEO factor. Googlebot has a "crawl capacity limit" for every site, based on server speed and site reputation.

Prioritizing Content

Use robots.txt to ensure Googlebot isn't wasting its capacity on:

Redundant URL Parameters: Session IDs, tracking parameters, and unnecessary sort orders.
Auto-generated Content: Tag pages with very few posts or archive pages that offer no unique value.
Legacy Files: Old PDF catalogs or outdated documentation that you no longer want to promote.

Monitoring Server Load

If your server logs show that a specific bot (like a rogue AI scraper) is hitting your site 100 times per second, use robots.txt to block it immediately. This is a functional use of the file to maintain site stability.

Example Rulesets for Common Scenarios

Use these templates as a starting point for your own configuration.

The "Standard Professional" Ruleset

This ruleset is ideal for most B2B and service-based websites. It allows all search engines, blocks admin areas, and provides the sitemap.

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /scripts/
Disallow: /tmp/
 
Sitemap: https://example.com/sitemap.xml

The "Privacy-First / No-AI" Ruleset

This ruleset is for publishers who want to maximize search visibility while strictly opting out of AI data ingestion.

User-agent: *
Disallow: /cgi-bin/
 
# Block AI Training Bots
User-agent: GPTBot
Disallow: /
 
User-agent: Google-Extended
Disallow: /
 
User-agent: CCBot
Disallow: /
 
User-agent: anthropic-ai
Disallow: /
 
User-agent: Claude-Web
Disallow: /
 
User-agent: Omgili
Disallow: /
 
Sitemap: https://example.com/sitemap.xml

The "E-commerce Optimization" Ruleset

Focused on saving crawl budget by blocking faceted navigation and internal search.

User-agent: *
Disallow: /checkout/
Disallow: /cart/
Disallow: /account/
Disallow: /search/
Disallow: /*?sort=
Disallow: /*?price=
Disallow: /*?filter_color=
Disallow: /*?filter_size=
 
Sitemap: https://example.com/sitemap.xml

Summary Checklist for Robots Txt Best Practices

Before you finalize your file, run through this checklist to ensure compliance with modern standards.

File Name: Is it robots.txt (all lowercase)?
Location: Is it in the root directory (e.g., domain.com/robots.txt)?
HTTP 200: Does the file return a 200 OK status code?
No Critical Blocks: Are you sure you aren't blocking CSS, JS, or high-value content?
Sitemap Link: Is the full, absolute URL to your sitemap included?
AI Strategy: Have you explicitly addressed bots like GPTBot or CCBot based on your data privacy preferences?
Clean Syntax: Have you removed any conflicting Allow and Disallow rules?
Case Sensitivity: Do your paths match the actual casing of your URLs?
Subdomains: Does each subdomain have its own file?
Testing: Have you verified the rules in Google Search Console’s tester?

By following these robots txt best practices, you create a clear, efficient path for search engines to discover your best content while protecting your server and data from the demands of modern AI crawlers. This file is a living document; review it at least once a quarter or whenever you make significant changes to your site's structure or business model.

To dive deeper, read our article about technical SEO for AI crawlers, which gives you a practical audit framework to ensure your content is not just found but truly understood by the next generation of search algorithms.

Frequently Asked Questions (FAQ)

Q1: Does robots.txt remove a page from Google search results?

No. It controls crawling, not guaranteed removal from the index. If a blocked URL is still linked elsewhere, Google can keep it in results without a snippet. Use noindex, proper status codes, or removals tooling when de-indexing is the actual goal.

Q2: Should I block CSS or JavaScript in robots.txt?

Usually no. Google needs access to key CSS and JavaScript files to render pages correctly. Blocking them can make pages harder to interpret and can create avoidable indexing or rendering problems.

Q3: Does `Crawl-delay` work for Googlebot?

No. Googlebot does not support Crawl-delay in robots.txt. If crawl rate is a problem, use Google Search Console settings where available and fix the server-side issue that makes crawling expensive.

Q4: Can I use robots.txt to hide private information?

No. Robots.txt is not a security tool. It only prevents search engine crawlers from accessing certain directories, but it does not prevent determined users from finding the content through other means. For true privacy, use authentication or IP restrictions.

Q5: How long does it take for Google to see changes in robots.txt?

Google typically caches the robots.txt file for up to 24 hours. If you need an urgent update, you can use the "Submit" feature in the Google Search Console Robots.txt Tester to ask Google to re-fetch it.

Q6: What is the difference between robots.txt and noindex?

Robots.txt controls crawling (whether a bot can access a page), while noindex controls indexing (whether a page can appear in search results). You can block crawling but still have a page indexed if it's linked from elsewhere.

Q7: What happens if my site has no robots.txt file?

If a robots.txt file is not found (returns a 404), crawlers assume there are no restrictions and will attempt to crawl all publicly accessible content. If you specifically want to allow all crawling, it is better to have an explicit file with an empty Disallow: and a Sitemap: directive than to let it 404.

Q8: How often should I update robots.txt?

Review it whenever you make significant changes to site structure, add or remove major sections, or change your AI bot access policy. At minimum, check it once per quarter and validate with Google Search Console after every update.