Robots.txt

Robots.txt is a text file placed in the root directory of a website (e.g., example.com/robots.txt) that instructs search engine crawlers which pages or directories should not be crawled. It uses the Robots Exclusion Protocol with directives like 'User-agent', 'Disallow', and 'Allow'. For example, 'Disallow: /admin/' blocks crawlers from accessing the admin folder. Important: robots.txt does not prevent indexing—only crawling. A page blocked by robots.txt but linked from elsewhere may still appear in search results (without a snippet). To prevent indexing, use a 'noindex' meta tag or password protection instead.