Robots.txt Generator
Control the bots. Generate a valid robots.txt file to manage how search engines crawl your site.
Leave empty for no delay.
Disallow GPTBot, ChatGPT, Claude, and Common Crawl.
User-agent: * Allow: /
Leave empty for no delay.
Disallow GPTBot, ChatGPT, Claude, and Common Crawl.
User-agent: * Allow: /
What is a Robots.txt File?
A robots.txt file is a simple text file placed in your website's root directory that provides instructions to web crawlers (like Googlebot) about which pages they can or cannot crawl. It is part of the Robots Exclusion Protocol (REP) and is the first thing a bot looks for before indexing your site. While it doesn't "hide" pages from the public, it is essential for managing yourcrawl budget and keeping private or duplicate content out of search results.
Essential Commands and Use Cases
Protecting Against AI Bots
Use User-agent: GPTBot with Disallow: / to prevent OpenAI from using your site content to train AI models without your permission.
Managing Admin Areas
Telling bots to ignore /admin or /wp-admin ensures they focus on your valuable content instead of login pages.
Prioritizing Sitemaps
Always include a reference to your sitemap URL (e.g., Sitemap: https://yoursite.com/sitemap.xml) to help bots find your content faster.
Preventing Duplicate Indexing
Block search parameters or staging environments to prevent search engines from indexing multiple versions of the same page.
How to Validate Your Robots.txt
After generating and uploading your file, use the Robots.txt Tester inside Google Search Console. This ensures your rules are being interpreted correctly and that you haven't accidentally blocked high-traffic pages that you want to rank.
Press ⌘K