Robots.txt Generator
Create search engine compliant robots.txt files to manage web crawler access, protect sensitive paths, and optimize crawl budget.
General Rules
Generated Robots.txt
What is a Robots.txt File and Why Do You Need It?
A robots.txt file is a plain text file hosted in your website's root directory that tells web crawlers (such as Googlebot, Bingbot, and AI bots) which pages or sections of your site they can or cannot request.
How to Install Your Robots.txt File
- Click Download robots.txt above to save the file.
- Upload the file directly to your site's root directory via FTP or File Manager (e.g.,
https://yourdomain.com/robots.txt). - Verify that the file is accessible in your browser at
https://yourdomain.com/robots.txt. - Test and submit your file inside Google Search Console.
Does robots.txt hide pages from the public?
No. Robots.txt instructs search engine crawlers not to index pages, but it does not act as security. Use password protection or authentication for confidential directories.
Should I block AI bots like GPTBot?
Blocking GPTBot or CCBot stops generative AI companies from scraping your content to train AI models, without affecting your Google search indexing rankings.