Robots.txt Generator
Create a robots.txt file in minutes. Choose which search engines and AI crawlers can visit your site, block private folders and add your sitemap.
- Includes AI crawlers
- Built-in URL tester
- Copy or download
How to Add Robots.txt to Your Website
- 1
Create your rules
Start with a preset, then choose what to block and which search engines and AI crawlers to allow.
- 2
Upload the file
Download robots.txt and upload it to your website’s root folder so it opens at yourdomain.com/robots.txt.
- 3
Check it in Google
Open the robots.txt report in Google Search Console to confirm Google can read your new file.
Frequently Asked Questions
01What is a robots.txt file?
Robots.txt is a small text file at the root of your website (for example, example.com/robots.txt) that tells search engine and AI crawlers which pages or folders they may visit. It is one of the first files crawlers check.
02Where do I upload robots.txt?
Upload it to the main (root) folder of your website, usually public_html, so it opens at yourdomain.com/robots.txt. It will not work in a subfolder. On WordPress, you can also edit it with an SEO plugin such as Yoast or Rank Math.
03Does robots.txt remove pages from Google?
No. Blocking a page in robots.txt stops Google from crawling it, but the URL can still appear in results if other sites link to it. To keep a page out of Google, allow crawling and add a noindex meta tag instead.
04Should I block AI crawlers like GPTBot?
It depends on your goals. Blocking AI training crawlers such as GPTBot, ClaudeBot and Google-Extended keeps your content out of future AI training, while allowing AI search crawlers such as OAI-SearchBot and PerplexityBot lets your business appear and be cited in AI answers. If you want visibility in ChatGPT, Claude or Perplexity, do not block their search crawlers.
05Does Google follow Crawl-delay?
No. Google ignores Crawl-delay and adjusts its crawl speed automatically. Bing and Yandex do respect it, so only add it if those crawlers are slowing down your server.
06How do I test my robots.txt?
Use the tester on this page to check whether a specific URL is allowed for a given crawler. After uploading, you can also check the robots.txt report in Google Search Console.
Want to Be Found in Google and AI Search?
Our SEO and AI SEO team makes sure the right crawlers can find, understand and recommend your business.