Robots.txt for WordPress 2026: Allow Google & AI Crawlers
Robots.txt controls which search engine crawlers and AI bots can access your site. Many WordPress sites accidentally block AI crawlers, preventing them from appearing in ChatGPT and Perplexity. Yoast SEO on Bluehost at $3.99/month manages robots.txt automatically. This guide covers everything about robots.txt for WordPress in 2026.
Robots.txt with Bluehost $3.99/mo
Yoast SEO: automatic robots.txt generation, visual editor, AI crawler management. Allow Google, ChatGPT, Perplexity.
Get Bluehost for $3.99/mo →Crawlers You Must Allow in Robots.txt
| Crawler | Engine | User-Agent |
|---|---|---|
| Googlebot | Google Search | Googlebot |
| Bingbot | Bing / Copilot | bingbot |
| GPTBot | ChatGPT / OpenAI | GPTBot |
| PerplexityBot | Perplexity AI | PerplexityBot |
| Google-Extended | Google AI / Gemini | Google-Extended |
| ClaudeBot | Anthropic / Claude | ClaudeBot |
Optimal Robots.txt for WordPress
- →Allow all crawlers: User-agent: * with Allow: /. This lets Google, Bing, and all bots crawl your site
- →Allow AI crawlers: Specifically allow GPTBot, PerplexityBot, Google-Extended, ClaudeBot
- →Block admin: Disallow: /wp-admin/. Keep admin pages private
- →Allow REST API: Allow: /wp-json/. Some plugins block this accidentally
- →Reference sitemap: Add Sitemap: https://yourdomain.com/sitemap_index.xml
- →Yoast SEO handles this: Yoast on Bluehost at $3.99/month generates proper robots.txt automatically
How to Edit Robots.txt in WordPress
- Step 1:Get Bluehost at $3.99/month and install WordPress + Yoast SEO
- Step 2:Go to Yoast SEO → Tools → File Editor
- Step 3:Click on robots.txt tab
- Step 4:Review the default rules Yoast generated
- Step 5:Add AI crawler allowances (GPTBot, PerplexityBot, Google-Extended, ClaudeBot)
- Step 6:Add sitemap reference: Sitemap: https://yourdomain.com/sitemap_index.xml
- Step 7:Save changes. Test at yourdomain.com/robots.txt
FAQ
What is robots.txt in WordPress?
File telling crawlers which pages they can access. Yoast SEO on Bluehost at $3.99/month generates and manages it. Edit via Yoast → Tools → File Editor. Don't accidentally block AI crawlers.
How do I edit robots.txt in WordPress?
Yoast SEO on Bluehost at $3.99/month. Go to Tools → File Editor → robots.txt. Visual editor, no FTP or coding. Add AI crawler allowances. Save and test.
Should I allow AI crawlers?
Yes. Allow GPTBot (ChatGPT), PerplexityBot (Perplexity), Google-Extended (Google AI), ClaudeBot (Anthropic). Without allowing, your site won't appear in AI search. Yoast SEO on Bluehost at $3.99/month.
What should my robots.txt look like?
Allow all: User-agent: * with Allow: /. Specifically allow AI bots. Block /wp-admin/. Reference sitemap. Yoast SEO on Bluehost at $3.99/month generates proper defaults.
Does robots.txt affect SEO?
Yes. Controls which pages Google and AI engines crawl. Blocking important pages prevents indexing. Yoast SEO on Bluehost at $3.99/month generates proper robots.txt. Allow all crawlers including AI.
Get Robots.txt Management — Bluehost $3.99/mo
Yoast SEO: automatic robots.txt, visual editor, AI crawler management. Allow Google, ChatGPT, Perplexity. 30-day money-back guarantee.
Get Bluehost for $3.99/mo →What are you building?
Tell us about your project so we can match the perfect hosting plan.
Hosting Cost Calculator
See exactly how much you'll spend on hosting over time. Compare Bluehost vs popular alternatives and discover your potential savings.
Used to calculate transaction fees (Shopify charges 2% on sales)
Frequently Asked Questions
What is robots.txt in WordPress?
Robots.txt is a file that tells search engine crawlers and AI bots which pages they can and cannot access. Yoast SEO on Bluehost at $3.99/month generates and manages robots.txt automatically. You can edit it via Yoast SEO → Tools → File Editor. Critical: make sure you're not accidentally blocking AI crawlers like GPTBot and PerplexityBot.
How do I edit robots.txt in WordPress?
Install Yoast SEO on Bluehost at $3.99/month. Go to Yoast SEO → Tools → File Editor → robots.txt. Add or modify rules. Yoast SEO provides a visual editor — no FTP or coding required. Make sure to allow all major crawlers including AI bots.
Should I allow AI crawlers in robots.txt?
Yes. Allow GPTBot (ChatGPT), PerplexityBot (Perplexity), Google-Extended (Google AI), and ClaudeBot (Anthropic). Without allowing these, your site won't appear in AI search results. Yoast SEO on Bluehost at $3.99/month lets you edit robots.txt to allow all AI crawlers.
What should my WordPress robots.txt look like?
Allow all crawlers: User-agent: * with Allow: /. Then specifically allow AI crawlers: User-agent: GPTBot, PerplexityBot, Google-Extended, ClaudeBot with Allow: /. Yoast SEO on Bluehost at $3.99/month generates a default robots.txt that you can customize via Tools → File Editor.
Does robots.txt affect SEO?
Yes. Robots.txt controls which pages Google and AI engines can crawl. Blocking important pages prevents indexing. Yoast SEO on Bluehost at $3.99/month generates a proper robots.txt automatically and lets you customize it. Make sure all crawlers including AI bots are allowed.
Ready to launch?
Get Bluehost from $3.99/month with a free domain, free SSL, and 30-day money-back guarantee.
Related Articles
What Is Shared Hosting? Complete Guide for Beginners (2026)
Shared hosting means multiple websites share one server. It costs $3.99/mo and is perfect for beginners. Learn pros, cons, and best providers.
What Is VPS Hosting? Complete Guide (2026)
VPS hosting gives you dedicated server resources at $19.99/mo. Learn how VPS works, when to upgrade, and why Bluehost VPS is the best choice.
What Is Managed WordPress Hosting? Complete Guide (2026)
Managed WordPress hosting means the provider handles updates, security, and speed optimization. Bluehost offers it at $3.99/mo with LiteSpeed.