Free BuilderAI & GEO Ready

AI-Ready robots.txt Generator

Create an optimized robots.txt file in seconds. Control which AI bots (GPTBot, ClaudeBot, Gemini) can cite your website.

1. Select a Preset Strategy

2. Site Information & Paths

3. Individual Bot Permissions

14 Supported Crawlers
GPTBotAI Training

Trains ChatGPT & OpenAI models

ChatGPT-UserAI Search

Live search requests initiated by ChatGPT users

ClaudeBotAI Training

Trains Claude 3.5 models

Google-ExtendedAI Training

Controls training for Gemini & Vertex AI

PerplexityBotAI Search

Retrieves live web sources for Perplexity citations

Applebot-ExtendedAI Training

Trains Apple Intelligence foundation models

Meta-ExternalAgentAI Training

Meta AI Llama model training crawler

BytespiderAI Training

ByteDance Doubao & TikTok AI scraper

CCBotAI Training

Bulk web crawler feeding open LLM datasets

AmazonbotAI Training

Amazon Bedrock & Alexa AI crawler

GooglebotSearch Engine

Main Google search indexing crawler

BingbotSearch Engine

Microsoft Bing & Copilot web crawler

AhrefsBotSEO Tool

SEO backlink and audit crawler

SemrushBotSEO Tool

Semrush keyword & site audit crawler

Live Output

robots.txt Preview

# robots.txt generated by UI Pirate (https://uipirate.com/tools/robots-txt-generator)
# Domain: example.com
# Last updated: 2026-08-24

User-agent: *
Allow: /
Disallow: /admin/
Disallow: /api/
Disallow: /checkout/

# Explicitly Allowed AI & Search Crawlers
User-agent: GPTBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: Google-Extended
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Applebot-Extended
Allow: /

User-agent: meta-externalagent
Allow: /

User-agent: Amazonbot
Allow: /

User-agent: Googlebot
Allow: /

User-agent: bingbot
Allow: /

User-agent: AhrefsBot
Allow: /

User-agent: SemrushBot
Allow: /

# Explicitly Blocked AI Training Scrapers
User-agent: Bytespider
Disallow: /

User-agent: CCBot
Disallow: /

# AI Infrastructure & Discovery
# llms.txt: https://example.com/llms.txt
Sitemap: https://example.com/sitemap.xml
Save as: robots.txtLocation: /public/robots.txt
robots.txt Architecture

How to Properly Configure robots.txt for AI Search

Standard robots.txt directives follow RFC 9309. Understanding how AI crawlers interpret wildcard fallback rules prevents accidental search exclusion.

01

Order of Specificity

Specific bot rules (e.g. User-agent: GPTBot) take priority over the global wildcard User-agent: *.

02

Explicit Allow Overrides

Always pair disallow blocks with explicit allows for public documentation and sitemaps so AI citation agents can verify your domain authority.

03

Sitemap Declaration

Declare absolute URLs for all XML sitemaps at the very bottom of your robots.txt to ensure complete multi-engine crawl coverage.

Frequently Asked Questions about robots.txt

Where should I place robots.txt?

robots.txt must live in the root directory of your website (e.g. https://yourdomain.com/robots.txt). For Next.js projects, place it in /public/robots.txt or use app/robots.ts.

Can I block AI training without hurting Google SEO?

Yes. By targeting named user-agents (GPTBot, ClaudeBot, CCBot) and leaving Googlebot and Bingbot allowed, your Google rankings remain 100% unaffected.

UI Pirate Ecosystem

Suggested Tools for Your Stack

Explore related utilities in this workflow or discover cross-category tools.

View All Tools →