Free tool
Robots.txt generator
A robots.txt in three parts: what you keep every crawler out of, where your sitemap is, and a choice for each AI crawler, named one by one with what it does, because blocking the ones that train a model and blocking the ones that answer a customer's question are different decisions. Nothing you type leaves your browser.
robots.txt
Everything open, no sitemap yet.
The file appears here as you type. This needs scripting on.
What was checked
What the file can do, and the line that empties a site
A robots.txt is a request to crawlers about which paths to fetch. It does not hide a page that is already known, it does not remove one from the index (a noindex tag does that, and only if the page can be crawled), and it does not stop a scraper that chooses to ignore it. What it does well is keep crawlers out of the pages that waste their time, such as carts, search results and admin areas, and point every one of them at the sitemap.
The line that matters most is the one this tool refuses to write: a slash on its own under the wildcard group. That keeps every search engine out of the whole site, and it is how a new site launches invisible, usually because the file was copied from the staging copy where it belonged. If you need a path blocked, name the path.
The AI section is where most published advice is careless. "Block AI" is two different decisions. A training crawler reads your pages to teach a model; blocking it changes nothing about where your business appears. An answering crawler fetches your page because a customer asked an assistant a question, or to build that assistant's search index; blocking it means the answer names your competitor instead. The middle preset blocks the first kind and keeps the second, which is the setting a business that wants enquiries from AI answers usually wants. Google-Extended sits in the first group and is the one people misread: it governs Gemini and Vertex AI, and Google Search and AI Overviews carry on through ordinary Googlebot either way.
Where it fits in the rest of the technical work
Robots.txt is the first line of the crawling section on the SEO audit checklist, and one of the things a site move most often breaks, which is why it has its own step on the migration checklist. What we check and fix in that layer is described on the technical SEO page.
Questions about robots.txt and AI crawlers
Should I block AI crawlers?
Decide it per crawler, because they do different jobs. The ones that train models (GPTBot, ClaudeBot, Google-Extended, CCBot) can be blocked without changing where your business appears. The ones that fetch pages to answer a live question or build an assistant’s search index (OAI-SearchBot, Claude-SearchBot, PerplexityBot and the on-demand user agents) are how a business gets cited in an AI answer, and blocking them takes you out of those answers. The preset that blocks training and keeps answering is the one most businesses want.
Does robots.txt stop a crawler?
It asks. The well-known crawlers from Google, Bing, OpenAI, Anthropic, Perplexity and Apple honour it; a scraper that ignores it will ignore it. Keeping something genuinely private needs a server rule or a login, not a line in this file.
Will blocking Google-Extended hurt my Google rankings?
No. Google-Extended only controls whether your pages are used for Gemini and Vertex AI. Google Search and AI Overviews are fed by ordinary Googlebot, which this tool never blocks.
Where does the file go?
At the root of the host, so that it answers at yourdomain followed by /robots.txt. One file per host: a www version and a bare version of the same site each need their own, or one should redirect to the other, and a subdomain is a separate host.
Is anything I type here sent anywhere?
No. Everything happens in your browser. There is no form submission, no analytics event and no network request of any kind, which you can confirm in your browser’s network tab.
Book a call with us
A 30-minute call in your working hours. Tell us what you sell and where, and we will tell you what we would do first and what it involves. Prefer email? Send your website through the form and the Passieon Team will reply within two working days.