What to block in robots.txt, and what to leave alone
By Hamza Rivera, updated
Short answer: A robots.txt file tells crawlers which URLs they may request, mainly to avoid overloading your site. Google says it is not a way to keep a page out of search: a blocked URL can still be indexed if other sites link to it. To keep a page out, use a noindex directive or password protection, and do not block files Google needs to understand the page.
Robots.txt is a small text file at the root of your site. It is easy to edit and easy to get wrong, because it looks like a switch for hiding pages and is not one. Here is what Google says it does.
If you want a second pair of eyes on your crawl setup, see our technical SEO services.
What robots.txt is for
Google describes robots.txt as a file that tells crawlers which URLs they can access on your site. Its main use is to avoid overloading the site with requests. It manages crawler traffic. That is the whole job.
What it is not for
Google states that robots.txt is not a mechanism for keeping a web page out of Google. If another site links to a URL you blocked, Google can still index the address, and sometimes the link text, without ever reading the page. That result shows up with no description. If you want a page kept out of search, Google points to a noindex directive, set with a meta tag or a response header, or to password protection. For private content, use a password, not robots.txt, because crawlers choose whether to obey the file.
Do not let the two work against each other
Google warns that mixing crawling and indexing rules can cancel each other out. If you block a page in robots.txt, Google cannot fetch it to see a noindex tag on it. Pick the tool that matches what you want: robots.txt to limit crawling, noindex to keep a page out of results.
What is reasonable to block
Google says you can block images, scripts or styles that are not important, as long as the page still works without them. The catch is the other direction: do not block resources Google needs to understand the page. Blocking the files that draw your layout can leave Google with a worse view of your pages than a visitor gets.
Check before you edit
Not every search engine supports robots.txt, and crawlers can read the syntax differently. If your site runs on a hosted builder such as Wix or Blogger, you may not be able to edit the file at all. Google says to use the builder's search visibility settings instead.
What not to expect
Changing robots.txt does not remove a page from Google and does not make a page rank. If a page you care about is missing from results, our guide on how long a new page takes to appear in Google covers the usual causes before you touch this file.
Where to go from here
If you want us to look at how your site is crawled, see our technical SEO services. Passieon is an independent agency and is not affiliated with Google.
Sources
- Google Search Central: Introduction to robots.txt, read November 10, 2026.
Frequently asked questions
What does a robots.txt file do?
It tells crawlers which URLs on your site they may request. Google says its main use is to avoid overloading your site with requests.
Does robots.txt keep a page out of Google?
No. Google says it is not a mechanism for that. A blocked URL can still be indexed, without a description, if other sites link to it.
How do I keep a page out of search results?
Use a noindex directive, set with a meta tag or response header, or protect the page with a password. Google also lets you remove the page.
Can I block scripts and stylesheets?
Google says you can block unimportant ones if the page works without them, but you should not block resources Google needs to understand the page.
Why can I not edit robots.txt on my site?
Some builders such as Wix and Blogger do not give access to the file. Google says to use the search visibility settings in that platform instead.
Do all search engines follow robots.txt?
Not all of them support it, and crawlers can read the syntax differently. The rules are not enforced, so crawlers choose whether to obey them.
Want this done for your site? See what our technical SEO services page covers, or read the client results.
Read next
- Landscaper marketing on Google: what to set up first A landscaper can show up on Google through a verified profile, a website with a page per service and photos of real work. Here is the order we would do it in.
- Google Business Profile for a cleaning company: address and service area Many cleaning companies work from home and go to the customer. Here is how to set the address, the service area and the services on a Google profile.
- Setting up a Google Business Profile for a pest control company A pest control company should pick one clear category, list the services it really offers and set a service area that matches where the trucks go.