Robots.txt and noindex: what each one does

A single broad rule can make a useful part of a website invisible to Google. Before changing robots.txt, decide whether the goal is to reduce crawling, remove a page from Search, or protect private information. Each has a different tool.

Short answer: Robots.txt tells compliant crawlers which URLs they may request. It does not reliably hide a URL from Search. Use noindex on a crawlable page to keep it out of Search, or require a login for private content.

Written by Hamza Rivera. Last reviewed November 8, 2026

Hamza works in SEO and GEO/AEO. He writes the Passieon guides and checklists and does the SEO work for the businesses Passieon serves.

Use robots.txt for crawler access

The file at the site root can tell Googlebot not to request unimportant URLs, such as a large set of internal search results. It is a request to compliant crawlers, not a security boundary. Google may still show a blocked URL if another site links to it, although it cannot read the blocked page content.

Use noindex for a public page you do not want listed

Put a noindex meta tag or response header on the page and allow Googlebot to crawl it so the rule can be seen. Blocking that same URL in robots.txt can stop Google from reading the noindex instruction. For private records, use authentication rather than either search instruction.

Check the exact rule before publishing

Test a service page, a blog post, the contact page and each pattern the proposed rule matches. A rule meant for a staging folder can catch the live site if its path is too broad. Keep a copy of the old file and check Search Console after the change for unexpected exclusions.

Make a safe choice for each URL type

A page that should earn inquiries needs to be crawlable and indexable. A duplicate filter page may need a canonical or another inventory decision. A private client file needs access control. If the goal is a temporary removal, use the appropriate Search Console removal process while the lasting change is made on the site.

Sources for technical rules

Google's documentation is the authority for its own search systems. This guide explains the work in our words.