Robots txt no index no follow: What Each Directive Does and When to Use It

Quick Answer
If you searched for “robots txt no index no follow,” the key is to separate three different controls: robots.txt controls which URLs crawlers can access, noindex tells Google not to include an accessible page in its search results, and nofollow communicates how links on a page should be treated. They are not interchangeable. To request removal from Google Search, let Google crawl the URL so it can read the noindex instruction.
Key Takeaways
- robots.txt Disallow controls crawler access, not guaranteed search-result exclusion.
- noindex tells Google not to include an accessible page or resource in its index.
- nofollow concerns links on a page; Google treats it as a hint rather than an absolute guarantee.
- Blocking a page in robots.txt can prevent Google from discovering its noindex instruction.
- Use an HTML robots meta element for HTML pages or an X-Robots-Tag HTTP header for resources such as PDFs.
- Use actual authentication and access restrictions to protect confidential information. Robots.txt is not a security system.
Robots.txt, noindex, and nofollow: What's the difference?
These three instructions are often discussed together because they affect how search engines interact with website content. However, each addresses a different part of the process.
Think of a website as a building:
- Robots.txt helps determine which doors a crawler may enter.
- Noindex tells a search engine not to include an accessible page in its search index.
- Nofollow communicates how the search engine should treat links found on a page.
Here is a practical comparison.
| Directive | What it controls | Where it is implemented | Intended effect | Important limitation |
|---|---|---|---|---|
| robots.txt Disallow | Crawler access to URL paths | /robots.txt | Prevents compliant crawlers from accessing matching URLs | Does not reliably prevent URL indexing |
| noindex | Search-result inclusion | HTML meta tag or X-Robots-Tag HTTP header | Requests exclusion from supported search indexes | Google must crawl the resource to discover the instruction |
| nofollow | Links on a page | HTML meta tag or X-Robots-Tag HTTP header | Communicates how links on the page should be treated | Not a guaranteed way to prevent all link discovery or consideration |
The important distinction is that crawling, indexing, and link handling are separate activities. Changing one instruction does not automatically achieve the result associated with another.
Google documents these differences in its robots.txt introduction and robots meta tag documentation.
What robots.txt actually controls
Robots.txt is a text file that gives instructions to web crawlers about which parts of a website they may access. It is normally placed at the root of a hostname.
For example:
``text
User-agent: Googlebot
Disallow: /private-section/
``
This rule tells Google's crawler not to crawl URLs matching the specified path under the applicable rules.
You can also create rules for all crawlers:
``text
User-agent: *
Disallow: /temporary/
``
The asterisk represents all user agents covered by the rule. Different crawler groups can have different instructions.
Does robots.txt prevent Google from indexing a page?
Not necessarily.
A crawler may be prevented from accessing a URL while Google still knows that the URL exists. For example, Google might discover it through links on other websites or other available signals.
If Google cannot crawl the page, it cannot inspect its normal page content or discover a noindex instruction inside that content. The URL may therefore appear in search results without the usual page information.
This is why robots.txt Disallow should not be treated as a reliable deindexing method.
Google explains this distinction in its official robots.txt documentation.
Robots.txt is not website security
A Disallow rule does not require visitors or crawlers to authenticate. It is a crawler instruction, not an access-control system.
If a folder contains confidential documents, private customer information, or sensitive files, protect it using authentication, authorization, and appropriate server-side restrictions.
The IETF Robots Exclusion Protocol, RFC 9309, explicitly distinguishes crawler exclusion rules from access authorization.
What noindex does and how Google reads it
Noindex is a directive that tells supported search engines not to include a page or resource in their search indexes.
For an HTML page, you can use a robots meta element inside the document's head section:
``html
<meta name="robots" content="noindex">
``
This tells supported crawlers that can access and read the page that it should not be included in search results.
For example, imagine that your website has an old promotional landing page. The page is still accessible to visitors, but you no longer want it appearing in Google Search.
You can leave the page accessible and add a noindex directive. When Google recrawls the URL and processes the directive, it can remove the page from its search index.
Why Google must be able to crawl a noindex page
This is one of the most important technical details to understand.
Google needs to fetch the page before it can read a robots meta instruction in the HTML. If robots.txt blocks the URL, Google may not be able to discover that instruction.
Consider this configuration:
``text
User-agent: Googlebot
Disallow: /old-campaign/
``
And the page contains:
``html
<meta name="robots" content="noindex">
``
Although the page contains noindex, the robots.txt rule can prevent Google from accessing the HTML. The crawler therefore cannot reliably discover the instruction.
If your goal is to remove the page from Google Search, allow crawling and use noindex rather than hiding the directive behind a Disallow rule.
Google explains this requirement in its guide to blocking search indexing with noindex.
Does noindex remove a page immediately?
No. Google must recrawl the URL, discover the instruction, and process the change. The time required is not guaranteed.
Requesting another crawl through Google Search Console does not guarantee immediate removal. A successful technical configuration also does not guarantee that every search engine will interpret the instruction in exactly the same way.
What nofollow means: Page-level versus individual links
Nofollow is different from both robots.txt and noindex.
A page-level nofollow instruction communicates that links on the page should not be followed in the usual way. Google currently treats nofollow as a hint, not an absolute guarantee that it will never consider or discover those links through other sources.
For an HTML page, the instruction can look like this:
``html
<meta name="robots" content="nofollow">
``
This is a page-level instruction. It concerns links across the page rather than one particular link.
Page-level nofollow versus rel=nofollow
There is another important distinction: the rel="nofollow" attribute on an individual link.
For example:
``html
<a href="https://example.com/partner" rel="nofollow">
Visit partner website
</a>
``
This qualifies that particular outbound link.
By comparison:
``html
<meta name="robots" content="nofollow">
``
applies at the page level.
| Implementation | Scope | Typical purpose |
|---|---|---|
<meta name="robots" content="nofollow"> | Links on the page | Communicate a page-wide link instruction |
rel="nofollow" | One individual link | Qualify a specific outbound link |
Google also documents rel="sponsored" for paid or sponsored links and rel="ugc" for user-generated content links. These attributes help describe the relationship associated with individual links.
See Google's documentation on qualifying outbound links.
Do not use page-level nofollow when your actual requirement is simply to qualify one particular external link. Choose the implementation that matches the scope of your requirement.
Can robots.txt, noindex, and nofollow work together?
Yes. Different directives can express different instructions, but their interaction matters.
The biggest potential mistake is combining a robots.txt block with a page-level noindex instruction when the objective is to remove a URL from Google Search.
Here is a practical conflict-diagnosis matrix.
| Configuration | What happens | Correct interpretation |
|---|---|---|
| Disallow in robots.txt plus noindex in HTML | Google may be unable to crawl the URL and discover the HTML instruction | Conflicting configuration for a deindexing objective |
| Crawling allowed plus noindex | Google can fetch the page and read noindex | Appropriate for requesting search-result exclusion |
| Crawling allowed plus nofollow | Google can read the page-level link instruction | Does not itself prevent page indexing |
| Crawling allowed plus noindex,nofollow | Google can read both instructions | Requests search exclusion and communicates the page-level link instruction |
| X-Robots-Tag on a PDF or other non-HTML resource | The HTTP response communicates the directive | Useful for controlling search inclusion of supported non-HTML resources |
Scenario 1: Disallow and noindex together
Suppose you add a noindex tag to a page and also block the page in robots.txt.
``text
User-agent: Googlebot
Disallow: /private-offer/
``
The page contains:
``html
<meta name="robots" content="noindex">
``
The crawler may be prevented from fetching the HTML, so it cannot discover the noindex instruction.
If you want the page removed from Google's index, remove the conflicting crawl restriction for that URL and allow Google to process the noindex directive.
Scenario 2: Crawling allowed with noindex
This is the appropriate approach when you want Google to access a page but exclude it from search results.
``html
<meta name="robots" content="noindex">
``
Google can fetch the document, read the directive, and process the exclusion request.
Scenario 3: Crawling allowed with nofollow
A page can remain eligible for indexing while carrying a page-level nofollow instruction.
``html
<meta name="robots" content="nofollow">
``
The directive concerns links on the page. It does not mean that the page itself must be excluded from search results.
Scenario 4: Crawling allowed with noindex,nofollow
You can combine the two directives in one meta element:
``html
<meta name="robots" content="noindex, nofollow">
``
Google can read both instructions when it crawls the page. Noindex concerns search-result inclusion, while nofollow concerns links on the page.
Combining them does not make the instructions interchangeable or guarantee a particular outcome across all search engines.
How to implement the directives correctly
Choose the implementation based on the type of resource and the result you want.
1. Use an HTML robots meta element
For an HTML document, place the instruction inside the head element.
Noindex only:
``html
<meta name="robots" content="noindex">
``
Nofollow only:
``html
<meta name="robots" content="nofollow">
``
Both instructions:
``html
<meta name="robots" content="noindex, nofollow">
``
You can also use separate meta elements:
``html
<meta name="robots" content="noindex">
<meta name="robots" content="nofollow">
``
Google documents both comma-separated and separate directive implementations.
2. Use X-Robots-Tag for HTTP responses
An HTTP response header provides another way to communicate robots instructions.
For example:
``http
X-Robots-Tag: noindex, nofollow
``
The server sends this header with the resource response.
This method is particularly useful for non-HTML resources, including PDF files, because those files do not have an HTML head section where you could place a robots meta element.
For example, if you have an outdated PDF guide that should not appear in Google Search, the server can return an appropriate X-Robots-Tag header with that PDF response.
Google provides implementation details in its robots meta tag and X-Robots-Tag documentation.
3. Use robots.txt for crawler access rules
For crawler access management, use a robots.txt file.
Example:
``text
User-agent: Googlebot
Disallow: /temporary-area/
``
This is separate from the HTML meta and HTTP-header mechanisms. Do not put noindex into robots.txt and expect Google to treat it as a supported indexing directive.
Google announced the retirement of its unsupported robots.txt noindex interpretation in 2019. See its official announcement about unsupported robots.txt rules.
Which directive should you use?
The correct choice depends on the result you want.
| Your objective | Appropriate approach | Why |
|---|---|---|
| Prevent Googlebot from crawling a specified URL path | robots.txt Disallow | Controls crawler access |
| Keep a page accessible but exclude it from Google Search | noindex | Communicates the indexing exclusion instruction |
| Communicate how links across a page should be treated | Page-level nofollow | Applies to links on the page |
| Qualify one individual outbound link | Link-level rel attribute | Applies to that specific link |
| Exclude a PDF from search results | X-Robots-Tag: noindex | Works through the HTTP response rather than HTML markup |
| Protect confidential website information | Authentication and server-side access controls | Prevents unauthorized access rather than merely requesting crawler exclusion |
A practical example
Imagine that your website contains three resources:
- An outdated promotional page that visitors can still access.
- A temporary server directory that crawlers should not access.
- A PDF document that should not appear in search results.
You would approach them differently:
- For the outdated promotional page, allow Google to crawl the page and use noindex.
- For the temporary directory, consider an appropriate robots.txt Disallow rule, provided the objective is crawler access management rather than confidentiality.
- For the PDF, configure an X-Robots-Tag response header with noindex if the goal is search-result exclusion.
There is no need to apply all three instructions to every URL. Use only the directives that match the actual requirement.
How to check whether your directives are configured correctly
After implementation, verify the relevant response and page markup.
For an HTML page, check that:
- The page is accessible to Googlebot when Google needs to read a noindex instruction.
- The robots meta element appears in the rendered HTML.
- The page does not have an unintended conflicting robots directive.
- The relevant robots.txt rules do not prevent the required crawl.
For a PDF or other non-HTML resource, inspect its HTTP response headers and confirm that the X-Robots-Tag is being returned as intended.
If you manage the website, Google's Search Console provides URL Inspection information about Google's indexed view and a separate Live Test of the current page. These are different checks and should not be confused with a guarantee of indexing or immediate removal.
You can also use the IndexNowFast Google Index Checker to review observed Google index status for individual URLs or larger URL lists. Its results are point-in-time observations, not a diagnosis by themselves. Check robots.txt, meta robots, X-Robots-Tag, accessibility, and other relevant technical signals before deciding what to change.
For a broader explanation of indexing problems, see the Google indexing troubleshooting guide.
Frequently Asked Questions
Does robots.txt noindex work in Google?
No. Google does not support noindex as a robots.txt directive. Use an HTML robots meta element or an X-Robots-Tag HTTP response header instead.
Can Google index a page blocked by robots.txt?
Yes, a blocked URL may still appear in search results if Google discovers it through other sources. Google may not have access to the page content, so the result may lack normal descriptive information.
Should I use noindex and nofollow together?
Only when both instructions match your objective. Noindex requests search-result exclusion, while nofollow communicates how links on the page should be treated. You do not need nofollow simply because you use noindex.
Does nofollow prevent Google from crawling links?
It is not an absolute guarantee. Google treats nofollow as a hint, and it may discover URLs through other sources. Do not use it as a guaranteed crawler-blocking mechanism.
What is the difference between meta robots nofollow and rel=nofollow?
Meta robots nofollow is a page-level instruction concerning links across the page. The rel=nofollow attribute qualifies one individual link.
Can I use X-Robots-Tag for PDFs?
Yes. X-Robots-Tag can communicate robots directives through HTTP response headers, making it useful for PDFs and other non-HTML resources.
How long does Google take to remove a noindex page?
There is no guaranteed removal timeframe. Google must recrawl the URL, discover the noindex directive, and process the change. A recrawl request does not guarantee immediate removal.
Final Decision: Choose the Directive That Matches Your Goal
The difference between robots.txt, noindex, and nofollow becomes clear when you separate their three responsibilities.
Robots.txt manages crawler access. Noindex communicates whether an accessible resource should appear in search results. Nofollow communicates how links on a page should be treated.
The most important practical rule is simple: If you want Google to remove a page from search results, do not block Google from reading the noindex instruction. Keep the URL crawlable, implement the appropriate directive, and allow time for Google to recrawl and process the change.
Choose each instruction deliberately, verify where it is implemented, and remember that Google's indexing decisions are not guaranteed by any single technical directive.