← Blog

Robots txt no index no follow: What Each Directive Does and When to Use It

2026-11-16 · IndexNowFast Team
Robots txt no index no follow: What Each Directive Does and When to Use It

Quick Answer

If you searched for “robots txt no index no follow,” the key is to separate three different controls: robots.txt controls which URLs crawlers can access, noindex tells Google not to include an accessible page in its search results, and nofollow communicates how links on a page should be treated. They are not interchangeable. To request removal from Google Search, let Google crawl the URL so it can read the noindex instruction.

Key Takeaways

  • robots.txt Disallow controls crawler access, not guaranteed search-result exclusion.
  • noindex tells Google not to include an accessible page or resource in its index.
  • nofollow concerns links on a page; Google treats it as a hint rather than an absolute guarantee.
  • Blocking a page in robots.txt can prevent Google from discovering its noindex instruction.
  • Use an HTML robots meta element for HTML pages or an X-Robots-Tag HTTP header for resources such as PDFs.
  • Use actual authentication and access restrictions to protect confidential information. Robots.txt is not a security system.

Robots.txt, noindex, and nofollow: What's the difference?

These three instructions are often discussed together because they affect how search engines interact with website content. However, each addresses a different part of the process.

Think of a website as a building:

  • Robots.txt helps determine which doors a crawler may enter.
  • Noindex tells a search engine not to include an accessible page in its search index.
  • Nofollow communicates how the search engine should treat links found on a page.

Here is a practical comparison.

DirectiveWhat it controlsWhere it is implementedIntended effectImportant limitation
robots.txt DisallowCrawler access to URL paths/robots.txtPrevents compliant crawlers from accessing matching URLsDoes not reliably prevent URL indexing
noindexSearch-result inclusionHTML meta tag or X-Robots-Tag HTTP headerRequests exclusion from supported search indexesGoogle must crawl the resource to discover the instruction
nofollowLinks on a pageHTML meta tag or X-Robots-Tag HTTP headerCommunicates how links on the page should be treatedNot a guaranteed way to prevent all link discovery or consideration

The important distinction is that crawling, indexing, and link handling are separate activities. Changing one instruction does not automatically achieve the result associated with another.

Google documents these differences in its robots.txt introduction and robots meta tag documentation.

What robots.txt actually controls

Robots.txt is a text file that gives instructions to web crawlers about which parts of a website they may access. It is normally placed at the root of a hostname.

For example:

``text User-agent: Googlebot Disallow: /private-section/ ``

This rule tells Google's crawler not to crawl URLs matching the specified path under the applicable rules.

You can also create rules for all crawlers:

``text User-agent: * Disallow: /temporary/ ``

The asterisk represents all user agents covered by the rule. Different crawler groups can have different instructions.

Does robots.txt prevent Google from indexing a page?

Not necessarily.

A crawler may be prevented from accessing a URL while Google still knows that the URL exists. For example, Google might discover it through links on other websites or other available signals.

If Google cannot crawl the page, it cannot inspect its normal page content or discover a noindex instruction inside that content. The URL may therefore appear in search results without the usual page information.

This is why robots.txt Disallow should not be treated as a reliable deindexing method.

Google explains this distinction in its official robots.txt documentation.

Robots.txt is not website security

A Disallow rule does not require visitors or crawlers to authenticate. It is a crawler instruction, not an access-control system.

If a folder contains confidential documents, private customer information, or sensitive files, protect it using authentication, authorization, and appropriate server-side restrictions.

The IETF Robots Exclusion Protocol, RFC 9309, explicitly distinguishes crawler exclusion rules from access authorization.

What noindex does and how Google reads it

Noindex is a directive that tells supported search engines not to include a page or resource in their search indexes.

For an HTML page, you can use a robots meta element inside the document's head section:

``html <meta name="robots" content="noindex"> ``

This tells supported crawlers that can access and read the page that it should not be included in search results.

For example, imagine that your website has an old promotional landing page. The page is still accessible to visitors, but you no longer want it appearing in Google Search.

You can leave the page accessible and add a noindex directive. When Google recrawls the URL and processes the directive, it can remove the page from its search index.

Why Google must be able to crawl a noindex page

This is one of the most important technical details to understand.

Google needs to fetch the page before it can read a robots meta instruction in the HTML. If robots.txt blocks the URL, Google may not be able to discover that instruction.

Consider this configuration:

``text User-agent: Googlebot Disallow: /old-campaign/ ``

And the page contains:

``html <meta name="robots" content="noindex"> ``

Although the page contains noindex, the robots.txt rule can prevent Google from accessing the HTML. The crawler therefore cannot reliably discover the instruction.

If your goal is to remove the page from Google Search, allow crawling and use noindex rather than hiding the directive behind a Disallow rule.

Google explains this requirement in its guide to blocking search indexing with noindex.

Does noindex remove a page immediately?

No. Google must recrawl the URL, discover the instruction, and process the change. The time required is not guaranteed.

Requesting another crawl through Google Search Console does not guarantee immediate removal. A successful technical configuration also does not guarantee that every search engine will interpret the instruction in exactly the same way.

What nofollow means: Page-level versus individual links

Nofollow is different from both robots.txt and noindex.

A page-level nofollow instruction communicates that links on the page should not be followed in the usual way. Google currently treats nofollow as a hint, not an absolute guarantee that it will never consider or discover those links through other sources.

For an HTML page, the instruction can look like this:

``html <meta name="robots" content="nofollow"> ``

This is a page-level instruction. It concerns links across the page rather than one particular link.

Page-level nofollow versus rel=nofollow

There is another important distinction: the rel="nofollow" attribute on an individual link.

For example:

``html <a href="https://example.com/partner" rel="nofollow"> Visit partner website </a> ``

This qualifies that particular outbound link.

By comparison:

``html <meta name="robots" content="nofollow"> ``

applies at the page level.

ImplementationScopeTypical purpose
<meta name="robots" content="nofollow">Links on the pageCommunicate a page-wide link instruction
rel="nofollow"One individual linkQualify a specific outbound link

Google also documents rel="sponsored" for paid or sponsored links and rel="ugc" for user-generated content links. These attributes help describe the relationship associated with individual links.

See Google's documentation on qualifying outbound links.

Do not use page-level nofollow when your actual requirement is simply to qualify one particular external link. Choose the implementation that matches the scope of your requirement.

Can robots.txt, noindex, and nofollow work together?

Yes. Different directives can express different instructions, but their interaction matters.

The biggest potential mistake is combining a robots.txt block with a page-level noindex instruction when the objective is to remove a URL from Google Search.

Here is a practical conflict-diagnosis matrix.

ConfigurationWhat happensCorrect interpretation
Disallow in robots.txt plus noindex in HTMLGoogle may be unable to crawl the URL and discover the HTML instructionConflicting configuration for a deindexing objective
Crawling allowed plus noindexGoogle can fetch the page and read noindexAppropriate for requesting search-result exclusion
Crawling allowed plus nofollowGoogle can read the page-level link instructionDoes not itself prevent page indexing
Crawling allowed plus noindex,nofollowGoogle can read both instructionsRequests search exclusion and communicates the page-level link instruction
X-Robots-Tag on a PDF or other non-HTML resourceThe HTTP response communicates the directiveUseful for controlling search inclusion of supported non-HTML resources

Scenario 1: Disallow and noindex together

Suppose you add a noindex tag to a page and also block the page in robots.txt.

``text User-agent: Googlebot Disallow: /private-offer/ ``

The page contains:

``html <meta name="robots" content="noindex"> ``

The crawler may be prevented from fetching the HTML, so it cannot discover the noindex instruction.

If you want the page removed from Google's index, remove the conflicting crawl restriction for that URL and allow Google to process the noindex directive.

Scenario 2: Crawling allowed with noindex

This is the appropriate approach when you want Google to access a page but exclude it from search results.

``html <meta name="robots" content="noindex"> ``

Google can fetch the document, read the directive, and process the exclusion request.

Scenario 3: Crawling allowed with nofollow

A page can remain eligible for indexing while carrying a page-level nofollow instruction.

``html <meta name="robots" content="nofollow"> ``

The directive concerns links on the page. It does not mean that the page itself must be excluded from search results.

Scenario 4: Crawling allowed with noindex,nofollow

You can combine the two directives in one meta element:

``html <meta name="robots" content="noindex, nofollow"> ``

Google can read both instructions when it crawls the page. Noindex concerns search-result inclusion, while nofollow concerns links on the page.

Combining them does not make the instructions interchangeable or guarantee a particular outcome across all search engines.

How to implement the directives correctly

Choose the implementation based on the type of resource and the result you want.

1. Use an HTML robots meta element

For an HTML document, place the instruction inside the head element.

Noindex only:

``html <meta name="robots" content="noindex"> ``

Nofollow only:

``html <meta name="robots" content="nofollow"> ``

Both instructions:

``html <meta name="robots" content="noindex, nofollow"> ``

You can also use separate meta elements:

``html <meta name="robots" content="noindex"> <meta name="robots" content="nofollow"> ``

Google documents both comma-separated and separate directive implementations.

2. Use X-Robots-Tag for HTTP responses

An HTTP response header provides another way to communicate robots instructions.

For example:

``http X-Robots-Tag: noindex, nofollow ``

The server sends this header with the resource response.

This method is particularly useful for non-HTML resources, including PDF files, because those files do not have an HTML head section where you could place a robots meta element.

For example, if you have an outdated PDF guide that should not appear in Google Search, the server can return an appropriate X-Robots-Tag header with that PDF response.

Google provides implementation details in its robots meta tag and X-Robots-Tag documentation.

3. Use robots.txt for crawler access rules

For crawler access management, use a robots.txt file.

Example:

``text User-agent: Googlebot Disallow: /temporary-area/ ``

This is separate from the HTML meta and HTTP-header mechanisms. Do not put noindex into robots.txt and expect Google to treat it as a supported indexing directive.

Google announced the retirement of its unsupported robots.txt noindex interpretation in 2019. See its official announcement about unsupported robots.txt rules.

Which directive should you use?

The correct choice depends on the result you want.

Your objectiveAppropriate approachWhy
Prevent Googlebot from crawling a specified URL pathrobots.txt DisallowControls crawler access
Keep a page accessible but exclude it from Google SearchnoindexCommunicates the indexing exclusion instruction
Communicate how links across a page should be treatedPage-level nofollowApplies to links on the page
Qualify one individual outbound linkLink-level rel attributeApplies to that specific link
Exclude a PDF from search resultsX-Robots-Tag: noindexWorks through the HTTP response rather than HTML markup
Protect confidential website informationAuthentication and server-side access controlsPrevents unauthorized access rather than merely requesting crawler exclusion

A practical example

Imagine that your website contains three resources:

  • An outdated promotional page that visitors can still access.
  • A temporary server directory that crawlers should not access.
  • A PDF document that should not appear in search results.

You would approach them differently:

  1. For the outdated promotional page, allow Google to crawl the page and use noindex.
  2. For the temporary directory, consider an appropriate robots.txt Disallow rule, provided the objective is crawler access management rather than confidentiality.
  3. For the PDF, configure an X-Robots-Tag response header with noindex if the goal is search-result exclusion.

There is no need to apply all three instructions to every URL. Use only the directives that match the actual requirement.

How to check whether your directives are configured correctly

After implementation, verify the relevant response and page markup.

For an HTML page, check that:

  • The page is accessible to Googlebot when Google needs to read a noindex instruction.
  • The robots meta element appears in the rendered HTML.
  • The page does not have an unintended conflicting robots directive.
  • The relevant robots.txt rules do not prevent the required crawl.

For a PDF or other non-HTML resource, inspect its HTTP response headers and confirm that the X-Robots-Tag is being returned as intended.

If you manage the website, Google's Search Console provides URL Inspection information about Google's indexed view and a separate Live Test of the current page. These are different checks and should not be confused with a guarantee of indexing or immediate removal.

You can also use the IndexNowFast Google Index Checker to review observed Google index status for individual URLs or larger URL lists. Its results are point-in-time observations, not a diagnosis by themselves. Check robots.txt, meta robots, X-Robots-Tag, accessibility, and other relevant technical signals before deciding what to change.

For a broader explanation of indexing problems, see the Google indexing troubleshooting guide.

Frequently Asked Questions

Does robots.txt noindex work in Google?

No. Google does not support noindex as a robots.txt directive. Use an HTML robots meta element or an X-Robots-Tag HTTP response header instead.

Can Google index a page blocked by robots.txt?

Yes, a blocked URL may still appear in search results if Google discovers it through other sources. Google may not have access to the page content, so the result may lack normal descriptive information.

Should I use noindex and nofollow together?

Only when both instructions match your objective. Noindex requests search-result exclusion, while nofollow communicates how links on the page should be treated. You do not need nofollow simply because you use noindex.

Does nofollow prevent Google from crawling links?

It is not an absolute guarantee. Google treats nofollow as a hint, and it may discover URLs through other sources. Do not use it as a guaranteed crawler-blocking mechanism.

What is the difference between meta robots nofollow and rel=nofollow?

Meta robots nofollow is a page-level instruction concerning links across the page. The rel=nofollow attribute qualifies one individual link.

Can I use X-Robots-Tag for PDFs?

Yes. X-Robots-Tag can communicate robots directives through HTTP response headers, making it useful for PDFs and other non-HTML resources.

How long does Google take to remove a noindex page?

There is no guaranteed removal timeframe. Google must recrawl the URL, discover the noindex directive, and process the change. A recrawl request does not guarantee immediate removal.

Final Decision: Choose the Directive That Matches Your Goal

The difference between robots.txt, noindex, and nofollow becomes clear when you separate their three responsibilities.

Robots.txt manages crawler access. Noindex communicates whether an accessible resource should appear in search results. Nofollow communicates how links on a page should be treated.

The most important practical rule is simple: If you want Google to remove a page from search results, do not block Google from reading the noindex instruction. Keep the URL crawlable, implement the appropriate directive, and allow time for Google to recrawl and process the change.

Choose each instruction deliberately, verify where it is implemented, and remember that Google's indexing decisions are not guaranteed by any single technical directive.