Googlebot-Video: How to Identify It, Configure robots.txt, and Fix Crawling Issues

Quick Answer
Googlebot-Video is Google's dedicated crawler for video-related Search features. Its HTTP user-agent is Googlebot-Video/1.0, while its robots.txt identifier is Googlebot-Video. You can control its access through a dedicated robots.txt group, but allowing it to crawl does not guarantee that Google will index your videos.
Key Takeaways
- Google's documented Googlebot-Video user-agent is
Googlebot-Video/1.0. - Use
Googlebot-Videoas the user-agent token in robots.txt. - Googlebot-Video and GoogleOther-Video are separate crawlers.
- A specific robots.txt group is not automatically combined with the wildcard group.
- A user-agent string alone cannot prove that a request came from Google.
- Video crawling, resource retrieval and video indexing are different outcomes.
What Is Googlebot-Video?
Googlebot-Video is a Google crawler associated with video-related Google Search features. It fetches video resources so Google can process information needed for its video search experiences.
Like other search engine crawlers, it sends HTTP requests to publicly accessible website resources. Your web server, content delivery network (CDN), or hosting provider can record these requests in server logs.
What makes this crawler important for technical SEO is its separate identity. Website administrators can use its dedicated robots.txt user-agent token to control which resources it may crawl.
Google documents its crawler identities in its official common crawlers reference.
There is an important distinction: Googlebot-Video visiting a resource does not mean the associated video will appear in Google Search. Google must also be able to access and understand the video, and the content must meet its indexing requirements.
Googlebot-Video versus other Google crawlers
Google operates several crawlers for different purposes. Their names can look similar, but their functions and configuration options are not necessarily interchangeable.
| Crawler | Documented HTTP user-agent | General purpose |
|---|---|---|
| Googlebot-Video | Googlebot-Video/1.0 | Product-specific video crawling for Google's video-related Search features |
| GoogleOther-Video | GoogleOther-Video/1.0 | Generic fetching of publicly accessible video URLs |
| Googlebot Smartphone | Googlebot Smartphone user-agent containing its documented Googlebot signature | General web crawling using a smartphone-oriented crawler |
| Googlebot Desktop | Googlebot Desktop user-agent containing its documented Googlebot signature | General web crawling using a desktop-oriented crawler |
GoogleOther-Video deserves particular attention. It is not another name for Googlebot-Video. Google documents it as a separate generic video-fetching crawler rather than a crawler dedicated to a specific Google product.
For exact signatures and current crawler descriptions, consult Google's common crawlers documentation.
How to Recognize a Googlebot-Video Request
The documented HTTP user-agent for Googlebot-Video is:
``text
Googlebot-Video/1.0
``
A web server may record a request containing this value in its access logs. For example, a log entry might contain a timestamp, requested video URL, response status, source IP address and user-agent string.
The user-agent is useful for identifying requests that claim to come from Googlebot-Video. However, it is not proof of the request's origin.
Any client can send an HTTP request with a copied user-agent string. A malicious bot, monitoring service or ordinary script could therefore identify itself as Googlebot-Video without being operated by Google.
To establish whether a request is genuine, verify its source IP address using Google's documented verification procedure. This is particularly important before using crawler logs to make technical SEO decisions.
HTTP user-agent versus robots.txt token
These two values serve different purposes:
| Where you use it | Correct identifier |
|---|---|
| HTTP user-agent identification | Googlebot-Video/1.0 |
| robots.txt user-agent group | Googlebot-Video |
The HTTP value helps identify the crawler making a request. The robots.txt token tells Google which crawler-specific rules apply.
Do not copy the full HTTP user-agent string into the robots.txt user-agent declaration. Use the documented product token instead.
How to Allow or Block Googlebot-Video in robots.txt
The robots.txt file is a text file placed at the root of a website's host. It tells compliant crawlers which URL paths they may or may not request.
For example, a website might have video resources under /videos/ and private resources under /private-videos/.
A dedicated Googlebot-Video group could look like this:
``text
User-agent: Googlebot-Video
Allow: /videos/
Disallow: /private-videos/
``
This illustrative configuration allows Googlebot-Video to crawl the specified video directory while disallowing the private directory.
It is not a universal configuration. Replace these paths with the actual directory structure of your website and review the complete robots.txt file before making changes.
What happens when Googlebot-Video encounters these rules?
User-agent: Googlebot-Videoidentifies the crawler group.Allow: /videos/permits matching URLs under that path.Disallow: /private-videos/prevents the crawler from requesting matching URLs under that path.
When multiple rules match a URL, Google generally applies the most specific matching rule. If matching rules have the same length, an Allow rule takes precedence over a Disallow rule.
Does Googlebot-Video inherit wildcard rules?
This is one of the most important robots.txt details to understand.
Consider the following file:
```text User-agent: * Disallow: /media/
User-agent: Googlebot-Video Allow: /media/ ```
It might appear that the second group simply adds an exception to the first group. However, Google's robots.txt group-selection rules work differently.
Google selects the most specific matching user-agent group. It does not automatically combine the wildcard group with a more specific group. Multiple groups matching the same specific user-agent can be combined, but the global wildcard group is not automatically inherited by the specific Googlebot-Video group.
Therefore, the dedicated group must be reviewed as a complete set of instructions for that crawler. Do not assume that restrictions written only under User-agent: * also apply to Googlebot-Video.
Read the official Google robots.txt specification before deploying crawler-specific rules.
Can robots.txt protect private videos?
No. Robots.txt is a crawler instruction mechanism, not an access-control system.
A disallowed URL may still be discovered through links or other references. Its URL can potentially appear in search results even when Google has not crawled its contents.
If a video must remain private, protect it using appropriate authentication, authorization or server-side access restrictions. Do not rely on robots.txt to keep confidential resources secure.
Also remember that robots.txt and noindex solve different problems. Robots.txt controls crawler access, while a noindex directive tells a crawler that can access the page not to include it in search results. Google must be able to crawl a page to see its noindex directive.
How to Verify Genuine Googlebot-Video Traffic
Finding Googlebot-Video/1.0 in your access logs is a useful starting point, not a completed verification.
Google recommends two approaches for checking whether a request genuinely originated from its crawlers:
- Reverse DNS lookup followed by forward DNS confirmation.
- Checking the source IP against Google's published crawler IP ranges.
The reverse-and-forward DNS approach checks whether the IP address resolves to an appropriate Google hostname and whether that hostname resolves back to the original IP.
A practical verification workflow
Step 1: Find the request in your server logs.
Look for requests whose user-agent contains Googlebot-Video/1.0.
Record the following information:
- Source IP address.
- Request timestamp.
- Requested URL.
- HTTP response status.
- User-agent string.
Step 2: Perform a reverse DNS lookup.
Use a DNS lookup tool to find the hostname associated with the source IP address.
For example, on a system with standard DNS utilities, an administrator can use:
``bash
dig -x SOURCE_IP
``
Replace SOURCE_IP with the actual address from the log. This is an illustrative command, not a result from a real crawler request.
Step 3: Confirm the hostname through forward DNS.
Resolve the returned hostname to its IP addresses. Check that the original source IP appears among the returned addresses.
Step 4: Check the hostname against Google's documented verification requirements.
A reverse lookup result alone is insufficient. Confirm that the hostname belongs to the appropriate Google crawler domain and that the forward lookup matches the original address.
Step 5: Alternatively, check Google's published crawler IP ranges.
Google also provides published IP-range information for crawler verification. Compare the source IP against the relevant official range rather than relying on an unrelated third-party IP database.
Google explains both approaches in its official guide to verifying Google requests.
Once verification succeeds, you can classify the request as genuine Google crawler traffic with greater confidence. If it fails, do not attribute the request to Google merely because its user-agent looks correct.
Why Googlebot-Video May Access a Page but Not Index Its Video
A common technical misunderstanding is treating a successful crawl as proof of video indexing.
These are separate stages:
- Crawler identity: Which crawler is making the request?
- Crawl permission: Does robots.txt allow it to access the resource?
- Resource retrieval: Can Google successfully fetch the page, video and associated assets?
- Video eligibility: Does the content satisfy Google's video indexing requirements?
- Video indexing: Does Google actually include the video in its video-related Search features?
A problem at any stage can affect the final result.
For example, Google might access a watch page successfully but fail to retrieve the video file because a CDN restriction blocks the media request. Similarly, a video may be accessible but fail to qualify for indexing because the page does not meet Google's requirements.
Google's video SEO best practices explain the technical and content requirements that matter for video discovery and indexing.
Check the watch page and video resource separately
Start by checking whether the page containing the video is publicly accessible.
Then investigate the actual video resource. A working HTML page does not prove that its embedded video, thumbnail or associated media files can also be fetched.
Review these potential problems:
- robots.txt restrictions: The page, video file or thumbnail may be blocked for the relevant crawler.
- Authentication: The video might require a login, subscription or session cookie.
- CDN restrictions: Hotlink protection, geographic restrictions, bot filtering or firewall rules may prevent Google from retrieving the video.
- Unstable URLs: Temporary or frequently changing video and thumbnail URLs can interfere with discovery and retrieval.
- Player visibility: The video should be clearly presented on an appropriate watch page rather than hidden behind an inaccessible interface.
- Rendering problems: Important video information may not be available in the rendered page.
- Content eligibility: The page and video must satisfy Google's video indexing requirements.
Do not assume every indexing problem is caused by Googlebot-Video being blocked. A crawler can have permission to access a resource while other technical or content issues prevent successful indexing.
Use Search Console to investigate video indexing
Google Search Console provides a Video indexing report for sites with eligible data. It helps identify pages where Google has indexed a video and pages where it has detected a video but has not indexed it.
Use the report to understand the status and any reported issues. For an individual URL, URL Inspection can provide additional information about Google's stored indexing data and the results of a current live test.
These views answer different questions. Stored indexing information describes Google's known indexed state, while a live test checks current conditions. A successful live test does not guarantee that the video will subsequently be indexed.
See Google's Video indexing report documentation.
IndexNowFast also provides a Google Index Checker for point-in-time URL status checks and bulk CSV exports. Its results can help identify URLs that need investigation, but they do not authenticate Googlebot-Video requests or establish that a particular video has been indexed. For owned websites, use Search Console for Google's own diagnostic information.
For a more detailed explanation of the distinction between indexed information and live testing, see Google Search Console URL Inspection Tool: Results and Fixes.
Googlebot-Video Troubleshooting Decision Matrix
Use this table to connect what you observe with the next useful investigation. It separates crawler identification, access configuration, resource retrieval and indexing rather than treating them as one problem.
| What you observe | What it tells you | What it does not prove | Recommended action |
|---|---|---|---|
Log contains Googlebot-Video/1.0 | The request declares Google's documented video crawler user-agent | That Google actually sent the request | Verify the source IP through DNS or Google's published IP ranges |
Dedicated group contains Disallow: / | The group blocks Googlebot-Video from matching paths on that host | That other Google crawlers are also blocked | Review each relevant user-agent group |
| Wildcard group blocks a directory, but a specific group exists | Google selects the most specific matching group | That wildcard restrictions automatically apply to the specific crawler | Inspect the complete specific group |
| Watch page loads, but video retrieval fails | Page access and media access are different | That the video resource is crawlable | Check media URL responses, CDN restrictions and authentication |
| Google detects a video but does not index it | Detection has occurred without confirmed video indexing | That robots.txt is necessarily responsible | Review video eligibility and Search Console reports |
| Apparent Googlebot-Video requests come from an unverified IP | The user-agent claims a Google identity | That the request is genuine Google traffic | Authenticate the source before classifying it |
The matrix provides a practical starting point. For difficult cases, correlate server logs, CDN logs, robots.txt, the actual media response and Search Console information rather than relying on one signal.
Frequently Asked Questions
What is the Googlebot-Video user-agent string?
Google documents the HTTP user-agent as Googlebot-Video/1.0. You may encounter this value in server or CDN access logs when a request identifies itself as Google's video crawler. Verify the source IP before treating the request as genuine.
What robots.txt token controls Googlebot-Video?
Use Googlebot-Video in the robots.txt user-agent declaration. Do not confuse this token with the HTTP user-agent string, which is Googlebot-Video/1.0.
Is Googlebot-Video the same as GoogleOther-Video?
No. Googlebot-Video is associated with Google's video-related Search features. GoogleOther-Video is a separate generic crawler optimized for fetching publicly accessible video URLs. Google documents both separately.
Can I block Googlebot-Video without blocking Googlebot Smartphone?
Yes. You can create separate robots.txt groups for the two crawler identities. However, check the complete file carefully because a specific crawler group is not automatically combined with the wildcard group.
Does Googlebot-Video follow wildcard robots.txt rules?
Google selects the most specific matching user-agent group. A specific Googlebot-Video group is not automatically combined with the global wildcard group. Multiple groups matching the same specific user-agent can be combined.
Does allowing Googlebot-Video guarantee video indexing?
No. Allowing crawling only removes a potential access restriction. Google must also retrieve and process the video, and the content must meet its video indexing requirements. Google ultimately decides what appears in Search.
How can I verify whether Googlebot-Video traffic is genuine?
Use reverse DNS followed by forward DNS confirmation, or compare the source IP with Google's published crawler IP ranges. A matching user-agent string alone is not sufficient proof.
Final Thoughts
Understanding Googlebot-Video is about more than recognizing a crawler name in server logs. Website owners need to know which crawler is requesting their resources, which robots.txt rules apply, whether the actual video can be retrieved and whether the content qualifies for indexing.
Start with the documented Googlebot-Video/1.0 user-agent and the Googlebot-Video robots.txt token. Verify suspicious requests, review crawler-specific rules, test the actual media resources and use Search Console to investigate video indexing outcomes.
Most importantly, keep crawling permission, successful retrieval and indexing separate. That distinction makes Googlebot-Video troubleshooting more accurate and helps prevent unnecessary robots.txt changes when the real issue lies elsewhere.