can ahrefs crawl site urls? Site Audit Setup, Limits, and Troubleshooting

Quick Answer

Yes, Ahrefs can crawl site URLs through its Site Audit tool. You can audit website pages, XML sitemaps, custom URL lists, and URLs found through backlink data. AhrefsBot also crawls the public web for Ahrefs research tools. Neither type of Ahrefs crawl guarantees that Google will crawl, index, or rank your pages.

Key Takeaways

Ahrefs offers different tools for different crawling goals. The right choice depends on whether you need a technical site audit, a check of known URLs, backlink data, or Google's indexing records.

• Ahrefs Site Audit crawls website URLs to find technical SEO issues, including access, linking, and page-level problems.
• Site Audit supports five starting URL sources: Website, Auto-detected sitemaps, Specific sitemaps, Custom URL list, and Backlinks.
• Custom URL lists accept direct entry or CSV and TXT uploads, with a documented 16 MB file-size limit.
• To restrict a crawl to sitemap URLs, select Specific sitemaps, disable other sources, and set maximum depth from seed to zero.
• Ahrefs documentation dated September 3, 2026, lists 5,000 monthly Site Audit crawl credits per verified project under its free allowance.
• Project scope, robots.txt rules, server access, crawl limits, and JavaScript settings can all affect which URLs Ahrefs reaches.
• An Ahrefs crawl does not prove that Googlebot visited a URL or that Google added it to search results.

Start with the URLs you need to check. Then choose the crawl method that matches that goal. Use Google Search Console when your real concern is Google indexing.

Which Ahrefs Tool Should You Use to Crawl URLs?

Use Ahrefs Site Audit to crawl your own website for technical issues. Use Site Explorer for backlink research. Use Google Search Console to investigate Google's crawl and indexing records.

These tools collect different kinds of data. Choosing the wrong one can waste time and lead to false conclusions.

Ahrefs operates two main crawlers. The official Ahrefs crawler documentation identifies AhrefsBot and AhrefsSiteAudit as separate systems. AhrefsBot collects public-web data for Ahrefs research products. AhrefsSiteAudit checks URLs within a configured technical audit.

A Site Audit crawl does not force AhrefsBot to revisit those pages. It also does not update Site Explorer's backlink records on demand.

The following matrix matches common SEO tasks to the correct tool. It also shows what each result can establish.

The main distinction is between an audit you configure and a database you research.

Ahrefs Site Audit gives site owners control over crawl inputs and technical settings. Site Explorer shows data collected through Ahrefs' separate web research process. Google Search Console reports information from Google's own systems.

Ahrefs also confirms in its manual URL submission guidance that users cannot send URLs or XML sitemaps to Site Explorer or Content Explorer for immediate on-demand crawling.

If you simply want to check your site's technical health, begin with Site Audit. There is no need to wait for Site Explorer to discover every URL first.

How to Crawl Website URLs With Ahrefs Site Audit

To crawl website URLs with Ahrefs, create a Site Audit project, set its scope, choose URL sources, and run the audit. Review the completed URL results before drawing conclusions about site coverage.

You should know which domain or site section you want to inspect. Decide whether you need broad URL discovery or a defined group of pages. This choice determines the best starting source.

Choose the Right Starting URL Source

Ahrefs documents five options in its Site Audit configuration guide. Each option gives the crawler a different place to start. You can use suitable sources together, subject to your project settings.

• Website: Start with website discovery and follow eligible links within the configured scope.
• Auto-detected sitemaps: Let Ahrefs use XML sitemaps it detects for the website.
• Specific sitemaps: Supply selected sitemap locations for a more controlled audit.
• Custom URL list: Enter known URLs or upload a supported file containing them.
• Backlinks: Use URLs associated with known backlink data as an audit source.

Ahrefs documents Website and Auto-detected sitemaps as default source options. These are a sensible starting point for a general site audit.

Custom lists serve a different purpose. They help when you already know which pages need checking. Specific sitemaps are useful when you want to assess a defined sitemap set.

For a broader overview of available tools, see our guide to crawling a website online for free and finding SEO errors.

Set Scope, Crawl Depth, and Rendering

Set the project scope before starting the crawl. A URL outside that scope may be ignored even if it appears in your source list.

Scope can matter when your website uses several subdomains or separate URL paths. An audit aimed at one store section should not be assumed to cover the whole domain.

Crawl depth controls how far the crawler can follow links from its starting URLs. A page limit controls how much work the audit can attempt. Both settings affect coverage.

JavaScript also needs attention. Some websites load important content after the initial HTML response. Ahrefs provides an Execute Javascript setting for sites that depend on rendered content.

Rendering can help the crawler inspect such pages. It does not guarantee that every dynamic URL will be discovered.

Review exclusions before the crawl. Ahrefs supports URL filters, parameter-related controls, and settings for following nofollow links. Its guide to excluding pages from Site Audit explains these options.

Do not exclude whole URL patterns without checking their role. A broad filter may hide pages you need to audit.

Run the Crawl and Review the Results

Start the audit after checking sources, scope, limits, and access settings. Then review the crawl's completion status and URL-level findings.

Technical reports may reveal failed requests, redirects, broken links, and other site issues. But a completed audit is not proof that every intended page was checked.

Compare the crawled URL list with the source you meant to audit. If you uploaded 200 distinct product URLs, check those same URLs in the results. Don't rely only on the total number of pages crawled.

If pages are missing, record which ones failed to appear. Some may fall outside the project scope. Others may be blocked, excluded, or absent from the selected discovery sources.

Save the findings, fix the relevant settings or website issues, and run a follow-up crawl. Keep the original URL inventory so you can compare results using the same scope.

Crawl a CSV, TXT File, or Specific URL List

Yes, Ahrefs Site Audit supports custom URL lists. You can enter URLs directly or upload CSV and TXT files. The documented file-size limit is 16 MB, and the URLs must belong to your project scope.

This method is useful when you already have a list of pages. It removes the need to rely only on links from the homepage or on sitemap discovery.

Consider an online store that has exported its product URLs from its content management system. The owner wants to check a selected set of products before a major site update.

The owner can prepare that list, select Custom URL list as the source, and use the project scope that covers those products.

A practical import workflow follows these steps:

1. Export the URLs you want to audit from your CMS or another trusted inventory.
2. Remove duplicate entries and check for URLs outside your intended scope.
3. Create or open the relevant Ahrefs Site Audit project.
4. Select Custom URL list under the available URL sources.
5. Enter the URLs directly or supply a supported CSV or TXT file.
6. Review crawl limits, exclusions, and access settings before starting.
7. Run the crawl and compare results with the original URL list.

The comparison is the most useful part. It shows which planned pages appeared in the audit and which need further checks.

A 16 MB file limit refers to the size of the uploaded file. It does not promise a fixed number of successfully crawled pages. URL length, project settings, credits, and site access can affect the outcome.

Also check redirects. A submitted URL may lead to a different final URL. Record both addresses when reviewing the results. Otherwise, the crawl may appear to have missed a page that redirected elsewhere.

For repeat audits, keep the same URL list and note any deliberate changes. That makes it easier to compare results after a site migration, product update, or technical fix.

Crawl Only the URLs in an XML Sitemap

To crawl only URLs from a chosen XML sitemap, select Specific sitemaps, disable other starting sources, and set maximum depth from seed to zero.

This setup follows Ahrefs' official sitemap-only crawl instructions.

The depth setting matters because the crawler can otherwise follow links from the sitemap pages to other URLs. That may expand the audit beyond the set you intended to check.

A sitemap-only audit can answer a narrow question: which URLs in this selected sitemap can Ahrefs crawl under the current settings?

Compare the following configurations before starting your project.

These two crawls answer different questions. A general crawl tests what the selected discovery methods can reach. A sitemap-only crawl tests the listed sitemap URLs without following extra links.

Neither method proves that your sitemap contains every important page. A website can have valid URLs that were never added to its XML sitemap.

A sitemap URL may also fail to load, redirect, or be restricted by server settings. The presence of a URL in an XML file is not proof that it will return a successful page response.

For useful follow-up work, compare the sitemap URLs with your CMS export and Site Audit results. Keep track of which URLs are missing from each source. That helps separate sitemap maintenance issues from crawl access issues.

Why Ahrefs Misses Some Website URLs

Ahrefs may miss a URL because it never found the page, excluded it through crawl settings, or could not fetch it. JavaScript, crawl depth, project limits, and available credits can also affect the results.

Start with the observed failure rather than changing every setting at once. A URL that was never discovered needs a different fix from one that returned an access error.

URL Discovery and Scope Problems

A website may contain pages with few or no internal links. If the crawl starts from the homepage, those pages may never enter the crawler's path.

A sitemap or custom URL list can help expose known pages. But these sources work only for URLs they contain.

Scope is another common source of confusion. A project restricted to a URL path may not crawl related pages elsewhere on the domain. Exclusion patterns can also remove pages from the audit.

For an independent inventory, our guide to finding all URLs on a website explains why relying on one discovery source can leave gaps.

Robots.txt, Firewall, and Access Failures

A URL that loads in your browser may still fail for Ahrefs. Website security tools can treat crawler requests differently from normal visits.

The Ahrefs Site Audit access troubleshooting guide covers robots.txt restrictions and other access problems.

Check whether the intended crawler is allowed to fetch the page. Review server responses and any firewall or bot protection rules that may apply.

Do not remove intentional restrictions without understanding their purpose. Private pages, test areas, and sensitive paths may be blocked for good reasons.

For a more detailed diagnosis, follow our guide to testing website crawlability and finding errors.

JavaScript, Crawl Depth, and Credit Limits

A page can load its main content through JavaScript. If Site Audit does not render that content, the resulting report may be incomplete.

Enable JavaScript execution where it is needed. Then check whether the expected content appears in the crawl results.

A low crawl depth may stop discovery before the crawler reaches deeper pages. Page limits and available credits may also end an audit before all intended URLs have been checked.

AhrefsBot has separate discovery behavior. According to Ahrefs' explanation of incomplete website coverage, website popularity and URL priority can affect what its public-web crawler collects.

Use the following diagnostic matrix to choose the first useful check. The causes are possibilities, not confirmed findings about any particular website.

The table helps prevent the wrong repair. For instance, raising crawl depth will not fix a firewall block. Enabling JavaScript will not add URLs missing from every selected source.

Classify the failure first. Then change one relevant setting or site condition and compare the next crawl against the same URL inventory.

Ahrefs Free Crawl Credits, Speed, and Practical Limits

Ahrefs' free allowance includes 5,000 Site Audit crawl credits per month per verified project, according to its documentation updated September 3, 2026. Check the current Ahrefs free-feature rules before planning a crawl.

Crawl credits are a usage allowance. They are not the same as 5,000 unique pages found, 5,000 successful URL fetches, or 5,000 pages indexed by Google.

A project can include redirects, errors, excluded URLs, or pages that need rendering. The number of URLs in a source list may differ from the number successfully audited.

Crawl speed has its own limits. Ahrefs documents a normal maximum rate of 30 URLs per minute for its Site Audit crawler, while verified owners may have different configuration options.

That figure describes crawler rate behavior, not guaranteed real-world throughput. Slow server responses, rendering work, access restrictions, and account settings can affect how much work finishes in a given period.

For a large website, start with the pages most relevant to your audit goal. A targeted crawl can produce useful findings without assuming the whole domain must be covered in one run.

Keep separate records for the number of submitted URLs, successfully crawled URLs, and URLs that returned errors. Those counts answer different questions.

If your real concern is how often Googlebot visits the website, Ahrefs' crawl rate is not a substitute. Google follows its own scheduling and resource decisions. Our guide to finding the bottleneck in Google's crawl rate covers that separate process.

Paid-plan allowances and advanced controls can change. Use the limits shown in your current Ahrefs account when setting crawl size and frequency.

Does Crawling With Ahrefs Help Google Index a Page?

Crawling a URL with Ahrefs does not force Google to crawl or index it. Site Audit helps find technical problems that may affect access, but Google makes its own crawling and indexing decisions.

Three separate paths explain the difference.

AhrefsSiteAudit crawls URLs for technical audit reports. AhrefsBot collects public-web information for Ahrefs research products. Googlebot visits URLs as part of Google's search crawling process.

A successful Site Audit result tells you something about Ahrefs' access under that audit configuration. It does not show that Googlebot reached the URL.

Google's official recrawling guidance describes URL Inspection for individual pages and sitemap submission for larger URL groups. Google states that a crawl request does not guarantee indexing.

For a single important page, use our guide to requesting indexing in Google Search Console. Check its current status before submitting another request.

When a page remains absent from search, the issue may involve discovery, access, indexing rules, or Google's own decisions. Our guide to why Google is not indexing your pages helps separate those cases.

Google also provides official crawling error troubleshooting guidance. It covers issues involving site access, discovery, and server behavior.

Fix technical problems that affect both crawlers when the evidence supports doing so. But avoid treating an Ahrefs success message as a Google indexing confirmation.

Ahrefs' documented IndexNow-related support is also separate from Google's general indexing workflow. Do not assume that an IndexNow submission or an Ahrefs crawl will lead to Google index inclusion.

Check URL Coverage After an Ahrefs Crawl

The best way to measure Ahrefs crawl coverage is to compare its results with a defined list of expected website URLs. A crawl total alone does not show whether the right pages were checked.

First, build a list of distinct URLs that belong to the audit scope. You might use an authorized CMS export, selected XML sitemaps, or a maintained URL inventory.

Define how you will treat redirects, duplicate URLs, and alternate URL forms before counting. Otherwise, the same page may appear several times in your records.

Calculate Coverage Against a Defined URL Set

Use this simple editorial measurement framework:

URL coverage (%) = Successfully crawled distinct expected in-scope URLs ÷ Total distinct expected in-scope URLs × 100.

The denominator comes from your documented expected URL list. The numerator counts URLs that meet your chosen successful-crawl rule.

Consider a hypothetical website with 100 distinct expected URLs. Suppose its Site Audit results show 80 successful crawls under the chosen rules.

The measured coverage is 80 ÷ 100 × 100 = 80%.

This is an arithmetic example, not a reported Ahrefs result or a typical industry benchmark. It is also not Google's indexing rate.

A site owner can repeat the same measurement after making fixes. Keep the same expected URL list unless the website has genuinely added or removed pages.

A Worked Coverage Diagnosis

Suppose the hypothetical 100-URL inventory produces the following results. All figures below are illustrative, not live crawl data.

The support group needs closer review, but the numbers do not explain why those URLs were missed. The next task is to classify the 20 unsuccessful or missing URLs by their actual observed causes.

Some might be outside the selected sources. Others could have crawl errors, redirects, or restrictions. Use the audit results and server evidence to distinguish those cases.

The following record fields make the comparison repeatable. They also give a team enough context to assign a suitable fix.

This record is more useful than a single crawl count. It shows which URLs need work and what evidence supports each action.

A successful crawl does not prove that the page is indexable. For instance, a crawler can fetch a page containing a noindex directive. That page may be accessible while still being excluded from search indexing by design.

Likewise, noindex is not normally something a crawler can confirm before it has fetched the relevant page content. Keep access and indexing checks distinct.

After resolving issues, rerun the audit with matching settings. Compare the same expected URLs and note which outcomes changed. If you change the denominator or scope, label the new comparison clearly.

The resulting percentage is your own coverage measure. It is not an official Ahrefs metric or a Google indexing score. Its accuracy depends on how complete and well-defined your expected URL inventory is.

Frequently Asked Questions About Ahrefs Crawling

Ahrefs can check website URLs in several ways, but each crawler has its own purpose and limits. Finding a URL, gaining access to its content, and getting it indexed by Google are separate tasks. These answers cover pages missing from sitemaps, CSV imports, JavaScript content, crawl limits, and AhrefsBot's discovery process. They also explain what an audit result can prove and when to check Google's own records instead.

Can ahrefs crawl site urls that are not in a sitemap?

Yes. Site Audit can use website link discovery, custom URL lists, and other supported sources. A page does not have to appear in an XML sitemap to be included in an audit. It must still fall within the project scope and be accessible under the chosen crawl settings.

Can Ahrefs crawl all URLs on my website?

Ahrefs can crawl many eligible URLs within a configured project. It cannot guarantee that every existing URL will be discovered. Pages without reachable links, omitted sitemap entries, exclusions, and access restrictions can create gaps. Compare the results with a defined URL inventory when completeness matters.

Can I upload a CSV file containing website URLs?

Yes. Ahrefs Site Audit supports CSV and TXT uploads through Custom URL list. The documented upload size limit is 16 MB. Make sure the listed URLs fall inside your project scope. Then compare the results with the submitted list to identify missing or unsuccessful crawls.

Can Ahrefs crawl JavaScript websites?

Site Audit provides an Execute Javascript setting. It can help inspect pages whose content depends on JavaScript rendering. You still need suitable URL sources, adequate crawl limits, and site access. Rendering alone does not guarantee discovery of every dynamic page.

Can I force AhrefsBot to crawl my backlinks immediately?

No general on-demand submission workflow is provided for Site Explorer or Content Explorer. A custom URL list in Site Audit does not force the public-web AhrefsBot database to update. The two systems collect data for different purposes and follow different crawl processes.

Is Ahrefs Site Audit free?

Ahrefs documented a free allowance of 5,000 Site Audit crawl credits per month per verified project as of September 3, 2026. Eligibility and current limits should be checked in your account. Credits are a usage measure, not a promise of indexed pages or complete coverage.

Will an Ahrefs crawl get my page indexed by Google?

No. An Ahrefs crawl can reveal technical issues, but Google controls its own crawling and indexing. Use Google Search Console to inspect Google's recorded URL information. You can request a recrawl where appropriate, but Google does not guarantee index inclusion after a request.

Recommended Next Steps for a Reliable Ahrefs URL Audit

Start with a clearly defined URL set and the Ahrefs Site Audit source that best matches it. Keep the crawl scope stable, then compare the results with the pages you expected to check.

For a whole-site technical review, combine suitable website and sitemap discovery sources. For a product launch, migration, or selected-page audit, a custom URL list may provide a clearer comparison. Use a sitemap-only crawl when you need a controlled check of sitemap entries.

Review missing URLs by cause. Correct unintended access blocks, source omissions, and crawl settings before running the same test again. Keep a record of changes so the follow-up results are meaningful.

Finally, separate technical crawl success from search indexing. Use Ahrefs to find site problems and Google Search Console to examine Google's own URL information.

The useful outcome is not simply a larger crawl count. It is a clear record of which intended URLs were checked, which were missed, and what action should come next.