How to Crawl a Website Using Python: Code, Robots.txt, and CSV Export

Quick Answer

To crawl a website using Python, use Requests to download permitted HTML pages and Beautiful Soup to find internal links. Add a URL queue, robots.txt checks, request limits, and a CSV export. This works well for small HTML sites. Use Scrapy for larger projects or browser rendering when JavaScript hides needed links.

Key Takeaways

A good Python crawler does more than collect links. It must control requests, handle errors, and show exactly what it found.

• Requests and Beautiful Soup offer a simple starting point for crawling links on small HTML websites.
• A URL queue and visited set help the crawler follow new links without fetching the same URL repeatedly.
• Check robots.txt, approved hosts, timeouts, and request delays before crawling a website.
• Follow redirects with care, and preserve query strings that may represent different pages.
• Export actual HTTP results and errors to CSV rather than treating every discovered URL as a successful fetch.
• Scrapy offers more crawl-management tools, while browser rendering helps with links created by JavaScript.
• A Python crawler finding a URL does not prove that Google has crawled or indexed that page.

For most small projects, start with a limited HTML crawler. Review its output before increasing the crawl size or adding more tools.

Choose the Right Python Crawling Method

Use Requests and Beautiful Soup for a small HTML website. Choose Scrapy when you need a larger, repeatable crawl. Use browser rendering when key links are missing from the HTML returned by the server.

The right choice depends on the site and your goal. A basic script may be enough to collect internal links. A larger SEO audit may need sitemaps, detailed error records, and several sources of URL data.

Requests and Beautiful Soup for Small HTML Websites

The Requests HTTP library downloads pages and gives your code access to their responses. You can read status codes, response headers, and HTML content.

Beautiful Soup reads that HTML. It can find anchor tags and extract their href values.

These tools work well together, but neither manages a full website crawl for you. Your code must decide which links to follow, which URLs to skip, and when to stop.

This approach suits a small site with ordinary HTML navigation. It also helps developers learn how crawling works before moving to a larger framework.

Scrapy for Larger or Repeatable Crawls

Scrapy provides spiders, request scheduling, data extraction, and other crawl tools. Its official spider documentation also covers SitemapSpider and sitemap-based discovery.

A Scrapy project needs more setup than a short Requests script. That setup can pay off when you need to run the same crawl often or collect structured data from many pages.

Scrapy does not remove the need for careful scope and traffic rules. You still need to choose allowed sites, manage request rates, and review the output.

Browser Rendering for JavaScript-Created Links

Some websites build menus or page content after JavaScript runs. A simple HTTP request may receive only the starting HTML.

Google's JavaScript SEO documentation explains the difference between fetching a page and rendering its content.

A browser tool such as Playwright can run page scripts and inspect the resulting HTML. This adds setup and uses more resources.

Before switching tools, compare the raw HTML with what appears in a browser. If the needed links already exist in the HTML, browser rendering may add little value.

The comparison below shows which method fits each job. It is a guide to features, not a tested speed ranking.

For a first Python crawler, Requests and Beautiful Soup are enough to learn the core steps. Move to another method only when the site or report needs it.

For more detail on rendering and search engines, see how Google crawls websites with JavaScript.

Prepare Python and Set Safe Crawl Limits

Install the two Python packages, choose a website you have permission to crawl, and set limits before making requests. The tutorial uses Python's standard library for URL handling, CSV files, queues, and robots.txt parsing.

The settings below are examples for a small authorized test. They are not universal safe limits for every server.

Install the Required Packages

Use Python 3.11 or a compatible recent Python version with supported package releases. Open a terminal and run:

``bash python -m pip install requests beautifulsoup4 ``

On systems where Python uses a different command, you may need python3 instead.

Requests handles network requests. Beautiful Soup handles HTML parsing. Modules such as csv, collections, datetime, urllib.parse, and urllib.robotparser come with Python.

Save the crawler as crawler.py. Run it only after changing the example starting URL to an approved site and checking its access rules.

Set Hosts, Timeouts, and Page Limits

A crawler should know where it may go and when to stop. Without these controls, ordinary navigation can lead to hundreds of pages or unexpected hosts.

For this tutorial, use the following settings:

• Start URL: One approved HTTP or HTTPS page on the target site.
• Allowed scope: The exact hostname of the starting URL.
• Maximum recorded URLs: 25 per run.
• Base delay: Two seconds between page-request attempts.
• Request timeout: Five seconds for connection and 15 seconds for read inactivity.
• User agent: A clear crawler name with a real contact address.
• Output: A CSV file containing requested URLs, responses, and errors.

The delay and page limit are tutorial choices. Always follow the site's stated rules and adjust traffic based on permission and server behavior.

A Requests timeout does not create a guaranteed deadline for the entire download. A slow response may still take longer than expected.

Check robots.txt Before Fetching Pages

Robots.txt gives crawlers rules about which paths they may request. The Robots Exclusion Protocol in RFC 9309, published in September 2022, defines the main protocol behavior.

Python includes RobotFileParser. Its can_fetch() method checks whether a user agent may request a given URL under parsed rules.

The same module provides crawl_delay(), request_rate(), and site_maps() methods. Your application must decide how to use these values.

An unavailable robots.txt file and an unreachable robots.txt server are not the same. RFC 9309 requires crawlers to initially treat access as disallowed when server or network failures prevent access to robots.txt.

The code below uses a conservative policy. It allows an absent robots.txt file reported as HTTP 404 or 410. It stops when a robots response is unclear or cannot be reached. It also stops on robots.txt redirects rather than assuming permission.

This policy may reject some sites that a more complete RFC 9309 implementation could crawl. That is a deliberate limit for a beginner example.

Robots.txt is not a password system. Follow website terms, access rules, and data-handling duties as well. These duties may vary across US websites and services.

Build a Python Website Crawler Step by Step

A basic crawler starts with one URL, requests the page, finds links, and adds permitted internal URLs to a queue. It repeats this process until the queue is empty or the page limit is reached.

The script below puts those steps together. It also checks robots.txt, records failures, handles redirects without blindly following them, and exports a CSV file.

The code passed a Python syntax check during preparation. A complete live crawl and full network test were not performed. Test it on an authorized staging site before using it for regular work.

Complete Python Crawler Code

Copy the full script into crawler.py. Replace the starting address and contact details before running it.

```python import csv import time from collections import deque from datetime import datetime, timezone from urllib.parse import urldefrag, urljoin, urlsplit from urllib.robotparser import RobotFileParser

import requests from bs4 import BeautifulSoup

START = 'https://example.com/' USER_AGENT = 'ExampleAuditBot/1.0 (contact: you@example.com)' MAX_PAGES = 25 DELAY = 2.0 TIMEOUT = (5, 15)

session = requests.Session() session.headers['User-Agent'] = USER_AGENT host = urlsplit(START).hostname queue = deque([(START, '', 0)]) seen = set() robots_cache = {} rows = [] last_request = 0.0

def clean(url): url, _ = urldefrag(url) p = urlsplit(url) if p.scheme not in ('http', 'https') or p.hostname != host: return None if p.username or p.password or p.port not in (None, 80, 443): return None return url

def robots_for(url): origin = f'{urlsplit(url).scheme}://{host}' if origin in robots_cache: return robots_cache[origin] try: r = session.get(origin + '/robots.txt', timeout=TIMEOUT, allow_redirects=False) if r.status_code in (404, 410): result = (True, None) elif r.status_code == 200: parser = RobotFileParser() parser.parse(r.text.splitlines()) result = (True, parser) else: result = (False, None) except requests.RequestException: result = (False, None) robots_cache[origin] = result return result

def allowed(url): ok, parser = robots_for(url) return ok and (parser is None or parser.can_fetch(USER_AGENT, url))

def fetch(url): global last_request current = url for _ in range(6): if not clean(current) or not allowed(current): return None, current, 'Blocked by scope or robots.txt' delay = DELAY _, parser = robots_for(current) if parser is not None: delay = max(delay, parser.crawl_delay(USER_AGENT) or 0) time.sleep(max(0, last_request + delay - time.monotonic())) last_request = time.monotonic() try: r = session.get(current, timeout=TIMEOUT, allow_redirects=False, stream=True) except requests.RequestException as exc: return None, current, type(exc).__name__ if r.status_code in (301, 302, 303, 307, 308): target = clean(urljoin(current, r.headers.get('Location', ''))) r.close() if not target or target == current: return None, current, 'Unsafe or invalid redirect' current = target continue return r, current, '' return None, current, 'Too many redirects'

while queue and len(rows) < MAX_PAGES: original, parent, depth = queue.popleft() url = clean(original) if url is None or url in seen: continue seen.add(url) response, final_url, error = fetch(url) status, ctype = '', '' if response is not None: status = response.status_code ctype = response.headers.get('Content-Type', '') if status == 200 and 'text/html' in ctype.lower(): try: html = response.content for tag in BeautifulSoup(html, 'html.parser').find_all('a', href=True): found = clean(urljoin(final_url, tag['href'])) if found and found not in seen: queue.append((found, final_url, depth + 1)) except requests.RequestException as exc: error = type(exc).__name__ response.close() rows.append({ 'url_original': original, 'url_normalized': url, 'discovered_from': parent, 'crawl_depth': depth, 'http_status': status, 'final_url': final_url, 'content_type': ctype, 'fetched_at': datetime.now(timezone.utc).isoformat(), 'error': error, })

with open('crawl_results.csv', 'w', newline='', encoding='utf-8') as f: fields = ['url_original', 'url_normalized', 'discovered_from', 'crawl_depth', 'http_status', 'final_url', 'content_type', 'fetched_at', 'error'] writer = csv.DictWriter(f, fieldnames=fields) writer.writeheader() writer.writerows(rows) print(f'Saved {len(rows)} results to crawl_results.csv') ```

Run the crawler from the folder containing the script:

``bash python crawler.py ``

If it completes normally, it creates crawl_results.csv in the current working folder. The output reports observed requests and failures, not Google indexing results.

The script assumes a normal, public website operated within your authorized scope. Its exact-host checks do not provide a full network security boundary. They do not defend against DNS changes, every private-network route, malicious servers, or hostile redirects during connection setup. It also does not enforce a total download-size limit, a full-run deadline, or all robots.txt rules. Do not use this beginner code against untrusted targets or sites you do not control.

Start with a Queue and Visited Set

The crawler begins with one URL in a queue. Each queued item contains the URL, the page where it was found, and its crawl depth.

A visited set stores normalized URL values already processed. Before fetching a page, the script checks that set.

Suppose the starting page links to /about and /services. The crawler adds those two URLs to the queue. When it reaches /about, it looks for more internal links.

If /about links back to the starting page, the visited set stops the crawler from processing that same URL again.

The script may still add the same new URL to the queue more than once before processing it. The visited check prevents a later duplicate fetch. For larger crawls, a separate queued set would also reduce duplicate queue entries.

Fetch Pages and Record HTTP Results

Each request uses a stated timeout. The crawler records the returned HTTP status code and content type when a response is available.

It extracts links only from successful HTTP 200 responses with an HTML content type. PDF files, images, and other content types are not parsed for links.

When a request fails, the code stores the exception type in the error field. It then moves to the next queued URL.

HTTP 403, 404, 429, and 500 responses may appear in the report. They do not trigger link extraction. The script does not attempt to bypass access controls or automatically retry rate-limited pages.

Because the script reads HTML into memory, large documents need extra safeguards. A production crawler should also cap response size and total download time.

Extract and Resolve Internal Links

HTML links often use relative addresses. A page at https://example.com/services/ might link to /contact instead of giving the full URL.

Python's urljoin() turns that relative path into an absolute address. Beautiful Soup supplies the href values from HTML anchor tags.

The script accepts HTTP and HTTPS links on the approved hostname. It ignores links using unsupported schemes, including mailto, tel, and javascript.

Some websites store navigation targets in forms, scripts, or data files. This basic parser will miss those destinations unless they also appear as normal HTML anchor links.

Normalize URLs Without Hiding Valid Pages

The code removes URL fragments from the identity used for fetching. A fragment such as #pricing usually points to a section inside the same document rather than a separate HTTP resource.

It deliberately keeps query strings. For example, /products?page=1 and /products?page=2 may show different product lists.

Removing every query string could hide useful pages from the report. Keeping every parameter can also create many URL variations.

The site's guide to crawling URL parameters explains why parameter rules deserve their own review.

The example does not sort query parameters, remove tracking parameters, or merge trailing-slash variants. Such changes require an explicit rule based on the target site.

Enforce Host and Redirect Boundaries

The example stays on the same hostname as its starting URL. A link to another hostname is skipped, even if it belongs to the same business.

Redirects need special care. Requests normally follows them, but this script turns automatic redirects off for page requests.

It inspects each new destination before making another request. A redirect to a different hostname or unsupported port is rejected. The code also limits the redirect chain.

This reduces unexpected page requests, but it is not full protection against server-side request forgery or private-network access. A security-hardened crawler needs DNS and IP checks tied to the actual connection, along with network-level controls.

The simple hostname rule may also miss valid redirects from a non-www host to www. For a known site migration, approve and implement each allowed host explicitly rather than removing the boundary.

Stop at the Limit and Preserve Errors

The crawler stops when it records 25 URL results or runs out of queued URLs. This limit counts rows, including blocked and failed URLs. It is not a count of successful HTML pages.

The example applies a two-second minimum delay between page-request attempts and uses any larger parsed crawl-delay value. It does not implement the parser's request-rate value.

A reliable production crawler should also handle robots.txt caching rules, malformed files, response sizes, session limits, and safer network resolution.

Do not raise the crawl limit merely because the first report contains fewer pages than expected. Find out why those pages were missed first.

Export Crawled Website URLs to CSV

A useful crawl report should show what the script actually observed. Each row should identify the requested URL, where it came from, the response, and any error.

The example code writes nine fields. It does not guess canonical URLs, noindex values, Googlebot crawl events, or index status.

Understand the CSV Columns

Every CSV field has a clear role. This makes it easier to filter failures and compare results from later runs.

The final URL is not a complete redirect history. A richer SEO crawler should record each redirect hop, its status, and its destination.

The timestamp marks the result record, not the exact start of the HTTP request. For most basic reviews, this is enough to tell when the run took place.

Canonical URLs and noindex directives need their own extraction rules. They must not be inferred from HTTP 200 alone.

Completed Example CSV Results

The following rows are fictional examples. They show the kind of outcomes the script can record. They are not results from a live crawl.

The example shows why a blank status matters. It is not the same as HTTP 200 or 404. It means the crawler did not obtain a final HTTP response to record.

Redirect outcomes need care too. The script records the final response and destination, but not every hop. A complete redirect audit requires a longer event record.

Use the CSV for a Basic SEO Review

Open the CSV in a spreadsheet or load it into a Python data tool. Filter the status and error fields to find pages that need attention.

You can group successful HTML pages, missing pages, access problems, and request failures. Review links pointing to those URLs before changing the website.

The file is a crawl record, not an SEO score. It describes this script's requests under its settings. It does not claim that every page exists in the report.

For repeat runs, keep the date, starting URL, code version, and crawl settings with each saved report. Otherwise, a change in the number of results may reflect changed crawler settings rather than a website problem.

Find More URLs with Sitemaps and Scrapy

A link crawler finds URLs reachable from its starting page. A sitemap can reveal other URLs, including some that the crawler did not reach.

Use both sources when broader discovery matters. Google explains that XML sitemaps help search engines discover URLs, but submitting a sitemap does not guarantee crawling or indexing.

Compare Sitemap URLs with Internal Links

Build two sets: URLs listed in a sitemap and URLs found through the crawler. Normalize them with consistent rules before comparing the sets.

For illustration, suppose a fictional website has 120 unique sitemap URLs and 100 unique crawler URLs. Suppose 90 appear in both sets.

The comparison would look like this:

These numbers are an arithmetic example, not measured site data.

A sitemap-only URL is not automatically an orphan page. The crawler may have stopped early, missed JavaScript links, or skipped another valid starting page.

The site's broader guide to crawling a website for all URLs covers how different URL sources fit together.

Use Scrapy SitemapSpider

Scrapy's SitemapSpider reads XML sitemaps and sitemap indexes. It can also use sitemap references from robots.txt.

A small example looks like this:

```python import scrapy

class SitemapAuditSpider(scrapy.spiders.SitemapSpider): name = 'sitemap_audit' allowed_domains = ['example.com'] sitemap_urls = ['https://example.com/sitemap.xml']

def parse(self, response): yield { 'url': response.url, 'status': response.status, } ```

This is a starting example, not a full crawl-audit project. Replace the domain and sitemap with authorized values. The code has not been tested against a live sitemap in this article.

SitemapSpider can organize URL discovery from declared sitemap files. It cannot find URLs that are absent from those files unless you add another source or discovery method.

Control Scrapy Traffic with AutoThrottle

Scrapy offers AutoThrottle settings to adjust request delays based on observed response times.

The documented extension includes a five-second default starting delay and a target concurrency of 1.0. The AutoThrottle extension itself is disabled by default.

A basic configuration can enable it:

``python AUTOTHROTTLE_ENABLED = True AUTOTHROTTLE_START_DELAY = 5.0 AUTOTHROTTLE_MAX_DELAY = 60.0 AUTOTHROTTLE_TARGET_CONCURRENCY = 1.0 ``

These are example settings, not a promise that a site can safely handle a set request rate. Check permissions, robots rules, and server behavior before choosing traffic limits.

Scrapy can help manage a larger project, but it does not make every URL discoverable. Sitemap content, starting points, access rules, and rendering still shape what you find.

Fix Common Python Website Crawling Problems

Most crawler problems become easier to solve once you separate link discovery, HTTP access, rendering, and reporting.

Start with the script's status and error fields. Then inspect the actual page or restriction behind the result. Do not assume that every missing URL has the same cause.

The following table links common symptoms to the next useful check.

A symptom does not prove a single cause. For instance, an empty report could come from a robots block, a bad starting URL, or a failure before useful links were found.

Test the Crawler Before a Larger Run

Use an authorized test site with known links. Start with a small page limit and inspect every output row.

A useful test plan includes the following cases:

1. A homepage with two valid internal links should produce the expected queue entries.
2. A relative URL should resolve against the page where the link appears.
3. A fragment link should not create a second fetch of the same document URL.
4. Distinct query strings should remain distinct under the tutorial's URL rules.
5. A link cycle should not cause endless fetching.
6. A blocked robots.txt path should not be fetched as an ordinary page.
7. An unreachable robots.txt file should cause the conservative policy to stop requests.
8. A redirect to another hostname should be rejected.
9. A non-HTML response should be recorded without HTML link extraction.
10. A failed request should create an error record without pretending it returned HTTP 200.

These are required checks for a stronger deployment, not claims that all ten tests passed. The code received a syntax check, but the full cases need controlled execution.

For deeper Googlebot testing, use the site's website crawlability and error guide. That task is different from testing whether this Python script works.

Check Robots and Redirect Failures Carefully

A robots.txt problem deserves special attention because the crawler may stop before reaching useful pages.

The tutorial rejects robots.txt redirects. Some legitimate sites redirect that file, so this conservative choice can cause false blocks. A more complete crawler can implement the RFC's redirect rules while keeping host and network safety controls.

Robots rules may also be cached. A long-running application must decide when to refresh them under the relevant standard and site needs.

For redirects on normal pages, the script checks the new hostname before another request. It does not keep a complete redirect history or block every private IP path. Those controls belong in a hardened crawler.

If a page returns 403 or 429, do not change user agents, rotate addresses, or bypass barriers just to force a successful fetch. Review the site's allowed access and request policy instead.

Understand What a Python Crawl Cannot Prove

A Python crawler can prove what its own requests observed. It cannot prove that Googlebot visited those URLs or that Google placed them in its search index.

Keep discovery, fetching, indexing eligibility, and index inclusion as separate states. Combining them into one success label can create a misleading SEO report.

Crawled URLs Versus Indexed URLs

A discovered URL is a link the crawler found. A requested URL is one the script attempted to fetch. A fetched URL is one for which it obtained a response.

A page may be fetched but still be unsuitable for indexing. It may contain a noindex directive, be a duplicate, or have other issues.

Google's robots.txt guidance explains that robots.txt is mainly a crawl control. A URL blocked from crawling can still appear in search results through other signals.

Noindex has a different role. It tells supporting search engines not to index a page. A crawler blocked from fetching the page may be unable to read that instruction.

A basic Python CSV should therefore use labels such as fetched, failed, blocked, and discovered. It should not add an indexed label without separate index evidence.

Google's Indexing API Has Limited Eligibility

The official Google Indexing API documentation describes support for pages with JobPosting structured data or BroadcastEvent embedded in VideoObject.

That is not a general submission method for ordinary blog posts, product pages, or every URL found by Python.

A crawler can still help you prepare a URL inventory. For standard website discovery, use the appropriate sitemap and Search Console tools rather than assuming API submission is available for every page.

Check Google Index Status Separately

Google Search Console provides the site's own indexing and inspection reports. Use those reports when you need Google's view of a URL.

The IndexNowFast Google Index Checker is another site-provided option for checking URL index status. Its public page describes the service, but it is not independent proof that any particular URL was indexed.

Keep the data sources clear. A Python crawl result, a third-party check, and Google's own Search Console report may answer different questions or reflect different times.

For an accurate SEO review, save those results in separate fields with their check dates and data sources.

Frequently Asked Questions About Python Website Crawling

These answers cover the main choices and limits that readers face after building their first crawler. The best method still depends on the website's structure and access rules. Start with what the crawler must find, then check how the site serves those pages. A small HTML site may need only Requests and Beautiful Soup. A large site, or one built around JavaScript, may need more tools. The answers below also explain why a crawl report cannot confirm Google's index status.

Can Python Crawl Every Page on a Website?

Not by using one simple link crawler. It can find URLs reachable from its starting points under its settings. Pages may be missing because of weak links, blocked paths, JavaScript, or crawl limits. Compare sitemaps and other authorized sources for wider coverage.

A page absent from the CSV is not proof that it does not exist. First check the crawl limit and allowed hostname. Then review sitemap entries and pages that may have links to it. You may find the missing page without making the crawler more complex.

Is Beautiful Soup Enough to Crawl a Website?

Beautiful Soup parses HTML. It does not fetch pages, follow links, or manage the crawl queue by itself. Combine it with Requests or another HTTP client and add rules for scope, permissions, errors, and limits.

For a small project, that pair is often enough. Requests downloads the HTML. Beautiful Soup finds links in it. Your script then decides which links to visit next. Each tool has a clear job.

Should I Use Scrapy Instead of Requests?

Choose Scrapy when you need a larger, repeatable crawl with framework tools. Requests is often simpler for a small HTML site. Scrapy needs more setup, but its spiders and traffic controls can reduce custom code in larger projects.

Do not switch to Scrapy just because your first script misses URLs. Check the cause first. If links appear only after page scripts run, a normal Scrapy spider may miss them too. The tool must fit the site's actual structure.

Can Python Crawl a JavaScript Website?

Yes, with the right tools. A normal Requests call does not run page JavaScript. If links appear only after scripts run, use a browser-rendering workflow. Check the original HTML first because many JavaScript sites still include crawlable links there.

Browser rendering takes more time and computer resources. It may also load images, fonts, and other files that a plain HTML request avoids. Use it for pages that need it rather than for every URL by default.

How Do I Respect robots.txt in Python?

Read the applicable robots.txt file, apply the rules for your user agent, and check each URL before fetching it. Python's RobotFileParser can help. Handle server failures and unclear responses safely, and follow any other site access rules.

A permitted path does not give you unlimited access. Keep a clear user agent, reasonable request delays, and a firm page limit. If the server starts rejecting or slowing requests, stop and review its rules before continuing.

How Do I Save Website URLs to CSV?

Use Python's built-in csv module. Create a DictWriter with fields matching the data your script collects. Write the header once, then one row for each recorded result. The tutorial script saves its report as crawl_results.csv.

Keep errors in the same report. A missing HTTP status can show that the request never got a final response. Do not replace it with 200. Also save the final redirect URL so you can spot pages that moved.

How Fast Should My Python Crawler Run?

There is no universal safe crawl speed. Site rules, server capacity, response times, and permission all matter. Begin with conservative delays and small limits. Increase traffic only after controlled testing and approval where needed.

The two-second delay in the sample is a teaching setting, not a general rule. A server may need longer pauses. On a site you own, review server logs and load before changing the request rate. A faster crawl is not useful if it causes errors.

Can Python Tell Me Whether a URL Is Indexed by Google?

Not from a normal HTTP request alone. A successful fetch shows that your script obtained a response. Use Google Search Console or another clearly identified index-status source for a separate check. Record the source and date of that result.

Keep these fields apart in your report. Your crawler can record the page's HTTP status and fetch time. Google's index status needs its own evidence. Mixing them can make a crawl report look more complete than it is.

Final Python Crawler Checklist and Next Steps

Before using the report for SEO decisions, check that the crawler stayed within its approved scope and recorded useful results.

A short final review can prevent common mistakes:

• Confirm that the starting website is approved for crawling.
• Check the exact hostname, access rules, and robots.txt behavior.
• Set timeouts, delays, and page limits before each run.
• Make sure relative URLs and fragments are handled as intended.
• Keep meaningful query strings until you have reviewed their role.
• Inspect redirects and error rows instead of looking only at HTTP 200 pages.
• Confirm that every CSV field comes from observed data.
• Test failure cases on an authorized site before scaling up.
• Keep Python crawl results separate from Google indexing reports.

For a small HTML site, the Requests and Beautiful Soup example is a useful first step. Run it on a permitted test site, review the CSV, and compare its URL list with available sitemaps.

For larger jobs, use Scrapy or add stronger controls. A production system may need persistent queues, retries, robots caching, DNS and IP safety, memory limits, and structured logging.

Most of all, report what the crawler truly found. A smaller, well-defined URL list is more useful than a larger list with unknown gaps and misleading status labels.