I didn’t need an SEO crawler. I needed a list of every PDF and external link.

Websites quietly accumulate documents and outbound links. Policies get replaced but remain accessible. Forms move. Partner resources disappear. PDFs survive multiple redesigns without anyone being entirely sure where they are linked from.

The practical question is not simply what links exist.

During a migration or content audit, an isolated list of links is only half useful. The actionable question is: which page contains each link?

That source-to-destination relationship tells an editor exactly where a document needs to be reviewed, where an obsolete partner link appears, and which pages will need updating if an asset moves.

If we remove, replace or move this document, which pages do we need to update?

The obvious option was bigger than the job.

Screaming Frog is an excellent SEO crawler. Its free edition is limited to 500 URLs and restricts parts of the wider crawling and configuration feature set.

For this audit, buying a complete SEO crawling suite felt disproportionate. The requirement was not dozens of reports, integrations or technical SEO diagnostics. It was a focused, repeatable export containing:

• The page where a link was found
• The PDF or external URL it points to

So I built a small tool that does exactly that.

What the Sitemap Link Scraper does.

The scraper is a local Python web application. Give it a public XML sitemap and choose one of two audit modes:

It follows sitemap indexes, visits each listed page, reads its links, resolves relative URLs and removes duplicate results. The output is deliberately simple:

That CSV can be handed directly to a content editor, project manager or developer. It is useful for migration planning, document inventories and link remediation without requiring them to interpret a crawler interface.

Designed as a working utility.

The scraper runs through a small browser interface rather than requiring command-line knowledge. While a crawl is running, it reports progress and skips pages it cannot retrieve instead of abandoning the whole job.

Results are held in memory and downloaded directly as a CSV. The tool can be rerun whenever the website changes, producing the same understandable output each time.

Deliberate limits and safeguards.

The utility is intended for authorised audits of public websites, not for bypassing authentication or crawling private systems. It therefore includes explicit controls:

• Only public HTTP and HTTPS destinations are accepted
• Private, loopback and reserved network addresses are rejected
• Redirect destinations are checked rather than trusted automatically
• Response sizes, redirect chains and request times are limited
• Sitemaps containing more than 10,000 URLs are rejected

This is not intended to replace Screaming Frog.

It replaces one narrow workflow for which a full SEO crawler can be unnecessary: turning a sitemap into a clean, traceable inventory of documents or outbound links.

That focus is the point. Small operational tools are often most valuable when they remove one stubborn piece of manual work without introducing another large platform to own and maintain.