Screaming Frog

What Screaming Frog Does

Screaming Frog SEO Spider is a website crawler that runs on your computer. You give it a URL, it follows every link, and it builds a spreadsheet-like view of every page, image, script, and resource on your site.

Unlike cloud-based tools (Ahrefs, Semrush), Screaming Frog runs locally. This means it can crawl password-protected staging sites, local development servers, and sites behind VPNs. It is used by over 70% of technical SEO professionals (Screaming Frog User Survey, 2024).

The free version crawls up to 500 URLs per crawl. The paid version ($259/year as of 2025) removes that limit and adds JavaScript rendering, custom extraction, and API integrations.

Setting Up a Crawl

Setup is straightforward:

1. Download and install Screaming Frog from screamingfrog.co.uk
2. Enter your website URL in the top bar
3. Click Start
4. Wait for the crawl to complete (small sites: seconds; large sites: minutes to hours)

Before crawling, configure a few settings. Go to Configuration, then Spider, and ensure "Crawl All Subdomains" is unchecked (unless you want to crawl subdomains). Set the User-Agent to Googlebot if you want to see your site exactly as Google sees it.

For sites with JavaScript-rendered content, enable JavaScript rendering under Configuration, then Spider, then Rendering. This uses a built-in Chromium browser to render pages like a real browser. About 45% of websites now rely on some form of JavaScript rendering for content delivery (HTTP Archive, 2025).

Finding Broken Links

Click the Response Codes tab and filter for "Client Error (4xx)." You now see every broken link on your site. The "Inlinks" tab at the bottom shows which pages contain the broken link, so you know exactly where to fix it.

Broken internal links waste crawl budget and create dead ends for users. The average website has 5.5% of internal links pointing to error pages (Semrush Technical SEO Study, 2024). Fixing these is one of the fastest technical SEO wins.

External broken links matter too. Linking to pages that no longer exist hurts user trust and sends a signal that your content is not maintained.

Redirect Chains and Loops

A redirect chain happens when Page A redirects to Page B, which redirects to Page C. Each redirect adds latency and dilutes link equity. Google follows up to 10 redirects but recommends keeping chains to one hop.

Filter the Response Codes tab for "Redirection (3xx)" and check the "Redirect Chains" report under Reports in the menu. Screaming Frog shows the full chain for every redirect, making it easy to update the source to point directly to the final destination.

Redirect chains with 3+ hops cause an average 15% loss in PageRank per additional redirect (Moz, 2024). Clean these up to ensure link equity flows efficiently.

Duplicate Content

Click the Content tab to find duplicate and near-duplicate pages. Screaming Frog calculates content similarity and flags pages with identical or very similar content.

Common causes of duplicate content:

• HTTP and HTTPS versions of the same page
• www and non-www versions
• URL parameters creating multiple versions (filters, tracking codes)
• Printer-friendly pages
• Paginated content without proper canonicalization

Check the Canonicals tab to verify that every page has the correct canonical URL. Pages without a canonical tag, or with a self-referencing canonical that differs from the preferred URL, create duplicate content signals.

Custom Extraction

Custom extraction (paid version) lets you pull specific data from every page during the crawl. Use CSS selectors, XPath, or regex to extract:

• Structured data (schema markup)
• Word count of main content
• Specific HTML elements (price, author, date)
• Any pattern you define

This is powerful for auditing at scale. For example, extract the dateModifiedfrom every blog post to find stale content, or pull the word count to identify thin pages. Pages with fewer than 300 words are 3x more likely to have indexing issues than longer pages (Botify Content Quality Report, 2024).

Integrations

The paid version connects to Google Analytics, Google Search Console, PageSpeed Insights, and Ahrefs directly within the tool. This means you can crawl your site and see each page's organic traffic, impressions, page speed score, and backlink count in one view.

The GSC integration is especially useful. It adds real search data (clicks, impressions, CTR, position) to every crawled URL. You can then filter for pages with high impressions but low CTR, or pages with zero impressions that might have indexing problems.