Screaming Frog SEO Spider is a desktop application that crawls your website and identifies technical SEO issues. This guide covers how to run your first crawl, how to read the results, and which issues to prioritise fixing.
Step 1 — Download and Install
Download Screaming Frog from screamingfrog.co.uk. It is available for Windows, Mac, and Ubuntu. The free version installs without any account or licence key — just download and run.
System requirements: The more RAM your machine has, the larger the crawls you can run. Screaming Frog recommends at least 8GB RAM for crawling medium-sized sites. For very large sites (100K+ URLs), 16GB or more is advisable.
Step 2 — Run Your First Crawl
Open Screaming Frog, type your website URL into the search bar at the top, and click Start. Screaming Frog will begin crawling your site, following internal links and building a complete picture of your site’s structure.
Before starting, check these settings:
- Configuration → Spider → check “Crawl all subdomains” if your site spans multiple subdomains
- Configuration → Spider → Limits — set a crawl limit if you want to test on a sample first
- Configuration → User-Agent — Screaming Frog defaults to its own crawler user agent; you can switch to Googlebot to simulate how Google sees your site
For a typical site with a few hundred pages, the crawl will complete in a few minutes. Larger sites take longer depending on server response times.
Step 3 — Understanding the Interface
Once the crawl completes, data is displayed across several tabs:
- Internal — all internal URLs found, with status codes, titles, descriptions, and metrics
- External — all external links found on your site
- Images — all images, including those missing alt text
- CSS, JavaScript, Fonts — resource files
- Sitemaps — sitemap data
The most useful tab for auditing is Internal. Sort by Status Code to immediately see all broken pages (404), server errors (500), and redirect chains (301/302).
Step 4 — The Key Issues to Look For
Broken links (404 errors)
Filter the Internal tab by Status Code 404 to find all broken pages. These need to be either restored or redirected to a relevant live page. Broken internal links waste crawl budget and create a poor user experience.
Redirect chains
Filter by Status Code 301 or 302 and look for chains — where page A redirects to page B which redirects to page C. Each hop in a redirect chain loses a small amount of link equity and slows page load. Fix chains by pointing the original URL directly to the final destination.
Missing or duplicate meta titles
Click the Page Titles tab. Filter by “Missing”, “Duplicate”, “Too Long” (over 60 characters), or “Too Short” (under 30 characters). Every page should have a unique, descriptive title within the recommended length.
Missing or duplicate meta descriptions
Click the Meta Description tab and apply the same filters. While meta descriptions are not a direct ranking factor, they affect click-through rate from search results.
Missing alt text on images
Click the Images tab and filter by “Missing Alt Text”. Every meaningful image should have descriptive alt text for accessibility and SEO.
Duplicate content
Click the Content tab and filter by “Duplicate”. Duplicate page titles, descriptions, or content can confuse Google about which page to rank. Each page should be unique.
Step 5 — Using the Overview Panel
The right-hand panel in Screaming Frog shows details for any selected URL. Click any URL in the main table to see its inlinks (pages linking to it), outlinks (pages it links to), and all associated data in one view. This is useful for understanding the internal link structure around any specific page.
Step 6 — Exporting Data
All data in Screaming Frog can be exported to CSV via Bulk Export in the top menu. Key exports for a standard audit:
- Bulk Export → Response Codes → Client Error (4xx) — all broken pages
- Bulk Export → Response Codes → Redirection (3xx) — all redirects
- Bulk Export → Page Titles → Missing — pages without titles
- Bulk Export → Meta Description → Missing — pages without descriptions
- Bulk Export → Images → Missing Alt Text — images without alt text
Frequently Asked Questions
How long does a Screaming Frog crawl take?
Crawl speed depends on your site size and server response times. A 100-page site typically takes 1-2 minutes. A 10,000-page site may take 30-60 minutes. Screaming Frog crawls at a default rate that respects server load — you can increase or decrease the crawl speed in Configuration → Speed.
What is the best way to use Screaming Frog for free?
Configure Screaming Frog to exclude images, CSS, and JavaScript files from the crawl count (Configuration → Spider → Limits → uncheck non-HTML file types). This maximises how many HTML pages you can audit within the free 500 URL limit.
Can Screaming Frog crawl JavaScript websites?
Yes, but only on the paid version (£199/year). JavaScript rendering must be enabled in Configuration → Spider → Rendering. The free version crawls HTML only and may miss content loaded dynamically by JavaScript.
How do I find all 404 errors with Screaming Frog?
After completing a crawl, click the Response Codes tab at the top, then use the filter dropdown on the right to select “Client Error (4xx)”. All 404 and other client error pages will be displayed. You can export this list via Bulk Export → Response Codes → Client Error (4xx).
Does Screaming Frog work on Mac?
Yes. Screaming Frog is available for Windows, Mac (Intel and Apple Silicon), and Ubuntu Linux. Download the appropriate version from the Screaming Frog website.
“`html
How to Identify Broken Links with Screaming Frog
Short answer: Use Screaming Frog’s “Response Codes” feature to find broken links.
To identify broken links on your website, start by running a crawl of your site with Screaming Frog. Once the crawl is complete, navigate to the “Response Codes” tab, where you can filter the results to show only the “404 Not Found” errors. This will give you a clear view of all the pages that are returning a broken link response. Fixing these broken links is essential, as they can negatively impact user experience and your site’s SEO performance.
After identifying the broken links, you can take steps to rectify the situation. This may involve updating the link to direct users to the correct page, redirecting the broken link to a relevant page, or even removing the link altogether if it’s no longer necessary. Regularly checking for broken links using Screaming Frog can help maintain the integrity of your website and improve its overall SEO health.
Utilizing Screaming Frog for Meta Data Analysis
Short answer: Analyze your site’s meta titles and descriptions using Screaming Frog.
Screaming Frog is a powerful tool for analyzing meta data, such as titles and descriptions, which are critical for SEO. After crawling your website, head to the “Page Titles” and “Meta Description” tabs. Here, you can quickly identify issues like missing, duplicate, or overly long meta titles and descriptions. A well-optimized meta title and description not only helps search engines understand your content but also influences click-through rates from search results.
Once you’ve identified areas for improvement, you can systematically update your meta data to align with SEO best practices. Aim for unique titles and descriptions for each page, keeping them concise and relevant to the content. By regularly using
| Feature | Details |
|---|---|
| Starting Price | See pricing section above |
| Free Plan | Check official site for current offer |
| Best For | See verdict section |
Bottom Line
This tool offers genuine value for the right use case. Check current pricing on the official site before committing to a plan.
