Screaming Frog is an SEO tool, it is an ultimate solution for a website’s technical SEO audit. The program crawls and scans a website’s links, images, and codes to collect valuable SEO insights and provide detailed reports. The free version allows us to track up to 1,000 log lines and up to 500 website URLs, but doesn’t include customer support. But the program is super easy-to-use and its limited options are enough for small and medium-sized projects.
In regular crawl mode, the SEO Spider will crawl the subdomain you enter and treat all other subdomains it encounters as external links by default (these appear under the ‘external’ tab). In the licenced version of the software, you can adjust the configuration to choose to crawl all sub domains of a website.
One of the most common uses of the SEO Spider is to find errors on a website, such as broken links, redirects and server errors.
The SEO Spider tool crawls from sub folder path forwards by default, so if you wish to crawl a particular subfolder on your site, simply enter the URI with file path.
For example, if it’s a blog, it might be – https://www.screamingfrog.co.uk/blog/, like our own blog. By entering this directly into the SEO Spider, it will crawl all URI contained within the /blog/ sub directory. As well as crawling a website by entering a URL and clicking ‘Start’, we can switch to list mode and either paste or upload a list of specific URLs to crawl.
This can be particularly useful for site migrations when auditing redirects. If you wish to perform a particularly large crawl,its necessary to increasing the RAM memory allocation in the SEO Spider first. If you receive ‘you are running out of memory for this crawl’ warning, then you will need to save the crawl, increase the RAM allocation, open the crawl and resume the crawl. The number of URLs the SEO Spider can crawl is down to the amount of memory available on the machine and whether it’s allocated.
You May Also Like: