Find us at BrightonSEO 25/26 April - STAND 18! Fill your swag bag!

How to crawl in list mode

By default Sitebulb is set to 'crawl' a website from the start URL you supply - which means finding all the links on the page and following these in turn. However, in some circumstances you may wish to only crawl a specific set of URLs on a website without following the links.

In such cases, you would need to utilise Sitebulbs's 'list mode.'

Selecting List Mode from the Crawl Sources

When you first open up the audit setup screen, the left hand side will have 'Audit Data' selected, and the options will show on the right hand side. There are lots of Audit Data options, so we have a dedicated documentation page for this.

Audit Settings

Select the 'Crawl Sources' option from the left hand menu. This is the only audit setting in Sitebulb that is not optional. Sitebulb needs at least one crawl source, otherwise it cannot crawl!

The default setting is for Sitebulb to crawl the website, so this will always be ticked by default. However it can be configured to also crawl XML Sitemap URLs, and/or a provided URL List.

Crawl Sources

To get Sitebulb to 'crawl' based on a list, check the 'URL List' option. To add a URL List, simply upload a .csv or .txt file from your local computer.

It isn't strictly crawling, as links from the pages will not be followed, but the data will be collected and analysed for all URLs contained in the list. Typically URL Lists are used when you DON'T also crawl the website, and are used to crawl a specific area or section of the site. If you wish to do this, make sure to uncheck the 'Crawl Website' option.

URL List Crawl Source

Important note

Please note that Sitebulb will only crawl URLs that match the subdomain of the start URL provided (so you can't just upload a massive list of URLs from lots of different sites).