Crawler limits

You may wish to limit Sitebulb, in terms of how deep it will crawl, or in terms of how fast it will go. By default, Sitebulb is set up to cater for most sites very well, but occasionally you may find that you need to limit the crawl, and you can do this via the Advanced Settings.

To get to Advanced Settings, you scroll to the bottom of the main Audit setup page and hit the grey Advanced Settings button.

The Limits section is under Crawler -> Limits

The first two options give you various ways to limit the crawler so that it crawls less (or more) pages/content:

• Maximum Crawl Depth - The number of levels deep Sitebulb will crawl (where the homepage is 0 deep, and all URLs linked from the homepage are 1 deep, and so on). This is useful if you have extremely deep pagination that keeps spawning new pages.
• Maximum URLs to Audit - The total number of URLs Sitebulb will Audit - not that this will include external URLs, and page resource URLs. Once it hits this limit, Sitebulb will stop crawling and generate the reports.