Find us at BrightonSEO 25/26 April - STAND 18! Fill your swag bag!

Global crawler settings

Sitebulb has a range of 'global settings' which act as the default settings for every new audit you start. From the top navigation menu, choose the 'Settings' option.

Global settings menu

From here, the 'Crawler' tab allows you to adjust the following things the default crawler type so you can switch between HTML & Chrome crawling you can also select the default device type either Mobile or Desktop, the default User Agent and default Accept-Language, both of which are sent through on HTTP Header requests when Sitebulb crawls each URL.

Global crawler settings in Sitebulb

You can adjust these values using the drop-down menus, as above. Bear in mind that the values you select here may have an impact on how the server decides to return content to you. If you have trouble crawling a website, such as URLs returning 403 HTTP responses, it is always worth trying to change the user-agent and/or the accept-language values to see if the server treats these requests differently.