Crawler Settings

Sitebulb has a range of 'global settings' which act as the default settings for every new audit you start. From the Dashboard, choose the 'Settings' option from the left-hand menu.

Global Settings Option

From here, the 'Crawler' tab allows you to adjust the default User Agent and default Accept-Language, both of which are sent through on HTTP Header requests when Sitebulb crawls each URL.

Global Crawler Settings

You can adjust these values using the drop-down menus, as above. Bear in mind that the values you select here may have an impact on how the server decides to return content to you. If you have trouble crawling a website, such as URLs returning 403 HTTP responses, it is always worth trying to change the user-agent and/or the accept-language values to see if the server treats these requests differently.