Authentication for crawling staging sites

Some websites utilize authentication, meaning that you need to login using a username/password in order to access the page content. Most often you will encounter this when dealing with dev sites or staging servers.

There are two different ways which this can be implemented:

  1. HTTP authentication (the server will 'challenge' a client request and prompt the client to provide authentication information, so you don't even get to access the page until you provide authentication).
  2. Forms-based authentication (the page will respond with a 200 status and you can access the page, but will include an embedded form which requires authenticating before you can access the full page content).

Sitebulb allows authentication via either of these options, however they are handled in different places in the tool. 

How to crawl sites with HTTP authentication

For HTTP authentication, you will need to add the authentication credentials via the Advanced Settings.

To get to Advanced Settings, you scroll to the bottom of the main Audit setup page and hit the grey Advanced Settings button.

The Authentication section is under HTTP -> Authorization. On the Authorization tab you will see a checkbox for HTTP Authentication, tick this and enter your HTTP Username and HTTP Password. When you start your audit Sitebulb will use these details to login into http authentication page.


Here is a quick example of what this looks like in practice:

How to crawl sites with forms-based authentication

For forms-based authentication, you actually do not add the authentication credentials via the Advanced Settings. This is because the Start URL returns with a 200 status, so Sitebulb does not 'know' there is a form involved - you need to tell Sitebulb that this is the case.

Start a new project and check the Forms authentication required checkbox. This will open a small browser window taking you to the site you are going to be crawling.

Navigate to your login form, if you need to change the URL of where your login form is you can change the URL at the top of the window. Enter your site's login details. Once you have successfully logged in click the Add Authentication button at the bottom of the page.