How to crawl sites with HTTP authentication

Do you need to crawl a website with http authentication (not to be confused with form authentication)? Sitebulb makes it a breeze!

Start a new crawl/project, enter all your sites details – wait while the domain validates… press Save and Continue. This will take you to the Audit setup page, scroll all the way to the bottom of the page and click the Advanced Settings button.

At the top of the page you will see 3 tabs Crawl, URLs and HTTP. Click the HTTP tab – to reveal some more tabs below, click the Authorization tab. On the Authorization tab you will see a checkbox for HTTP Authentication click it and enter your HTTP Username and HTTP Password. When you start your audit Sitebulb will use these details to login into http authentication page.