Crawling subdomains

By default, Sitebulb will not crawl subdomain URLs it encounters, and will effectively treat them as external URLs. You can control how Sitebulb treats subdomain URLs in the Advanced Settings.

To get to Advanced Settings, you scroll to the bottom of the main Audit setup page and hit the grey Advanced Settings button.

The Subdomains section is under Crawler -> Subdomains

Advanced Settings - Subdomains

The default option is for the top box to be ticked, 'Check Subdomain Link Status' - this means that subdomain URLs will be treated as external URLs, and Sitebulb will run a HTTP status check on any it encounters. This allows you to find any broken or redirected subdomain links. If you untick this box, Sitebulb will not run the status check on subdomains links.

How to crawl all subdomain URLs

The second option is a bit more involved, 'Treat Subdomains as Internal' - this means that any subdomain URLs that Sitebulb encounters will be treated as internal URLs. On other words, ticking this box is 'how to crawl all subdomain URLs on a website'. Sitebulb will run all the normal 'internal checks' on these URLs, including internal link calculations.

When you do tick the box, the top tickbox will disappear (because it is now redundant) and you will be faced with the option of explicitly adding the subdomains, one per line in the box underneath.

If you tick the box and don't add any subdomains into the box, ALL subdomain URLs will be crawled. However if you do add some subdomains to the box, ONLY those subdomains will be crawled.

Simply enter the subdomains, one per line. In the example below, Sitebulb would crawl and and treat those subdomains as internal (but NOT or any other subdomain).

Enter subdomains

When you come to look at the completed audit results, you will see that URLs from subdomains will now appear in the 'Internal URLs' lists, and they will have URL Rank (UR) scores.

Subdomains in internal