Excluded URLs

Using Excluded URLs is a method for restricting the crawler, and this method allows you to specify URLs or entire directories to avoid.

Any URL that matches the excluded list will not be crawled at all. This also means that any URL only reachable via an excluded URL will also not be crawled, even if it does not match the excluded list.

As an example, if I were crawling the Sitebulb website and wanted to avoid all the 'Product' pages, I would simply add the line:

To get to Advanced Settings, you scroll to the bottom of the main Audit setup page and hit the grey Advanced Settings button.

The Excluded URLs section is under URLs -> Excluded URLs

Excluded URLs