Using Included URLs is a method for restricting the crawler, and this method allows you to restrict the crawl to only the URLs or directories specified.
As an example, if I were crawling the Sitebulb website and only wanted to crawl the 'Product' pages, I would simply add the line:
/product/
To get to Advanced Settings, you scroll to the bottom of the main Audit setup page and hit the grey Advanced Settings button.
The Included URLs section is under URLs -> Included URLs
It is worth noting a couple of things:
- Excluded URLs over-ride Included URLs, so ensure your rules do not clash.
- Your Start URL must contain at least one link to an Included URL, otherwise the crawler will simply crawl 1 URL and then stop.