Confirm the site is, in a general sense, ready for auditing via a crawl. Set up a Sample Crawl, run it with default settings, then look for the critical errors below. These should be resolved before any additional auditing.
Note: for very small sites (under ~10k pages) you can skip the sample crawl and run these checks on a Standard Crawl instead.
Flag if the robots.txt is not available at all (4xx or 5xx error).
Flag if the entire site, or large important sections of it, are Disallowed for crawling in robots.txt.
Flag if the entire site, or large important sections, are set to noindex and/or nofollow in meta robots.
Flag if the entire site, or large important sections, are canonicalized to alternative or incorrect URLs.
Flag if the entire site, or large important sections, are issuing blanket 5xx or 4xx status codes (site down).
Flag if the crawl finalizes after far too few URLs, indicating URLs may not be crawlable.
Note: if the robots.txt file doesn't exist, the Robots.txt configuration dashboard won't show green arrows, and no content will show up on the Robots.txt tab.
These are site-wide blockers. If any are present, your crawl data will be unreliable and any downstream findings will be based on a broken picture of the site, so they must be caught and fixed first.
A recent migration or deployment changed robots.txt, canonical, or meta robots rules site-wide.
A staging configuration (blanket noindex or disallow) was pushed to production by mistake.
None.
Sitebulb
Find, fix and communicate technical issues with easy visuals, in-depth insights, & prioritized recommendations across 300+ SEO issues.
Get all the capability of Sitebulb Desktop, accessible via your web browser. Crawl at scale without project, crawl credit, or machine limits.