The Sitebulb MCP is live. Bring your crawl data into Claude & ChatGPT. The Sitebulb MCP is live. See how it works

Critical
This Hint requires immediate attention, as the issue may have a serious impact upon crawling, indexing, or ranking.

Check for critical errors

What to check

Confirm the site is, in a general sense, ready for auditing via a crawl. Set up a Sample Crawl, run it with default settings, then look for the critical errors below. These should be resolved before any additional auditing.

Note: for very small sites (under ~10k pages) you can skip the sample crawl and run these checks on a Standard Crawl instead.

  • Flag if the robots.txt is not available at all (4xx or 5xx error).

  • Flag if the entire site, or large important sections of it, are Disallowed for crawling in robots.txt.

  • Flag if the entire site, or large important sections, are set to noindex and/or nofollow in meta robots.

  • Flag if the entire site, or large important sections, are canonicalized to alternative or incorrect URLs.

  • Flag if the entire site, or large important sections, are issuing blanket 5xx or 4xx status codes (site down).

  • Flag if the crawl finalizes after far too few URLs, indicating URLs may not be crawlable.

How to check

  1. Auditing indexable, noindex & nofollow pages (video instructions).

  2. Auditing canonical pages (video instructions).

  3. Review Robots.txt content (video instructions).

Note: if the robots.txt file doesn't exist, the Robots.txt configuration dashboard won't show green arrows, and no content will show up on the Robots.txt tab.

Why this matters

These are site-wide blockers. If any are present, your crawl data will be unreliable and any downstream findings will be based on a broken picture of the site, so they must be caught and fixed first.

When this typically occurs

  • A recent migration or deployment changed robots.txt, canonical, or meta robots rules site-wide.

  • A staging configuration (blanket noindex or disallow) was pushed to production by mistake.

Further reading

None.

Tools

Sitebulb

Contents

    Related Sitebulb hints

    None.

    Sitebulb Desktop

    Find, fix and communicate technical issues with easy visuals, in-depth insights, & prioritized recommendations across 300+ SEO issues.

    • Ideal for SEO professionals, consultants & marketing agencies.

    Sitebulb Cloud

    Get all the capability of Sitebulb Desktop, accessible via your web browser. Crawl at scale without project, crawl credit, or machine limits.

    • Perfect for collaboration, remote teams & extreme scale.
    ;