The Sitebulb MCP is live. Bring your crawl data into Claude & ChatGPT. The Sitebulb MCP is live. See how it works

High
This Hint is very important, and definitely warrants attention.

Review robots.txt instructions

What to check

Check that essential pages are accessible to crawlers, and that relevant search engine user agents are not disallowed from crawling the site.

  • Flag any user agents and URL patterns that are disallowed when that's a problem (e.g. valuable pages are not accessible).

  • Flag the reverse: should any URL patterns be disallowed but are not blocked today?

  • Flag any disallow rules that appear to be development or staging leftovers.

  • Flag if any key page-building resources are blocked (file types or directories containing CSS, JavaScript, or image resources).

How to check

Review Robots.txt content (written instructions).

Review corresponding Sitebulb Hints.

Why this matters

robots.txt is the first thing crawlers read. A stray disallow rule can hide valuable pages or block the CSS and JavaScript needed to render them, quietly undermining crawling and rendering across the whole site.

When this typically occurs

  • Staging disallow rules were carried over to production.

  • CSS/JS/image directories are blocked, preventing correct rendering.

Further reading

Tools

Sitebulb Robots.txt Validator and Testing Tool

Sitebulb Desktop

Find, fix and communicate technical issues with easy visuals, in-depth insights, & prioritized recommendations across 300+ SEO issues.

  • Ideal for SEO professionals, consultants & marketing agencies.

Sitebulb Cloud

Get all the capability of Sitebulb Desktop, accessible via your web browser. Crawl at scale without project, crawl credit, or machine limits.

  • Perfect for collaboration, remote teams & extreme scale.
;