Check that essential pages are accessible to crawlers, and that relevant search engine user agents are not disallowed from crawling the site.
Flag any user agents and URL patterns that are disallowed when that's a problem (e.g. valuable pages are not accessible).
Flag the reverse: should any URL patterns be disallowed but are not blocked today?
Flag any disallow rules that appear to be development or staging leftovers.
Flag if any key page-building resources are blocked (file types or directories containing CSS, JavaScript, or image resources).
Review Robots.txt content (written instructions).
Review corresponding Sitebulb Hints.
robots.txt is the first thing crawlers read. A stray disallow rule can hide valuable pages or block the CSS and JavaScript needed to render them, quietly undermining crawling and rendering across the whole site.
Staging disallow rules were carried over to production.
CSS/JS/image directories are blocked, preventing correct rendering.
Introduction to robots.txt by Google
Blocking Special Files in robots.txt by Matt G. Southern (John Mueller)
Sitebulb Robots.txt Validator and Testing Tool
Find, fix and communicate technical issues with easy visuals, in-depth insights, & prioritized recommendations across 300+ SEO issues.
Get all the capability of Sitebulb Desktop, accessible via your web browser. Crawl at scale without project, crawl credit, or machine limits.