Review whether the site's robots.txt includes instructions for LLM and AI crawler user agents. This is an emerging area with no universally correct answer; the goal is to confirm a deliberate decision has been made rather than leaving it to default behaviour.
List the AI user agents to look for.
Annotate any that are present and understand what the instructions mean.
Confirm which are not present.
Present the results either way and confirm it is working as intended.
Review Robots.txt content (video instructions).
Review Robots.txt content (written instructions).
AI crawlers increasingly access site content for training and AI search. Whether to allow or block them is a business decision, and this check simply ensures that decision has been made consciously rather than by default.
robots.txt was written before AI crawlers were a consideration.
No one has decided the site's stance on AI training data or AI search.
AI/LLM User-Agents: Blocking Guide by Robotstxt.com
Should You Block Google Extended by Marie Haynes
How Do AI Bots Crawl the Web & What Does It Mean for Your Site? by Tory Gray
Sitebulb
Find, fix and communicate technical issues with easy visuals, in-depth insights, & prioritized recommendations across 300+ SEO issues.
Get all the capability of Sitebulb Desktop, accessible via your web browser. Crawl at scale without project, crawl credit, or machine limits.