The Sitebulb MCP is live. Bring your crawl data into Claude & ChatGPT. The Sitebulb MCP is live. See how it works

Low
This Hint is of the lowest significance, and should only be addressed if there are not more serious issues which have not been handled.

Check robots.txt for LLM user agent instructions

What to check

Review whether the site's robots.txt includes instructions for LLM and AI crawler user agents. This is an emerging area with no universally correct answer; the goal is to confirm a deliberate decision has been made rather than leaving it to default behaviour.

  • List the AI user agents to look for.

  • Annotate any that are present and understand what the instructions mean.

  • Confirm which are not present.

  • Present the results either way and confirm it is working as intended.

How to check

Review Robots.txt content (video instructions).

Review Robots.txt content (written instructions).

Look for AI user agents.

Why this matters

AI crawlers increasingly access site content for training and AI search. Whether to allow or block them is a business decision, and this check simply ensures that decision has been made consciously rather than by default.

When this typically occurs

  • robots.txt was written before AI crawlers were a consideration.

  • No one has decided the site's stance on AI training data or AI search.

Further reading

Tools

Sitebulb

Contents

    Related Sitebulb hints

    None.

    Sitebulb Desktop

    Find, fix and communicate technical issues with easy visuals, in-depth insights, & prioritized recommendations across 300+ SEO issues.

    • Ideal for SEO professionals, consultants & marketing agencies.

    Sitebulb Cloud

    Get all the capability of Sitebulb Desktop, accessible via your web browser. Crawl at scale without project, crawl credit, or machine limits.

    • Perfect for collaboration, remote teams & extreme scale.
    ;