Work for an agency? Our next webinar is for you. Register now.

Dealing with Indexability issues?

Crawl your website with Sitebulb for 300+ tech SEO checks

Try for Free
Insight This Hint is neither an issue nor an opportunity, and is brought to your attention as it may provide a useful avenue of investigation.

Has noindex and nofollow directives

This means that the URL in question has both noindex and nofollow directives.

Why is this important?

If a URL is noindex,nofollow, this means that search engines are being instructed not to include the URL in their index, and to not schedule and crawl any of the links found in the HTML (similarly, crawler software like Sitebulb will not follow these links, unless 'respect robots' is unchecked in the crawl settings).

Using these type of robots directives is a common way to control what content search engines can crawl and index (e.g. a user login area). As such, there is potentially nothing wrong with these URLs, Sitebulb is simply notifying you that they are there.

What does the Hint check?

This Hint will trigger for any internal URL which has noindex,nofollow attributes, either in the HTML or in the HTTP header.

Examples that trigger this Hint

The Hint would trigger for any URL that had the following:

Meta noindex,nofollow in the <head>,

<meta name="robots" content="noindex,nofollow">

OR noindex,nofollow in the HTTP header:

HTTP/... 200 OK

...
X-Robots-Tag: noindex,nofollow

Why is this Hint marked 'Insight'?

This Hint is an 'Insight', which means there isn't necessarily any action that needs to be taken - the Hint is intended to alert your attention to something, rather than flagging up an issue that needs fixing. Setting URLs as noindex,nofollow is very common, and there is often nothing wrong in this situation.

However it is worth double checking that there are no URLs using these directives that you actually want to be properly crawled an indexed. If there are, then the solution for this issue would begin with removing the offending directives.

Sitebulb Desktop

Find, fix and communicate technical issues with easy visuals, in-depth insights, & prioritized recommendations across 300+ SEO issues.

  • Ideal for SEO professionals, consultants & marketing agencies.

Sitebulb Cloud

Get all the capability of Sitebulb Desktop, accessible via your web browser. Crawl at scale without project, crawl credit, or machine limits.

  • Perfect for collaboration, remote teams & extreme scale.