JavaScript SEO for AI Search: Tutorial with Sam Torres & Tory Gray. JavaScript SEO for AI Search: Sam & Tory. Sign up now!
- Home
- Hints
- Indexability
- Mismatched noindex directives in HTML and header
Mismatched noindex directives in HTML and header
This means that the URL in question has a noindex directive in both the HTML and in the HTTP header, where the directives do not match.
Why is this important?
This means that one location uses 'index' and the other uses 'noindex'. The net result of this is that the page would remain 'noindex', even if efforts had been made otherwise. Google clearly states that if you have multiple directives which conflict, they will select the option that is most restrictive (and it is likely that the other search engines follow suit).
It is considered best practice to only specify robots directives once on any given URL. This is because doing it multiple times makes the configuration more open to error, as is the case in this scenario.
You can avoid such catastrophic futures by only specifying robots directives in one location.
What does the Hint check?
This Hint will trigger for any internal URL which contains mismatched index/noindex directives in the HTML and in the HTTP header.
Examples that trigger this Hint
The Hint would trigger for any URL that had both of the following;
Meta noindex in the <head>,
<!doctype html>
<html lang="en">
<head>
<title>example</title>
<meta name="robots" content="noindex,nofollow">
...
</head>
<body>...</body>
</html>AND an index directive in the HTTP header:
HTTP/... 200 OK
...
X-Robots-Tag: index,nofollowSimilarly, the Hint would also trigger for the inverse;
Meta index in the <head>,
<!doctype html>
<html lang="en">
<head>
<title>example</title>
<meta name="robots" content="index,nofollow">
...
</head>
<body>...</body>
</html>AND a noindex directive in the HTTP header:
HTTP/... 200 OK
...
X-Robots-Tag: noindex,nofollowHow do you resolve this issue?
This is a scenario where the robots directives are conflicting, and ultimately are completely broken. To resolve the issue, you will first need to determine the 'correct' directive, then go and fix the incorrect one so that the directives match.
Following on from this, to help avoid such issues again in the future, it would also be prudent to adjust the page template so that it only uses one method to set robots directives.
To do this you may need developer help, as you will need to adjust page templates, plugins or HTTP headers - removing the duplication so that robots directives are only defined once.
Sitebulb Desktop
Find, fix and communicate technical issues with easy visuals, in-depth insights, & prioritized recommendations across 300+ SEO issues.
- Ideal for SEO professionals, consultants & marketing agencies.
Sitebulb Cloud
Get all the capability of Sitebulb Desktop, accessible via your web browser. Crawl at scale without project, crawl credit, or machine limits.
- Perfect for collaboration, remote teams & extreme scale.