Work for an agency? Our next webinar is for you. Register now.

Dealing with Rendered issues?

Crawl your website with Sitebulb for 300+ tech SEO checks

Try for Free
Critical This Hint requires immediate attention, as the issue may have a serious impact upon crawling, indexing or ranking. Issue This Hint represents an error or problem that needs to be fixed.

Noindex only in the HTTP response HTML

This means that the URL in question contains a noindex in the response HTML, and not in the rendered HTML.

Why is this important?

Rendering webpages is a resource intensive task, and it takes significantly longer than simply grabbing source (response) HTML content.

This is why Google essentially crawl URLs in a two-stage process: their 'first look' is of the HTML response, then they render the page and have a 'second look' at the rendered HTML, then they update the index based on what they found in the rendered HTML.

Google rendering

The 'Processing' stage is the most complex to get your head around, because things are happening in parallel. It is also necessarily more complicated, because Google are making decisions on which pages to render based on the result of the meta robots check.

Specifically, from their documentation Understand the JavaScript SEO basics;

"Googlebot queues all pages for rendering, unless a robots meta tag or header tells Googlebot not to index the page."

Essentially what this means is: noindex = no render

So this means that these pages will not get queued for rendering, and therefore Google will not see in the rendered HTML that the noindex was not present, and the page will not get indexed.

Additionally, this mismatch means it is unclear whether the page should contain noindex or not. Left unchecked, these pages will remain noindex - even if the rendering process was supposed to make them indexable.

What does the Hint check?

This Hint will trigger for any internal URL that contains a noindex robots directive in the response HTML but not in the rendered HTML.

Examples that trigger this Hint

This Hint will trigger for any URL that has meta robots=noindex in the <head> of the response HTML, that is not present in the rendered HTML.

For example, if the response HTML looked like this:

<!doctype html>
<html lang="en">
<head>
<meta name="robots" content="noindex,follow">
...
</head>
<body>...</body>
</html>

And the rendered HTML looked like this:

<!doctype html>
<html lang="en">
<head>
<meta name="robots" content="index,follow">
...
</head>
<body>...</body>
</html>

How do you resolve this issue?

This Hint is marked 'Critical' as it represents a fundamentally breaking issue, which may have a serious adverse impact upon organic search traffic. It is strongly recommended that Critical issues are dealt with as a matter of high priority.

You will first need to establish the desired state for these pages - if they should be noindex or not. Once this has been clarified, ensure that whatever robots directives appear in the HTML response are also the same after JavaScript rendering.

Further Reading

Sitebulb Desktop

Find, fix and communicate technical issues with easy visuals, in-depth insights, & prioritized recommendations across 300+ SEO issues.

  • Ideal for SEO professionals, consultants & marketing agencies.

Sitebulb Cloud

Get all the capability of Sitebulb Desktop, accessible via your web browser. Crawl at scale without project, crawl credit, or machine limits.

  • Perfect for collaboration, remote teams & extreme scale.