JavaScript SEO for AI Search: Tutorial with Sam Torres & Tory Gray. JavaScript SEO for AI Search: Sam & Tory. Sign up now!
Nofollow only in the HTTP response HTML
This means that the URL in question contains a nofollow in the response HTML, and not in the rendered HTML.
Why is this important?
Rendering webpages is a resource intensive task, and it takes significantly longer than simply grabbing source (response) HTML content.
This is why Google essentially crawl URLs in a two-stage process: their 'first look' is of the HTML response, then they render the page and have a 'second look' at the rendered HTML, then they update the index based on what they found in the rendered HTML.
The 'Processing' stage is the most complex to get your head around, because things are happening in parallel. It is also necessarily more complicated, because Google are making decisions on which pages to render based on the result of the meta robots check.
During this stage, they are parsing the response HTML, looking for followed links, and then passing these URLs back into the crawl queue - this occurs before rendering has happened.
Important elements, such as meta robots tags should be consistent between the response and rendered version of the HTML. When the nofollow is present in the response but not the rendered version of the page, the mismatch makes it unclear whether the page should contain nofollow or not.
If links are nofollow in the response HTML, this may slow down how quickly Google finds and indexes the linked URLs, even if this was not the intention.
Additionally, some LLMs do not yet render JavaScript, so the meta robots present in the response HTML could determine which content is ingested and surfaced in AI-powered results.
What does the Hint check?
This Hint will trigger for any internal URL that contains a nofollow robots directive in the response HTML but not in the rendered HTML.
Examples that trigger this Hint
This Hint will trigger for any URL that has meta robots=nofollow in the <head> of the response HTML, that is not present in the rendered HTML.
For example, if the response HTML looked like this:
<!doctype html>
<html lang="en">
<head>
<meta name="robots" content="index,nofollow">
...
</head>
<body>...</body>
</html>And the rendered HTML looked like this:
<!doctype html>
<html lang="en">
<head>
<meta name="robots" content="index,follow">
...
</head>
<body>...</body>
</html>How do you resolve this issue?
You will first need to establish the desired state for these pages - if they should be nofollow or not. Once this has been clarified, ensure that whatever robots directives appear in the HTML response are also the same after JavaScript rendering.
Further Reading
Sitebulb Desktop
Find, fix and communicate technical issues with easy visuals, in-depth insights, & prioritized recommendations across 300+ SEO issues.
- Ideal for SEO professionals, consultants & marketing agencies.
Sitebulb Cloud
Get all the capability of Sitebulb Desktop, accessible via your web browser. Crawl at scale without project, crawl credit, or machine limits.
- Perfect for collaboration, remote teams & extreme scale.