Indexability

Go to hints

Indexability relates to the technical configuration of URLs so that they are either Indexable or Not Indexable.

Search engines generally take the stance that any successful URLs (i.e. HTTP status 200) they find should be indexed by default - and they will, in the main, index everything they can find. However, there are certain signals and directives you can give to search engines that instruct them to NOT index certain URLs.

Setting URLs so that they are Not Indexable is a relatively common task, and straightforward to do in most modern CMSs. You might want to set a URL to noindex, for instance, if it is useful to website users, but is not a page that would represent a useful search result (e.g. a 'print' version of a page).

However, indexing signals often get misconfigured, or set up incorrectly, which can result in important URLs not getting indexed. An important thing to note is that if a page is not indexed, it has no chance to generate any organic search traffic.

Robots Directives & Canonicals

There are 2 main ways you can signal to search engines that a page should not be indexed - robots directives and canonicals. Accordingly, Sitebulb's Indexability Hints are split in two to reflect this.

Robots Directives

Sitebulb's Robots Hints deal with the robots.txt file, meta robots tags and the X-Robots-Tag, and how robots directives may impact the way in which URLs are indexed by search engines.

Click through to read more about robots directives, or check out the Robots Hints below.

Directive insights

There are 3 Hints that relate to potential issues caused by the directives themselves, and how they are used in conjunction with internal linking practices.

  1. Has noindex and nofollow directives
  2. Internal Disallowed URLs
  3. URL only has nofollow incoming internal links

Disallowed resources

There are 3 Hints that relate to rendering issues caused for disallowed resource files:

  1. Disallowed image
  2. Disallowed JavaScript file
  3. Disallowed Style Sheet

Multiple robots directives

There are 6 Hints that relate to issues caused by robots directives being specified multiple times:

  1. Mismatched nofollow directives in HTML and header
  2. Mismatched noindex directives in HTML and header
  3. Multiple nofollow directives
  4. Multiple noindex directives
  5. Nofollow in HTML and HTTP header
  6. Noindex in HTML and HTTP header

Canonicals

Sitebulb's Canonical Hints deal with how canonicals impact the way in which URLs are indexed by search engines, and help you unpick canonical issues.

Click through to read more about canonicals and how they are implemented, or check out the Canonical Hints below.

Issues with the canonicalized URL

There are 11 Hints that relate to the canonical URL itself:

  1. Canonical loop
  2. Canonical points to a different internal URL
  3. Canonical points to a disallowed URL
  4. Canonical points to a noindex nofollow URL
  5. Canonical points to a noindex URL
  6. Canonical points to a redirecting URL
  7. Canonical points to a URL that is Error (5XX)
  8. Canonical points to a URL that is Not Found 404
  9. Canonical points to another canonicalized URL
  10. Canonical points to external URL
  11. Canonical URL has no incoming internal links

Conflicting protocol issues

There are 2 Hints that relate to mismatched HTTP/HTTPS canonicals:

  1. Canonical points to HTTP version
  2. Canonical points to HTTPS version

Implementation issues

There are 8 Hints that relate to the implementation of canonicals:

  1. Canonical is a relative URL
  2. Canonical is malformed or empty
  3. Canonical only found in rendered DOM
  4. Canonical outside of head
  5. Canonical tag in HTML and HTTP header
  6. Mismatched canonical tag in HTML and HTTP header
  7. Multiple canonical tags
  8. Multiple, mismatched canonical tags

Pagination issues

There are 3 Hints that relate to pagination and pagination canonicals:

  1. Next/Prev Paginated URL is canonicalized to different URL
  2. Noindex found on rel Next/Prev Paginated URL
  3. Paginated URL missing next/prev canonicals

Ready to try Sitebulb?
Start your free 14 day trial now

Start Free Trial