How to re-audit failed URLs

How to re-audit failed URLs

One of the new features added with version 3.0 was 're-audit failed URLs', a much requested feature. This is a short guide to show you how to use this function to help when auditing websites.

We recently moved some of our Hints content around, and there was a brief period where this resulted in some broken links on the site, which was the perfect opportunity to showcase this new feature.

A straightforward audit of the site shows that we have 4 broken links after the content move:

Broken URLs

These relate to Hints pages that were linked internally, and have now moved to new URLs, before any redirects have been set up:

Internal URLs not found

Since these URLs will still be 'live' links in older (pre-3.0) versions of Sitebulb, we then need to set up 301 redirects for all of these URLs, pointing at their new homes - so that they still work for users who use the old versions of the software.

Once all these redirects are in place, we could simply recrawl the site to check that everything is as we expect. And on a site this size, this would only take a couple of minutes. But imagine if you had a site with 10,000 pages, would you really want to recrawl the whole site in order to check 4 broken links?

That is where the new feature comes in. In order to only re-audit the broken URLs, all we need to do is press the Re-Audit Failed URLs button on the Audit Overview.

Re-audit failed URLs

This will trigger a pop-up confirming how many URLs will be re-audited. In our case we had 4 internal URLs and 1 external URLs that were Not Found, so 5 in total:

Confirm re-audit

Once you click Yes to this, you will be taken to the familiar crawl progress screen, and this time only the failed URLs will be queued up and crawled.

This will complete as normal, and then it will rebuild all the reports, before finally taking you back to your completed audit.

Now when we check our internal URLs, all the 'Not Found' ones have happily disappeared. In this case, they were replaced by redirects, but that's for another guide post.

Sitebulb after crawl

Important things to note

It may not be obvious, but this process does not result in a new audit. So in your project, you don't have 2 audits where previously you had 1 - you will still only have 1. In effect, you have replaced the old audit with this new one.

Since all the reports are rebuilt, this means that the graphs, Hints and data tables will also all be updated - including things like internal links and the Link Equity Score.

Use cases

Most of these ones will be obvious:

  • If you have done some work fixing broken URLs, and want to check if you got them all.
  • If your client claims to have done some work fixing broken URLs, and you want to check if they actually did anything at all!
  • If you have done some work re-instating old content, and want to check if you still have any 404s.
  • If your audit suffered from some timeouts or forbidden URLs, and you want to double check that they are still 'broken.'

Bonus secret use case

This one most definitely is not obvious. When queuing up the failed URLs to re-audit, Sitebulb will also queue in any 'Uncrawled URLs', and crawl those as well.

So, for example, let's say you are crawling a 200,000 URL website, and you get about half-way in and really want to take a look at the data, or the Crawl Map perhaps. Now, you can 'Stop' the audit (top right option on the Crawl Progress screen) so that Sitebulb moves on to creating the audit report.

It will create the report as if the audit has finished, and you can explore the results at this mid-way point. Then, to pick up crawling the site from where you left off, hit the Re-Audit Failed URLs button on the Audit Overview, as above.

This will cause Sitebulb to queue up any failed URLs, along with the 100,000 or so 'Uncrawled' that you had left in your queue, along with any new URLs it discovers along the way. This will allow you to complete the full audit as you originally intended, with all the graphs and Hints based on the updated and complete crawl data.

Eventually we will probably split these out into 2 logical features/options, but for now, this is a nifty little hack you can keep up your sleeve!

Patrick Hathaway

I am a technical SEO, a father and a part-time bread baker. For Sitebulb, I do the marketing, support and 'customer success' (which, among other things, means writing posts like this).

Ready to try Sitebulb?
Start your free 14 day trial now

Start Free Trial