Mastering 4XX Errors: A Comprehensive Guide Using Sitebulb

Can’t believe it’s taken us this long to publish a guide to 4XX errors! Thank you to Chidinma Itsuokor for rectifying this travesty for us.

I remember the first time I had to urgently find statistical data to complete a project. As I searched for information, I found a link perfect for my needs. Sadly, after clicking it, I received a message that said, "404 Not found." I was devastated. Since a reputable blog had referenced the URL, I was convinced the link would be correct. However, no matter how many times I clicked on the link, it returned the same message, "not found."

Now, imagine your users encountering a similar experience on your website. Even though you worked hard to design a seamless user experience, clicking an incorrect link can result in a "404 Not Found" error.

What happens next? They leave, and chances are, they may never return.

A 4XX error—whether a 404 Not Found or a 403 Forbidden—can happen for many reasons, from mistyped URLs to broken redirects.

So, how do you find and fix these errors?

This guide will explain 4XX errors, and how Sitebulb can detect and resolve any error, so you can maintain a user-friendly website.

Contents:

• What are 4XX errors?
• Using Sitebulb to detect 4XX errors
• Prioritizing 4XX errors
• How to fix internal and external 4XX errors
• Designing user-friendly 404 pages
• Case studies: Fixing URL issues with Sitebulb
• From frustration to fixed: keeping your website error-free

What are 4XX errors?

4XX errors are a group of HTTP status codes that tell a user something went wrong while processing the request. Although Google has stated that 4xx errors do not directly affect a website's search rankings, excessive 4xx errors like "404 Not Found" can lead to a loss of valuable link equity and wasted crawl budget.

Examples of 4XX error codes

Below is a table summarizing some of the most common 4XX errors and typical scenarios:

Using Sitebulb to detect 4XX errors

Sitebulb is an intuitive auditing tool that can detect and fix 4XX errors and other issues lurking on your website. This SEO audit tool has gained the trust of many SEO experts for improving their websites. 

Here's an interesting review about this software by Casey M that highlights its ease of use, people-first approach, and issue prioritization.

Now, here’s my 2-step guide for detecting 4XX errors using Sitebulb. 

1. Crawl your website with Sitebulb

1. Download and launch the Sitebulb application and follow the installation guidelines. Next, register your account with Sitebulb. Once this is complete, you will see a dashboard where you can manage all your SEO projects.
2. Click the "Create New Project" button to initiate a crawl. Then, give your project a name, enter the URL you want to crawl and click "Save and Continue." Sitebulb will analyze your site's configuration before allowing you to set up the audit.
   
3. Configure crawl settings: A great feature of the Sitebulb audit tool is its ability to customize crawl settings. To detect 4XX errors, navigate to the "Crawl Configuration" tab and make sure the "HTTP Status" option is enabled so the tool can identify and report 4XX errors.
4. Run the crawl: After configuring your settings, hit the "Start Crawl" button. Sitebulb will scan your website and log every URL it encounters.

2. Leverage Sitebulb Hints to analyze error details

Once the crawl is complete, Sitebulb does something brilliant—it breaks down the data into easy-to-digest insights called "Hints." These Hints allow you to spot and fix errors, including 4XX ones, quickly.

Here’s how you do it:

1. Open the audit generated by Sitebulb for your website. 
2. Navigate to the 'Hints' tab. Here, you can see all the issues flagged during the crawl, such as broken links, redirect chains, and 4XX or 5XX status codes.
   
3. You can filter the Hints by status codes or click the different types of Hints under the All Hint tab; for example, when I click on the Link tab, I can see the status of internal and external links with 4XX errors. 
   
4. Click on the specific Hint and review the details on the URLs affected. For instance, Sitebulb flagged a potential issue with a URL on this website, which showed a link to a non-HTTP protocol. You can see the Hints and suggestions provided to fix this issue. For each Hint, there’s a page on the Sitebulb website with further explanation.
   
5. Apply fixes and re-crawl the website: Once you've made the necessary changes, re-crawl your website with Sitebulb, and then check the updated Hints to confirm that the errors are no longer flagged. 

Prioritizing 4XX errors

Staring at a long list of errors can be overwhelming, but knowing which 4XX errors to address first can save you time and resources. Also, not every 4XX error is equally harmful, but addressing the important ones will ensure your site remains crawlable, user-friendly, and trustworthy. 

Here are four types of pages to consider when prioritizing 4XX errors:

1. High-traffic pages: Prioritize fixing 4XX errors on pages that attract a considerable amount of traffic. These pages are critical for maintaining both your rankings and user engagement.
2. Pages with backlinks: Consider pages with valuable backlinks, because these links help pass link equity and any broken links can lead to a loss of authority.
3. Conversion pages: Remember your conversion pages, like product pages or service pages, are your money pages and should be error-free. Any 4XX errors here can cause missed sales opportunities. Here's a perfect example of how a service page can lead to better revenue. Imagine if you had a 4XX error on this page due to an incorrect redirect; it could lose you a lot of leads.
   
   (Image source: The practical marketer webinar, How To Build an $8 Million Inbound Machine featuring Andy Crestodina by MOZ).
   
   
4. Pages with high user engagement: Focus on fixing 4XX errors on frequently visited pages on your website, “like landing pages”.

How to fix internal and external 4XX errors

1. The first step is to spot where these errors occur within your site. But how do you find those invisible cracks? Use tools like Sitebulb or Google Search Console (check the coverage report section) to crawl your website and see the pages that return 4XX errors that need immediate action.
2. Replace broken links with correct URLs: Once you have identified the broken links (internal or external), the next step is to replace the URLs. Don't ever leave users stranded!
3. Set up 301 redirects for moved pages: Don't let pages vanish into the void when they move. You should instead set up a 301 redirect that tells search engines “that the site has moved, but here is the new address’’. 
   
   The new address should be as relevant as possible to the old page. For example, avoid redirecting users to the homepage when moving a product page. Search engines would interpret this as a soft 404. Instead, redirect them to a similar product page to improve the user experience.
   
   Barry Schwartz, CEO of RustyBrick, quotes Google's John Mueller in his article Google PSA: Do Not 301 Pages that Should 404 To Your Home Page:
   
   “There is no benefit in using a 301 redirect for a page you killed off when you redirect it to a page irrelevant to that 404 page. You should only 301 redirect pages to other pages with a one-to-one relationship and not unrelated ones. So, if you have a page about apples and you decide to get rid of it, redirecting your apple page to your home page is not useful."
4. Remove external links that point to permanently deleted pages. You can contact the site's webmaster containing the broken external link and request its removal. If you don't get a response from the webmaster, you may need to use Google's Disavow tool to prevent any negative impact.

 

Designing user-friendly 404 pages

Even with the best efforts, some 404 errors will still occur. This is where you need to be creative with your 404 pages. Make them user-friendly by offering clear call-to-action options to navigate to other popular content. 

One great example is Nespresso’s 404 page, which features a powerful image of a coffee cup, transforming an error into an opportunity to reinforce its brand identity. As you can see from the image above, the page provides clear navigation options, guiding users back to popular sections and keeping them engaged with the brand.

As Alan Perkins, Managing Director at Qoob, mentioned in a LinkedIn discussion with Gary Illyes, Analyst at Google, it's essential to have "a friendly 404 handler, telling people it's an error and helping them to continue somehow”. He added that if your website has a good search feature, making it prominent on the 404 page can help users quickly find what they want. 

Remember, your 404 page is more than an apology; it is an opportunity to re-engage and retain users on your website.

Case studies: Fixing URL issues with Sitebulb

PeerSpot's Migration with Sitebulb

During PeerSpot’s domain migration, Jake Gronsky, an associate of SEO expert Eli Schwartz, relied on the Sitebulb tool to uncover crawl depth issues and orphaned directories. This led to the discovery of a major growth opportunity: internal linking.

One of the key challenges during the migration was managing 301 redirects, which can quickly turn into redirect chains or loops, causing link equity loss. Sitebulb preemptively flagged thousands of broken redirects, allowing the team to update link paths and avoid serious SEO pitfalls.

This case perfectly highlights how Sitebulb’s auditing tool provides valuable hints and opportunities to enhance user experience on a website by preventing 4XX errors, optimizing internal link structure, and ensuring a seamless transition during complex website migrations.

Read the full PeerSpot case study on Sitebulb.

How Zoopla Tackled SEO Issues Using Sitebulb

Zoopla, a leading property website in the UK with over 60 million visitors monthly, faced challenges in managing millions of pages. They needed a reliable tool to diagnose SEO issues at scale. Using the Sitebulb Cloud crawler, Zoopla was able to identify and fix critical problems efficiently.

According to Robyn Lodge, Zoopla's SEO Manager, a recent Sitebulb crawl highlighted critical issues like duplicate title tags, H1s, and unintended URL generation. These insights allowed the team to act swiftly by creating tickets to address the problems and blocking unnecessary URLs immediately while engineers developed long-term solutions. 

Additionally, the Sitebulb cloud crawler helped Zoopla uncover and address issues with internal linking and no-index logic, contributing to a smoother and more optimized website and making it an indispensable resource for Zoopla's SEO team.

Read the complete Zoopla case study on Sitebulb

To view more case studies from Sitebulb, check here.

From frustration to fixed: keeping your website error-free

When searching for important information, a broken link is the last thing anyone wants to see. So fixing a 4XX error isn't just about improving a technical glitch - it shows your users that you value their time and maintain their trust. With Sitebulb, you have the insights and tools to transform broken links into better experiences.

The next time users visit your website, make sure they have a pleasant experience. From the first click to the final conversion, let their journey be error-free, smooth, and worth their time. 

Ready to improve your site? Try Sitebulb today and keep your users engrossed with your website.

You might also like:

• Beginners Guide to Technical SEO Factors
• Guide to Technical SEO for Ecommerce
• Ultimate Guide to XML Sitemaps for SEO