Sitebulb Release Rants

It's that time of year again. One last Sitebulb update, full of bug fixes and festive cheer.

Bug Fixes

• The organic search report had completely stopped showing data. Like, at all. The data was there in the audit, Sitebulb just did not want to show it. As soon as we saw this issue, we knew it was a problem - benefits of a classical education - and we knew we had to get a hotfix out pronto.

• Code Coverage report was making some audacious claims. For example, it reckoned there was 1GB of unused code in a file that was only 8KB in total. Err... that isn't how maths works. Used must be less than or equal to total (not 125,000 TIMES the total). Come out to the coast, we'll get together, have a few laughs...

• On a number of deeper reports, when trying to view 'referencing URLs', Sitebulb was showing a SQLite error. If you're not completely sure what this means, it means that something went drastically wrong, so you could not get to the data at all, and before you knew it the work Christmas party had gone very badly sideways and you'd found yourself trapped in an air duct using a lighter to see.

Ten points if you got it on the first one, five on the second, but only one if it took you until the third.

Happy trails.

• The 'Re-audit failed URLs' button had disappeared on the desktop version of site, which meant that users were... unable to re-audit failed URLs. If you didn't need or want to re-audit failed URLs during this time period, you probably would not have noticed - but I assure you that it did indeed disappear.

• In the last update, the Add/Remove Columns pop-up modal had decided to shift quite far to the left, in protest to the far right populism resurgence. While this is not the forum to comment on political leanings, we must draw the line at the functionality failing at a base level - in this case the modal was falling off the left hand side of the screen, when viewing Sitebulb on smaller screens/laptops.

• You know how sometimes when developers create sitemaps, they serve the sitemap index file with the incorrect MIME type? Yeah well that was stopping Sitebulb from parsing the sitemaps properly. Since we don't want to punish you, cherished SEO, from the foolish incompetence of such devs, Sitebulb now gracefully ignores the error and parses the sitemaps anyway. It also flags a warning about the invalid MIME type, so you can shout at the dev team to get their fucking shit together. Honestly.

• Sitebulb was flagging certain URLs with confusing Hints in the International report, such as 'Missing hreflang annotations'. When we investigated, it turns out that the pages in question were non-indexable URLs, and so they should not really have been considered eligible for hreflang in the first place (because hreflang is an indexing signal). In order to make the audit data cleaner and clearer, Sitebulb will now only consider indexable URLs for the following Hints:

  • Invalid HTML lang attribute

  • Mismatched hreflang and HTML lang declarations

  • Missing hreflang annotations

  • Missing HTML lang attribute

  • Has hreflang annotations without HTML lang

New Feature: Lost and Found

• New crawl comparison feature

  • Tracks granular changes between audits

  • Available on both desktop and cloud (all plans)

  • Requires at least two audits to activate

• Audit-level URL change tracking

  • Categorises URLs as:

    • Remained (present in both crawls)

    • Found (new URLs)

    • Lost (URLs no longer present)

  • Provides clearer insight into net URL changes between crawls

  • Clickable metrics to view affected URLs

• Integrated into Audit Overview

  • No need for separate comparison tools or reports

  • Automatically available within existing workflow

• Hint-level change tracking

  • Embedded within Hints section for deeper analysis

  • Tracks how specific issues evolve between crawls

• New Hint-level statuses

  • Regressed: Existing URLs that now trigger an issue

  • Fixed: Existing URLs that no longer trigger an issue

  • Also includes Remained / Found / Lost at Hint level

• Improved issue analysis

  • Helps identify newly introduced vs resolved problems

  • Enables more precise debugging of technical SEO changes

Google Search Console Integration Update

We had a few reports of Sitebulb's GSC data being off (mostly on clicks and impressions). These had come from some of our bigger customers with massive traffic numbers, and we thought initially it must be a sampling issue.

But when we looked into it more deeply, we discovered that Google was sending back a LOT less data than we expected in some cases. There were two key things in our requests that influenced what the GSC API returned - traffic split by device (i.e. desktop/mobile/tablet) and keywords data - both of which Sitebulb requested by default. 

When we request keywords, the data returned by the GSC API is sampled, and some queries are removed to anonymise the data - this was having a particularly noticeble impact in thecase of large websites and specific industries. 

Frankly, it was making Sitebulb look like it didn't know wtf it was talking about :(

As a result, we have:

• Simplified our Google Search Console integration by removing device-specific data breakdowns (Desktop, Mobile, Tablet). Instead, you'll see aggregate metrics across all devices - All clicks, impressions, CTR, and position data now represents combined data from all devices.

• Keywords data is only requested when the keywords analysis report is enabled. So your search traffic metrics will be most accurate when Keyword Analysis is disabled.

Additional benefits of this change:

• ~30% reduction in GSC data processing time

• Faster audit completion for GSC-enabled audits

• Reduced database storage requirements (saving your hard drive space 1 MB at a time)

But the main one, of course, is that the data is no longer completely fucking wrong.

• When running Content Extraction, in the overview Sitebulb was reporting both 'Total Found' and 'Found on URLs' as identical, which would be awesome if they were measuring the same thing and/or happened to be the same value. Unfortunately they are not measuring the same thing and rarely happen to be the same value, which means this was sub-awesome, so we fixed it.

• On certain exports that include the XML Sitemap URL, Sitebulb was including a bunch of random numbers in the export, instead of the actual 'XML Sitemap URL.' While the numbers look random to me, I expect it was actually an Easter Egg that Gareth left in the tool intentionally for people like him that only like motor sports rather than real sports (like the 'biggest torque value' ever recorded for an F1 car or some other such nerdy bollocks).

• So... a user tried to use Bingbot as their user-agent, and Sitebulb refused to crawl. This is because we had a non-ASCII character in the UA string, which means we had fucked up. Does this mean that we encountered the very first usage of Bingbot as a user-agent, over 8 years since Sitebulb first launched?!