Sitebulb Release Rants

Version 9.0.275

Released on 29th October 2025 (hotfix)

Fixed a ridiculous issue when opening Sitebulb on desktop, where it would open up then immediately do a 180 and shut itself down. Computer says no.

This bug is only present on the original 9.0 version, if you encounter it, please update immediately!

Version 9.0

Released on 20th October 2025

URL Crawl Limits Now Based on HTML URLs

Up to now, Sitebulb's 'crawl limit' has been calculated on all pages crawled - which would include external links and page resource URLs, in addition to internal HTML URLs.

The limit I'm referring to is found in the 'Crawler Settings':

But most folks base their understanding of 'how big is my website?' only on internal HTML URLs (i.e. unique pages) and so our implementation of these limits was confusing. As an example, you might have an ecommerce store with about 10,000 pages - based on around 9,000 product pages and 1,000 other pages (categories, subcategories, blog, etc...). But if you run a crawl, Sitebulb could easily find another 1000 external URLs, and maybe 50,000 page resource URLs (images, CSS, JavaScript etc...).

So if you set a crawl limit of 10,000, you might actually find that the crawl would stop when only a couple of thousand internal HTML pages had been crawled (as the rest had been external links or page resources).

While no one actually complains at us about these sort of things, we know how annoying they must be for day-to-day usage. So we completely changed our philosophy on how we calculate it, so it's now based entirely on the number of HTML pages crawled.

We also updated the Crawl Progress UI to add more clarity around how many HTML pages have been crawled or are due to be crawled:

In practice, what this means is that all of Sitebulb's plans have become more generous in terms of 'how many pages you can crawl', and it should be easier to set appropriate crawl limits as there is less guesswork involved.

Project Tags

One of the extremely cool benefits of a tool like Sitebulb is that there's no project limits in place at all - you could literally add an infinite number of projects if you could count that high (you can't) - so it's not uncommon to have hundreds of projects in play at any one time.

This is particularly true if you've been using Sitebulb for years, or if you have a team on Sitebulb Cloud with lots of users all adding their own projects. 

Over time, this can become unwieldy, so to help tidy things up a bit, we've introduced 'Tags' to project settings. Tags introduces a flexible, intuitive way to organize and navigate your projects with your own organizational logic.

Once you have some tags up, you can easily filter by them using this new dropdown:

The above example might work well for consultants who do different types of work for different clients. Here's some other examples:

• Agencies could tag based on client name, account manager, or region

• In-house teams could tag based on locale (e.g. US/UK/DE) or brand family

• Larger teams could combine tags with standardized taxonomies (e.g. client name + project type)

To add a tag to a project, navigate to the project page and hit the Add Tags button:

Then in the modal window, hit the dropdown and choose tags you wish to add to that project.

To create new tags, simply start typing in the box, then hit enter:

Projects can have multiple tags, so you can group or filter projects across different dimensions.

Please reach out to us if you have ideas or feedback on how you are using this feature and what we can do to make it more useful.

Batch Delete Projects

Several years ago I made the mistake of mentioning 'you'll never add infinity projects to Sitebulb', and a number of users took this as a personal challenge.

Sooner or later they would realise the folly of their ways, as their computer would routinely whinge about 'low disk space' and other such complaints.

The support tickets we'd get revealed the depths of their frustration, 'why can I only delete projects one by fucking one?'.

We heard you, check this out:

Simply select the projects you wish to delete, then hit the button to get rid of them forever.

No more angry emails (at least about this specific thing...).

Batch Delete Audits

Not content with one batch delete option, we've actually added two: this one allows you to delete multiple audits at once from within a single project.

And why might you one need to delete lots of audits at once? Erm...

Even for smaller sites, if you crawl them shitloads of times, the overall size really can add up.

Deleting audits works almost identically to the deleting projects method - simply navigate to the project in question, select the audits that you wish to delete on the left, then hit the 'Delete Audits' button:

New Authentication Menu & Better Shopify Support

I've got to admit it has become very pleasing when I go to buy yet more niche cookware I absolutely don't need but absolutely must have, and the ecom store happens to run on Shopify - that checkout process is just slick.

In the last few months we've seen an uptick in support tickets from folks trying to crawl Shopify sites and being hit with a 429 response - which causes Sitebulb to stop the crawl early.

This most likely reflects the widespread concern that LLM crawlers are aggressively scraping website page content with little regard for the servers that power them, friendly crawlers like Sitebulb are unfortunate collateral damage.

To their credit, Shopify have addressed this issue directly and published guidance on how to crawl your Shopify store, which allows you to generate a HTTP message signature and authenticate your crawl requests using Web Bot Auth.

We too recognise the growing focus on authentication for crawling - which for the overwhelming majority of the entire history of SEO has been necessary only for staging sites - and so we have a new 'Authentication' menu item which groups all the authentication methods we support.

This comprises the 3 authentication methods you can add to Sitebulb:

• HTTP Authentication

• Forms Authentication

• Custom Headers

• PLUS a new section 'Shopify Settings', which is a specific application of Custom Headers

So this is where you need to head if you wish to add authentication to Sitebulb, and specifically in regards to Shopify, you can follow along with our doc: Crawling Shopify websites with Sitebulb, which you can use in conjunction with Shopify's own instructions to generate the HTTP message signature and add this into Sitebulb;

Improved HTML Template Recognition

This is one of those updates that we kinda had to do because the feature was getting worse over time, but might cause a few of you out there to go 'oh for fuck sake, why did you change that?' I'm sorry.

What have we done?

We have improved the way HTML templates are classified, which should give you a more accurate page template classification.

Why was this necessary?

Over time, the way that HTML pages are created within CMS platforms has made them become increasingly dynamic, with sections and elements loading in with more unique identifiers - which was making it harder for Sitebulb to identify and group them accurately.

Why might you be pissed off?

If you have previously customized the template names within your projects as part of your workflow... you will need to do this again. Once you run a new audit post-update, your pages will be re-classified and re-grouped, and you will notice the template names in the list revert back to default.

Who can you blame?

As always, please direct your curses towards Gareth - he was the one insisting on this change, and in general can be blamed if Sitebulb ever causes any negative sentiment.

Duplicate Projects

To make it quicker an easier to create new projects based on the settings used in another project, we have added a 'Duplicate Project' button:

Once you press this button it will bring you to the 'New Project' screen, so you will then need to update both the project name and the start URL:

After this, when you get through to the pre-audit stage you will notice that the duplicated project will have inherited all the audit settings from the original project - so you won't need to go through and set these up again.

'Learn More' Links Added to PDFs

We've added an option when generating PDFs to include links to Sitebulb's Learn More content when printing Hints:

If you select this option, the PDF will then include clickable buttons to the corresponding 'Learn More' page on our website:

WARNING: If you have not told clients that you use a magical piece of software called 'Sitebulb' to make you look like a literal fucking superhero, using this option may give the game away... 

Archives

Access the archives of Sitebulb's Release Notes, to explore the development of this precocious young upstart:

• The Beta Notes (January-September 2017)

• Version 1 (September 2017 - March 2018)

• Version 2 (April 2018 - May 2019)

• Version 3 (October 2019 - July 2020)

• Version 4 (August 2020 - June 2021)

• Version 5 (July 2021 - December 2022)

• Version 6 (December 2022 - March 2023)

• Version 7 (March 2023 - July 2024)

• Version 8 (August 2024 - October 2025)