Announcing the launch of Sitebulb

We are proud to announce the official public launch of our new product: Sitebulb.

Sitebulb is a website crawler that helps users carry out technical SEO audits. It is a desktop product which works on Windows and Mac.

Why does the world need another crawler?

Quite frankly, it doesn't. There are dozens of excellent crawler products on the market already. But the problem is, all they do is crawl. Well, they crawl and then spit out endless data tables for the user to try and negotiate themselves.

Sitebulb is fundamentally different to this - it actually audits the data it finds and presents these results instead.

We built Sitebulb with 3 core differentiating features in mind, which are aligned with the biggest pain points:

  1. Reporting - how do you build a report following a crawl?
  2. Data Visualization - how do you communicate complex issues?
  3. Scale - how do you deal with really big websites?

We've not approached it from the perspective of someone carrying out a crawl, but carrying out an audit.

#1 Reporting

Most website audits are carried out because the user needs to produce a report. Perhaps they are producing a technical audit report as part of an initial deliverable, or running a monthly health-check report, or simply reporting to their boss.

Understanding that 'crawling a website' is simply one step in the larger task of creating a report, has allowed us to approach our product design in a different manner to the other tools on the market.

At every step, we've tried to focus on the experience of writing a report, and how Sitebulb can help with this process.


In each report section, Sitebulb produces a set of contextual 'Hints', which essentially flag potential issues to the user. This can translate into a big win when you think about how much time you can save manipulating data in Excel, as Sitebulb will pre-process all the data for you.

For example, the Hints below can be found in the Indexability report:

Indexability Hints

Issues such as 'Canonical points to a URL that is Not Found (4XX)' is potentially very damaging from an SEO perspective, and Sitebulb gives you everything you need to fix it, without having to unpick tons of data in Excel.

Sitebulb also includes lots of Hints and reports that you simply won't find in any other tool. The Site Speed section is a prime example of that - Sitebulb is the only software that will provide speed scoring recommendations for every single page on your website.

Site Speed Audit

As you can see above, this view allows you to access either the URLs affected, or the resources that are the root cause - again demonstrating a commitment to give you everything you need to solve the problem.

#2 Data Visualization

Crawling a website produces a ton of data. The job of an SEO is to interpret that data and translate it into meaning - something which Sitebulb is set up specifically to help you with.

There is no better example of this than Crawl Maps, which provide an intuitive, interactive visualization of website structure.

Consider the Crawl Map below for an ecommerce store. We can see immediately that the homepage (big green dot on the left) basically links to every single product directly, with no tiered category structure or hierarchy. The blog has its own little hub (over on the right), with some pagination issues to sort out (the crazy whip like tentacles).

This sort of visualization makes it both easy to comprehend a fairly complex idea, and also to explain and demonstrate this idea to others.

Crawl Map

Each reporting section also includes several graphs, each of which reveal a different aspect about the site (or the crawl itself), which can be crucial in understanding how a site is set up.

URLs Crawled By Depth

Or simply so you can say to your client: 'this doesn't look right, here take a look.'

Indexability Status

Again, Sitebulb makes it easy to export graphs with a click of a button, so that adding them to a report is a straightforward task.

#3 Scale

The problem with scale is that most desktop crawlers aren't built to crawl big sites. Since they hold everything in memory, the crawler will stop when it runs out of RAM, which can be a limitation when trying to crawl sites over 100,000+ URLs.

The logical alternative has always been to use a cloud crawler, which have access to more resources and so do not come up against the same types of limitation. However, a cloud crawler can become very expensive, very quickly, and as such do not provide an affordable alternative to most people.

Sitebulb neatly fills this niche, as any user can comfortably crawl over 500,000 URLs on a normal desktop computer (up to a hard maximum of 2 million URLs). It does this by writing data to disk, instead of storing it in RAM, making it much less resource intensive, while allowing this sort of thing to be the norm: 

Crawl 100k Site

The reality is that most websites that people need to audit are simply not that large. Typically it is the addition of one or two larger client sites that force users to move away from desktop solutions, potentially paying over the odds for a cloud-based product simply to complete crawls for these one or two bigger clients.

Sitebulb however is an every-down crawler - it will handle the small to medium sites AND the larger sites. While it should not be considered an enterprise solution, it can handle most websites you throw at it.

Who are we?

We're nobody special. We're not a cool startup that's just secured funding, we're a bootstrapped, 2 man team and we've built both our products from the ground up.

We have an obsessive focus on improving our product, and our singular goal on customer support is making sure that every customer is happy (even if that means telling them a different product is a better fit for them).

And we are ever thankful for the amazing, supportive SEO community that underpins our industry and helps inspire us all to bigger and better things. We wouldn't be close to having a launch-worthy product without the help of thousands of beta testers putting it through its paces.

Software products are our way of giving something back to this community. Our goal is to help agencies and consultants of any size carry out amazing technical SEO work, and we hope Sitebulb can help you.

Ready to try Sitebulb?
Start your free 14 day trial now

Start Free Trial