How is Sitebulb Different to Other Crawlers?

I was recently asked the following question on Twitter:

Looks interesting! How does it differ from Screaming Frog, Deepcrawl et al? Any particular features?

— Blink. (@Blink_Norwich) July 11, 2017

I tried and failed to write my answer in 140 characters, so I gave up and went all in with a full scale blog post.

TL;DR - it's actually not a 'crawler.' It's a website auditor.

Desktop vs SaaS

First things first, Sitebulb is a desktop product (like Screaming Frog), and not a cloud/SaaS product (like DeepCrawl). This means that you install it on your own computer (Windows or Mac) and it runs using your own local resources (CPU/RAM).

We see this as a positive. Let me explain...

Reason #1: It's not prohibitively expensive

We had a goal in mind when we built our crawler, we really wanted to help small agencies and consultants to do amazing technical SEO work, at scale, without it costing a fortune.

We actually spent 3 months building the product as a cloud crawler, before realising there would be no way to keep the product reasonably priced. 

The reason cloud crawlers are expensive is because crawling websites produces a LOT of data. Sitebulb can have up to 300 columns of data for any one URL. Say you've crawled a site with 100,000 URLs... you're talking 30 million fields of data. And that's just for URL data, for a site with 100 links on each page, you could have another 10 million links, all of which needs storing - perhaps indefinitely - on your cloud architecture. That's a lot of bytes right there.

So any pricing package for a cloud crawler needs to cover both the potential bandwidth you will use crawling, but also the potential storage. Notice that this isn't actual storage, but potential storage, just in case you need to use it.

We realised that with a cloud setup we could not satisfy our goal of producing software that could work at scale, without costing a fortune.

So we had to slightly redefine our goal:

-> "To help small agencies and consultants do amazing technical SEO work, at scale, using desktop software."

And by scale, we mean this sort of thing:

Reason #2: It's much more convenient

With a cloud crawler, a number of things need to happen when you start a new crawl. Once you press the 'Go' button, it needs to go off and spin up a new instance on AWS (or Azure/Google Compute/etc...). This can take anything from 5 to 10 minutes just to get initialized so that it's ready to go. This means that it's not particularly easy to 'take a quick look' at a website.

In contrast, this is what desktop software is brilliant at. Screaming Frog, in particular, is like lightning. Within a couple of clicks you can be off and running and looking at data in realtime as you go.

Sitebulb isn't quite as straightforward and this, there are a few necessary setup steps due to the way that Projects are organized, but for a quick look at a site you can be off in about 60 seconds, so it's more than adequate for quick exploratory crawls.

Sitebulb is more than just a crawler

Ok. It's 2018. Every SEO worth their salt knows what a crawler is. Every SEO worth their salt already pays for crawler software, often from multiple providers.

We knew this when we set out building Sitebulb, so we knew we couldn't build 'just another crawler'.

We knew we had to build more than just a crawler. We felt that, in order to compete in this market, we had to build something that would deliver answers rather than questions, would deliver opportunities rather than problems, and would deliver shortcuts to vital insights. And, importantly, would give you the ability to communicate all this to your clients or bosses.

We also know that there are lots of advanced technical SEOs who love to manipulate the raw data themselves. They like nothing more than a massive CSV export or SQL file. To be honest, if this is you, you're not really our target customer.

Why? Because Sitebulb will do a lot of this manipulation for you - saving you loads of time in the process. It then presents the data in a huge variety of ways; there are data tables and graphs galore, filtered URL Lists, pre-designed bulk exports, and context specific Hints to highlight potential issues.

So what we've actually built is a product that doesn't simply crawl your website and spit out the data for you to analyse yourself. It will do a ton of the analysis for you. It would be much more accurate to call Sitebulb a 'website auditor' than a 'crawler.'

And then there are the Crawl Maps. This is easily our most popular feature, if beta feedback is to be believed. It tackles that big question that has been gnawing at SEOs for years: "how do we explain site architecture to clients (or other non-SEO muggles)?"

Crawl Maps provide a fantastic, visual, intuitive way to do this, which is one of the reasons they are so popular. The other, of course, is that they help the SEO themselves to better understand a website. One quick look at a Crawl Map can tell you information it would have taken hours to extract manually.

And yes, you can build your own Crawl Map if you prefer with any old crawl data and a copy of Gephi. But it takes ages and doesn't look as good. And who wants to do that?

This point really sums up Sitebulb, as one of our beta testers picked up on:

If you love getting your hands dirty with the data and building all the reports yourself, then Sitebulb is probably not for you. But if you’re like Tony, and you want to save yourself a ton of time on unnecessary processes, then Sitebulb could be exactly the solution you are looking for.

The best of both worlds

A LOT of work has gone into the Sitebulb interface and presentation of data. Sitebulb has been in active development for over three years, and a significant portion of that time was figuring out what data we wanted to present, and how we could do this in a meaningful way.

We were aiming for something that was intuitive enough for a junior SEO to pick up and play, yet could communicate complexities that an advanced SEO could dig deep into.

Essentially, we wanted to build a tool that had the cost and convenience benefits of a desktop crawler, yet with the reporting power of a SaaS crawler.

I may be slightly biased, but I think we achieved it.

So do these guys <humblebrag>.