If you've found your way here, this means you are probably interested in the Sitebulb Server beta.
The purpose of this post is to introduce you to Sitebulb Server and clarify what it is and what it is not. Hopefully by the end of it you will have a clear idea if you think it is for you or not (to be clear: it is not for everybody).
First things first, Sitebulb Server is not a cloud crawler.
In fact, it is a solution which means that cloud crawling is not necessary.
Crawling websites uses a lot of computer resources, and requires a lot of storage, so this is mostly what you are paying for when you use a cloud crawler.
Cloud crawlers have stringent limits on the number of projects/domains you can crawl and the number of URLs you can crawl - this is because of the elastic way in which cloud hosting prices grow. If there were no limits, the prices would explode out of control.
Sitebulb Server is different to this - it is designed to be run on a dedicated server where you are working with fixed (or capped) costs, that means your usage does not need to be limited by arbitrary thresholds.
It is a much cheaper way to achieve the same thing.
And it does this by ticking off all the things you want from a cloud crawler in the first place:
Sitebulb Server version is designed with certain types of users in mind:
Of course there will be some amount of crossover here, but in general most users will fall into one of these categories, and there will be some features of Sitebulb Server that are more important to you than others.
And if you don't need any of the things above, you probably don't really need Sitebulb Server.
Sitebulb Server is designed so that after the initial install, you don't need to keep logging back onto the server again via Remote Desktop Connection.
Instead, you can connect to the server on your desktop or laptop machine running Sitebulb Pro. This means it is perfect if you are already using Sitebulb Pro - but if you have not already got a Sitebulb Pro subscription, you will need to get one if you want to try Sitebulb Server.
You will need a Pro license for every user that wants/needs to connect to the server.
To get set up, you need to follow these three steps, in order:
As with everything in SEO, 'it depends.' The options range from 'essentially free' to hundreds of dollars a month in terms of cost. Which option is suitable for you will depend on what resources you currently have available, your budget for technical SEO, and how 'mission critical' website crawl data is to your business.
These are the options you can go for:
If you happen to have an spare Windows desktop computer lying around in the office, this could be a good starting point. This machine can be set up to act as the server, and as long as it stays on (and connected to the internet), you'll be able to connect to it from your other desktop/laptop machines.
This spare computer would need an absolute minimum spec of 8 cores with 16GB RAM and SSD hard drives - if you happen to have a more powerful machine available, bigger is definitely better. More cores will allow you to crawl faster or support concurrent crawling, and more RAM will allow you to process bigger websites.
Please check out our server recommendations guide for more details about suitable machine specs.
AWS/Google Compute is a really quick option that will allow you to test Sitebulb Server without a monthly commitment to a rented server.
They literally take 10 minutes to set up (see our video guides) so they are a great short-term or test solution.
However if you want to use Sitebulb Server on an ongoing basis, a cloud solution can cost as much as 5X more.
This is why our top recommendation is to:
If you don't have a spare machine, or want a more dedicated resource to handle your crawling, renting a dedicated server is the best bet here. These days it is pretty straightforward to get a decent spec 'out the box' server set up in a matter of hours, for a few hundred dollars/pounds a month.
The benefits of renting a server like this is that you can avoid an initial outlay on buying your own hardware and the ongoing management/maintenance. It also means you will end up with a machine connected directly to a fast internet connection, without sucking up all your office bandwidth.
Also, a server can be rented on a monthly contract, so you can switch it off if you no longer need it (e.g. large client cancels) or you can easily upgrade to a bigger server as you scale.
In order to move forward with this solution, we suggest you read our server recommendations guide.
This option won't be for everyone, but if you have invested in your in-house hardware stack, it is probably the most effective long-term solution.
This is suitable for publishers or larger in-house brands where you can define exactly the right server you need for your specific crawling needs, with more clarity about the long-term requirement (versus an agency situation where the requirement is dependent on changeable client needs).
Please contact us on [email protected] if you are planning to purchase your own servers, so we can help make sure you get the right spec.
Sitebulb will put together a customised server build specifically for your needs, that is fully managed and taken care of for you.
This is the most expensive option, as it does not require you to set up or run servers at all, and instead let us take care of it.
This is suitable for large enterprise brands or agencies who need to regularly crawl large websites, but don't want to pay the eye-watering fees that DeepCrawl, OnCrawl or Botify charge them.
If this sounds like you, please get it touch with us via [email protected] so we can discuss your requirements. We'll have a meeting to properly understand your needs, then put together a bespoke quotation for you to consider.
Having read all of the above, you now need to make a decision about the right option for your business.
To help give you further guidance in this, please read our server recommendations guide.