Sitebulb Version 2.0

#1 New JavaScript Crawler: Headless Chrome

We've ripped out our old JavaScript crawler and replaced it with a brand spanking new version of headless Chrome.

Our old JavaScript crawler was built using PhantomJS, a very popular headless browser (which is essentially a web browser without a graphical user interface), which has been a stable solution for us as we've developed the software over the last 2 years.

However, in the middle of last year, two important things happened:

  1. Google announced they were shipping headless Chrome in the public release of Chrome 59
  2. The PhantomJS developers, upon learning about headless Chrome, ceased developed of Phantom.

The upshot was that our JavaScript crawler has been getting stagnant, and was completely unable to crawl certain sites using particularly recent frameworks. But the new Chrome release also gave us an exciting opportunity to significantly improve our product.

Integrating headless Chrome has given us the ability to add some unique new reporting features, and report on data that was otherwise unavailable to us. These include a full front-end and performance audit, a code coverage audit, and an accessibility audit (you can read more about all of these below). In short, it's like being able to run Lighthouse on every single page on your site.

You can select the Chrome Crawler on the main audit setup page:

New Chrome Crawler

You may notice from the illustrative image above that we've also re-named our crawlers, the new one being 'Chrome Crawler' (because what does 'JavaScript Crawler' even mean?) and the other one being 'HTML Crawler' (which is exactly the same as the old 'Non-JavaScript Crawler').

#2 New Report: Front-end

Front-end is a new report you can get through Sitebulb, which is bundled in with Page Speed and Mobile Rendering in the audit selection audits. In combination, you end up with a full performance audit on every page on the site, pretty sweet huh?

The Front-end Hints include HTML validation, CSS validation and reporting of JavaScript errors, in addition to a ton of other useful stuff.

Front end

Additionally, it will also give you a Cookies report, which shows you all the cookies set by the website, along with the expiration on them. This gives the ability to do a complete cookie audit, to help you set the right consents in order to be GDPR compliant (25th May is coming, people!).

Cookies GDPR

#3 New Datapoint: First Meaningful Paint

Using the Chrome Crawler, Sitebulb can also now collect Time to First Meaningful Paint, which lives in the Page Speed report. First Meaningful Paint is possibly the most user-centric page timing datapoint anyone has ever come up with, as it is based on the user's perception of load time, identifying the time at which the user feels that the primary content of the page is visible.

First Meaningful Paint

By the way, if you've never heard about First Meaningful Paint, watch the first 10 minutes of this video and you'll understand it well enough to throw it around in a sentence and make everyone else look dumb when they don't know what you're talking about.

#4 New Report: Code Coverage

For those not satisfied by the performance optimizations suggested by the Front-end audit, the Code Coverage report will allow you to squeeze even more juice out of your site.

The idea here is that Sitebulb will show you where CSS and JavaScript code is being loaded in but not being used. It will do this for every page on the website, so you end up with a really clear view of 'dead code.' Cleaning this up so you only include the code you need will allow you to reduce the size of your pages and improve load time.

Code Coverage by file

You can also click through for each resource file and Sitebulb will pick out the unused lines of code from each CSS and JavaScript file. This combination allows you to find files that are not being used, and to optimize the code on files that are being used.

Code wastage highlighter

A few months ago Ian Lurie published a typically enjoyable post on using the Code Coverage report in DevTools to improve page speed, which should act as an excellent primer if you are new to the topic. Sitebulb takes the same DevTools methodology and aggregates it across the whole website. 

#5 New Report: Accessibility

Web accessibility sits beyond the realms of what is generally considered digital marketing, but is becoming more important in the development world as the web matures.

Accessibility is about making your website more inclusive, ensuring that all of your potential users, including people with disabilities, have a decent user experience and are able to easily access your information.

If you select 'Accessibility' from the audit options, Sitebulb will run over 35 automated accessibility checks, as it crawls, across your entire website - identifying the worst performing URLs and the most common violations.

You can pick out any URL and Sitebulb will build the DOM for you and highlight the violations, so you can inspect them directly without leaving the software.

Accessibility Violations

#6 New Report: Keywords

This new report is not made possible by Chrome, but by the Search Analytics API, which you can connect to Sitebulb through your Google Search Console account.

I'll show you what it looks like first, then how to set it up:

Keywords Report

There's a few other charts, and of course you can click through to a list of all the queries for every page. We haven't added any Hints yet, but we'll be soliciting feedback on this and developing this report further over the next few months (so if you have any ideas, please share!).

You add the Keywords report as an optional extra in the Project setup, once you've added a Google Search Console account/property. To get the breakdowns on branded/non-branded queries, you'll need to enter some brand variations in the box so that Sitebulb knows how to classify the keywords it pulls back from the API.

For example:

Keywords Setup

#7 New Report: Security

No matter what audit options you select when setting up, Sitebulb is going to give you the new Security report whether you want it or not.

But you should pay attention to it, because security is kinda a big deal yo.

Not Secure Message in Chrome

HTTPS and mixed content issues grab all the headlines, but Sitebulb will also alert you of a ton of vulnerabilities you've probably never even thought about, which will now keep you up at night. You're welcome.

Advanced Security Checks

There is also a quick 'Insecure Content' export, which includes triggered Hint data from all instances of insecure content: mixed content, HTTPS URLs linking to HTTP, page resources being loaded with protocol relative URIs and links vulnerable to tabnapping.

Insecure Content Export

#8 List mode!

For years we've sold a product, in URL Profiler, that works exclusively in 'list mode'. People would often ask us, 'when are you guys gonna build a crawler onto this thing?'

As soon as we release our crawler product, we get the opposite! 'Yeah but I want to just crawl a list, not the whole website.'

We can't f%$&ing win.

So here it is. You can now select a URL List as a URL Source to audit. You can audit the list on it's own, or combine it with a crawl, or combine it with sitemaps as well. 

List Mode Baby

Despite my sarcastic tone above, this is actually a feature we're pretty excited about. It allows you complete control over which URLs are 'crawled' (although they are not really crawled in the strict sense of the word), and allows you to check smaller subsections of a site without having to crawl the whole thing.

Either way, list enthusiasts should very pleased.

#9 New URL Details panel

This one is underwhelmingly awesome. We've totally rebuilt the URL Details view, and it's very cool.

The new panel slides out from the right of the screen, and shows more data about each URL, which you can navigate via the left hand menu.URL Details panel

You may also notice the blue 'Live View' button under the menu. Clicking this causes Sitebulb to go and fetch the page 'live' and render the page content (using the aforementioned headless Chrome).

This gives rise to some really useful data:

  • The response HTML, the rendered HTML, and any differences between the two (super useful for understanding how JavaScript can change HTML content, which may affect crawling and indexing).
  • Live HTTP Headers (request and response), so you don't need yet another Chrome extension.
  • Rendered screenshots for desktop, tablet and mobile, so you can quickly compare and contrast (or save for later, should you wish).

Since words are rarely enough, this can be better communicated via the medium of gif:

Live view

You can access the new URL Details panel by clicking the blue 'URL Details' buttons you'll find on the left in URL Lists.

URL Details

#10 More helpful Hints

Probably my favourite of all the new additions (because I do the support and it should hopefully make my life easier!), we have added contextual help for a number of Hints, and given ourselves the ability to scale this out across the coming months, so that every Hint contains more support.

Firstly, we have added new 'Hint Details' panels to a number of Hints, that work very much like the 'URL Details' panels discussed above. In particular, we think these can help with Hints that highlight on-page issues that are difficult to dig into without searching through source code.

For example, Images with Missing or Empty Alt Text, which previously would just show you a list of URLs that have more than one image with no alt text, and leave you to fend for yourself when figuring out exactly which images were the offending ones. 

Now, for any URL, you can click through to view the Hint Details from the URL List, which will open up a new frame showing you the HTML, with the images without alt text highlighted in the code.

Hint Details images missing alt text

A whole lot easier, right?

To support you further in your quest for knowledge, we're also creating Knowledge Base articles for each and every Hint, which will be linked up via the Hint descriptions in the tool, and published on the website for general consumption. You can check out an example here, and as you can see they explain what the Hint is actually checking, why it's important, and how to fix the issue.

Roughly 50% of the Hints currently have these Knowledge Base pages, but the rest will be finished in the next couple of months. We think this should make Sitebulb's Hints more transparent, help experienced users better understand what they are looking at, and provide an invaluable resource for less experienced SEOs who won't always appreciate the significance of every issue they come up against.

#11 URL List overhaul

Ok, maybe 'overhaul' is a bit of stretch, but we've made a bunch of changes.

The much requested 'sticky header' when you scroll:

URL List Sticky Header

The ability to freeze the URL column:

Freeze URL Column

Multi-level filtering (also much requested):

Filter URL List

#12 New Audit setup page

Not exactly a feature, but something that will look a bit different to what you're used to. 

We've moved around the audit setup page, and added the new options - all the new audit types mentioned above and the new URL source options.

The orange notices indicate that some of the report options will only work with the Chrome Crawler (and that Front-end works better with Chrome). We hope that users will still be selective about their audit choices and only turn on the data that they actually need, but in our hearts we know that SEOs will just press all the buttons anyway. 

New audit setup page

#13 New AMP Hint: AMP Page has validation errors

Sitebulb will now perform AMP validation on all your AMP pages, so it's checking to see if a given AMP page is;

  • missing mandatory elements
  • including disallowed, depreciated or duplicated elements
  • containing style, layout or templating errors

From the URL List associated with this Hint you can click the 'Hint Details' (see above) to check out the AMP validation errors:

AMP Validation

Ready to try Sitebulb?
Start your free 14 day trial now

Start Free Trial