Released on 7th February 2018
Improved crawling speed
Everyone likes FAST. Fast is awesome. My little boy is constantly debating who is faster: Sonic, The Flash, Usain Bolt or Catboy (from PJ Masks). My money's on Catboy.
We've made Sitebulb's task scheduling more efficient, which has made it crawl faster. You can safely file this one under 'performance enhancements.'
View folder link
Sitebulb writes data to disk, which means there's a directory on your local hard drive which contains all the data and export files. Occasionally, folk come to us asking how to find these directories. And to be honest, trying to explain it is a massive pain in't arse. So we just made a little button on the Audit Overview that magically takes you there.
XML Sitemap Report pulling in Sitemaps from other Audits
A number of users noticed this peculiar issue, where Sitebulb would display the XML Sitemap details for a completely different website, when you looked at the XML Sitemap report.
This was an issue with the user interface, rather than the crawler itself, and has now been rectified.
Truncated URLs and anchor text on URL Details page
We noticed that on some websites, the URLs and/or the anchor text was too long to fit in the allotted space we had allocated for it, causing them to tumble horizontally off the side of the page, rendering the page nigh on useless.
Enter, that versatile and truly magnificent typographical cliffhanger itself... the ellipsis.*
*Used in this instance merely to truncate. A waste if you ask me.
Disable cookies now works properly
Sitebulb has cookies enabled by default, but you can turn them off in the Advanced Settings. Well you can now anyway - if you tried to do it before today you may have noticed that it actually did not turn them off. Whoops.
Tiny typo fixed in audit progress text
Annoyingly, no users reported this glaring error, where the report building copy for 'Exporting Multiple <h1> tags' was showing the HTML:
Fortunately, I spotted it. Lucky someone is paying attention.
HTML also in Compare Audits export
Just like the issue above that no one could be bothered to notice, we also discovered that HTML had ended up in the export from the Compare Audits function.
More shit and bollocks!
Time of audit changed to only display hours:minutes
As sticklers for precision, we've always liked to display the 'time of audit' to the nearest second. Upon hearing industry rumours about one member of the Sitebulb team and his 'obnoxious pedantry', we've made the controversial decision that this degree of accuracy is wasted on, and quite frankly, unappreciated by, the entire Sitebulb community.
Typo on Internal URLs Hint
There was a typo on one of the Internal URLs Hints, where the word 'usual' should have read 'usually'. Disappointingly slack on my part.
Pagination was broken on the Dashboard
Curiously, the pagination links at the bottom of the Dashboard stopped working, so you could not browse through old projects. This was particularly annoying for users such as myself, having 320 Projects in my list.
The Hint: '<head> contains invalid HTML elements' was firing for commented out elements
One of our most popular Hints* was giving false positives when encountering commented out elements. Some schoolboy shit right there.
*This Hint scored in the top 3 Hints of all time, based on a survey of 3 people.
Pause and update settings was forgetting the crawl limit
If you set a crawl limit (e.g. 100,000 URLs) before crawling a site, then mid-crawl decided to pause and update the crawl setting, Sitebulb would very annoyingly forget the crawl limit and just keep crawling beyond the limit.
URL Details view showing URL listed in the same sitemap multiple time
If you clicked through to the URL Details page, and had XML Sitemaps turned on, the same sitemap would be listed multiple times. This led to a number of confused users, asking irritating support questions such as 'what am I doing wrong?' and 'why is this happening to me??' and 'what did I do to deserve this?'
Exporting large audits was not always working
Very few people will have seen this, but if you tried to export very large audits the software would sometimes throw an error.
Released on 9th January 2018 (hotfix version)
#1 Meta description length default changed
Since the beginning of time, every SEO on the planet has been conditioned to write all meta descriptions in the range: 140 characters < x < 160 characters. It has got to the stage where it is impossible to even write a regular sentence that falls outside these strict bounds (check these ones and see for yourself).
But then Google came along and decided to change everything – 'meta descriptions can now go up to 320 characters', they said – which I’m sure you’ll agree makes for preposterously long, awkward, unwieldy sentences that just go on and on and feel completely unnatural to both reader and writer, present company included.
So, Sitebulb has followed suit, changing the default 'too long' setting for meta descriptions to now be > 320 characters, following Danny Sullivan's tweet (Please note: this affects only the default setting, which was previously set at 170 - if you have already overwritten the default, yours will not change to 320).
#1 Sitebulb now correctly opening on startup
Occasionally, users would end up in a situation where Sitebulb would not always open up properly first time around, meaning they had to go and start the app up for a second time. If this never happened to you, you're in luck, but let's all just agree that it sounds pretty fucking frustrating. We've resolved it by completely rebuilding the startup procedure, for both Windows and Mac.
Released on 1st December 2017
#1 We sold our souls to the Apple
When we first started building Sitebulb, it was our single guiding vision to have one universal user interface that looked exactly the same on both Windows and Mac. This was the one fundamental principle we knew we must stick by. We spent hundreds of hours perfecting the design - everything from the stunning report graphs to the delightful little X button in the top right hand corner.
Windows users positively fawned over Sitebulb's exquisite design.
Mac users, however... well, that was a different story. "We really love it guys", they'd say, "but..." (there's always a but). "I can't get enough of those graphs, but...", "Those crawl maps - magnificent! But..."
"...BUT WHERE ARE MY TRAFFIC LIGHTS?????"
Day in, day out. Relentless. 400 emails a day about it. They'd tweet me, Facebook messenger me, Slack me (is that a verb yet?), they'd send me SMS messages like they were actually my friends. One guy hand delivered a letter.
I'm ashamed to say, we gave in. Here. Here are your bloody traffic lights:
Yes, this will make half our users happy. But we have paid a toll, a heavy toll indeed. We have abandoned the very principles we lived by, the values that stood at our very core. And for what? Commercialism. Shame! Shame! Shame!
Peer pressure can do this to a man.
You take care out there, kids, it's a tough world.
#2 Added user agent and language data to preferred domain check
You may come across scenarios during your day-to-day technical SEOing, where the preferred domain results do not match your expectations. 'Redirecting to m.example.com, but why?!' you may be known to cry.
Well, Sitebulb now gives you more details of the HTTP request used - the User Agent and the Accept Language - both of which may impact how the site handles the request. You can configure these by clicking change settings, which will take you to the Global Settings page, where you can adjust both of these things.
#3 Added Accept Language to configurable Global Settings
Following #2 above, we couldn't very well tell you the language but not allow you to change it. So now you can, from the Settings -> Crawler options. The first time you use the software, it will auto-detect you language settings and set the default language accordingly (but it won't do it again after that).
#4 On the All Hints page, changed the 'All Hint Data' export to 'Export All Data'
The old export was near useless, with about a million worksheets you needed to tab through. The new export is actually a collection of all the individual hint exports, along with all the reports for each section of the audit. Way more usable.
#5 'Crawl alternates' is no longer on by default
In the 1.6.0 update, we added the ability to crawl alternate URLs, and set this as 'on' by default. Turns out, this is pretty fucking annoying. WordPress sites, for example, will spew out oembed alternate links for every page on the site. So from one crawl to the next, you would see page totals doubling. This feature is best reserved for when you actually need it, so going forwards you'll need to turn it on in the Advanced Settings.
#1 Reduced timeouts to preferred domain checks
Some users reported an issue with the little check Sitebulb does when setting up a new Project, to determine the preferred domain, where the check would take a long time (> 30 seconds). This would happen when one (or more) of the 4 options was completely inaccessible, and was timing out. We've reduced the timeouts on this check, so these edge cases will still take a little longer than normal, but only a few seconds in total.
#2 Finally resolved occasional issues with exporting/importing
As we've been improving Sitebulb, we've been making it faster wherever we can. Turns out we made it too fast in some places, so the export/import overlay could not catch up. We added a slight delay to the export building process, which has resolved this issue.
#3 Split out Hint exports to be indexable/non-indexable
We recently added exports for every single Hint, but in the individual exports they would include both indexable and non-indexable URLs. Doh! We've split them out now to only include the 'right' stuff.
#4 Fixed domain resolution for South African TLDs
If you entered a website with a South African TLD, after the preferred domain checks, Sitebulb was suggesting you crawl https://co.za. The fuck, Sitebulb??
#5 Fixed mis-firing of "<head> contains invalid HTML elements"
Our new ohgm-inspired Hint was accidentally firing when it found references to <p> in a script in the head. Our bad.
#6 Fixed crappy word counts
Occasionally Sitebulb's algorithm for figuring out the content area (and thus, the 'Content Words' and the 'Template Words) would get one mixed up with the other, the other one mixed up in both, and both mixed up in all. Normally such errors are due to really shit HTML, but in this case we found an example that was entirely Sitebulb's fault.
#7 Duplicate Content export button now wired up correctly
From the 'All Hints' page, the export button associated with Duplicate Content did nothing, because it was not wired up to do anything. It is now.
Released on 24th November 2017
#1 Preferred domain check
When you start a new Project, Sitebulb will go off and check that the start URL is the right URL, right after you enter it. It checks the http/https/www/non-www versions to see how each responds, and advises you which it thinks is the best option.
If you tend to copy/paste URLs from your browser, you may not notice a lot of difference here. But if you're a URL-typer, this could save you a ton of wasted time.
Of course, this may also reveal some issues that you need to get fixed!
#2 Crawl faster (with threads)
We've been rather reluctant to add the ability to crawl with threads, for very good reason, as this method of crawling is notoriously bad for crashing servers and pulling down websites. But it's such a regular request that we figured we'd better do something about it.
So we've added the ability to crawl with threads (up to a maximum of 25, beyond which point the tool does not really work any faster).
We've written a guide on How to Crawl Really Fast as well as its counterpoint: How to Crawl Responsibly, which emphasises all the reasons we were reluctant to do this in the first place. We'd encourage all users to read the second piece, as it should also give you a better understanding of how the tool works.
#3 Better 404 Testing
Up until now, we've had 404 tests visible on the URL Resolution section on the Audit Overview. The results they've thrown up have caused us a number of questions, so it's clear that what we had before was not clear enough. Clearly.
In order to be clearer, we have moved the 404 Tests to a tab on the Internal URLs report. Not only that, we've added a lot more tests, and more details about what we are testing and what the tests show.
This constitutes much more thorough 404 testing - checking for pages, folders, images, CSS, text files and XML. Each should respond with a 404 response.
We have *ahem* deliberately left our site misconfigured, for the purpose of this demonstration. The sacrifices I make for you guys eh?
#4 More crawl control: crawl canonical, pagination and alternate URLs
Some users have asked for more control of what Sitebulb will crawl, specifically relating to canonicals and pagination links, so we've added some new crawl controls in Advanced Settings (under the 'Robots' tab).
By default, Sitebulb WILL schedule and crawl any canonical URLs, alternate URLs or pagination URLs that it finds - either in the <head> or in HTTP headers. In order to STOP Sitebulb crawling any of these URLs, you'll need to tick the appropriate box.
Note that if a URL you wish to stop Sitebulb crawling is also linked to via anchors, it will still get scheduled and crawled. You'd need to use 'Excluded URLs' in Advanced Settings.
#5 Pre-audit notifies you about site features
#6 Loads of new Hints
Gareth's favourite saying is 'the internet is broken.' While most blokes while away their weekends watching sport or drinking beer, Gareth prefers to sit in a darkened room, trawling the internet to find new examples of shitty web pages. He's been known to exclaim, without warning, 'JUST LOOK AT THE SIZE OF THAT HEAD.'
New Hints as follows:
- HTML is missing or empty - literally, pages with absolutely nothing on (consider exhibit 1 and exhibit 2)
- Has link with a URL referencing a local or UNC file path - people do some stupid shit when putting together web pages, like linking to files that no one in the world can actually see.
- Has link with a URL referencing LocalHost or 127.0.0.1 - say it with me, 'the internet is broken.'
- Has a link with whitespace in href attribute - like, if someone accidentally put a space at the end of the URL, à la href="https://example.com/page/%20". How embarrassing.
- Next/Prev Paginated URL is canonicalized to different URL - I mean, what are you even playing at with this shit?! If you canonicalize a paginated page, Google is not going to crawl the rest of the paginated series. Dumbass.
- Noindex found on rel Next/Prev Paginated URL - Oh. This one is not that bad actually. But nice to know, I guess.
- Internal/Resource URL is part of a chained redirect loop - This is more like it. Redirect chains that go round in a big loop, like 1 -> 2 -> 3 -> 4 -> 1. Internet = broke.
I'm not finished yet, we have three more. These were all inspired by serial-internet-breaker @ohgm in his latest escapade 'Breaking the Head (Quietly)'. Stop reading my drivel, go and read his instead. Then come back and appreciate these 3 new Hints:
- <head> contains a <noscript> tag
- <head> contains a <noscript> tag, which includes an image
- <head> contains invalid HTML elements
We'll be getting some Sitebulb t-shirts printed with #internetisbroken if enough of you start tweeting out the hashtag. I'll be the judge of when enough is enough, thank you very much.
#7 Search for a Hint
You can view all Hint data via the All Hints screen (top right nav) or via the Hints tab on the Audit Overview. Both of these now have a 'search for a Hint' box. Note that this only searches triggered Hints, so if you search for a Hint and it doesn't come up, then that means it was a green tick pass.
#8 Autoscroll to previous view scroll location
You're using Sitebulb in full blown investigation mode, you're out to cracks some heads together today. You're checking all the reports, inspecting each and every graph, looking for something, something. You know it's there somewhere. A string for you to pull on. Then all of a sudden, you feel your spidey sense tingle. A graph. A pattern. That might be it. It might just be. You have to know more. So you click, and you're there, IN the data. You're Neo now, and you can see everything for what it truly is. You can feel it, you're right on top of it. You pause, not wanting to get ahead of yourself, trying to slow the heart beating out of your chest. You want to double check the graph, make sure you've got it right. You hit 'back' and WAIT. WHAT?! WHICH GRAPH WAS I LOOKING AT AGAIN?? NOOOOOOOOOOOOOOOOOOOOOO!
No one can be told what the Autoscroll to previous view scroll location is. You have to see it for yourself:
#9 Select Google Data for up to 90 days
Thus far, Sitebulb has offered a paltry 30 days worth of data to check. With this 3X update, you can now select up to 90 (ninety) days worth.
#10 Added Google Analytics Page Timings Data
Sitebulb has a Google Analytics integration. It also does Site Speed testing. We figured, why not mash these things together and pull out the GA Page Timings data?
This is what you'll get in the Site Speed report (on a separate tab):
You'll get this by ticking 'Site Speed' and selecting a Google Analytics profile, when setting up the Project.
We plan to iterate on this feature to make it more useful, so please hit us up with any ideas you have about it.
#11 Added Happiness feedback to toolbar
Here in the world of desktop SEO software, we live in a vacuum for most of our lives. And no, I don't mean we live in a fucking vacuum cleaner, I mean that we don't really get to talk to our customers very often.
So we are always looking for new ways to elicit feedback. Our latest idea is a happiness button in the toolbar:
#12 Low disk space warning
Not everyone is aware that Sitebulb does not hold data in RAM (like most desktop crawlers) but writes to disk instead. This means that if you're crawling a big site and you don't have much disk space, bad things are going to happen.
To mitigate this risk, Sitebulb will now warn you if you've got less than 5 GB space remaining - which is where you might want to start thinking about it.
If you're the type who likes to live life on the edge, you can simply dismiss this message with the handy 'X'.
#13 New column added: "No. Outgoing Navigation Links"
This is part of a wider plan, to offer more visibility to internal linking. It starts with this acorn, splitting out navigation links on a page. You'll find it as a new column in URL Lists, so feel free to have a play with it and tell us what you think.
Otherwise... watch this space.
#1 Fixed Bosnian language code on hreflang check
In the International report, the language code 'bs' was being incorrectly labelled as invalid. Thanks to the helpful user who pointed out that Sitebulb was talking complete BS! (sorry)
#2 Corrected colour coding on compare audits
Our vaunted compare audits feature includes helpful colour coding, so you can quickly see what has improved or disproved unproved got worse (using the universal colours for good and bad: green and red). Previously it was doing this in a 'dumb' way, where any increase was green. Now, using a ground breaking combination of AI and machine learning, it correctly figures out things like 'less 404 errors is actually a good thing.' Ground = broken.
#3 Fixed the All Hints page showing SEO and HTML Hints grouped together
The Sitebulb community has been up in arms about this one! Hundreds of you pointed out that the On Page section is split into two Hint groups: SEO and HTML, and combining them as one on the All Hints page was an abomination. Put your pitchforks back in the pitchfork cupboard, people, for they are now separate.
#4 Fixed Hint for 'Base URL Malformed'
Sitebulb was incorrectly claiming that <base href="/"/> was an illegitimate base URL.
#5 Changed sheet name for Hint: Has only one followed internal linking URL
On the export for this Hint, the sheet name used to be called 'Has 1 incoming link'. This has been changed to 'Pages with only 1 linking URL', which is more accurate, and because Sitebulb users are magnificently anal and we love them for it.
#6 Syntax support in exclusion list
One of our users noticed that $ symbols were not being correctly recognised when used in the 'Exclude URLs' setting, so URLs were not being excluded as they should. This has been resolved, so feel free to throw your $$$ around like a mother fucking gangster bro.
#7 Fixed the back button on Audits
If you followed this path: Dashboard -> Project -> Audit, and then hit the Back button, you'd be returned to the Dashboard, rather than the Project.
#8 Fixed: Multiple GA codes Hint
This Hint was occasionally firing false positives, for instance if a GA code was referenced in a script. We've now fixed it so that it only reports if 2 or more different GA codes are found.
#9 Fixed Hint: URL receives follow and nofollow links
In some cases, this was not correctly reporting the nofollow links, making the Hint pretty useless.
#10 Fixed: Cache headers reporting invalid
Cache headers were not being reported correctly, which was incorrectly firing this Site Speed Hint: Set long expires cache headers.
Released on 3rd November 2017 (hotfix version)
#1 Fixed export on compare audits
Per update 1.5.0, we added a ton of new exports, changing the way exports are built in the process. This managed to break the export associated with 'Compare Audits', which was pointed out to us 13 seconds after we launched 1.5.1 (thus inspiring our new hotfix release).
#2 URL resolution checks (404, Non-WWW, etc) on some sites were being rejected by the server
You probably won't notice much difference here, but the little URL Resolution checks on the Audit Overview were failing on some sites and showing inaccurate results. Rest assured that everything is ok now.
Released on 2nd November 2017.
#1 Historical trend data via sparklines
How awesome is this?
Carry out more than one audit within a project, and you'll be presented with little sparklines everywhere which show changes over time. They appear on all the main 'Insight' numbers as below, and alongside all the Hints.
Up and to the right people!
3 important things to note:
- This will work for new AND existing audits, so you check it out right now if you have a Project with a few audits on.
- Even if you delete your old audits, this history will persist so the sparklines will still show your trend data.
- If you're just dialing in your technical SEO, your sparklines will remain flat forever, making you look like the lazy, no good, piece of shit that you are.
#2 New report: Duplicate Content
Duplicate Content is one of those things you have to check on pretty much every single audit, and when it gets out of hand can cause a whole bunch of problems. Previously, we had a tiny little section for this, squashed into the On Page report. We've gone the other way with it now, and it's got a full blown section all for itself.
This is the type of thing you can enjoy seeing in the duplicate content section (plus a whole load more):
A word of warning, however. Despite all the pretty graphs, you can't click through and view duplicate content data in URL Lists. You will need to grab the export file instead (the green button at the top), which will give you everything you need to fix your dup content problems.
The reason for this is because URL Lists are entirely inadequate for communicating duplicate content data. It's not intuitive why not, so I'll explain a little further. URL Lists are built to display 1-to-1 data. There is 1 URL per row, and all the data on that row relates only to the unique URL in question.
However, it we take duplicate Page Titles as an example, in that instance we need to say 'here is a URL with the page title of '10 Cat Pictures that are so cute you will just cry and then you won't believe what happened next' (for example), but here are also 73 other URLs with the exact same title.' So it is more like a 1-to-many relationship that we need to communicate, where the 73 are somehow grouped and associated with the 'original URL.'
Anyway, we are working on a better way of displaying this, so for now just use the export instead.
#3 New charts added to the On Page report
To fill the void left by duplicate content, we've added some new charts to the On Page report, highlighting critical on-page SEO elements.
There are 6 new pie charts in total, displaying data about titles, meta descriptions and H1s - to do with both length and presence thereof - which should hopefully make it easier to pick out optimization opportunities.
Additionally, from the global settings area, users can define the values for 'too short' and 'too long', so you can be captain of your ship, vis-à-vis the length of on-page SEO elements.
#4 Sitebulb users are no longer forced to download the latest version!
Jonny Rockstar, a self-proclaimed SEO guru, takes a call from a client in his swanky attic office:
Jonny: You've reached Rockstar SEO, I'm Jonny Rockstar, how can I help you today?
Client: Hi, I was wondering if you could help me with my SEO?
Jonny: Today's your lucky day my friend, you're reached the right place! I am the world's top expert on SEO don't you know?
Client: Oh wow, you sound amazing. I wish I was you. So you can help me?
Jonny: I can do anything. What's your website, I'll take a quick look now for you.
Client: Great! My site is secondhandsocks.com
Jonny: Sure, I'll just fire up my...Oh.
Client: What's wrong?
Jonny: FOR FUCK SAKE. I NEED TO UPDATE AGAIN?! I JUST DID IT LAST WEEK. YOU STUPID PIECE OF SHIT.
Client: (hangs up)
Sound familiar? Well, from this version onwards*, you won't be forced** to update if you don't want to***.
* The change only exists in the 1.5.0, so you kinda will be forced to update this one. And by kinda I mean completely.
** We will however notify you of new versions in future, and strongly suggest you install them.
*** Also if we release a critical update, we actually will force you do update it. Because critical.
#5 Export internal links directly from URL Lists
We're going to town with exports in this update. This one is a cheeky little update to URL Lists, that allows you to instantly export incoming internal linking URLs (to the URL in your list).
Of course you could just click the blue 'View URLs' link and export from there, but this saves you 1 less click. Think of the dozens of seconds this will save you across a whole year! This thing is like a freaking time machine!
#6 Added export button to individual Hints
For even more exporting joy, you can now export individual Hint data directly from URL Lists.
...and on the Hint page itself (#7)
The important thing to note about both these Hint exports is that this is not always equivalent to exporting the rows from the URL List. This is because not all Hints are created equal.
Some Hints need special treatment (a bit like duplicate content, see above), so those Hints have customised exports. Some examples: broken links, images without alt text, and redirects. These are more cases where there is a many-to-1 relationship, so the exports are built to handle this.
#8 New copy experiment when building PDF reports
PDF reports take about 30 seconds to build, and we used to have a message up there encouraging feedback while you wait. We've just changed it for something a little more fun. Let us know what you think!
#9 Scroll to adjusted column
This is one of those tiny UX changes that new users will not notice at all, but Sitebulb veterans will love (Aside: can you be a veteran for a product that's less than 6 weeks old?).
Anyway, whenever you adjust a column in URL Lists (i.e. sort or filter), Sitebulb will now scroll you along back to that column. Previously it would dump you back on the first column, which was très annoying.
I'll demonstrate through the medium of gif:
#10 Free users can now download exports
We have a free plan, for crawling sites up to 500 URLs. This is perfect for users with small sites, or for people who hate paying for things. We'd left a restriction on there meaning free users could not process any exports. We decided this was a bit tight, so we've now enabled exporting for all free users.
Who's tight now eh?
#1 Fixed pause and resume!
Sorry! I know, I know. Pause and resume was broken, so if you paused an audit, you couldn't resume :(
We feel partially responsible for this as well.
Maybe in time we can forgive each other...?
#2 AMP Hints had a (small) makeover
A number of people pointed out that these two AMP Hints were bogus:
- AMP URL is not indexable
- AMP URL is not in a sitemap
#1 was happening because of the way general URLs are classified as indexable or not indexable, which is of course impacted by canonicals. Since AMP URLs are MEANT to have canonicals, this Hint did not make a lot of sense.
#2 was only advisory anyway, but we removed it because I got fed up of people pointing out to me that 'John Mueller said you don't need AMP URLs in sitemaps therefore you're wrong.' I don't need the hassle in my life.
We also removed AMP URLs from appearing in a lot of the other reports, as they really need to be treated in their own AMP-esque context.
#3 HTTP headers now correctly parsing canonical link
We came across a site that was setting and image source link element AND a canonical link element in the HTTP headers. Sitebulb was not correctly identifying the canonical, happily firing off warnings left, right and centre.
Magnificent as this Hint is, Sitebulb was actually wrong to fire it in this instance, as the canonical setup used was perfectly valid.
#4 You can once again click on nodes in Crawl Maps
I have a bone to pick with you. All of you. No one bloody told me that the 'click node to view URL' feature in Crawl Maps had stopped working.
Or is this your collective way of telling me you didn't know it could do it??
It's awesome, let me try and sell it to you... -> Click on any node and BOOM straight to the URL details!
How did I do?
#5 Special characters removed from all export sheetnames
We should have seen this one coming. Mainly because we'd already spotted it once before relating to slashes (/s). Some Hint exports were failing, because we'd left some special characters in the sheetnames. F$&k my life.
#6 Maximize no longer cuts off scrollbars
Some people are overly fixated on size. Almost like they're covering up for something...
Anyway, some users like to maximize Sitebulb so it takes up the whole screen. Which is FINE. Unfortunately Sitebulb was penalising these size-obsessed freaks by cutting off the scrollbars on the URL Details page, making it look like you couldn't scroll down.
#7 Fixed issue where some exports were getting stuck
In the last release, we optimized the exporting process to make it a lot faster. Turns out we made it too fast, and the UI was actually struggling to keep up with it. This will have resulted in some exports getting stuck (either at 2% or 95%, for some reason) on some audits, some of the time. If I had to put a number on it, I'd say it affected precisely 4.1279% of all exports carried out.
#8 Typo fixed in BOM Hint
A couple of days ago I shared something cool with the world - the BOM hint that Sitebulb triggers (hat tip to Glenn Gabe for his work on this). Whilst most were suitably awestruck, one individual took it upon himself to publicly shame us for our child-like spelling mistake.
What gives? Nerd.
10 new features and 8 fixes in one update?! A big one indeed (that's what she said).
Released on 13th October 2017.
- We've made crawling and processing data more efficient, so Sitebulb will now crawl faster, and on most websites, is better able to approach the max URLs/second limit you set. We've also changed the way crawl speed is reported, switching to a cumulative average, and added a new metric 'Current TTFB.' If you see Current TTFB increase, you will see a decrease in crawl speed, for the two are as entwined as a dragon queen and a Northern bastard.
- Since speed is on the agenda, we've also improved the way the user interface queries data, making it imperceptibly faster. You're welcome.
- It appears that no one is really using the report exports we spent a million hours trying to get right (thanks, guys). So we have tried to make them more visible by building a 'Bulk Exports' page, which explains each one and allows you to download them all (including a 'download all' button as well).
- Since we needed to add another button, we also figured you'd appreciate a massive UI change. If you look to the top right, you'll now find Filtered URL Lists, Bulk Exports, Printable Reports, Crawl Maps and All Hints.
- Advanced settings now pops up underneath the main settings when you click the 'Advanced Settings' button, instead of shifting you to a new screen, which some users were finding unclear (and we're all about clarity here at Sitebulb).
- Occasionally, if a server hates you crawling it, it will return a 429 HTTP status (too many requests). This is the server's was of saying 'would you kindly fuck off now?' Sitebulb will now take this advice, and stop trying to crawl the site.
- Sitebulb was claiming the hreflang nn-no was invalid, when it is in fact perfectly valid (nn is Norwegian Nynorsk, as you were no doubt already aware). Tilgje meg.
- The Indexability export was not pulling through the Indexability Hint data, which is should have been. This is because it was still looking for the 'indexation' data, which was renamed recently to avoid Barry Adams losing his shit every time the word was uttered.
- Sitebulb was forgetting crawl and analysis settings when re-auditing, which was super annoying. Like when you go upstairs to get your phone charger and get distracted by a fly relentlessly smashing itself against a closed window and in a blind rage you chase it across the house with the sport section of Saturday's Guardian, swinging aimlessly (and frankly, embarrassing yourself) and you have no idea what you were writing about when you first started this sentence.
- Tidied up a few of the exports, which had got a bit unruly.
Released on 3rd October 2017.
- New feature alert! Printable PDF Exports are now in town. You asked (again and again and again), so we delivered. Since this is a real big boy update, it deserves more than a snarky comment from me, so it's got it's own fully fledged user guide. Because my little comment isn't good enough.
- I mean, let's be frank, the user guide literally says 'click this button and save it', you can't work that out on your own?
- I Am Jack's Inflamed Sense of Rejection.
- Mac users! You can now hide the app with command+H, and quit the app with command+Q. Now, please, enough with the death threats ok?
- Regex enthusiasts, we've got something for you too! You can now filter columns using regex. It's a bit slower than the normal filtering method, but at least it's there now, ok?
- Adjusted the Dashboard to allow you switch between Projects/Imported Audits/Paused Audits/Queued Audits/Interrupted Audits, which makes it a lot clearer which state each audit is in currently.
- Added a link to our new Feature Requests board - in the left navigation on the dashboard.
- Adjusted the robots.txt checking, so that it now treats this sort of rule: Disallow */example-path/ in the same manner as Google. Note that this is not listed as a fix, because Sitebulb has been doing it correctly this whole time. Robots commands should always start with a / but it seems Google knows what you are going for so just lets it fly. Sitebulb will do the same, but only because it wants to. A dragon is not a slave.
- Fixed an annoying issue that has been lingering around for a while, but we couldn't find out what was causing it. From the Audit Progress screen, if you viewed Realtime URLs and then clicked the back button, Sitebulb would start another audit. Yeah, it's not really meant to do that.
- This one stumped a few of you as well (sorry). The hint HTTPS URLs links to HTTP was wired up totally wrong. Y'all were emailing me asking what you'd done wrong, and it was not you, but us. Stop blaming yourself. It isn't your fault!
- URL Details were not showing nofollowed incoming links. This is fixed now. Page Rank Sculptors, breathe easy.
- Adjusted the duplicate content checking algorithm to stop it flagging some false positives. It was claiming some URLs were duplicate, that had completely different content on them. Hardly Panda arousing.
- Improved checking for empty or missing H1s, as it was missing some in certain circumstances.
- Fixed a typo on the setting page, we'd gone all Slumdog 'Who Wants to Be a Millonnaire.'
- Over the last week we've had a few instances that have experienced database corrupting, which is a massive ballache. We've not been able to recreate this issue, but we've made a couple of critical updates to the way we write and read data. Fingers crossed that's it sorted. Symptoms of this are: Sitebulb doesn't fucking work at all. Please let us know if this happens to you.
Released on 27th September 2017.
- We've added a new all singing all dancing Hints section to the Overview so you can see exactly how many Hints were triggered across all the reports (like this). Mind = blown.
- Also added a little flag to show how many Hints were triggered on each report (like this) so you know when you done fucked up. If you happen to actually be good at your job, you'll earn Sitebulb's respect (like this). By the way did you see whose site that was? Not so cocky now are you?
- We have adjusted the Hint 'Contains links with no anchor text', so that Sitebulb now only looks at internal links, where previously it was looking at any old link. This was adjusted because all Sitebulb users that we surveyed said they give zero fucks about optimising anchor text on external links.
- The Link Equity Score has been changed from having 4 decimal places to 2 decimal places, because brevity > precision 99.7645% of the time.
- When you pause an Audit and go to 'Update Settings', we've added a cancel button down at the bottom, because one user reported being frightened by the Save button (note: those may not have been his exact words).
- When you export any of the charts as images, they will now contain the name of chart in the filename, rather than the accurate-though-pretty-fucking-useless alternative: 'chart.png.'
- Fixed a crazy error which happened if you enabled Site Speed and looked at the export, where there would be tons of additional 'B Score Worksheets'. I'll be honest, it wasn't really meant to do that, so now it doesn't.
- In a CRO experiment, we tried to encourage users to sign up for multiple subscriptions by opening 31 browser tabs when you clicked 'Update to Pro' (when on a Sitebulb Trial). Astonishingly, the experiment failed.
- On some URL lists, if you added/removed columns, Sitebulb would totally ignore you and just displayed whatever it wanted to show, like a petulant teenager. We told Sitebulb to stop being an obnoxious little shit, and this behaviour has miraculously improved (for now...)
- The protocol column was displaying everything as http, even when it was actually https. This is because Gareth, in his infinite wisdom, had trimmed the database column down to 4 characters, because he's stuck in 2007 and literally 'forgot' about https. In other news, 2007 was TEN YEARS AGO. We know it's shocking but don't worry, we're here for you.
- Fixed a small typo that one of our users noticed - in the settings tooltip for XML Sitemaps - it said 'and' instead of 'any'. I was tempted to change it to 'anal.'
- In the 'mixed content' Hint, we'd built the export to include a slash (/) in the filename. Anyone who's ever used Windows ever knows that WINDOWS HATES SLASHES. We should have known better.
- Some servers were throwing a hissy fit when reading our Accept request headers and responding with a Internal Server Error 500 (which meant you couldn't even do the pre-audit, never mind crawl the site). 99.9999% of you would never have noticed, but we've fixed it for the 0.0001%, because precision is very important at all times.
Released on 18th September 2017.
- Added some more data to the hover state on Crawl Maps. Now, when you hover over a node, you'll also see the page title, 'First found on URL' (i.e. the parent URL) and the Link Equity Score. Because more is always better.
- Changed the name of the 'Indexation' report to 'Indexability' instead. Internally referred to as 'the Barry Adams update.'
- Added the option to specify a custom user agent. You can do this either from the Advanced Settings for a single Audit (comme ci) or from the global settings for every Audit (comme ça). Et voilà!
- When you queue Audits, you'll now see a red icon alongside the 'Queued Audits' left navigation button, which shows the number of queued Audits you have. So it will show 1 for when you've got one queued Audit, and then 2 when you've got two. Basically it will increase by one every time you queue another Audit. You see? One more every time. It's just math(s).
- When you terminate an audit, the old warning message we had was 'Terminate the audit and delete data collected'. Some people thought we were bullshitting about the 'delete data collected' bit and called our bluff, only to be bitterly disappointed when their data wasn't there. Since 'I told you so' does not make for a favourable customer experience, we've re-jigged the warning message to make it clearer what will happen. We included some words in bold and everything.
- After our previous update a number of beta users were unceremoniously dumped onto our 'free' tier. They should have ended up on a 'Free Trial', which is a different thing entirely, and is pretty much indistinguishable from beta. Remember that 'soft launch' we were going on about? Yeah, this is why.
Released on 14th September 2017.
- Period fans, rejoice! (no, not that kind of period). I mean these little badgers ->. We call them 'full stops' in good old Blighty. Anyway, as per beta feedback, you can now have a period in your Project name, should such urgent need arise. Read that last sentence as you see fit.
- Several months ago Gareth and I had a very lengthy 'meeting' about whether we should use the magnificent 'log in/out' or the banal 'sign in/out'. The support for both options was fierce, and the argument raged on into the wee hours. Bloodied and limping, Gareth finally announced 'sign in' as the victor, before collapsing in a heap. Despite losing an arm in the battle, my belief in 'log in' was unwavering, and I managed to sneak her onto the 'Account' screen in the tool. It would be an understatement to say I suffered Gareth's wrath when he recently realised. I remain, persisting in a swimming pool of catharsis, overcome with grief for my fallen friend, as 'log in' is no more :(
- Added a 'Support' button to the top right of the user interface, so you can always easily get in touch with support.
- Curiously, Sitebulb had stopped asking for authentication details when trying to audit a site which uses HTTP authentication. The sharp witted among you will realise this means it couldn't actually crawl these sites, which is a bit of a problem for a crawler product like Sitebulb. It will once again ask you for authentication details (which are then saved against future re-audits, and available via Advanced Settings - FYI).
- In very specific conditions, the HTML parser was breaking, and it isn't anymore. That's all we know.
- When you Pause an Audit and go to 'Update Settings', we had a devious Back button on that screen. This duplicitous fellow would sometimes take you back to the Audit, and sometimes back to the Dashboard. The wily trickster was proving difficult to control, so we banished him completely (don't worry, you can still return to the Audit by pressing 'Save Settings' at the bottom of the screen).
The Beta Notes
Version 1.0.19 (beta)
Released on 4th September 2017.
- Added a new global settings area (Dashboard -> Settings), which currently looks a little sparse, having only one thing in it, but in the future we will use this new real estate to house some important global settings some of you have been asking for (e.g. a global proxy setting). Not to get ahead of ourselves, the current global setting we have just added is 'Default User Agent'. This allows you to over-ride the default User Agent that Sitebulb uses (handily named 'sitebulb') and instead choose whichever one you want! This will then become the User Agent used for the little pre-audit check at the start, and the default for all audits, unless you change it in Advanced Settings. Muy emocionante!
- Very occasionally Audits would get interrupted, because your computer would restart, or the internet would drop out, like when it's 1999 and you're playing jittery-as-fuck Grand Theft Auto online against your mate on your 56k dial-up modem, and the line would drop because your mum picked up the phone so she could ring her sister, who she just spoke to yesterday anyway. These sort of Interrupted Audits should always allow you to resume, but sometimes they wouldn't, as Sitebulb would mis-classify them as Errored Audits, and the 'Resume' button would be missing. This shouldn't happen anymore. GOURANGA!
- Fixed a very annoying issue where pause/resume would cause Sitebulb to skip the rest of the crawl and move onto building reports once you pressed resume. I say annoying because this would only happen if the number of queued URLs happened to be larger than the crawl limit, and I'd tried many many many MANY MANY MANY many tests before I realised it was this specific scenario that was causing it. In fact it wasn't even me who realised it, it was one of our magnificent beta testers. I might as well give up and go home now.
- Related to the above, we also fixed an issue where the crawl limit would get accidentally triggered early and stop the crawl (it wasn't strictly pause/resume related, yet did result in a similarly embarrassing premature climax).
- In an Audit, if you were on the Overview and you searched for a specific URL and then clicked through to view the URL Details page, you could then not navigate properly within the modal window. Specifically, if you went to look at Incoming Links, for example, you then couldn't go back to the 'Overview' of the URL Details. According to Gareth this is because the URL Details view opens in a modal, and so Sitebulb would have two references to 'Overview', incorrectly believing it was already on the modal Overview. I suggested it would be straightforward to just rename one of the Overviews, to 'Jimmy' perhaps, or 'Roger Federer'. But he decided to fix the underlying problem instead. Spoilsport.
Version 1.0.18 (beta)
Released on 30th August 2017.
- To make crawl limits more transparent/understandable, and to prevent unnecessarily saving tons of unwanted data on bigger sites, Sitebulb will no longer save disallowed URLs by default. If you want this data, you can switch on a new setting in the Crawl & Audit Setup called 'Save Disallowed URLs.' Good name huh? I came up with that. Ok I didn't really, but I could have if I wanted to.
- Improved the speed that Link Equity Score is calculated. Yes, you could argue that it was simply TOO SLOW before and we have simply fixed the thing that was making it slow, but I'm a glass half full kinda guy.
- When aforementioned Link Equity Score was taking fucking ages on larger sites, users were trying to go back to the Dashboard and look at other Audits, as is their wont. But they were getting the Spinny Spinner of Death, followed by the Grey Screen of Death (just an empty Dashboard). When this happened to me, I asked Gareth WTF was going on here, and he said it was because locks. So now you know.
- Follow-up: when I asked him to explain it in non-geek, he said that basically, when Sitebulb was calculating the link graph (for the Link Equity Score), it was reading and writing to the database, and locking the database from being accessed by other areas of the tool, such as the bit that houses the reports. He re-jigged the locks so that it only locks a certain bit of the database, and doesn't lock all of it. Because locks, QED.
Version 1.0.17 (beta)
Released silently (like a ninja) on 29th August 2017.
- Totally groovy new Hint for the Links section: 'Only has links from URLs that do not pass Link Equity'. This is for obscure situations where you have URLs which ONLY have links from canonicalized URLs, for example. I know it doesn't sound like much now, but it'll be there for you when you need it most. Like that old bottle of Baileys in the back of the cupboard, leftover from a work Christmas party several years ago, which has almost definitely gone off and you can barely open because the lid is crusted over.
- On website, updated 'Changelog' to 'Release Notes'. Because meta.
- There was an issue where Sitebulb didn't want to crawl sites protected by Incapsula. It wouldn't even perform a HTTP header check on external links to sites protected by Incapsula, FFS. This was completely unacceptable, so we didn't accept it.
- In the Links report, the anchor text would crash into the side of the page if it was particularly long (e.g. if you had a link with anchor text: "I'm here with my dad. And we never met. And he wants me to sing him a song. And, um, I was adopted. But you didn't know I was born. So I'm here now. I found you, Daddy. And guess what? I love you. I love you. I love you." - that would cause the anchor text to crash). Now we just use ellipsis...
- Changed the way included/excluded URLs are handled on the crawl progress screen. Depending on the website, and the inclusion/exclusion criteria, the numbers recorded on the crawl progress would jump around like a mother fucker. This was to do with the way they were being counted and recorded by the software. To make life easier for everyone, we no longer count them at all.
- If you had a double slash in the URL (e.g. movies.com/is-die-hard-really-a-christmas-movie//discuss) this would trigger BOTH the Hint 'URL contains a double slash' AND 'URL contains repetitive elements', since the double slash was being classed as a repetitive element. Whilst not technically wrong, this wasn't exactly useful, so we've changed them now to be properly unique. The answer, by the way, is clearly YES. I don't even know why there's a discussion about it.
- I know most of you haven't noticed, but there is a 'Feedback' tab in the left hand navigation. On there is a feedback box (I'm explaining because you probably haven't seen it) which you're supposed to write feedback into. If, on the Mac, you tried it drag and drop an image into this box (the one that is meant for feedback), the image would take over the screen and lock Sitebulb completely. It no longer does this, it just rejects the image outright. Not that you'll notice anything different anyways.
Version 1.0.16 (beta)
Released on 25th August 2017.
- You know how, once or twice in your lifetime, you have to admit that you were wrong about something. Well, this first update is one of those moments for me. We've introduced a new metric, 'Link Equity Score', which is a score from 0 to 10 for every internal URL, and is based on an internal linking algorithm. Gareth put this in the tool about a year ago, and then at some point I decided it needed to go (I can't even remember why). But it's back, by overwhelming public demand, and I was forced to utter the unholy words 'I was wrong'. I hope you appreciate my sacrifice.
- If you know anything about SEO, you'll know that Alt text on images is Kind of a Big Deal. Sitebulb has always had the Hint, 'Images missing Alt text', but up until now, has refused to tell you what these images actually are (you tease, Sitebulb). Now, however, we've moved the Hint to the 'Page Resources' section, and given it the respect it deserves. From the URL List associated with the Hint, you can click through to see the images associated with each page (check it out). Even better, in the Page Resources export you can see the same data, but this time grouped by image, with counts and example images (check it), so you can easily pick out template images that are on lots of pages.
- On the Overview, Sitebulb will sometimes display messages like 'Audit stopped early because crawl limit was hit.' When you only get one of these warnings it looks ok, but sometimes multiple warnings will trigger, so you'll get 2 or 3 of these boxes, and it looks jangly as fuck ('af', if you're down with the kids). So we've added a nifty X button in the top right hand corner of each box, so you can dismiss any that displease your eyes.
- Moved the crawl setting 'Maximum URLs to Audit' onto the first screen of the settings, underneath the Crawler Type selection. This is to give you, dear user, more transparency and control when it comes to bigger Audits. We have also fixed an issue where the maximum URL limit was incorrectly including Disallowed URLs in its calculation (this is really a Fix and should by rights go in the list below, but I do what I fucking well like and don't let anyone tell you otherwise).
- Sitebulb will now parse XML sitemaps that are compressed, so feel free to compress sitemaps to your heart's content. Middle-out for the win, Bro.
- Sitebulb will now pass HTTP headers when content fails to load, so it can at least tell you something about the URLs.
- Added fancy.com to the list of global excluded domains, because if you Audit a site that links to a unique fancy.com share page on every single URL, it's fucking annoying, trust me.
- In the previous update we added a clever little change for the mobile rendering checks, which essentially used just a sample of pages rather than the whole lot. In doing this, we managed to fuck everything up. The data was being collected but never saved, and instead duplicate URLs were just inserted into the database! Cue many reports of odd looking issues, such as duplicate content Hints, missing GA codes and the like. If you've seen anything that sounds similar... then this is probably why.
- On re-audits, Sitebulb was going rogue, resetting the settings for crawl speed and maximum URLs, even if you'd changed them previously. We reined Sitebulb in, so it's no longer doing this. Valar Dohaeris.
- A few users reported an issue where the tool was occasionally crashing when switching between the Crawl Map and the Overview. We believe we have fixed this now, but we are not 100% sure. If it happens to you, it would be awfully good of you to let us know about it.
- AMP URLs were being counted as duplicates, because canonicals on AMP pages were not being classified correctly. I'm so sorry.
- Kinda specific, but we fixed an issue where Sitebulb would refuse to crawl sites where the server returned the Status Code as '200' AND the Status Description as '200' - Sitebulb was looking to find 'OK', and would throw a shit-fit if it didn't.
Version 1.0.15 (beta)
Released on 7th August 2017.
- Quite a few people asked us to make the Crawl Maps more integrated with the rest of the product. So now you'll find the Crawl Maps don't open in a new window, but within the main Sitebulb window. Not only that, but you can now click any node and be taken directly to the URL Details page for that URL. Wait, there's more... you can also now scroll out so far that your Crawl Map becomes a dot (I don't know why people want to do this, but it was a feature request). AND we've also moved the 'Save Crawl Map' button so that it doesn't print onto the captured image, which I think we can all agree looked pretty shit. The eagle eyed among you will also notice that we've tweaked the spacing a bit for how Crawl Maps display, which seems to make most Crawl Maps a bit more legible, but if you think we've made them worse please tell us!
- We've added the option to update crawl and audit settings while an Audit is paused. When you click 'Pause' you should see an 'Update Settings' button alongside 'Resume' on the top right. Bear in mind that whatever settings you change will only come into affect for new URLs. So if you click 'Site Speed' halfway through, then the speed analysis will only come into play for half the URLs. Similarly, if you untick 'Crawl Parameters', then no new parameterized URLs will be added, but there will still be some in the queue that will get crawled. Please make sure you read and remember this forever so I don't have to keep answering the same fucking questions about why it's still crawling parameterized URLs.
- If you've tried crawling a Shopify site with over 500 internal URLs, you may have noticed that those rat bastards eventually stonewall you with 430 HTTP errors (TOO MANY REQUESTS). It only seems to kick in around the 500 URL mark, so if you have a 1000 page site, just delete half your content and you're sorted... Alternatively, you can crawl at 1 URL/second, which is slow enough to stop Shopify blocking you. In the pre-audit, Sitebulb will detect if it's a Shopify site, and give you a cheeky little message to tell you to slow down, lest you blow your load early.
- The big one - searching for Projects/Audits in the search bar top right will now also search imported Audits. Whoa.
- While looking at reports in the Audit, the report you are 'in' will now be a darker shade of grey, to help you understand 'where you are.'
- Sitebulb also now matches on the URL (www and non-www version) when trying to match the Google Analytics code to pre-select the right View, although the UA- code will take precedence, if found (see above).
- Sitebulb is now checking a few more places to find sitemaps to include in the pre-audit. I could tell you where we check, but I'd have to kill you.
- If you delete a Project and then want to start a new Project with the same name, it wouldn't let you do it ('you already have a Project with that name', which WASN'T TRUE!). We coerced Sitebulb to relent in this tyrannical nomenclature.
- The Hint 'URL received search traffic but 0 goal conversions' was showing a negative number, instead of 0. We conducted some research on this topic, and actually found that it is impossible to have negative URLs in real life, so we've changed this to show 0 instead.
- Some URLs were being incorrectly flagged for the Hint 'Set mobile viewport.' This was a coding error by Gareth, as it happens. Rest assured that he has been soundly thrashed.
- There was a tiny little typo on one of the compare audits fields - an extra, unwelcome lower case 'k' - which has now been forcibly rejected.
- The exclude URLs function, and the robots.txt parser, were not working correctly when handling wildcards (*). They are now.
- Astonishingly, no one noticed this bug, despite it being there from day dot: when doing pause and resume, you'd end up with some duplicate URLs. Whilst we've fixed the duplication issue, we can only conclude that no one's even looking at the fucking data, you're all just distracted by the pretty graphs. FML.
- For Site Speed Hints, if the affected URL figure was more than 1000, it would change to 1k, which then did not look like a number and so you wouldn't get the buttons on the right to click through and see the URLs. Now we do the lookup differently to avoid this issue.
- Not all images are created equal. Some images are embedded page resources, while some are anchored images that have their own image URL. Even if you don't select 'Page Resources', Sitebulb will still crawl anchored images and report on them. However, the reports weren't all hooked up right, so you'd click to view resources, and it would tell you that you hadn't turned them on! Touche, Sitebulb, but not very helpful.
- These self same anchored images, if hosted on a subdomain, would still get crawled even if you had unchecked 'Check subdomains', which was bafflingly annoying. This don't happen no more.
- Hint filters were wrong for the Hints 'Canonicalized URL received organic search traffic' and 'Noindex URL received organic search traffic', so the data looked like bollocks. Fixed the filters and now the data looks sweet as a nut (not that kind of nut!).
- Subdomains URLs weren't being identified when there was a protocol mismatch, which caused significant consternation for almost no one. Rest easy people, we've fixed it.
- If the start URL redirects, it is now updated to the final destination URL found by the pre-audit. I know what you're thinking, 'didn't it do that before?'. Apparently not.
- Fixed an issue where audits that got Interrupted (e.g. computer says no) were not always starting up again - they would say they were running, when they actually weren't (which was frightfully annoying).
- Sitebulb would forget what render timeout you set when re-auditing a site with new settings. It would remember everything else, but just forget this. We added 'Arya Stark mode', and now, The North Remembers.
- Google Analytics code was getting reported as not there, when it was there really.
- We experimented a bit with the selection options for 'results to display' on URL Lists. We tried 1000 and 500, but both of these would make Sitebulb hang occasionally on bigger audits. So we've abandoned this and gone back to the stable max option of 250.
- Fixed and issue where Audits were skipping the current process when resuming. We actually already fixed this bastard problem once before, but then something else started causing the same issue.
- If the tool crashes, as is its wont occasionally, the scheduled URLs (max 300) no longer get dumped/lost, so Sitebulb can pick up and recover more smoothly.
- Fixed issue where some exports (e.g. URL Details) were not generating properly. They were getting to 2% and then stopping indefinitely, which is no use to man nor beast.
- Fixed issue where some exports (e.g. URL Details for external URLs) were not saving when you chose 'Save and Open'. They'd just sit there, like a fucking lemon.
- Can now crawl Medium sites without adding the twatting 'GI' query parameter to every URL.
- When filtering URL Lists for 'Content Type', it would only show a few options. Now we show all the options. There's too many, if you ask me.
Version 1.0.14 (beta)
Released on 21st July 2017.
- When using the software for the first time, users were not prompted to create an account. This meant they could use the software up until the point of creating a Project, at which point a license check would fire, and the software would enter a never ending state of blind panic. It was the shittest back door ever invented, and now it's been removed.
Version 1.0.13 (beta)
Released on 20th July 2017.
- The one you've all been waiting for... Added the 'Content Type' column to a bunch of Hints: URL contains whitespace, URL contains upper case characters, URL contains repetitive elements, URL contains non-ASCII characters and URL contains a double slash. I know right? Holy fuckballs.
- Fixed the timeout issues which was causing Sitebulb to shit itself. If the tool has hung or crashed for you since you installed 1.0.12, this is probably why.
Version 1.0.12 (beta)
Released on 19th July 2017.
- As promised, Mac users can now happily dock the app again, without fear of Armageddon.
- Train mode - you can now view completed Audits while on a train (note, you can only start new Audits if you have access to the internet on said train).
- Plane mode - you can now view completed Audits while on a plane (note, you can only start new Audits if you have access to the internet on said plane).
- Offline mode - you can now view completed Audits while not connected to the internet (i.e. you no longer need to be on a train or plane to take advantage of this feature).
- Sped up the pre-audit process for some websites. Websites, in particular, with the defining characteristic of being utterly shite (e.g. bloated or broken HTML).
- Optimized HTML parsing. Translating from nerd language, this just means that when Sitebulb comes across crappy HTML, it can pick out the bits it needs and ignore all the junk. Sitebulb was previously handling this stuff badly, so CPU would sometimes spike when it came across a few of these shitty pages.
- Added a few default reply parameters to the 'Excluded URLs' tab in Advanced Settings. This stops the tool crawling 'replytocom' URLs, or other such worthless chuff.
- We dropped a bollock on this one - a bug that meant if you paused the Audit while it was crawling XML Sitemap URLs, and then hit 'Resume', it would skip the crawling and move on to building reports. It would also do the same for GA and GSC... which might explain some issues where users were getting big lists of 'uncrawled URLs' with no reason whatsoever.
- Fixed all the annoying issues with closing, opening, double opening and crashing the tool. In short, Sitebulb has an application element, which does all the work, and a separate user interface which handles the presentation. We like to be original, so we refer to these as 'the App' and 'the UI'. What we have done is basically made the App and the UI talk to each other better, so each one knows when the other is open, and when it is not.
- Fixed a pretty big issue on the Mac, where if you right clicked the app icon and clicked 'Quit', the application would no longer open again from the Applications folder. This is related to our change above to 'make the App and UI talk to each other better'.
- Fixed a few UI bugs that were throwing errors in our bug tracker, which meant we couldn't see the wood for the trees. You won't notice anything different, but rest assured we've diligently fixed them.
Version 1.0.11 (beta)
Released on 14th July 2017.
Attention Mac Users: Don't run from the dock
We have had to temporarily disable the option to run the program from the Dock. So if you have it docked already and install the new version, it WILL NOT RUN FROM THE DOCK.
Even if you dock the new version, it won't run from there. It will just show you the spinny loader thing. We're fixing this, but it's quite complicated, so bear with us.
- Lots of users were asking for a button to export and save the crawl map (apparently some people haven't heard of Jing). Anyway, button = added (which saves the crawl map as a PNG).
- This one is very cool. You can now highlight any text in the tool (from almost anywhere) and copy it. Anything: Hint copy, URLs from URL Lists, Titles, hreflang, etc... Highlight the text, right click, then press 'Copy'.
- If you click on a Hint, then export the data from the URL List, it now passes the Hint name into the name of the exported Excel spreadsheet (e.g. sitebulb_com_has_upper_case_characters_20170714102236.xlsx). Word up to that.
- Added a natty check to see if the Project name you've entered has already been used. If so, the user is warned to buck up their ideas.
- One confused beta tester thought that Sitebulb was crawling external links and shooting off to crawl the whole entire internet. Fret not, friends, it only does a HTTP status check on external links. It does this by default, but you can switch off external links in the Advanced Settings. We've updated the copy for this to make it clearer for said confused tester.
- Within Projects, if you hover over each Audit in the Website Audit History you will now see a little box showing details of the crawl (success, error, etc...). This will help you tell one Audit apart from another, it really is quite a handy feature. I wish I'd thought of it myself to be honest.
- Increased the maximum view to 500 rows, in the URL lists. Apparently 250 wasn't enough for some people.
- In the Links report, the 'Top Pages' tab in the tool shows the top 50 pages based on internal links. In the export you also got 50 pages, which seemed a bit paltry. So we upped it by 24,500 (in the export only).
- Added a new global setting (Dashboard -> Settings) called 'Excluded External URLs.' This is pre-loaded with a bunch of hosts and paths that, most of the time, there is no benefit to crawling (e.g. instagram or pinterest on every single page of your site). You can delete, change or add new options, should you wish.
- Sitebulb is now saving the version number of the software against the Audit, so it will warn you if you open an Audit from an older version of the software. In case you see some mistakes which we've already fixed. You might refer to this document in fact to find out what's changed (and that way someone will actually read all this shit).
- Added some comments to Hints related to Google Tag Manager to state that they might be redundant (e.g. To stop this: "there is no GA code - yeah but there is a GA code in GTM so fuck off").
- Similarly, added some comments to some of the Performance Hints which are redundant with HTTP/2. Because people kept having a go at us for being behind the times. We are nothing if not down with the kids.
- Added the 'HTTP Error' column, and added some additional clarification, for all lists of 'Errors' in the tool. One user found some URLs that returned a 200 status code, but were being listed as errors. This is because the content had failed to download, and it wasn't clear why, or if the URLs were being incorrectly classified. They weren't, but Sitebulb was just doing a shitty job of explaining what was going on. Naughty Sitebulb.
- Changed the 'multiple H1s' Hint from 'Issue' to 'Advisory', as one user pointed out that in HTML5 you could have perfectly valid multiple H1s (for different sections, for example). Since we still think it could help indicate an issue/misconfiguration, we've kept the Hint in.
- Removed the ability to use any special characters in Project names, as they were getting cut off or causing the software to crash. So no more Projects named 'This client is a c#£$!', sorry.
- Some Crawl Maps were coming back with a single node. While there are some legitimate reasons for this happening (e.g. homepage is canonicalised), in the examples we looked at it should NOT have been happening. Since the Crawl Map is my favourite feature, and a single node Crawl Map is as underwhelming as an England performance in a World Cup, we prioritised this fix above all others. Well, I wrote it here above the others anyway.
- There is an 'anti-sleep' function built into Sitebulb that means your computer won't sleep or hibernate while it is crawling. This was not kicking in properly on the report building element, so your computer would go to sleep. And crash Sitebulb. So you could spend 2 days crawling a 1 million URL site, then it would crash out at the last bit. Sorry if this happened to you! Anyway, 'insomnia mode' now stretches over the entire duration of the audit (note that your computer will still sleep when not running an Audit, and it still allows your display to turn off).it
- On the Audit Overview, we had been pulling back the number of indexed pages in Google, based on a site: search. Since this uses your local machine, it does the check using your own IP address. If you happen to be a massive spammer and blocked by Google, this check will show 0 pages indexed. Since some of our beta testers are massive spammers, they were seeing 0 and thinking the tool was buggy. We can't have that, so we have removed this feature for now. This is why we can't have nice things.
- Changed the Hint text slightly for 'URL is orphaned', to clarify that is checking for internal followed links. Apparently it wasn't obvious to some people...
- Pausing Audits has been a bit of a pain in the arse. Despite previous changes, some people were still having issues, so we've done some more work to help the figures update correctly.
- Fixed a peculiar issue which occurred when you had a Sitemap index, which included XML Sitemaps that redirected (the example we had they were going from having no trailing slash, to having a trailing slash). Sitebulb would follow the redirect and tell you about it, but it would ignore all the URLs it found! Whoops.
- Imported Audits no longer have an associated Project. Well they never originally had an Associated Project, but you could try and click on 'View Project', which would crash the software.
- The XML Sitemaps export button had an identify crisis, and was claiming to be an AMP export. We gave it a dry slap, and it's come to it's senses.
- If you stop an audit early, but let XML Sitemaps complete, you won't get a Crawl Map. Don't ask me why, because I do not know, but it happened, and that is why I'm bothering telling you so. It is fixed, so don't fret.
- A few versions ago I mentioned that we'd made the UI fit better on really shitty old Windows 7 work laptops with tiny screens. Well this was true, except for one part of the onboarding, where you needed to click somewhere off the fucking screen, and would get stuck in the onboarding forever and ever. Well, we've fixed it now so the onboarding actually works on really shitty old Windows 7 work laptops with tiny screens.
Version 1.0.10 (beta)
Released on 10th July 2017.
- A biggie - we've rebuilt all the report building to be much faster, which in particular affects LARGE AUDITS. So if you think size matters, we invite you to smash Sitebulb's back doors in.
- In line with the update above, we also added an Advanced Settings option to 'not crawl parameters', which can make a regular sized website seem a lot larger than it really is (you can also specify some parameters to still be included). It's all a question of perspective, really.
- Added a cancel button to the Import Audit overlay, just in case you change your mind in those crucial few seconds while the Audit is actually being imported, as is your wont.
- A few of you came across this issue - if you had an incomplete Audit (e.g. paused or interrupted), you'd still have the opportunity to click 'View Latest Audit' from the dashboard. Since the Audit hadn't actually been created yet, this caused Sitebulb to throw a hissy fit. Now it will say 'Incomplete Audit', and you can go to the 'Paused Audits' screen to resume or cancel them off.
- Occasionally, when running larger Audits (a few thousand URLs), the main progress screen would just freeze. In the background it was still crawling, but the numbers wouldn't move. Similarly, if you went back and forth from the Dashboard to the Crawl Progress a few times, the UI would often freeze. We felt it would be better if the UI didn't occasionally freeze, so Gareth spent all weekend rebuilding the script for this, and it seems to have done the job.
- At the same time, we also wanted to fix the issue with pause/resume (that you would know about if you actually read the emails we send), where the UI wasn't updating new crawl numbers, and would revert back to the total when you originally paused it.
- On the Mac, re-running a previous audit using the same settings, for any Audit you ran before version 1.0.9 would progress to 102% completion, and only show you half the reports. This may well have been a developer error, and has now been resolved. Any Audits that were affected should have gone back to normal as well.
- One of our testers found a bug whereby, instead of queuing two Audits up, the tool would just crash instead. That isn't exactly the way it's supposed to work. We think this may have been caused by some slowdown to the computer or the network, and the jobs would not get queued up properly. So we've fixed THAT issue, and hope it solves the crashing thing too.
- An eagle-eyed tester spotted this: 'The "page has more than one GTM code" seems to fail on pages which only have one but use the current implementation method of <head> component and <body> component.' Fixed.
- The very same eagle-man also spotted this: 'The <body> GTM script has a frame which flags as fail for "page has no frames". If GTM is in use, it's likely this will always fail.' We deliberately left this misconfiguration in for him to find - an Easter Egg if you will. Our warg friend did not disappoint. Now we're not flagging anything with 'noscript.'
- The sample audit that we pre-load in (which most people delete without looking at) is not exportable. But we still had the button there screaming 'export me!', which, if you succumbed to its request, would duly make the software shit itself. So we took the fucking attention seeking button away now.
- In the Advanced Settings, if you were changing max depth or max URLs to crawl, you could click the down button a few too many times and set these as negative values. Now you can't go to negatives using the down button. You can still enter negative values using your keyboard, but if you do this then you are definitely a prick, and deserve everything you get.
- Fixed an annoying, rare onboarding bug that occurred when you viewed your very first Audit, if you tried to view a report that you didn't actually switch on.
Version 1.0.9 (beta)
Released on 5th July 2017.
- On a report, if you go into the Hint data and click through to look at some of the URLs, the back button will now take you back to the Hint tab you were on before (rather than just back to the report overview, which we can all agree was fucking annoying).
- Added Gravatar to our ever-growing list of automatically suppressed external links, so Sitebulb will no longer check the status of 62,000 near identical gravatar.com links. Soon we'll add this as a configurable global setting, so you can switch it back on (if you really want to check 62,000 near identical gravatar.com links).
- When the crawl has finished and all the reports are built, Sitebulb also builds a lot of the Excel exports that you find within the reports. Previously it would do a faux 'loading' screen, which was largely, if not completely, pointless. Now it just instantly gives you the file to download or open (FYI, in our dev queue, Gareth added this ticket as 'Exports are file aware', as if they are some sort of sentient being).
- If you switched on 'International' in the Audit settings, but Sitebulb couldn't find any hreflang, when you then went to view the International report, the tool would hang forever on the spinny loading thing. Obviously we have fixed this bug, but in reality you never should have been selecting 'International' in the first place, if you knew that there was no hreflang on there (and if you didn't know this, well, you should have know this).
- On Crawl Maps, when you hovered over the homepage it would claim that it had 0 children URLs. Other than single page websites, this was utter bullshit. Suffice it to say, we had words, and it is now a more responsible parent.
- Fixed small typo on the pre-audits checks, 'Web Frameworks' had gone full Jonathon Ross (Web Fraweworks).
- In the International export, one of the columns was misaligned on the hreflang cluster matrix. No one actually noticed this issue, but it doesn't mean it wasn't wrong (if a bear shits in the woods, can the Pope hear it?).
- On the Redirect report, the headers went missing on the top graph when you switched to the 'Data View'. Fortunately, we managed to find them and return them to their graph.
- Fixed (we think) a curious issue where Google Analytics would select the wrong property, meaning the view would be wrong. Only in rather bizarre circumstances would his happen, so it won't have affected many of you.
Version 1.0.8 (beta)
Released on 3rd July 2017.
- If you clicked to start a new Audit from an existing Project, but then had a change of heart and decided you in fact did not want to do a new Audit after all, you'd end up trapped on the overlay, with the only option to start a new Audit (which you just realised you no longer want to do). We added a 'close' button, so you will not longer be trapped thus.
- Changed a few bits of the new account creation workflow, which is nothing to do with the tool anyway so does not concern you. Stop reading this bit please.
Version 1.0.7 (beta)
Released on 30th June 2017.
No updates this time, sorry to be such an enormous disappointment.
- This one is a biggie... on the Mac, if you docked the app, when you next tried to run it from the dock, it stayed on a loading screen forever. To clarify: IT DIDN'T FUCKING WORK AT ALL. Thankfully that shit show was spotted by an eagled-eyed beta tester, and it now works as intended.
- My wife has a really shitty, old Windows 7 laptop from work, that has the tiniest screen you've ever seen. When I tried Sitebulb on there you couldn't actually click on some of the menu options from the navigation. It was an... oversight. So we optimized the user interface to still work on really shitty old Windows 7 work laptops with tiny screens.
- Some of the Site Speed statistics data was incorrect, and is now not incorrect. It was so incorrect, by the way, that nobody actually noticed. What is it we are not paying you beta testers for anyway?
- The filtered list for Broken Page Resources did not include the column 'No. URLs Referencing Resources.' This shocking omission has been duly corrected.
- One beta user complained that the Uncrawled URLs button says 'View' when it should say 'Export', so we updated this button copy to satiate his pedantry.
Version 1.0.6 (beta)
Released on 23rd June 2017.
- In URL Lists, the Add/Remove Columns selector panel was too large for smaller screens, and the bottom was getting cut off. So we made it smaller (size isn't everything y'know).
- For larger audits, when displaying Realtime URL data, there can be a delay loading and indexing the data. Previously we just had a blank screen, which looked really shit, so we added our ubiquitous loading spinner thing.
- Some unfortunate beta users found themselves in a disastrous install-restart-uninstall-restart-install experience with 1.0.5. To avoid this, we added a version number to the installer, and instructions to uninstall old versions.
- Previously, if you did an Audit on a website using a really long Start URL, when displaying Projects on the Dashboard the URL column would be so wide that you could neither see nor click any of the action buttons on the right. We didn't think this looked very good, so when we encounter long URLs now we use ellipsis instead (...).
- Sitebulb stores crawl data on your hard drive. If you don't have enough space left on your hard drive, Sitebulb has nowhere to write the data to. Up until now, Sitebulb has handled this poorly, repeatedly smashing its head against a brick wall and spamming our bug tracking software 57,000 times. Now, however, Sitebulb gracefully crashes if it runs out of space (*ahem* user messaging to follow).
- We added a new 'Realtime URL' view so you could see the crawl progress in a URL List. But we left in the Column Selector, which wasn't that smart, and would have allowed you to query the database as it was being written to. So we removed it.
- We also removed the onboarding from this Realtime URL view, because you probably don't need it.
- MAJOR change to the UI - we moved a couple of the left hand menu items around on the Dashboard, so that all the 'Audit' items were adjacent (this was our most requested update).
- In the Redirect report, we'd forgotten to put a legend on the stacked area chart (the one at the top), so that is now in.
- Also on the Redirect report, we found occasions where the two pie charts were null (had no data), and displaying a pie chart with no pie is just not cricket. So now we just don't show them when there is no data.
- One of the Advanced Settings options is called 'Included URLs'; we've just added some examples to the copy to make it clearer how you actually use this function.
- In some of the 'Hints' sections, we split up the Hint into Indexable/Not Indexable, and the buttons on the right allow you to view the associated URLs. Apparently this was not at all intuitive, so we made it marginally clearer by adding a barely perceptible hover state tooltip.
- If you did not switch on one of the Analysis options when setting up your Project (e.g. you did not tick 'XML Sitemaps'), you would not get to see the corresponding report in your Audit. However we were still producing an empty export for you to not look at, which is a massive waste of resources and entirely unnecessary. So now we don't produce the export, and we've even removed the redundant 'export' button to boot.
- For Sample Audits, link data is not processed, but Crawl Maps are. Normally Crawl Maps include some link data about each URL when you hover over a node, but since link data is not switched on, these overlays were full of 0s, which was confusing and stupid. So we've just stripped out the link data from the overlay on Sample Audits (and also on Standard Audits where 'Link Analysis' is turned off in the Advanced Settings).
- Added the text 'Redirect Loop' to URL Details page for URLs that are in a redirect loop, to make it clearer that they are in a redirect loop.
- Some of the graphs on the International report were not hooked up to URL Lists correctly. When performing hreflang checks, Sitebulb is only looking at Indexable URLs (which filters out a lot of the noise). This 'Indexable' filter was not getting correctly applied, but now it is.
- The Pagination graph on the Internal URLs report was not hooked up to the URL List correctly. In fact it was completely broken, and is now completely unbroken.
- Ditto a bunch of graphs on the Search Traffic report (they are now unbroken too).
- Fixed issue where incorrect data was being displayed for 'Not Indexable XML Sitemap URLs' (it was showing Indexable as well, which is just plain wrong).
- In URL Lists, the Meta Description (and other fields with many characters) often displays a 'read more' eye icon, which shows the full copy when you hover over it. This hover function had stopped working, and just looked like a weird emoji character. We fixed the hover state.
- One of the Advanced Settings options is called 'Excluded URLs'; which contained a disgraceful typo in its copy ('different' instead of 'difference'). This was confusing users left right and centre, so we had to fix it pronto.
- When you close Sitebulb while a crawl is running, the modal window that displays had a button that was slightly too far left. You'll be pleased to hear this has been shifted to its rightful place (on the right).
- In the Indexation report is a warning message for when your robots.txt redirects elsewhere. This message was not firing correctly, so users were seeing it when they shouldn't have, and they were understandably up in arms about this. The problem has been resolved and order has been restored to the realm.
- Resolved obscure issue when re-auditing a site with deleted Google accounts - Search Traffic report is no longer generated (which would probably never actually come up in real life).
Version 1.0.5 (beta)
Released on 21st June 2017.
- Split Links report up by adding new 'External' section, so Links report now only contains internal link data.
- Added new Redirects report, and canonicalized data about redirects to only live in this report.
- Added option to 'View Realtime URL Data' while crawl is in progress.
- Added Filtered URL Lists, which is now the top menu item in the Report. These are pre-filtered URL lists so you can quickly jump to e.g. Internal Indexable HTML URLs.
- Added link from URL list to view all internal linking URLs to each URL (screenshot)
- Changed 'Performance' report to break out into two - 'Site Speed' and 'Mobile Rendering', now both available in the left hand menu in a Report.
- For Site Speed Resource Hints, added 'View URLs' link to column 'URLs Referencing Resource', so you can see exactly which Resources are problematic (e.g. JS not minified) and which URLs they affect.
- Added Indexable/Not Indexable counts for all Hints where it makes sense (screenshot).
- Added HTTP Status for outgoing links on single URL view.
- Moved some Hints from Links Report into Indexation Report. Links Report now only contains Hints that pertain to the distribution of link equity/PageRank (or lack thereof).
- Changed Hint icons to be actual words instead of icons (Issue/Advisory).
- Moved Project List to the dashboard front and centre - you can now view the project or jump straight into the latest audit. 'Recent Audits' is still accessible via the left menu.
- Changed default columns for all Hints and Filters.
- If you have some disallowed page resources, you will see a warning message on the Indexation report. This is now hooked up to the actual offending resource files (screenshot).
- Added messaging to Crawl Comparison page to tell you when settings have changed between Audits.
- On Site Speed report, added links to URL Lists from Insights for 'Slowest TTFB' and 'Slowest Download Time'.
- Added 'Internal Linking URLs' to all filtered URL Lists (including Hints and Insights).
- Added message to tell you when you have some paused Audits.
- Added new link to Dashboard for 'Queued Audits', which displays all Audits currently queued.
- Fixed issue where Sitebulb can't match up domain with GA account (now looking for UA code on page).
- Fixed issue where some resource sizes (in Kb) were not being collected properly.
- Fixed issue where the Hint 'Does not contain Google Analytics code' was being fired incorrectly.
- Fixed issue where some On Page Hints were sometimes showing false positives (H1 Not Set and Missing Meta Description).
- Fixed: Reading time now displays in mm:ss (instead of integers).
- Fixed issue on URL Lists where columns would default if you changed number of results to display.
- Fixed issue where Total Page Size was not being collected properly.
- Fixed issue on URL List where your filter resulted in only a few rows, where editing that filter was then impossible.
- Fixed issue where crawl comparison export stopped working.
- Fixed issue where Sitemaps Report filter 'Not in XML Sitemaps' was picking up non-HTML content types.
- Fixed issue where Crawl Maps data 'links from X% of URLs' wass using percentile instead of percentage.
- Fixed column alignment issue on URL Details export.
- Fixed duplicate content issue where redirects were not being checked - now URLs must be internal, HTML, status 200, indexable in order to be checked for duplicate content.
- Fixed issue where top pages would show external URLs.
Version 1.0.1 - 1.0.4 (beta)
Never publicly released, the very early beta releases are widely considered as Sitebulb's formative years.