One of the features in Sitebulb that people seem most excited about is Crawl Maps, which, if you've not seen them yet, are interactive visualizations of your website architecture.
And they look awesome.
Table of contents:
- Example Crawl Maps
- Flat site architecture
- Duplicate content
- Pagination chains
- Content Explosion
- Interactivity
- Attribution & Learning More
- Crowd-sourced Examples
- Manipulative
- Content Mess (Where to begin...?)
- Before and Afters
- Content Explosion
- Extreme Pagination
- Crawl Map Videos
- Trolling
- Nice site structure (huh? what?!)
- Calling all Crawl Map fanatics!
- Create Your Own Beautiful (or ugly) Crawl Maps
Example Crawl Maps
Basically, Sitebulb will take your crawl data and map it out using a force-directed crawl diagram, displaying URL 'nodes' as dots, with links represented by the connecting lines ('edges').
The result is an interactive graph that can be incredibly useful for technical SEO audits, often revealing patterns in the site architecture that you'd struggle to spot otherwise. Visual site mapping can be a great way to take a high-level view of your website's structure.
I'll stop talking about it, and show you some examples instead.
Flat site architecture
This is like a classic 'SEO friendly' flat website architecture, with almost every page no more than 2 clicks from the homepage. You're looking at the big green dot in the middle as the homepage, then the smaller green dots are at crawl depth/level 1 (i.e. they are linked to from the homepage).
The orange dots are at a crawl depth of 2. In this instance, they represent product URLs, and the depth 1 nodes are sub-category URLs.
If you haven't guessed already, this is an ecommerce site with a pretty extensive mega menu.
Duplicate content
Not an easy one to digest at first glance, but this is a significant case of duplicate content. In the upper left is the homepage (big green circle) and all the proper site content. But this also links off to two similar structures down at the bottom...duplicate homepages and in fact a duplicate website (twice over!).
Pagination chains
The long whip thing coming out the side is a string of paginated pages.
I've got another example of pagination actually, this one even weirder:
That one was actually caused by some legacy pagination markup that wasn't even being used anymore! (hence the 'bare branches' with nothing coming off them).
Content Explosion
I like this one a lot. Shared by one of our beta testers, Gareth Edwards from Wolfgang Digital. It shows a relatively small 'product' site, with a large and complex blog.
The homepage is the big green circle at the bottom, and everything coming down off that is the marketing site which lists their products and services. The little green dot in the centre is the blog homepage, with posts, sections, categories and pagination coming off that.
Content marketing FTW.
Interactivity
At this point you might be asking yourself how I'm so sure what I'm looking at. That's because of the bit that doesn't come across so well with static images - the Crawl Maps are also interactive.
If any particular 'node' piques your interest, you can hover over it to find out which URL it represents, along with data about its crawl depth and internal inlinks.
Attribution & Learning More
It would be remiss of me to not give credit where it's due, to the innovative marketers who inspired us to build this feature in the first place.
Finding a way to visualize website architecture has been pretty much at the top of our wish list when building the tool, and the method we were most keen to replicate was the one we first saw demonstrated by Ian Lurie of Portent, in his article 'SEO Using Force-Directed Diagrams'.
As you can see, the crawl maps produced by Sitebulb are very similar to Ian's, and I don't believe for a second that we'd have been able to come up with something this awesome on our own, so thank you Ian (and sorry for nicking your idea!).
To produce his visualizations, Ian used Gephi, and we were actually first introduced to Gephi many moons ago by Justin Briggs, when he set about visualizing external links in his blog post 'How to Visualize Open Site Explorer Data in Gephi.'
This was where we first learned about the concept of using graph theory to represent link data, and going back over his old posts helped us solidify our ideas for implementing Crawl Maps in Sitebulb.
As ever, we are extremely grateful to the wonderful members of the SEO community for consistently sharing such inspirational ideas.
The posts I linked to above can also serve as education pieces, if you are interested in learning more about this kind of data visualization. We have also published our own 'Crawl Maps FAQ' which gives some more specific insight into how Sitebulb's Crawl Maps are built.
Crowd-sourced Examples
Below you will find a collection of Crawl Map examples from the community, which have been grouped together loosely by yours truly, with their own colourful commentary included as extra sauce.
Manipulative
We've only got one of these so far, but it's a doozy:
@sitebulb #sitemaps
— Simon Cox (@simoncox) February 15, 2018
This sitemap is from a black hat site where they were framing and hiding high ranking sites and injecting their own SEO services into. Bad boys.
Site now gone. pic.twitter.com/L8wyB6KZfl
Content Mess (Where to begin...?)
Crawl Maps are perhaps best for quickly showing you that you've got a real mess to untagle.
95,000 URLs. Staging server. Migration imminent. Client committed to launch date. Just gained access. 2 days left. Panic. @sitebulb pic.twitter.com/p8cSjZjtYJ
— Yanni PapaSomething (@YanniTweets) February 23, 2018
I created this crazy Crawl Map with @sitebulb - a badly done CMS migration that included available content both on htpp and https, duplicate products everywhere, a blog with only a link to the main site and other horrors... #whatamess!!!! pic.twitter.com/dkXE5qleKt
— Juan González Villa (@seostratega) February 21, 2018
Yeah, so I don't even know where to begin to untangle this site's architecture @sitebulb #SEO :) pic.twitter.com/rjmNZmuAEM
— Micah Fisher-Kirshner (@micahfk) February 17, 2018
The rest of the site is a little messy though. pic.twitter.com/43dxoIXjPO
— Marie Haynes (@Marie_Haynes) February 17, 2018
@sitebulb #sitemaps
— Simon Cox (@simoncox) February 16, 2018
Canonical issues fixed from earlier and we can see a lot more pages - but the home page is a little isolated! Much more to do on this site! pic.twitter.com/Ap9dqRJ7tk
The crawl map of nightmares via @sitebulb (full screen version)
— Andy Drinkwater (@iqseo) February 15, 2018
A combination of pages, posts and a healthy dose (very healthy!) of duplication. #SEO #nightmare pic.twitter.com/mQTCGM1ROq
Before and Afters
At Sitebulb we pride ourselves on stocking an eclectic range of before and after Crawl Map tweets.
Here's my "before and after" featuring the easiest SEO win probably ever. I found a missing slash before a relative URL in the site's navigation links. BOOM. #TechnicalSEO #SoDamnTechnical @sitebulb pic.twitter.com/XLWpQXuUTR
— Tomislav Lukinić (@tlukinic) February 19, 2018
Update to a site this morning has brought a lot more visibility to it. A small and often overlooked issue of urls having trailing slashes and the canonicals not.
— Simon Cox (@simoncox) February 22, 2018
Images are before and after adding the trailing slash in the canonicals. https://t.co/SXqNCjNvuy
@sitebulb pic.twitter.com/hV9swHcp9n
@sitebulb sitemaps compo:
— Simon Cox (@simoncox) February 15, 2018
Same site for these two - 1st is at start of opening crawl map - looks a right old spaghetti bowl! 2nd is when it has settled. This is https://t.co/WOSkbUEQgV - @alistapart - a fine publication. pic.twitter.com/gXRL3Ix1MR
Content Explosion
It's probably what The Big Bang looked like.
Crawl map of a large ReactJS site which only prerendered the navigation. Sitebulb Javascript rendering was turned on otherwise nothing would be crawled.
— + AM 👾🤖👓 (@Adoubleagent) February 15, 2018
A lot of problems but crawl map showed issues with URL canonicalization, architecture and similar/duplicate content. @sitebulb pic.twitter.com/lnLh0xCYPQ
Extreme Pagination
Pagination, but like you've never seen it before. Like on top of a mountain.
@sitebulb Last one from me - A website of a building that the British Army burnt down in 1814. pic.twitter.com/qTe5PqQeNT
— Simon Cox (@simoncox) February 25, 2018
Here's a "close-up" of the clusterf**k. pic.twitter.com/yhbnVio9Kb
— Yanni PapaSomething (@YanniTweets) February 22, 2018
The http/https crab with long pagination pincers?? Thanks @sitebulb for the lovely way to convey to the client that their site is, ummm, jacked!!! pic.twitter.com/mljl8xUFJI
— Scott Nickels (@scott5cents) February 21, 2018
While the blues and purples are pretty, the meaning behind them is not. Looking forward to wrapping up our site IA restructuring soon and getting these little flowers pruned. @sitebulb pic.twitter.com/Y1mU5l6XNo
— Kyle Faber (@regal_kyle) February 19, 2018
I just love how visually appealing these are. Gives client a clear view of how there website actually look. Great tool! pic.twitter.com/Stt2s20MyU
— Hampus Nyström (@hnysse) February 16, 2018
I created this crazy Crawl Map with @sitebulb - check out those antennas! Looks like i'm trying to pick up SEO signals from space. pic.twitter.com/2o555RBnC6
— JD Prater 🚴 (@jdprater) February 15, 2018
@sitebulb #sitemaps
— Simon Cox (@simoncox) February 15, 2018
This one from an audit I did recently. Looking forward to seeing this get cleaned up a bit! pic.twitter.com/o4FqrYFp7O
Videos
Some Crawl Map enthusiasts are not satisfied with a static image, and instead created epic multimedia extravaganzas for your viewing pleasure.
Was making a video export of a sitemap using @sitebulb and I just had to share how beautiful this explosion is. Small things like that can entertain me for hours #EasilyAmused pic.twitter.com/UVsJDxVrt8
— Shawn Harding | YouTube, SEO & Graps (@shawnbandv) February 22, 2018
A @sitebulb crawl map too beautiful for this world.
— Oliver H.G. Mason 📉 (@ohgm) February 16, 2018
😢🌏
This represents a 14k sample of infinite pagination on an otherwise uncrawlable JS site, condensed into 10 wiggly nodes.https://t.co/ZJ1Nn8MSgp
Motion and text added because you people can't *just appreciate things*. pic.twitter.com/UPeUZxbwSw
Difficult to show this all in one screenshot, so have a terrible video that show how the WWE is wrestling with their massive catalog of content. I know... that was terrible, even by my standards 😓 @sitebulb #RecordedOnAPotato pic.twitter.com/LhSONKO1MB
— Shawn Harding | YouTube, SEO & Graps (@shawnbandv) February 22, 2018
Trolling
At Sitebulb we believe in freedom of speech, even trolls are welcome here.
@sitebulb is a really useful tool for analysing sites with complex internal linking structures. pic.twitter.com/kUdx7CpH1N
— Zack Neary-Hayes (@NH_Zack) February 15, 2018
Nice site structure (huh? what?!)
I know, I know. You didn't come here to see sites with a nice structure. That's why they're at the bottom.
I love seeing a nice site structure. You know it is going to result in something easier to crawl for Google and users to find their way around via @sitebulb pic.twitter.com/YzI3IdkzfT
— Andy Drinkwater (@iqseo) February 23, 2018
Too bad these well-structured parts of this site are auto-generated, thin tag pages. @sitebulb - cool visualizations. Adding this to my #SEO toolset. pic.twitter.com/gpetA7WszT
— Katherine Watier Ong (@kwatier) February 20, 2018
I'm quickly becoming addicted to @sitebulb's crawl maps. This is the forum on a site we are currently doing a site assessment for. Beautiful. pic.twitter.com/09NY159GHC
— Marie Haynes (@Marie_Haynes) February 17, 2018
@sitebulb #sitemaps
— Simon Cox (@simoncox) February 15, 2018
This one from an e-commerce site. Still needs some work on it to balance the product topics. pic.twitter.com/AyQPEiUEs0
Look at this beautiful crawl map - created with @sitebulb @HathawayP #LoveTheCrawl Not many better ways to visualise site architecture. pic.twitter.com/RXuqlUFvO3
— Craig Gilhooly (@CraigGilhooly) February 15, 2018
Crawl maps generated via @sitebulb provides useful technical insights about your site's architecture.
— Murat Yatagan 📈 (@muratyatagan) February 15, 2018
If you want to identify problematic patterns of your site structure take advantage of well visualized crawl maps feature. This image represents a site with flat architecture. pic.twitter.com/8B8zVkWBQW
Calling all Crawl Map fanatics!
If you want to see your name and tweets up in lights on our illustrious tweet wall, get involved!
Simply tweet at us @sitebulb with a picture of your favourite Crawl Map(s), and you too can be featured on this page.
Please be warned, however, as some folks struggle to deal with the worldwide fame and celebrity that awaits.
Create Your Own Beautiful (or ugly) Crawl Maps
If you've not yet indulged in the joy that are crawl maps of your website's architecture - now is the time to unleash the nodes!
If you're already a Sitebulb user, here's a quick guide on where to find crawl maps in Sitebulb.
Not signed up for Sitebulb yet? You can get a 14 day free trial here, with full access to the crawl map functionality.
Patrick spends most of his time trying to keep the documentation up to speed with Gareth's non-stop development. When he's not doing that, he can usually be found abusing Sitebulb customers in his beloved release notes.
