}
Jojo Furnival
Sitebulb's Patrick Hathaway was joined on November 27 by Mark Williams-Cook, Nikki Halliwell & Chris Lever to discuss website migrations & redirect mapping dos & don’ts.
Watch the recording
Please subscribe to the Sitebulb YouTube channel so we can keep you updated with future webinars. You can also sign up here to be the first to find out about upcoming webinars.
Further reading:
- Speed Up Site Migrations with AI-Powered Redirect Mapping (Mark Williams-Cook)
- How Bring Digital Aces Complex Migrations with Sitebulb (Chris Lever)
- Support Your Deployment Workstream with a Crawling Strategy (Katie Cunningham)
Webinar transcript
Patrick Hathaway:
Hi everyone. Thank you all for joining us today for our final webinar of the year where we'll be taking a deep dive into the world of website migrations. My name is Patrick and I'm the co-founder and CEO of Sitebulb. We also have Sitebulb's marketing manager, Jojo with us as well. She's hanging out in the chat. And we have a fantastic panel for you today. Please welcome Chris Lever, Nikki Halliwell and Mark Williams-Cook.
We have a few things to go through before we get started today, and as I run through this, hopefully Jojo will drop some links into the chat for you. So first and foremost, we've got plenty more webinars already planned for 2025, and if you'd like to get on our email list to find out when they drop, please sign up via our website, Sitebulb.com/resources/ webinars. Whenever you're ready, Jojo with a link for that one. And then secondly, we are still taking applications for guest panellists for webinars in 2025 and beyond. So if you're interested in joining a webinar like this to share some wisdom with the world, please fill in our survey. I'm not going to try and read this one out because it's a crazy long URL. So again, just drop that link in the chat for us Jojo, when you get a second. And then the final call out is about Sitebulb itself.
I can see plenty of you in the chat already Sitebulb customers, but for anyone out there who isn't familiar already, Sitebulb is a website auditing and crawling tool. Coincidentally enough, one of the things that can be useful is helping with website migrations. And if you've been considering making the switch to Sitebulb as your call of choice, I'd like to congratulate you for your wisdom. I'd also like to point out that we have a Black Friday offer available on the website that secures a 20% recurring discount off all our yearly plans, and that is for both our desktop and cloud subscriptions. So again, another link will be hitting the chat very shortly with the link to our pricing page, which has actually already got the discount applied. That sale ends on Monday and that's when the coupon expires.
Okay, so final bit of housekeeping then we can crack on. We are recording the webinar today and we'll be sending out the recording tomorrow including all the links we just mentioned. So don't worry if you're not able to make it for the whole thing. And we will have some time at the end of the webinar for your questions. So please put them in the Q&A tab next to the chat, not actually in the chat but the chat box itself, there's a little Q&A tab on the left. If you hover over it, you'll see. And on there you can also up vote other people's questions if you can't think of any yourself.
So let me introduce our incredible guests today, all of whom it turns out do lots of things. We have Chris Lever, he's the head of technical SEO for Bring Digital, an award-winning digital marketing agency, specialised in helping e-commerce businesses grow. Chris also works as a freelance SEO and has created over a dozen free SEO tools that he publishes on his website. We also have Nikki Halliwell, who is the technical SEO manager for Journey Further, an international performance marketing agency. Nikki also speaks at events all over the world, runs her own freelance consultancy, runs a website auditing service and publishes her weekly newsletter Tech SEO Tips.
And finally, please also welcome Mark Williams-Cook, who is the digital marketing director at Candour, a full-service digital marketing agency. Mark is also the founder of the SaaS tool, AlsoAsked, runs an e-commerce site selling dog leads and harnesses, publishes a weekly SEO newsletter called Core Updates, and is well-known on LinkedIn for his daily unsolicited SEO tips. So welcome and thank you all so much for joining us today.
Mark Williams-Cook:
Super pleased to be here. Patrick,
Patrick Hathaway:
I can't believe how much you all do to be honest. Right. So we'll get on with the questions then. And I want to start with what we actually mean when we say website migration because it's kind of an ambiguous term, so I want to know how you would describe what it is, what it isn't, and could you kick us off with that one please, Nikki?
Nikki Halliwell:
Yeah, a small question to start off Patrick, you dived right in. Yeah, I mean a website migration is a move of any site, of any type really is how I would define it. Any time you're making a big change, it could even be elements to do with a blog depending on what you're doing with that, that could be a type of migration. Most commonly when we talk about a migration, we're talking about things such as moving to a new CMS. You might be doing a site redesign, you might be moving to a completely new domain, launching an international area or a combination of all of the above, which does happen, wouldn't really recommend it. Would suggest doing one at a time, but unfortunately I've been in SEO long enough where I've encountered projects where they're doing a combination of all of those things and that's always fun to handle.
Patrick Hathaway:
Yeah, I think that's a really interesting point. To be honest, it was going to be my next question anyway, it's like how do you help restrict the scope so it doesn't end up being so much?
Nikki Halliwell:
Shall I take that, should I carry on?
Patrick Hathaway:
Feel free.
Nikki Halliwell:
And then come down to somebody else? I'd love to know what Chris and Mark think, but it starts off with just setting expectations with the clients. And I think a lot of it is depending on the stakeholders that you're communicating with, they don't understand why you would not do all of those things at once. It's like, oh well we're doing the redesign, so let's do this as well and let's do this and let's do this. So it's just for me is setting those expectations, having those conversations, not jumping in and flat out saying no, because then you're going to have an upset client and there's going to be a lot more backwards and forwards, but just understand why they want to do that and then you can manage those expectations and agree on an alternative, whether that's a staged migration, depending on what it is that they're requesting, or you can work something out in other ways. But I think it's important not just to say flat out, no.
Mark Williams-Cook:
I generally think unless I've upset someone at some point I'm probably not trying hard enough. I think Nikki you mentioned, in terms of what is a migration, I think you mentioned pretty much everything there, like domain URL, design hosting platform. I guess the one I would tack on the end now recently for me is kind of like an entity migration if you're changing your company. And that really came home when I saw that WooCommerce to Woo migration, which had had it looked like all the kind of generic stuff done, but because the name had changed, I think that really kind of upset Google. And in terms of restricting what part of a migration you're doing, as Nikki said, I think clear communication's super important and it always makes sense from an SEO perspective to separate and do those things one at a time, but it doesn't always make business sense to do that.
So I think it's finding that middle ground, if you are redesigning, moving CMS for instance, the design's probably going to change anyway. People have limited resources. So I would say like with so much in SEO it's just about setting expectation and sometimes just saying, yeah, we can do these things, these three things all at once, but here is the risk and here's kind of what is at risk, which I think maybe we'll talk about later, but working out kind of the size of the thing you're risking by doing the thing you want to do is important.
Patrick Hathaway:
Anything to add onto that, Chris?
Chris Lever:
I think really trying to tag on the end of that is communication is key. So for example, if it's a replatform, like who are the stakeholders who need to get involved, who's responsible for say lift and shift of the content, who's responsible for the schema, who's responsible for the site structure, I think once you get those all aligned, it just makes those conversations a little bit easier with the stakeholders as well, so there's fewer pushbacks.
Patrick Hathaway:
Does any of this make a difference in what stage you get kind of brought in to the process?
Chris Lever:
I think for myself, I think the earlier you get involved with the migration, I think the better. Quite often we get involved with migrations in the four weeks from launch and you can see that there's lots of maybe issues that still need to be rectified. I suppose really as earliest you can get on is basically when the developers are starting to scope out the project. If it's a replatform, where are they moving to? If it's being reskinned, what technologies are there being used? And then you can start to build a shopping list and help steer the migrations go right down the line.
Patrick Hathaway:
Okay. I guess, so assuming you do get involved at that early stage, do you guys have a playbook, like a sort of planning process that you always follow every single time? Or is it very much custom to what the situation is?
Nikki Halliwell:
Yes and no. It depends. Oh, very SEO. I always start off with what I call a migration briefing document, especially if we're brought on really early and it's literally just a questionnaire. What's the timeline? What resources do you have? When is the launch? Why are you doing this? All these sorts of questions.
And then that feeds into the quote of course, but it then feeds into the next steps that I take and how I plan that out in terms of the resource and the conversations that I may or may not need to have with the various stakeholders and other people that are involved in the project. So yes, I do, but it's very much tailored to when we're brought on. If it's a very last minute project which often comes across in freelance work, then I probably don't have the time to put that sort of document in front of them and get those answers. But I'll do a very short version of it and make sure I have the information that I need to be able to manage it.
Mark Williams-Cook:
And what Chris said there as well about at the stage you're brought on. Both in my experience, fairly common situations, which is you are either brought on when the new site is already pretty much made and you are just helping essentially damage mitigation in the migration, or you're there to help actually build the thing they're migrating to. And how I explain that to clients is it's like if you're moving house and you currently live in a massive mansion and then you've asked me to help you move house, but then you just inform me you're moving to a second floor, two bedroom flat and we've got to try and squeeze all your furniture in, then it's like a damage mitigation exercise. Whereas if you say, actually we want to move to over here, then we can help you design the house and make sure everything kind of fits in and fits in well.
The other thing we do during that planning process is an assessment on what we call value at risk during the audit, which is we'll look at the current site, where the traffic's coming from, how much is branded, how much is non-branded, because generally branded traffic, comparatively, will be quite safe for most brands compared to non-branded traffic, unless you do something horrendously wrong.
And then we look at some of the factors that play into the difficulty for the audit. So say you're getting 30,000 visitors a month, but almost all of those are to four different pages on your site, it's probably going to be a fairly straightforward migration compared to 30,000 visitors spread over two and a half thousand pages of which are buried in different places in a nav. And if they don't have any internal metrics on the value of their traffic, they haven't got that far as a pinch, I tend to use just the paid search as a rough guide for the cost per click.
And it gives you then you can work out a, okay, your unbranded traffic is roughly worth this much per year if you were to buy it on paid search. So then you at least know how much traffic is on the table to be lost and a value of that. And as a rough guide, obviously we don't want the cost of the migration work generally exceeding that figure because to give an extreme example, if you had a website with 200 visitors a month that you made two sales from, you wouldn't spend 10,000 pounds on that migration because you could just afford to start over.
Nikki Halliwell:
Yeah, I think that's really important and I think that's something that a lot of people miss or don't think about, especially on the client side when you're doing the migration, is talking about what is the risk. A lot of the talk is the reasons to migrate, and we had a similar thing recently at Journey Further with a charity website. We were working with an animal charity and they wanted to get rid of this section on their website that talked about the names of the various animals that they work with because they didn't think it was valuable because they don't use it internally. Completely forgetting about the fact that they were charity, and when you look at the analytics, that's one of the most visited pages on the site. You're an animal charity, you want people to, I nearly said associate, that's the wrong word, empathise with the animals that you want people to sponsor.
If you're asking them to sponsor Dave, that's a lot easier than sponsor this animal. But they almost didn't think about that and they were just like, "Oh yeah, we don't use it, let's get rid of that." And we had to have those exact conversations like what Mark was talking about, to tell them why that was probably not a good idea to just jump in and get rid of that section.
Patrick Hathaway:
Yeah, so my next question was actually about what metrics do you want to be benchmarking? So that is a really good starting point. Would you say, Chris, is there any other metrics that you are looking at from a benchmarking perspective?
Chris Lever:
I would imagine site speed would be a good metric to start with. So for example, if you're already passing Core Web Vitals, really you want to ensure that you're going to be passing Core Web Vitals on the other side. I think between Mark and Nikki, who have mentioned using cost pay-per-click traffic to get an idea of how much traffic or risk is at play. I'd also think about the other channels as well. So for example, social traffic, traffic from say Facebook, et cetera, I'd want to get an idea of how much traffic I get from those as well. So if you see a decline in say traffic post-migration from those as well, it's worth to keep track of.
Patrick Hathaway:
Anyone else got any other metrics that they're-
Mark Williams-Cook:
Yeah, loads.
Patrick Hathaway:
Loads? All of them.
Mark Williams-Cook:
Loads. So even if we're brought in after the site's been built, one I don't see too many people using that I use quite a lot is trying to work out whether you'd like simulate page rank or look at how important pages are in terms of internal linking. That's one thing I always have a metric for. So if we know we have these 20 pages generate 80% of our traffic, I want to fully understand how they're linked to internally because the last thing I want is this page brings us loads of traffic because it's linked to in the main menu, kind of being demoted on the new site. So there's various different ways you can score that, but I will always get some kind of internal link score and Sitebulb does one as well for those pages.
Basics like the number of indexable pages as well. Sounds obvious, but I want to see that both sites are roughly the same size, because I certainly had it before where you're told everything is just being moved over one-to-one, and then it's like, well hang on a minute, your new site is one 20th of the size, what's going on here? If we're brought in at the time they're building the site, then I would fire up Sitebulb, full audit and every single SEO metric that we're looking at, like Chris said, Core Web Vitals, everything, make sure that we are performing at least as well and we've got parity in other things.
Patrick Hathaway:
Awesome. And then I guess finally thinking about that prep stage, is there anything else you normally like to have in place before things start happening? Again, assuming you're kind of involved at least at some stage before the migration work actually starts then.
Chris Lever:
So in terms of preparation as well, where Mark said getting a full crawl of the site, you could use your multiple tools as well, because multiple tools give out different metrics that you can keep an eye on. I would also start crawling the site regularly as well. If there's multiple changes going on on the build up to migration, for example, if it's a fast fashion site and they're switching seasons, obviously the website structure is going to change significantly. So I'd also keep an eye on not just having a benchmark crawl but crawl regularly as well.
Patrick Hathaway:
Yeah.
Nikki Halliwell:
For me as well, beyond crawling and all that sort of things, I want them set up regular lines of communication. Like whose building the site? Can I have direct contact with the devs? Or if not, who do I need to be having those conversations with? Who's going to be that middle person? Can we get standups in? Are there any other stakeholders that I need to be aware of and how do I get in touch with them? Who's going to have sign off over the redirects? Who do I need to speak to if there's an issue with the build and just all these sorts of admin questions I guess that often get overlooked. But a lot of the biggest issues in migration tend to come from and can be resolved by just making sure that you've got those clear and regular lines of communication open.
Often that can just be something as simple as a Slack channel. That might be all you need to get people in the same room, but it could also be a call at 9:00 every Monday or something similar, especially as you get more towards the business end of migration.
Patrick Hathaway:
It's remarkable actually how many of these webinars that we do that communication comes back again and again and again, is such a key skill that it's not always the first thing you think about when you think about SEO, right? And it's just being so important. I think it will play a big part in my next question as well, which is really about when you're during the development stage or that phase of the project when things are being built and things are getting rolled out on the staging site, how do you stay on top of which things are being changed or outdated? And I think importantly the sort of nuance with this as well is that as we've mentioned before, sometimes multiple things are changing all at once and you're not always necessarily looking for parity, but you might be looking for, well the expectation is this thing will have completely changed. How do you go about actually staying on top of all that.
Mark Williams-Cook:
Silence. Nobody does. We just wing it.
Patrick Hathaway:
Just leave it.
Mark Williams-Cook:
I mean from my point of view, because we've got an agency, for me, firstly and primarily a good project manager who should know and it should be decided ahead of time during the prep, what stages you are involved at. So when we set out we're involved in a build project, we have specific stages, like there is an SEO involved in the very first spec of the site, there is an SEO involved at wireframe stage, there is an SEO involved when things go live on the staging site. With clients, things obviously they can ride a bit looser. So especially with things like the current live site, I will always set up something like content king to monitor changes
Because even things like I've had it where we're doing content migration, the client themselves is doing content migration and will put their content on their current site into the staging site, and then someone else in their team will change what's on the live site. So we need to agree when there's a freeze there and even when they've told me there's a freeze, I don't trust anyone and I set up that monitoring. So I've got receipts when something does change. But for me, I rely very heavily on project managers. Everyone's sticking to that and making sure wherever possible I can monitor things myself.
Chris Lever:
Which is very interesting because it's really hard to keep on top of changes as well on the site. For example, ourselves at Bring Digital, we tend to work on e-commerce sites the most and the product catalogue is changing daily. Even PLP pages can change. They're included in the navigation, they're hidden from the navigation. So yeah, those alerts are key and it's responsibility as well. So for example, if you just take a snapshot of the website and the content was lifted to say the staging site, very often that a lot of the content does change down the line because timelines and migrations are never linear, there's always a delay of some form. So you've got to have some sort of roles and responsibility. Project managers, weekly check ins. Even say shared spreadsheets like a live working document. If someone's going to make a change that you can see when the rollout was and then you can just go recheck, recall the site if you need to.
Nikki Halliwell:
Yeah, we're similar at Journey Further, the only difference being is that we don't have typical project managers because we have the director analyst model, but what we do make sure that we have is robust project management tools, and there's various ones that you can use like Asana being a common one. Personally in my own work I quite like Monday.com just because I prefer the flexibility of that, but whatever you are using, it's making sure that you've got things documented in that and all these sorts of decisions and changes can very, very easily get lost in email chains, especially when you've got lots of other clients and conversations going on on your emails. Whereas if you've got a good PM system then it makes it a lot easier and you can see who said what, when, without having to spend half an hour digging through your emails to find out when that was agreed.
Patrick Hathaway:
So I guess that leads on nicely to the next question. So how do you know when to step in when you think something's going sideways?
Chris Lever:
I would say virtually immediately. If you can say there's something off plan, and for example something that could be off plan, so for example, e-commerce builds, you usually scope out what modules, what components that you're going to use early on, and quite often is the case is modules don't work well together and modules get shifted around. So the things we look at for say, let's say it's a blog module, for example, the blog module, it doesn't have the ability to have product carousels et cetera, where it's already in plan, and say the developers put something in and it's not in the plan, then it's good to call it out as soon as possible. And again, it comes down to that roles and responsibilities. Us as SEOs probably call it out more than say the project management assigned to the migration.
Patrick Hathaway:
Anyone else on when to pull the fire alarm?
Nikki Halliwell:
As soon as you see something that you think is not quite right, the longer you leave it, the bigger the problem's going to get. It also goes back to the point that I was making earlier about making sure that you've got those lines of communication open. If you've got a seat at the table already, it's much easier for you to raise the fire alarm when you spot something that isn't quite right. But even if it's not a big red flag but it's something that you are unsure about, raise it straight away. Don't wait because like I say, that'll get bigger and only become more confusing as the project goes on. So yeah, straight away.
Patrick Hathaway:
Cool. All right. What tools do you use to help ensure that staging site is suitably similar to the live site, and what metrics are you looking at to help determine this? We already talked about this a bit earlier. We are only looking for the correct answers in this section. I do have the power to mute you. Who wants to lead off first with their tool set for migrations?
Nikki Halliwell:
My favourite is Sitebulb, of course.
Patrick Hathaway:
That is the correct answer. Excellent.
Nikki Halliwell:
But I'd be saying that even if you weren't here Patrick and you wasn't part of the podcast that we're on. I think it's, you know, a very powerful tool, it makes it a lot easier to do all of the things that we would otherwise struggle. Even with a standard call, Sitebulb makes it a lot easier for you to get the metrics that you need. And yeah, we spoke a lot earlier about the different metrics that we'd be paying attention to, but if you've got several ones running as Chris spoke about, not just the one at the start, with Sitebulb, it's easy to see the dates and times of when that was one so you can compare the calls to one another in that way as well.
Mark Williams-Cook:
The other thing I'd add is I actually use probably more than a lot of people the crawl visualisations on Sitebulb. I think it's a little bit like tea-leaf reading in that I think you need some experience from SEO and of the tool to get the most of it. But it's funny what you can tell very quickly just by looking at those visuals sometimes where there might be issues with the site, when you see those silly long tendrils going off down somewhere and it's like hang on a minute, that shouldn't look like that.
So I actually do use those even outside of migrations. Basically when I'm looking at a brand new site, whether it is one I'm going to be migrating to or from or just any site, that's one of the first things I kind of look at because it just gives me some feel for what's going on with the site from an internal link point of view, which is, as I said earlier, it's super important when it comes to migrations. And as Nikki said, when you're doing the multiple crawls in the same project, that's really helpful to see the process of the staging site and know when you've got things that have been changed that you've brought up as well. So you don't have to go manually check everything, you can just see it's been of knocked off there from the crawl.
Chris Lever:
Yeah, and just one final point to add on that as well. Some migrations have multiple staging environments and with Sitebulb you can compare crawls against projects, which is super useful as well. It's one of my favourite call-outs for Sitebulb. That you can compare multiple sites, staging sites.
Patrick Hathaway:
Yeah, well that's interesting. I don't know if I've ever come across that before myself. So all right, so redirects is obviously part of the topic and a necessary part of almost every migration. What's your process for mapping them out, how, and at what stage when I suppose are you doing them in the process?
Chris Lever:
And starting with at what stage, I used to try and map out redirects early on in a migration, but I often find you end up having to do the redirects again because at some point the structure will change, something will change down the line. So in terms of when the mapping process should be done, obviously at the scoping session you've got your key structural points, but when you get to say the nitty-gritty of remapping all the URLs, I would say probably do it two to three weeks prior to the migration, just to be on the safe side. If you do it too soon, you pull the trigger too soon, you'll probably end up with lots of missing redirects or the redirects or land on pages that do not exist anymore.
Patrick Hathaway:
Nikki, anything to add there?
Nikki Halliwell:
Yes and no. In terms how I approach it. I still tend to do by quite earlier on in the process or have a version of it as early on as I can in terms of when the content is on staging and I'm relatively happy with it. I don't mean straight away, because yeah, of course there's going to be a lot of changes at that point, but I at least have version one of the redirect mapping then, knowing that I will have to do several other versions. But it gives us the chance to monitor it as we've been talking about, and flag anything that might be changing in terms of the structure. Because if there's a lot of people working on the staging site, they can change things that they shouldn't be changing, and if I've got an early version of the redirect mapping, I can quickly tell them what it should be and what not to do.
But yeah, I keep that updated as much as possible and try to get it signed off in terms of a final version exactly as Chris said, two to three weeks before. And that gives me and my team as SEOs chance to review it, make sure we're happy, and then there's also the chance for me to pass that back over to the client and for them to go through it and sign it off. This is all under the hopeful agreement that there's no more changes happening at this point. Ideally two weeks out, something like that, there's a change freeze and while the devs are able to dot the I's and cross the T's and that's when you can get your final versions crossed off.
In terms of how I do it, it's a mix of processes again, so I still prefer the manual method because there's no replacement for that insight that we have about the websites that we're working on. However, if it's a large site, of course that's not feasible and it'll take you a month solid to be able to do it if you were going to be able to do that completely manually. But there's lots of tools now that people are building to help us do just that.
One of my colleagues at Journey Further built an automated AI mapper that would get us to a decent stage using the Facebook AI actually, and it's called, what's it called? I think it's called Face. It's like the Facebook AI similarity search or something like that. And we use that to get to a good level of the likeness if it has the confidence level of the pages and you can use things like not just the URLs, but the page title and the H1s and all these other elements to help match them up. And then we have the manual insight later on as well, but it's a good way to do it on mass and get you to a good level of mapping.
Patrick Hathaway:
Yeah. Well I mean on the topic of AI, Mark, you also recently wrote a piece for SEMRushland (!) about using AI to help with redirect mapping. Any differences in your process to the one that Nikki just outlined?
Mark Williams-Cook:
Well, to begin with, I don't think we've mentioned yet, the first thing I'll do as well is get all the skeletons out the closet in terms of existing redirects. So if we've already got redirects set up and we just go off search console maybe or crawl of the site, we might end up redirect chains, so get them.
Lists of non-canonical pages also helpful. So things like links with query strings that we want to preserve or not break. So it might be you've got an email list of 20,000 people, you've emailed loads of people, that's got you 20 links to a page that has a query string in and because of how your current site works, you're maybe not going to pass query strings. So make sure we don't lose those in the redirect. And then the actual process of redirection, it depends on the size of the site. If it's a small site, so like 100 or so pages and the client's changing content, I tend to make them do it because it keeps them honest in terms of if they're rewriting the content anyway, I can say, okay, then put in the old new URL and it is actually a good way to make them check what they're doing.
If it's a large site. We actually now, so very similar to what Nikki mentioned, so we've got a script that scrapes the sites and you can use as Nikki mentioned, any element. And the first step is it uses mini LM, which is a Microsoft vectorization. So basically it transforms that element of the page into a set of numbers that encapsulate the semantic meaning and then the Face to Facebook AI similarity search specifically for that. And as you said, it gives you the confidence scores. In my experience, if that's set up well, that's been far more effective than the older methods that we used to use to try and do automated redirect mapping because it used to kind of work, but then you could catch yourself out if you weren't very, very careful. In my experience, this is a lot better and we've used it for some very big sites. Of course it has to be manually checked.
And a lot of it is like 0.7, 0.8% confidence up tends to be right. And then how to do the redirects is also super important depending on the number. So there's a misconception as well that things like a redirect rule for instance might be quicker than a list and it really depends on the logic you use because I've seen some awful very inefficient redirect logic that's being run on every single page load that slowed things down and the same, it's not great if you had 100,000 list of one-to-one URLs.
Sometimes it's best just to let URLs die. You don't always have to redirect everything. So working out who's going to do that logic, how it's going to be written, what the most efficient thing to do is, and then depending on their tech stack, trying to get it as high up as possible.
So the ideal is we do redirects at kind of edge level. If you're on cloud or anything like that, tends to be the most performant and easiest to manage. And then we have some processes later on where, because they tend not to be the most user-friendly redirects, if the team have to do redirects themselves throughout the year, they can do them maybe at a CMS database level and then they're reviewed annually and then migrated if they're permanent kind of up the chain.
Patrick Hathaway:
Nice. Awesome.
Chris Lever:
And just to add to that as well. I think how you collect URLs for the redirects is important as well. So for example, legacy redirects. If you're working with a site that's been around for numbers of years, it's quite a large site, it could be a case of you run into millions of URLs. You could also ask for the server logs as well. Ask if any, so you can see if any of those URLs are actually being hit by Googlebot. You could extract URLs from CDN because there's often multiple places where you can collect URLs ready for your redirect mapping. So you've got search console, which is an obvious one, if you can for the existing redirects from the CMS, you could check the CDN. If there's any at the CDN, you can use Google Analytics as well to pull a wider window. You could do off the back link tools as well and collect them all together.
We work with relatively large e-commerce sites and can run into millions of URLs. So we've attempted to use a AI for example, and we found it's too slow, it could take weeks to run over the URLs. So we've attempted to build tools as well, such as say, use a fuzzy lookup and do it as a confident score, run it at say 90% to begin with, then drop it down, drop it down to say six to 10. Obviously we're looking at things to match up, for example, H1s, titles, et cetera, for PLPs, CMS pages, blogs. But I think a quick one for say SKUs is just get an export of each and then map SKU to SKU. That's a really fast way of doing it as well just to speed it up. So those are tips as well that could potentially help to speed things up.
Patrick Hathaway:
I like that. Treating the product catalogue almost differently than you might all the other pages as well. All right, cool. We've got about five more minutes of my questions then we've got the audience questions, so please remember to go in there and up vote. We might not get a chance to do all of them, so up vote any you're interested in. So I guess I kind of want to move on almost through the stages of the migration process. So what would you say are the final things you're checking before you go live with the new site and how do you make sure things haven't slipped through the cracks?
Mark Williams-Cook:
Can I get in first and say that they've taken the no index off? It still happens.
Patrick Hathaway:
Yep. And disallow, right?
Chris Lever:
Yeah, the other thing I would check as well if they've moved from say... If they're using say staging environments that are wrong, a cloud instance, because that cloud instance, if you're going to crawl that cloud instance and do the benchmarks against speed, the performance is likely not going to be the same performance as it would be on the production environment. So it is probably good to do the final checks once they've moved from the production to environment where the live one will sit as well.
Patrick Hathaway:
Yeah.
Nikki Halliwell:
And I think ideally this is a question that you would note early on in the project, but just being clear as to whether there is any planned downtime as part of the migration, I think that's something that often gets missed, and it depends on the type of migration of course, but there can be things where we just assume it's going to be a straight switch and it'll be fine, but then actually there's downtime, it might be for an hour or it could be a couple of hours, but you need to be clear at the start of the project, or as clear as you can be, if there'll be any at all and how long it's expected to last.
If it exceeds that, then there's potentially an issue and you need to start asking the necessary questions of the developer teams. But yeah, the most common one as Mark says is no index tag canonicals as well. I've seen websites go live where they're still canonicalized back to the staging URL, and that's something that can be easily missed because the first thing as SEOs that we look forward is removal of password protection and no index and need to make sure that you don't forget about those pesky canonicals.
Patrick Hathaway:
When you're sort of putting your quotes together and stuff, do you tend to factor in, right, I want to have two, three, four weeks also tacked on the end of this for monitoring, troubleshooting, making sure that everything's okay? Assuming that they're not like an ongoing client, it's kind of like a project work and then they've moved on. How do you handle that sort of thing?
Nikki Halliwell:
Yes, I do. That section I tend to do it based on the deliverables rather than the time that it will take. It might be that if I was only to do two weeks, that might look fine, but then it might take Google a couple more weeks to crawl the rest of the site depending on the size of it. And if I was to only do it on that then we could actually miss a lot of problems. So it's a difficult one to get the balance right, but I tend to do it based on this will be the deliverables, these are the things that I will look out for and unless it starts to get a bit ridiculous, which so far hasn't been a problem, then it takes as long as it takes until the case of when we're happy with it. If it does get to that point where I am getting concerned, then yeah, I'll have those conversations where it might be that yeah, we need to spend a bit more money to keep an eye on it and these are the reasons why.
But yeah, I've found that that's been the best way so far in ensuring that things don't get missed and don't start to become a problem. Just because that was two weeks ago, it's not my problem anymore. It is, because I worked on a migration and if someone found out that it was mine or if it was Mark's and it's all gone wrong, Mark's going to get the blame or same with me or Chris. So yeah, it can be a difficult one to handle, but it's certainly worthwhile.
Chris Lever:
I find as well, you tend to find... Of course for migration there are functionalities that get missed. So you could have a shopping list of what you'd expect from the site to go live. Nearer to the time you would see that some of the functionality that you've requested doesn't necessarily get implemented. And it's usually prioritise post launch as well. So there usually is a window of monitoring site post launch as well and just seeing through any final functionality pieces that need adding through.
Patrick Hathaway:
Nice. All right, I've got one more question than before we move on to the audience. I want to hear if you've got any horror stories, just the worst migration that's gone wrong that you've seen. It doesn't have to have been caused by you or obviously it won't have been caused by you, but what's the biggest mess that you've come into and gone bloody hell, what's happened here? Who's going first?
Mark Williams-Cook:
I have one that we avoided, which was one we were working on and we did all of these things that we said and we put the site live and guess what day the site went live?
Chris Lever:
Friday.
Mark Williams-Cook:
Friday, yeah, of course because that was completely non-flexible to do that, and everything was actually fine. And then our content king pinged me on Saturday morning, because after we had done everything, the devs decided to push one more small change, and this particular website had user agent detection to basically show a client side rendered version to normal users and a static version to search engine bots, and their entire site to search engines was just blank. Unfortunately we got some alerts that page title has changed and it started filling up. And it was actually really difficult to motivate them to get that fixed out of hours.
So I was texting the manager at the time sort of saying, "This is a most important, most urgent thing that it could be." And they were like, "Well, we QA'd it and it's fine." Because obviously their QA process didn't involve changing the user agent. And I was trying to explain to them to Google your site is blank. And they did eventually change it, but then they were just really kind of blasé about it on Monday. They were kind of like, "Oh yeah, thanks for picking that up." And it was a big site they were doing in seven figures of revenue a month on this eCom site. So yeah.
Patrick Hathaway:
Bloody hell. That was a good one. Anyone else?
Nikki Halliwell:
Yeah, definitely a good spot. Mine's not as dramatic, but it's an example that included by an SEO talk actually, but it's one that we spotted that was a competitor of one of our clients.
So the competitor was a wallpaper site and the client come to us by saying, "We've noticed that these guys have dropped off the face of the earth, what's happened? Can you figure it out for us?" And we had a look into it and we saw that they'd done a migration themselves, but it looked as if they'd bulk redirected all of their pages to the home page. Instead of mapping individually, they just did everything blanket to the home page. And there was an interesting graph that we found in Sistrix when we compared the domains. They had done a .com to .co.uk or was it the other way around? Yeah, I think it was .com to .co.uk, and the graph had gone from up here to straight down here and it was quite the dramatic drop. But it's not the first time I've seen that sort of thing happen either where you just do bulk redirects because what could go wrong, right?
Chris Lever:
Sounds eerie familiar that Nikki. We took on a client this year, wallpaper brand, that had a failed migration. They did a replatform and they did a domain consolidation. The main domain was a .com. I think they did go to .co.uk. And what they also did is with the domain consolidation, they parked the other domain onto Shopify as well. So with Shopify from the other domain, caused redirect change for every single one of those URLs. So what we did is we broke off that domain, we got a GoDaddy host account, and we mapped all the URLs so they was direct, so they were not going through any redirect chains, and that really had a positive impact on the fail migration. It didn't go back to pre-levels because there was other things going on, but it did make a big difference.
Patrick Hathaway:
Do you think you might be talking about the same website?
Chris Lever:
I think. Nikki, we'll catch up.
Nikki Halliwell:
Yeah, we might have to. We'll compare notes afterwards, but yeah, that would definitely be interesting.
Patrick Hathaway:
Excellent.
Nikki Halliwell:
But there's other examples that I'm sure we could go over. I remember one from quite a few years ago with Bayern Munich, the football team, they did a migration. Obviously this was nothing to do with me, but I definitely saw it online, but they'd gone from FC Bayern.de to FC Bayern.com. I'm not entirely sure what went wrong with theirs, but again, you could see the drop-off between when it happens. And again, we can suspect that there was redirect issues and I don't know if they did a platform change at the end, but I guess the point I'm making is that migrations can and do go wrong for websites of all sizes and just because you're a big name like that or any of the seven figure bands that we've alluded to today, you're not immune to these sorts of issues.
Patrick Hathaway:
Absolutely. Absolutely. Right.
Chris Lever:
Yeah, and I think one more horror story is one [inaudible 00:50:04] migration.
Patrick Hathaway:
Chris, we've got loads of questions in Q&A and I'd love to sit here and do more horror stories, but I think we're better get on with these because there's loads that have been up voted as well. So yeah, a reminder if there's any you want to see that are in there that you like the look of, go and up vote them. I will start with the first one here though. Sorry to interrupt you there Chris, we're down to 10 minutes left.
Simon Cox, a serial webinar attendee. In smaller sites, let's say under 10,000 pages, are there any good reasons not to redirect all the URLs? No. Is that the answer?
Nikki Halliwell:
I love the silence. You could probably hear all of us thinking as well.
Chris Lever:
Yeah, I've seen some of Simon's questions previously, so first of all, I'm trying to ascertain is it a genuine question?!
Nikki Halliwell:
It could be interesting. I mean, none that come to mind if I'm being honest, other than the fact that they're pages that are being retired. If it's something that's like a short timeline, maybe they're pages that are being retired so you don't think that it's worth them being redirected and things like that.
Mark Williams-Cook:
Yeah, so if you've got a slight structural change, for instance where in the new site you've merged some categories and the URLs on the current site aren't getting any traffic, they don't have any links or anything, then I would probably not bother migrating the URLs. But Simon specifically has said pages here, so I guess if the question is assuming all the content we want to keep, it comes down to it being a value question at the end of the day and because of the tools we've got to do migrations, it can be done quite quickly. I would always say the target is to do them all because that's what you want to achieve in a migration, a one-to-one if you can.
Patrick Hathaway:
Cool. All right, let's move on to the next question. So Yordan said, how do you overcome unexpected pushback from external stakeholders like developers in the migration?
Chris Lever:
I think to start with this, I don't think there's any unexpected pushback from developers. I think all of it is pushback from developers. We give them a hard time, we push back on fixes, implementations that they do. I think sometimes they reciprocate that as well and just make life a little bit harder for ourselves, but surely so because it just has a better outcome in the end.
Nikki Halliwell:
Yeah, I'd want to find out what their resistance is, why are they pushing back? Once you understand their objections, you can usually find a workaround or understand what they're saying. And it could even be that both of you are actually saying the same thing. You're just saying it in SEO speak and dev speak. Yeah, I don't think there's any unexpected pushback in my experience. With migrations, I always expect the unexpected, but if you just find out what the resistance is and the reasons for it, then it makes life easier.
I think there's significant value in both understanding and explaining the why of things, which again comes back down to that communication thing again, right?
Mark Williams-Cook:
Sometimes you do just get obstinate people though that you need to crush with facts and objective information and data! But yeah, most of the time it's the motivation thing.
Patrick Hathaway:
Crush with facts. Excellent. Okay, let's move on to this one then. So Salman says some people believe that having too many redirects can slow down a website because the server to process each rate redirect in the list to find a match destination. What's your opinion on this? So we did actually at least half cover this in the questions already. It kind of fits I suppose a little bit with Simon's questions as well. At what point, how big do you have to get before it starts becoming an issue to just have loads and loads of redirects just one to one?
Chris Lever:
The bigger the side, if it's a really large site, I would say migrate the redirects to the CDN. If you've got a very large remapping file, if it's a NGINX file, it can and does slow down the site. So at what number? I would say half million upwards is probably better for the CDN to handle all the redirects. Anything below you could probably rely on your cloud infrastructure or if it's like, say if it's a smaller site such as a VPS, et cetera, try to keep within a file limit. Maybe anything below 10 megabyte should be okay. But it depends on the size of the site and it depends on the infrastructure investment. But yes, redirect files can and do slow down websites.
Patrick Hathaway:
Cool. Thank you for that one. Right, I'm going to put this one on stage because it's just a bit different to what we've talked about already. From Ben. How do you effectively migrate a site with a high volume of image impression, sorry, and backlinks while minimising loss, especially when the images lack a proper naming convention?
Mark Williams-Cook:
So I'll handle this one if no wants to jump in immediately. So the image thing, so we had a site that was basically getting all new photography done. So same products but just different photos. And the issue I had was they ranked really well for lots of the current images and it was very challenging to try and match those up on a one-to-one basis. So what I ended up doing was putting the new site live with the new images, just ignoring everything else, and then literally just uploaded, kept all of the same images onto the server that existed because they were staying in Google. And obviously now when you're do an image search in Google and you click, it takes you to the HTML page that they are on, which being redirected.
So actually what happened was we kept pretty much all those images and started getting other ones listed as well. So that worked really well. If you're having an issue where you want to migrate images, they're the same images and it is just like a file name kind of parity thing, there's actually loads you can do now with APIs for hashing images. So you can upload them programmatically, generate kind of a fingerprint for that image and generate the new URL. So there are ways you can do that programmatically as well. I don't know if anyone else wants to answer the kind of links thing?
Patrick Hathaway:
Do you know what, we've still got loads more questions and we're drastically running out of time. So thanks for handing the image aspect of that. I mean, this one is similar to what we covered and similar to the last question as well, I suppose. Emran asks, how would you approach handling redirects for PDPs where both the URLs and the product name have been completely changed?
Nikki Halliwell:
I'd look at the other elements that you could use like Chris talked about using the SKUs earlier. Hopefully they've stayed the same as well, so you can do something with that. You can use H1s and other elements on the page if absolutely necessary as well. Perhaps you could extract product descriptions and use them with a custom extraction. There's a few ways that you could do it, but basing them off things like the confidence score that we spoke about earlier as well. And if you get to a good level of confidence with them, then you're usually right, but then obviously make sure that you're using manual oversight as well.
Patrick Hathaway:
Yeah, I mean, also just forcing the client not to also change the product name when you're doing a migration, that's not particularly helpful is it? Right. We've got time for a couple more. So do you guys care about redirects that are less than five to 10 hops given Googlebot follows up to 10 redirect hops? From Shaikat.
Mark Williams-Cook:
That's contrary to my knowledge because I believed it was five, so someone else can answer that. So I wouldn't let it be more than five. I'm pretty sure unless it's been updated, that's what we were told by Google, although I haven't tested that.
Chris Lever:
I think with each of those redirect hops as well, the equity pass starts to diminish as well.
Nikki Halliwell:
Yeah, I'd say five. I've certainly not heard of 10 from Google. I mean, Mark talked earlier about as part of the migration process is looking at your existing redirects and trying to update them as part of the process. So I care about them in that respect and trying to include them in my redirect mapping and minimise them as soon as possible. And hopefully that would help with the question that we had just earlier as well about reducing the size of the redirect file. If you're including them in your mapping, that will also help to keep that file size two somewhat of a minimum.
Mark Williams-Cook:
It's extra latency as well for users for no reason whatsoever.
Patrick Hathaway:
Yeah. Okay. Right. Let's have one more last one. So Joe asks. If URLs are changing, what's your thoughts on changing URLs to the target URL pre-migration? Or is it better to change URLs with the migration? Okay, I would answer myself, I don't really know what the point would be in doing that. Can anyone support why you would want to do this?
Mark Williams-Cook:
So I'm assuming this is if we're changing CMS or something and the URLs are going to change, that we're trying to change one thing at once. But my kind of answer to that would be if you're changing the structure of the sites of the URLs are going to change because maybe you're changing something like the CMS, making your current CMS follow that URL structures probably going to be a huge amount of effort. So I guess in theory that might be a useful thing to do. I've never done it and I would struggle to think about, it might be my shortcoming practically how to do that in an easy way.
Chris Lever:
It doubles the length of the velocity. In Google search as well, it takes a lot longer for the URLs to settle down.
Nikki Halliwell:
Yeah, because they then have to pick up two changes. I would struggle to see the benefits really in that and you're just making your work twice as hard and twice as long, so although-
Mark Williams-Cook:
We get to charge twice as much.
Chris Lever:
Yeah.
Patrick Hathaway:
Yeah, so don't do it unless you can charge for it twice. I like it. That'll do. All right, so that's all we've got time for today folks. Thank you ever so much for watching and for all your fantastic questions. Sorry, we didn't get to all of them. There were still more in there. Huge thanks of course go to Mark, Nikki and Chris are so generously giving up their time and expertise.
We'll be publishing the webinar over the next few days on our website and emailing that out to everyone who registered. So if you miss the start, don't worry you can catch up. As a reminder, this is our final webinar of the year, but please sign up to hear about our webinars in 2025 as soon as we announced them via the link, Sitebulb.com/resources/webinars. If you sign up there, you'll get an email as soon as we announce a new webinar with the opportunity to save your seat. For everyone that celebrates it, happy Thanksgiving, and for everyone who doesn't, happy Black Friday week and please don't forget about Sitebulb's Black Friday offer. Thanks again for watching. See you all next year.
