Enterprise SEO Audit: From Crawl to Prioritised Action Plan

This week, we welcome Kylee Della Volpe, sharing her playbook for turning a sprawling enterprise crawl into a prioritized roadmap your business partners will actually buy into.

Picture this - you just ran your first crawl on over a million pages on your site. You’re digging into the crawl report and seeing a lot of opportunities, which is exciting, but it’s also making your head spin. Suddenly you realize:

• There’s too much content for you to go page by page to fix
• Your crawl doesn’t tell you what business partners you need to work with to implement your strategies
• You have no idea what to prioritize or how to get started.

Anyone who works as an in-house SEO has been there. Let’s break it down to help you go from overwhelm to action plan on your most recent crawl.

Segments and exclusions: the backbone of an effective crawl

When your crawl is hundreds of thousands or millions of pages, you can’t possibly begin to see patterns and diagnose issues without segmentation. Segmentation is going to be part of the plan for every issue you come across or every enhancement you want to try. Why? Because every time you do something on an enterprise site you’ll be asking yourself two questions:

1. Is this a template level fix or a page level fix?
   If you see the issue happening across all the pages of a certain template (like your PDPs or your blog) then you know that there’s potential for this to be a template level fix. While this type of fix is a little trickier to implement because you’ll need to engage your engineering team, it typically has a bigger impact because you can roll it out across all the pages in that type at once.

   If you’re only seeing the issue here and there on pages within the same template, then you know this is likely a page level issue. This fix may require more manual effort, but you typically would not need to engage engineering on it.

2. What segments will provide the biggest impact when I implement this work? The answer to this is going to be entirely dependent on your business and its goals. But your segmentation will give you visibility into what fixes need to be made on the sections that ladder up to your business priorities. For example, if you’re trying to grow your business in EMEA, your regional segments will be key. If you’re trying to grow revenue in your women’s shoewear line, then your business segments are going to help give you the insights you need to create a strategy.

When you’re setting up your segmentation, also be sure to adjust your crawl settings to filter out any additional noise and to focus on exactly what matters for your business.

Useful crawl segments for enterprise SEOs

Try out these segments on your next crawl:

Segment by page type/intent

AI search has fundamentally changed the traditional marketing funnel, but as SEOs we still care deeply about search and page intent.

• Informational

• Commercial

• Transactional

This segment can be useful for:

• Prioritizing work

• Determining where in the funnel people enter the site and where people fall off (if your crawl is linked to analytics)

Segment by product type

Your product type segmentation is a subset of your transactional intent segments, but this additional layer can be a lifesaver when it comes to prioritization. Let’s say you work for an outdoors company that sells both camp stoves and fuel canisters. The stoves bring in a higher average order value, but you sell a lot more fuel canisters, making it the more profitable item each month.

This segmentation is helpful for:

• Prioritizing fixes for specific product types

• Interlinking

Segmenting by business/partner

When you’re working on a large site, the likelihood that you can touch every page is very low. Part of your job is supporting your business partners and empowering them to understand what has to happen on the site.

Depending on the size and complexity of your company, creating these segments might end up being more difficult than it sounds. You’ll likely need to talk to your business partners to understand which pages belong to which portfolios. If you aren’t able to make headway with that approach, a good work around is to find the segmentation used for analytics dashboards and mirror that.

This segmentation is helpful for:

• Providing insights to empower business partners

• Knowing who to work with to get buy in for optimizations

• Knowing what pages to focus on when you’re working with a specific business partner

Segmenting by region

For a global company, internationalization is always going to be something a technical SEO is keeping an eye on. These segments will help you hone in on any region-specific issues. They can also help you cut down on erroneous crawl errors that might be caused by these pages.

For example, many websites have an automatic redirect in place where you land on a regional page but your IP isn’t in that region. Makes sense for a human, but your web crawler is only going to be crawling from one IP address in one specific location. All those 301s for your international pages are going to look overwhelming in the crawl and it's going to be hard to suss out what’s a true 301 and what’s the regional redirect. That’s where your segmentation can help.

This segmentation will be useful for:

• Addressing any href lang issues

• Removing noise from crawls so you can isolate issues

• Determining if bots crawl pages differently in different regions

Be sure to keep a file with the rules for all of your segments so that you can easily apply it in the filter settings of your crawls. For example, you could filter all of your US English educational content pages using the following filter.

Content search and content extraction to see the scale of your issues

A huge website increases the chances that multiple content owners have touched content – and those touches and updates might not be consistent. Content search and content extraction can help you find all of the pages that have similar elements, see if those elements are applied consistently, and determine if any optimizations can be done at a template level or a page level and what the scale of those optimizations are.

Example extractions for enterprise SEOs

Find all pages with FAQs

With everyone now focused on AI visibility, there’s a renewed focus on FAQ content. There’s a good chance you have a lot of FAQs on your site and that some of them are applied on the page without using a specific template or content module.

You can use the content search setting in Sitebulb to find any content that reads “FAQ” or “Frequently Asked Questions” to understand the full scope of your FAQ content. From there you can build out your FAQ strategy to ensure the right FAQs are on the right pages, they follow a standardized format, and schema is applied.

To do this, first navigate to the content search section:

Then enter all the variations of your text that you want to search for during your crawl.

Find all instances of video without video schema

In the past year or two you’ve likely had an increased focus on a sound schema strategy for your site. But with a large site where there might be a lot of people updating content, it can be difficult to understand exactly how much work you need to do to implement your strategy for specific content types.

Again, using content extraction, you can identify pages where your target content type is present. To do this, you’ll need to find an example page that includes a video. From there you can use the built in testing tool in the content extraction settings to find the correct path to search for.

To do this, first navigate to the context extraction section of the audit settings.

Add a new rule:

1. Enter a URL from your site that contains the item you want to extract in the crawl

2. Click on the element you want to extract. In this example, I used the video play button.

3. Use the drop down to select how you want to extract the data

4. Check that your test is checkmarked green

Once your crawl is complete, you can join this data with your structured data report. You’ll be able to see exactly where the schema is applied, if it's applied correctly, and what pages are missing the necessary schema.

Finding all of the product mentions in unstructured content

This can be particularly helpful if you’re sunsetting a product or changing a product name. For example, if your company is moving from the SuperTrailShoe 4 to the TrailShoeMax 5. You plan on eventually sunsetting the four and want to focus on the five. You can use your content search to find all content mentions of the SuperTrailShoe (linked or unlinked) and you can update the copy so that you now reference the TrailShoeMax.

What we can see about AI visibility

If you had a dollar for every time you’ve been asked how you’re doing in AI search then you could probably just go ahead and retire now. While citation and visibility tools are getting better and better, there are a few ways a site crawl can help you diagnose potential visibility issues and prioritize fixes before you even touch the content on the page.

Review Javascript rendering

We finally got to a place where Google could fairly reliably render our Javascript pages and then AI search had to come along and blow that all up.

While this isn’t a hard and fast visibility metric, it’s something that helps to remove the very first hurdle an AI crawler would face – can they even see the page.

Use Sitebulb’s Javascript rendering features to determine what pages, sections, or content types are rendered client side vs. server side. If you have important content that is rendered client side, then it’s time to work with your engineering team to see how you can fix that so that AI crawlers can read the content.

Log files

As we move to a new world of search that’s getting harder and harder to measure, log files can give us some grounding into which AI bots are coming to our pages, what their jobs are, how often they’re coming. It’s a good directional proxy to tell us if optimizations are working without focusing on prompt level tracking or optimizations.

While Sitebulb doesn’t currently do log file analysis, it’s a feature that’s on the way, so be on the lookout for the update.

This type of analysis is great for understanding what content and page types are getting more traction in AI, identifying site architecture issues that may be preventing bots from accessing content deep in your site, and understanding if your pages become more visible after optimizations.

Breaking down silos with internal linking

The larger your company is, the more people are likely to be creating and updating content. This presents a huge blind spot when it comes to interlinking. Often, the content creators are focused on their own content and that can create silos when the content is not linked to other relevant pages. Remember, your customers don’t care about your internal structures or silos, they care about solving their problems so it’s your job to help make sure they can do that.

Your crawl can help you find link issues like:

• Redirect chains

• Paginated content implementation errors

• Pages that are underlinked

Those underlinked pages can be the most time consuming to fix because it's not always immediately clear what pages should be linked together. Luckily, AI has made this much more accessible for all SEOs as it's easier than ever to extract your content’s embeddings using out of the box solutions from crawlers like Screaming Frog.

Once you have the embeddings you can use a number of algorithmic approaches to determine what content should be linked where. Thanks to the wonderful, open nature of our SEO community there are lots of guides and free Google Collab notebooks out there. So you don’t need to be a data scientist to implement this work.

When your data is clean and ready, you can use your segmentation to isolate and prioritize the areas that are most important to your business and that will make the biggest impact.

How to move from ideas to implementation

Getting all these fixes in place will not only take time, but it will also take attention and work from other teams. You need to be able to communicate why those folks need to focus on your project over the million other initiatives they have on their plate.

How to prioritize with the Impact - Effort Matrix

Using the impact effort matrix is especially helpful for SEOs. The four quadrant decision making tool takes into account the amount of time and resources a fix will take and what you expect the ROI will be. You then plot that information within the appropriate quadrant to help you visualize exactly where you should be putting your efforts.

How to use the model

In order to effectively use this model, you’ll need to be able to estimate both your effort and your business impact.

Let’s work through an example.

Your company: An outdoor equipment company

The business focus: The company is launching a new cooler backpack that’s great

Your recommendations:

1. Update the product category page templates for these activities to dynamically recommend the new product.

2. Update any blogs that have unlinked mentions of packing a cooler, taking a day trip, or beating the heat.

Your approach: Set up a regex for terms related to those mentions and do add them to your content search settings to find them all so that you can link this product on those pages.

Time investment:

Estimating impact

In order to estimate this we need to know three things:

1. Current traffic to the pages where we want to place the link

2. CTR to similar products linked on these pages types

3. Conversion rate of PDPs of similar products

4. Cost of the product

Once you have these numbers, you’ll need to do three simple equations to get your final number.

• Traffic X CTR = Estimated traffic to the PDP

• Estimated traffic to PDP X Conversion rate = Estimated number of conversions

• Estimated number of conversions X cost of product = Estimated revenue

You can then take that estimated revenue and use it to move this project to the appropriate square.

How to measure impact post implementation

When you’re making your business case for this work, you should be sure to document:

• The exact metrics you hope to influence

• The measurement source of truth is

• What the model used will be (Last touch, full touch, etc), and

• How the metrics will be reported (specific dashboards, regular report outs, etc).

This will ensure that you’ve set expectations and everyone is on the same page when it comes to measurement.

Create a feedback loop on the project outcome

After the updates are made, you’ll want to make sure you measure the ROI you actually got vs. what you estimated. Be sure to update any changes in time investment when you’re doing these calculations and keep notes for future time investment calculations. You’ll want to record this information and review your estimates to see what was correct and what might need to be tweaked next time you estimate this type of work.

As you are gathering this information, make a plan to share it out with stakeholders. Being transparent on both your wins and losses will help improve your credibility with your stakeholders for future projects.

How to advocate for more resources

There are going to be times where you want to do a big project, but the resources aren’t available. Using a model like the effort-impact matrix will be a big help in advocating for those resources because you’ll be able to clearly show the business impact of your work. Before you bring your case to leadership, there are a few more questions you’ll need to have answers for:

• How do you plan on overcoming the lack of resources: Can you hire a contractor or onboard a new tool to help?

• The cost of your plan: Scope out the cost of hiring contractors or the cost of the tool. You’ll need to add both the monetary cost and the time investment to onboard this solution into your ROI equation.

• The cost of waiting - is there a cost associated with waiting? Will you be losing money by not implementing this work? Will a competitor be taking over market share while you wait? These are all things that leadership will want to know and that will impact their decision to say yes to your resourcing question.

The more you practice your prioritization focused on business impact, the better you’re going to get. Over time you’ll be able to feel confident that you can take what you find in your site crawls and create clear, actionable roadmaps that actually get buy-in and prioritization.

Editor’s Note: You might also want to check out Sitebulb’s free Enterprise SEO Course, which breaks down the challenges unique to enterprise SEO and provides solutions to help you overcome them.

TL;DR

• Segmentation is the backbone of an effective crawl. You should consider creating segments that align with how your business works, who you work with, and how content is logically grouped together

• Content search, content extraction, and structured data reports provide a narrower view of your content but can span across multiple segments. It’s great for finding like content, similar wording, or like schemas

• Your crawls can also give you some insights into AI visibility to help answer those burning questions from your business partners

• Getting your roadmap is only the first step, you need to align all of your recommendations to a specific business impact and be prepared to measure your outcomes in order to get buy in