SEO Testing, An Argument for Using Data in SEO

What if I told you there was only a 48% chance that your page “improvements” would lead to an increase in organic clicks from Google?

According to our tests at SEOtesting.com, those are just about the odds of making impactful SEO page changes.

Also revealed in those tests?

• 18% of page changes led to no change in clicks
• 34% of page changes led to a decrease in clicks

Basically, you have a 50/50 (okay, slightly less) chance that your changes will improve SEO results.

All the more reason to focus your time and energy on testing which changes actually move the needle, and then creating a workflow around the results of those tests.

How to do that? We’ll discuss this in more detail.

What is SEO testing?

Also referred to as SEO experimentation, SEO testing is the process of isolating a given set of figures, called the "test set," against another set of figures, called the "control set," to determine what changes do or do not have an impact on SEO.

SEOs can use this method to understand the causes of change in ranking, website traffic, visibility, and other SEO metrics. It can also help mitigate risk when planning site migrations, structural changes, on-page optimizations, internal linking exercises, content removal, and more.

Similar to the process of data science or even grade school science, SEO testing involves the process of hypothesis, testing, and analysis.

The most common time frames for SEO testing are 2, 4, and 6 weeks, although you can test as long as you'd like depending on your circumstances. You'll start to see the impact of your test in as little as two days, although testing should give the best results at least four weeks for measurement purposes.

There are two types of SEO tests: the Time-Based Test and the Split-Test.

Time-based tests

Time-Based Tests involve making changes to a singular page and comparing results over time. It requires a control period before making the changes, then a test period after making the changes.

Split-Tests

Split Tests, on the other hand, are groups of pages, split into 2 (hence, the name) and tested as a "control" group and a "test" group.

When to use Time-based tests vs Split-tests

The primary difference between using Time-based tests vs. Split-tests is the complexity.

For smaller, simpler tests, you'll want to use the Time-based model since it's an easier setup. Additionally, in a Time-Based test, you're only tracking the results of one page at a time.

On the other hand, Split-tests require splitting multiple pages into two groups: the control group and the test group.

When you're testing something with less overhead time, like titles and meta descriptions, multiple pages aren't an issue. But if you're making more considerable changes or changes across lots of pages, that's something you'll want to split test first.

Finally, Split-tests help overcome seasonality and algorithm updates issues since there is a "test" group and a "control" group versus the Time-based test, which only measures the impact over a period of time.

Limitations, Influences, and Considerations

Like other forms of experimentation, there are limitations to SEO testing. Here's a bit more information on what factors might influence the results of an SEO test.

• Timing. SEO testing can mean waiting to understand results before moving forward with all your changes. Ideally, SEO testing is done singularly, with one change made at a time.
• Seasonality. Thanksgiving, Black Friday, Football season, basically anything that may affect your test or control period.
• Algorithm updates. Be wary of any algorithm or SERP updates that can skew results during testing periods.
• Data-driven. You need to already have some data to work with before beginning a test. You want to make sure you're working with pages and keywords with enough volume for testing periods.

Why is testing important for SEO?

In short, the reasons for using the SEO Testing Method are:

• Data and Facts vs. Opinions
• Prioritization and Process
• Context and Industry

Data and Facts vs. Opinions

Often in SEO, individuals are subject to the mercy of their boss, client, or another party.

SEO testing approaches SEO from a scientific, factual perspective. It forces you to think through your reasoning, process, and statements.

Rather than getting someone an immediate answer to "will this impact our site?" and "what will happen if we change this?" or "here are some changes we want to make," SEO testing allows you the much needed time and evidence that your opinion may need to help support or deny a bad request from a client or colleague.

It also gives you a process to follow, a standard that everyone must abide by before making any changes.

• Does the client want to remove all category-x pages? Run a test.
• Are you thinking about changing the format of all your page titles? Run a test.
• Are you making changes to the sitemap? Run a test.

There's a subtle but powerful shift in approaching SEO from a scientific perspective. It's no longer guessing based on everyone else's best practices but knowing based on your evidence-based tests.

“There are hundreds of different ways to do SEO, including sitemaps, link-building, search-engine-friendly site design and so on. The best strategy for successful SEO can differ by product, by page and even by season. Identifying what works best for each case helps us move fast with limited resources. By running a large number of experiments, we found some well-known strategies for SEO didn’t work for us, while certain tactics we weren’t confident about worked like a charm.”
Julie Ahn - Pinterest engineer, Growth

It’s easy to argue with Bill, the SEO Consultant, but difficult to argue with the data and facts.

Prioritization and Process

As mentioned, creating a process for SEO testing has many benefits.

Prioritization, for one. When you have a more prominent website or multiple client sites to manage, it's not always realistic to roll out a new change based on someone's opinion (even your own), especially when those changes require engineering or developer time.

With SEO testing, you can run a miniature version of your suggested changes by creating a hypothesis, setting a test, and measuring its impact. With this approach, you can avoid massive mistakes or changes with little risk. After all, it's just as important to know what doesn't work as what does.

Further, setting SEO testing as a standard practice creates a cushion for you to say no. OR perhaps just not right now. With SEO testing, you can be proactive in your approach versus saying yes to every request, change, or idea without knowing the impact until after it's done.

Context and Industry

Finally, implementing a formal SEO testing process (or even a better informal process) can allow for context, competition, industry, and other factors to be considered before claiming victory in your efforts.

It's not enough to look at a graph and assume that what you've done is effective. Too often, people make changes but do not measure the results. With SEO experimentation, you consider your changes on a case-by-case basis. You think, document, and justify your results the same way a scientist, mathematician, or doctor might, with reasoning and data.

SEO testing allows you to move beyond SEO as a passive practice and into the forefront of innovation by depending on your data, not a general blog post or case study that does not consider your unique client or company.

How to find SEO testing opportunities

Depending on your organization, initiatives, and daily work, you'll be able to find SEO testing opportunities in a wide variety of cases. The goal is to move away from random guesswork to a more strategic, thoughtful approach. This is especially important in more prominent organizations or companies where multiple stakeholders are making website changes.

With this approach, you're making decisions based on tests, results, and data.

Here are a few ideas to get you started on finding opportunities for SEO testing:

• CTR optimization
• Structured data
• Internal linking
• Page speed
• Page layout
• External links
• Javascript changes
• Content optimization
• Architecture updates
• And more.

SEO Testing Steps

So how exactly does it work?

The steps for SEO testing generally look something like this.

1. First, understand what you’re looking to test and why. This will help you form your hypothesis.
2. Next, form your hypothesis. What are your changes and what do you expect will happen when you make your changes?
3. Then, document your method for testing. Make sure it’s easy for another person on your team (or even future you) to understand or replicate tests in the future. This will also help you work through any possible obstacles or gaps in your reasoning.
4. Now you’re ready for testing. Ready your change and determine your testing period. Publish the change and resubmit your site in Search Console and Webmaster Tools.
5. Finally, let the test run. When your testing period is over (assuming nothing catastrophic has happened as a result), you’ll be able to confirm if your hypothesis is correct or incorrect. Take the time to work through explaining your findings, and document any other observations during your testing period, such as changes made by your competition. This will help you understand if the change was influenced by any other factors.
6. If needed, share your results with your team or client, and decide what your test results mean as you move forward.

This entire process can be done in a spreadsheet or a tool like seotesting.com.

Getting stakeholder & client buy-in

The SEO industry is excellent at sharing work that saw positive results but not so great at sharing its failures. There are many times where guesses lead to traffic or visibility loss, deindexation, or revenue drops. If there was a way to mitigate these failures, wouldn't you want to take them?

Here are some benefits to adopting SEO testing as an approach.

More transparency.

It can be a challenge to justify specific initiatives as an SEO professional. When we suspect a project will benefit the user, and therefore, the website, we have to create transparency around those assertions.

Instead of making changes, hoping for positive results, and then maybe showing those results to stakeholders, SEO testing allows for you to say "This is my prediction, this is the test we're running, here's what that will mean when we get back the results" and "These are the results, here are the next steps. Based on the data, it suggests we will see an improvement of X amount by X date."

This is much more transparent than running a monthly report and assuming improvement or decline based on what you did or didn't do during the month.

Less risk.

Are you confident that a particular change will improve site performance? The reality is that you don't know what you don't know until you know.

Yes, it's a great idea to clean up your blog posts or design a new site that's more user-focused. But what if that design didn't have the proper technical framework? You'd risk losing all the previous traffic and go far, far in the wrong direction. Your testing period would show you your blockers before you rolled out tons of changes and then discovered the issues.

When you use SEO testing as your standard, you have less risk in managing your portfolio.

Failures become learning opportunities.

Cliché, but true. Failures are learning opportunities.

Taking the time to test your hypothesis with SEO testing helps you learn from failure and avoid what could have been a significant mistake.

For example, say you want to change the structure of all your blog posts. You could run a test before making the changes. If the results are good, then roll it out across all the blogs. If, however, you're noticing a dramatic decrease in traffic within the first week, you can choose to revert your changes and prove the failure of your hypothesis.

Congratulations, you just saved yourself a ton of work and crappy results.

Documentation for industry improvement.

Have you seen other SEO's sharing their brilliant work and everyday findings with the world? That could be you, too.

SEO testing is excellent for sharing process, reasoning, and experimentation with the SEO industry. Agencies or consultants looking to prove the value of their work can easily use SEO testing to measure the success or learnings of their campaigns.

It's a great way to show off an idea in a blog post, webinar, or conference.

Data-driven results.

As mentioned, SEO Testing is a great way to prove the value of your work to clients and within your organization. It focuses more on data and logical thinking than it does on industry best practices.

Don't get me wrong, there's a place for best practices. But they're a starting point, an educated guess made by educated people, not the result of a carefully crafted test for your unique situation.

“I find it fascinating how there is not a one-size-fits-all or universal SEO strategy. There are best practices, but no one client will ever have the same SEO strategy. A lot like a medical diagnosis, each client is unique, and what works for one client won't work for another.”
Tess Voecks, Senior Director of Project Management, Local SEO Guide

In this way, SEO testing lends itself to the idea rather than commanding the idea. It's a way to support your creative thinking, strategies, and ideas vs. making false claims and then covering up mistakes.

Not that you'd ever do that... right?

Clear expectations.

I’m not sure about you, but I love saying no (under the right circumstances). Relish in it, in fact.

No.

I’m not making that change to our entire site.

Why?

Because here are the results of the test. They don’t meet our benchmarks. Enjoy your day.

Just imagine.

Seriously though, setting clear expectations for what you're doing and why will help save you time, money, resources, energy, and mental capacity. Developing and following a process for SEO work can help you and your team/ client focus on the work that's worth doing.

Rather than assume your opinions are absolute, prove them with tests, then move forward with your changes.

Final thoughts

Thankfully, SEO testing removes opinion and ego from the equation and relies on proof, data, and justification for moving forward with SEO recommendations. Hopefully, we can move toward SEO testing as the standard in our industry as more professionals become aware of the process and practice.

Whether you’re making recommendations as part of an SEO audit, or simply looking for opportunities to increase performance, testing can give confidence that you’re taking the right approach (or avoid taking the wrong one!). 

Hey, if Pinterest and SurveyMonkey are doing cool stuff like SEO testing, it's probably what I'm going to do, too (they know a few things, I hear).