With more and more companies making 9+ figures of online revenue ($100m or more), their leaders are constantly challenged to understand and grow their digital presence. It’s increasingly likely that folks at this level will have had close exposure to digital marketing earlier in their careers, but what level of detail do they need to know about their SEO program, and SEO testing in particular?
I have written before about what I think marketing leadership needs to know about how SEO has changed since they were hands-on with the channel. Today, I’m writing about a facet of the big question: what the C-suite should know about SEO testing.
Here’s what they should know:
SEO is changing, and effective teams running SEO for large websites are investing heavily in testing capabilities. The investment required to do it well is significant, but the results are powerful.
(If you’re operating at an even bigger company, with a large online business embedded in even larger offline operations, you can read this whole post as being about what the GM of the online business unit needs to know).
At a high level, I think the C-suite should know:
- SEO testing is possible - there are methodologies for testing on large websites that can give statistical confidence ranges on the scale and direction of SEO impact of on-site changes to large site sections
- SEO testing is increasingly necessary - many SEO programs are held back by rolling out too many changes that hurt performance in small enough ways that they aren’t detected and rolled back, and it is increasingly difficult to figure out which changes are going to be positive from best practices and experience alone
- There is a lot of value to unlock - beyond the obvious, the benefits of aligned product, engineering and SEO teams drives a multiplier effect
- SEO tests are complex - it’s hard to select good control and variant groups, and roll out the tests, and effective measurement takes significant work
- A good SEO testing program can help you beat the competition - it enables a qualitatively different SEO strategy that helps you and the team effectively explore and exploit the ways you can get ahead
Let’s start with the first point:
1. SEO testing is possible
Regular readers of the SearchPilot blog and case studies won’t be surprised by this news, but it’s still early days for the adoption of SEO testing across enterprise organisations. If there’s one single message I’m trying to get out there, it’s that SEO testing is possible and you can have an SEO program that gives measurable results connected to the activities carried out by your team.
What excites me about this space is that our work to innovate edges us closer to the dream of SEO getting investment comparable to the investment in paid search.
A few features of paid search have led to the incredible growth in so many companies’ investment in the channel, including:
- Return is clear and obviously connected to spend - you pay for clicks which result in visits, which result in sales
- Reporting can draw an obvious line between efforts to optimise spend and improvements in measurable outcomes
- As a result, the channel is worth spending money on and it’s worth spending money to improve the channel’s performance
In contrast, until recently, SEO hasn’t been as easy to budget for:
- While the value of organic traffic is huge and almost as easy to quantify as paid search traffic, the value of efforts to improve it has been harder to measure
- It’s been hard to tie activity to changes in organic search performance directly
It’s hard for anyone to have missed the incredible investments companies make in paid search (just look at Google’s revenue). Many of us in the SEO space believe that the difficulty of tying activity to outcomes in SEO is one of the reasons why large companies have historically underinvested in SEO compared to paid search.
By combining regular SEO testing with conversion rate and usage data into what we call full funnel testing, we can connect the activities of an SEO team and outcomes of an SEO program directly to the core metrics the business cares about - going as far as directly-attributed revenue.
2. SEO testing is (increasingly) necessary
In my article marginal losses, I explained how too many SEO programs underperform as a result of inadvertently rolling out changes that have negative impacts. Often these impacts are individually too small to be spotted immediately and corrected, but over time, the compound effect can be a disastrous friction and drag on SEO performance.
If you’re not performing controlled tests, there is essentially no way of spotting these drags on performance since their effects are camouflaged in the regular week-in-week-out fluctuations of traffic and rankings.
I also, in my article SEO is hard, wrote about how this problem is getting worse as it gets ever harder to figure out which changes are a good idea from first principles or prior experience.The size and scale of the web and the complexity of the search engines means you can’t just work out the answer and have confidence you’re right.
The addition of machine learning, with its lack of easy explanations and ability to customise “the algorithm” to different situations, verticals, and even queries, has made it even harder to know the right things to do.
3. There is a lot of value to unlock
Hopefully I don’t need to sell the general value of organic search traffic or of increasing it, but when it comes to testing, you do seek to capture the value of the winners (see, for example our article on 7 winning SEO tests) and avoid the losers (see 8 losing SEO tests).
In addition to that, there are a range of additional benefits that you and your team can capture:
- You can unlock a hugely more functional environment by enabling product, engineering, and SEO to work together more productively. It builds team trust to know that engineering is only asked to implement valuable SEO changes, and the quantification of SEO benefits is great for all involved
- Proof of the benefits of the program is not only beneficial to management, but also to overall team morale - everyone wants to know their work is worthwhile and makes a difference
- Improved prioritisation leads to a more effective workflow even beyond the benefits of the selection of important changes to make
4. SEO tests are complex
There are a bunch of reasons that most organisations have not yet embraced structured SEO testing as part of their investment program.
Before SearchPilot, the complexity was daunting for all but the largest and most technical of dedicated marketing engineering teams.
While most executives won’t need or want to know every detail about how SEO tests work, it’s good to know the shape of the complexity. Some areas to be aware of:
It’s important to select the right site sections
Site sections must be large enough (> 30,000 organic sessions / month, hundreds of pages) and have similar page templates.
It’s important to select the right kinds of tests
Tests should be bold enough to have a chance of making a real difference to the site’s performance and either be backed by a solid SEO hypothesis or be designed to evaluate product changes that are desired for other reasons.
Selecting control and variant groups is hard to do well
The most-visited pages in a site section can be visited hundreds of times more than the average page. It’s very easy to get uneven splits between control and variant groups as a result.
We search for control and variant groups with similar statistical properties to maximise the sensitivity and robustness of our tests.
Full funnel testing (conversion rate and SEO testing together) is even more complex, as you introduce even more degrees of freedom with varying conversion rates and seasonality.
Rolling out tests is rarely simple
Few content management systems make it easy to roll out a change to a randomly-selected variant group of pages.
Full funnel tests are even harder to deploy, requiring cookies to deliver a consistent site-wide user experience.
Measurement and analysis benefits from the latest innovations
If you want the brainy stuff, we moved from causal impact to a neural network model to increase the power, sensitivity, and confidence of our statistical analysis. Custom neural networks need a good amount of engineering and data science resource to run effectively.
You need highly sensitive statistics to detect small SEO uplifts with confidence.
5. You can beat the competition
One of the greatest benefits of frequent testing is unlocking the “explore and exploit” strategy: rapid iteration combined with confidence in not regularly implementing losing experiments.
You can beat the competition if you are able to explore high variance changes - highly uncertain hypotheses that would be impossible for competitors to roll out if they couldn’t be confident in their benefits.
By exploiting different approaches, you can stand out even in crowded competitive markets.
In addition, as mentioned earlier, you can iterate rapidly without fear of the marginal losses effect.
Ongoing information and updates
OK, so you know all of that, and you are running, or are on the verge of running, an SEO testing program. What information should you want delivered to you by the team on an ongoing basis for you to monitor, steer, and feedback on progress?
Depending on your distance from the details, you might request access to:
- Cadence / number of tests run in each major site section per quarter
- A classification of their effectiveness - the number of winners / losers / inconclusive tests - remembering that each of these is valuable:
- Winners are obviously good for the business, build morale, and deliver results
- Identifying losers prevents marginal losses, focuses effort on productive outcomes, and avoids more effort being expended in unproductive areas
- Inconclusive tests tell us things we can avoid wasting time doing - and are often “best practice” type changes that your competitors are wasting time on. By avoiding this, you build trust between your product, engineering, and SEO teams, and free up resource for more important product development and higher priority SEO changes
- Big wins / anecdotes / stories to shape the narrative of SEO investment
I hope this overview of executive-level SEO testing information has been useful. Feel free to send it to your boss if you are considering a testing program, get in touch if you would like to see SearchPilot in action, and drop me a line on twitter if you want to discuss any of the points I raise here.
Image credit: Nastuh Abootalebi.
