Return to Articles 7 mins read

Build vs Buy: Which is better for SEO testing?

Posted August 28, 2024 by Will Critchlow

I speak to a lot of folks in and around the SEO industry, and occasionally I meet people who are building in-house SEO testing capabilities. Two years ago I wrote up the build versus buy debate because it kept coming up in those conversations.

It still comes up, and now it comes with another surface attached. Teams are not only asking whether they can build SEO testing themselves. They are asking whether they can build something that measures AI search at the same time.

So here is the updated version: the benefits and drawbacks of building and buying, and what GEO and SEO A/B testingtogether does to the decision. 

A quick detour into the world of conversion rate optimization (CRO)

An informative place to start is by thinking about conversion rate optimization.

Testing has always been considered an integral part of the CRO process. SEO, on the other hand, was around for decades before structured, rigorous testing became common, and SearchPilot was among the pioneers.

For that reason, the CRO testing and tool market is more mature than its SEO counterpart. Observing the more advanced market lets us anticipate what might lie ahead for search testing, and if you want to go deeper on that, Ronny Kohavi'straining materials are worth your time.

Here is what is new. If SEO testing is a decade or so behind CRO, then GEO testing is a couple of years behind SEO testing. 

The case for partnering with a third-party vendor

I came across a presentation by Ryan Lucht, and I am really aligned with his thinking here. It helps that he works on something close to what we do without being a competitor, so I have no problem highlighting it. We have since had him on for a longer conversation about what search testing can learn from more mature experimentation programs.

Ryan's experience with enterprise customers typically involves them opting for third-party tools to increase the number of live experiments they can run. That matches my experience in SEO testing exactly. Our biggest pitches tend to revolve around a single idea: running more tests leads to more winners.

The reason is the shape of the returns. The concept of fat tails shows that a significant portion of wins comes from a small number of experiments. Over 90 per cent of the positive impact from a testing program comes from the top 5 per cent of experiments.

Most of your tests will be flat. The program pays for itself on the few that are not, and the only reliable way to find those is to run enough experiments to reach them. That is why we treat testing cadence as a top-level SEO KPI rather than as a nice-to-have.

A program running six tests a year is not a slower version of one running sixty. In fact, it is a program that may never see a big winner at all.

What that looks like in practice

One global sports brand we work with had already proved that SEO testing worked before they came to us. They had run tests, they believed the results, and they wanted more of them.

What they did not have was cadence. Manual setup and analysis kept the number of tests so low that the program wasn't producing enough insight to justify the effort.

That is the outcome nobody plans for when they decide to build. Not something that fails, but something that works and runs too slowly to reach the experiments where the money is.

The same program has since passed $20 million in return, including a single product detail page schema test worth an estimated $6 million a year.

What AI search changed

If you are considering building, AI discovery makes the job harder in three specific ways.

There is no best practice to fall back on. For traditional SEO, you can at least follow published guidance while you work out how to measure things. For AI discovery, that level of knowledge does not exist yet, so the choice is between testing and opinion. It also means the cost of a slow testing program is higher than it used to be, because there is no accumulated wisdom filling the gap while you wait.

Client-side implementations may measure nothing. Most AI assistants still don't execute JavaScript when they fetch a page. Andre Alpar's test of twelve AI assistants, published in June 2026, found that all seven US-built assistants tested (ChatGPT, Claude, Gemini, Perplexity, Meta AI, Copilot and Grok) answered from raw HTML when fetching a page live. That includes Gemini, even though Googlebot renders JavaScript when indexing pages for search. A variant deployed client-side can be invisible to the bots whose behavior you are trying to change, leading to inconclusive tests.

The volumes are small, and the surfaces are noisy. AI referral data arrives in far smaller quantities than Google organic, and model responses are not deterministic. That raises the bar on your statistical model rather than lowering it. It also means a page split that is well balanced for Google organic sessions is not automatically well balanced for AI referrals, so multi-metric balance has to be designed in from the first test rather than added at the tenth.

And then there is the finding that makes the case better than any of the above.

Our customer, Omio, ran two tests on the same question: what makes content more visible to AI search? The first changed a brand USP module and lifted LLM sessions by 18 per cent, with no clear effect on Google organic. It was rolled out. The second added a key takeaways module. It also lifted LLM traffic, but it would have cost 6.5 per cent of Google organic sessions, so it was not.

Two tests, two increases in AI visibility, one rollout. The second change would have looked like a win on any AI visibility dashboard on the market. Only a controlled experiment measuring both surfaces showed what it cost.

If what you build only measures one surface, that change ships and you never find out.

When you should build and scale your own testing capabilities

Partnering with a third party isn't always the right path, and some organizations with the scale and resources should build their own experimentation tools. Companies like Amazon, Netflix, and Meta buck the "buy is better" advice for several reasons:

  • They have CEO involvement. Their CEOs are deeply involved in experimentation and regularly talk about key terms like velocity and impact. Reed Hastings at Netflix taught Experimentation 101 to new hires.
  • Their founder or leadership team created the first iteration of their testing infrastructure themselves. This is something Mark Zuckerberg apparently did at Meta in the early days.
  • The philosophy and ethos of testing are baked into the organizational structure. These topics come up at board meetings and on investor calls.

We aspire to that level of executive engagement. Part of our product vision is to see our data, dashboards, and work gain visibility higher in the organizations we work with, including on earnings calls and in public company filings.

One thing I would add in 2026. Even in organizations like Amazon, Netflix, and Meta, search testing is a different problem to the experimentation they already do well. User testing platforms split users. Search testing splits pages, because crawlers only ever see one version of a page, which changes the bucketing, the forecasting, and the analysis. Even in a highly integrated case, you're likely starting from scratch on the SEO testing capability itself, which is exactly why many teams in the same position choose to buy in that expertise rather than build it.

If you are borrowing an internal experimentation platform for search, that difference is the first thing to work through, and the maths is worth reading properly.

Build vs Buy: The final verdict

Most companies operating today do not have the resources in place to realize that kind of experimentation culture. In that case, partnering with a third-party vendor is the most practical choice.

The all-in cost of building tools internally is high and often underestimated. When you factor in the costs of engineers, data scientists, and other high-skill resources, these expenses can quickly exceed even the largest tool budgets.

Focusing solely on cost, which is a temptation in any business, also overlooks the revenue and growth opportunities on the other side. Effective experimentation drives organic search traffic and, increasingly, visibility in AI search. For the teams responsible for those outcomes, the thing that matters most is the capability to design, implement, analyze, and deploy tests end to end, at a scale that reaches the winners.

There is one more cost that belongs in the comparison. Time spent building the measurement is time not spent running experiments (and the returns are in the experiments).

Learn more about building and executing successful SEO tests

If you are looking to get started, we have tons of material available: a free SEO A/B testing course, detailed guides on how testing works, what your engineering team will want to know, and a catalogue of test results. Sign up to the Flight Log to get the results of two of our most surprising experiments every month.

And if you would like to take SearchPilot for a test flight, get in touch to schedule a demo. 

Sign up to receive the results of two of our most surprising SEO experiments every month