What we learned from GEO A/B testing for Omio
As more people turn to LLM-powered tools like ChatGPT for product discovery and search, many teams are making a sweeping assumption: the changes they are making for SEO purposes will benefit them in LLMs too.
Our A/B testing shows that this assumption isn’t reliable. SEO and GEO aren’t the same thing.
Download the full
customer success story
Our key discovery
The same on-page change can elevate LLM-driven discovery while harming your organic performance on Google or vice versa. A win in one channel does not necessarily equal a win in the other.
We have been applying our A/B testing approach to the world of AI and GEO. The testing method stays the same. The pages you’re testing stay the same. It’s your testing focus that shifts from top-of-funnel visibility to bottom-of-funnel influence. You're no longer testing against a single search result, but against an entire AI-driven research journey.
About Omio
Omio is a leading global travel booking platform. By unifying the fragmented world of transport into a single, easy-to-use app, travellers can find, compare, and book trains, buses, flights, and ferries from thousands of partners – and find the journey that’s right for them. Omio is available for travel in 47 countries across Europe, the US, Canada, Southeast Asia, Japan and Brazil. Every day, more than 100,000 people travel with Omio.
The challenge
AI-powered discovery is moving fast and tools are developing rapidly, with new capabilities and better results. But it is still a relatively new phenomenon, and how it interacts with SEO is not yet clearly-defined. Businesses want to increase visibility by appearing in AI overviews and in LLM recommendations, but without testing, it’s all guesswork.
Being able to test website changes and understand their impact on AI-powered search is vital to improving AI visibility. But our customers don’t want to do it at the expense of SEO. For businesses, this means experimenting with Generative Engine Optimization (GEO) while protecting organic traffic from Google. That is where SEO and GEO testing with SearchPilot come in.
The hypothesis
We tested whether adding brand USPs to its site would have a positive impact on LLM traffic because LLMs tend to surface content that clearly explains what makes a brand different. USPs quickly communicate why a user should choose one provider over another.
The change
We added clear, well-structured brand USPs to make it easier for LLMs to understand and summarise the page’s value. Our hypothesis was that this would increase the likelihood of these pages being cited or linked to in LLM-driven responses, uplifting LLM sessions on variant pages compared to control pages.
The outcome
The test showed a clear uplift in LLM sessions, indicating that adding the brand USPs made these pages more visible and useful within LLM-driven experiences. The impact on Google organic sessions was inconclusive.
The decision
Given the clear benefit observed in LLM traffic and no evidence of harm to Google's organic performance, the result supported rolling out the USPs module across relevant pages. This is a good example of a GEO-first change that can add incremental value without harming SEO.
The solution
Measure SEO and GEO impact of changes at the same time
With SearchPilot, a single test can measure the impact of a website change on both traditional organic search and on AI discovery. Our findings? What works for LLMs isn’t always what works for Google (and vice versa). We have found hypotheses that have benefited one and are harmful to the other, as well as cases where the impact is aligned. Businesses must consider the NET impact of website changes on all organic channels and conversion rates to make the best rollout decisions.
Run robust page-split tests with an AI-specific hypothesis
We applied our scientific page-splitting SEO testing approach to the world of AI and GEO. Thousands of similar pages are divided into statistically similar control and variant groups. We then apply GEO-specific hypotheses to the variant group, such as formatting content specifically for LLM scannability. Then, we measure the referral traffic from LLM-powered sources (like ChatGPT) alongside organic search traffic and the impact on clicks through to the site from the different sources with statistical confidence.
Prove the ROI of AI discoverability to leadership
With explosive growth in AI use posing the first real threat to Google’s dominance in organic discovery in over two decades, marketing leaders are asking their teams to be ready. Teams need to be able to defend search visibility, understand the tradeoffs between AI and SEO, and ultimately identify the ROI of AI optimisation. SearchPilot has unlocked new insights, made intentional changes to boost impressions and visibility in AI search, and maintained the safety of Google organic search as the key traffic driver.
The hypothesis
LLMs may favor structured, summary-style content when interpreting and referencing pages. Our hypothesis was that adding ‘key takeaways’ would positively impact LLM traffic and improve user engagement, but we would have to monitor any secondary impact on Google organic sessions.
The change
We added bulleted, quantitative and comparative on-page data above an existing summary table in the HTML.
The outcome
While the change performed positively for LLM-driven traffic, it would likely hurt Google organic sessions by -6.5%. The test delivered mixed results across traffic sources, which could be due to content overlaps, reduced prominence of crawlable data or other important content, or increased AI overview exposure leading to fewer clicks.
The decision
Because the primary goal was to improve how the page gets interpreted and summarised by LLMs, Omio considered the test positive. However, Google organic traffic is Omio’s largest traffic source, so the decision was made not to roll out this change. Instead, we have developed iterations of this test to see whether a more balanced result can be achieved.
