AI Content Is Cheap, Lost Traffic Isn't: A Roadmap for Testing What Works

When teams talk about AI content, the conversation often starts in the wrong place.

I am Demetria Spinrad, a consultant at SearchPilot, and this webinar was my attempt to slow everyone down before they flood their site with AI content and then try to explain the fallout later.

They ask whether AI content can work at all. At this point, that is not the most useful question. We already know AI-generated content can improve organic traffic in some situations. We have seen that in testing. The harder question now is which AI content is worth publishing, which model should create it, which prompts shape it well, and which quality checks keep it from hurting long-run performance.

That was the focus of this webinar. I wanted to move past the shallow version of the conversation and give a practical roadmap for testing AI content the same way we would test any other meaningful SEO change: with a clear hypothesis, a structured sequence of experiments, and enough rigor to avoid rolling out content that it's useless for both users and search engines.

View webinar slides here.

Key takeaways

AI content can lift organic traffic, but results from early tests (like 2023-era rollouts) may not hold today because models, Google, and user expectations have all shifted.
Start with a hypothesis and treat AI content as a testing program, not a one-off rollout: each result should lead to the next, tougher question.
Prove a content element is worth refreshing before you rewrite it by running a value series: remove it, hide it on load, or move it down the page and measure the impact.
Use multi-metric and full-funnel thinking: content changes can affect more than blue-link clicks, and they can help or hurt user behavior, so pair SEO testing with CRO signals.
Do not default to the most popular LLM. Compare models head-to-head (across vendors and within a vendor) using the same prompts to find what fits your content needs.
Test prompts, not just models. Small prompt changes (keywords, inputs, specificity) can produce meaningfully different content and different SEO outcomes.
Translation and localization are high-risk at scale. NMT and LLM translation behave differently, and quality varies by language and region, so test workflows and add guardrails plus human QA.

The old AI content question is no longer enough

Some of the earliest AI content tests we ran were done back in 2023, when public LLMs were new enough that most teams were still asking the basic question: if we add AI-generated content to pages, will traffic go up?

In one of those early tests, the answer was yes. Adding AI-created content improved organic traffic. That result mattered because it proved something important: search engines were not dismissing content simply because it came from an AI workflow.

But that does not mean the same content would still hold up today. Models have changed. Users' brains haven't changed, but search engines' ability to evaluate content has changed. So the goal cannot be "publish AI content because it worked once." The goal has to be testing your way toward content that is genuinely useful, well-placed, well-written, and worth keeping.

But that does not mean the same content would still hold up today. Models have changed. Google's ability to judge quality has changed. A content block that looked helpful two years ago may now read like filler. So the goal cannot be "publish AI content because it worked once." The goal has to be testing your way toward content that is genuinely useful, well-placed, well-written, and worth keeping.

A better roadmap starts with better questions

The framework I shared is built around five stages:

checking value,
choosing models,
choosing prompts,
optimizing content,
translating or localizing it for global sites.

Each stage starts with a question.

Is this content element valuable at all?
Am I using the best model for this job?
Are my prompts producing useful, relevant output?
Is the content the right length and targeting the right terms?
Will it still work when translated into other languages?

That matters because SEO testing should never start with "let's generate a bunch of text and hope." It should start with a question you are trying to answer. Then, once you get your first result, you keep going. A positive result is not the end of the road. It is the start of more focused questions about placement, wording, model choice, query targeting, and long-term usefulness.

Before you create new content, prove the old content deserves a refresh

One of the biggest mistakes teams make is assuming an existing content element deserves to be rewritten, expanded, or replaced before they have proved that it matters.

That is why I recommend starting with a content value series. Before you spend internal time generating AI or human-written content, test the element you already have. Remove it. Hide it on page load. Move it lower on the page. If Google treats that content as important, those tests should hurt performance. If the result is flat, or even positive, you have learned something useful: the content may be stale, weak, misplaced, or simply less relevant than you thought.

This is one of the most valuable places to begin because it prevents waste. If a large chunk of text can be removed without consequence, that tells you not to spend more time polishing it. If hiding it changes nothing, that suggests Google is not treating it as central to the page anyway. If moving it lower has no downside, maybe your best page signals belong somewhere else.

A negative result in these tests is good information too. It tells you the element still carries value, which means a refresh is more likely to matter.

Content value is about more than blue-link traffic

Content does not only affect classic organic clicks.

That is why I stressed multi-metric testing. A content change might look flat if you only measure one familiar traffic source, while still changing how often you appear in product search features, how often people interact with listings before reaching your site, or how much traffic arrives through LLM-driven discovery. Search has become too crowded, and too fragmented, to pretend there is only one useful outcome to watch.

I also pushed for full-funnel thinking. If a content block exists only for SEO, that is already a warning sign. Users still see that content. They still react to it. So if you remove a block and rankings stay steady but user behavior improves, that matters. If you add content and rankings improve but engagement drops, that matters too. SEO testing and CRO testing should work together because a page that satisfies both search engines and users is stronger than one that only performs for one audience.

Choosing the right model is not the same as choosing the most famous one

Many teams default to the same few names. They use ChatGPT or Gemini because those are the tools everyone talks about.

That is understandable, but it is not a testing strategy. If you are only using one model from one company, you may be missing a better fit for your actual task. Some models are stronger at structured writing. Some are better at multilingual work. Some are lighter, cheaper, or more consistent. Some are better at following instructions and worse at tone. The only way to know is to compare them.

I recommended two simple model comparison routes. First, generate the same content with models from different companies using the same prompt. Second, generate the same content with different models from the same company. That second one matters more than people expect. Even within one provider, model differences can change how well the output fits the page, the keyword target, or the brand voice.

The point is not to crown a universal winner. The point is to stop treating model choice like a cosmetic preference. It is a meaningful variable in the testing process.

Prompts deserve testing too

A lot of teams test models and then treat prompts as if they are just instructions typed quickly into a box.

That is a mistake. Prompts shape the output as much as the model does, and sometimes more. If you use the same model with three different prompts, you may get three very different SEO outcomes.

I spent time in the webinar breaking this down because "AI" still gets talked about as if it has human understanding. It does not. A large language model is not sitting there thoughtfully considering your business goals. It is predicting text. It can be impressively good at that, but it still does not understand your brand, your users, or your product the way your team does.

That is why prompts need their own testing roadmap. One prompt might ask the model to rewrite existing title tags. Another might require it to include one new relevant keyword. Another might ask it to generate titles from product descriptions rather than old title tag content. Those are not minor changes. They can change the quality, specificity, and usefulness of the output in ways that affect performance in search.

The rule here is simple: do not test only what model you use. Test what you ask it to do.

AI can also help you cut content down

The popular idea is that AI helps you create and expand. It can, but it can also help you simplify.

That matters because longer is not always better. Some pages are carrying heavy, bloated content blocks that do not help users and do not send especially good signals to search engines either. In those cases, AI can be useful for reducing length while preserving the important ideas.

I shared test ideas like shortening an existing content block while preserving relevant keywords, turning a long product description into a clean bulleted feature list, or summarizing user reviews into a compact, readable block that highlights what customers consistently care about. These are all examples of AI helping make content more usable, not merely more abundant.

That is an important shift in mindset. Good content optimization is not always about addition. Sometimes it is about clearer structure, sharper relevance, and less clutter.

Translation and localization are where things get risky fast

This is one of the places where teams are most tempted to scale before they understand the downside.

If you run a site in many languages, AI and machine translation can look like the perfect shortcut. And in one sense, they are. They let you produce or adapt content at a speed no manual process can match.

But they also create one of the easiest ways to publish text your own team cannot properly review.

That is why I spent so much time on multilingual testing. Even if you are confident you found the right model for English, you should not assume it will be the right model for French, German, Japanese, Chinese, or even regional variants of English. Many of these systems are strongest in US English because that is where the biggest training volume sits. A model that sounds sharp and natural in one variety of English may sound awkward, outdated, or subtly wrong in another.

I also walked through why not all translation systems work the same way. Statistical machine translation, neural machine translation, and LLM-based translation each have different strengths and weaknesses. NMT can be more literal and often more dependable. LLM translation can sound more natural and more human, but it also increases the risk of inaccuracies and hallucinations. That makes it attractive and dangerous at the same time.

So the right way to approach translation is through comparison. Test NMT against LLM translation. Test machine translation against a human translator. Test workflows where one system generates and another checks for drift or errors. If your team cannot read the output, you need more than blind trust. You need guardrails.

This is not one test. It is a test cycle.

The most important idea in the whole webinar may be that there is no finish line here.

You are not going to run one model test, one prompt test, and one translation test, then declare your AI content strategy solved. Search engines will keep shifting how they judge quality. Models will keep changing. User patience will keep changing too. Content that works today may look weak a year from now. A prompt that produces useful output now may become sloppy after a model update.

So the right frame is a cycle. Check value. Compare models. Refine prompts. Optimize content. Test translations and localization. Then loop back through it again as conditions change.

What I hope teams take away

The cheapest part of AI content is the generation step. That is exactly why teams get into trouble.

What costs you is the traffic lost when low-value content goes live at scale, the cleanup when inaccurate content gets published in markets your own team cannot evaluate, and the time spent polishing content elements that never mattered in the first place.

So my advice is simple. Slow down before you scale up. Prove value before you refresh. Compare models before you standardize. Test prompts before you automate. Add human QA before you publish. And if you operate globally, treat translation as a serious quality problem, not a box to tick.

AI content can work. It can help a lot. But only if you test it like it matters.

Put Search in Control Mode with SearchPilot

If there is a theme underneath all of this, it is that publishing first and learning later is a bad bet. The safer path is to turn content decisions into measurable experiments, so you can see what is helping, what is neutral, and what is quietly hurting performance before it spreads across the site.

That is exactly where SearchPilot fits. Search is often the biggest channel and the least understood, and SearchPilot makes SEO and GEO testable so teams can move from guesswork to evidence. We run controlled experiments across page templates, content elements, navigation, and key commercial surfaces, then help teams read the result with confidence.

We run controlled experiments across category pages, product detail pages, navigation, and content, then deliver clear uplift with timelines and confidence. Teams move from quick validation to a steady test cadence to full control, turning search into a performance channel you can plan and fund.

For ecommerce teams focused on product grids and Merchant Center feeds, the first step is a focused test plan. Measurement tracks impressions, clicks, and revenue so leaders can see the real impact.

Stop trying to predict the future. Experiment to discover it. If you want tailored test ideas for your top PLPs and PDPs, schedule a demo and we will share a starter list and a clear path from validation to velocity to control.

Share this post