Rand Fishkin’s team ran 2,961 prompts across ChatGPT, Claude, and Google AI. 600 volunteers, 12 different prompts, two months of runs. They wanted to answer one question: how often do you see the same list of brand recommendations twice, even with the exact same prompt?
The answer? Less than 1% of the time. The odds of seeing the same list in the same order are closer to one in a thousand.
Most conversations about AI inconsistency treat it as a measurement problem: how do I know if my brand is showing up? That’s a legitimate question. But it’s not the only question. And it might not even be the most important one.
If AI systems give different recommendations essentially every time, the same inconsistency is already baked into every AI chatbot you’ve deployed — your hotel chat widget, your B2B sales assistant, your customer service tool. Most teams have never measured it. And some of those inconsistent answers are already driving negative reviews for your brand and business.
This episode connects three stories Tim has covered over the last three weeks — the AI value gap, the uncertain timeline of agentic commerce, and now AI inconsistency — showing that they all stem from the same underlying condition. It also explains what City of Hope, appearing in 69 of 71 AI responses for "West Coast cancer care hospitals," tells us about how you can fix this problem for your business.
Key Insights for Strategic Leaders to Close the Gap
In this episode, Tim Peter breaks down:
- The full SparkToro/Gumshoe.ai research — and what it actually means. Rand Fishkin and Patrick O’Donnell ran nearly 3,000 prompts with 600 volunteers. The list of brands recommended changed more than 99% of the time. Here’s why that reframes everything about how you should be tracking AI visibility.
- The operational problem most people are missing. AI inconsistency isn’t only a marketing measurement challenge — it’s a liability inside your own deployed tools. Your AI chatbot may be giving materially different answers to different customers right now. And, it’s almost certain that no one on your team is measuring that.
- City of Hope: what 97% consistency looks like. Why City of Hope appeared in 69 of 71 AI responses for "West Coast cancer care hospitals" and what that reveals about how AI decides which brands it’s willing to commit to — and which ones it isn’t.
- Why "post more content" is the wrong strategy. How AI actually works: triangulation across independent sources, why your own website is a low-weight signal, and what "digital witnesses" means for building prompt brand equity that holds up.
- The King, Queen, and Crown Jewels operating model. Content is king, customer experience is queen, and data is the crown jewels, not just as a branding concept, but as the mechanism that drives the AI’s confidence in your brand.
- Four moves to make this week.
- Shift from rank to frequency measurement.
- Audit your deployed AI tools for consistency before worrying about external AI visibility.
- Build credible witnesses, not content volume.
- And treat review velocity as a strategic input, not just a reputation metric.
Whether you’re in hospitality, retail, or B2B, this episode is for anyone who’s deploying AI in a customer-facing role… or who’s who’s being asked to report on AI visibility and wants a better sense of what they’re actually measuring.
The AI Coin Flip: Why AI Gives Every Customer a Different Answer (Digital Reset Episode 488) — Headlines and Show Notes
Show Notes and Links
Buy the Book — Digital Reset: Driving Marketing and Customer Acquisition Beyond Big Tech
Tim Peter has written a new book called Digital Reset: Driving Marketing Beyond Big Tech. You can learn more about it here on the site. Or buy your copy on Amazon.com today.
Past Appearances
Rutgers Business School MSDM Speaker: Series: a Conversation with Tim Peter, Author of "Digital Reset"
Free Downloads
We have some free downloads for you to help you navigate the current situation, which you can find right here:
Subscribe to Thinks Out Loud
Contact information for the podcast: podcast@timpeter.com
Past Insights from Tim Peter Thinks
Technical Details for Thinks Out Loud
Recorded using a Shure SM7B Vocal Dynamic Microphone and a
