Episode 93: Many analysts

Episode 93: Many analysts

Author: Tom Chivers and Stuart Ritchie January 13, 2026 Duration: 1:18:44

Here’s a cheery one for our first episode of the year. Guess what happens when you give several sets of scientists the same dataset and ask them to answer the same question? Well, they all find the same results, right? Right!?

Sadly not. This “Many Analysts” problem has been analysed and debated in multiple different scientific fields and across several papers. We cover them in this episode. What does it tell us about the objectivity of science if different teams draw different conclusions from the exact same data?

The Science Fictions podcast is brought to you by Works in Progress magazine. Their excellent new article on how we’re living in “the golden age of vaccine development”, as discussed on the show, can be found (along with the rest of their articles on science, history, and technology), at worksinprogress.co. We’re very grateful that they support the podcast.

Show notes

* 2015 Nature commentary article on “crowdsourced research” (on racism in football)

* And the full 2018 writeup titled “Many Analysts, One Data Set

* Gelman and Loken on the “Garden of Forking Paths

* 2020 many-analysts neuroscience (fMRI) paper

* And the plan for the similar study on EEG

* 2022 PNAS many-analysts paper on the “hidden universe of uncertainty”

* 2026 critique on ideological bias from George Borjas

* 2023 critique on effect sizes vs. statistical significance

* 2025 ecology & evolution many-analysts paper on blue tits and eucalyptus

* 2025 economics many-analysts paper with results on data cleaning

* 2024 PNAS critique of many-analysts research

* Julia Rohrer’s critique of multiverse analysis

Credits

The Science Fictions podcast is produced by Julian Mayers at Yada Yada Productions.



This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit sciencefictionspod.substack.com/subscribe

Every week, Tom Chivers and Stuart Ritchie pull apart the biggest arguments and most confusing claims making headlines. Science Fictions isn't just about celebrating breakthroughs; it's a necessary dive into the messy, contested, and often surprising realities of how science actually works. You'll hear them unpack heated debates, examine questionable studies, and explore why even solid research can sometimes lead to public confusion. This podcast serves as a guide through the noise, separating compelling evidence from overblown narratives. Each episode feels like a conversation with two deeply informed friends who aren't afraid to ask tough questions, offering clarity on topics that matter. For anyone curious about the stories behind the science headlines, this is an essential listen. Tune in for thoughtful analysis that goes beyond the press release, grounded in a genuine fascination with how we know what we know.
Author: Language: English Episodes: 100

Science Fictions
Podcast Episodes
Paid-only Episode 17: The lab leak hypothesis [not-audio_url] [/not-audio_url]

Duration: 10:47
This is a free preview of a paid episode. To hear more, visit sciencefictionspod.substack.comIt’s been five years since the start of the COVID pandemic (yes, you read that correctly—five years). And the debate still rage…
Episode 64: IQ [not-audio_url] [/not-audio_url]

Duration: 1:14:58
Every so often there’s a controversy related to IQ. The latest was caused by [checks notes] the new Vice President of the US attacking the IQ of a political podcaster on Twitter.You could argue that the VP should have be…
Episode 63: Philosophy of science [not-audio_url] [/not-audio_url]

Duration: 1:10:24
It had to happen eventually: this week The Studies Show is all about philosophy. As we look at science in general, how do we decide what those studies are actually showing? Tom and Stuart take a look at the Big Two of ph…
Episode 62: Violent videogames [not-audio_url] [/not-audio_url]

Duration: 1:04:55
Before the panic over social media—but after the panic over “video nasties”—was the panic over violent videogames. Was Pac-Man causing little Johnny so much frustration that he’d take it out on his siblings with his fist…
Paid-only Episode 16: Recycling [not-audio_url] [/not-audio_url]

Duration: 11:17
This is a free preview of a paid episode. To hear more, visit sciencefictionspod.substack.comIs recycling worthwhile? Is sending your rubbish to landfill actually so bad? Grab your cotton tote bag and join Tom and Stuart…
Episode 61: Conflict of interest [not-audio_url] [/not-audio_url]

Duration: 1:07:21
We want scientists to be paragons of objectivity. At the very least, we want them to tell us who’s paying their bills. But it turns out that in some fields of research, the norms about reporting financial conflicts of in…
Un-paywalled: Youth gender medicine & the Cass Review [not-audio_url] [/not-audio_url]

Duration: 1:15:24
This week, as a gift for New Year’s Eve, we’re opening up a previously-paywalled episode so that everyone can listen. It’s our episode from April 2024 on “Youth gender medicine & the Cass Review”. Since the show notes we…
Episode 60: Best and worst science of 2024 [not-audio_url] [/not-audio_url]

Duration: 1:05:41
In this final episode of 2024, Tom and Stuart talk about the most exciting scientific breakthroughs of the year… but temper it with some of the worst episodes of scientific fraud and misconduct, too. Then, just as a bonu…
Episode 59: The apocalypse [not-audio_url] [/not-audio_url]

Duration: 1:04:32
In this “fun”, festive episode of The Studies Show, Tom and Stuart discuss two ways—one man-made, one natural—that our species might be wiped off the planet.The first is “mirror life”, a science-fiction-sounding threat t…
Paid-only Episode 15: Sex and sport [not-audio_url] [/not-audio_url]

Duration: 10:45
This is a free preview of a paid episode. To hear more, visit sciencefictionspod.substack.comRather unexpectedly, the idea of separate sports for males and females has become massively controversial—a major flashpoint in…