Episode 93: Many analysts

Episode 93: Many analysts

Author: Tom Chivers and Stuart Ritchie January 13, 2026 Duration: 1:18:44

Here’s a cheery one for our first episode of the year. Guess what happens when you give several sets of scientists the same dataset and ask them to answer the same question? Well, they all find the same results, right? Right!?

Sadly not. This “Many Analysts” problem has been analysed and debated in multiple different scientific fields and across several papers. We cover them in this episode. What does it tell us about the objectivity of science if different teams draw different conclusions from the exact same data?

The Science Fictions podcast is brought to you by Works in Progress magazine. Their excellent new article on how we’re living in “the golden age of vaccine development”, as discussed on the show, can be found (along with the rest of their articles on science, history, and technology), at worksinprogress.co. We’re very grateful that they support the podcast.

Show notes

* 2015 Nature commentary article on “crowdsourced research” (on racism in football)

* And the full 2018 writeup titled “Many Analysts, One Data Set

* Gelman and Loken on the “Garden of Forking Paths

* 2020 many-analysts neuroscience (fMRI) paper

* And the plan for the similar study on EEG

* 2022 PNAS many-analysts paper on the “hidden universe of uncertainty”

* 2026 critique on ideological bias from George Borjas

* 2023 critique on effect sizes vs. statistical significance

* 2025 ecology & evolution many-analysts paper on blue tits and eucalyptus

* 2025 economics many-analysts paper with results on data cleaning

* 2024 PNAS critique of many-analysts research

* Julia Rohrer’s critique of multiverse analysis

Credits

The Science Fictions podcast is produced by Julian Mayers at Yada Yada Productions.



This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit sciencefictionspod.substack.com/subscribe

Every week, Tom Chivers and Stuart Ritchie pull apart the biggest arguments and most confusing claims making headlines. Science Fictions isn't just about celebrating breakthroughs; it's a necessary dive into the messy, contested, and often surprising realities of how science actually works. You'll hear them unpack heated debates, examine questionable studies, and explore why even solid research can sometimes lead to public confusion. This podcast serves as a guide through the noise, separating compelling evidence from overblown narratives. Each episode feels like a conversation with two deeply informed friends who aren't afraid to ask tough questions, offering clarity on topics that matter. For anyone curious about the stories behind the science headlines, this is an essential listen. Tune in for thoughtful analysis that goes beyond the press release, grounded in a genuine fascination with how we know what we know.
Author: Language: English Episodes: 100

Science Fictions
Podcast Episodes
Episode 99.5: Candidate genes [not-audio_url] [/not-audio_url]

Duration: 1:13:09
Here’s another one for the annals of “entire scientific field becomes totally misguided for decades”. How could it have been possible that so many scientists fell for the idea of candidate genes—that there were individua…
Episode 99: Power posing [not-audio_url] [/not-audio_url]

Duration: 1:13:01
For a while in the early-to-mid 2010s, the most prominent psychology research in the world was on power posing. Harvard’s Amy Cuddy did a TED talk that reached tens of millions; her exhortation to “fake it til you make i…
Paid-only episode 28: Tourette's syndrome [not-audio_url] [/not-audio_url]

Duration: 10:21
This is a free preview of a paid episode. To hear more, visit sciencefictionspod.substack.comHave you seen the award-winning film I Swear, about a Scottish man with Tourette’s? (The less said about what happened at the c…
Episode 98: Dark oxygen [not-audio_url] [/not-audio_url]

Duration: 58:38
In 2024, researchers claimed to find something mysterious at the bottom of the ocean. It was “dark oxygen”—oxygen produced where there’s no chance of photosynthesis. So what could possibly be producing it?Natural batteri…
Episode 97: The 2D:4D digit ratio [not-audio_url] [/not-audio_url]

Duration: 1:03:23
The last few episodes have been pretty heavy. So here’s… well, here’s the 2D:4D ratio. Does the difference in length between your index finger and your ring finger reveal a huge amount about your personality (and much mo…
Paid-only episode 27: Antidepressants [not-audio_url] [/not-audio_url]

Duration: 9:06
This is a free preview of a paid episode. To hear more, visit sciencefictionspod.substack.comAnd now… following last week’s episode on ECT, here’s part two of our double episode on depression treatments. This time we’re…
Episode 96: Electroconvulsive therapy [not-audio_url] [/not-audio_url]

Duration: 1:13:25
Open up some scientific papers, and you’ll hear electroconvulsive therapy described as the most effective treatment for depression (especially very severe depression). But open up others, and you’ll see it described as c…
Episode 95: Critical thinking [not-audio_url] [/not-audio_url]

Duration: 1:01:00
This episode is dedicated to Justin Eldridge.We like to think that, in often hamfisted ways, we’re applying critical thinking on this show. But what even is “critical thinking”? Can you measure it? Can you teach it to ki…
Episode 94: Medical marijuana [not-audio_url] [/not-audio_url]

Duration: 1:02:58
Here’s another episode that revists a topic we’ve covered before. A while back, we did an episode on the downsides of cannabis (for example, the risk of psychosis). But of course, a lot of people claim there are medical…
Paid-only episode 26: Microplastics redux [not-audio_url] [/not-audio_url]

Duration: 10:47
This is a free preview of a paid episode. To hear more, visit sciencefictionspod.substack.comIt’s rare that we return to a topic, but it’s also nice to have been right. In 2024 we did an episode on microplastics, and cas…