Vox’s Future Perfect newsletter reports:
“Just carefully reading a paper — even as a layperson without deep knowledge of the field — is sufficient to form a pretty accurate guess about whether the study will replicate.
Meanwhile, DARPA’s replication markets found that guessing which papers will hold up and which won’t is often just a matter of looking at whether the study makes any sense. Some important statistics to take note of: Did the researchers squeeze out a result barely below the significance threshold of p = 0.05? (A paper can often claim a “significant” result if this threshold is met, and many use various statistical tricks to push their paper across that line.) Did they find no effects in most groups but significant effects for a tiny, hyper-specific subgroup?
“Predicting replication is easy,” Menard writes. “There’s no need for a deep dive into the statistical methodology or a rigorous examination of the data, no need to scrutinize esoteric theories for subtle errors—these papers have obvious, surface-level problems.”
This is important work and I get the point but in a way it’s studying things backwards. It’s assessing whether laypeople can do better than random at predicting which studies will replicate, which, again, is important. But the test of the studies’ usefulness is really whether they can help people improve their judgments, not the other way around.
The study I’d like to see would work like this: A group of people is asked to predict the result of a forthcoming study which, unbeknownst to them, is a replication of a past study. They’re asked to predict the effect that some intervention has on some outcome variable. One group, the control, makes this prediction just based on their knowledge of the world. The other group, the treatment, gets access to the original study. They can read it, see its result and methodology, and then incorporate that (if they want to) in making their prediction.
Would access to the original studies improve peoples’ predictions?
beyond the times 