Economics studies tend to replicate at a higher rate than psychology studies. Why? One possibility is that economics has a more unified theoretical framework to help guide researchers toward hypotheses that are more likely true, whereas theories in psychology are numerous and not well integrated.
Joseph Henrich has made this argument, and wants psychology to root itself in evolutionary theory. And Matt Clancy, an innovation researcher, lays out the case at Works in Progress. He argues that theory helps lead researchers to hypotheses that are more likely to be true and broad, unified theoretical frameworks allow more chances for a new study to support or refute an old study without it being an actual replication.
I’ve been skeptical of the idea that more theory is the answer to the replication crisis, mostly because I think the dominant unified framework in economics has a lot of downsides. One of the biggest movements in the field in the last 30 years was behavioral economics, which largely just pointed out empirically the many ways that the dominant theoretical framework failed to capture economic behavior. It borrowed from psychology, wasn’t that theoretically unified, and represented real progress for the field of economics. In macro, meanwhile, large chunks of the profession seemingly failed to understand the great recession because of their preference for theory — for beauty over truth in Krugman’s estimation.
But maybe theory does help with replication. Maybe that’s what you get in return for these other limitations; sure you miss a huge chunk of human behavior but where your theories do apply you produce stable results.
Clancy writes on his substack about a study looking at how theory affects publication bias. It presents evidence that when a theory predicts a specific relationship (rather than being ambiguous and allowing for multiple results) there’s more publication bias. Microeconomic theory predicts that less of a good is demanded at higher prices (demand curve slopes down). So:
Studies that estimate demand exhibit much more selection bias than those that don’t… In other words, when economists get results that say there is no relationship between price and demand, or that demand goes up when prices go up, these results appear less likely to be published.
So what do we make of this?
If you’re really skeptical of the empirical turn in economics and think there are a laundry list of other problems with these papers outside of publication bias, you might argue that this “bias” is what’s helping economics papers replicate better. Publication bias sounds bad, but in this view empirical social science is so screwed up that theory is serving as a useful line of defense. You have a paper finding that the demand curve actually slopes up? That defies basic theory, so get that out of here. One more spurious result saved from publication.
The more straightforward response, I think, is to view this as a risk of too much deference to theory. Yes, theory saves some bad, spurious papers from being published, but it’s a real problem if a theory is allowed to capture the publication process. Theory is essentially banning results that contradict it!
My hunch is that this is another reason to favor a “many models” approach. Sure, maybe you need more theory than psychology. But rather than aspiring to one dominant unified framework — everything is optimizing, self-interested agents! everything is grounded in evolution — I think you’d more realistically want a manageable collection of theoretical frameworks. For example, economics needs models that can account for irrationality and cooperation, even if they aren’t perfect fits with the basic workhorse micro models.
“Models” — the abstract, typically mathematical frameworks that economists use to make sense of the world — form the heart of the book. Models are both economics’ strength and its Achilles’ heel; they are also what makes economics a science — not a science like quantum physics or molecular biology, but a science nonetheless.
Rather than a single, specific model, economics encompasses a collection of models. The discipline advances by expanding its library of models and by improving the mapping between these models and the real world. The diversity of models in economics is the necessary counterpart to the flexibility of the social world. Different social settings require different models. Economists are unlikely ever to uncover universal, general-purpose models.
(Posts on economic models here, here, here, here, here.)
This is all very speculative, but my sense is that the many model approach still allows theory to inform hypotheses while also allowing data that challenges one or more of those theories to get published.
beyond the times 