Analysis vs. science

The work of an analyst and the work of a scientist have some things in common. They’re both fundamentally truth-seeking endeavors, and they both rely on versions of the scientific method. But they’re also quite different. What explains those differences? Among other things, I’d venture it’s that analysis is designed to maximize truth-seeking in the near-term while science is set up to maximize it in the long-term.*

Take, for example, how an analyst and a scientist might set their bar for the quality of evidence. A scientist might say that for a finding to be taken seriously it needs to employ some plausible method of establishing causality, to have gone through peer review, etc. (These standards will vary by field.) Of course, the scientist wouldn’t say that evidence that didn’t meet those requirements was worthless. But they likely would treat these other sources of evidence as inputs and inspiration for more rigorous research that does employ the highest standards of their field. And when they assess the state of knowledge on a subject they’re more likely to emphasize what’s been established using that higher level of rigor.

The scientist’s norms are set up to encourage the steady progress toward truth over the long-term, which means gradually adding high-quality evidence and understanding to a field of knowledge. They can afford to treat less-rigorous evidence as input and inspiration because they’re focused on that long-term progress.

The analyst, by contrast, might have a far lower bar for rigor and might sample from a wider base of evidence. That’s because the analyst’s norms are set by the need to reach the best possible answer quickly, which often means reaching a conclusion in the face of scant or highly imperfect evidence. Analysis is a skill of its own.

One interesting example is the fact that Wall Street analysts and policymakers shied away from DSGE models of the macroeconomy even as they became popular within much of academic economics. The appeal of these models was (supposedly) that they improved on serious theoretical shortcomings of previous models and that they did a better job of connecting macro thinking to the economy’s micro-foundations.** You can see how these things would appeal to scientists, in the abstract at least. Over time, a science needs to improve its theories by improving their coherence, their ability to track reality, and their connection to other branches of science.

Macroeconomic analysts, meantime, were preoccupied by the near-term.***  They wanted to know what would happen, and how it would be affected by all sorts of variables — and they wanted the best available understanding now. The older class of models turned out to be better at this.

One implication of this is that scientists won’t always be the best guides to the empirical side of policymaking. Yes, they’re deeply informed and they often do put their “analyst” hat on when advising policymakers. But the skills they develop as scientists (researchers) are subtly distinct from the skills that analysts develop. Policymakers often need the best answer now, and that’s not always the same as the best answer that science has to give.

*Yes, Kuhn, paradigms — I know. But pragmatically speaking this still holds as at least one useful way to think about science.

**I think this holds whether or not you think the DSGE models were ultimately a misstep for macroeconomics as a field.

***This sometimes gets treated merely as the difference between “prediction” and “explanation” but that’s incomplete.