More on why data helps

I’ve written before about the puzzle of why data helps. Why do even really basic, descriptive analytics seem anecdotally to be so useful?

First, just one more time evidence that data sure seems to help. From a paper in Nature about forecasting social change:

We compared forecasting approaches relying on (1) no data modelling (but possible consideration of theories), (2) pure data modelling (but no consideration of subject matter theories) and (3) hybrid approaches. Roughly half of the teams relied on data-based modelling as a basis for their forecasts, whereas the other half of the teams in each tournament relied only on their intuitions or theoretical considerations… Forecasts that considered historical data as part of the forecast modelling were more accurate than models that did not… There were no domains where data-free models were more accurate than data-inclusive models.

(Amazingly the data-only models did even better than hybrid models.)

By contrast:

The results from two forecasting tournaments conducted during the first year of the COVID-19 pandemic show that for most domains, social scientists’ predictions were no better than those from a sample of the (non-specialist) general public.

The only time the experts did better than the public? When they used data:

Which strategies and team characteristics were associated with more effective forecasts? One defining feature of more effective forecasters was that they relied on prior data rather than theory alone. This observation fits with prior studies on the performance of algorithmic versus intuitive human judgements²¹. Social scientists who relied on prior data also performed better than lay crowds and were overrepresented among the winning teams

Why does data work? Why does quantifying seem to be so useful?

Here’s a totally separate study at Voxeu that compares stories to data and illustrates a key driver of the systematic biases that drive human judgment awry: memory.

To examine the belief impact of stories versus statistics, we conducted controlled online experiments. The key idea of these experiments is to compare the immediate belief impact of stories and statistics to the belief impact after some delay, to isolate the role of memory. Participants in our experiment were informed that hypothetical products received a number of reviews. The task of participants was to guess whether a randomly selected review is positive. Before stating their guess, participants either received news in the form of a statistic, a story, or no information. We conceptualise statistics as abstract summaries of multiple data points (multiple reviews). Stories, by contrast, contain one datapoint (one review), but in addition provide contextual qualitative information about the review. Each participant saw three different product scenarios across which they were presented with one story, one statistic, and once no information. Crucial to our experimental design was that we elicited beliefs from participants twice, once immediately after they received the information and once following a delay of one day. This allows us to track the belief impact of stories versus statistics over time…

…both stories and statistics have an immediate effect on beliefs. On average, subjects immediately adjust their beliefs by about 20 percentage points for statistics, and by about 18 percentage points for stories. This pattern, however, looks markedly different after a one-day delay. While there remains a substantial belief impact for stories (about 12 percentage points), the belief impact of statistics drops to about five percentage points. In other words, we document a pronounced story-statistic gap in the evolution of beliefs. While the impact of statistics on beliefs decays rapidly, stories have a more persistent effect on beliefs. Using recall data, we confirm that the reason for this dynamic pattern is that stories are more easily retrieved than statistics.

As I wrote in my summary of behavioral economics, “We rely heavily on the information we can easily recall.” Memory gives us a biased view based on the stories we can most easily recall. But what comes easily to mind may have little to do with what’s actually going on: we’re misled by what’s most unusual or extreme or striking. Data works because it focuses us on what usually happens, not what is most memorable, and so has a de-biasing effect.

The amazing thing is that our judgment is so poor that a lot of the time we can’t do any better than just totally deferring to a super basic, content-free extrapolation from that data. Quantification has its own problems, of course. It helps not because it’s so great but because of how limited human reason can be.