beyond the times

I recently read Radical Uncertainty by the economists John Kay and Mervyn King. A few notes, then a bunch of block quotes that stood out to me…

Notes

• I strongly disagree in practice with their argument against probabilistic reasoning. Only economists who’ve spent time in finance and business schools could possibly think that probability and expected value-based thinking were overvalued; in practice they seem far undervalued. Kay and King tell the story of Obama’s advisors telling him numerically what they think the chances are that Osama bin Laden is in the house — a scene Phil Tetlock describes in his book as a model case of probabilistic reasoning. Kay and King think this is useless and actively damaging: The analysts are using numbers to hide that they just don’t know. I think Tetlock has it right here, and that summarizes how I felt about most of the book.
• That said, Kay and King’s basic point that sometimes it’s pointless to put a probability on something and we should just admit “I have no idea” — that seems right. What will US GDP be in the year 5,000? I’m not sure it’s helpful to try and put numbers and confidence intervals to that sort of question.
• They also stick up for the art of reasoning to the best explanation (abductive reasoning), and they frequently come back to the question, borrowed from a business professor: “What is going on here?” Again, overall I’m mostly skeptical. The evidence seems to suggest this is an overvalued starting point — we’re more likely to zoom too far in than too far out, which is why it’s often wise to step back, look for data and take the “outside view.” But it’s also possible to go too far in that direction and pay too little attention to what’s unique about a single case (I’ve done it plenty). And they’re right that explaining individual cases requires judgment. Sometimes broader data is nonexistent; sometimes conditions are such that broader comparison sets aren’t useful; sometimes diving into the details is what’s required to truly understand a topic. “What is going on here?” is a good animating question.
• Their dismissal of behavioral economics was unpersuasive to me, but the discussion of narratives in decision making was intriguing. They argue that people craft “reference narratives” about how they hope or expect their lives to go, and then they make decisions so as to bring reality as closely into line with the narrative as possible. I was left wanting more on this subject.

Quotes

Plato sought and found truth in logic; for him there ws a sharp distinction between truth, which was axiomatic, and probability, which was merely the opinion of man. In premodernt hought there was no such thing as randomness, since the course of events reflected the will of the gods, which was determinate if not fully known. The means of resolving uncertainty was not to be found in mathematics, but in a better appreciation of the will of the gods.

p. 54

At the end of the nineteenth century, Charles Sanders Peirce, a founder of the American school of pragmatist philosophy, distinguished three broad styles of reasoning. Deductive reasoning reaches logical conclusions from stated premises… Inductive reasoning … seeks to generalise from observations, and may be supported or refuted by subsequent experience… Abductive reasoning seeks to provide the best explanation of a unique event… Deductive, inductive, and abductive reasoning each have a role to play in understanding the world, and as we move to larger worlds the role of the inductive and abductive increases relative to the deductive. And when events are essentially one-of-a-kind, which is often the case in the world of radical uncertainty, abductive reasoning is indispensable.

p. 137-138

Kahneman offers an explanation of why earlier and inadequate theories of choice persisted for so long — a ‘theory-induced blindness: once you have accepted a theory and used it as a tool in your thinking, it is extraordinarily difficult to notice its flaws’. We might say the same about behavioural economics. We believe that it is time to move beyond judgmental taxonomies of ‘biases’ derived from a benchmark which is a normative model of human behaviour deduced from implausible a priori principles. And ask instead how humans do behave in large worlds of which they can only ever have imperfect knowledge.

p. 147-148

In many colleges, students of law are taught to follow a structure described as IRAC: issue, rule, analysis, conclusion. The impressive skill of a top lawyer is to identify the issue; to give structure to an array of amorphous facts, freqnetly presented in a tendentious manner — that is to establish ‘what is going on here.’ … IRAC is a useful acronym for anyone engaged in the search for practical knowledge. In the legal context it leads naturally to thetow next stages of effective practical reasoning — communication of narrative and challenge to the prevailing narrative.

p. 194-195

The legal style of reasoning, essentially abductive, involves a search for the ‘best explanation’ — a persuasive narrative account of events relevant to the case. The great jurist and US Supreme Court Justice Oliver Wendell Holmes Jr. began his exposition of legal philosphy with the observation that ‘The life of the law has not been logic; it has been experience… The law embodies the story of a nation’s development through many centuries, and it cannot be dealt with as if it contained only the axioms and corollaries of a book of mathematics.’

p. 211

A ‘good’ explanation meets the twin criteria of credibility and coherence. It is consistent with (most of) the available evidence and the general knowledge available to judges and jurors… A good explanation demonstrates internal coherence such that, taken as a whole, the account of events makes sense. The best explanation can be distinguished from other explanations and is not compatible with these other explanations. Statistical reasoning has its place but only when integrated into an overall narrative or best explanation.

p. 212

In pressing the case for probabilistic reasoning, Philip Tetlock and Daniel Gardner, the appraisers of forecasting and architects of the ‘good judgment project’, argue that ‘For decades, the United States had a policy of maintaining the capacity to fight two wars simultaneously. But why not three? or four? Why not prepare for an alien invation while we are at it? The answer hinges on probabilities.’ No it doesn’t. There is no basis on which one can form probabilities of an invasion by aliens… The attempt to construct probabilities is a distraction from the more useful task of trying to produce a robust and resilient defence capability to deal with many contingincies, few of which can be described in any but the sketchiest of detail.

p. 294-295

The mark of science is not insistence on deductive reasoning but insistence that observation trumps theory, whatever the purported authority supporting the theory.

p. 389

Acknowledging radical uncertainty does not mean that anything goes. Look to the future and contemplate the ways in which information technology will be deployed in the coming decades, or consider the ways in which the growth of prosperity and political influence in Asia will affect the geopolitical balanc.e They are all things about which we can know something, but not enough; we see though a glass, darkly. We can construct narratives and scenarios to describe the ways in which technology and global politics might develop in the next twenty years; but there is no sensible way in which we can refine such dialogue by attaching probabilities to a comprehensive list of contingiences. We might, however, tlalk coherently about the confidence we place in scenarois and the likelihood that they will arise. As we have ephasised, the words ‘confidence’, ‘likelihood’ and ‘probablitly’ are often used interchangeably but they have different meanings. We do not enhance our understanding of the future by inventing facts and figures to fill in the inescapable gaps in our knowledge. We cannot rely on forecasts in planning for the future…. We are not afraid to answer these questions with ‘we do not know.’

p. 403

I recently left Quartz after ~2.5 rewarding years as an editor there. The most gratifying editorial aspect of that work was editing hundreds of interesting features from nearly every reporter on staff. There are too many of those pieces to try and select favorites. But I wrote a bit, too. And I wanted to link to a few of my favorite pieces I wrote during my time there:

• I wrote a feature on Phil Tetlock’s research into counterfactual forecasting, including the use of the video game Civilization V as an experimental setting
• I collaborated on a tool to calibrate your confidence (credit to David Yanofsky for building it; I wrote the piece)
• I wrote about Facebook’s experimental app “Forecast”
• I laid out the best and worst cases for the VC market in 2022 and earlier wrote about when and whether the tech boom would ever end
• I wrote about the changing geography of startups (and here)
• I interviewed an economist about the long-run future of remote work
• I used the war in Ukraine as an example of two different ways analysts think about conflict
• I interviewed an ethicist about how companies should respond to the invasion of Ukraine
• I wrote an explainer / “Obsession” email on the concept of Bureaucracy
• I wrote advice for learning a new skill online
• I wrote about antitrust, especially merger rules that need to change and how Google’s antitrust defense compares to Microsoft’s

I’ll always be grateful for my tenure there, working with some truly excellent people.

…Among my principal guides are Ralph Waldo Emerson and John Dewey. If there’s an implicit allegiance here, a school of thought in which I might claim membership, it’s some version of pragmatism. That is, I believe that our understanding of art emerges from our experience of it, and that our notions of beauty and value are the result of our arguments about them, rather than the conditions of such arguments. Truth is the lovely echo of our noisy, contending ways of being wrong.

And one big way to be wrong–perhaps one I should have addressed at more length–is to buy into the fantasy of critical authority, to confuse the decidedly democratic power of persuasion with other kinds of power.

“Better Living Through Criticism: How to Think About Art, Pleasure, Beauty, and Truth,” A.O. Scott, pg. 275-276

“When I speak of myself as a journalist,” he wrote, “I do not of course mean that I have always dealt with current events or that I have not put into my books something more than can be found in my articles; I mean that I have made my living mainly by writing in periodicals. There is a serious profession of journalism, and it involves its own special problems. To write what you are interested in writing and to succeed in getting editors to pay for it, is a feat that may require pretty close calculation and a good deal of ingenuity. You have to learn to load solid matter into notices of ephemeral happenings; you have to develop a resourcefulness at pursuing a line of thought through pieces on miscellaneous and more or less fortuitous subjects; and you have to acquire a technique of slipping over on the routine of editors the deeper independent work which their over-anxious intentness on the fashions of the month or the week have conditioned them automatically to reject, as the machines that make motor parts automatically reject outsizes.”

“Better Living Through Criticism: How to Think About Art, Pleasure, Beauty, and Truth,” by A.O. Scott, pg. 228

There’s a lot of buzz and debate about ESG and corporate social responsibility lately, so I wanted to post something I worked on recently on this topic. I helped Quartz put together a page describing its mission to “Make Business Better,” and most of that reflects group work bridging a range of perspectives within the organization.

I’m pleased to take credit for one bit, though, where I tried to articulate my own thinking on the topic:

Econ 201

Milton Friedman’s theory of shareholder capitalism kept things simple: Companies should not break the law, and other than that they ought to serve shareholders. By contrast, “stakeholder capitalism” or “corporate social responsibility” can seem hazy, unfocused, and not rigorous. 

So here’s a short checklist for what a better form of capitalism would require of businesses, in plain English and in economic jargon:

Solve a real problem for someone (in econ terms, create “surplus”)

Share the rewards fairly (customers, workers, suppliers share in the surplus—and not just those who are members of privileged groups)

Don’t pollute (minimize negative externalities, or at least pay for them)

Follow the rules (follow the law in spirit, not just in letter)

Be a good corporate citizen (compete fairly, and don’t undermine the institutions on which you depend—including government and civil society) 

That’s it. A little more complicated, sure, and a little harder to write down as a mathematical model. But we believe it’s what’s needed in a world beset by inequality, climate change, and (one more econ term) countless market failures. We can’t just leave these problems to the market, and we can’t just leave them to governments or philanthropists either.

This draws on stuff I started to sketch out in my post “What are organizations for?” and this interview I did with an ethicist also in my opinion adds a lot of good context around these questions.

A recent David Autor review paper sums up the evolution:

I began by asking what the role of technology—digital or otherwise—is in determining wages and shaping wage inequality. I presented four answers corresponding to four strands of thinking on this topic: the education race, the task-polarization model, the automation-reinstatement race, and the era of AI uncertainty. The nuance of economic understanding has improved substantially across these epochs. Yet, traditional economic optimism about the beneficent effects of technology for productivity and welfare has eroded as understanding has advanced. Fundamentally, technological change expands the frontier of human possibilities, but one should expect it to create many winners and many losers, and to pose vast societal challenges and opportunities along the way.

What are the policy implications of these observations? The question is so broad that almost any answer is bound to appear vague and inadequate. One can reliably predict that technological innovations will foster new ways of accomplishing existing work, new business models, and entirely new industries, and these in turn will generate new jobs and spur some productivity gains. But absent complementary institutional investments, technological innovation alone will not generate broadly shared gains. Autor et al. (2022) sketch a long-form policy vision of what form these investments may take, focusing on three domains: education and training; labor market institutions; and innovation policy itself.

This is a good tweet:

Lessons of last 17 years:
1. Smart generalists (superforecasters) out-perform smart specialists in forecasting tournaments
2. So skepticism is justified but populists take it too far. Expertise grounded in reliable causal knowledge & applied to well-defined problems works wonders

— Philip E. Tetlock (@PTetlock) May 27, 2022

The value of deferring to experts depends on the alternative. If the alternative is deferring to a market or the consensus of smart generalists with good incentives or to a carefully calibrated statistical model, then deference to experts might not look so good–or at least is likely incomplete.

But a lot of the time the alternative is leaning on your own biases or those of your group, or deferring to pundits or to prevailing views shaped by attention algorithms that no one fully understands. In those cases, deference to experts looks pretty good!

Ultimately, the value of deferring to experts is in tying yourself to the mast, in epistemological terms. You defer to avoid your own biases. But as always, getting it right is about choosing wisely whom to trust.

In a great post defending economist Lisa Cook’s appointment to the Fed Board of Governors, Nobel-winner and famed theorist Paul Romer gets into the role of theory vs. empirics in social science:

There is a role for the type of theory that John [Cochrane, another theorist to whom he is responding…] and I do. Theorists build tools. Some of these tools turn out to be useful because they fit the facts. Many do not. Little harm comes from positing an imaginative new theory that turns out to be wrong provided that empiricists check it against the facts before someone uses it to make an important decision. John is a glider pilot so he understands the importance of this intellectual division of labor. When a passenger plane runs out of fuel – and yes, this can happen – neither John nor I would rely on a theorist skilled in computational aerodynamics to choose between landing at a nearby airport that has a short runway or a more distant one with a longer runway. We’d both want to give the judgment call about where to land to someone who knows the facts about the landing speed and glide ratio of the plane.

From The Will to Believe in 1896:

Objective evidence and certitude are doubtless very fine ideals to play with, but where on this moonlit and dream-visited planet are they found? I am, therefore, myself a complete empiricist so far as my theory of human knowledge goes. I live, to be sure, by the practical faith that we must go on experiencing and thinking over our experience, for only thus can our opinions grow more true; but to hold any one of them — I absolutely do not care which — as if it never could be reinterpretable or corrigible, I believe to be tremendously mistaken attitude, and I think that the whole history of philosophy will bear me out…

…But please observe, now, that when as empiricists we give up the doctrine of objective certitude, we do not thereby give up the quest or hope of truth itself. We still pin our faith on its existence and still believe that we gain an ever better position towards it by systematically continuing to roll up experiences and think. Our great difference from the scholastic lies in the way we face. The strength of his system lies in the principles, he origin, the terminus a quo of his thought; for us the strength is in the outcome, the upshot, the terminus ad quem. Not where it comes from but what it leads to is to decide. It matters not to an empiricist from what quarter an hypothesis may come to him: he may have acquired it by means fair or foul; passion may have whispered or accident suggested it; but if the total drift of thinking continues to confirm it, that is what he means by its being true.

Pragmatism: A Reader, p. 79-81

For the past few weeks, The Ezra Klein Show has been doing episodes about Russia and Ukraine from a variety of perspectives. In the most recent one, Ezra described his approach:

I want to begin today by taking a moment and getting at the theory of how we’re covering Russia’s invasion of Ukraine on the show. There is no way to fully understand an event this vast, where the motivations of the players and the reality on the ground are this unknowable. There’s no one explanation, no one interpretation that can possibly be correct. And if anyone tells you they’ve got that, you should be very skeptical. But even if all models are incomplete, some are useful. And so each episode has been about a different model, a different framework, you can use to understand part of the crisis.

I approve. And I tried my own attempt of a many-model explanation of the conflict in a piece for Quartz a couple weeks back. I tried to balance structural and game-theoretic explanations with historical and personality-driven ones, and to present the outside view as well as the inside one.

For the outside view, I relied on Chris Blattman’s excellent forthcoming book Why We Fight. Here’s the summary:

In Why We Fight, Blattman uses game theory to explain why war does and doesn’t happen. His starting point is that war is rare because it’s expensive. But five factors can overwhelm the incentives for peace:

💪 Unchecked interests. War is more likely when the people in charge don’t pay the price for it. That’s almost always true to an extent, but some leaders are more or less insulated from the costs of conflict.

🎲 Uncertainty. Neither side knows for sure how strong the other is. One side could be bluffing about its strength or resolve, so sometimes the other side calls.

🗝️ Commitment problems. When one side is growing stronger, the other may want to attack before its adversary gets too powerful. The growing power might promise not to attack later on when it’s the dominant power, but that commitment can’t be trusted.

🤔 Misperceptions. Decision makers are overconfident and don’t understand how their adversaries think.

🖤 Intangible incentives. Sometimes people care about things that can’t be bargained for and go beyond costs and benefits—like vengeance, glory, or freedom.

You can read the rest here.