Let’s get one thing straight: I am not a “superforecaster.”
Over the past decade, I’ve written about forecasting research and forecasting platforms. And I’ve participated in them as well. In this post I’ll share some of my results to date. Though I’m nowhere near superforecaster level (the top 2% of participants) I’m pleased to have been consistently above average.
Here are my results:
- Good Judgment Project (~2017): 23 questions, 68th percentile
- Good Judgment Open (2015-2017): 9 questions, 60th percentile*
- Good Judgment Open (2021): 4 questions, 76th percentile*
- Foretell/Infer (2021): 2 questions, 90th percentile
- Update Aug. 2022: On INFER first half of 2022, 7 questions, 76th percentile; lifetime INFER ranking 87th percentile.
The number of questions is not the number of forecasts: in many cases I made several forecasts over time on the same question. I’ve given percentiles rather than relative Brier scores or other measures because a) they’re more intuitive and b) the GJ Project setup I did was a market (no real money), and so the results were given in terms of total (fake) dollars made and where that scored by percentile. The latter is more comparable to the other scoring systems.
(*) GJP and Infer report percentile scores across an entire season so I used those above. GJ Open doesn’t, best I can tell, so in these cases I’ve averaged my percentile scores for each question, which is a bit different than percentile in total score.
Here’s another view, this one excluding my Good Judgment Project results because I don’t have percentile scores for each question.
For Good Judgment Project, not included in the chart, I “made money” (again: no actual money involved) on 17 of 23 questions, lost money on 4 and was basically even on 2.
Some of my worst scores across all of this involved the 2016 election (including primaries). One of my best involved venture capital. My impression is that, although subject matter knowledge is nice to have, time spent is the major limiting factor. Spending more time and updating forecasts more regularly pays off, even in areas where I’m coming in fairly fresh.
To close out, here is some of my writing about forecasting:
- What a study of video games can tell us about being better decision makers (Quartz; on counterfactual forecasting research)
- Use this tool to overcome the biggest decision-making mistake (Quartz; a brief confidence calibration tool)
- Facebook is shutting down its experimental app Forecast (Quartz; news piece)
- What Research Tells Us About Making Accurate Predictions (HBR; research summary)
- Question Certainty (HBR; review of books on forecasting and decision making)