Last year I had the opportunity to be a “Pro” forecaster on INFER, the crowd forecasting tournament run by Cultivate Labs and the University of Maryland (formerly Foretell of Georgetown’s CSET). Basically you get a small stipend to participate each month. It was fun and I recommend it!
Ultimately, I decided not to keep going as a Pro in 2023. I started Nonrival in August, which I’ll blog about one of these days, and since then my forecasting time has been focused on that project.
I still plan to collaborate with the INFER team in some other capacities (more on that soon too perhaps) but I won’t be paid to make forecasts there.
I’ve “exited” the three questions that I’d forecasted on and weren’t yet resolved, but I’ll still be scored eventually for the period of time I was active on them — so this assessment isn’t quite a true scoring of my time there, but it’s close. How’d I do?
- In the 2022 season, 344 users made at least 5 forecasts (that resolved), and by default that’s the cutoff INFER uses on its leaderboard so I’ll use it here, too.
- I finished 76th, with 8 questions resolved and scored. That puts me at the 78th percentile.
- On the “all-time” leaderboard for INFER (which for me counts my two questions forecast in 2021) I’m 71st of 620, which puts me at the 89th percentile.
- Lifetime on INFER, I’m better-than-median on 9 out of 10 questions (7 out of 8 for 2022 season), with my one blunder being a forecast of Intel’s earnings where I seemingly underrated the chance of an out-of-sample result.
Overall, my MO seems to be consistently being just a tiny bit better than the crowd. Not bad! But that leaves plenty of room for improvement. Some of it is I think I could do better by simply spending more time and updating more regularly on news and shifts by other forecasters.
But there’s also a “tenaciousness” that Tetlock describes when talking about the superforecasters that includes a willingness or even an eagerness to sift through details when necessary until you find what you need. I saw some of that with teammates during my year as a Pro. And that’s something I’ve not had the time or maybe the inclination for. I think I’ve done a pretty consistent job of avoiding the basic mistakes that lead to poor forecasts: I look for quantitative data, seek out multiple perspectives, I often blend my own judgment with others’, etc. But if I want to get to the next level I need to immerse myself more in the details of a topic, at least some of the time.
beyond the times 