## Metadata - Author: [[Philip E. Tetlock, Dan Gardner]] - Full Title: Superforecasting - Category: #books - [[∙Melvil Decimal System (DDC)]] - [[3 - Social sciences]] - [[30 - Social Sciences; Sociology and anthropology]] - [[303 - Social Processes]] - [[303.4 - Social change]] - [[303.49 - Social forecasts]] ## To look into The book was a winner in the “CMI Management Book of the Year”. See what other books have won. *** ## Notes Using percentages when making forecasts is important. It forces the forecaster to think harder and come up with an actual number. Language is easier. “Slight chance”, instead of “30% chance”. Also words are more malleable after the forecasted event: > If the event happens, “a fair chance” can retroactively be stretched to mean something considerably bigger than 50% – so the forecaster nailed it. If it doesn’t happen, it can be shrunk to something much smaller than 50% – and again, the forecast nailed it. With perverse incentives like these, it’s no wonder people prefer rubbery words over firm numbers. P58 *** Weather forecasting is lends itself to quickly working out if the forecaster is accurate. There’s new weather everyday. Most forecasts are rarer. Take the U.S. Presidential Elections. They’ve been going on for over 200 years, but there’s been less than 60 of them. If you’re forecasting them, that’s not a lot of data. And that’s a lot of gaps between new data. P61-62 *** ## Highlights *** > "I have been struck by how important measurement is to improving the human condition," Bill Gates wrote. "You can achieve incredible progress if you set a clear goal and find a measure that will drive progress toward that goal.... This may seem basic, but it is amazing how often it is not done and how hard it is to get right." He is right about what it takes to drive progress, and it is surprising how rarely it's done in forecasting. Even that simple first step-setting a clear goal-hasn't been taken. (Page 15) *** > hen physicians finally learned to doubt themselves, they turned W to randomized controlled trials to scientifically test which treatments work. Bringing the rigor of measurement to forecasting might seem easier to do: collect forecasts, judge their accuracy, add the numbers. That's it. In no time, we'll know how good Tom Friedman really is. > > But it's not nearly so simple. Consider a forecast Steve Ballmer made in 2007, when he was CEO of Microsoft: "There's no chance that the iPhone is going to get any significant market share. No chance." > > Ballmer's forecast is infamous. Google "Ballmer" and "worst tech predictions" or "Bing" it, as Ballmer would prefer-and you will see it enshrined in the forecasting hall of shame, along with such classics as the president of Digital Equipment Corporation declaring in 1977 that "there is no reason anyone would want a computer in their home." And that seems fitting because Ballmer's forecast looks spectacularly wrong. As the author of "The Ten Worst Tech Predictions of All Time" noted in 2013, "the iPhone commands 42% of US smartphone market share and 13.1% worldwide." That's pretty "significant." As another journalist wrote, when Ballmer announced his departure from Microsoft in 2013, "The iPhone alone now generates more revenue than all of Microsoft."2 But parse Ballmer's forecast carefully. The key term is "significant market share." What qualifies as "significant"? Ballmer didn't say. And which market was he talking about? North America? The world? And the market for what? Smartphones or mobile phones in general? All these unanswered questions add up to a big problem. The first step in learning what works in forecasting, and what doesn't, is to judge forecasts, and to do that we can't make assumptions about what the forecast means. We have to know. There can't be any ambiguity about whether a forecast is accurate or not and Ballmer's forecast is ambiguous. Sure, it looks wrong. It feels wrong. > > There is a strong case to be made that it is wrong. But is it wrong beyond all reasonable doubt? > > I don't blame the reader for thinking this is too lawyerly, a tad too reminiscent of Bill Clinton's infamous "it depends on what the meaning of the word 'is' is." After all, Ballmer's meaning seems plain, even if a literal reading of his words doesn't support that. But consider his full statement, in context, in an April 2007 interview with USA Today: "There's no chance that the iPhone is going to get any significant market share. No chance. It's a $500 subsidized item. They may make a lot of money. But if you actually look at the 1.3 billion phones that get sold, I'd prefer to have our software in 60% or 70% or 80% of them, than I would to have 2% or 3%, which is what Apple might get." That clarifies some things. For one, Ballmer was clearly referring to the global mobile phone market, so it's wrong to measure his forecast against US or global smartphone market share. Using data from the Gartner IT consulting group, I calculated that the iPhone's share of global mobile phone sales in the third quarter of 2013 was roughly 6%. That's higher than the "2% or 3%" Ballmer predicted, but unlike the truncated version so often quoted, it's not laugh-out-loud wrong. Note also that Ballmer didn't say the iPhone would be a bust for Apple. Indeed, he said, "They may make a lot of money." But still there is ambiguity: how much more than 2% or 3% of the global mobile phone market would the iPhone have to capture to be deemed "significant"? Ballmer didn't say. And how much money was he talking about when he said Apple could earn "a lot of money"? Again, he didn't say. > > So how wrong was Steve Ballmer's forecast? His tone was brash and dismissive. In the USA Today interview, he seems to scoff at Apple. But his words were more nuanced than his tone, and too ambiguous for us to declare with certainty that his forecast was wrong-much less so spectacularly wrong it belongs in the forecasting hall of shame. (Page 46) > All this brings us to the final feature of winning teams: the fostering of a culture of sharing. My Wharton colleague Adam Grant categorizes people as "givers," "matchers," and "takers." Givers are those who contribute more to others than they receive in return; matchers give as much as they get; takers give less than they take. > > Cynics might say that giver is a polite word for chump. After all, anyone inclined to freeload will happily take what they give and return nothing, leaving the giver worse off than if he weren't so generous. > > But Grant's research shows that the pro-social example of the giver can improve the behavior of others, which helps everyone, including the giver-which explains why Grant has found that givers tend to come out on top (Page 208) *** > One involves the provocative phrase "diversity trumps ability," coined by my colleague (and former competitor in the IARPA tournament) Scott Page." As we have seen, the aggregation of different perspectives is a potent way to improve judgment, but the key word is different. Combining uniform perspectives only produces more of the same, while slight variation will produce slight improvement. It is the diversity of the perspectives that makes the magic work. Superteams were fairly diverse-because superforecasters are fairly diverse-but we didn't design them with that in mind. We put ability first. (Page 209) *** > "In war, everything is uncertain," wrote Helmuth von Moltke.¹ In the late nineteenth century, Moltke was famous the world over after he led Prussian forces to victory against Denmark in 1864, Austria in 1866, and France in 1871-victories that culminated in the unification of Germany. His writings on war-which were themselves influenced by the great theorist Carl von Clausewitz-profoundly shaped the German military that fought the two world wars. But Moltke was no Napoleon. He never saw himself directing his army organization was entirely different. > > visionary leader as a like chess pieces. His approach to leadership and The Prussian military had long appreciated uncertainty-they had invented board games with dice to introduce the element of chance missing from games like chess-but "everything is uncertain" was for Moltke an axiom whose implications needed to be teased out. The most urgent is to never entirely trust your plan. "No plan of operations extends with certainty beyond the first encounter with the enemy's main strength," he wrote. That statement was refined and repeated over the decades and today soldiers know it as "no plan survives contact with the enemy." That's much snappier. > > But notice that Moltke's original was more nuanced, which is typical of his thinking. "It is impossible to lay down binding rules" that apply in all circumstances, he wrote. In war, "two cases never will be exactly the same." Improvisation is essential.² Moltke trusted that his officers were up to the task. In addition to their military training, they received what we today would consider a liberal arts education with an emphasis on critical thinking. Even when the curriculum focused on purely military matters, students were expected to think hard. In other nations of that era-including the United States-instructors laid out problems, told students the right answer, and expected them to nod and memorize. In Germany's war academies, scenarios were laid out and students were invited to suggest solutions and discuss them collectively. Disagreement was not only permitted, it was expected, and even the instructor's views could be challenged because he "understood himself to be a comrade among others," noted the historian Jörg Muth. Even the views of generals were subject to scrutiny. "German junior officers were regularly asked for their opinions and they would criticize the outcome of a large maneuver with several divisions before the attending general The Leader's Dilemma 215 The acceptance of criticism went beyond the classroom, and under extraordinary circumstances more than criticism was tolerated. In 1758, when Prussia's King Frederick the Great battled Russian forces at Zorndorf, he sent a messenger to the youngest Prussian general, Friedrich Wilhelm von Seydlitz, who commanded a cavalry unit. "Attack," the messenger said. Seydlitz refused. He felt the time wasn't right and his forces would be wasted. The messenger left but later returned. Again he told Seydlitz the king wanted him to attack. > > Again Seydlitz refused. A third time the messenger returned and he warned Seydlitz that if he didn't attack immediately, the king would have his head. "Tell the King that after the battle my head is at his disposal," Seydlitz responded, "but meanwhile I will make use of it." > > Finally, when Seydlitz judged the time right, he attacked and turned the battle in Prussia's favor. Frederick the Great congratulated his general and let him keep his head. This story, and others like it, notes Muth, "were collective cultural knowledge within the Prussian ofcorps, recounted and retold countless times in an abundance of variations during official lectures, in the officer's mess, or in the correspondence between comrades." The fundamental message: think. > > If necessary, discuss your orders. Even criticize them. And if you absolutely must-and you better have a good reason-disobey them.* All this may sound like a recipe for a fractious organization that get anything done, but that danger was avoided by balancing those elements that promoted independent thinking against those ficer can't that demanded action. (Page 213) *** > How the Kennedy White House changed its decision-making culture for the better is a must-read for students of management and public policy because it captures the dual-edged nature of working in groups. Teams can cause terrible mistakes. They can also sharpen judgment and accomplish together what cannot be done alone. Managers tend to focus on the negative or the positive but they need to see both. As mentioned earlier, the term "wisdom of crowds" comes from James Surowiecki's 2004 bestseller of the same name, but Surowiecki's title was itself a play on the title of a classic 1841 book, Extraordinary Popular Delusions and the Madness of Crowds, which chronicled a litany of collective folly. Groups can be wise, or mad, or both. What makes the difference isn't just who is in the group, nedy's circle of advisers demonstrated. The group is its own animal.