Matt Ridley: There are no experts on the future

Michael Gove was mocked during the referendum campaign for saying that “I think people in this country have had enough of experts.” Critics asked pointedly if he dismissed the expertise of doctors when ill. But subsequent weeks have left economic experts, at least, looking a bit less than the full Nostradamus.

The expert pollsters told the hedge funds Remain would win right up till when it lost, so the pound and the FTSE 100 rose, then crashed. The expert financial forecasters then told investors the FTSE 100 would fall further, but it quickly recovered all its lost ground and more. The expert analysts told us we should watch the FTSE 250 plunge instead, but that has now returned to the level it was at a week before the referendum.

Meanwhile, the IMF experts have abandoned their prediction of a recession, and all talk of a punishment budget from the Treasury experts has been forgotten. By contrast, Friday’s bad snapshot from the purchasing managers index was all the more credible for being a non-expert survey.

Yet of course it is still true that we turn to doctors, accountants, plumbers, motor mechanics and weather forecasters for their expertise, and rightly so. Why are some experts indispensable some of the time and others not? I suspect the answer is simple: there are no experts on the future. Explaining the present and the past requires expertise: “it’s your carburettor/prostate”. In forecasting the future, experts are generally no better than everybody else. They might be worse.

Even weather forecasters, who are very good these days, have a short time horizon. More than five days out and they struggle. The Met Office, badly burned by the failure of its long-range forecasts a few years ago, now adopts a more humble tone. Half way through May it said this about the coming summer: “the outlook suggests the chances of above – or below – average rainfall are approximately similar.” Which covers all possibilities, like an astrologer. Even then, it added a disclaimer: “this is not a normal weather forecast. It’s an experimental and complex outlook based on probabilities”. Whatever that means.

No wonder it is cautious. In 2007 the Met Office said “there are no indications of a particularly wet summer." Floods followed. In 2009 it forecast a “barbecue summer” with “rainfall average or below average” before one of the dullest and wettest Julys. In September 2009 it forecast that the winter “is likely to be milder than last year” before the coldest winter in 30 years. In October 2010, it forecast a 60-80 per cent chance of “warmer-than-average temperatures this winter” before the coldest December in 100 years. A blindfolded person throwing darts at a chart would have done better than this. A little expertise is clearly a dangerous thing. See http://news.bbc.co.uk/1/hi/8173533.stm and  http://www.express.co.uk/news/uk/130909/Winter-will-be-mild-say-Met-Office-goofers

Beginning in the 1980s, Philip Tetlock, now of the University of Pennsylvania, ran a tournament testing 28,000 specific predictions from 284 experts over 20 years. He found that on average expert forecasters were only slightly more reliable than chance, and that simple extrapolation was usually more accurate. The more famous the forecaster, the worse his or her performance. (He has since found there are a very few “superforecasters”, generally rather self-effacing types, who do more consistently get things right.) See https://www.amazon.co.uk/Expert-Political-Judgment-Good-Know/dp/0691128715 and https://www.amazon.co.uk/Superforecasting-Science-Prediction-Philip-Tetlock/dp/184794714X/ref=sr_1_1?ie=UTF8&qid=1469351293&sr=8-1&keywords=superforecasters

The reason for this lack of expertise about the future among experts is partly that their forecasts rely too heavily on pet arguments or assumptions. The Met Office’s computer had been programmed to expect faster global warming than was happening. Economists get certain bees in the bonnet. Famous people succumb to their own fame.

There is also the temptation to get media attention for a forecast by indulging in excessive pessimism. Paul Samuelson famously joked that stockmarkets forecast nine of the last five recessions. Doom has been selling newspapers for decades, in the form of impending war, famine, pollution, disease or economic collapse. Cassandras generally get more coverage than Pollyannas.

Here’s Robert Heilbroner, a famous economist, in 1973: “The outlook for man, I believe, is painful, difficult, perhaps desperate and the hope that can be held out for his future prospects seems to be very slim indeed.” Here’s a famous ecologist, Paul Ehrlich, speaking at the Institute of Biology in London in 1971: “If I were a gambler, I would take even money that England will not exist in the year 2000.”

Simple extrapolation is more reliable. If you had defied the gloomsters and said in 1960 that the world economy would grow by 2-6% a year every year with few exceptions for more than 50 years, you would have been laughed at, but you would have been right. There was only one year in that period when the world economy shrank – 2009 – and even that merely brought it back to its long-term trajectory. This is much steadier than any one country. I find that rather strange and don’t fully understand it.

After all, the economy shares an essentially unpredictable feature with the climate: they are both chaotic. That is to say, they have many small causes, which affect each other and result in complex feedback loops. Small perturbations in initial conditions can create big later changes: in the cliché, a butterfly’s wing flap can lead to a hurricane.

And unlike in climate forecasting, prediction of human trends is difficult because discovery and innovation keep throwing spanners in the works. “Science is the belief in the ignorance of experts,” said the physicist Richard Feynman, reflecting on the tendency of research to explode complacency and embarrass experts who tell you what’s impossible.

Experts are notoriously bad at seeing technological change coming. When they do, they often expect it in the wrong area. Fifty years ago, after dramatic changes in transport but not much change in communication, futurologists were all babbling about personal gyrocopters, regular supersonic flights and routine space travel – none of which have yet materialized. Very few of them saw mobile phones coming, let alone the internet, search engines or social media.

So trust experts, yes, but never about the future. The inventor, James Lovelock, aged 96, put it rather well in an interview with the Bournemouth Echo at the weekend: “I think anyone that tries to predict more than five to ten years ahead is a bit of an idiot, so many things can change unexpectedly.”

Matt Ridley, a member of the British House of Lords, is an acclaimed author who blogs at www.rationaloptimist.com.