Matt Ridley: The wisdom of crowds

My Times column on the wisdom of crowds, published the day before election day in the US:

‘In these democratic days, any investigation into the trustworthiness and peculiarities of popular judgments is of interest.” So begins an article entitled Vox Populi, which is not about Donald Trump but was published in 1907 by Francis Galton, a pioneer of statistics, by then 85 years old. He had analysed the results of a sweepstake competition held at the West of England Fat Stock and Poultry Exhibition in Plymouth.

An ox was on display. Visitors could buy a postcard for sixpence and write their guess as to the weight of the ox, once slaughtered and dressed. Of 800 cards filled out, Galton rejected 13 as illegible and averaged the rest. The arithmetic mean of the 787 guesses came to 1,197lb. The true dressed weight of the ox was — yes — 1,197lb (Galton reported slightly different results, but recent reanalysis by Kenneth Wallis of Warwick University finds the match was exact).

The message is that a crowd is at least as wise as any expert (only one guess was spot on). In a large group of people, ignorance in one direction cancels out ignorance in another. Thus, to take another example, roughly ten million people will eat lunch in London today. Working out exactly what they will decide to eat, where, and in what quantities, is a vastly complicated exercise. Fortunately, we do not entrust the problem to a very well-paid and highly qualified London Lunch Commissioner, but to something called the market, which uses millions of signals of supply and demand to “crowd-source” the answer. And it works remarkably well every single day.

With Donald Trump possibly on the brink of election to the most powerful job in the world, many people are tempted to lose faith in the wisdom of crowds. It is common now to hear the argument that democracy is giving voice mainly to the ignorant and must therefore be somehow curtailed, with power handed back to the knowledgeable.

“It’s time for the elites to rise up against the ignorant masses,” read the headline of an article by James Traub in Foreign Policy in June this year, referring to both Trump’s nomination and the Brexit referendum result. I think he is wrong. The crowd still has a wisdom that no individual can match, and the results of modern elections do not contradict this. We need more, not less, crowd wisdom.

A new book called Against Democracy by an American political philosopher, Jason Brennan, argues that democracy is “the rule of the ignorant and the irrational” and that “political participation and democratic deliberation actually tend to make people worse — more irrational, biased, and mean”. (You can see why he published the book this year.) In its place he recommends “epistocracy”, where you should have to earn the right to vote by showing a modicum of knowledge.

Professor Brennan’s Plato-like proposal is a bit vague as to detail. We might have to sit an exam to get a vote. Those with degrees might get extra votes (as Oxbridge once did). A council of epistocrats might veto certain applicants. If you think these ideas ludicrous and dangerous, it might be worth reminding you that this is how a great many decisions are indeed taken: a committee of experts is set up; a quango is staffed; a civil servant allocates a budget.

To a large extent we do live in such an epistocracy. Fortunately, we cling to the idea that every few years, and on certain constitutional questions, the populi — all of them — can have a vox. And rightly so, because the ideal future government of a country is too complicated a question for any expert, even if, like Mr Brennan, he is the associate professor of strategy, economics, ethics and public policy at the McDonough School of Business, at Georgetown University. Besides, there is no such thing as general ignorance or general expertise. Every brilliant person I know is also astonishingly ignorant on certain matters.

Paul Flynn, a Labour MP, says the result of the European referendum was illegitimate because most of the people who voted were ignorant. A slightly more sophisticated argument against plebiscites was common among the intelligentsia in the wake of the Brexit vote. Richard Dawkins, among others, argued that he should not have had to vote on a matter he did not understand. He says that is what parliament is for: to hand such decisions to experts, who understand the details.

But members of parliament are not experts, let alone omniscient ones. Whether Britain is right or wrong to leave the European Union is a question that nobody, however clever, can possibly know the right answer to. That is precisely why, like the weight of Galton’s ox, it is a question that should be decided by averaging popular opinion, and 34 million guesses are better than 650. The ignorances, biases, prejudices and hunches of everybody should be thrown into a giant blender. During the campaign some were persuaded that the ox was heavier, some that it was lighter than it was.

So what about Donald Trump? If tomorrow he becomes president-elect, am I arguing that this is a good decision because the crowd wills it? No, and here is why. The American people are not being asked, en masse, “who is the best person you can think of to occupy the White House?” They are being asked: “From among a very small group of very rich, very famous people, winnowed down for you to two, and that winnowing done by the less than 10 per cent of the population extreme and unrepresentative enough to belong to and turn out to vote in the party primary elections, which is the least awful?” That is very different from crowd-sourcing . . . I think!

A minority has foisted its choice on the majority. Likewise, Jeremy Corbyn is leader of the Labour Party, not because a large crowd has decided in its wisdom that he is the best leader of the opposition, but because a small and unrepresentative crowd, with the megaphone of social media at its disposal, has so decided.

If presidential elections were more, not less, like ox-weight guessing competitions, they might produce better results. The paradox of today is that technology, which ought surely to make testing the wisdom of crowds by giving everybody equal weight easier, may instead be giving undue weight to a minority of extreme voices.

Matt Ridley, a member of the British House of Lords, is an acclaimed author who blogs at www.rationaloptimist.com.