"Doxa often varies, a fool is that who relies on it" - frustrated, at Château de Chambord. If he were still among us, I think that François 1er would have expressed frustration with the conventional wisdom on polls jumping so easily from one extreme to the other. Except he would have posted it to his Facebook instead of engraving it on his bedroom walls.
And how could we disagree? The lack of understanding displayed by a vast majority of the public and the French media when it comes to interpreting polls is fascinating. On the eve of the April 2017 presidential elections, a Pavlovian reflex of poll relativization reigned. Polls were supposedly useless after the debacles of Brexit and the American presidential election.
Remember though that British polls indicated an extremely tight referendum, around 50-50 and that American polls gave Clinton a +4 point advantage - an error of 2 points since she won by 2 points . In both cases, the blame lies more with our inability to consider a large number of scenarios in an uncertain universe than with polls themselves.
Then came April 23, 2017, the first round of the French presidential elections. Pollsters performed particularly well and our questioning of them largely disappeared. Again, a mistake, since 2 weeks later they had one of their worst performances.
The problem is that this underperformance was only slightly noticed, contrary to the outperformance of the first round. As a result, during European elections in May 2019, the public and the media expectations may still be disconnected from the actual performance of pollsters.
To properly analyze these performances, we have studied nearly 800 first round polls , spanning 15 elections since 1974. Here is an overview of our results.
Mean absolute error of all polling institutes, by type of election
|Type of election||Far left||Left||Greens||Center||Right||Far right||All parties|
The absolute error measures the distance between the weighted average of the pollsters and the election results. PollsPosition calculations on nearly 800 polls in 15 elections. See our method for more details.
There is a lot of information in this table, but let's start with the last line - the absolute error of all pollsters (what we will refer to as "the market") for all elections. The absolute error measures the distance between the pollsters mean and the election results: if polls indicate the left at 25% and it lands at 23%, it is a 2-point error (the same as if the end result is 27%).
First lesson: French pollsters are quite accurate. Taking samples from approximately 1000 people and getting a maximum average error of 2 points over all elections for a population of 48 million voters, is quite remarkable.
However, this is an average, which means that the margin of error (the famous 95% confidence interval) should be around 5 points. And if you compare two candidates, you have to double that margin of error , which is a 10-point margin.
This means that you should not be surprised when a candidate who was leading by 6 points in the polls ended up losing by 1, 2 or even 4 points. This is more than the traditional margins of error indicated by pollsters, which only take into account sampling errors and only relate to each candidate's share, not to their difference.
The error also varies by type of election, albeit weakly. In broad terms, European and legislative polls are more accurate than regional polls, which are more accurate than presidential polls. But the situation is different depending on the party. Polls of the left are more accurate during presidential and legislative elections than in European and regional ones – same thing for the Green party. The situation is reversed for the far right.
Remember this when it comes to European elections: if some polls will be very accurate, others will differ from the result by 2, 3, or even 7 points. And that will be perfectly normal.
French pollsters' accuracy probably turned us into spoiled children who consider that polls are useless from the moment they are not within a point of the result. In this case, it is obviously our expectations that are unrealistic – historically, a one-point error is perfectly normal.
And spoiled kids with binary reasoning too, since the media focuses on the winner. While the error was greater in France than in the United States, the US presidential election led to an excessive relativization of polls, while the French one did not raise many eyebrows. It is probably because French pollsters had called the winner, while their US counterparts had put Clinton in the lead.
We strongly discourage this kind of reasoning: what matters is the distance between the poll and the result, not the fact that it called the right winner. And using this criterion, pollsters aren’t worse than before - but they aren’t better either. The graph below illustrates the stability of errors since 1974.
Evolution of the error of all pollsters for all parties
A methodological precision: what you are seeing here is the raw error, which allows to see the direction of the error. A positive error means that the market overvalued a given party, while a negative error indicates an undervaluation. We can see that the errors mostly happen between -3 and +3, with peaks between -5 and +5, but there is no noticeable downward or upward trend.
Our findings corroborate those of researchers Will Jennings and Christopher Wlezien, who show that the accuracy of polls in European, American and Asian democracies has been rather stable for several decades.
Since I know you are insatiably curious, I’ve enclosed this table as a bonus, which details the figures for the graph you just saw:
Raw error for all polling firms, by election
|Election||Far left||Left||Greens||Center||Right||Far right|
The raw error measures the distance between the weighted average of pollsters and the election results but also indicates the direction of that error. A positive error means that pollsters overvalued a given party, while a negative error indicates an undervaluation. PollsPosition calculations on nearly 800 polls in 15 elections. See our method for more details.
One point we often hear is that of the "shy voters", especially when it comes to the Front National (FN). Simply put, many voters would not dare to confess their FN preference to pollsters and would only reveal themselves on Election Day. If this were the case, we would observe a systematic undervaluation of the far right.
If you simply look at distribution of errors of all pollsters for every election, you notice that it is not the case. All parties combined, the average error is around 0, indicating that there is no bias in the market.
This is a good sign for the quality of French polls, but incites us to be cautious with our inferences: the error can go both ways and it is almost impossible to predict which way.
Distribution of raw errors for all pollsters, by party
However, we can see that the left and the far right stand out: their distribution is wider than that of the other parties, which means that the errors are more fragmented. Why do pollsters find it harder to estimate these two parties? Perhaps their electorate is more volatile? This seems credible for the left, currently in deep reconstruction, and subject to shifting coalitions with the Green party, the center or the far left, depending on the type of election.
It is harder to believe for the National Front, whose electorate is extremely loyal . Perhaps are the misfires - real or imagined – of Western pollsters for populist candidates pushing French institutes to overcompensate, creating new errors but in the opposite direction?
The graph below shows that the current trend is to overvalue the FN after a period of undervaluation. In our database, over the last 5 elections, the FN has been overvalued 4 times. And the two biggest overvaluations took place during the last presidential and legislative elections - respectively 2.3 and 4.7 points. A trend we should keep in mind for Europeans elections, especially as pollsters are much less criticized when they overvalue the far right than when they undervalue it. In any case, this overvaluation makes it possible to firmly set aside the hypothesis of "shy voters" 1 .
Evolution of the raw error of all pollsters for the National Front and best fit line
In a nutshell, polls are not as bad as what the post-Brexit bashing suggested, but they are also not as good as what the first round of the 2017 French presidential election hinted. The public and the media must learn how to interpret polls probabilistically , keeping in mind a wide range of scenarios, in order to distinguish between a very close election and a real polling failure.
1 : Not only were there no Marine Le Pen shy voters, but there were also none for the FN - otherwise one would observe an overvaluation only during presidential elections.