General Election Opinion Poll Inquiry publishes report
The Inquiry set up to investigate inaccuracies in the opinion polls during the 2015 General Election campaign, Chaired by University of Southampton Professor, Patrick Sturgis, has released its full report.
Twelve recommendations are made to the polling industry in the Report of the Inquiry into the 2015 British General Election opinion polls. The Inquiry recommends a number of changes to British Polling Council (BPC) rules. These include requiring members to pre-register ‘vote intention’ polls before conducting fieldwork and providing statistical significance tests for changes in vote shares between polls. The report also makes recommendations for improving the representativeness of polling samples.
Pre-election polls in the 2015 election significantly under-estimated the size of the Conservative lead over Labour, with the final polls incorrectly suggesting a statistical ‘dead heat’. Preliminary findings of the Inquiry, released in January 2016, showed the primary cause of the failure of the polls was unrepresentativeness in the composition of the poll samples. The methods of sample recruitment used by the polling organisations resulted in systematic over-representation of Labour voters and under-representation of Conservative ones. Statistical adjustment procedures were not effective in mitigating these errors.
The recommendations made in this new, full report, include measures to improve the representativeness of quota samples and for better prediction of voter turn-out. The Inquiry identifies probability sampling as a more accurate way of collecting samples of voters, but acknowledges the practical and cost barriers for the industry of using this approach.
In addition, the report suggests an academic led pre and post-election random probability survey for the 2020 General Election, to be funded by the Economic and Social Research Council (ESRC).
Chaired by Patrick Sturgis, Professor of Research Methodology at the University of Southampton, the independent Polling Inquiry was set up last May by the British Polling Council (BPC) and the Market Research Society (MRS). It was tasked with establishing the extent of inaccuracies in the polls, the reasons for inaccuracies, and whether the polls were properly communicated to the general public.
Commenting on the report, Professor Patrick Sturgis said: “Opinion polls are subject to many different sources of error and no set of procedures will ever guarantee total accuracy. But how polls are conducted and reported in the UK can and should be improved. We hope the recommendations set out in this report will be taken on board by the polling industry in order to reduce the risk of these kinds of polling errors being repeated in future elections.”
Lord David Lipsey, co-chair of the All-Party Parliamentary Group on Statistics (APGS), said: "Parliamentarians in the APGS warmly welcome Professor Sturgis's authoritative report. Its scrupulous attention to the statistical facts supersedes the uninformed speculation that followed the opinion poll disaster of 2015. In view of the political impact of the polls, Parliament is sure to take a close interest in how the polling industry responds to Professor Sturgis's careful and cautious recommendations."
The Polling Inquiry report will now be published by the BPC and MRS who will set out their responses to the findings and recommendations.
Notes for editors
The 12 recommendations made in the Report of the Inquiry into the 2015 British General Election opinion polls are:
BPC members should:
1. include questions during the short campaign to determine whether respondents have already voted by post. Where respondents have already voted by post they should not be asked the likelihood to vote question.
2. review existing methods for determining turnout probabilities. Too much reliance is currently placed on self-report questions which require respondents to rate how likely they are to vote, with no strong rationale for allocating a turnout probability to the answer choices.
3. review current allocation methods for respondents who say they don’t know, or refuse to disclose which party they intend to vote for. Existing procedures are ad hoc and lack a coherent theoretical rationale. Model-based imputation procedures merit consideration as an alternative to current approaches.
4. take measures to obtain more representative samples within the weighting cells they employ.
5. investigate new quota and weighting variables which are correlated with propensity to be observed in the poll sample and vote intention.
The Economic and Social Research Council (ESRC) should:
6. fund a pre as well as a post-election random probability survey as part of the British Election Study in the 2020 election campaign.
BPC rules should be changed to require members to:
7. state explicitly which variables were used to weight the data, including the population totals weighted to and the source of the population totals.
8. clearly indicate where changes have been made to the statistical adjustment procedures applied to the raw data since the previous published poll. This should include any changes to sample weighting, turnout weighting, and the treatment of Don’t Knows and Refusals.
9. commit, as a condition of membership, to releasing anonymised poll micro-data at the request of the BPC management committee to the Disclosure Sub Committee and any external agents that it appoints.
10. pre-register vote intention polls with the BPC prior to the commencement of fieldwork. This should include basic information about the survey design such as mode of interview, intended sample size, quota and weighting targets, and intended fieldwork dates.
11. provide confidence (or credible) intervals for each separately listed party in their headline share of the vote.
12. provide statistical significance tests for changes in vote shares for all listed parties compared to their last published poll.