Why are the polls so different?
February 8th, 2005
-
Could it be the way they ask the questions?
With the current claims by the Tories that the polls have a systemic bias in favour of Labour eveybody seems to ignore the fact that each firm puts the voting intention question in a unique way. There are subtle differences that could have an impact on their figures.
MORI which has been in this game for longest, put it very simply: “How would you vote if there were a General Election tomorrow? Conservative/Labour/Liberal Democrat/Other”. This is blunt, straightforward, and designed for a face-to-face interview.
ICM which also has been doing this for a long time, put it slightly differently recognising, perhaps, that people are not always so certain about these things. Their question is: “If there were to be a general election tomorrow which party do you think you would vote for? Conservative/Labour/Liberal Democrat/Other” We like the “..do you think..” element because it softens the question without being too wordy.
YouGov is very similar to MORI and, of course, respondents read it on a computer screen. “If there were a general election tomorrow, which party would you vote for? Conservative, Labour, Liberal Democrat, Scottish Nationalist/Plaid cymru, some other party, would not vote, don’t know.” In recent months YouGov has seen very high figures for “others” particularly UKIP and maybe this first screen helps that. Does the “would not vote, don’t know” element encourage users to go to the “others” page so boosting UKIP/Veritas?
Populus is fairly new to this business and has rather a wordy question which must be quite a mouthful for its interviewers to get out clearly. It goes: “On the basis of your view of the parties at the moment, if there were a general election tomorrow, which party would you vote for? Conservative, Labour, Liberal Democrat, some other party, or would you not vote at all?” We do not like this at all. The first phrase - “on the basis of your view..” seems superflous and the “some other party” hardly encourages you to admit to not supporting the main groupings. The final “would you not vote at all” seems to add to the conclusion.
Communicate Research is also fairly new to the business and ask its very simple question without offering prompts on party choice. “Which party would you vote for if there were a general election tomorrow?” A simply straightforward question let down by the lack of party choice prompts. This has been shown to produce higher ratings for the two main parties.
NOP uses the fairly simple” If you do vote in the next general election, which party will you vote for - Conservative, Labour, Liberal Democrat, or some other party?” We are not sure about the opening phrase - “if you do vote” - which seeks to be non-judgemental but perhaps dillutes the impact.
All the pollsters that name the parties present them in alphabetical order with the Conservatives first. Does this have an impact? Would doing it “Labour, Conservative or Liberal Democrat” produce different numbers?
Our favourite is ICM which seems to get the tone just right. What do other people think?
The latest Populus poll has had some impact on the spread markets with the Tory and Lib Dem numbers being reduced and Labour being increased. IG Index spread prices: LAB 355-362 (+3) : CON 190-197 (-2) : LDs 68-72 (-2) .
© Mike Smithson 2005
MessageSpace Advertising
Obviously there’s no perfect question, but I agree with you Mike that ICM seem to have the best balance
ICM has the best question and You Gov the best method.
I can’t agree that Yougov has the best method when such a large percentage of the population don’t have home computers. I have just read that they got the US elections badly wrong and they are usually out of kilter with all the other pollsters. Which elections other than the 2001 General election did they get right? Or is their reputation based on that one poll? And did they get the turnout right?
They were the only pollster (I believe) to get the Scottish elections right. Also they were within the margin of error for the US election as well. They called for Kerry but heavily caveated and pointed out that Bush could win as their poll was so neck and neck. I dont think they can be justly criticised for that!
They used to be known as anything YouwantGuv because they were prepared to ask anything their clients wanted them to. Something the others wouldn’t do. Such as “Which wife of which party leader is most likely to persuade you not to vote for their husband”?. (And I’m not joking!)
Rik. For the US elections they called it 50 to 47 the wrong way (So I have just read on the other board) . That’s quite a long way out.
Roger re 5 can you be careful with the terms of your criticism. To attack the methodology or question their performance is fine and a proper issue for debate. Anything that casts doubt on their probity could take us into different territory and I might have to moderate.
Interesting piece in the Times today about the Alastair Campbell dodgy email scandal. The interesting bit about the article is that they caught him dining with a Tory MP, when trying to find him. The MP was Andrew Mitchell. So there are now more rumours flying around about defections, though looking at his profile on the BBC I would guess it is unlikely.
As for YouGov I don’t see why people object to them asking questions the newspapers want. They are a business and they obviously need to generate income so ergo they ask questions about leaders wives etc. These polls should be taken with a massive pinch of salt anyway. By the way they also do more serious questions. They were some v.interesting surveys about British attitudes to sport in the Telegraph over New Year.
How can psephology show that it is an empirical science and not just a an interminable string of of tiresome a priori speculations (such as we have now) and post-hoc justifications (such as we will have post 05/05)? Simple: the main players must agree to fund a controlled experiment where the same electors are asked their intentions in the gamut of formats outlined above within a short time span. alternatively, they should agree to use the same question, which might be cheaper.
My own opinion is that the right question to ask is the traditional one: ‘if there was a general election tomorrow, how would you vote?’ No choice list, no cues, just a straight question.
A kind of uber tracking poll, I take it. That might be difficult considering that polling is a competitive business.
I use the version in 9. Most people don’t hear my first introduction words saying who I am as they are too busy recovering from the shock of someone ringing the doorbell after dark. So their response tends to be fairly accurate - apart from the LibDems who we all know don’t like to tell you how they are going to vote - bit of a give away now. And as another LD’er posted a few days ago - we do know that some like to engage with us canvassers to slow us down - but we know about that tactic as well.
Today’s update of the Populus poll adds two interesting sidelights. First, the reported Labour surge doesn’t appear to have been a “Baghdad bounce”: the poll suggests that people are moderately encouraged by the Iraq elections, but that it hasn’t changed very many minds about the war itself, with a 3:2 majority against using Populus’s version of the question (which historically has tended to produce more “opposed” replies than other polls). Second, the poll shows for the first time a clear margin for “yes” in a European Constitution vote, though explicitly using a change in methodology: it offers a “wouldn’t vote” option, and numerous potential “no” voters seem to have gone for that. “I dislike the EU so much that I refuse to vote about it”? I’m not sure how realistic this is, but it shows the fluidity of opinion.
Nick
I have always taken the majorities against the EU constitution with a large pinch of salt. I’m not saying it will win, but when the time comes the “Yes” camp will be able to rubbish the idea that we can cherry-pick those bits of EU institutions that suit us and throw away the rest - it will be easy to present the “no” camp as all take and no give.
I was old enough to vote in the 1975 referendum (i.e. I am ancient) and had intended to vote “no” until Heath trounced Barbara Castle in a televised debate (from the Oxford Union :)) and in any case I was very unhappy about putting my X in the box the fascists (National Front then, BNP now) wanted it in.
I think the left “no” vote is very soft and I doubt that the “no” campaign will have the savvy to keep it on board.
Hitherto, polls have shown that people who are opposed to the EU constitution are more likely to vote than those in favour.
I’d say that last year’s Euro elections offer quite a clear pointer to the likely outcome (roughly 60:40 opposed on a 42% turnout).
13 - well, the BNP opposed the Iraq war. I don’t think that means you have to support the war to be a non-fascist.
The danger with what you suggest is that if you present a choice between withdrawal and swallowing the whole EU pill without any “picking and choosing”, withdrawal will turn out to be much more popular than you expect.
Mike re 7. Apologies. But I would like to emphasise that I didn’t make up the question about Prime Minister’s wives it was a real question. I was one of their panel and the answer did appear in the Mail on Sunday. As for the name “Anythingyouwantguv” that was taken from an article about polling in the Guardian.
What is the support for withdrawal from the EU? About 20%? At the end of the day it will come down to whether people fear quitting the EU more than they fear its constitution.
Maybe 20% now, but my point was that it will rise much higher if you seek to close off the option of retaining EU membership without the obligation to keep on and on driving towards “ever closer union”.
Following on from Nick’s comments at 12 the other interesting detail of the poll is the new concern with ‘International Poverty’. It now scores 17% with the public as an important issue as against 10% for Europe! Among women the figures are even more extraordinary, 19% against 5%. This might just show how fadish respondants are from the comfort of their own homes. Or it might show a more caring electorate. I’m not sure how this squares with the attitudes towards asylum seekers though.
McHack @ 11. I have talked to canvassers of all parties and they all say the same about the supporters of others. i.e. You could rearrange the names of the parties in your sentence and it would still hold true.
If you’re going to prompt with a list of parties, I would have thought that having the same order for every interview in your sample would be a pretty bad idea. Surely they should either be in a randomised order for each individual interview, or be in the order they will appear on the ballot paper in the constituency of the interviewee?
Nick at 12, I haven’t seen the poll, but possible ideas for the unexpected Euro referendum figures.
- The whole poll is a big fat outlier, they just hit a big chunk of Labour loyalists in their sample this time, so they get both a high Labour figure and lots of people saying they’ll either vote with the government line on the referendum or stay at home because even tho’ they may not like the Euro constitution, they *don’t* want to give Labour a good kicking.
- The ‘won’t vote’ people are the ones really don’t care that much about the EU as it’s a really low priority for them. They might very well say ‘no’ if asked for an opinion, but only because you asked them and that seems to be the mainstream opinion. They’re not really interested. Certainly not interested enough to go out and vote about it.
- The ‘won’t voters’ do have an opinion on the EU constitution, but are embarrassed enough about it that they don’t want to tell a pollster what it is, so they say they’re not going to vote even tho’ they are. No idea whether these would be ’shy-Euroskeptics’ or ’shy-Euroenthusiasts’. You could argue it either way.
- The ‘won’t voters’ are people who feel that the constitution isn’t really going to change very much — that it actually is pretty much a codification of the status quo — so whether they think the EU is good or evil, they don’t think it matters one way or the other whether the constitution passes, so why bother to vote?
- The ‘won’t voters’ firmly believe ‘Resistance is futile, we will be Euro-assimilated’, it’s all rigged anyway, voting never changes anything, so why dignify the process by taking part?
- The ‘won’t voters’ are Euroskeptics of the ‘get out, get out now!’ variety, rather than the ‘renegotiate into a Europe of nation-states’ flavour, who’ve noticed that the constitution has a clause that actually provides a formal mechanism for getting out. So, even tho’ they’re can’t bring themselves to vote in favour, the constitution passing arguably moves them towards their goal. Therefore they’re sitting this one out.
No vote soft - well we’ll see of course but I think it is harder than the yes vote. As any canvasser knows almost all the strong opinions one hears about the EU are negative.
MORI conducted an exhaustive survey in July which suggested that 4% are strongly supportive of the constitution and certain to vote in a referendum, and 14% are strongly opposed and certain to vote.
I would guess that between them, those two elements make up about 40% of those who vote in the referendum.
As a matter of actual fact Populus (and I assume most other companies) randomly rotates the order in which the names of the parties are listed in the voting intention question prompt. Not prompting with party names has been shown clearly to result in understatement of the Liberal Democrat share. I think this whole debate about how the question is worded is overblown: polls using slightly different voting intention question but doing fieldwork at the same time, by the same method, using the same weightings will get the same result. You need to keep always in mind margin of error: polls are blunt instruments, not precision measuring tools.
Re 24. Thanks Andrew. It would be interesting to know if changing the oder in which the parties are presented has an impact.
A friend of mine who runs a resaurant told me that when they want to push a particular dish the waiter would reel of the lists of specials today - with the last one listed the one they want you most to order. And, he says, it works.
Re [13] [14] [17] [18] I think that it is worth pointing out the Constitution actually does end the “ever closer union” form of words, and in several areas it also sets out limits of EU powers. Under conditions where all the other 24 member states ratify and the UK fails to do so, then effectively you are actually talking about at the very least a limited withdrawl from the EU. Given the experience of the 1975 referendum- at the beginning of the campaign the polls showed 2-1 against and as we know the vote ended up nearly 2-1 in favour, I could certainly see circumstances where the electorate vote Yes. Indeed, despite the current poll ratings, I even think it likely that the UK will vote Yes. Which brings me back to the thread- polls can vary widely and still be accurate at the point that they are taken. There are after all a snapshot rather than a realtime feed.
27 - point taken about the wording but my reading of the sense of the Constitution is not that it formalises a concept of limited European government: it may hold its participants to a line in the sand now, but it doesn’t discourage the starting gun from being fired very shortly.
I need to go back to the text to justify this, so it will take me a while to firm it up.
Yes indeed James [26] - it is the 1975 precedent which was in my thinking in my ealier post, although arguably the consequences of a “no” vote then were more fear-inducing. However, it is not a foregone conclusion that all the other states will vote “yes”.
One point on polls that has not been discussed here (forgive me if it has and I’ve missed it) is the relative costs of the various methods. Is there anything about this on Anthony’s site - I do try to keep up with it but again I may have missed something. I would assume that automated polling is cheaper and therefore more practicable for constituency or sub-regional polling.
[27] As I think people on all sides of the debate agree, the Constitution is a fairly nebulous document. However, in terms of the debate on the issue as it stands, it probably limits as much as it proposes. Whatever happens next, after ratification, is different- and pretty difficult, I would have thought, to convince the electorate for or against.
From the point of view of the campaign though, the Yes guys will probably focus on the point that if just as independently minded states as the UK are still happy to sign, but the UK does not, then de facto Britain is withdrawing from critical parts of the EU.
When people say, well we can reject the constitution but still remain full members of the EU, that doesn’t look credible if everyone else has ratified- which is why, of course the UK is so late in the ratification calender. So in fact, refering to points above, it is quite possible that the polls do drift in the Yes direction- indeed the trends seem to support that even now.
Re [28] Yes, and of course if there is a rejection elsewhere, than things are probably quite different and much harder for the Yes camp.
IA. Generally face-to-face surveying is far and away the most expensive. In order to get that number of surveys in that short space of time is incredibly labour intensive (is that why it overestimates Labour
). Telephone surveys are less so because the interviewers can contact people extremely quickly (especially with modern autodialling). The internet should be the cheapest because there is no need for labour costs. However, the costs of maintaining a panel and of employing people to run the technicalities may offset this partially.
IA - like Graham says, face-to-face is the most expensive, then phone polling, then internet polling. My guess would be that saving on wages would mean that Rasmussen style automated polling would also be cheaper than normal phone polling, but that is just a guess.
29.
As against that, rejection of the Euro by this country has not led to our exclusion from the EU.
I can’t help thinking that Labour might be better going for an early referendum if they win the next election. Otherwise, any boost they may get from winning will have disappeared.
re [33] Yes, beacuse the UK had already negotiated opt-outs on the Euro and the Social Chapter. The constitution is designed to be a package of measures to improve overall decision taking. By definition there are no opt-outs, so it really is all or nothing.
Well it seems 2006 it is though.
Anthony. Yes, except the installation costs of the system may be prohibitive. In the long run, the biggest fixed cost of a research company is labour. However, as the industry is technologically based then the cost of keeping ‘up to date’ is relatively high. Like any investment, you have to make a decision based on how much use you are going to get out of it. As opinion polling is actually a relatively small part (if high profile) of any research company’s business it is important that the methods you are able to employ are also accpetable / useful to a wide range of clients. For example, many of our clients are in the public sector. Many of them are fairly averse to ideas that may be considred intrusive. Automated messages are considered synonymous with dody sales, so they wouldn’t use it - so nor do we.
Thanks fir all your replies on the economics of polling. What on earth are “dody” sales, I wonder… must be a typo for “dodgy” - gosh, you have to stay awake round here
In the contest for the best pollster my vote goes to ICM.
Why?It has the longest continuous history from 1990 and a proven record of being the most accuratet at the last 2 elections(it was Bob Worcester who said “How do you do it Nick”).ICM seems particularly accurate on the Tory vote share, and also seems less volatile from month to month.
So what does ICM suggest for the 2005 election?
Comparing Jan 2005 with Jan 2001 polls gives Con -3,Lab-6,Lib +5,.If these differences persisted to the election then shares would be Con 30,Lab 36,lib 24.
If instead we took current poll as difference to last election result shares are con 30,lab38,lib 21.
In looking at seat with relatively small swings ,a uniform swing model seems the best rather than tactical voting model.On this basis a relatively small change of seats would be expected.
ICM’s final poll for the Evening Standard in 2001 was well wide of the mark. They gave Labour a lead of 17% (compared to an outcome of 9%) nationally, and a lead of 27% (compared to 17%) in Greater London. Many thanks to Mike Smithson for referring to this from time to time, as it is usually overlooked (though to their credit ICM have it on their website).
rogerheape, following on from your 4 years ago calculation
From their website, ICM did 9 polls in the 3 months prior to the 2001 election with an average score of
Lab 47.0 (5.0 higher than the actual result)
Con 31.8 (0.9 underestimate)
LD 16.1 (2.7 underestimate)
Oth 6.1 (0.4 underestimate)
(The figures dont match because of rounding)
Re 37 - Because the 2001 General Election took place on June 7 you should reallt compare January polls now with February polls then to get the same time difference. This makes a big difference.
40 - but Mike we dont yet know when the GE will be -last time it was thought to be in May - so arguably it is comparable!
I think I have done all this before. The general trend is Lab higher than final, Con about right, Lib Dems lower. It is best to use MORI’s site as it has all the polls on there.
Re 37ff. ICM set the standard for UK pollsters - first to adapt their methods after the 1992 debacle and most right in both 1997 and 2001. All polls are subject to margin of error and one in 20 will, other things being equal, be rogue to a degree. So it is always best to read polls in batches - if you average, say, the ten ICM polls before polling day in 1997 and 2001 they were very close indeed. And the average Tory share in ICM polls throughout those Parliaments was within 1% of their subsequent share (that’s the flatline for you!)
slots Men first feel necessity, then look for utility, next attend to comfort, still later amuse themselves with pleasure, thence grow dissolute i
swing trading strategy…
swing trading strategy She had recommissioned from Months storgee in her perfect freedom of swallow’d, disinfected independence ; had modestly re-published liason in her business trunk-shafts and the shaltst to support hers…