Pew and the Polling Average

The final Pew poll is out today and it is very, very bad news for Mitt Romney. Obama is up by 3 among likely voters.

The polling averages are always the best indicator of where the race stands, especially relative to any single poll. But the Pew poll does have all the characteristics of a good poll.

1) Pew has a good historical track record
2) Pew has a very big sample; 2,709 likely voters were interviewed
3) Live interviews
4) Large numbers of cell-phone only voters were interviewed

This one poll can be wrong because of sampling error. The margin of error among likely voters is +/- 2.2 points which means the margin between Obama and Romney is highly likely to be within the range of Obama +7.4 and Romney +1.4. So there is room to interpret this poll as consistent with a small lead for Romney. But it is a narrow space.

The possibility of bias exists here too. But Pew's methodology is very, very solid in every way a pollster can control. Romney's hopes increasingly depend upon some "X factor" that pollsters are not currently able to identify.

Rasmussen, Systemic Polling Failure, and "Truth"

There's quite a brouhaha spinning around the interwebs with conservatives complaining that Nate Silver is cooking the books in his polling model because he's a liberal shill. Here's the argument in its original full form. It is pure drivel. Silver himself had this brilliant retort today ...

But, anyway, the whole thing got me to thinking about the state of the polls and the state of the race. Conservatives are reading Rasmussen and Gravis and Gallup and celebrating what is sure to be a big victory for Romney next Tuesday. Liberals are reading ... well, everything else and are sweating the national vote polls that have it close (actually, if you take Gallup and Rasmussen out of the naional poll averages, Obama is very slightly ahead) but feeling pretty comfortable about the state-level polls that show Obama leading in more than enough states to win a majority in the Electoral College.

What is the state of the race and whose polls are right? The truth is we don't really know for sure which set of polls is right. But, if I had to bet, I'd say Rasmussen and Gallup are wrong (for different reasons). Let me lay out the case for each side and explain why I think Rasmussen has it wrong but it is important to acknowledge that ...

Rasmussen could be right. Rasmussen has been identified as a pollster that has a "Republican lean" but this is really a mislabeling of what they are doing. It suggests that the pollster is just putting his thumb on the scale and cooking the books. That's not what Rasmussen is doing. At least I'm pretty sure they're not. What Rasmussen does is to use "dynamic party weighting." They weight their sample of voters so that they get to a designated number of Republicans and Democrats in their sample. The party weighting is "dynamic" because Rasmussen comes up with the relative weights of Republicans and Democrats based on some recent previous period of polling. Effectively, Rasmussen is making an educated guess about what the partisan makeup of the electorate will be (nationally or in a state) and then they make sure their sample looks like that.

So is it possible Rasmussen is right? Absolutely, it is possible. If their dynamic weights are accurate, then their polls will be accurate. And the flipside is also true. If their weights are wrong, they'll be wrong.

So what's the case against Rasmussen? A couple of days ago, Nate Cohn wrote that, if the polls stay where they are up to Election Day, Mitt Romney's only hope will be "systemic polling failure." This means that the vast majority of polls must have done something (question wording, poor sampling, etc.) that led them all to the wrong answer. Nate Cohn is right about this.

Systemic polling failure does happen. To understand why it is unlikely, we have to understand what other pollsters are doing differently than Rasmussen to get such systematically different results. First, other pollsters are not weighting by party ID at all. To most pollsters, party ID is something to be discovered in a poll, not something to use to weight the sample. This is because party ID is a fluid construct. People tend to change their response to a party ID question based on how they feel about the competing presidential candidates or based on how enthused they are to vote for their preferred candidate. Other pollsters simply ask respondents who they are planning to vote for and then use likely voter screens (of different varieties) to determine who is likely to vote and who is a non-voter.

It is entirely possible that this methodology will lead to a sample that is not reflective of the population as a whole. One reason is simply statistical variation. Maybe you just happened to get a disproportionate number of Romney supporters in your sample. This happens but we know the precise likelihood it will happen and this problem is corrected for by the various polling averages out there. Only a very small number of polls will be outliers for this reason and, when they are averaged into the rest, they will not have a substantial effect on the overall average number. A second problem is some kind of problem with the way we are sampling. One problem along these lines is the undersampling of certain kinds of populations like cell phone users or Hispanic voters who do not speak English, either of which may not be called by pollsters for different reasons. Some pollsters do call cell phones and some call Spanish-only voters. Here's Stan Greenberg talking about why undersampling of cell phone voters is causing Obama voters to be under-represented in the public polls we see:



Other pollsters who are good at what they do, use some kind of weighting mechanism to try to get a representative sample without calling them. This is increasingly problematic but is done well by some pollsters (like PPP in my view).

The point is that these pollsters are either getting a truly representative sample or they are using demographic weighting (not partisan ID weighting) to make sure they don't have a problem with their sample. The fact that so many pollsters are coming to basically the same conclusions (for instance, that Barack Obama has a small lead in Ohio) is evidence that they are likely right.

Could Ohio (and other states) still shift? Yes, but it is getting very late.

Could Rasmussen be right and the other polls wrong? Yes. I don't think so but it could happen and, if it did, it would be exactly the kind of "systemic polling failure" Nate Cohn was talking about.

We'll know next Wednesday morning.

Where are the National Polls?

We all see the divergence between the national tracking polls, which generally show Romney with a small lead on average, and the state-level polls, which generally show Obama with small leads in most of the key swing states on average.

How could these two things be true at the same time? There's actually two things going on here:

First, it is entirely possible that the President is winning by narrow margins in swing states like Ohio and Colorado but is losing by more in "solid Romney" states than he is winning in "solid Obama" states. That would translate into a national popular vote deficit but, potentially, an Electoral College victory. Most of the sites that create election-prediction models like Sam Wang's Princeton Election Consortium and Drew Linzer's Votamatic use state-level polls either exclusively or more heavily than national polls because the state-level polls provide more information that is useful. They are generally right to do this in terms of predicting what matters - the Electoral College outcome. But those national tracking polls are still unpleasant to look at ... which brings me to my second point ...

Second, it is really striking to me how few "full" national polls there have been in this campaign, especially in the final weeks of the campaign. There are at least 8 different national "tracking" polls that are released just about every day. But these are not the same as a regular poll. I can't claim perfect expertise on the impact (or not) of this phenomenon but it does seem to me there is some important effect of looking at only tracking polls as opposed to regular "full" national polls. Tracking polls take a smaller sample every night for 3 or 4 or 5 or even 7 (in the case of Gallup) nights. They put these together to create a national polling number. Then, when a new night comes on board, the sample from the oldest night drops out. What you end up with is a "rolling average" of the most recent nights. One potentially problematic aspect of this is that, because each night is a distinct sample and because the pollster wants to keep the demographics in the full sample correct from night to night, you have to weight some very small samples night to night before adding them to the sample. This would seem to introduce some error as opposed to doing a regular 2 or 3-night poll and then weighting the whole sample from there. In effect, you'd need to do less weighting as you'd have stronger sub-samples to work with.

Now, I'm just kind of spit-balling on all that and I could be wrong. But I do think that regardless, it is notable that there have been so few full national polls. Indeed, there has not been one full national poll that I can find since the third debate. I'd be interested to see one of those from a good pollster who does live interviews with cell phones included.

UPDATE: In a post today, Nate Silver speaks to the question of the split between the national tracking polls and the state-level polling. The bottom line is that he says the state-level numbers are suggesting leads for Obama in swing states like Ohio that are statistically very meaningful. Mark Blumenthal makes some similar arguments in Pollster.com

One Poll Makes you Happy ...

... and one poll makes you sad. That's where we are this morning. Let's start with the poll that makes me sad.

NBC/WSJ released a national poll yesterday that has Obama and Romney tied at 47. This poll worries me a bit for a couple of reasons. First and foremost, NBC/WSJ is a good poll (Peter Hart and Bill McInturff), they do live-interviews, and they call cellphones. Second, it worries me because Obama's number is 47. As Chuck Todd points out, if it was 49-49, that wouldn't be so bad for the President. But 47 is not really the topline number you're looking for as an incumbent. Now, all of that said, this is the first national NBC/WSJ poll since late September (before the first debate). That poll had Obama ahead by 3 (49-46) so this is not a massive shift by any means and is technically in a range that could simply be statistical noise.

So what poll makes me happy this morning? Quinnipiac has a poll out this morning that has Obama ahead in Ohio 50-45. Quinnipiac is also a good pollster and they also do live interviews and this poll had a very large sample (over 1,500 likely voters). Like the NBC/WSJ national poll, Obama's 5-point lead here is down from where it was in a Quinnipiac poll of Ohio in late September (it was 53-43 then) but Obama still leads by a healthy margin. Perhaps more important than any of that, if you go back through all the polls of Ohio, the last time Romney led in any poll there was in a Gravis Marketing poll (they've got a strong Republican lean from what I can tell) conducted just after the first debate and Romney led by just 1 point in that poll. The bottom line is that Ohio looks good for the President.

What if both of these polls are perfectly accurate? If so, I think it would be likely that Obama would win the Electoral College and lose the popular vote by just a hair.

But there's still two weeks to go and tonight's debate may yet move the needle in one direction or another.

UPDATE: As if on cue, ABC/WP unveiled their new tracking poll today and they have Obama up 49-48. This is not a great result for Obama. But it is better than the NBC/WSJ poll. Obama doesn't need to get to 51. He needs to get to 50. And getting there is easier when it is 49-48 than when it is 47-47. Am I reading too much into one poll? You bet I am. That's what happens this time every 4 years. Now back to your regularly scheduled program.

Cell Phones and Polling

Not-so-surprising news that cell-phone only voters are more supportive of Obama than landline voters. It is fair to say that, if Obama wins, it will only be because of cell phone voters.

Cell Phones

By the way, in the recent SurveyUSA poll of North Carolina, the pollster called cell phones and landlines. They found that, in North Carolina, Obama is ahead by 15 points among cell phone respondents (52-37) and it is basically a tossup (Obama leads 46-45) among landline respondents.