Outlier polls

Every so often you may see a poll which is commonly regarded as an outlier poll, ie well outside where other polls  are. A reader has asked me to comment on what do pollsters do, when they get one.

A recent Bloomberg poll in the US had Obama 13% ahead of Romney. The average of the public polls has the gap at around 1%. Mark Blumenthal writes:

The most likely possibility is that this poll simply represents a statistical outlier. Yes, with a 3 percent margin of error, its Obama advantage of 53 to 40 percent is significantly different than the low single-digit lead suggested by the polling averages. However, that margin of error assumes a 95 percent level of confidence, which in simpler language means that one poll estimate in 20 will fall outside the margin of error by chance alone.

This is worth remembering. One in 20 polls will fall outside the quoted margin of error. One of the challenges to being a pollster is what to do if you think one of your polls may be an outlier.

Bloomberg have done a statement on their poll, talking about what they did and didn’t do. This is a good idea, and they look at theories on why they got the result they did, such as:

Maybe you had a higher-educated respondent pool and they tend to like Obama. Maybe we did. Of all the theories, this one holds some water. The 2008 exit poll shows 24 percent with a high school education or less, compared to our 20 percent among likely voters; the 2008 electorate had 31 percent with some college, and we had 23 percent. In 2008, 28 percent of voters had a college degree and 17 percent more had some postgraduate education; we had 34 percent with a college degree and 21 percent with some postgraduate exposure. However, in our poll, every education subgroup votes for Obama over Romney.

We played around with the data to test whether our findings would have changed had our education distribution looked more like the 2008 exit poll. By “played around,” I mean we created a 52-cell weight variable accounting for age, race, and education. The presidential contest becomes a 10-point race: 51 percent for Obama and 41 percent for Romney. It is still a double-digit lead for Obama and would likely have created as much stir as our 13-point lead.

In the end they conclude:

In the end. We will soon know whether this poll is, in fact, an outlier. Potentially, this poll caught the electorate when the wind was at Barack Obama’s back for a brief moment in time.

If a pollster gets a response which is “way out there”, there are a number of things you can do.

1. Check the raw data. Audit a higher proportion of calls than normal.
2. Check the numbers add up. Sometimes human error can sneak in.
3. Look at your sample – is any group seriously under or over represented? Maybe all your Auckland responses came from Parnell?
4. Play with your weightings. As Bloomberg did, see if different weightings will materially impact it.
5. Check your questions. Was the script different to last time.
6. Check what was happening when you were in the field. If an All Black game was on one night, then your polled no rugby fans, and that could make a difference. Was there a final of a popular TV show?
7. Look at your cross-tabs. If one indicator moves (say party vote) but not another (say right vs wrong direction) then that may imply an unusual sample (the one in 20)
8. In extreme cases, you can always redo the poll. But it is a dangerous thing to throw a poll away just because you assume it is an outlier. In fact, you may be first to pick up a change or trend.

Ultimately, as Bloomberg says, it will become apparent in time if the poll was an outlier or not.