Poll Weighting

A reader e-mails:

I notice a number of US pollsters appear to adjust their samples to achieve a predetermined proportion of Republican/Democrat voters.

Would you be kind enough to do a blog post on this subject, citing the nature of the practice, the reasons for it, it’s legitimacy and whether or not it skews results.  It seems to me it does.  Also, to your knowledge is such a practice carried out in NZ?

Weighting is very common to help ensure the results are representative of the overall population. If (for example) 20% of adults are aged under 30 and they make up 15% of your responses (as say young people are out more and harder to get hold of), then you weight their responses by 20/15 or 1.33.

Normally weighting is done on demographics such as gender, age, area, income, ethnicity etc.

In the US some pollsters do weight by party identification. This is not really possible in NZ, as most NZers do not publicly declare their identification. The purpose of such weighting is to make sure a sample is not under-represented by Republicans or Democrats.

However there are dangers in doing so. What do you weight to? You see party identification levels change over time. As party gets more or less popular, more or fewer people identify with them, so if you weight by party ID you may miss that there is a trend of decreasing or increasing support.

In NZ there is no polling by party ID, but sometimes you may see a poll weighted by vote at the last election. This is an attempt to have the results reflect the voting population. The problem with this, is people do not always correctly recall who they voted for at the last election. There is a tendency for some to say they backed the winning party, even if they did not.

In my opinion the best thing is to have your sampling done in a way which means you do not have to weight by party identification. If your sample is picking up (for example) 65% Democrats and 35% Republicans there is probably an issue with your sampling, and while weighting mitigates that, it can mean you ignore the underlying issue.

There is another issue with weighting. Sometimes it means you can exaggerate the error in a small sample. Let’s say you have a poll of 500 and 10% of the population is from (say) Timaru. Now if for some reasons only 20 responses were from Timaru, then they would get a weighting of 2.5. Now that small sample of 20 has a huge margin of error, and you have just compounded it by weighting.

Overall weighting is (in my opinion) a good and necessary thing to do. But I prefer to do it only on known factual demographics.