Horizon Polls

Late last year the Sunday Star-Times started running polls by Horizon Research. I commented on them in September and again in December. I had concerns over the results they were coming up with. Unlike all the other public parliamentary pollsters, they use an Internet panel for their polling, rather than randomly selected phone numbers.

There is of course no sure fire way to know whether a poll is accurate or not, except when measure against an actual election. The only election they have previously publicly polled is the Auckland Council. Horizon predicted Len Brown would win the Auckland Mayoralty with a 31%  margin, and in fact the actual margin was just under 14%.

I deliberately have not blogged again on them, because I think you should not rush to judgement off just one or two polls. No pollster is 100% reliable. Even with a margin of error, that is only to 95% confidence. That means that around 1 in 20 polls will be outside the sampling margin of error.

The katest Horizon poll is here. They helpfully include their results for the last six months, so I have done a comparison of the results for the Horizon polls with the average of the polls for the four other companies which do public parliamentary polls. They are Colmar Brunton, Reid Research, TV3 and Digipoll. Rob Salmond found that the average of the polls for the four public pollsters was highly accurate at the 2008 election.

In doing the comparison, I compare percentages of decided voters, as that is what is commonly reported and is comparable to an actual election result.

As one can see the Horizon Poll has National around 10% lower than the average of the four other pollsters. I’ve also included the lowest and highest poll score for National in each month, and Horizon poll tends to have National 5% to 10% below anoy other individual poll.

This is difficult to reconcile. It is possible that Colmar Brunton, Reid Research, Digipoll and Roy Morgan (who have long established track records) are all getting it wrong, and that this brand new company by using Internet Panels is getting it right. But I know where I would put my money.

With Labour, there is less of a gap. They tend to have Labour 2% to 3% lower than the average of the other polls, but pretty close to the lowest scores each for the other polls.

Horizon has had the Greens significantly higher than the average of the other polls for five out of six months. On average by about 3%.

With NZ First, Horizon have had a huge gap, often being more than double the average of the other polls. Around a 5% gap on average which is huge. Also worth pointing out there is a certain self-fulling aspect to polls. The increase in January in the other polls was probably partially due to publicity over them being at 9% in the Horizon poll.

The latest (May 2011) poll has ACT at over 5%. This isn’t as extraordinary as the fact their April 2011 poll had ACT at over 4%. In fact the Horizon poll says ACT gained more support in March and April under Rodney Hide (a 2% gain) than they gained in May under Brash.

I would not be unhapy with ACT at 5%, but to believe ACT are at 5% in May, you need to believe that they were at 4% in April prior to the leadership change.

The discrepancies between the Horizon polls and the other four pollsters are huge. I do not see a way to reconcile them. Now this is not to say Horizon are necessairly wrong. It is possible the other four pollsters are all wrong. Now the average of the other four pollsters got the 2008 election close to spot on, and Horizon were out on the Brown-Banks margin by around 17%. But despite that, it is possible that the other four companies have just fluked their results over the last few years, and only Horizon is applying an accurate methodoogy to their sampling.