Why I have resigned from the Research Association of New Zealand
I (and Curia) have just resigned as members of the Research Association of New Zealand. It is something I never thought I would do, and do with great sadness.
I have been an enthusiastic member of the industry body.
I joined in 2004 when I formed Curia Market Research. It was an honour to be nominated by (Professor) Jack Vowles and Gabriel Dekel.
I served on the executive of its predecessor. I was the lead author on the NZ Political Polling Code. I was a guest speaker or panelist at AGMs, including last year’s one.
So what happened?
Up until around two years there had been no issues. But over the last two years the complaints process has been weaponised against me. I can’t even count up how many complaints there have been. I will cover in some detail the significant ones.
Now lots of people have to put up with complaints processes. I wish I could be like Mike Hosking who probably doesn’t even blink if someone complains about him to the BSA.
But these numerous complaints have caused me huge stress. I won’t go into the details of it here, but it has been very significant.
Yesterday I saw an e-mail from RANZ that, not only had the PSG upheld the latest complaint against me, but that it was considering a recommendation that could involve suspension to expulsion.
I was absolutely stunned. I could not believe how what I saw as a subjective dispute over question design could possibly meet the threshold for such a move. I’ll get into the details later, but basically it was over whether a client should be allowed to ask:
The UK health service (the NHS) has stopped the use of puberty blockers, which begin the gender transition process, for children under 16 as it deemed they are too young to consent. Do you support or oppose a similar ban in New Zealand on the use of puberty blockers for young people 16 or younger?
My first response was considering getting people to complain against polls from other companies (I have a large file of potential complaints). But I calmed down and tried to remember my motto of don’t be an arsehole or deliberately harm others. That is why I have never ever encouraged a complaint against another company, despite many many opportunities. I have even tried to talk people out of them.
I decided I would trust the process and argue my case to an independent panel. I was very confident that there is no way the current dispute could be seen as resulting in a sanction that I don’t think has been used in recent history. I engaged a lawyer and I started collating information. I looked at processes for other professional bodies, and also obtained detailed advice on whether the actions of RANZ could breach the Bill of Rights by putting pollsters in a position where they have to refuse questions on controversial issues, because if they accept them they get suspended or worse.
So I had my plan. I was confident my chances of not having a decision of suspension or expulsion was pretty damn good (but of course you can’t know). I was all set to spend what would have been tens of thousands of dollars, confident in my belief that I should trust in the process.
But then I reflected what happens if I do win. It won’t stop. I’ll still have the usual people complaining about most of my polls and I would be having to run Curia with the massive anxiety that every single poll I prepared would have to be defended, arguing over what is an incredibly subjective area of question writing.
This is insane! I do political polling and many people oppose some of what my clients stand for. But this is weaponising an appeals process. It forces people like myself into the impossible position of either not serving my clients, or trying to satisfy a professional body that should be objective and neutral, but I sadly don’t think is being either.
So, with the deepest regret I resigned from RANZ today.
Since I made the decision I feel so much better, that I know it is the right thing to do. My stress levels have improved knowing I don’t have to spend the future fighting endless complaints against me
In the rest of this post I want to detail a bit of history of Curia, how important reputation is to me, the whole area of question design, and details of some of the complaints (including where I could have done better), and some suggestions for RANZ.
Finally I am going to cover a proposed accountability scheme for Curia going forward, so people can still raise issues in good faith and get them resolved.
20 years of Curia
Curia for many years was most well known as being National’s pollster. I first got involved in electorate polling for National as a Young National volunteer in 1993. I was one of many callers, but had a real interest in it so Al McLauchlan and Jack McFaull who ran the programme for National showed me what they did, and answered my many questions.
In 1996 I ended up working in Parliament and Al retired from his polling roll. With MMP coming the party engaged a professional polling form for most of their polling, but I was asked to take over from Al and do occasional electorate polls using a team of volunteers .
The results were good enough that for the 1999 election ,my internal team top volunteers was asked to supplement the polling from the external pollster and in 2002 I was doing most of the polling for National. I was very proud that they placed equal trust in what I could do with a team of volunteers, with companies that had been operating for decades.
In 2004 I had been in Parliament for 8 years and was exhausted (when I left they replaced me with three staffers) and I had so enjoyed doing the polls, I thought I would take a leap in the dark and set up my own polling company – Curia. National agreed to become a client (my decision wasn’t dependent on that though) and Curia was born.
Since then I have done work for over 300 different clients. I have never had to advertise – it has all been word of mouth. I literally could not count up how many actual polls I have designed and analysed, but it is probably over 3000
I’m very proud to have provided polling services to five NZ Prime Ministers and four Opposition Leaders.
And while I am known as a National supporter, I provide a professional service regardless of politics. I have actually provided polling services to 10 different political parties that range the entire political spectrum.
I have done polling for Labour MPs and Ministers. I have polled for Green Party members and Alliance members standing in local body election. I have polled for lobby groups I agree with and lobby groups I totally disagree with. I have even polled for groups on opposing side of an issue such as euthanasia. I told both groups that I had been asked to poll for the other side also, and they both consented as they really wanted to use Curia and trusted me to be absolutely professional.
Reputation
It is hard to understate how important reputation is to me. As I said, I have never had to advertise for clients. I am able to have discussions with senior Labour MPs where they trust me not to ever quote them. I have sat on numerous cross-party parliamentary groups on particular issues where MPs from ACT to the Greens take party and we have some very undiplomatic conversations.
I also take pride in my relationship with many in the media. I think I have declined just one interview request in the last eight years or so. I always give my honest opinion to them, however I do of course keep some things to myself.
I believe Curia has a good reputation for accuracy. In the 2023 election we were one of only two companies that had all parliamentary parties within the margin of error. In 2014, John Key kindly called me the best pollster in New Zealand in his victory speech.
In 2023 an independent researcher (https://public.flourish.studio/visualisation/12585241/ ) found that Curia had the least divergence from the all-firm trend between 2020 and 2023.
The public polls for the Taxpayers’ Union in 2023 correctly predicted the winner in Northland, Auckland Central, Napier and Ilam. The Tamaki poll had Brooke (who had been regarded as a long shot) just 2% behind, which arguably helped her momentum to win the seat.
The polls for the Māori electorate seats were less accurate but they are notoriously difficult to poll (and if I get to poll them again, I am going to try some new things to identify Māori roll voters) but they did point to the risk of Nanaia Mahuta losing, which wasn’t widely expected.
By contrast in 2017 one RANZ member produced a poll which was 22% out on the National-Labour gap and in 2011 another RANZ member had a poll which was 16% out on the National-Labour group. I wonder aloud what does more damage to the industry – polls that get a general election result completely and massively wrong, or arguments over the wording of questions?
No polling company should be judged by their last pre-election poll.as all companies have polls that turn out not to to be very accurate Curia, like all companies, have polls that are not as accurate as one would like. In 2023 I had National below their actual result (but within the margin of error) and in 2020 I had them higher (as almost everyone else did as their support fell away in the last week).
The vast majority of polling companies in NZ work very hard to get accurate results. I often recommend my competitors if people don’t want to use me. I have sent clients to UMR/Talbot Mills, Research NZ, Colmar Brunton etc.
Question Design
Two of the three upheld complaints I received related to the wording of questions on controversial or sensitive issues. Question design is a very subjective area. Every day in the world there are thousands of people critiquing a poll question, claiming it is leading or biased etc. Just as people critique political arguments and fact checkers critique claims (and some fact checkers have been shown to do a horrible job).
What is absolutely key is that any poll report must include the exact question asked. This allows people to judge the results in light of the wording. I actually think people are quite good at working out if question wording has had an impact on results.
Clients commission polls for a number of reasons, and this impacts question design. Some just want to know what opinion is on an issue without any context given, for their own decision making. Some try to explain the pros and cons of an issue to then see what a more informed rather than instinctive reaction is. Some have a firm view on an issue and wish to test what language or facts are most likely to persuade people.
For example one client asked me in 2019 to poll four different phrases in a poll on euthanasia – assisted dying, euthanasia, assisted suicide and assisted killing. We tested support and opposition on each phrase so they would know which phrase was most helpful to them.
The client wants questions that are of value to them. This does not mean they get to ask any old question, and nor does it mean they get the result they want (many clients have received results which were unhelpful to them, and didn’t publish them) but it does mean as a pollster you need to understand the client. I have described polling as both an art and a science and question writing is an art. I have no idea how many polling questions I have written but if I’ve done 2,000 or so polls, it could well be many thousands.
It is extremely rare for me to just accept the questions the clients propose. I almost always suggest changes, and/or warn that their proposed wording will make it easier for people to downplay the results.
In fact, the client, whose poll was the subject of the final complaint, has said they use me despite the fact I actively campaign and oppose them on probably around 80% of the issues they campaign on because they like the fact I will give pushback on the questions, to make them more defensible.
I also fact check questions, so that any assertion in the question is robust. For the question on puberty blockers for under 16 year olds I actually spent almost two hours reading the Cass Report.
A poll by another company for the Better Public Media Trust claimed that 60% of NZers supported the TVNZ/RNZ merger, but the actual question was:
The government is planning to merge TVNZ and RNZ into a new state-owned public media service, with an extra $109 million per year, which equals to $22 per person per year. If this organisation provided new content for niche, minority and regional audiences while keeping the current TV, radio and online services as well, would you support it?
Now I think it is a pretty ridiculous question as it rests on a premise that is hypothetical. It’s like saying would you support merging the DHBs together if it meant every hospital kept providing their current services, plus you get 50,000 more operations. Who would say no to that?
I would strongly advise the client against the question, on the basis that, while some media might run it without scepticism, it will have zero impact on the actual decision makers. But if they insisted, I would be okay with running it, because clients do have the right to ask questions.
But now we seem to be in a situation where a decision to run wording like the above can result in potential expulsion from the Research Association. This should be of concern to many in the industry.
I would have thought the threshold for finding a question is biased is if it is factually incorrect or uses incredibly inflammatory language. I guess examples would be.
Will you vote for the former war hero John McCain or Bill Clinton who dodged the draft? Or, would you rather spend $500 million on new cancer drugs to save lives or on new frigates to help the Defence Force kill enemies more efficiently?
But the two upheld against Curia have been ones where I think reasonable people could agree to disagree.
The complaints
The first of the significant complaints was in 2022 over a poll for the Auckland Mayoralty. It was because I had only listed the major candidates, not all 15 of them. Basically, a guy who had a miniscule chance of winning (he was not in the top eight finishers) was complaining that he had not been included in his own right.
Now with polls on voting, there are three ways broadly you ask the voting question.
- Unprompted. You don’t even give them the options. You just ask who they will vote for. This works well for parliamentary polls about parties as most people know the parties and have a preference. It is not very effective for local government polls where many don’t even know who is standing.
- Prompt major candidates and include another option. It is often clear who the major candidates are. Clients basically just want to find out the relative support amongst those in with a shot. This is absolutely normal in political polling. Do you think many US pollsters ask people if they are voting for Chase Oliver or Randall Terry for President?
- Provide a list of all candidates. This is feasible for online panels but for phone polls it just isn’t practical to read out 15 or 20 names. You just end up with many more undecideds.
Now this particular complaint was not upheld, but I had to spend considerable time justifying why we hadn’t included a candidate who had no chance of winning. I was surprised the complaint was even accepted as substantive.
The second complaint was about a survey of academics done by the Free Speech Union, which I analyse for them. Now I am quite happy to say there were things I could have done better with that poll. Curia is far from perfect.
A lot of the issue was whether a response rate should have been included and whether it was a probalistic sample or not. My practice is to include response rates for polls of communities where the client has a relationship with them. So, when I poll members of an industry group, I give a response rate. For polls which are regarded as probabilistic you do a margin of error. As the poll of academics was not on behalf of the universities (they sadly had no interest in knowing the views of their staff on academic freedom) it was done by the FSU trawling e-mails off university websites and I saw it as an unsolicited poll, akin to a public poll.
Now I should have, in hindsight, included the response rate. It was sort of there implicitly, but it should have been explicit because the low response rate does mean it is difficult to know how well those who responded reflect all academics. Because many academics hate the FSU, they were arguably less likely to take part.
Also due to a miscommunication, I did not see the final report from the FSU and their report placed more reliance on the actual percentages than was justified. However they did have a cover note I insisted on noting there was a low response rate.
After this complaint was upheld, I reviewed all our templates against the Code of Practice and developed a new client policy stating explicitly that any information they release must be run by Curia in advance, and this policy is now automatically attached to quotes for new clients.
I also included extra details of methodology on our standard report template, and even go so far as to provide a paragraph summarising the methodology that clients are requested to include in media releases, with an explicit request for media to include the statement.
So what I am getting at is I took the finding very seriously. I made significant changes.
The next (third) complaint was about the wording of a poll on the Golden Mile. The wording was: “Do you think the Wellington City Council should commit to spending $139 million, (of which the outgoing Government pledged $71 million) on the Golden Mile project considering the blowout in the Town Hall renovation and growing Council debt?”
The Professional Standards Group found against it because it introduced two different concepts – the Town Hall blowout and the Golden Mile project. I understand their point, but I believe that is quite justifiable in this case– there is only one pot of money. All of politics is about trade-offs. If the Government is running a surplus you might support a spending project while you wouldn’t if they have huge deficits.
So I thought the question was fine, and the PSG didn’t. Its fine to ask about an imaginary outcome for merging TVNZ and Radio NZ together, but not to mention Council spending blowouts with relation to another major Council project.
. Here is a poll questions asked by another company:
After previously signing up to international commitments to protect vulnerable marine ecosystems, home to deep sea coral and sponges, the New Zealand government has this year lobbied for bottom trawling to continue in these sensitive areas.
Some warn that bottom trawling is known to destroy deep sea coral ecosystems, with New Zealand being the only country still using this method in the South Pacific.
You could argue that is leading as it doesn’t provide any context in support of bottom trawling. But no complaints about that.
The fourth complaint was from an individual who has complained literally scores of times on almost every single poll I have published in the last year. I have probably had over 200 e-mails from him. He is the obvious vexatious litigant, but I still had to go through the formal process of having one of his complaints assessed.
The final complaint was about this question earlier this year:
The UK health service (the NHS) has stopped the use of puberty blockers, which begin the gender transition process, for children under 16 as it deemed they are too young to consent. Do you support or oppose a similar ban in New Zealand on the use of puberty blockers for young people 16 or younger?
I have not seen the decision of the PSG yet but I am told they decided the question was biased. This staggers me as the decision by the NHS was not a different concept but directly relevant to the question. I carefully considered the decision on the Golden Mile complaint and concluded this was not at all similar.
The complainant said we could have chosen a country that hadn’t banned them. Now the UK decision on the back of the Cass review was major world news. It probably will lead to other countries changing their policies. To suggest it is not relevant just seems crazy to me.
One reason I was so surprised by the latest decision is that RANZ declined to uphold a complaint against a poll which said:
There is a proposal to sell off the Auckland Port operating business which is currently 100% owned by Auckland Council and Aucklanders, via a lease. How strongly do you support or oppose the sale of the port operations?”
Now this question is simply factually wrong. There is no sale. It is a lease. A sale transfers ownership. A lease does not. The question is obviously written to get more people to be opposed. Yet RANZ said it was absolutely fine. This is why I think there is a double standard. It might not be deliberate, but it is there.
Now I don’t think the company that did that poll should be sanctioned for it either. It isn’t totally egregious, and as I say I don’t think RANZ should be second guessing question design. But after having read that they said that was okay, I was pretty surprised to have them apparently rule the totally factually correct question on puberty blockers was unacceptable.
I could provide many more examples of what I see as inconsistent decisions, but my aim isn’t to get into tit for that. I wasn’t party to their discussions and I am sure that from their backgrounds they genuinely think there was a difference. Different people have different worldviews.
I have detailed these cases so I can’t be accused of hiding what the complaints have been, but also to stress than when a complaint has been upheld I take it very very seriously.
RANZ
I have no ill will towards RANZ. I have many friends in RANZ and there’s not a single person there I dislike (even if the same may not be true in reverse). The Officers have been very supportive iof me, and the PSG Complaints Officer has always been professional, despite the obvious divergence in views.
I wish them well, but I do have some genuine advice for their complaints procedure going forward, because it has been weaponised and there are many other companies that could be targeted. They may not have owners with the same stress levels as me, but I’d rather they don’t have to go through it.
- Make it easier to dispose of vexatious complaints
- Consider a limit for the maximum number of complaints you will accept from one person
- Change the process so that the PSG can meet with the member and discuss the issues, rather than merely deciding based on one written response. It was extremely frustrating to not be able to discuss the issues, the intentions, the interactions in detail with back and forth.
- I really think it is very dangerous to have RANZ/PSG become the arbiter of what is a good question, unless the question is so horribly biased. This is a subjective area and it has a chilling effect. You have to write hundreds of questions every few months, with the worry that because three or four people don’t like it, you can then suffer severe consequences. The consequences of bad questions should primarily be public scrutiny, and less impact on decision makers.
- If you are going to remain the arbiter of what is an acceptable question, then consider far more detailed guidance. You could do case studies at the AGM. You could even have an advisory service where a panel of fellows could provide feedback.
- The threshold for suspension or expulsion should be clearly defined and reserved for conduct that is along the lines of other professions, ie accountants who steal, lawyers who shoplift, doctors who kill, engineers whose buildings collapse. Disputes over question wording should not be at that level.
I will remain a supporter (not a member) of RANZ and the work they do for the industry. I will also strive to uphold best practice of their codes and the global industry codes.
Future Accountability
One benefit of RANZ membership was the code of practice and complaints procedure, that helped as quality assurance. So what will Curia do, without being a member of RANZ?
I plan to have a three stage process. They are:
- Discussion. If you have a question or concern about a poll, then talk to me. I’m happy to meet in person. I often take questions about methodology at the many conferences I speak to. I am speaking on radio next week about the challenge of polling Pacific communities. I am always looking for how to improve also.
- Complain to Curia. If you are unhappy with any discussion, then there will be the ability to send in a complaint about a Curia poll through our website. This models what the Press Council and BSA do, where you have to complain to the member first. If your complaint is not vexatious, I will respond on our website. I will allow you to respond and vice-versa up to a maximum three rounds. I may agree that some things could have been done better, or I may defend why I did it the way I did. People will be able to see the exchange on the same site as the results. I could even link the exchange to the results. This may also build up a useful FAQ over time, on issues such as why don’t you publish a response rate.
- Finally I am looking at a peer review ability. If you are really unhappy with Curia’s response, then I will ask an expert in polling (which is a specialised subset of market research) to do a peer review where they can critique the poll, and I will publish all peer reviews favourable and unfavourable. Now this will have a cost as the peer reviewer will be paid. As I will be saving around $1,000 a year in RANZ membership I will use that to pay for half the cost of any peer reviews, and the complainant the other half. This also means the peer reviewer is not paid solely by me, making them more independent. It will also mean that people will only go to this level if they really feel strongly I have it wrong.
This isn’t the same as the RANZ procedure, but I do think it will demonstrate my commitment to best practice and transparency. In one way it will be more transparent as all complaints will be published, not just those upheld.
I do a bit of work in other countries, and may also look at whether I can join an overseas industry group, and be subject to their code of practice and complaints procedure. Also if RANZ changes their complaints procedure, then I would even look at rejoining, if they thought it was desirable (which they probably won’t as I imagine I cost them a huge amount of time and money dealing with all this).
I hope this post fully explains my decision.