The University of Auckland report for the Ministry of Transport seeks to identify public support for new methods of financing transport infrastructure. Fuel excise and road user charges will apparently no longer be viable ways to fund transport. The reason is that electric cars will replace petrol and diesel.
But while the Herald reported on the work as though it were an opinion poll, it really was something different.
The researchers used Twitter, LinkedIn, and transport blogs to invite “transport stakeholders” to take part in a novel online conversation and consensus building tool.
It was not a poll of any representative sample of Kiwis. It was a new way of running online focus groups. Participants could put their own views into the mix while approving or disapproving of other participants’ statements.
The conversation included only 436 respondents, which is a small sample. And it was certainly not representative. The researchers admitted it was biased towards cycling advocates.
If you ask cycling advocates about their preferred transport policies, they are hardly likely to favour motorways.
And where advocates write their own statements, don’t expect unbiased wording.
Consider a statement that 88% of participants agreed with: “Use innovative mechanisms such as congestion charges and pollution pricing to encourage people out of cars and into other forms of environmentally sustainable transport.”
Hardly anyone would prefer using outdated mechanisms over “innovative” ones. Similarly, would anyone favour environmentally unsustainable transport? There were many such examples – and who would expect otherwise, given the method?
In fairness, the researchers clarified that this was a pilot study to test their fancy new conversation tool. It allowed them to find ‘clusters’ of beliefs among participants. But those clusters might simply be the different groups of activists drawn to the survey at different times.
One lesson might be that these kinds of focus groups need more structure. Activists proposed and supported alternative transport funding mechanisms that make little sense. It is hard to see the point in finding out that a high percentage of self-selected activist participants support odd policies.
The cost of the report is unknown, but whatever was spent, was wasted – at least as far as transport research is concerned. It yielded no results that would have been of value to the Ministry of Transport.
It is non-reports like this that give external consulting a bad reputation.
Dr Oliver Hartwich is the Executive Director of The New Zealand Initiative think tank. This article was first published HERE.
It was not a poll of any representative sample of Kiwis. It was a new way of running online focus groups. Participants could put their own views into the mix while approving or disapproving of other participants’ statements.
The conversation included only 436 respondents, which is a small sample. And it was certainly not representative. The researchers admitted it was biased towards cycling advocates.
If you ask cycling advocates about their preferred transport policies, they are hardly likely to favour motorways.
And where advocates write their own statements, don’t expect unbiased wording.
Consider a statement that 88% of participants agreed with: “Use innovative mechanisms such as congestion charges and pollution pricing to encourage people out of cars and into other forms of environmentally sustainable transport.”
Hardly anyone would prefer using outdated mechanisms over “innovative” ones. Similarly, would anyone favour environmentally unsustainable transport? There were many such examples – and who would expect otherwise, given the method?
In fairness, the researchers clarified that this was a pilot study to test their fancy new conversation tool. It allowed them to find ‘clusters’ of beliefs among participants. But those clusters might simply be the different groups of activists drawn to the survey at different times.
One lesson might be that these kinds of focus groups need more structure. Activists proposed and supported alternative transport funding mechanisms that make little sense. It is hard to see the point in finding out that a high percentage of self-selected activist participants support odd policies.
The cost of the report is unknown, but whatever was spent, was wasted – at least as far as transport research is concerned. It yielded no results that would have been of value to the Ministry of Transport.
It is non-reports like this that give external consulting a bad reputation.
Dr Oliver Hartwich is the Executive Director of The New Zealand Initiative think tank. This article was first published HERE.
4 comments:
Another example of couching questions to achieve the desired outcome.
The fact that the cycling advocates were biased away from motorways (not surprising in Auckland) is not in itself surprising.
One would get a similar biased response from muscle drivers.
Would be interesting to see the results of a similar survey in rural and small town New Zealand, but with a much larger sample.
Quite right Oliver. One could just as easily, and certainly more cheaply, posed those same questions to Chatgpt and received much the same response - most certainly if framed from a perspective of the suspected party(ies) answering.
But really, 436 respondents and even then far from a diverse random sample - what a complete waste of time and money. Naturally, commissioned by some faceless clown, or cluster of them, using others money. Why is it not
surprising who commissioned it?
Michael Wood is the minister of transport every thing he gets reports on he ends up scrapping it after much expense. He is as thick as two planks!
It's important to remember that this was not a survey. The report shows the results of using a new opinion formation platform. If UoA is trying to propose policy formulation based on this, their hod of statistics should be asked to affirm such a blatant abuse of this discipline.
Post a Comment