Thursday, 24 September 2009

Should PoliticsHome suspend its insider panel? A question of methodology

The PoliticsHome website yesterday faced the challenge for any media outlet when it becomes the story: can it remain objective, or does defending the organisation's interests and reputation take priority?

Editor Freddie Sayers set out a "business as usual" approach to restoring the trust lost by the controversial takeover in which Conservative party deputy chairman Lord Ashcroft has taken a majority stake in the site.

But this demonstrated a serious methodological question over ploughing on with the "insider panel" - yesterday asking about the Ashcroft question itself. The results on the past neutrality of the site were very good; the results on future confidence, though breezily presented, were very mixed.

Even so, only 42% of those still participating were confident that there will be no post-Ashcroft change in editorial line, with 21% believing editorial will be affected and fully 37% not sure. That total is boosted by 78% of Conservatives (21 out of 27) having such confidence. Of the 24 Labour-identifying panellists who remained still, only four expressed confidence in no editorial influence while twenty can not express that confidence at this stage. Seven believe that the editorial line will change; and thirteen don't know. Only 2 of the 6 LibDems could express confidence that the editorial line would not change: the other four were in "Don't Know" mode; and only one in five of the non-aligned participants expressed confidence, with 60% being don't knows.

So there is a significant divide - with low trust among on the left, a sceptical centre and a sanguine right. And, of course, the confidence numbers were affected (and boosted) by up to 20 resignations (around 10% of the panel), giving 10-20 less responses than usual.

I would like to make a rather boring point of methodology. This doesn't matter if the panel is "just a bit of fun" like a poll of SkyNews viewers or Sun/Guardian readers, though that might mean that there was little point in it. But they are questions which need to be addressed were the "insider panel" to continue to be marketed as a credible, balanced and reportable snapshot of 'what the political class' thinks?

The panel's continued veracity and credibility depends on the remaining voices from the centre-left being credible as a broadly representative sample across centre-left opinion.

Having a similar number of left and right participants is not enough to achieve this. It may be necessary, but it is far from sufficient.

This is true if there is no significant difference (on ideology, strategy or policy issues,) between people who would regard it as difficult or impossible to continue to participate in the panel and those who would have no problem doing so.

But it is false if that is not true, should controversy among significant numbers of people or segments of opinion as to whether they could or could not be involved create a "sampling bias". (This was almost certainly the case with regard to the questions the panel were asked yesterday about Ashcroft).

For broader issues, that can't yet be proved either way - but there are good reasons to ask the question.

But why could this be the case?

1. What if views about the ownership of PoliticsHome - and Andrew Rawnsley and Nick Assinder's decisions to resign - were correlated with major political questions which the panel might be asked?

Could there, for example, be a significant correlation with views about state funding of political parties, and/or with other constitutional reform or 'cleaning up' politics questions more generally, or perhaps media issues, such as the role of the internet, or the libel laws? It seems to me quite intuitively plausible that there could be such a link, perhaps particularly about party funding.

This might extend to other "cleaning up politics" and reform issues. To take the major open issue dividing the Cabinet and Labour party, the resigning group strike me as distinctive in containing many people who do not just support electoral reform, but give it a high political priority (with Tom Harris an honourable exception). It reads like a roll call of the Vote for a Change supporters. So a poll on "should Gordon Brown announce an electoral reform referendum" could well, I think, give significantly different read-out of left opinion next week than it would otherwise have done.

2. What if there were differences from different 'wings' of the Labour Party. Might an "Ashcroft controversy" make it more difficult to get sufficient 'left' voices from the more Compass soft left or Labour Representation Committee wing of the party?

So far, the evidence might be against this thought. The initial group of resigners includes Tom Harris, Denis MacShane, Matthew Taylor and Charles Clarke who are all strong advocates of New Labour, soggy centrists such as myself and leftist voices such as Neal Lawson. (Though several of them may share a non-membership of the Gordon Brown fan club).

However, the question arises as to whether there might be any methodological bias if, for example (to make up an example), Dianne Abbott of the Campaign Group proved to be willing to contribute but, say, Chris Mullin or Michael Meacher from the left of the party were not, if that correlated with different views about some issues. (I have no idea whether any of these either do or would contribute: it is an illustrative example).

But, if the electoral reform problem were an issue, that would also affect broad questions of party strategy and party relationships - such as the approach to other parties, or to a hung parliament.

3.There might also be other 'sociological' biases: it is notable that several of those who were but are not now in the panel are either the main Guardian/Observer opinion formers (such as Andrew Rawnsley, Polly Toynbee and Nick Cohen) or those who are current or former Guardian/Observer staffers (including Rafael Behr, Observer leader writer, myself as an ex-Obs leader writer, Martin Bright and others). This no doubt reflects the high level of respect current and former colleagues have for the views of Martin Bright and Andrew Rawnsley, who encouraged or invited many of them to participate.

Other journalists - very notably Nick Assinder - have also resigned. And there might still be many Guardian/Observer voices still on the panel. I simply don't know. But, again, the question would arise as to whether either the media section or the left-liberal sections of the panel could represent media opinion with very limited participation from the most prominent of the two left-liberal media groups. (This might be correlated with 'concern about media ownership' or 'political reform' or it might not be).

4. There could be other effects. For example, would it matter if those on the right tended to be better known and more senior than those on the left? This could be the case if there was a significant 'refusenik' tendency on the left and among neutral media voices, but not on the right, as seems likely given yesterday's results.

Nobody doubts that PoliticsHome can continue to run large-scale public opinion surveys. (I am grateful to Oranjepan for providing a very plausible account of why the PHI5000 survey data might be among the most attractive reasons for Michael Ashcroft to be involved with the site, in answer to my Ashcroft mystery piece, which seems plausible given how much emphasis Michael Ashcroft paid to extensive private polling in the 2005 campaign, with enormous influence on the party, indeed perhaps excessive influence, in the view of Tim Montgomerie.

But I wonder whether they can continue with the Insider Panel. The professional approach may well be to suspend it for now while trying to put a credibly balanced panel back in place, or perhaps to simply drop it. (And I am not sure how central it is to its public offering of the site).

But how could these concerns be dealt with?

To test them academically, you would need to poll left participants against refuseniks. Short of that, I think you would need to ask a polling pointy-head from outside the organisation - a Peter Kellner type figure - to look at the robustness of the panel, and how to address these issues, or perhaps to get the British Polling Council or a similar body to scrutinise the issues.

To be clear, I do not think that the problem is not that editorial staff might deliberately "bias" or "fix" the panel to give particular results on major political controversies - like the Brown leadership or the Lisbon Treaty. But I think that there is obviously a risk that the pressure to maintain a sufficient Labour "quota" (as stressed in yesterday's poll report) might lead to these methodological questions being under-addressed or overlooked (as they were yesterday).

While these are partly issues of qualitative political judgement, I think the continued credibility would therefore depend on once again having senior independent external voices - with profiles similar to Martin Bright and Nick Assinder - with deep knowledge of party factions and strands of opinion in the different parties, with a good deal of editorial input so that they were willing to vouch publicly on those issues. Without that, I think the results carry little value.

Another possibility would be for PoliticsHome to attempt a significant rapprochement with its critics - for example, by revealing more about the board structure; setting out what guarantees of editorial independence were written into the deal, and once again putting in place organisational structures which credibly protect these.

But until that happens I do not think it has an Insider Panel from which it can credibly issue results - especially during the Labour Party conference next week - without issuing a health warning with the findings.

And I would argue too that how far it is seen to take these straight methodological questions seriously is now one early test of how far its independence, journalistic integrity and professionalism have survived under new ownership.

1 comment:

J said...


If petulant lefties want to take their ball home, that's their lookout.