Wednesday, October 25, 2006

Evolution of a Survey Headline: From Data to Dishonesty

pinion pollster Social Weather Stations asked a random sample of 1200 respondents the following question in its 3rd Quarter National Survey :

in English:
Recently the Sigaw ng Bayan and ULAP raised their petition to the Supreme Court. Do you have much trust, are unsure, or have little trust that the Supreme Court would make a fair decision as to whether or not to grant the petition of Sigaw ng Bayan and ULAP regarding people's initiative?
Now take a look at SWS asking it in Filipino:
Kamakailan ay inakyat ng Sigaw ng Bayan at ng ULAP ang kanilang petisyon sa Korte Suprema. Kayo po ba ay malaki ang tiwala, hindi sigurado, o maliit ang tiwala na ang Korte Suprema ay makapagbibigay ng makatarungang desisyon kung dapat o hindi dapat na pahintulutan ang petisyon ng Sigaw ng Bayan at ng ULAP ukol sa people's initiative
24% of respondents said "little trust".
13% said they had "much trust"
61% were "unsure".

But look at several HEADLINES announcing these results:

SWS (Raw Data Page):
62% unsure if SC will decide fairly on 'people's initiative' petition

SWS (Media Release on Main Home Page):
Third Quarter 2006 Social Weather Survey: 62% unsure if SC will decide fairly on 'people's initiative' petition

Philippine Daily Inquirer (Front Page Headline):
60% of Filipinos doubt SC fairness, says poll

Notice how the obvious and correct description (that a 60% majority of Filipinos are UNDECIDED about the case) was transmogrified, apparently for propaganda purposes, by both SWS and PDI in an obvious breach of some kind of Ethics. Also, the Media Release people at SWS are not the statisticians, that is fairly obvious.

The Loaded Fairness Question itself is problematic, as pointed out in yesterday's post. I note also that the English and Tagalog versions of the question are NOT exactly the same, but even leaving that out, my faith in SWS has taken a mighty big hit these past weeks. They have evolved into a genre of propaganda, or are manipulable in that direction.

The question itself is badly flawed in a number of respects:

The English and Tagalog versions are not exactly the same, and it is not just the fact that something is always lost in translation. The premises and presentation of the question are complex and convoluted, requiring some knowledge of many current events and issues that even the Justices of the Supreme Court themselves are unsure of.

In fine the question asks the respondents to make a prediction about how the Supreme Court will rule; whether it will rule fairly or on the merits; and on behalf of some obscure organizations that are meaningless acronyms to most people randomly selected.

No wonder the majority answered loud and clear UNDECIDED.

But that is the Macchiavellian thing about this question (or is it Orwellian?), the way the question was phrased gives the Media Release folks license to SAY the people are unsure IF the Supreme Court will rule fairly, yet NOT SAY that the data also means the people are unsure if the Supreme Court will rule UNFAIRLY. It's like announcing that a half-full glass is half-empty, as IF there were a difference!

This is self-justifying propaganda based on scientifically collected statistical data packaged to say something the data plainly does NOT.

It's intellectual dishonesty.

5 comments:

Dave Llorito said...

actually that sws survey says that 62 percent have no opinion at all about the supreme court on how it will act of the sigan ng bayan petition. it means that survey has no meaning whatsoever, no information value at all. now these pdi guys dont probably understand how to read the survey and they misinterpreted it as "doubt" re the supreme court. stupid really, but its unfair to the supreme court.

Dave Llorito said...

why in the first place did sws frmaed that question? do they anticipate that SC will not be fair? or will be totally fair? if that's the purpose that survey indeed is loaded with biases. and then when they found out that 62 percent have no answer, they issued a press release saying 6 of ten pinoys are "unsure" about the impartiality of the high court. its either that sws doesn’t know how to interpret its own survey or its simply too dense that it found nothing.

domingo said...

A "self-justifying propaganda" indeed.

So, if the manner in which SWS phrased the poll question is "loaded" and the analysis of the poll results can after all be "transmogrified," how sure are we now of the fairness, integrity or honesty of the individual poll takers tasked or hired (outsourced) to record the reply of every respondent in order to be regarded as "scientifically collected data"?

What safety nets are employed to guarantee that the respondent’s reply is indeed the same reply the poll taker recorded?

A respondent’s "Yes" might be "transmogrified" to become "No" when a "biased" or "dishonest" poll taker gets to record it.

Is there a trail poll takers leave behind we can follow to verify?

Rizalist said...

In this case, there is nothing wrong with the data (there usually isnt!). What is wrong is the interpretation and reporting, which I am saying is flies right in the face of the obvious: the people are UNSURE how the court will rule, therefore how can they even judge whether it is FAIR?

this is a case of bad question design and even worse reportage and interpretation for the public and the media.

john marzan said...

Recently the Sigaw ng Bayan and ULAP raised their petition to the Supreme Court. Do you have much trust, are unsure, or have little trust that the Supreme Court would make a fair decision as to whether or not to grant the petition of Sigaw ng Bayan and ULAP regarding people's initiative?

The answers are:

24% of respondents said "little trust".
13% said they had "much trust"
61% were "unsure".


the banners:

SWS (Raw Data Page):
62% unsure if SC will decide fairly on 'people's initiative' petition

SWS (Media Release on Main Home Page):
Third Quarter 2006 Social Weather Survey: 62% unsure if SC will decide fairly on 'people's initiative' petition

Philippine Daily Inquirer (Front Page Headline):
60% of Filipinos doubt SC fairness, says poll


sabi ni tito dean:

Notice how the obvious and correct description (that a 60% majority of Filipinos are UNDECIDED about the case)

i think you read it wrong DJB. the survey is not about the PI case. It's about the Supreme court and whether they trust the Supreme Court to give a fair decision (for or against) the PI.

24% of respondents said "little trust".
13% said they had "much trust"
61% were "unsure".

I think this should be the headline.

"Only 13% trust the SC to render a fair decision"

ang dami kasing undecideds eh. eniwey, ang pangit ng pagkakagawa ng survey.