Thursday, July 31, 2008

Pulse Asia SONA Survey Was Propaganda, Just Like the SONA

After looking at the survey questions, the methodology and the purported analytical results of the July 2008 Pulse Asia Survey poll on President Arroyo's State of the National Address, I am forced to the inexorable conclusion that it was basically propaganda, not a valid scientific public opinion poll. The main numerical data are contained in Tables 1, 2, 3, and 4.

To begin with, the pollster admits that only 60% of the 1200 respondents to the survey were qualified to participate and be tallied because only that fraction of them claimed to have "heard or read about any previous SONA." Thus the proper random sample size is 720 and the one-sided statistical error is closer to 4% than 3%. But this is hardly its biggest or most egregious flaw.

If you look under a magnifying glass at the bottom of Table 4 you will discover that the first question asked of each respondent was whether they had heard or read about any previous SONA ("Kayo ba ay nakarinig na, or nakabasa na, sa anumang nakaraang Ulat Sa Bayan o SONA ni Pangulong Arroyo?"). To those who answered YES, two further questions were asked soliciting an opinion about the truthfulness of the 2007 and the likely truthfulness of the 2008 SONAs. There is a further microscopic notation that "truthful" means "mostly or completely truthful" while "untruthful" means "mostly or completely untruthful."

This is a badly flawed question because the 2008 SONA, for example, has over 4600 words, and 500 sentences spread out over many dozens if not hundreds of possibly truthful or untruthful individual assertions by the President. But the follow-on question forces the respondent to decide whether he or she thinks they were (in a previous SONA) or would be (in 2008) "mostly or completely true or untrue." Well, no wonder 46% of the 720 qualified respondents said they were UNDECIDED about the likelihood of the 2008 SONA's truthfulness, while 52% were UNDECIDED about the actual truthfulness of the 2007 SONA.

It is a rule of thumb, at least among scientific, professional pollsters that when a question elicits such large percentage of UNDECIDED responses, there is something badly wrong about the question itself, either its design, content or meaning for the respondents. Among physical scientists this is equivalent to having half of one's data set being corrupted, unreadable or unusable because we are measuring the wrong parameter or using the wrong proble. We usually throw away such tests and their resulting data sets as being unreliable or positively useless.

What the analysis keys on, as presented here and by Ana Marie Tabunda on ANC last evening, was the large percentage (40% of 720 respondents) opining that the 2008 SONA would likely be untruthful while only 14% of 720 respondents opined it would likely be truthful. It is of course not surprising that 46% thought the 2007 SONA was untruthful while 13% thought it was truthful.

With respect to the 2008 SONA, this is numerically gussied up ASTROLOGY and not statistical opinion polling. With respect to "any previous SONA", I doubt very much that any of the respondents could've made a fair mental evaluation of the totality of the President's assertions. Rather, human nature being reflexively suspicious and skeptical towards politicians (especially one as opaque and plastique as Gloria Macapagal Arroyo, who truly deserves it, in my own opinion), there is a natural tendency to remember some unbelievable or untruthful statement and ascribe dishonesty to the entire speech, rather than to mentally maintain an inventory of a large number of possibly true statements. After all, when a person tells even one lie in such a forum, it's easy to believe such a person is telling many other lies.

Since the nature of the basic question asked of the respondents, requires of them to make a global judgment ("mostly truthful" or "mostly untruthful" but nothing in between) of some past or future SONA, and given that she has most verifiably lied on a number of very important and public occasions in her checkered political career, the results might've been predicted without all the trouble of a full blown statistical survey. After all, the respondents were qualified to answer these questions if they had merely read or heard about some SONA or other -- most likely from an habitually skeptical or even hostile media.

Nevertheless, two wrongs just don't make a right. This Pulse Asia Survey was what we call a setup or a "gimme" for anti-Arroyo forces. It is another example of the cynical use of science for propaganda purposes.

Deliciously malicious propaganda at that, since of course propaganda can itself be truthful or not, just like the SONAs themselves!

Tee-hee, but shame on you Pulse Asia! I bet if somebody did a survey on the surveys, the results would run against the pollsters with similar percentages being cynical and skeptical about them as propagandists.
UPDATES: Cogent discussion of the brewing bribery brouhaha in the Court of Appeals is to be found at La Vida Lawyer.

9 comments:

Anonymous said...

Susmaryosep, mali talaga yoong questionnaire... fer cryin' out loud the spits errr speech was 12 page long totaling 28K+ characters who would have the patience to go through all that and then remember the previous lies? The question should have been did you listen to the entire spiel err speech or read the entire text, then follow it up with a why with a fill in the blank answer.

Deany Bocobo said...

pogb, Ana Marie Tabunda and Pulse can only hurt the polling industry with this. It exposes the true nature of their outfit as a journalistic, and not a scientific enterprise. It was more like, astrology than statistics.

rc said...

DJB,

The same can be said of the Obamessiah polls happening in the US right now...by the US MSM who will stop at nothing to see him elected (see the NYT reponse to McCains response to the Obamessiah!). BTW, did you know Obama plans on being president for the next 10 years?!? WTF?!? (look the quote up on google) This dude doesn't even know how long the POTUS is allowed by the contitution to remain in office...or if he does, he doesn't care. Sound familiar? No, he's a 'post-racial patriot'..didn't think it would.

Richard

Deany Bocobo said...

Richard,
Messiah, or Rock Star? Gotta admit, the dude's on a roll man. Saw the recent ad comparing him to Britney Spears and Paris Hilton. Maybe the GOP shoulda gotten Mitt.

domingoarong said...

DJB

Nobody I think is competent (in the sense that one has all the data and information) to answer the questions the survey asked, if the truth sought meant the fact or the reality of every item (bridges or classrooms) Arroyo recited as her accomplishments in previous SONAs.

Under this survey, truth simply becomes subject to one’s opinion.

rc said...

DJB,

"Gotta admit, the dude's on a roll man."

Dude, not so much lately...

http://www.gallup.com/poll/109177/Gallup-Daily-Obama-45-McCain-44.aspx

Just like Britney and Paris....

rc said...

DJB,

Hate to pile on old boy, then there is this...even the adoring media has to admit it now...just like they had to do with Iraq:

http://latimesblogs.latimes.com/washington/2008/07/obama-mccain-po.html

Cheers,

Richard

EQ said...

What's the best reason for Gloria's all-time low popularity?

* research funded by Opposition
* research companies(SWS,Pulse Asia) do not conduct scientific research!
* it's just propaganda disguised as research!
* wrong sample base
* they didn't research and question the whole population(85 million people)

Anonymous said...

What has been the most defining moment of her Presidency?

* U.S. Junket with 59 congressmen
* Spanish Junket with 34 congressmen
* 2008 Nobel Peace Prize Nomination for the Erap pardon!
* Withdrawing our troops from Iraq!
* Her quick and decisive response to Typhoon Frank!