"SWS" Survey: Fallacy of the Leading Question In Aid of Propaganda

(MP3) Before I tackle the main subject of this post, I want to first share with Philippine Commentary readers a very short, but precious audio recording I was able to make yesterday of President Corazon Aquino. Here is the opinion of one woman that matters to many of us: Cory Aquino, once more at Club Pilipino...from the heart!

Meanwhile, public opinion pollster, Social Weather Stations reports on its March 2006 Survey, Tests of Political Messages of the Administration, and explains the origin of survey data quoted in newspaper advertisements placed by the Philippine Information Agency (official Palace propaganda bureau) in two major broadsheets --

SWS: "The March 2006 Social Weather Survey included items privately commissioned by Mr. Pedro R. Laylo Jr. to test public reactions to political messages, which were the basis for the advertisements by the Philippine Information Agency entitled "Let The Numbers Speak: It's Time To Move Forward" on April 23 in the Philippine Star (p. 17) and Philippine Daily Inquirer (p. A17)."

THE WORST IS OVER? The latter-named newspaper yesterday, published an Agence France Presse dispatch, trumpeting the results of this privately designed, privately commissioned module of the March 2006 SWS survey, as showing that "Worst may be over for Arroyo -- poll" and went on to quote the "results" of the module survey, without making its provenance clear, which to its credit, SWS does:

SWS: The quarterly Social Weather Surveys are omnibus surveys that accept specific questions commissioned by sponsors; such results can be kept confidential for up to three years. Once findings are released by a sponsor, the confidentiality provision no longer applies, and SWS is at liberty to report the following: identification of the sponsor; dates of interviewing; method of obtaining the interviews; population that was sampled; size and description of the sample; complete wording of the questions upon which the release is based, and; the percentages upon which the release are based."

From a professional journalistic standpoint, I think this is a case where a news reporter has not made the "source" of the claimed "news" clear. Though it was the SWS that conducted the survey, SWS did not design the questions in the module -- since it was commissioned by the Philippine Information Agency.

Anyone with more than a passing interest in public opinion surveys, knows as a matter of pure scientific literacy, that quality statistical question design is all-important to the validity, merit and usefulness of a scientific public opinion poll. Optimally, survey questions are simple, easy to understand, and ideally they ought to be answerable directly with a Yes or No. They really should contain no argumentative features or patently leading or suggestive elements.

Pollsters probing public opinion on complex political and social issues often employ the technique of asking the survey respondents whether they agree, disagree or have no opinion on a series of statements. There is however, a common practice that largely renders worthless such surveys. That is when the statements that the respondents are asked to examine are actually in the form of leading or suggestive statments that even before SWS collects the data, the private commissioner of the survey already knows how the data will more or less emerge. Which is of course why, in this case, the survey results did end up in full page newspaper ads.

(1) The members of the opposition against PGMA should start helping to improve the country and stop too much politics.

This is as clever as asking, "So, have you stopped beating your wife?" To disagree with the statement is to disagree with the stopping of too much politics and with starting to help improve the country. The premise is an accusation that naturally leads to the desired agreement from the majority of decent respondents.

(2) Mining companies should be given a chance to show that it is possible to protect the environment even as they offer the needed jobs to the country.

Who would oppose "needed jobs" or the chance for mean, evil capitalistic firms to protect the environment?

(3) Whatever happend in the elections is over and it is time to move on and let the President focus on the real problems of the nation.

Implying of course that if you were to have the temerity to disagree with this statment, it means you don't want to focus on the "real" problems of the nation. (You disagreeable scoundrel!)

(4) PGMA has the right plan for the nation and the economy but it is not moving fast enough as expected by the average citizen.

Subtle: if you are an average citizen, and that is what "random samples" usually end up with, by virtue of the Central Limit Theorem in elementary statistics, then you really ought to agree with this statement right? Otherwise you would be abnormal wouln't you?

These are magnificent examples of the tricky art of designing public opinion survey questions. They are all LEADING or LOADED QUESTIONS. The reported results of the survey. have dubious scientific merit on their face.

The tip-off to such low-quality, subtly biased surveys is usually an unexpectedly high percentage of "UNDECIDED" responses to the survey questions. And sure enough, this module's results display this very signature of Bad Housekeeping -- in some of the questions the undecideds were bigger than 20%. Such quantities of undecided are quite often the sign of questions that the respondents don't understand, don't agree with as posed, don't know very much about, are confused about the premises, etc. Large undecideds usually proclaim suspicious question design in a public opinion survey.

One must not underestimate the huge importance of question design. Last year I had the opportunity to respond to a "BLEG" from Ramesh Ponnuru (National Review Online) and ended up analyzing the results of two polls on abortion in the United States which had completely opposite survey results yet the questions they asked were so similar it was a real Ponnuru's Puzzle: Gallup Poll asked respondents if they thought abortion ought to be legal in all cases" while ABC/Washington Post asked if they thought abortion ought to be legal in all "circumstances." Both were "statistical" proof that Americans were pro-abortion AND anti-abortion.

Lately I've noted a trend in both Pulse Asia and SWS to design questions that expect too much of the surveying process. They present to respondents some scenario of choices and ask them for their preferrence in a sequence of events and circumstances that are complex and can't possibly be comprehended in more or less the same way by all of the 1200 randomly selected sample of voters.

We should remember that surveys are most accurate when they ask questions that all the respondents can easily and automatically answer, almost without thinking, like: Who did you just vote for? (exit poll) , or Who will you vote for? (pre-election survey).

But if you ask them if they agree with some statements, that all ask whether one agrees with motherhood and apple pie, or some approximation or glimmer of them, including hints and recipes for achieving utopia, one is likely to get a predictably loaded, but utterly worthless survey result.

Since the point of the exercise was for PIA to be able to put out the survey data as FULL PAGE COMMERCIALS for the Palace, it is understandable that they got an associate of the SWS (Mr. Laylo) to write up questions they approved. Or maybe they just handed them to him directly.

RELATED POSTS:

How the Surveys Have Lost Their Sting

Public Opinion Polling as a Genre of Journalism

To the latter we must now also add that Public Opinion Polling can also be a genre of PAID ADVERTISING.

Speaking of which, the attendees at last week's Free Expression in Asian Cyberspace were treated to the heady insights of J.V. Rufino, a head honcho of INQ7 -- one of the leading news websites in the Philippines, upon which I rely heavily for hard, documented news information. He was telling of the perpetual battle he has with the marketing and advertising department of their website -- the same basic scramble for space (we call it "bandwidth") as occurs in the printed newspapers. Of course the dirty lil secret about the present-day Main Stream Media in the Philippines is that most of the major outlets in print, broadcast and online media are utterly beholden to two giant advertisers: the telecomm cartel's SMART and GLOBE cellphone service providers. These guys regularly buy full color multi page spreads touting the latest mobile phone devices, accessories and services. At broadsheets like PDI, they account for up to 75% or more of advertising revenues. Think we have no sacred cows in the Media. Why for example has Congressman Alan Peter Cayetano been like a voice crying in the wilderness about the alleged FREE 3G LICENSES granted to the carted by the "government? Everywhere else, billions of dollars in license fees were earned from such firms in Europe and Asia. But is it true Globe and Smart got theirs free and are about to turn the whole atmosphere into their own private telecomms microwave oven? I don't really know because the Media hardly ever touches stuff like that in their news or editorial items. I would be more impressed with newspapers and other media if they published not just who their owners are, but who their biggest ADVERTISERS are.