The Fine Art of the Loaded Question--by Social Weather Stations

Or, How a Survey Question Becomes Unscientific

To be called "scientific" a public opinion poll should have all scientific parts: (1) the random sampling technique must be used to select the respondents, no matter how few or many; (2) valid logic and reasoning must be used to arrive at the generalization from the random sample to the total population that will become headlines in the media; and (3) the survey question itself must be scientific. Huh? What does it mean for a survey question to be "scientific"? It must be well posed, unambiguously answerable, ideally with a Yes or No, with few if any modifying clauses, adjectives or adverbs that may introduce bias or color to the respondents' appreciation of the question being asked.

A little understood aspect of SWS and Pulse Asia public opinion polls is the design and exact phrasing of the survey questions that are presented to the random sample of respondents. Generally speaking, the subject matter of survey questions are of two types: those that will be borne out by subsequent events, like an election, plebiscite or other objective, independent process (like measureable market share or popularity), and those that won't be put to such a test. Check out my previous post, Ponnuru's Puzzle, for an example on abortion polling showing how one word can drastically change a survey result. Most people do not realize that the generalizations to the entire population made by the pollsters and the Mass Media into headlines after the survey data is analysed, are based purely and rhetorically on the survey questions asked.

Third Quarter 2006 Social Weather Survey: General Vote on Cha-Cha still 67% "NO"; Opposition to Specific Amendments Grows

The Social Weather Stations Media Bureau Headline above was echoed by the Mainstream media today. It was based the following survey question in the SWS Third Quarter questionnaire presented to 1200 respondents:

In English: If a plebiscite is held today, would you vote FOR or AGAINST the Constitution that PGMA wants?

In Pilipino: Kung gaganapin po ngayon ang plebisito para sa pag-aapruba ng isang panukalang bagong Konstitusyon na gusto ni Pangulong Arroyo, kayo po ba ay boboto ng OO SA BAGONG KONSTITUSYON NA GUSTO NG PANGULO o kayo po ba ay boboto ng HINDI SA BAGONG KONSTITUSYON NA GUSTO NG PANGULO?

Please notice that in the Pilipino version of the question, the phrase "na gusto ng pangulo" ("that PGMA wants") appears three times and is ALL CAPS in the original SWS questionnaire.

I think I have to agree with Administration Congressman Constantino Jaraula that there is something wrong with this survey question. It is what I call a "loaded question."

The emphasis upon and repetition of the phrase "that the President wants" is what loads this loaded question with a sly suggestion--As Rep. Jaraula pointed out--that ONLY the President supports the proposed new charter and because of that the President must want it for some selfish, illegitimate or hidden reason. Else why don't others want it too? The proposed Constitution is not described in any other way than that the President wants it. In effect, the Survey Question as most of the respondents probably took it to mean, would read as follows:

If a plebiscite is held today, would you vote FOR or AGAINST a Constitution just because PGMA wants it?

I'm surprised only 67% disapprove of such a dastardly and monstrous proposed Constitution "that the President wants."

Was this a "scientific survey question"? NO. Because it will never be put to the test of an actual plebiscite or election. It can never be proven to be RIGHT or WRONG because we are not having a plebiscite today, nor any time in the future so far as any of the respondents knew at the time they answered this purely hypothetical but loaded survey question.

Of course the headline is not implausible, given the highly charged political atmosphere over chacha. Although it is a loaded question, its subject matter is not obscure and many people have taken definite positions on it. The single digit undecided percentage on this question is evidence of that. Never mind if it's worthless as scientific DATA. All that matters is that it makes for wonderful propaganda, or entertainment. If you think about it, this matter of not being able to ever find out if SWS was right or wrong on this, if its 67% or 51% that would vote down chacha, --is the same thing that happens with the cheapest call-in poll on AM radio, which are usually derided as "unscientific". Et tu, SWS?

Nothing makes it more crystal clear that Public Opinion Polling is Genre of Journalism.

Whenever the pollsters trot out the results of their surveys on such questions, they are actually trading on their own good reputation and credibility from when they do the good scientific work they are capable of.

As an example of the latter, here is the 2007 Senate Horse Race culled from recent surveys of Pulse Asia, Inc. which will be validated by the 2007 Midterm elections in May next year.

Rank	July, 2006	% of Voters	Nov, 2006	% of Voters
1	Loren Legarda	48.6%	Loren Legarda	52.9%
2	Kiko Pangilinan	39.0%	Ping Lacson	41.1%
3	Ping Lacson	34.9%	Kiko Pangilinan	36.6%
4	Manny Villar	34.2%	Manny Villar	31.4%
5	Ralph Recto	33.1%	Alan Cayetano	30.5%
6	Tito Sotto	31.0%	Tito Sotto	30.3%
7	Koko Pimentel	29.9%	Ralph Recto	28.7%
8	Alan Cayetano	29.5%	Korina Sanchez	27.0%
9	Gringo Honasan	27.7%	Koko Pimentel	24.8%
10	JV Ejercito	23.8%	Gringo Honasan	24.1%
11	Imee Marcos	23.1%	JV Ejercito	22.1%
12	John Osmena	22.7%	Ed Angara	20.3%
13	NoyNoy Aquino	21.8%	Noynoy Aquino	19.3%
14	Loi Estrada	21.7%	Chiz Escudero	19.3%
15	Joker Arroyo	21.0%	John Osmena	17.8%
16	Ed Angara	20.9%	Rufino Biazon	17.8%
17	Chiz Escudero	20.2%	Imee Marcos	17.6%
18	Kit Tatad	18.9%	Joker Arroyo	17.0%
19	Mike Defensor	18.7%	Loi Estrada	16.9%
20	Rufino Biazon	18.5%	Mike Defensor	15.7%