Monday, November 27, 2006

The Fine Art of the Loaded Question--by Social Weather Stations

Or, How a Survey Question Becomes Unscientific

To be called "scientific" a public opinion poll should have all scientific parts: (1) the random sampling technique must be used to select the respondents, no matter how few or many; (2) valid logic and reasoning must be used to arrive at the generalization from the random sample to the total population that will become headlines in the media; and (3) the survey question itself must be scientific. Huh? What does it mean for a survey question to be "scientific"? It must be well posed, unambiguously answerable, ideally with a Yes or No, with few if any modifying clauses, adjectives or adverbs that may introduce bias or color to the respondents' appreciation of the question being asked.

A little understood aspect of SWS and Pulse Asia public opinion polls is the design and exact phrasing of the survey questions that are presented to the random sample of respondents. Generally speaking, the subject matter of survey questions are of two types: those that will be borne out by subsequent events, like an election, plebiscite or other objective, independent process (like measureable market share or popularity), and those that won't be put to such a test. Check out my previous post, Ponnuru's Puzzle, for an example on abortion polling showing how one word can drastically change a survey result. Most people do not realize that the generalizations to the entire population made by the pollsters and the Mass Media into headlines after the survey data is analysed, are based purely and rhetorically on the survey questions asked.

Third Quarter 2006 Social Weather Survey: General Vote on Cha-Cha still 67% "NO"; Opposition to Specific Amendments Grows

The Social Weather Stations Media Bureau Headline above was echoed by the Mainstream media today. It was based the following survey question in the SWS Third Quarter questionnaire presented to 1200 respondents:
In English: If a plebiscite is held today, would you vote FOR or AGAINST the Constitution that PGMA wants?

In Pilipino: Kung gaganapin po ngayon ang plebisito para sa pag-aapruba ng isang panukalang bagong Konstitusyon na gusto ni Pangulong Arroyo, kayo po ba ay boboto ng OO SA BAGONG KONSTITUSYON NA GUSTO NG PANGULO o kayo po ba ay boboto ng HINDI SA BAGONG KONSTITUSYON NA GUSTO NG PANGULO?
Please notice that in the Pilipino version of the question, the phrase "na gusto ng pangulo" ("that PGMA wants") appears three times and is ALL CAPS in the original SWS questionnaire.

I think I have to agree with Administration Congressman Constantino Jaraula that there is something wrong with this survey question. It is what I call a "loaded question."

The emphasis upon and repetition of the phrase "that the President wants" is what loads this loaded question with a sly suggestion--As Rep. Jaraula pointed out--that ONLY the President supports the proposed new charter and because of that the President must want it for some selfish, illegitimate or hidden reason. Else why don't others want it too? The proposed Constitution is not described in any other way than that the President wants it. In effect, the Survey Question as most of the respondents probably took it to mean, would read as follows:

If a plebiscite is held today, would you vote FOR or AGAINST a Constitution just because PGMA wants it?

I'm surprised only 67% disapprove of such a dastardly and monstrous proposed Constitution "that the President wants."

Was this a "scientific survey question"? NO. Because it will never be put to the test of an actual plebiscite or election. It can never be proven to be RIGHT or WRONG because we are not having a plebiscite today, nor any time in the future so far as any of the respondents knew at the time they answered this purely hypothetical but loaded survey question.

Of course the headline is not implausible, given the highly charged political atmosphere over chacha. Although it is a loaded question, its subject matter is not obscure and many people have taken definite positions on it. The single digit undecided percentage on this question is evidence of that. Never mind if it's worthless as scientific DATA. All that matters is that it makes for wonderful propaganda, or entertainment. If you think about it, this matter of not being able to ever find out if SWS was right or wrong on this, if its 67% or 51% that would vote down chacha, --is the same thing that happens with the cheapest call-in poll on AM radio, which are usually derided as "unscientific". Et tu, SWS?

Nothing makes it more crystal clear that Public Opinion Polling is Genre of Journalism.

Whenever the pollsters trot out the results of their surveys on such questions, they are actually trading on their own good reputation and credibility from when they do the good scientific work they are capable of.

As an example of the latter, here is the 2007 Senate Horse Race culled from recent surveys of Pulse Asia, Inc. which will be validated by the 2007 Midterm elections in May next year.

Rank

July, 2006

% of Voters

Nov, 2006

% of Voters

1

Loren
Legarda

48.6%

Loren
Legarda

52.9%

2

Kiko Pangilinan

39.0%

Ping
Lacson

41.1%

3

Ping
Lacson

34.9%

Kiko Pangilinan

36.6%

4

Manny Villar

34.2%

Manny Villar

31.4%

5

Ralph Recto

33.1%

Alan
Cayetano

30.5%

6

Tito
Sotto

31.0%

Tito
Sotto

30.3%

7

Koko
Pimentel

29.9%

Ralph Recto

28.7%

8

Alan
Cayetano

29.5%

Korina Sanchez

27.0%

9

Gringo
Honasan

27.7%

Koko
Pimentel

24.8%

10

JV
Ejercito

23.8%

Gringo
Honasan

24.1%

11

Imee
Marcos

23.1%

JV
Ejercito

22.1%

12

John
Osmena

22.7%

Ed
Angara

20.3%

13

NoyNoy
Aquino

21.8%

Noynoy
Aquino

19.3%

14

Loi
Estrada

21.7%

Chiz
Escudero

19.3%

15

Joker Arroyo

21.0%

John
Osmena

17.8%

16

Ed
Angara

20.9%

Rufino
Biazon

17.8%

17

Chiz
Escudero

20.2%

Imee
Marcos

17.6%

18

Kit
Tatad

18.9%

Joker Arroyo

17.0%

19


Mike Defensor

18.7%

Loi
Estrada

16.9%

20

Rufino
Biazon

18.5%

Mike Defensor

15.7%

2 comments:

Dave Llorito said...

people have been asking mahar the same questions and im surprized he doenst seem to be interested in answering such valid questions.

Dave Llorito said...

try analysing the results and interpretations of his "self rated poverty" survey and you will find many interesting things.