What Makes a Survey Scientific?

There are two kinds of public opinion poll or survey: the statistically SCIENTIFIC poll and the statistically ENTERTAINING poll. To explain the difference requires a number of preliminary considerations...

First the following important distinction. Public opinion surveys can be divided quite neatly into two kinds--(1) those surveys that make definite predictions about how some FUTURE event will turn out, and (2) those that don't. In both types of survey, typically 1200 randomly selected respondents are asked to fill in a multiple choice questionnaire, but as the following representative examples will show, only the first type of surveys lead to testable predictions of the survey results and pollsters interpretation of them. The second type of survey does not make testable predictions about future events but only produces "snapshots" of how the respondents answer some arbitrary survey question. These questions are often commissioned by clients of the survey outfits and scientifically polled and analysed by them, but which are never tested by some independent objective EVENT that will either verify or confirm the scientific validity of the survey, its relevance, materiality or utility.

An example of the first kind of public opinion survey are the periodic voter preference surveys conducted by both survey outfits in the run-up to national elections. Based upon the series of random samples that the pollsters ask to see how the candidates are doing with the voters before the elections, trends may be discerned and pollsters can predict how the elections will actually turn out. Thus the scientific integrity of the survey is tested on all critical technical aspects by the actual results of the elections such as (1) the question design; (2) the random sample data acquisition procedure; (3) the data analysis, interpretation and prediction of likely winners and losers. Generally speaking, the work of SWS and Pulse Asia on this type of survey has been outstanding and world class. In almost all cases their predictions of how elections eventually turn out have been within the bounds of expected statistical variation based on their typical respondent sample sizes and resulting margin of error. In my judgment the voter preference surveys trending upcoming elections in the Philippines conducted by SWS and Pulse are statistically SCIENTIFIC.

Another very good example of statistically SCIENTIFIC polls are consumer product preference surveys conducted by marketing research outfits on behalf of clients who are launching new products or new packaging of old products and who are testing various alternatives and options. These surveys are also validated by how well they do against actual results after product launch.

What about the second type of public opinion polls that don't make testable predictions and are only statistically ENTERTAINING? Which surveys of SWS and Pulse Asia involve questions to respondents that don't lead to trends and predictions and therefore are never tested by some independent, objective and decisive event?

In the case of SWS the most famous of these are the SELF-RATED POVERTY and HUNGER surveys, which make for great big banner headlines about every three months screaming that hunger is at "peak record levels" and poverty incidence is causing little girls to commit suicide. But of course none of these numbers can ever be challenged or tested in any effective or automatic way like voter preference surveys which have the elections to keep them credible. In the case of Pulse Asia, the recent example is the survey on who is the most corrupt President (cleverly and craftily designed and commissioned by Senator Serge Osmena) which turned up Mrs. Arroyo as being even more corrupt than Ferdinand Marcos.

Although this second category of surveys are scientifically conducted, they are not proper scientific statistical surveys, in my humble opinion, because they do not present a TESTABLE HYPOTHESIS that the subsequently collected data is supposed to affirm or deny. They may be professionally conducted by SWS and Pulse, but any conclusions, predicitons or interpretations of such data are merely STATISTICALLY ENTERTAINING since there is no future even to test their validity.

Objective, independent testability of a survey's results, predictions and analysis are what make it SCIENTIFIC.

Although science can be ENTERTAINING, don't be fooled when SWS or Pulse conducts and reports on the second type of survey. They are just paying the bills, not advancing science. In this sense, Public Opinion Polling as such really IS a genre of Journalism--it is a mix of hard data (news), opinion and editorializing by the pollsters designers and commissioners (views) and entertainment based on fun with numbers and questionnaires.

Over at the Philippine Daily Innuendo, which through the years has largely practiced and promoted the use of surveys as a genre of propaganda and innumeracy, there is at least John Nery (Newsstand blog) a PDI senior editor who has done a good deal to correct the situation by taking the trouble to understand Statistics 101 and actually attempt to apply it to real surveys. I hope he will continue to encourage his colleagues in journalism to study math and statistics too. Please note that John Nery's Five Rules of Thumb for understanding surveys refers ONLY to election surveys.

Surveys that are tested may be trusted. Those that are not tested by some external event like an election are really NOT scientific surveys at all, but they do trade upon the reputation of the pollster. Such surveys do have some validity in the sense that they are the result of random sampling techniques. However, I think that survey questions which will not be put to the test in a real objective event have mainly JOURNALISTIC and not scientific value.

---

Manuel L. Quezon III turns in a thoughtful lamentation on the matter of alleged public apathy and indifference starting with some choice quotes from Ninoy Aquino. I say alleged because he may really be referring to the apparent inability of the fiercely anti-GMA opposition to scare up even the shadow of an Edsa Dos style regime change. Having used that trick to get into power, GMA sees any attempt at flattery-by-imitation a mile away and nips it in the bud. Despite one devastating scandal and controversy after another hounding the President, the magical combination of civil society outrage, military withdrawal of support and a judiciary willing to savage the Constitution has so far eluded the Edsa-envious and the scattered Opposition. Compared to Erap's kanto boy plunder of jueteng, GMA is possibly guilty of big league crimes like political murders and enforced disappearances, suppression of human rights and Press Freedom. No less than UN Human Rights Rapporteur Philip Alston has turned in a largely negative Final Report on her administration on the extrajudicial killings issue. The President countered brilliantly by getting King Juan Carlos of Spain to give her a Gold Medal for Human Rights, trading largely on her abolition of the death penalty in 2006.

So far, the Public has tolerated Arroyo, who is entering eight years in power, even if Pulse Asia says they think she is their most corrupt leader of the last five. But her enemies have simply never gotten it together like Erap's enemies once did. Having survived so much already, GMA seems to have turned into pure teflon-on-steel, untouchable and immutable, treating all opposition news, views and rebellions as political noise to be ignored or suppressed. Or ruthlessly exploited for political and propaganda advantage as in the Manila Peninsula incident when Trillanes and the Mass Media played a lethal Game of Chicken with the Authorities, but blinked.

How has GMA gotten away with so much and for so long? It may have to do with the problem of the alternative. Consider that the "Civilian Component" of the Manila Peninsula Rebellion of Sonny Trillanes, such as it was, included the likes of Linggoy Alcuaz, Mentong Laurel, Dodong Nemenzo, Bishop Julio Labayen, Tito Guingona. Together with the CPP national united front organization Kapisanan ng Makabayaang Ekonomista, these same individuals had been calling for the immediate resignation of the President, Veep, Senate President and House Speaker to make way for a transition government to be headed by Chief Justice Reynato Puno. Not even snap elections, the bums! Puno reportedly issued a statement saying he is "neutral" on the proposed Treason, instead of roundly denouncing it as another illegal Regime Change a la Davide 2001. But Puno should set his sights higher--on Geneva and the international justice scene. Those human rights Rules of Court under his watch are absolutely brilliant. Writ of amparo and habeas data by Puno should trump even Davide's judicial coup d'etat on Erap and the Separation of Powers.

There is another reason I say "alleged public apathy" -- I don't think the Filipino people have been sitting around just taking it from their Failed Ruling Classes -- past, present and future. They've been busy as OFWs (what the Sour Grapers call "toilet bowl cleaners of the world") raking in and sending home the over ONE BILLION DOLLARS A MONTH which is the rising tide raising all boats in this God-forsaken archipelago. We are care-givers to humanity and the world. We are not indolent apathetics. And as we send back and return the fruits of our labor to our families and children, whom we have not at all abandoned or forgotten, as the Sour Grapers grouse, the spell is broken, the voices of political correctness and social authority lose their omniscience, lose their ability to abash us. The solutions to our own problems lie in our own hands, and not of saviors, or leaders with vain ideologies.

MLQ3 renews a favorite pitch of his (and mine) for "a common political vocabulary"--which I locate in the Constitution, and urge upon him and other utilitarians the greater appreciation for the Social Contract tradition of interpreting constitutions, citizen's and institution's rights, duties and privileges.

MLQ3 ends with a quote from Inday Varona Espina's recent essay in Graphic Magazine

“The young agree about hell lying in wait. But they think the solution lies elsewhere: Collective tinkering with a wheezing vehicle, the dismantling of roadblocks and a more discerning choice of routes. These acts require sober dialogue, not the hissy fits coming from all shades of the political spectrum these days.” So we must ask ourselves, how well do we understand the engine we intend to tinker with?

I really like the dark, quirky, unpredictable column pieces that Ms. Inday Varona Espina has produced over the years as The Scarred Cat. But that was mainly before she became chairman of National Union of Journalists of the Philippines (NUJP) and editor of Philippine Graphic Magazine. (By the way, I once shared a double bunk bed with her predecessor, Luis Mauricio, editor of the same venerable publication in September, 1972, when Ferdinand Marcos arrested us both along with about a hundred others including Ninoy Aquino and Chino Roces from both the Mass Media and the Political Opposition, on the night Martial Law came down.)

Gloria Macapagal Arroyo has a ways to go in order to become Ferdinand Marcos, in my book, which also says on Page One: Eternal Vigilance is the Price of Freedom!