Tuesday, December 18, 2007

What Makes a Survey Scientific?

There are two kinds of public opinion poll or survey: the statistically SCIENTIFIC poll and the statistically ENTERTAINING poll. To explain the difference requires a number of preliminary considerations...

First the following important distinction. Public opinion surveys can be divided quite neatly into two kinds--(1) those surveys that make definite predictions about how some FUTURE event will turn out, and (2) those that don't. In both types of survey, typically 1200 randomly selected respondents are asked to fill in a multiple choice questionnaire, but as the following representative examples will show, only the first type of surveys lead to testable predictions of the survey results and pollsters interpretation of them. The second type of survey does not make testable predictions about future events but only produces "snapshots" of how the respondents answer some arbitrary survey question. These questions are often commissioned by clients of the survey outfits and scientifically polled and analysed by them, but which are never tested by some independent objective EVENT that will either verify or confirm the scientific validity of the survey, its relevance, materiality or utility.

An example of the first kind of public opinion survey are the periodic voter preference surveys conducted by both survey outfits in the run-up to national elections. Based upon the series of random samples that the pollsters ask to see how the candidates are doing with the voters before the elections, trends may be discerned and pollsters can predict how the elections will actually turn out. Thus the scientific integrity of the survey is tested on all critical technical aspects by the actual results of the elections such as (1) the question design; (2) the random sample data acquisition procedure; (3) the data analysis, interpretation and prediction of likely winners and losers. Generally speaking, the work of SWS and Pulse Asia on this type of survey has been outstanding and world class. In almost all cases their predictions of how elections eventually turn out have been within the bounds of expected statistical variation based on their typical respondent sample sizes and resulting margin of error. In my judgment the voter preference surveys trending upcoming elections in the Philippines conducted by SWS and Pulse are statistically SCIENTIFIC.

Another very good example of statistically SCIENTIFIC polls are consumer product preference surveys conducted by marketing research outfits on behalf of clients who are launching new products or new packaging of old products and who are testing various alternatives and options. These surveys are also validated by how well they do against actual results after product launch.

What about the second type of public opinion polls that don't make testable predictions and are only statistically ENTERTAINING? Which surveys of SWS and Pulse Asia involve questions to respondents that don't lead to trends and predictions and therefore are never tested by some independent, objective and decisive event?

In the case of SWS the most famous of these are the SELF-RATED POVERTY and HUNGER surveys, which make for great big banner headlines about every three months screaming that hunger is at "peak record levels" and poverty incidence is causing little girls to commit suicide. But of course none of these numbers can ever be challenged or tested in any effective or automatic way like voter preference surveys which have the elections to keep them credible. In the case of Pulse Asia, the recent example is the survey on who is the most corrupt President (cleverly and craftily designed and commissioned by Senator Serge Osmena) which turned up Mrs. Arroyo as being even more corrupt than Ferdinand Marcos.

Although this second category of surveys are scientifically conducted, they are not proper scientific statistical surveys, in my humble opinion, because they do not present a TESTABLE HYPOTHESIS that the subsequently collected data is supposed to affirm or deny. They may be professionally conducted by SWS and Pulse, but any conclusions, predicitons or interpretations of such data are merely STATISTICALLY ENTERTAINING since there is no future even to test their validity.

Objective, independent testability of a survey's results, predictions and analysis are what make it SCIENTIFIC.

Although science can be ENTERTAINING, don't be fooled when SWS or Pulse conducts and reports on the second type of survey. They are just paying the bills, not advancing science. In this sense, Public Opinion Polling as such really IS a genre of Journalism--it is a mix of hard data (news), opinion and editorializing by the pollsters designers and commissioners (views) and entertainment based on fun with numbers and questionnaires.

Over at the Philippine Daily Innuendo, which through the years has largely practiced and promoted the use of surveys as a genre of propaganda and innumeracy, there is at least John Nery (Newsstand blog) a PDI senior editor who has done a good deal to correct the situation by taking the trouble to understand Statistics 101 and actually attempt to apply it to real surveys. I hope he will continue to encourage his colleagues in journalism to study math and statistics too. Please note that John Nery's Five Rules of Thumb for understanding surveys refers ONLY to election surveys.

Surveys that are tested may be trusted. Those that are not tested by some external event like an election are really NOT scientific surveys at all, but they do trade upon the reputation of the pollster. Such surveys do have some validity in the sense that they are the result of random sampling techniques. However, I think that survey questions which will not be put to the test in a real objective event have mainly JOURNALISTIC and not scientific value.

Manuel L. Quezon III turns in a thoughtful lamentation on the matter of alleged public apathy and indifference starting with some choice quotes from Ninoy Aquino. I say alleged because he may really be referring to the apparent inability of the fiercely anti-GMA opposition to scare up even the shadow of an Edsa Dos style regime change. Having used that trick to get into power, GMA sees any attempt at flattery-by-imitation a mile away and nips it in the bud.
Despite one devastating scandal and controversy after another hounding the President, the magical combination of civil society outrage, military withdrawal of support and a judiciary willing to savage the Constitution has so far eluded the Edsa-envious and the scattered Opposition. Compared to Erap's kanto boy plunder of jueteng, GMA is possibly guilty of big league crimes like political murders and enforced disappearances, suppression of human rights and Press Freedom. No less than UN Human Rights Rapporteur Philip Alston has turned in a largely negative Final Report on her administration on the extrajudicial killings issue. The President countered brilliantly by getting King Juan Carlos of Spain to give her a Gold Medal for Human Rights, trading largely on her abolition of the death penalty in 2006.

So far, the Public has tolerated Arroyo, who is entering eight years in power, even if Pulse Asia says they think she is their most corrupt leader of the last five. But her enemies have simply never gotten it together like Erap's enemies once did. Having survived so much already, GMA seems to have turned into pure teflon-on-steel, untouchable and immutable, treating all opposition news, views and rebellions as political noise to be ignored or suppressed. Or ruthlessly exploited for political and propaganda advantage as in the Manila Peninsula incident when Trillanes and the Mass Media played a lethal Game of Chicken with the Authorities, but blinked.

How has GMA gotten away with so much and for so long
? It may have to do with the problem of the alternative. Consider that the "Civilian Component" of the Manila Peninsula Rebellion of Sonny Trillanes, such as it was, included the likes of Linggoy Alcuaz, Mentong Laurel, Dodong Nemenzo, Bishop Julio Labayen, Tito Guingona. Together with the CPP national united front organization Kapisanan ng Makabayaang Ekonomista, these same individuals had been calling for the immediate resignation of the President, Veep, Senate President and House Speaker to make way for a transition government to be headed by Chief Justice Reynato Puno. Not even snap elections, the bums! Puno reportedly issued a statement saying he is "neutral" on the proposed Treason, instead of roundly denouncing it as another illegal Regime Change a la Davide 2001. But Puno should set his sights higher--on Geneva and the international justice scene. Those human rights Rules of Court under his watch are absolutely brilliant. Writ of amparo and habeas data by Puno should trump even Davide's judicial coup d'etat on Erap and the Separation of Powers.

There is another reason I say "alleged public apathy" -- I don't think the Filipino people have been sitting around just taking it from their Failed Ruling Classes -- past, present and future. They've been busy as OFWs (what the Sour Grapers call "toilet bowl cleaners of the world") raking in and sending home the over ONE BILLION DOLLARS A MONTH which is the rising tide raising all boats in this God-forsaken archipelago. We are care-givers to humanity and the world. We are not indolent apathetics. And as we send back and return the fruits of our labor to our families and children, whom we have not at all abandoned or forgotten, as the Sour Grapers grouse, the spell is broken, the voices of political correctness and social authority lose their omniscience, lose their ability to abash us. The solutions to our own problems lie in our own hands, and not of saviors, or leaders with vain ideologies.

MLQ3 renews a favorite pitch of his (and mine) for "a common political vocabulary"--which I locate in the Constitution, and urge upon him and other utilitarians the greater appreciation for the Social Contract tradition of interpreting constitutions, citizen's and institution's rights, duties and privileges.

MLQ3 ends with a quote from Inday Varona Espina's recent essay in Graphic Magazine
“The young agree about hell lying in wait. But they think the solution lies elsewhere: Collective tinkering with a wheezing vehicle, the dismantling of roadblocks and a more discerning choice of routes. These acts require sober dialogue, not the hissy fits coming from all shades of the political spectrum these days.” So we must ask ourselves, how well do we understand the engine we intend to tinker with?
I really like the dark, quirky, unpredictable column pieces that Ms. Inday Varona Espina has produced over the years as The Scarred Cat. But that was mainly before she became chairman of National Union of Journalists of the Philippines (NUJP) and editor of Philippine Graphic Magazine. (By the way, I once shared a double bunk bed with her predecessor, Luis Mauricio, editor of the same venerable publication in September, 1972, when Ferdinand Marcos arrested us both along with about a hundred others including Ninoy Aquino and Chino Roces from both the Mass Media and the Political Opposition, on the night Martial Law came down.)

Gloria Macapagal Arroyo has a ways to go in order to become Ferdinand Marcos, in my book, which also says on Page One: Eternal Vigilance is the Price of Freedom!


Eric said...

It's the lack of a clear alternative that keeps GMA in power. While columnist like de Quiros will simply say something to the effect of, "Let's oust her first, then worry about what to do afterwards," is not enough to send Filipinos back into the streets to rally against her. The fact that the opposition is so inept is another factor in her favor.

I agree with you that GMA has a ways to go before getting close to Marcos. That survey result only goes to show how short our collective historical memory is.

john marzan said...

i don't think the current opposition is all that bad. the anti-marcos groups were worse. marcos ruled for almost 20 years before the peoples got all worked up and ousted him.

i don't think the the current opposition is the problem vs arroyo. i don't think the leftists are the problem too. they were part of edsa dos hindi ba, and nobody told them then that their participation was unwelcome.

i think the real problem are the hypocrites who railed against corruption when it was convenient, and advocated people power when it was convenient.

victory has many fathers and defeat is an orphan, and i'm sad to know that you don't feel the same way about arroyo anymore and has criticized the very thing you supported back in 2001 and feb 2006.

john marzan said...

EDIT: convenient for them...

Sef said...

Simply put, the recent survey has no value. Sad to say mass media plays on it as if it is the most important news. Rather than pursue leads and evidences on PGMA's and all the other politician's crimes, they rely on cheap propaganda ploys.

Had our mass media been NOT as it is, PGMA and the rest of our corrupt politicians would not have survived a bit.

tiki said...

DJB questions this survey and yet asks anyway, "How has GMA gotten away with so much and for so long?" which in some ways supports the results of the survey.

He gives several answers to the question. First, he argues that there is no alternative amongst the opposition. Doesn't DJB consider himself as one who opposes GMA? That is what I gather from the question above. And aren't there others like DJB? If so, then DJB's first point is questionable.

Finally, he argues that Filipinos are too busy making too much money overseas and that the opposition is merely sour-graping. Is DJB aware that most Filipinos are working to fill in jobs abroad? What happens if a global credit crunch and a depreciating dollar makes those jobs less lucrative?

Also, if you divide OCW earnings by the total number of OCWs (around 8 million) and assume that each OCW has to support five other people, then you are looking at around $25 a month per person, or $300 a year. At PhP45 to the dollar, that's PhP13,500 a year, or roughly a thousand pesos a month. And that's given the current exchange rate: what will happen if the dollar depreciates further? And that money will be spent in a country that has some of the highest costs for petrol, medicine, electricity, and other needs relative to income in the region. On top of that, petrol and food prices are increasing significantly worldwide and it appears that they will continue doing so in the long run.

Finally, the WB reported in the past that up to 50 percent of government revenues is lost to corruption: that money could have gone to public education, health care, military equipment, and so on. Just recently, it was reported that the WB wants to suspend loans to the country due to various anomalies, and the _Economist_ reports that government corruption is the main reason why the Philippine economy is not taking off. On top of that, it's likely that lots of red tape and a complex tax system is making it difficult not only for foreigners but even for locals to do business here.

Gabby said...

dear DJB,

i need to understand your differentiation between scientific and unscientific surveys. if i understand, scientific surveys are surveys that (eventually) agree with some kind of test (like a voluntary election, which in itself is a survey too, or sales).

in this case, many surveys that sociologists and economists use aren't scientific. the FIES isn't. The U of michigan (i think is the institution) Survey of Consumer Sentiment isn't one either. The PSID isn't. So are the hordes of surveys regarding policy/program evaluation.

is this your opinion?

Jego said...

How to test the the 'second type' of public opinion polls? Have SWS do the survey again using the same methodology used by Pulse, and using the same survey materials. If the results are the same, then that would validate the survey results scientifically. You can keep doing this and see if the results are the same.

This second type polls make an implicit prediction: that the results are the same with any random sample when using the same methodology. They can be tested by doing the survey again.

DJB Rizalist said...


All I am saying is that even if one conducts a survey in a scientific manner, its scientific value may be nil if the survey question is such that the validity of the survey will never be tested, ie falsified OR verified by some independent event or process.

Such event or process must be capable of deciding the question for the entire population, such as an election, not just another survey.

There is no free lunch in public opinion polls. they are not an omniscient oracle. there are real stringent scientific limits to their powers as statisticians. they are turning statistics into a genre of propaganda, a headline generating machine.

The main point here is you can't conduct a scientific survey on just any old survey question that comes to mind. MOST questions can't be reliably answered by the surveys, but that only INCREASES their entertainment value when they can be tantalizingly touted as being just as accurate as the voter preference polls.

Not all survey questions are created equal. Some fail the basic test of what constitutes a good survey question.

Gabby said...


i'm not sure about your response about scientific vs nonscientific.

i do heartily agree with your comments and some of your previous posts on certain questions, such as Hunger and (net) satisfaction ratings. Other examples are the 'happiness', 'valuation' and 'culture' questions in other surveys.

These are subjective questions which require interpretation and analysis. The problem with people with an agenda may hijack the figures and force them to tell a story that is sympathetic to their world view. The media needs to be aware of how to interpret these kinds of questions and answers, which is not as easy or straight forward as questions about income or prices.

DJB Rizalist said...


As gabby points out, every survey measures subjective states of the respondents, and extrapolates to the entire population from the random sample.

The point I am raising is that certain types of survey questions are suited for scientific surveys because the question means more or less the same, simple and unequivocal thing to all respondents because it is related to a common subjective experience like deciding who they will vote for in an election.

However, if we ask about who we consider the most corrupt president, it's not so clear how each respondent may apprehend such a question. Also, there may be a sampling bias in that all possible respondents are more familiar with the current President.

The statistical error or variation that results from bad question design cannot possibly be measured by the standard margin of error, which depends entirely on the sample size alone.

Surveys of the second kind are like news reports that do not cross check or attempt to verify claims by a single source.

Amadeo said...

A noteworthy discussion on surveys.

And if I may add, surveys done properly and impartially are good measures of people’s perceptions and preferences, which then become the facts for the day. But at the same time, the findings do not necessarily translate to specific or general truths. Because they are after all simply the perceptions and preferences of the sample tested.

Kahiyaka said...


DJB Rizalist said...


Ang GinagawaMo said...