Wednesday, November 7, 2007

Why Self-Rated Poverty and Hunger Statistics Are Scientifically Meaningless

pinion pollster Social Weather Stations needs to find something better to do in between elections. Their quarterly reporting of SELF-RATED poverty and hunger "statistics" seems designed to ensure that they will have some kind of feel-bad news for the headline writers of the scoop-addicted and sensation-dependent Press to report. The general public, unfortunately, isn't savvy enough to see through the fog of dire-sounding results when even the normally savvy scientists who work at SWS are complicit in the fallacious tsk-tsking.

Now whenever I see a headline like "52 Percent Feel Poor" or "20 million Pinoys say they are going hungry" I am not really impressed. To see why neither should you, substitute some negative self-describing adjective for "poor" and it would make sense but not be sensational. For example, consider a headline like "52 Percent Think They Are Short."

In fact, if one considers any "normal distribution (bell shaped curve) that represents some kind of social good or human preference, like wealth, access to food, health care, insurance, good looks, political influence, etc. it is unavoidable that half of the population will be below the Median Level and half above the Median Level. And so it is always possible to measure a "feel bad" and therefore "newsworthy" statistic merely by reporting the percentage of the population that describes itself as being one, two or three standard deviations below the Mean of that distribution. In the case of wealth for example, there are always "the poor" and "the rich" at either end of the distribution, even though the poor in one country may be far better off than even the rich in another country in terms of all the primary social goods that are actually available to them.

In the self-rated poverty and hunger surveys recently reported, SWS and its client news outlets like PDI, try very hard to make it seem deeply profound that,
"Household heads' ratings of general poverty, food poverty and experience of hunger were "internally consistent," SWS noted. Nationwide, the proportion of households who experienced hunger during the past three months was at 31 percent among those who rated themselves "food poor" and 28 percent among the self-rated poor..."
Considering that food, shelter and clothing are basic necessities and primary social goods, I wouldn't be surprised either if they next report that among the self-rated poor, a near-equal percentage would rate themselves as not having enough clothes to wear or good enough homes to live in.

The gentle reader will please notice that I am not questioning the scientific integrity of the data on "self-rated" statistics, or the accuracy of its measurement, merely on its real significance.

And why does SWS have to trot out an utterly meaningless and tendentious idea like "internal consistency?"

Well, because unlike the voter preference surveys that have made SWS deservedly famous, there is simply no independent, objective event or process by which their findings on self-rated anything can be falsified or verified. And so they make self-rated hunger stats and self-rated poverty stats, each other's verification. Because of that, self-rated statistical polls cannot be considered to be scientifically meaningful polls in the same way that voter preference polls are scientifically meaningful, which are falsified or verified by the results of the elections. The potential for egg on their face keeps SWS honest. But in the case of the self-rated stats, there is no such verification or falsification, and so merely reporting them allows people to draw their own conclusion, to which they are often led by the nose by the pollsters when "record levels" of hunger are noted and highlighted. I have criticized these results in the past because of seasonality and periodicity in the data indicates there may be nothing more than statistical variation to such "record levels."

The only reason SWS can get away with these numerically meaningless polls about hunger and poverty, is that most people cannot deny that there must be SOME level of poverty and hunger in the land, just like there have to be some short people if there are tall people and a bell shaped curve or normal distribution of the height attribute to begin with.

Because of the essential use of rigorous statistical and mathematical techniques to produce their raw data, SWS and other public opinion pollsters like Pulse Asia, Inc. have a just claim to being "scientific organizations" in much the same way as the weather bureau Pagasa.

But this can be very deceptive. Just as most newspapers and broadcasters have fairly rigorous news gathering components, with ethical standards that call for verification and substantiation of what gets reported as "news" -- which are the equivalent of the data gathered by public opinion surveys--the pollsters also offer "opinion". By this I mean that SWS and Pulse offer interpretations and explanations of why their data comes out a certain way, what factors they think account for the scientifically gathered results. Here things are not so scientific and often reflect the pollsters biases and ideology. In contrast the weather bureau does not offer personal opinions about WHY the weather is the way that it is, why a typhoon has suddenly appeared on the horizon.

Public opinion polling is unavoidably a genre of journalism. The data reported are the equivalent of news stories, but the attempt to explain the factors that led to the statistics are more like editorials and opinion columns, which may or may not be correct. In the case of the self-rated hunger and poverty stats, there is not really even the opportunity to check what the objective reality is relative to the numbers reported. And so they are really the equivalent of "letters to the editor"!

And so, as with the mass media, the advice is also: caveat emptor!

23 comments:

viking said...

"Well, because unlike the voter preference surveys that have made SWS deservedly famous, there is simply no independent, objective event or process by which their findings on self-rated anything can be falsified or verified."

Dean,

You're mistaken about falsifiability. The NSO can actually do a parallel survey on self-rated poverty simultaneously with the Family Income and Expenditures Survey. But that would cost a lot, considering the FIES sample is much larger, more than 40,000 households nationwide.
If such a parallel survey can be funded, then it would be a simply matter to do chi-square analysis and more sophisticated regressions whether self-rated poverty does correlate with reporter income and expenditure levels.

stuart-santiago said...

oh wow dean, it's not as if the self-rated hunger stats are not consistent with a long series of surveys not just by the sws but also by government's family and income expenditure survey, the latest of which shows that while gnp has risen (the rich are getting richer), incomes have fallen, the poor are getting poorer, which means increasing hunger, and yes, lack of clothing and decent shelter, which you seem to trivialize.

stuart-santiago said...
This comment has been removed by the author.
Roehlano said...

I won't agree that self-rating systems are numerically meaningless. Perhaps there is meaning there buried in the madness.

But in practice I stay away from that stuff. I'd rather go with something measurable and replicable, or "objective", like measures of malnutrition (weight for height, height for age, etc.), energy intake (kilocalories/day), or (for poverty) income and expenditure. That's the approach taken by most statistical agencies in the world, including our own.

viking said...

You're right econblogger. But in the greater scheme of things, the 'subjective' is part of the 'objective.'

Unknown said...

djb
First and foremost, these survey firms are not charitable institutions. Meaning, they have to create revenue in order to survive and become a viable enterprise. How do they earn money? Of course when different companies, political parties, interest-groups commission surveys, they have to tailor the questions in such a way that the end-result would be to the liking of the client. If not, then they lose business.

Although I am one with you that I will not refute the statistics and the mathematics side of surveys, we must remember surveys are still down the line mere perceptions of a target population most of which do not even reach 0.1% of the real population. (of course there are for practicality purposes)

Most Filipinos are very fond of playing the "blaming game" and these surveys are some of the toys that they employ to play the game. Instead of people saying, what can we do to help?, what can we do to contribute?, the usual line after surveys would be to sing songs of blame to high heavens.

What's funny is that some people see these surveys as gospel truths. I am saddened that there are a lot of people who don't understand the dynamics of public-relations and perception-driven endeavors.

It's all a game after all.

Deany Bocobo said...

Econblogger,

I agree. In fact, every public opinion poll is a self-rating system almost by definition!

The voter preference surveys are self-rating systems in which the respondent is basically deciding which one of two possible categories posed by the survey question he belongs in. For example, which candidate does he prefer for President.

The point I was trying to make is that a survey is scientifically (or as you say, numerically) meaningful if and when there is an independent event involving the entire population from which the survey's samples of respondents were drawn.

Deany Bocobo said...

Viking,
Of course the poverty survey is verifiable. but it requires a process as big as an election or census. There's the rub for there is no free lunch when it comes to statistics.

viking said...

Right Dean. And the head of your post is wrong and needs to be qualified. Right? I've used the services of the SWS, about 300 meters from my hole in the wall, and have also worked with NSO staff.

Hey, what's the score with Twink? Some of my friends are anxious.

Deany Bocobo said...

viking,
where is the event analogous to an election for the self rated poverty and hunger stats?

without it, they R scientifically meaningless, in the strict statistical sense. In other words, this is not a good survey question.

Deany Bocobo said...

stuart-santiago,
I submit that it is the SWS that may be trivializing or distorting the problem by presenting unscientific data that can't be verified or falsified.

in the recent surveys, SWS claims upwards of 19 million Filipinos are going hungry.

But I think it makes a big difference if this number is more like 4 million and not 19 million.

who knows now what number is right?

Ben Vallejo said...

Without a theory to explain it, statistics are really meaningless. These hunger statistics must be corroborated by other measures in health,economics etc.

Undergrad stat students are taught to be careful in interpreting means. In fact there is an essay on "How to Lie with Statistics". I don't think SWS or Pulse Asia or whatever is in the business of lying but the onus really falls on the blurbs,who have to make living on quoting these stats.

cvj said...

DJB, how about if the person in question commits suicide because of his/her self-perception of poverty. Will that make the data more meaningful?

stuart-santiago said...

i have no doubt that the number of filipinos going hungry is closer to 19 million than 4 million.

"Some 11 million Filipinos are among the 1 billion people across the globe living on less than $1 a day, the threshold defined as extreme poverty by the United Nations, according to the International Food Policy Research Institute (IFPRI)." (posted nov 8 2007)
http://services.inquirer.net/print/
print.php?article_id=99568

which doesn't count pinoys living on a dollar a day, that's 43 pesos these days. in my book that spells hunger and poverty too, given rising food costs and other living expenses such as, yes, clothes and shelter.

significant to me the coincidence of this discussion with the 11-year old girl's suicide and her self-rated poverty. but then such connections don't pass the scientific test, no?

Deany Bocobo said...

cvj,
C'mon, surely you cannot point to this tragedy and from it draw some kind of moral or political lesson about poverty. It is certainly not yet at the point in this country when 11 year old girls are so moved to suicide by the widespread poverty and hunger. Naah, something else went wrong with this young girl, and it is sheer political exploitation to connect it with the hunger and poverty stats.

Deany Bocobo said...

stuart-santiago,
It is a scientific question whether or not the number of starving people is 19 million or 4 million, not just a matter of our feelings. I think at 19 million we should have see civil disturbances and food riots long before this. by the way 4 million is what SWS calls "severe hunger." When you analyze the severe hunger data, we find it quasi stable and with no seasonalities and periodicities unlike the moderate hunger numbers (the 19 million) which shows hunger peaking on every other quarter, the one, curiously before harvest seasons in December and June!

BTW, the world's No. selling gin is Ginebra San Miguel. Do you think vices among poor Filipinos might have something to do with the overall level of "hunger" and poverty? How about shabu, cigarettes, prostitution and other vices?

Deany Bocobo said...

Folks,

I cannot yield to the essentially ad hominem argument that an insistence upon scientific rigor in our estimates of hunger and poverty is impersonal or insensitive to tragedies like the suicide.

I think it is moral imperative that we get the number right. Is it 19 million or 4 million??

Because the scale and earnestness of our long term response surely depends on how big the problem truthfully is.

As for the suicide of an eleven year old girl, if we are to accept the Davao data point as valid, then we really ought to see more such suicides among young heads of household, say in their 20s, who must surely bear the brunt of the alleged poverty and hunger incidence.

Perhaps there is such a trend not hitherto detected.

Suicide however is queer, queer meme. For how can an idea survive among human beings when its successful achievement of purpose results in the destruction of the very human being it has convinced and inhabits.

cvj said...

DJB, in the Social Sciences, self-observations have real significance and are as much an object of study as objective measures that are set by the 'experts'. I do accept that the seriousness of such self-rating responses can range from someone like Gloria Arroyo's who said that she also skipped a meal in the past 3 months all the way to a little girl actually hanging herself out of desperation. But isn't there a also a science to accepting the latter and throwing out the former? In any case, there is also a need to supplement poverty thresholds set by the experts since these are also arbitrary and are by no means immune from political agendas.

Deany Bocobo said...

cvj,

Knowing why Marianette killed herself is an entirely separate matter of which I actually know very little.

The point is, that knowing the true levels of real poverty and real hunger is still the most sympathetic and effective approach.

Knowing the truth always helps against any illegitimate political agenda.

Pushing a sort of fuzzy wuzzy feel-bad number that does not have the force of logic and reason that SWS voter preference surveys do, is a disservice to both the science and the nation. But I am not attacking self rating systems as such.

As I said early on in this thread, ALL public opinion polls, by practice and tradition, ask respondents to rate themselves on what is equivalent to a Yes or No question. This is true for voter preference AND the self-described hunger and poverty surveys.

But we have to understand the difference in the quality and reliability of the latter set of surveys as compared to the former. The more complicated and indefinite a survey question is, the less reliable are the statistics that result. And when there is no external objective event that falsifies or verifies the earlier survey results, the overall quality and reliability of the surveys drops and one wonders whether or not they are even scientifically meaningful

I think the real hunger levels are closer to 4 million than 19 million. Otherwise there would be a lot more trouble than we are seeing.

I wish I could rely on SWS for hunger and poverty statistics with as high a confidence level and as low an expected error level as I do with their voter preference surveys.

Unknown said...

Don't you find it funny that many 'poor' people,in the philippines own phones that are more expensive than those owned by middle-class earners. it says a lot why they are poor. A balut vendor having an ipod and the latest n-series? This is not even a exaggeration.

No wonder many are 'hunrgy' because they keep their non-necessities "full".

I'm not saying that there are no 'credible' poor but a lot of times, it's falsed by materalistic people

Jenijenjen said...

Hi Dean. Poverty statistics are indeed misleading. What does it really mean to be poor in this day and age anyway?

Sure there are hungry people.
But as Betcha aptly said, these "hungry" people are the same people we see walking around sporting fancy cell phones. Never mind if they are GSM (galing sa mandurukot).

Also, being called "food poor" can certainly be misleading. I mean the people surveyed may be those who think "food" are instant noodles, canned goods, and jollibee. If that is the case, then they will really say they are "food poor" and "malnourished" because in order to be able to acquire these kind of food, you'd need to have some level of purchasing power - something they actually lack.

But if they would define food as what they really are such as rice, corn, veggies and nutritional stuff that can be planted, then i do not believe that anyone will ever get hungry especially those living in the provinces.

Anyways, for all we know, these people who may have been surveyed are probably people or descendants of people who have left their provinces and opted to squat in Metro Manila. These might be the same people who prioritize cellphones, cellphone load, and Lucky Me over real nutrition. These may be the same people who have colored televisions and dvd players in their barong barongs but always complain that they have nothing to eat.

I guess what I'm trying to say is that bottom line, kasalanan din nila.

I'm sorry if this is going to sound a bit elitist but the reason why they are poor isn't really because they are "poor". They are poor because they want to have things that are beyond what is basic, and hence beyond their means. Then they participate in surveys and eventually become a statistic.

upoytao said...

"Gusto ko po sana magkaroon ng bagong sapatos at bag at hanapbuhay para sa nanay at tatay ko. Wala kasing hanapbuhay ang tatay at nagpa-extra extra lamang ang aking nanay sa paglalaba," she said in her "Wish Ko Lang" letter. [I wish for new shoes, a bag and jobs for my mother and father. My dad does not have a job and my mom just gets laundry jobs.]

"Gusto ko na makatapos ako sa pag-aaral at gustong-gusto ko na makabili ng bagong bike," she added. [I would like to finish my schooling and I would like very much to buy a new bike.] - Mariannet Amper

+*+*+*+*+*+*+*

Look kid you're not an "isolated" case as Malakanyang said short of making your death as irrelevant as hunger and poverty in this country.

I will never judge you nor tell you how wrong you are. No. I will never say that you don't own your life or that you owe it to your parents or the creator above.

My dearest Mariannet I believe It is your freedom to do so-to take your life is your greatest freedom. I believe that It is equally courageous to kill oneself as well as to live it.

They might say that you are a demented child a lonely girl looking for attention, that with your broken dreams and weak heart you choose to cut asunder your future.

How would they know? Have they experienced yor pain? Have they looked into your eyes? And why is it that moviestars and famous people have the right to de depressed and get treatment? But alas not you my dear Mariannet you have to face this life astride the back of poverty. Oh poverty that great killer which our government has kept mum about.

That while the government could give "Cash Gifts" left and right It has never given you and your kind a farts notice. Yes the government has thought that your poverty doesn't concern them. That your death is an "isolated" case.

I think that wherever you are you will never be hungry again. You will have food on the table and you will play with angels. I know you killed yourself because you love your family. That you would rather give them peace than have them look for something to give you. I know you don't hate them but you hate what this government and this society had driven your demise.

It is ok to die now Mariannet this country and this world doesn't deserve you. I would've followed your example if not for the hope of making more sense in this world.

But i find you brave and admirable.

Rest in peace my dear one. Rest in God and never be hungry ever again.

I'll see you soon.

I hope this link helps: http://www.pcij.org/blog/?p=2050

Dave Llorito said...

hi dean, try comparing the trends in self rated survey and the poverty threshold. first, sws says that self rated hunger/poverty has generally been down. second, the poverty threshold is declining, an indication, the sws said, that people are tightening their belts.

i find that interpretation interesting. if self rated poverty/hunger is down, necessarily one CANT say people are belt-tightening. maybe the threshold declined because people got higher incomes. when you say are you are less poor/less hungry, you dont say "im tightening my belt."