FYI: The True Uncertainty in the Net Satisfaction Rating

fter about twenty years of being exposed to the public opinion polls, both the Media and Public have, if only dimly, come to understand that there is a built in UNCERTAINTY to every statistic reported by the scientific pollsters like SWS and Pulse Asia. This uncertainty is sometimes called the MARGIN OF ERROR or the SAMPLING ERROR because it arises from the fact that every survey picks a finite random sample of the overall population, asks them the survey question and then PROJECTS the result to the entire population. The margin of error is the price we pay for surveying a small subset and then making a generalization that can become a screaming headline about the total population.

Typically, SWS and Pulse Asia use a random sample size of 1200 respondents. Now, get a nice calculator and compute the the reciprocal of the square root of 1200 (Enter 1200, press the button [1/x] then the button [SQRT]) and you will get a result equal to 0.02886751 which approximately equal to plus or minus 2.89 percent or about plus or minus 3 percent. If you go to the website of SWS or Pulse Asia you will see that they report this number, plus or minus 3 percent as the "sampling error" in their national statistics. If you do the same thing for the regional sample size of 300 respondents, you will find that the reciprocal of the square root or 300 is plus or minus about 5.77%.

So far so good. But they do something clever when they ask respondents if they are satisfied or dissatisfied with the President. They take the DIFFERENCE of the two statistics by subtracting the percentage of those saying they are dissatisfied from those who say they are satisfied and they label the result the NET SATISFACTION RATING (NSR). For example today, SWS reports that 39% claim to be satisfied while 43% claim to be dissatisfied resulting in an NSR of negative 4 percent.

But it cannot be denied that the sampling error in each of the two statistics being combined by taking their difference is plus or minus 3%. In other words 39% plus or minus 3% are satisfied whilst 43% plus or minus 3% are dissatisfied. But it is WRONG to think that the Net Satisfaction Rating also has a plus or minus 3% margin of error, because it is fundamental to the statistical sciences that when you combine statistics that have sampling errors, you must ADD together the individual sampling errors to calculate the equivalent error or UNCERTAINTY in the computed result, like the NSR.

This means that the uncertainty in the NET SATISFACTION RATING is actually PLUS OR MINUS SIX PERCENT. That is the reason why SWS in its media release said that the apparent improvement in the President's NSR was "SLIGHT" even though it apparently changed from negative 13% last quarter to negative 4% this quarter! Technically, the improvement of 9 percentage points in the NSR is "slight" because it is bigger than the uncertainty of plus or minus 6% in the NSR, but only "slightly." However, the best way to describe this most recent poll result is that, at the level of confidence cited (95%) it is not possible to decide if President's NSR is positive or negative. But it cannot be denied that it HAS improved, at least slightly, from the last survey.

If you compare plots of the NSR over time with plots of just the Satisfaction rating or just the Dissatisfaction Rating, you will notice that the NSR is about twice as erratic as the latter two statistics when plotted in a time series. This is because the NSR is a really lousy, almost useless statistic at random sample sizes of 1200 respondents. The NSR is however a very a commerciable statistic and often makes BIG BOLD HEADLINES, especially among ignorant mass media outlets.

The other important point is that if you add 39% to 43% you get 82%. In other words 18% of the respondents were UNDECIDED, which is in fact, to my mind, the real magnitude of the uncertainty in the NSR. But they INVENTED the NSR precisely to avoid this rather embarrassing fact about their polling.

By the way, the formula that relates random sample size to sampling error -- "the reciprocal of the square root of the random sample size"-- is only approximate. It depends on the level of confidence you want to have in the result and would unfortunately take a major detour into combinatorial and statistical theory to derive mathematically. So you will just have to accept this formula on "faith" from me and the pollsters.

It is a key concept however in understanding surveys and learning how to properly interpret them. Sampling error or margin of error or statistical uncertainty is also the reason why the SWS says that 17 candidates still have a statistical chance of making it to the Magic 12 in the Senate race. I could nitpick this statement too, but let's save it for the next time...

MLQ3 has the latest analysis of the Senatorial race based on today's Pulse Asia, Inc. poll.

---

New to my blogroll is INEVITABLE KARMA, a blog from the Philippine Science High School. Here's a video from some of the students there (Pisay Meets The World--We Hold the Future):


ROCKIN' SCIENCE!