The Skinny on ACM Accuracy Rating

ABSCBN News: SC orders Ombudsman to explain ACM ruling

The Supreme Court (SC) has ordered the Office of the Ombudsman to respond to the petition filed by opposition lawmakers seeking to nullify its decision clearing Commission on Elections (COMELEC) officials of any criminal liability over the voided P1.3-billion automated counting machines (ACMs) contract. The High Court has given Ombudsman Merceditas Gutierrez 10 days to answer the petition filed by Sen. Aquilino Pimentel Jr. Pimentel led other senators in filing the petition questioning the decision of the Ombudsman absolving the COMELEC led by its chairman Benjamin Abalos and officials of MegaPacific in connection with the purchase of the 1,991 ACMs amounting to P1.3 billion.

It certainly appears that the "Ombudsgirl" (in Sen. Joker Arroyo's colorful characterization) has plenty of explaining to do to the Supreme Court.

But one of the more interesting scientific aspects of this case is the matter of the ACCURACY RATING of the ACMs and the manner by which the Dept. of Science and Technology is alleged to have certified them.

From Ombudsman's Supplemental Resolution on MegaPacific Contract: The report shows that the failures on six (6) items, were not defects attributable to the machines. DOST after analyzing the causes of non-conforming accuracy rating using statistical analysis and investigation on the so called assignable causes of variation, concluded that the discrepancy was due to improper shading of ballots resulting to the failed marks obtained by the machines of Mega Pacific. More, in the same report of DOST, it is categorically stated that the results of the verification tests on the machines of Mega Pacific in fact yielded a one hundred percent (100%) accuracy rating for all three environment conditions.

I have previously pointed out that the ability to read shaded marks under a wide variety of conditions is surely the central point of testing an Optical Mark Reader, so the first part of above statement is beguiling and mystifying to say the least. But let me concentrate on the point about accuracy rating.

The Ombudsman's assailed Supplemental Resolution contains the following testimony in this regard from from the Science & Technology Dept. Secretary Estrella Alabastro--

From Ombudsman's Supplemental Resolution on MegaPacific Contract: "Secretary Alabastro testified that she was one of the members of the Advisory Council and the Technical Ad Hoc Evaluation Committee (TAHEC) who formulated the policies relating to the technical aspect of the automated election system. That when she was furnished with the list of twenty seven (27) key requirements to be used in the evaluation of the automated counting machines (ACMs) she noted that the accuracy rating that was required is 99.995%, whereas the Request for Proposal (RFP) had a higher accuracy rating of 99.9995%. She said however, that what was adopted in the meetings she had with the COMELEC and the Advisory Council is a 99.995% accuracy level and not the 99.9995% since the ACMs will be tested to read only 20,000 marks and not 200,000 marks."

What exactly did Sec. Alabastro mean when she said "the ACMs will be tested to read only 20,000 marks and not 200,000 marks."?

This has to do with the fact that an accuracy rating of 99.9995% given to an optical mark reader (OMR) means that it NEVER produces MORE THAN ONE erroneous reading for every 200,000 test marks fed for it to read. By the same simple decimal arithmetic, an accuracy rating of 99.995% means the OMR NEVER produces MORE THAN ONE erroneous reading for every 20,000 marks read. Sec. Alabastro was merely stating that their tests on the ACMs accuracy were based on the lower accuracy rating of 99.995%.

But suppose one feeds an ACM 20,000 test marks and it reads every single one of them (100%) correctly, does it logically follow that the ACM has been shown to have an accuracy rating of at least 99.995%?

NO! It only shows that the machine could have an accuracy rating as good as 100% or as low as 99.3%.

The 20,000 test marks fed to an ACM by DOST to estimate its Accuracy Rating, is entirely analogous to a random sample of 1200 respondents in an SWS or Pulse Asia public opinion survey, to estimate how public opinion is divided on a given question! Just as a finite random sample size of 1200 respondents results in a Margin of Error or statistical uncertainty of plus or minus 3 percent (actually 2.89%) in any generalization we care to make about the entire population, a sample size of 20,000 test marks also has an easily calculable Margin of Error of plus or minus 0.707%.

The secret formula for computing the Margin of Error in any Random Sampling process as a plus or minus percentage is simple. Just take 100 and divide it by the square root of the Number of Respondents or Test Marks.

Thus when the DOST fed a given ACM 20,000 optical test marks to read and it read all of them correctly, the DOST may only claim that the ACM has an apparent accuracy rating of 100% plus or minus 0.707%. In other words, the DOST had only tested the ACMs for an accuracy rating of 99.3% not 99.995%.

How many test marks should DOST have fed into an ACM in order to scientifically and statistically prove that it has an accuracy rating of 99.995%? This depends on what accuracy rating would cause the testing agency to reject a given ACM. Suppose Comelec decides that if a given ACM is tested by DOST and found to have an accuracy rating of 99.985% the DOST must reject the ACM. This means that the statistical precision or Margin of Error in the DOST test should be plus or minus 0.01% if the DOST will have a chance of detecting the difference between 99.995% and 99.985% accuracy rating required for such a Pass-Fail Test.

Now we can work the Secret Formula backward to get the required number of Test Marks! It is equal to 1 divided by the Square of the Margin of Error or 1/(0.01%)(0.01%) which comes out to 100 million test marks. In other words, to prove statistically that a given ACM has an accuracy rating of 99.995% it must not produce more than one error in a test involving 100 million test marks. That may seem like a lot of test marks, but there is no free lunch when it comes to statistical quality control.

But for the purposes of the intelligent layman confronted by statistical tests involving random samples, the important thing to remember is this. The size of the random sample--i.e., the number of respondents in an SWS survey or the number of optical test marks for an ACM--determines a Margin of Error in any measured statistic. This Margin of Error or Statistical Uncertainty in the data is actually the smallest real difference that can be detected by the statistical test. It is also called the statistical precision of the test. For SWS surveys of 1200 respondents the corresponding "tickmarks" are separated by 6% (Plus or Minus 3%). For 10,000 respondents or optical test marks the precision is 2% (Plus or minus 1%). For 20,000 optical test marks the statistical precision is 1.4% (Plus or Minus 0.707%).

I repeat: if you feed an ACM 20,000 ballots and it reads them all correctly, all you have proven, at 95% confidence level, is that the ACM has an accuracy rating as high as 100% or as low as 99.3% because the margin of error associated with a random sample of 20,000 is plus or minus 0.707%. The DOST needs to feed an ACM 100 million test marks and observe not more than one error to conclude that it passes the 99.995% accuracy rating.

Please read the previous post on the novel Constitutional issues that arise from a careful consideration of ACM Accuracy Ratings: ACM Accuracy Ratings and One-Man-One-Vote Suffrage.

UPDATES:

The short answer to Jose de Venecia on Con-Ass is this: A House Resolution is NOT an Act of the Congress. Even if it is signed by all 235 Members of the Lower House, ANY House Resolution is a mere Act of the Lower House of Congress. A mere House Resolution, even if unanimous, does not satisfy the basic requirement of the Constitution which plainly states:

Art XVII Section 1 Any amendment to, or revision of, this Constitution may be proposed by: (1) The Congress, upon a vote of three-fourths of all its Members; or(2) A constitutional convention.

A House Resolution submitted to Comelec under this provision would be insufficient in form.