The Ombudsman's Supplemental Resolution absolving Comelec commissioners of criminal fallout from ITF vs. Comelec and the Automated Counting Machines (ACMs) fiasco, contains the following on the testimony of Sec. Estrella F. Alabastro of the Dept. of Science andTechnology (DOST) and the tests they conducted to see how accurate the ACMs were as Optical Mark Readers:
(A) Secretary Alabastro testified that she was one of the members of the Advisory Council and the Technical Ad Hoc Evaluation Committee (TAHEC) who formulated the policies relating to the technical aspect of the automated election system. That when she was furnished with the list of twenty seven (27) key requirements to be used in the evaluation of the automated counting machines (ACMs) she noted that the accuracy rating that was required is 99.995%, whereas the Request for Proposal (RFP) had a higher accuracy rating of 99.9995%. She said however, that what was adopted in the meetings she had with the COMELEC and the Advisory Council is a 99.995% accuracy level and not the 99.9995% since the ACMs will be tested to read only 20,000 marks and not 200,000 marks.Her basic information is that DOSTtested the machines for a 99.995% accuracy rating by feeding them 20,000 test marks on sample ballots and measuring the error rate.
When they observed ZERO errors in such a test run of 20,000 marks, i.e., a given machine correctly read ALL 20,000 marks, the DOST concluded that the accuracy rating of the machine was 100%. Indeed, in the Ombudsman's Report we find:
(D) Engineer Rolando Viloria stated that as the Chairman of the DOST Technical Evaluation Committee he issued the Tests Certifications attesting to the fact that the ACMs of Mega Pacific had obtained a 100% accuracy rating during the verification tests.The MPC-supplied automated counting machines (ACMs) are based on Optical Mark Reader (OMR) technology, similar to that used by generations of students taking multiple choice exams. When Sec. Alabastro stated that "the ACMs will be tested to read only 20,000 marks and not 200,000 marks" it is because 99.995% accuracy implies that every time the machine is fed a mark to read, there is a 1-in-20,000 probability that it will make a optical mark reading ERROR; while an accuracy rating of 99.9995% corresponds to a 1-in-200,000 probability of an erroneous reading in which the machine either thinks there is a filled in mark when there is none in reality, or it thinks there is no filled in mark when there is.)
But was the DOST mathematically justified in claiming an accuracy rating of 99.995% (or even 100%) because a certain automated counting machine they tested was able to read without error a set of 20,000 optical test marks?
NO!! Here is an authoritative reference. And anyone who works with statistics, such opinion surveyors and quality control professionals knows why.
A test run with 20,000 optical test marks does not have enough STATISTICAL PRECISION to measure the required 99.995% accuracy rating because the MARGIN OF ERROR in such a measurement is TOO LARGE!
In order to measure an accuracy of 99.995% DOST must be able to distinguish the difference between 99.995% and 99.994%. That means that the STATISTICAL ERROR in their measurement of accuracy must be smaller than around 0.0005%
Most people are aware that the statistical margin of error in SWS surveys is equal to one divided by the square root of the NUMBER of respondents. So for their standard 1200 respondent surveys, the margin of error is plus or minus 2.89% which is usually quoted in news stories as plus or minus 3%.
If SWS used 20,000 respondents instead of 1200 respondents, their margin of error would be equal to plus or minus 0.7%.
And that number is also the margin of error of the DOST tests on the ACMs of Comelec because they fed each machine 20,000 test marks and counted the error rate.
So the correct, way for Alabastro or Viloria to have reported the DOST Test results was this: the counting machines were measured to have an accuracy rating of 100% plus or minus 0.7 percent. The DOST tests do NOT prove in a statistically acceptable fashion that the ACMs meet the required accuracy rating, even at the 99.995% level.
The correct number of test marks that should have been fed to each machine in order to be able to measure an accuracy rating of 99.995% plus or minus 0.0005% at confidence level 95% (Z=2) should have been:
n=200 billion optical test marks with E=1 million or less errors
This may seem like a lot of testing, but that is the consequence of even the REDUCED accuracy rating of 99.995%! With a proper appreciation of the challenge of such a verification test as DOST agreed to undertake, and with proper planning, such qualification testing could have been accomplished. Relaxing the confidence level to 68% (Z=1) would reduce the required number of test marks to 50 billion while maintaining the required .0005% margin of error.
CONCLUSION: It is possible that these ACMs DO have an accuracy even better than 99.995%, perhaps they could meet the original 99.9995% requirement. Or they could all be REJECTS at 99.3% which is within the margin of error of a 20,000 mark test from the reported 100% accuracy rating.
Based on testimony just revealed in the Ombudsman's Supplemental Report, the DOST did not conduct properly designed statistical tests to measure just exactly what the accuracy rating is of these ACMS. One thing for sure, 20,000 test marks was not enough to prove at 95% confidence level that a given ACM had an accuracy rating of 99.995% because the test was "TOO BLUNT" -- it lacked the statistical precision required.