Polling Bleg: Ponnuru's Puzzle

PINION POLLSTERS have been measuring American public attitude towards legalized ABORTION for many years. Abortion is a political and ideological issue that strongly divides and distinguishes politicians and political parties. In the National Review Online recently, Ramesh Ponnuru presented an intriguing puzzle about apparently divergent results on this hot-button issue from two reputable polling organizations, Gallup and ABC News-Washington Post:

Gallup routinely asks, "Do you think abortions should be legal under any circumstances, legal only under certain circumstances, or illegal in all circumstances?" For the last thirteen years, it has subdivided that middle option into "legal under most circumstances" and "legal only in a few circumstances." A pretty typical recent result: In May 2005, 23 percent of respondents said abortion should legal under "any" circumstances, 12 said "most," 40 said "only in a few," and 22 said "illegal in all circumstances." Pro-lifers find this polling data congenial, since it yields a 62 percent majority choosing the two most pro-life of the four options (40 + 22). (Some polling by other organizations makes these findings plausible.) Other polls, however, find results that appear to conflict with these. The ABC/Washington Post poll asks, "Do you think abortion should be legal in all cases, legal in most cases, illegal in most cases, or illegal in all cases?" This question has typically yielded a majority for the two most pro-choice options. In Dec. 2005, for example, 17 percent picked "legal in all cases," 40 "legal in most cases," 27 "illegal in most cases," and 13 "illegal in all cases." That's a 57 percent pro-choice majority. I'm looking for explanations for the divergent results--specific explanations, that is; something that goes beyond the point that many people are ambivalent about abortion policy. (Ideally with some evidence to back up the explanation.) Can switching from the word "legal" to the word "illegal"--that is, from "legal only in a few circumstances" to "illegal in most cases"--cause a 19-point shift in measured public opinion? Is there some difference between how people respond to "cases" and how they respond to "circumstances?" Note also that neither set of results appears to be a fluke: They are consistent over time in generating predictably divergent results. Also: Is there any reason to think that one of these ways of phrasing the question is better than the other, in terms of shedding more light on public opinion?

FORCED TO GOOD: I take the attitude that since we have no reason to doubt the competence of both Gallup and ABC to conduct public opinion polls, we must accept the data as reflecting SOMETHING about the public's attitudes towards abortion. That "something" must be different in each case because the divergences can't be "real" if the statistical data are valid. But first, I looked up the original data of Gallup and ABC News referred to by Mr. Ponnuru, from midyear 2005 and yearend 2005, and including previous data sets from each series in previous years. (I added these as bluetext links in the above quote from NRO.) In both polls, the results over time behave consistently. The apparent divergence is observable over the years that both polling firms have been conducting the same basic survey. Both use random sampling methods to conduct these periodic telephone polls on the target population. ABC News typically polled about 1000 people while Gallup chose 1200 or so respondents. Their respective error margins are plus or minus 4 percent and 3 percent. (I like using the simple rule of thumb that the margin of error is plus or minus the square root of the inverse of the number of respondents for a quick and dirty estimate of any poll's statistical precision.) Since there is no reason to doubt that the two polling firms go about their business in a thoroughly professional and scientific manner (to the extent that that is possible in the realsm of statistics) we are forced to the conclusion that there is no divergence at all and that the difference lies in the two survey questions. Something can be learned about survey design and question structure by assuming that the data are valid repeatable, statistical measurements. I've summarized the data sets in two tables which reflect the same basic results:

COMPARISON OF GALLUP AND ABC/WP POLLS MIDYEAR 2005

Gallup Poll	Percent		Percent	ABC/WP Poll
Legal under any circumstances?	23 %		20 %	Legal in all cases?
Legal under most circumstances?	12 %		36 %	Legal in most cases?
Legal only under a few circumstances?	40%		27 %	Illegal in most cases?
Illegal in all circumstances?	22 %		14 %	Illegal in all cases?

COMPARISON OF GALLUP AND ABC/WP POLLS YEAR-END 2005:

Gallup Poll	Percent		Percent	ABC/WP Poll
Legal under any circumstances?	26 %		17 %	Legal in all cases?
Legal under most circumstances?	16 %		40 %	Legal in most cases?
Legal only under a few circumstances?	39 %		27 %	Illegal in most cases?
Illegal in all circumstances?	16 %		13 %	Illegal in all cases?

DIVERGENCE? Do these data sets reveal a logical divergence such that one might have to doubt the integrity of one or the other pollster's methodology or data collection methods? I don't believe so because BOTH polls indicate that an overwhelming majority of the respondents, 80 % or more, think that SOME abortion should be legal (SOME includes ALL, MOST and FEW). Moreover, a plurality of respondents (less than or about equal to 40 %) think abortion should be legal only under a "few circumstances" but an equally large plurality think abortion should be legal in "most cases."

QUESTIONS AND SURVEY DESIGN: The apparent divergence in the results reveals something about the questions and survey design. It would be very interesting if respondents could be asked to list which "circumstances" justify a legal abortion and which do not. A half dozen categories of circumstances that cause pregnancies come to mind as possible answers: forgot the Pill; forgot the condom; juvenile experimentation; rape, incest, immaculate conception or got tricked into it, etc. In the case of ABC/WP --I wonder how many "cases" of potential abortion the respondents think there might be in any given year. The answers could range from hundreds of thousands to millions.

GALLUP: Both polls it seems to me have a built in "interrogative premise"-- Gallup is probing the question Do you believe that abortion is legally justified under SOME circumstances of unwanted pregnancy? Those who take the most extreme pro-life position will answer "No, it's never justified." Likewise, the most extreme pro-lifers say it is "Always justified." Based on the Gallup data, both groups are in the minority at about 20% or less. "Some circumstances" is a big category that covers ALL, MOST or FEW, which the Gallup survey then basically asks the respondents to select from if they have decided they do not belong to the NO category. The majority choice seems to be however, that abortion should be legal only in a few, if any, circumstances. This is interpreted as a "pro-life" stance.

The ABC/Washington Post Poll meanwhile uses the word "cases" which I think asks the respondents to consider the millions of women every year who might be considering abortion. ABC is probing public opinion on a question like "Do you believe that abortion should be legally available to some cases of women with unwanted pregnancies?" ABC presents a different choice though than Gallup: "Do you believe that abortion should be LEGAL for most or all of these women or should it be ILLEGAL for for most or all of them? The majority choice in the ABC poll is that abortion should be legal for most or all of the cases of women with abortable pregnancies. This is interpreted as a "pro-=choice" stance.

A POSSIBLE RESOLUTION: Is there a "divergence" between the two majority positions? I don't think so. People who think that abortion should be legal only for a few circumstances, (which I think translates to "causes of pregnancy" in the survey context), can still believe that most of the cases of women with unwanted pregnancy actually fall into those few categories of circumstance. In fact this seems to be a logical conclusion based on the fact the size of the apparent "pro-choice" majority in the ABC/WP poll is almost the same as the size of the "pro-life" majority in the Gallup poll, both being in the vicinity of 60%. If the data is accurate, the only conclusion we can come to is that both 60 percent slices of the population should be roughly congruent. The general impression one gets from the data interpreted this way, is that a slight majority of the public think it should be legal for most women with cases of unwanted pregnancy to choose abortion, but only if the circumstances that caused the pregnancy are among some limited but undefined set.

IS ONE POLL BETTER THAN THE OTHER? I think these two polls measure fundamentally different aspects of the public attitude towards abortion. Gallup probes people's thinking of what the Law should or should not allow based on the causes of unwanted pregnancy, while ABC/WP probes people's thinking on the rights of people with unwanted pregnancy to choose. One is asking about people's opinion on the Right to Life, while the other is asking about the Right to Choose. That they both seem to get a definite and consistent answer encourages the notion that the measurements are both accurate but that they xplore different territories of public opinion on the same large issue.

CAVEAT: Whenever statistics from separate categories of the possible answers to a given survey question are combined by addition or subtraction for analystical purposes, the inherent margin of error in the combined statistic is the SUM of the individual margins of error of the components. Thus in the ABC poll, where each statistic has a margin of error, or precision, of plus/minus 4 percent, the "percentage" of those who think abortion should be legal for ALL or MOST of the cases is 56% (midyear) and 57%(yearend) each with a margin of error or precision of plus/minus 8 percent. There's no free lunch when we start combining statistics for propaganda or conceptual purposes. The judgment of whether the observation of a particular combination leads to some definite conclusion about what the Majority thinks should always be tempered by the quantitative uncertainties in such statistical quantities.

A RELATED POST dealing with similar issues, but in the context of some Social Weather Stations polling commissioned by the Palace, is: Bunye & Goebbels, Masters of the Big Lie Technique