Bad News About Polimetrics

Mike Kellerman has some bad news about the utiity of econometrics:

Social Science Statistics Blog: Another paradox of turnout? (Part I) : Those of you who have followed this blog know that making reasonable causal inferences from observational data usually presents a huge challenge.... [I]t is worth mentioning "Comparing experimental and matching methods using a large-scale voter mobilization experiment" by Kevin Arceneaux, Alan Gerber, and Donald Green, which appears in the new issue of Political Analysis.... [T]hey base their analysis on a voter turnout experiment in which households were randomly selected to receive non-partisan phone calls encouraging them to vote.... This type of mobilization experiment suffers from a classic compliance problem; some voters either don't have phones or refuse to take unsolicited calls. As a result, in order to determine the average causal effect of the treatment on those who would receive it, they need to find a method to compare the compliers who received treatment to compliers in the control group. Since assignment to treatment was randomly assigned, they use assignment as an instrument in the spirit of Angrist, Imbens, and Rubin (1996). Using a 2SLS regression with assignment in the first stage, their estimates of the ATT are close to zero and statistically insignificant.... [T]heir experimental estimate... we might call the "truth".

The authors then attempt to replicate their experimental results using both OLS and various matching techniques. In this context, the goal of the matching process is to pick out people who would have listened to the phone call had they been contacted. The authors have a set of covariates on which to match, including age, gender, household size, geographic location, whether the voter was newly registered, and whether the voter turned out in each of the two previous elections.... [T]hey don't have much difficulty in finding close matches for the treated group based on the covariates in their data. Unfortunately, the matching estimates don't turn out to be very close to the experimental baseline, and in fact are much closer to the plain-vanilla OLS estimates.

Their conclusion from this result is that the assumptions necessary for causal inferences under matching (namely, unconfoundedness conditional on the covariates) are not met in this situation, and (at least by my reading) they seem to suggest that it would be difficult to find a dataset that was rich enough in covariates that the assumption would be met.

As a political scientist, I have to say that I like this dataset, because (a) it is not the NSW dataset and (b) it is not derived from a labor market experiment. What do these results mean for matching methods in political science?...