Marriage promotion: That’s some fine print

In a (paywalled) article in the journal Family Relations, Alan Hawkins, Paul Amato, and Andrea Kinghorn, attempt to show that $600 million in marriage promotion money (taken from the welfare program!) has had beneficial effects at the population level. A couple quick comments on the article (see also previous posts on marriage promotion).

After a literature review that is a model of selective and skewed reading of previous research (worth reading just for that), they use state marriage promotion funding levels* in a year- and state-fixed effects model to predict the percentage of the population that is married, divorced, children living with two parents, one parent, nonmarital births, poverty and near-poverty, each in separate models with no control variables, for the years 2000-2010 using the American Community Survey.

To find beneficial effects — no easy task, apparently — they first arbitrarily divided the years into two periods. Here is the rationale for that:

We hypothesized that any HMI [Healthy Marriage Initiative] effects were weaker (or nonexistent) early in the decade (when funding levels were uniformly low) and stronger in the second half of the decade (when funding levels were at their peak).

This doesn’t make sense to me. If funding levels were low and there was no effect in the early period, and then funding levels rose and effects emerged in the later period, then the model for all years should show that funding had an effect. Correct me if I’m wrong, but I don’t think this passes the smell test.

Then they report their beneficial effects, which are significant if you allow them p<.10 as a cutoff, which is kosher under house rules because they had directional hypotheses.

However, then they admit their effects are only significant because they included Washington, DC. That city had per capita funding levels about 9-times the mean (“about $22” versus “about $2.50”), and had an improving family well-being profile during the period (how much of an outlier DC is on the dependent variables they didn’t discuss, and I don’t have time to show it now, but I reckon it’s pretty extreme, too). To deal with this extreme outlier, they first cut the independent variable in half for DC, bringing it down to about 4.4-times the mean and a third higher then the next most-extreme state, Oklahoma (itself pretty extreme). That change alone cut the number of significant effects down from six to three.

Then, in the tragic coup de grâce of their own paper, they remove DC from the analysis, and nothing is left. They don’t quite see it that way, however:

But with the District of Columbia excluded from the data (right panel of Table 3), all of the results were reduced to nonsignificance. Once again, most of the regression coefficients in this final analysis were comparable to those in Table 2 (right panel) in direction and magnitude, but they were rendered nonsignificant by a further increase in the size of the standard errors.

Really. What is “comparable in direction and magnitude” mean, exactly? I give you (for free!) the two tables. First, the full model:

Then, the models with DC rescaled or removed (they’re talking about the comparison between the right-hand panel in both tables):

Some of the coefficients actually grew in the direction they want with DC gone. But two moved drastically away from the direction of their preferred outcome: the two-parent coefficient is 44% smaller, the poor/near-poor coefficient fell 78%.

Some outlier! As they helpfully explain, “The lack of significance can be explained by the larger standard errors.” In the first adjustment, rescaling DC, all the standard errors at least doubled. And all of the standard errors are at least three-times larger with DC gone. I’m not a medical doctor, but I think it’s fair to say that when removing one case triples your standard errors, your regression model is not feeling well.

One other comment on DC. Any outlier that extreme is a serious problem for regression analysis, obviously. But there is a substantive issue here as well. They feebly attempt to turn the DC results in their favor, by talking about is unique conditions. But what they don’t do is consider the implications of DC’s unique change over this time for their analysis. And that’s what matters in a year- and state-fixed effects model. How did DC change independently of marriage promotion funds? Most importantly, 8% of the population during 2006-2010 was new to town each year. That’s four-times the national average of in-migration in that period. This churning is of course a problem for their analysis, which is trying to measure cumulative effects of program spending in that place — hard to do when so many people moved there after the spending occurred. But it’s also not random churning: the DC population went from 57% Black to 52% Black in just five years. DC is changing, and it’s not because of marriage promotion programs.

Finally, their own attempt at a self-serving conclusion is the most damning:

Despite the limitations, the current study is the most extensive and rigorous investigation to date of the implications of government-supported HMIs for family change at the population level.

Ouch. Oh well. Anyway, please keep giving the programs money, and us money for studying them**:

In sum, the evidence from a variety of studies with different approaches targeting different populations suggests a potential for positive demographic change resulting from funding of [Marriage and Relationship Education] programs, but considerable uncertainty still remains. Given this uncertainty, more research is needed to determine whether these programs are accomplishing their goals and worthy of continued support.

*The link to their data source is broken. They say they got other data by calling around.

**The lead author, Alan Hawkins, has received about $120,000 in funding from various marriage promotion sources.