Tag Archives: Penn State

Marriage promotion: That’s some fine print

In a (paywalled) article in the journal Family Relations, Alan Hawkins, Paul Amato, and Andrea Kinghorn, attempt to show that $600 million in marriage promotion money (taken from the welfare program!) has had beneficial effects at the population level. A couple quick comments on the article (see also previous posts on marriage promotion).

After a literature review that is a model of selective and skewed reading of previous research (worth reading just for that), they use state marriage promotion funding levels* in a year- and state-fixed effects model to predict the percentage of the population that is married, divorced, children living with two parents, one parent, nonmarital births, poverty and near-poverty, each in separate models with no control variables, for the years 2000-2010 using the American Community Survey.

To find beneficial effects — no easy task, apparently — they first arbitrarily divided the years into two periods. Here is the rationale for that:

We hypothesized that any HMI [Healthy Marriage Initiative] effects were weaker (or nonexistent) early in the decade (when funding levels were uniformly low) and stronger in the second half of the decade (when funding levels were at their peak).

This doesn’t make sense to me. If funding levels were low and there was no effect in the early period, and then funding levels rose and effects emerged in the later period, then the model for all years should show that funding had an effect. Correct me if I’m wrong, but I don’t think this passes the smell test.

Then they report their beneficial effects, which are significant if you allow them p<.10 as a cutoff, which is kosher under house rules because they had directional hypotheses.

However, then they admit their effects are only significant because they included Washington, DC. That city had per capita funding levels about 9-times the mean (“about $22” versus “about $2.50”), and had an improving family well-being profile during the period (how much of an outlier DC is on the dependent variables they didn’t discuss, and I don’t have time to show it now, but I reckon it’s pretty extreme, too). To deal with this extreme outlier, they first cut the independent variable in half for DC, bringing it down to about 4.4-times the mean and a third higher then the next most-extreme state, Oklahoma (itself pretty extreme). That change alone cut the number of significant effects down from six to three.

coupdegrace

Then, in the tragic coup de grâce of their own paper, they remove DC from the analysis, and nothing is left. They don’t quite see it that way, however:

But with the District of Columbia excluded from the data (right panel of Table 3), all of the results were reduced to nonsignificance. Once again, most of the regression coefficients in this final analysis were comparable to those in Table 2 (right panel) in direction and magnitude, but they were rendered nonsignificant by a further increase in the size of the standard errors.

Really. What is “comparable in direction and magnitude” mean, exactly? I give you (for free!) the two tables. First, the full model:

tab2

Then, the models with DC rescaled or removed (they’re talking about the comparison between the right-hand panel in both tables):

tab3

Some of the coefficients actually grew in the direction they want with DC gone. But two moved drastically away from the direction of their preferred outcome: the two-parent coefficient is 44% smaller, the poor/near-poor coefficient fell 78%.

Some outlier! As they helpfully explain, “The lack of significance can be explained by the larger standard errors.” In the first adjustment, rescaling DC, all the standard errors at least doubled. And all of the standard errors are at least three-times larger with DC gone. I’m not a medical doctor, but I think it’s fair to say that when removing one case triples your standard errors, your regression model is not feeling well.

One other comment on DC. Any outlier that extreme is a serious problem for regression analysis, obviously. But there is a substantive issue here as well. They feebly attempt to turn the DC results in their favor, by talking about is unique conditions. But what they don’t do is consider the implications of DC’s unique change over this time for their analysis. And that’s what matters in a year- and state-fixed effects model. How did DC change independently of marriage promotion funds? Most importantly, 8% of the population during 2006-2010 was new to town each year. That’s four-times the national average of in-migration in that period. This churning is of course a problem for their analysis, which is trying to measure cumulative effects of program spending in that place — hard to do when so many people moved there after the spending occurred. But it’s also not random churning: the DC population went from 57% Black to 52% Black in just five years. DC is changing, and it’s not because of marriage promotion programs.

Finally, their own attempt at a self-serving conclusion is the most damning:

Despite the limitations, the current study is the most extensive and rigorous investigation to date of the implications of government-supported HMIs for family change at the population level.

Ouch. Oh well. Anyway, please keep giving the programs money, and us money for studying them**:

In sum, the evidence from a variety of studies with different approaches targeting different populations suggests a potential for positive demographic change resulting from funding of [Marriage and Relationship Education] programs, but considerable uncertainty still remains. Given this uncertainty, more research is needed to determine whether these programs are accomplishing their goals and worthy of continued support.

*The link to their data source is broken. They say they got other data by calling around.

**The lead author, Alan Hawkins, has received about $120,000 in funding from various marriage promotion sources.

28 Comments

Filed under Research reports

How many Black scholars does it take to have any Black scholars?

I had a very nice time at the 21st Annual Symposium on Family Issues at Penn State University, where I presented remarks in response to a paper by Sara McLanahan and Wade Jacobsen. The theme of the symposium was “Diverging Destinies,” or the growing differences in family experiences by social class in the US. The event has lots of time for discussion and debate, and much of that focused on poor people and their families, around contested terms such as choices, parenting, behavior, attitudes, orientation, and so on. I had plenty to agree and disagree with, there were lots of good talks, and it was a good conversation.

The scenic Nittany Lion Inn (photo by me)

The scenic Nittany Lion Inn (photo by me)

Here are two observations.

The first was a moment when Ron Haskins from the Brookings Institution, a long-time member of the welfare policy establishment (his bio describes him as “instrumental in the 1996 overhaul of national welfare policy”), responded to Harvard professor Kathryn Edin’s response to his presentation. She had spent most of her time talking about her new book, Doing the Best I Can: Fathering in the Inner City. For the book, Edin undertook years of ethnographic fieldwork and interviews, and emerged with a very sympathetic yet sobering description of the poor young men she studied, men who want more than anything to be good fathers — according to the contemporary ideals of both economic provision and emotional togetherness — but for many reasons usually can’t meet their own goals.

When they were both on the dais, Haskins said she was “too optimistic” about her subjects, in describing them as eager to do the right thing for their children. “I know these guys!” he said, before describing some anecdotal experiences from his (apparently distant) personal past. It struck me because it seemed profoundly disrespectful of not only her work, but of her kind of research. Of course ethnographers can do bad studies or misinterpret their data. But I would only discount a serious work of ethnography based on my personal experience if that experience were pretty deep. I suspect Haskins wouldn’t have struck that note if her work had been a quantitative demography, but I could be wrong. (Earlier, I had pointed out that welfare reform failed at its stated goal of making poor single mothers get married, and he countered that it had been successful at getting them to work, so “behavior modification does work” — and we should use that program as a model for future work-mandating reforms.)

Who's on that dais?

Who’s on that dais?

Anyway, the second observation was about the composition of the speakers. None of our 16 speakers this year was Black. When I grumbled about that on Facebook, someone said he felt the same way last year. That got me to check the previous programs. (Each year the organizers of the symposium produce a book from the papers — you can see previous editions here, where the contributors are all listed.) I had to go back to 2008 to find an African American speaker, according to my reading of their photos and bios (which is not the best way to identify race/ethnicity, obviously, so I maybe wrong). Overall, of the last 114 speakers going back to 2007, I think only one was Black.

I don’t know who decides on the topics or the invitations, or how the event has unfolded over time, so I can’t comment on the process or motivations of those involved. But I think this is not good. The symposium is a substantial endeavor, with grant money from various sources. An invitation to speak there is a line on your CV, it comes with a small honorarium and travel expenses, and it’s a chance to network with other family researchers, grant-makers, and policy people. There also are a lot of students attending the talks. So whatever the reasons, it’s a shame more Black scholars haven’t been there.

16 Comments

Filed under Me @ work