Tag Archives: cato

More marriage promotion failure evidence

broken lightbulb

Photo PNC / Flickr CC: https://flic.kr/p/9xNLad

I have another entry in the Cato Unbound forum on the Success Sequence. This is a response to a post by Brad Wilcox, which responded to my initial post, “The failure of the success sequence.”

With that context, I reprint it here.

In the second round of comments here, Brad Wilcox chose to focus on my argument that marriage promotion doesn’t work—that is, it doesn’t lead to more marriages. I have two brief responses to his comments.

First, Wilcox asserts that I have ignored salient evidence, and he mentions two studies. He writes:

But Cohen did not do justice to the existing literature on the HMI [Healthy Marriage Initiative] or of interventions like those used within it. For instance, he ignores evidence of modest success for the Oklahoma Marriage Initiative in fostering family stability (the longest running local effort working on this issue) and research that found that spending on the HMI was “positively associated with small changes in the percentage of married adults in the population” (italics in the original).

However, in my essay I linked to my book, Enduring Bonds: Inequality, Marriage, Parenting, and Everything Else That Makes Families Great and Terrible. There I dealt with the subject in much greater depth.

In particular, with regard to the claim that marriage promotion was associated with more marriage, the link is to this study (paywalled) in the journal Family Relations, by Alan Hawkins, Paul Amato, and Andrea Kinghorn. In my book I devote more than two pages to debunking this single study in detail. Since Wilcox appears not inclined to read my analysis in the book, I provide some key excerpts here:

[Hawkins, Amato, and Kinghorn] attempted to show that the marriage promotion money had beneficial effects at the population level.

They statistically compared state marriage promotion funding levels to the percentage of the population that was married and divorced, the number of children living with two parents or one parent, the nonmarital birth rate, and the poverty and near-poverty rates for the years 2000–2010. This kind of study offers an almost endless supply of subjective, post hoc decisions for researchers to make in their search for some relationship that passes the official cutoff for “statistical significance.” Here’s an example of one such choice these researchers made to find beneficial effects (no easy task, apparently): arbitrarily dividing the years covered into two separate periods. Here is their rationale: “We hypothesized that any HMI effects were weaker (or nonexistent) early in the decade (when funding levels were uniformly low) and stronger in the second half of the decade (when funding levels were at their peak).”

This is wrong. If funding levels were low and there was no effect in the early period, and then funding levels rose and effects emerged in the later period, then the analysis for all years should show that funding had an effect; that is the point of the analysis. This decision does not pass the smell test. Having determined that this decision would help them show that marriage promotion was good, they went on to report their beneficial effects, which were “significant” if you allowed them a 90 percent confidence (rather than the customary 95 percent, which is kosher under some house rules).

However, then they admitted their effects were significant only with Washington, D.C., included. Our nonstate capital city is a handy wiggle-room device for researchers studying state-level patterns; you can justify including it because it’s a real place, or you can justify excluding it because it’s not really a state. It turns out that the District of Columbia had per capita marriage promotion funding levels about nine times the average. With an improving family well-being profile during the period under study, this single case (out of fifty-one) could have a large statistical effect on the overall pattern. Statistical outliers are like the levers you learned about in physics—the further they are from the average, the more they can move the pile. To deal with this extreme outlier, they first cut the independent variable in half for D.C., bringing it down to about 4.4 times the mean and a third higher than the next most-extreme state, Oklahoma (itself pretty extreme). That change alone cut the number of significant effects on their outcomes down from six to three.

Then, performing a tragic coup de grâce on their own paper, they removed D.C. from the analysis altogether, and nothing was left. They didn’t quite see it that way, however: “But with the District of Columbia excluded from the data, all of the results were reduced to nonsignificance. Once again, most of the regression coefficients in this final analysis were comparable to those in Table 2 in direction and magnitude, but they were rendered nonsignificant by a further increase in the size of the standard errors.”

Really. These kinds of shenanigans give social scientists a bad name. (Everything that is nonsignificant is that way because of the [relative] size of the standard errors—that’s what nonsignificant means.) And what does “comparable in direction and magnitude” mean, exactly? This is the kind of statement one hopes the peer reviewers or editors would check closely. For example, with D.C. removed, the effect of marriage promotion on two-parent families fell 44 percent, and the effect on the poor/near-poor fell 78 percent. That’s “comparable” in the sense that they can be compared, but not in the sense that they are similar. Again, the authors helpfully explain that “the lack of significance can be explained by the larger standard errors.” That’s just another way of saying their model was ridiculously dependent on D.C. being in the sample and that removing it left them with nothing.

Oh well. Anyway, please keep giving the programs money, and us money for studying them: “In sum, the evidence from a variety of studies with different approaches targeting different populations suggests a potential for positive demographic change resulting from funding of [Marriage and Relationship Education] programs, but considerable uncertainty still remains. Given this uncertainty, more research is needed to determine whether these programs are accomplishing their goals and worthy of continued support.”

In short, this paper provides no evidence that HMI funding increased marriage rates or family wellbeing.

The other link Wilcox provides (“modest success for the Oklahoma Marriage Initiative”) goes to an essay on his website by the same Alan Hawkins. The evidence about Oklahoma’s “modest success” in that essay is limited to a broken link to another page on Wilcox’s site, and—I find this hard to even believe—an estimate of the effects of HMI funding in Oklahoma extrapolated from the paper I discussed above! That is, they took the very bad models from that paper and used them to predict how much the funding should have mattered in Oklahoma based on the level of funding there (and remember, Oklahoma was an outlier in that analysis). There was no estimate of the actual effect in Oklahoma. In fact, as I explained in a followup debunking, Oklahoma during this period experienced a greaterdecline in married-parent families than the rest of the country, even as they sucked up much more than their share of marriage promotion funds. This is, to put it mildly, not good social science. (The Oklahoma program, incidentally, is the subject of an excellent book by Melanie Heath: One Marriage Under God.)

Wilcox also argues that I am too demanding of federal programs, expecting demonstrable success. He concludes, “If the United States had adapted Cohen’s standard a half century ago, this would have resulted in the elimination of scores of federally funded programs that now garner hundreds of billions of dollars every year in public spending—from job training to Head Start.”

Amazingly, because Wilcox has made this argument before, I also addressed it in my book. Specifically, I wrote:

Of course, lots of programs fail. And, specifically, some studies have failed to show that kids whose parents were offered Head Start programs do better in the long run than those whose parents were not. But Head Start is offering a service to parents who want it, a service that most of them would buy on their own if it were not offered free. Head Start might fail at lifting children out of poverty while successfully providing a valuable, need-based service to low-income families.

As you can imagine, I am all for giving free marriage counseling to poor people if they want it (along with lots of other free stuff, including healthcare and childcare). And if they like it and keep using it, I might define that program as a success. But it’s not an antipoverty program.

Finally, in response to the idea that we just need more funding and more research to know if marriage promotion works, here’s my suggestion: in the studies testing marriage promotion programs, have a third group—in addition to the program and control group—who just get the cash equivalent to the cost of the service (a few thousand dollars). Then check to see how well the group getting the cash is doing compared with those getting the service. That’s the measure of whether this kind of policy is a success.


Filed under Me @ work

The failure of the success sequence

This essay was originally published as part of a forum on the success sequence sponsored by the Cato Institute, featuring Michael Tanner, Isabel Sawhill, and Brad Wilcox.

The success sequence is often (mistakenly) attributed to the 2009 book Creating an Opportunity Society by the Brookings Institution’s Ron Haskins and Isabel Sawhill. “First comes education,” they wrote. “Then comes a stable job that pays a decent wage, made decent by the addition of wage supplements and work supports if necessary. Finally comes marriage, followed by children.” They called for “marketing campaigns and educational programs to change social norms: to bring back the success sequence as the expected path for young Americans.”

The only issue here is marriage, as the rest is obvious to everyone. And in that regard this model of social change is wholly unproven and without precedent. Seat belt laws and anti-smoking campaigns, always cited by success sequence advocates, are not comparable. Those are daily habits easily addressed by legal regulations and tax policy (seat belts are required by law; with taxes, the price of cigarettes has more than tripled since 1980). The decline in marriage is a massive global trend driven by economic development and cultural adaptation. And the decline in teen pregnancy, to which success sequencers also point as a precedent for public information campaigns, flows with rather than against that underlying trend. As I detail in my new book, Enduring Bonds: Inequality, Marriage, Parenting, and Everything Else That Makes Families Great and Terrible, the drop in teen birth was part of the general increase in the age at which women have children, driven by the expansion of their educational and professional opportunities.

That idea of using public information campaigns to preach “marriage culture” echoed the futile proclamations of a previous generation. In a Hoover Institution symposium in 1996, former vice president Dan Quayle wrote that, “when it comes to strengthening families … we also desperately need help from nongovernment institutions like the media and the entertainment community.” Taking up the call with even more zeal, in 2001 Heritage Foundation fellow Patrick Fagan declared it was time to add three W’s to the common three R’s of schooling. “We need to stress something just as fundamental [as reading, writing, and arithmetic],” he wrote. “Call it the three W’s: work, wedlock and worship. … Put all three in the lives of parents and children, and they thrive.”* Five years later, another Heritage fellow said of the three W’s, “According to the social science data, if these three fundamentals are in place, government social policy is virtually unnecessary.” In 2012, the National Marriage Project, under director W. Bradford Wilcox, was again calling for “community-based and focused public service announcements” and a Hollywood “conversation” to promote marriage.

Meanwhile, slightly more liberal think tank denizens had discretely replaced “worship” with education, but they stuck to the basic idea that the problem with poor people is that they’re doing life wrong—and the “three somethings” formula. In a 2006 report for the National Campaign to Prevent Teen Pregnancy, Barbara Dafoe Whitehead and Marline Pearson wrote that it was time to “teach teens the rules of the success sequence,” which they defined as, “Finish high school, or better still, get a college degree; wait until your twenties to marry; and have children after you marry.” (Three things is a favorite formula of Chinese social engineers we well, as with Jiang Zemin’s “Three Represents” and Hu Jintao’s “Three Supremes”—but China combines such slogans with centralized education and state repression to increase their salience.)

Today, more than two decades after Quayle’s plea, 17 years after the three W’s, 12 years after the first “success sequence” proclamation, and one president after the National Marriage Project pitched its “President’s Marriage Agenda,” movement leaders are still calling for “Public and private social marketing campaigns on behalf of marriage and the ‘success sequence’,” to quote Wilcox and Wendy Wang’s latest report. Neither the policy nor the campaign to promote the policy have changed appreciably over the years, although the definition of the success sequence has varied from author to author. And in all this time, I could not find one academic study, outside of those published by think tanks, that seriously evaluates the claims of the success sequence.

What Could Go Wrong?

Today’s success sequence movement is puzzling in part because it fails to recognize—or admit—the extent to which its adherents already won. After the landmark 1996 welfare reform act, the federal government pumped more than $1 billion into national marriage promotion programs (the Healthy Marriage and the Responsible Fatherhood initiatives). This was cause for great celebration in the movement, as it should have been. In 2004, a Heritage Foundation report gushed, “The President’s Healthy Marriage Initiative is a future-oriented, preventive policy. It will foster better life-planning skills—encouraging couples to develop loving, committed marriages before bringing children into the world.”

It didn’t. The previous decade’s marriage promotion programs sent the same message the “success sequence” promoters do today. But where is the recognition that they failed? Rigorous evaluations of the marriage promotion efforts showed unequivocally that they produced no increase in marriage, not even among the people coerced into sitting for hours in relationship skills courses required to qualify for welfare benefits. As most readers probably know, in the years after welfare reform, marriage rates have continued to fall, and they have fallen fastest for those with less than a college education, the very population the programs were supposed to help. Even though pro-marriage billboards dotted the highways and FedEx delivered thousands of new-daddy care packages to hospitals. In fact, the only people more likely to marry after all these years of conservative activism are gays and lesbians. (This history is also reviewed in my book.)

Does this mean it’s bad advice to get an education, get a job, and find a permanent partner before having children? Of course not. But the success sequence is bad public policy, which is not the same thing at all. For public policy the question is, what will we accomplish with this money, compared with other things we could spend it on (or nothing at all)? Will the proposed campaigns have any positive effect on family outcomes? And if so, would they be better than some other way of spending money, like giving it to poor people, which is what most rich countries do, along with jobs, paid family leave, health care, and preschool education? Specifically, the rationale for spending money on these campaigns assumes that there are people who are on the fence about the success sequence, whose minds might be changed by the campaign, and that those altered decisions would lead to better outcomes in the future for those specific people. There is simply no evidence to support anything like that chain of events. Despite the ad nauseam repetition of the obvious fact that educated, employed, and (much less importantly) married people are less likely to be poor, there is no evidence at all that convincing people who are not one of those things of their importance will cause a reduction in poverty rates.

Given the well-documented desire of most young adults to finish high school, get a job, and get married—if the opportunity to follow that course presents itself—there is no reason to think the people reached by the proposed campaigns would not either already plan to follow the sequence or rightly suspect that it is not feasible for them. The decision to delay childbearing in hopes of marrying first rests on assumptions about the future—education, economics, relationships, health, stability—that the target population simply cannot makeabout their own destinies in today’s economic and social context. Improve the basic equation, the material expectations of young adults, and you won’t need a campaign to change behavior.

When women have more to lose, they delay parenthood. The college students in my classes, overwhelmingly women (I teach sociology of the family), almost all want to get married and then have children after they finish college. They understand that their marriage prospects will improve after college, and they don’t want children to interfere with their education or career launch. So, why shouldn’t we tell all women, especially those with poorer education and career prospects, to follow this course as well? Success sequencers believe it’s hypocritical to hoard this advice and only dispense it to the children of privilege. But you can’t wish away education, career, and marriage uncertainty or impose order on instability by force of will. If we’re not prepared to guarantee all women the same opportunities as those in my classes have, it’s not reasonable to demand the same attachment to the success sequence that those opportunities make feasible. In the absence of that guarantee, you’re simply asking, or requiring, poor people to delay (until “they’re ready,” in Sawhill’s terms, meaning not poor) or forego having children, one of the great joys of life, and something we should consider a human right.

In addition, what signals will a federal “success sequence” program send? What message will these campaigns send to people who are currently materially underserved by the welfare state, and people who don’t have the option to pursue the sequence because stable partners, education, or jobs aren’t available to them? What message will it send to the majority of Americans who are in a position to look down upon, and act against, those who become, in Sawhill’s chilling phrase, “norm breakers”?

And here race becomes especially salient. Black women have low marriage rates and black single mothers have high poverty rates. They face marriage markets with drastic shortages of eligible men, as Michael Tanner noted in the essay that opened this discussion. Not coincidentally, the history of welfare politics in the United States is intricately bound up with the history of racism against black women, who have been labeled pathological and congenitally dependent. The idea that delaying parenthood until marriage is a choice one makes is highly salient and prized by the white middle class, and the fact that black women often don’t have that choice makes them the objects of scorn for their perceived lax morals. The framing of the success sequence plays into this dynamic. For example, Ron Haskins has argued that welfare reform was needed to “[change] the values and the approach to life of people on welfare that they have to do their part.” The image of the poor welfare “taker” has a race and a gender in America.

In their book, Haskins and Sawhill proudly acknowledge that their cause was out of step with contemporary society. “To those who argue that this goal is old-fashioned or inconsistent with modern culture,” they wrote, “we argue that modern culture is inconsistent with the needs of children.” That may by a reasonable ideological position, but it’s no way to make public policy. The success sequence is a political meme repeated in highly similar form over more than a generation of public policy debates, without yet having any discernible impact for the better. The third “step” or “norm” in particular—marriage—has already been promoted with massive federal subsidies for almost two decades. The first two, education and jobs, are terrific ideas, obvious for good reasons, and not in need of much normative boosting, and we should turn our attention to improving the opportunity for more people to attain them.

* Thanks to Shawn Fremstad for this nugget.


Filed under Politics