Marriage promotion: That’s some fine print

In a (paywalled) article in the journal Family Relations, Alan Hawkins, Paul Amato, and Andrea Kinghorn, attempt to show that $600 million in marriage promotion money (taken from the welfare program!) has had beneficial effects at the population level. A couple quick comments on the article (see also previous posts on marriage promotion).

After a literature review that is a model of selective and skewed reading of previous research (worth reading just for that), they use state marriage promotion funding levels* in a year- and state-fixed effects model to predict the percentage of the population that is married, divorced, children living with two parents, one parent, nonmarital births, poverty and near-poverty, each in separate models with no control variables, for the years 2000-2010 using the American Community Survey.

To find beneficial effects — no easy task, apparently — they first arbitrarily divided the years into two periods. Here is the rationale for that:

We hypothesized that any HMI [Healthy Marriage Initiative] effects were weaker (or nonexistent) early in the decade (when funding levels were uniformly low) and stronger in the second half of the decade (when funding levels were at their peak).

This doesn’t make sense to me. If funding levels were low and there was no effect in the early period, and then funding levels rose and effects emerged in the later period, then the model for all years should show that funding had an effect. Correct me if I’m wrong, but I don’t think this passes the smell test.

Then they report their beneficial effects, which are significant if you allow them p<.10 as a cutoff, which is kosher under house rules because they had directional hypotheses.

However, then they admit their effects are only significant because they included Washington, DC. That city had per capita funding levels about 9-times the mean (“about $22″ versus “about $2.50″), and had an improving family well-being profile during the period (how much of an outlier DC is on the dependent variables they didn’t discuss, and I don’t have time to show it now, but I reckon it’s pretty extreme, too). To deal with this extreme outlier, they first cut the independent variable in half for DC, bringing it down to about 4.4-times the mean and a third higher then the next most-extreme state, Oklahoma (itself pretty extreme). That change alone cut the number of significant effects down from six to three.

coupdegrace

Then, in the tragic coup de grâce of their own paper, they remove DC from the analysis, and nothing is left. They don’t quite see it that way, however:

But with the District of Columbia excluded from the data (right panel of Table 3), all of the results were reduced to nonsignificance. Once again, most of the regression coefficients in this final analysis were comparable to those in Table 2 (right panel) in direction and magnitude, but they were rendered nonsignificant by a further increase in the size of the standard errors.

Really. What is “comparable in direction and magnitude” mean, exactly? I give you (for free!) the two tables. First, the full model:

tab2

Then, the models with DC rescaled or removed (they’re talking about the comparison between the right-hand panel in both tables):

tab3

Some of the coefficients actually grew in the direction they want with DC gone. But two moved drastically away from the direction of their preferred outcome: the two-parent coefficient is 44% smaller, the poor/near-poor coefficient fell 78%.

Some outlier! As they helpfully explain, “The lack of significance can be explained by the larger standard errors.” In the first adjustment, rescaling DC, all the standard errors at least doubled. And all of the standard errors are at least three-times larger with DC gone. I’m not a medical doctor, but I think it’s fair to say that when removing one case triples your standard errors, your regression model is not feeling well.

One other comment on DC. Any outlier that extreme is a serious problem for regression analysis, obviously. But there is a substantive issue here as well. They feebly attempt to turn the DC results in their favor, by talking about is unique conditions. But what they don’t do is consider the implications of DC’s unique change over this time for their analysis. And that’s what matters in a year- and state-fixed effects model. How did DC change independently of marriage promotion funds? Most importantly, 8% of the population during 2006-2010 was new to town each year. That’s four-times the national average of in-migration in that period. This churning is of course a problem for their analysis, which is trying to measure cumulative effects of program spending in that place — hard to do when so many people moved there after the spending occurred. But it’s also not random churning: the DC population went from 57% Black to 52% Black in just five years. DC is changing, and it’s not because of marriage promotion programs.

Finally, their own attempt at a self-serving conclusion is the most damning:

Despite the limitations, the current study is the most extensive and rigorous investigation to date of the implications of government-supported HMIs for family change at the population level.

Ouch. Oh well. Anyway, please keep giving the programs money, and us money for studying them**:

In sum, the evidence from a variety of studies with different approaches targeting different populations suggests a potential for positive demographic change resulting from funding of [Marriage and Relationship Education] programs, but considerable uncertainty still remains. Given this uncertainty, more research is needed to determine whether these programs are accomplishing their goals and worthy of continued support.

*The link to their data source is broken. They say they got other data by calling around.

**The lead author, Alan Hawkins, has received about $120,000 in funding from various marriage promotion sources.

23 Comments

Filed under Research reports

23 responses to “Marriage promotion: That’s some fine print

  1. vijay

    Why does the academy deride straight marriage and its proponents continuously (by asking for more and more assistance for families of unmarried people) but sings a different tune for gay marriage? In real life, most of the academia is happily married with two earners and minimal number of children. I can only conclude that this is because that this has no impact on the families of academia.

    The same criticism is true for economics; the academia has been a relentless cheerleader for globalization, economic outsourcing, and endless migration, because they are aware that it has limited impact on their families.

  2. Phil and Carolyn Cowan

    We’re concerned about the tone of the current rhetoric in discussions of “marriage promotion.” It’s true that the Bush Administration, some politicians, and some advocates like Brad Wilcox are advocating marriage as a solution to family poverty. But as a result of much discussion and re-thinking of that original moral imperative, much government money and some foundation money has actually been spent in support of interventions that strengthen the relationships between parents, regardless of whether they are married, cohabiting, separated, or divorced. I fail to see what’s wrong with that idea.
    The Building Strong Families intervention with about 4,500 families has been chosen as the poster-child example of a program that didn’t work. As you reported, there were no overall statistically significant differences between intervention and control participants. But the attendance at this 16-week program averaged around 35%. In fact, in 45% of the more than 2200 intervention participants, neither spouse attended even one group meeting. By contrast, in the only one of 8 sites in the unmarried Building Strong Families study that show significant effects of the intervention, the attendance rate was 70+ %. Furthermore. the analysis was an “intention to treat” strategy in which all couples assigned to the intervention condition were included in the intervention group, whether or not they participated in a single session. No wonder there was no detectable impact. What would it be like if a Head Start Program was declared a failure on the basis of a study in which the average attendance of the children was 35% with almost half of them never entering the Head Start door?
    Instead of picking on this already-battered program, a more thorough approach would be to consider the studies showing that good programs for couples, married or not, middle or low-income, can have positive effects on their relationships and on their children. For example, the Strengthening Healthy Marriage program for low-income married couples found small but significant effects on couple relationship quality and on children’s behavior. This is also a federally funded program, but there others funded by national, state, and local governments.
    We’re not quarreling with your analysis of the Hawkins et al article. We’re concerned about the fact that in this and other blogs, critics are failing to distinguish between advocates of marriage promotion and those of us trying to help strengthen relationships within families. The divorce rate is still close to 50%. If there was a medical disease that affected 50% of the population, wouldn’t you think that the government might take a role in preventing it from taking a toll on the population? Because there are so many who are worrying about the fragility of family relationships, and the negative impact of warring couples on their children’s development, we’re urging a more nuanced discussion of these issues rather than labeling and dismissing the whole enterprise of trying to strengthen families as marriage promotion.

    • Thank you very much for the comment. I am all for a positive tone. Also honesty, integrity, and diligence in research. Your comment stands on its own, but I have a few short responses:

      1. I have no problem with programs to support relationships. I do have a problem with spending welfare program money on that — and this is hundreds of millions of dollars — which was the ideological brainchild of the moralizing welfare reform, under the stated goal of promoting marriage. Here’s my suggestion: let poor people choose between cash and a heavily discounted relationship support program (e.g., $500 versus $2,000 worth of relationship support). As it is the implementation has been largely coercive and/or punitive, either taking money from welfare, or requiring people to attend relationship programs in order to get the money they need.

      2. The “intent to treat” research design is vitally important. Otherwise selection into the program undermines randomization completely. The failure of the programs to get people to participate is part of their failure. Those cases absolutely belong in the treated group. I don’t know anything about evaluations of Head Start (though people keep telling me I need to look at that – I don’t need to know about that in order to draw conclusions about Healthy Marriage.

      Thanks again!

      • Phil and Carolyn Cowan

        We agree that intent to treat designs are important and that the failure to get people to participate is part of the program failure. But it doesn’t mean that couple relationship strengthening interventions are failures. That’s part of our point. Supporting Healthy Marriage and other programs, including ours, have higher participation rates, improve family relationships, and benefit children. This point keeps getting lost when all of these efforts are described as “marriage promotion.”

        As to your point about letting families choose between cash and a heavily discounted relationship support program. The parents will probably choose cash, but the children who could benefit longterm from more collaborative parenting wouldn’t have a voice in this decision.

        • So this isn’t about helping married couples after all – this is the same right wing ideological framing of morally corrupt selfish poor people vs innocent abused babies .
          Also, how are gay families addressed in your program? You claim to distance your self from Wilcox et al but where is the evidence? Do you have a non-discrimination policy for sexual orientation and religion? Bush used these programs to funnel money to religious and right wing groups – it was basically political patronage. How do you address transparency issues to avoid the faults of previous efforts?
          It’s all well and good to claim you have changed but when science is corrupted by the likes of Wilcox, Regnerus and now apparently Amato in such blatant ways, it doesn’t inspire confidence.

        • Correction – my first sentence should have read just “families” not “married families”

  3. Pingback: Controversial claims about marriage promotion break the statistical rules of evidence

  4. Pingback: Today’s Links | VoteforAlice.com

  5. Pingback: Marriage promotion: That’s some fine print | Red Latinoamericana de Estudios de Familia - LAFAM

  6. For the readers who don’t know who Hawkins is, he is the guy who co-authored the “Get Rosenfeld” paper along with that economist woman PhD from Ava Maria University and finger snap, finger snap, the name just won’t come to me, the Canadian Economist who is an expert for the anti gay Ruth Institute. They wrote the infamous 3 economists paper attacking Demographer Rosenfeld out of Stanford’s paper that used the US census and showed no impact on education attainment of children with 2 gay parents.

    They sent their “Get Rosenfeld” paper to Regnerus to review and Regnerus complimented them on what a great idea they had, and wished he would have thought of it. That is who economist Hawkins is. So what is Sociologist/Demographer Paul Amato doing co-authoring a paper with Brigham Young Economist Hawkins? You tell me if this is frequent or not, sociologists & economists co-authoring papers.

    I’m sure everyone who reads this blog already knows that Paul Amato famously agreed to be a paid consultant to the Regnerus paper, agreed to be an “independent” peer reviewer and for the coup de grace also wrote an introductory article on the Regnerus Research. Hawkins I’m sure signed the “We support Regnerus” letter published on the Baylor University Religious Studies web page.

    Considering what I know of two out of the three co-authors I would dig through their findings with a fine toothed comb. We cannot let these right wingers take over family studies research, because they have already pre-determined that the ONLY way to have successful families and well adjusted children is 1Man + 1 Woman for life.

    I am very gun shy of social science research since I have learned more & more about it.

    • etseq

      Catherine Pakaluk is the economist you are thinking of but I’m pretty sure it was Joseph Price who was the third co-author rather than Hawkins. Both Price and Hawkins are BYU faculty but they have both signed onto to briefs in anti-gay litigation….

      • Phil Cowan

        And Alan Hawkins has his degree in Human Development and Family Relations, not in Economics. I’m not endorsing his views on gay marriage — just trying to keep the record straight and avoid McCarthy-like tactics of tarring people by their labels or associations.

        • etseq

          McCarthy-like? You do realize that McCarthy was a raving homophobe and gays along with commies were the subject of his witchhunts. Your personal views on gay marriage are irrelevant but you seem to lack any historical and moral perspective on matters when you portray religious conservatives as the victims of some paranoid gay conspiracy. You agree that Phillip was correct in his criticism of this paper as co-authored by Hawkins so on what basis are you claiming he is being tarred?
          Anyway, there was a good article in the Chronicle on this very matter

          http://chronicle.com/article/The-Great-Mom-Dad-Experiment/144027/

  7. I stand corrected. I’m at a disadvantage because I’m traveling and using my iPad instead of my regular computer with all my files on it. I know this Hawkins though, I wish I had my files with me. see that is what we have to do now, create files and keep track of the main anti gay academics.

    Duly noted the comment about McCarthyism, however if you were in the struggle for Equal Civil Rights for Sexual Minorities you would realize how these anti gay academics are involved in influencing Federal and State Court cases. They file anti gay amicus briefs for example to the United States Supreme Court, we HAVE to keep track of them, we ignore them at our peril. And Historically, historically the people pushing Civil Marriage as a panacea to all that ails us, have barred sexual minorities entrance. I know that Alan Hawkins name, I know I have that in my files, maybe he is an Advisory Editor of the Journal Social Science Research?

    If you academics just talked amongst yourselves I would be delighted to never read another research article. But you don’t, you are trying to influence public policy, public policy that produces winners and losers so perhaps you should reconsider use of the term McCarthism. What should gay rights advocates do? Ignore you, ignore the social scientists who pump out invalid research which is then testified to in court? You do know that somebody paid for example Mark Renerus to fly out to Hawaii and to testify to the Hawaii Legislature that they should not pass Civil Marriage for Sexual Minorities as the research shows having gay patents is bad for children. You know that, right? And the co-author of this here paper we are discussing Paul Amato was in that Regnerus band. Did you read the e-mails that were had via Freedom of Information. Act requests which shows just how sleazy this area of Family Studies research is? Dr. Cohen are you uniformed of the political reality of the influence of certain social scientists and why we keep track of them now?

  8. That last sentence should read “Dr. Cowan are you uninformed….”
    NOT Dr. Cohen, the author of this blog.

  9. I found what I could not put my finger on regarding that Alan Hawkins. He signed onto the Amicus Brief that was filed to our United States a Supreme Court in the DOMA case. He’s out of Penn State, Eggebeen who is also out of Penn State, as is Paul Amato Penn State, Hawkins & Eggebeen signed onto the Amicus Brief to the Supreme Court. Eggebeen also along with Paul Amato wrote a glowing article in the Journal when they published that Regnerus article. Are we supposed to ignore these “associations”.
    Here is a link to brief

    • etseq

      …its almost like there is a small but vocal clique of conservative christian “scholars” who manufacture ideological self-referential “social science” for political purposes. It doesn’t hurt that there is alot of money – both private and federal – to be made along the way! Mentioning these facts, however, is considered bad form and triggers hysterical reactions like “McCarthyism” and “Auto-da-fe” along with a recitation of academic bona fides.

  10. Dr Cohen, can you kindly list the funders of this study and the amounts?
    Thank you

  11. Everyting iss vry open with a very cleaar
    explanation of the issues. It was definitely informative. Your site is very helpful.

    Many thanks ffor sharing!

  12. Pingback: This ‘Supporting Healthy Marriage,’ I do not think it means what you think it means | Family Inequality

  13. Pingback: Response from Supporting Healthy Marriage supporters, with responses | Family Inequality

  14. Pingback: Does gay marriage make straight men hate children? | Family Inequality

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s