Church saves marriage, and produces curious coefficients

Things that make you say… “peer review”?

This is the time of year when I expect to read inflated or distorted claims about the benefits of marriage and religion from the National Marriage Project. So I was happy to see the new State of Our Unions report put out by W. Bradford Wilcox’s outfit. My first reading led to a few questions.

First: When they do the “Survey of Marital Generosity” — the privately funded, self-described nationally-representative sample of 18-46-year old Americans, which is the source of this and several other reports, none of them published in any peer-reviewed source I can find — do they introduce themselves to the respondents by saying, “Hello, I’m calling from the Survey of Marital Generosity, and I’d like to ask you a few questions about…” If this were the kind of thing subject to peer review, and I were a reviewer, I would wonder if the respondents were told the name of the survey.

Second: When you see oddly repetitive numbers in a figure showing regression results, don’t you just wonder what’s going on?

Here’s what jumped out at me:

If a student came to my office with these results and said, “Wow, look at the big effect of joint religious practice on marital success,” I’d say, “Those numbers are probably wrong.” I can’t swear they didn’t get exactly the same values for everyone except those couples who both attend religious services regularly — 50 50 50, 13 13 13 , 50 50 50, 21 21 21 — in a regression that adjusts for age, education, income, and race/ethnicity, but that’s only because I don’t swear.*

Of course, the results are beside the point in this report, since the conclusions are so far from the data anyway. From this figure, for example, they conclude:

In all likelihood,  the experience of sharing regular religious attendance — that  is, of enjoying shared rituals that endow one’s marriage with transcendent significance and the support of a community  of family and friends who take one’s marriage seriously— is a solidifying force for marriage in a world in which family life is  increasingly fragile.

OK.

Anyway, whatever presumed error led to that figure seems to reoccur in the next one, at least for happiness:

Just to be clear with the grad student example, I wouldn’t assume the grad student was deliberately cooking the data to get a favorable result, because I like to assume the best about people. Also, people who cook data tend to produce a little more random-looking variation. Also, I would expect the student not to just publish the result online before anyone with a little more expertise had a look at it.

Evidence of a pattern of error is also found in this figure, which also shows predicted percentages that are “very happy,” when age, education, income and race/ethnicity are controlled.

Their point here is that people with lots of kids are happy (which they reasonably suggest may result from a selection effect). But my concern is that the predicted percentages are all between 13% and 26%, while the figures above show percentages that are all between 50% and 76%.

So, in addition to the previous figures probably being wrong, I don’t think this one can be right unless they are wrong. (And I would include “mislabeled” under the heading “wrong,” since the thing is already published and trumpeted to the credulous media.)

Publishing apparently-shoddy work like this without peer review is worse when it happens to support your obvious political agenda. One is tempted to believe that if the error-prone research assistant had produced figures that didn’t conform to the script, someone higher up might have sent the tables back for some error checking. I don’t want to believe that, though, because I like to assume the best about people.

* Just kidding. I do swear.

7 Comments

Filed under Me @ work

7 responses to “Church saves marriage, and produces curious coefficients

  1. j

    Another possibility is that, sometimes, when folks graph coefficients, they plug in zeroes for the Nonsig coefficients (instead of putting in the value of the coefficient). I’m not saying this is the best practice, but it is done, and could explain the consistent numbers.

    Like

  2. What J said.

    I can replicate figure 12, based on the additional information provided in http://www.stateofourunions.org/2011/e-ppendix.php. (1) Run a logistic regression model with all of the predictors. (2) “Adjust” for the controls by ignoring their effect on predicted probabilities, even the significant ones. (3) Note that your intercept and coefficients for husband and wife attendance are not significant, so ignore them when computing predicted probabilities. (4) Since you are left with one coefficient (Both attend weekly), your predicted value is simply the inverse logit of this coefficient times the value. So invlogit(.56*0)=.5 for everyone who isn’t attending weekly with their spouses and invlogit(.56*1)=.636 for everyone who is. This is the wrong way to do it, and especially wrong for logit models and when there is an interaction effect. Wrong with the logit models because the logit transformation means that the same coefficient value has different effects on predicted probabilities depending on the value of other coefficients and the underlying mean/intercept. Wrong with the interaction effect because you are ignoring the value of their individual attendance.

    They note that, “If participants reported attending religious worship services at least once every week, they were coded as attending weekly. If they both reported that they jointly attended they were coded as both attending.” As such, the “true” impact of attending service is your individual effect plus the interaction effect. For women, this is .15+.56. For men, it is -.12+.56. Based on this (and assuming that the rest of their model is correct and that a better intercept–based on the other figures–is -.2 and not 0), my ball park estimate is something like a 45% happiness for those who don’t attend services, 50% for women who attend without their husbands, 42% for men who attend without their wives, and 62% for wives who attend with their husbands, and 56% for husband who attend with their wives. So people who do things with their spouses like their spouses more than people who do things without their spouses.

    And we will ignore for now while “martial satisfaction” is a dichotomous measure constructed from splitting the top value from the rest (five point scale), while “sexual satisfaction and “god at the center of marriage” are dichotomous measures constructed from splitting the top twos values from the rest (also five point scale) and “High Divorce Proneness” is the top four values of a ten point scale.

    Like

    • Wow, that’s great (what you did, not what they did). Thank you!

      I’m so appalled by this, though of course it’s just the tip of the iceberg.

      This is the list of members of the National Marriage Project Board of Advisors. There are some actual social scientists among them. Do they have anything to do with this? Do they even read this stuff?

      Richard Campanelli, Esq Attorney
      William J. Doherty University of Minnesota
      Kathryn Edin Harvard University
      Robert Emery University of Virginia
      Bill Galston The Brookings Institution
      Neil Gilbert University of California at Berkeley
      Ron Haskins The Brookings Institution
      Linda Malone-Colon Hampton University
      Elizabeth Marquardt Institute for American Values
      David G. Myers Hope College
      Isabel Sawhill The Brookings Institution
      Scott Stanley University of Denver
      Linda J. Waite University of Chicago
      James Q. Wilson UCLA (emeritus)
      G. Sim Johnson Author
      David Popenoe Rutgers University (emeritus)
      Christopher Ellison University of Texas at San Antonio

      Like

  3. Pingback: The Five Secrets of Happily Married Parents – TIME Healthland

  4. Brad Wilcox

    Readers who would like to see a response to Professor Cohen’s comments may go here: http://familyscholars.org/2011/12/12/explaining-the-figures-in-latest-state-of-our-unions/

    Like

  5. Pingback: National Marriage Project Does S***ty Research Again; People Always Thought Students Couldn’t Write; and More « Welcome to the Doctor's Office

  6. Pingback: Fatherhood’s transformations: What if someone checked the facts? | Family Inequality

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s