This is just a quick post to get down some of the obvious, fatal flaws in Douglas Allen’s new paper in the Review of Economics of the Household, “High school graduation rates among children of same-sex households.” (The article is paywalled, but since you’re a personal friend I will loan you my electronic copy.)
The short story is Allen reports children of gay and lesbian couples are less likel to graduate high school. But you can’t use this kind of data to answer this kind of question. So the results in the paper are meaningless. Here are the notes behind that conclusion.
Boost that sample
Like Regnerus, Allen talks about the benefits of a big, random, national sample. In praising the study, Mark Regnerus calls the dataset “massive.” (Regnerus background here.) However, like Regnerus before him, Allen ends up with a tiny sample of people from gay- or lesbian-parent households, and makes bad decisions to increase its size.
Allen says the law doesn’t permit him to release the sample size, but it’s a 20% file which should mean each respondent on average represents five people in the Canadian population. With a weighted population of gay-father kids of 423, and lesbian-mother kids of 969, that means Allen probably has about 85 gay-father kids and 194 lesbian-mother kids. I have no idea why these numbers are so low (Canada has 35 million people — almost the size or California — and homogamous marriage is legal there).
But to make his sample even that big, Allen says he included all 17-22 year-olds who live with their parents. With weights, that represents 1.97 million people, or (by my calculation) 77% of the 2006 Canadian population ages 17-22. (That seems very high to me, since in 2006 only 57% of American 17-22 year-olds lived with their parents, but I don’t know what’s going on in Canada.)
From the title through the end of the paper, Allen writes as if he were measuring the “graduation rate” of Canadian young adults. But that’s not the case. In Canada in 2006, 89% of people ages 25-34 had graduated high school (if I read their Census table right). In Allen’s sample, only 69% have graduated high school. Allen counts those in his sample who have graduated to calculate a graduation rate, but that’s not what it is, it’s the percentage of 17-22 year-olds (many of whom are still in high school) who have already graduated high school.
Allen throws up a smokescreen by analyzing the odds that people in his sample are attending school — which 74% of them are — as if this is the selection problem. (This analysis adds nothing, because some of his sample are attending high school and some are attending college.)
Who’s at home
All this is setup for the elephant in the room: selection into the sample. Who is living at home? Allen writes, “Children over the age of 22 were dropped because of a likely selection bias in children who live at home well into adulthood.” Age 22? That’s where you start to have selection bias in who lives at home? And then he’s got one of those throw-away footnotes that work if you trust the researcher:
There’s no reason to believe this selection bias would be correlated with family type, however. All regressions were run with various restrictions on the child’s age within the sample, including keeping everyone, and none of the gay or lesbian family results in the paper change, in terms of magnitudes or levels of significance, in an important way.
What were the “various restrictions on the child’s age”? Unless he got the same result with just the 17-year-olds, I think we can stop reading.
But what about “no reason to believe” the selection is correlated with family type? What drives the selection? There is no analysis comparing the people in his sample to the population of 17-22 year-olds who don’t live with their parents.
Think about the population like this: Here are some possible scenarios for 17-22 year-olds. The “live at home” column represent the people in Allen’s sample; the “doesn’t live at home” column represents threats to the validity of his sample. If the distribution across these columns is correlated with family structure, the study is wrong. What are the odds?
|Live at home||Doesn’t live at home|
|High school dropout||Happy and supportive family; or stuck at home with no exit plan||Successfully employed and independent; or unsuccessful and miserably kicked out of the house; married or not.|
|High school graduate||In college and living with happy and supportive family; in college and stuck at home because can’t afford rent; not in college and living with happy and supportive family; or not in college and stuck at home because too poor to move out.||Successfully employed and independent; independently poor and miserable (or married); successfully in college and living on parents’ money; in college but not supported by parents.|
I got an email from Kristi Williams, who suggested a hypothetical pattern in which gay and lesbian parents are more successful at launching their children from home after completing high school. In Allen’s analysis, that would be “troubling” evidence of a bad family outcome. That’s just one possible scenario, of course. But this problem alone completely invalidates the study, I believe. I can think of one other study that uses educational attainment among adults living with their parents to study high school dropout rates, but at least that paper included tests for differences between those living at home and not, and cautioned against generalizing to the non-living-at-home population. It’s just a bad idea unless you can solve that selection problem, and you probably can’t.
Who raised them?
That problem is so bad that you don’t need to worry about the problem of who raised these young adults, which is supposed to be the issue in the first place.
They live with their parents. But for how long have they done that, and for how long have their parents been in gay or lesbian relationships? We can’t know. Allen controls for whether the child has moved in the last year or five years, but we don’t know if the parents moved with them. Controlling for whether they have moved doesn’t address this. A full 60% of the lesbian-mother kids and 39% of the gay-father kids have moved in the last five years, compared with just 24% of the different-sex-married-parent kids. Their life stories are in these mobility histories, and the paper can’t say anything about that.
Interpreting the results
The paper says the children of gay and lesbian parents are “65% as likely to graduate,” a number Regnerus repeats, and Allen repeated in an interview. That’s just preying on the public, who don’t understand that in odds ratios (which I’ve discussed this here), that number would be even more dramatic if the graduation rates of the two groups were 99 and 96 percent. There is no good way to describe odds ratios, really, but they are useful in statistics. Anyway, the paper does provide the marginal effects, which show that the children living with gay parents have graduated from high school at an adjusted predicted percentage 6 points lower than those living with married different-sex parents, that number for kids of lesbian parents — which is not statistically significant with controls added — is 9 percentage points. But it’s not a meaningful result anyway.
The paper also splits the kids up by gender, and finds the worst “effects” are for girls living with gay dads. Since the analysis is all bogus, it doesn’t matter, but, in the interview he gave, Allen seems to forget that and think it’s the lesbians+daughter combination that’s worst, because he offers this “speculation” in answer to the question, “It’s particularly hard on girls, isn’t it?”:
Indeed, mothers may provide some parenting services that a father cannot provide, and fathers may provide parenting services that mothers cannot. These services may be necessary for girls but not necessary for boys. For example, I’ve been told by medical people that when a biological father is present in the home, daughters begin menstruation at an older age. Later menstruation is likely correlated with delayed sexual activity, etc., and this may lead to a better likelihood of high school completion.
Of course, girls in gay-father homes probably have a biological father in the home, which goes against his argument. Which is… really?
Believe it or not, there is some evidence that girls living without their fathers hit puberty earlier, which may be a kind of stress response. By earlier, I mean one month earlier on average, or maybe two months (as recorded by a retrospective question asked to adults). And it is true that earlier puberty increases the odds that girls will not finish high school, but that result comes from bigger differences than a month or two, as far as I can tell. If this were a true driver of family-structure effects on girls, we would get at it from studies of single mothers, not lesbian couples anyway.
But anyway, that’s neither here nor there in this study, which offers nothing of value.
I am willing to believe anything, if it’s true, even if I wish it weren’t true. I try to watch out for how my biases might distort my research (which I think is good) or the research that I criticize that I think is bad. Don’t hate on the methods because you hate the conclusion, hate on the conclusion because the methods are wrong.