Update: after Wilcox updated his report with the complete data, I now conclude the report is just dishonest, not complete bologna. See below.
Brad Wilcox and Nicholas Zill have a new report on Brad’s Institute for Family Studies website, about Arizona: “Stronger Families, Better Schools: Families and High School Graduation Across Arizona.” It bears a strong resemblance to a previous report, about Florida, “Strong Families, Successful Schools: High School Graduation and School Discipline in the Sunshine State.” Together these two give you a feeling like taking the first two bites of a leftover giant burrito that might have gone a little bad, and realizing there are probably about 48 more bites to go.
Anyway, this is about the Arizona one. I’ll first raise the possibility that it’s complete bologna – as in, fraudulent or error-ridden – and then discuss how it’s conclusions are dishonest at best even if the analysis is not technically wrong but rather just presented terribly.
Update: with the report corrected to show the complete data, the analysis now replicates fine. So I set aside the bologna issue. I leave this section here just so you can see the research design, but the main argument is in the next section.
First, the bologna issue
The report uses demographic data from 99 Arizona school districts to model graduation rates, and the gender gap in graduation rates. Their conclusion, based on two regression models using districts as the units of analysis and demographic indicators as the predictors, is this:
In Arizona, public school districts with better-educated and more married parents boast higher high school graduation rates. Gender equity is also greater in districts with more married parents. That is, boys come closer to matching the high school graduation rates of girls in districts with more married-parent families. Moreover, married parenthood is a better predictor of these two high school graduation outcomes than are child poverty, race, and ethnicity in public school districts across the Grand Canyon state.
To pad out the report, they also include appendix tables, so it’s theoretically possible to replicate their regressions. Unfortunately, unless I’m missing something, they don’t replicate. I wouldn’t normally bother rerunning someone’s regression, especially when the argument they’re building is so wrong-headed (see below), but just because we know from long experience that Wilcox does not behave honestly (in methods, ethics, and ethics) what the heck.
The report says, “Graduation rates and male/female graduation ratios for the 99 Arizona school districts in our study are shown in Table A1 in the Appendix.” Table A2 then lists the districts again, with the demographic variables. Unfortunately, table A2 only includes 83 districts – and the 16 missing are exactly those from Indian-Oasis to Paradise Valley in the alphabetical list of district names, so apparently an error handling the data. So I could only use 83 of the 99 for the regressions. Since I don’t know when they lost those 16 districts, I don’t know if it was before or after running the regressions (there are no Ns or standard errors on their regression tables).
For each of their dependent variables – graduation rate, and the male/female ratio in graduation rates – they list bivariate correlations, and adjusted betas from as multivariate regression. Here are their figures, with mine next to them. The key differences are highlighted:
If they’re using 99 cases and I have 83 (actually 81 for the gender cap because of missing data), you would expect some difference. But these are very similar, including the bivariate correlations and the R-squareds for the models.
The weird thing is that the biggest difference is exactly on their biggest claim: “married parenthood is a better predictor of these two high school graduation outcomes than are child poverty, race, and ethnicity…” That is based on the assertion that .29 is larger than -.28 (very luck for them, that tiny, insignificant difference in magnitude!). In my model the minority-size effect is more than twice as large as the marriage-parenthood effect. So, huh. It’s definitely possible Brad simply lied about his results and made up a few numbers. (And I’m just using the data they include in the report.) But now let’s pretend he didn’t.
Update: with the complete data I can report that those two betas are actually .2865 (.29!) versu .2847 (.28!). The idea that one is a “better” predictor than the other is clearly not serious. Further, for some reason (we can only guess), they combined percent Black, Hispanic, and American Indian together into “minority,” which produced the .28 result. If they had entered them into the model separately, they would find that Hispanic and American Indian effects are each bigger than the married parent effect, as I show here:
So much for the headline result. Anyway, back to the argument…
The point of the analysis is to make policy recommendations. They conclude:
If the state enjoyed more stable families, it might also see better educational outcomes among its children. It’s for that reason that Arizona should consider measures designed to strengthen and stabilize families.
Their recommendations to that end are vocational education and marriage promotion.
Private and public initiatives to provide social marketing on behalf of marriage could prove helpful. Campaigns against smoking and teenage pregnancy have taught us that sustained efforts to change behavior can work.
First, I’m not an education specialist (and neither are they), but shouldn’t there be some kind of policy variables in this analysis, like per-pupil spending, or teacher salaries, or something about curriculum or programming? It’s unusual to use only demographic variables and then conclude that what we need is a policy to change the demographics. It’s just not a serious analysis. (Please also please remember that “controlling for income” is not an adequate control for economic conditions and status.)
But second, given the first billion dollars of money spent promoting marriage produced absolutely no increase in marriage, is there any possible way Brad legitimately thinks this is the best way to improve graduation rates?
These are just two ideas. More should be explored. The bottom line: policymakers, educators, business leaders, and religious leaders in Arizona need to address the fragile foundations of family life if they hope for the state’s children to lead the nation in academic achievement.
Does this report really support that “bottom line”? Would it be better to spend money promoting marriage than to spend the same amount of money on some effort to improve schools? That’s obviously a dumb idea, but is it possible he really believes it? These are the only policies proposed. Maybe I’m wrong, but I doubt he believes it. I think he wants to promote marriage promotion programs for other reasons: to fund him and his compatriots, to support pro-marriage ideology, and so on. Not to improve graduate rates in Arizona schools. But, maybe I’m wrong.
And a laptop
I think what Brad is really doing is noise noise statistics statistics marriage-is-good expertise trust me fund me. The details clearly aren’t that important.
Meanwhile, not coincidentally, things are looking up for Brad at the Institute for Family Studies (IFS), the organization he created to handle the foundation-money rake. He started in 2011 as president / director of IFS at a salary of $35,000. After paying himself a paltry $9,999 and in 2012, he started improving his productivity, paying himself $50,000 in 2013, and then $80,400 in 2014 as a Senior Fellow, the last year for which I found a 990 form. Much of that money is coming from the Bradley Foundation (which also funded the Regnerus/Wilcox study) — their 2015 report lists $75,000 for IFS, so projections are good for next year. This is, of course, on top of what he gets for his service to the public at the University of Virginia.
The IFS disclosure forms also show purchase of a MacBook Pro. Which might or might not have been for Brad.
I do not make this case, and make it personally, because I disagree with Brad about politics. There are lots of people I disagree with even more than him, and I don’t spend all day criticizing them. The dishonesty offends me because it’s work and issues I care about, it hurts real people, I’m well situated to expose it, and his corporate-Christian-right megaphone is big, so it shouldn’t go unchallenged.