The ritualistic invocation of “policy implications” in sociology writing is puzzling. I don’t know its origin, but it appears to have come (like so much else that we cherish because we despise ourselves) from economists. The Quarterly Journal of Economics was the first (in the JSTOR database) to use the term in an abstract, including it 11 times over the 1950s and 1960s before the first sociology journal (Journal of Health and Social Behavior) finally followed suit in 1971.
That 1971 article projected a tone that persists to this day. In a paragraph tacked onto the end of the paper, Kohn and Mercer speculated that inflated claims about the dangers of marijuana “may actually contribute to dangerous forms of drug abuse among less well-educated youth” (although the paper was a survey of college students). “If this is the case,” they continued, “then the best corrective may be to revise law, social policy, and official information in line with the best current scientific knowledge about drugs and their effect.” The analysis in the paper had nothing to do with anti-drug policy, instead pursuing an interesting empirical examination of the relationship of ideology (rebellious versus authoritarian) and drug use. The “implications” are vague and unconnected to any actually-existing policy debate (and none is cited). Being in this case both banal and hopelessly idealistic — intellectual bedfellows that find themselves miserably at home in the sociological space many in the public deride as “academic” — it’s hard to imagine the paper having any policy effect. Not that there’s anything wrong with that.
Fifty years later, “policy implications” has become an institution in academic sociology — by no means universal, but a fixed feature of the landscape, demanded by some editors, reviewers, advisors, and funders. The prevalence of this trope coincides with the imperative for “engagement” (which I’ve written about previously) driven both by our internal sense of mission and our capitulation to external pressure to justify the existence of our work. These are admirable impulses, but they’re poorly served by many of our current practices. I hope this discussion of “policy implications,” and the suggestions that follow, help push us toward more productive responses.
How it’s done
Most sociologists don’t do a lot of policy work. It’s not our language or social or professional milieu, and often not part of our formal training. So what do we mean, in theory and practice, when we offer “policy implications” for our research? There is a very wide range of applications, from evaluations of specific local policies to critiques of state power itself. I collected a lot of examples which I’ll describe, but first a very prominent one, from “Social Conditions as Fundamental Causes of Health Inequalities: Theory, Evidence, and Policy Implications, by Phelan, Link, and Tehranifar (2010). Their promise of policy implications is right in the title. From the policy implications section, here is a list of policies intended to reduce inequality in social conditions:
“Policies relevant to fundamental causes of disease form a major part of the national agenda, whether this involves the minimum wage, housing for homeless and low-income people, capital-gains and estate taxes, parenting leave, social security, head-start programs and college-admission policies, regulation of lending practices, or other initiatives of this type.”
Then, in the conclusion, they explain that in addition to leveling inequalities in social condition, we need policies that “minimiz[e] the extent to which socioeconomic resources buy a health advantage” — in the U.S. context, this is interpretable as universal healthcare.
These are almost broad enough — considered together — to constitute a worldview (or perhaps a party platform) rather than a specific policy prescription. If this were actual policy analysis, we would have to be concerned with, for example, the extent which policies to raise the minimum wage, raise taxes, house the homeless, and expand educational opportunity actually produce reductions in inequality, and which of these is most effective, or important, or feasible, and so on. But this is not policy analysis, and none is cited. These are one step down from documenting wage disparities and offering socialism as “policy implications.” This is a review paper, mostly theory and summarizing existing evidence — which makes it more suitable than the implications attached to many narrow empirical papers (see below). It has been very influential, influencing thousands of students and researchers, and maybe people in policy settings as well (one could try to assess that), by helping to establish the connection between health inequality and inequality on other dimensions. Important work. But the way I read the term, this is too broad to be reduced to “policy implications” — it’s more like social implications, or theoretical implications.
127 more examples
To generalize about the practice of “policy implications,” I collected some data. I used a “topic” search in Web of Science, which searches title, abstract, and keywords, for the phrase “policy implications,” in articles from 2010 to 2020. This tree map from WoS shows the disciplinary breakdown of the journals with the search term, which remains dominated by economics.
I chose the sociology category, then weeded out journals that were very interdisciplinary (like Journal of Marriage and Family), and some articles that turned out to be false positives, and ended up with 127 articles in these 52 journals.*
First I read all the abstracts and came up with a three-category code for abstracts that (1) had specific policy implications, (2) made general policy pronouncements, or (3) just promised policy implications. Here are some details.
Of the 127 abstracts, only two had what I read as specific policy implications. Martin (2018) wrote, “for dietary recommendations to transform eating practices it is necessary to take into account how, while cooking, actors draw on these various forms of valuation.” And Andersen and van de Werfhorst (2010) wrote, “strengthening the skill transparency of the education system by increasing secondary and tertiary-level differentiation may strengthen the relationship between education and occupation.” These aren’t as specific as particular pieces of legislation or policies, but close enough.
I put 29 papers in the general pronouncements category. For example, I put Phalen, Link, and Tehranifar (2010) in this category. In another, Wiborg and Hansen (2018) wrote that their findings implied that “increasing equal educational opportunities do not necessarily lead to greater opportunities in the labor market and accumulation of wealth” (reading inside the paper confirmed this is the extent of the discussion). This by Stoilova, Ilieva-Trichkova, and Bieri (2020) is archetypal: “The policy implications are to more closely consider education in the transformation of gender-sensitive norms during earlier stages of child socialization and to design more holistic policy measures which address the multitude of barriers individuals from poor families and ethnic/migrant background face” (reading inside the paper, there are several other statements at the same level). I read three other papers in this category and found similar general implications, e.g., “if the policy goal is to enhance the bargaining position of labour and increase its share of income, spending policy should prioritise the expenditures on the public sector employment” (Pensiero 2017).
“Policy implications are discussed”
The largest category, 97 papers (76%) offered no policy implications in the abstract, but rather offered some version of “policy implications are discussed.” It is an odd custom, to mention the existence of a section in the paper without divulging its contents. Anyway, to get a better sense of what “policy implications are discussed” means, I randomly sampled 10 of the papers in this category to read the relevant section. (I have no beef with these papers or their authors, they were selected randomly, and I’m only commented on what may be the least important aspect of their contributions.)
The first category among these, with 5 of the 10 papers, are those without substantive policy contributions. Some have banal statements at the end, which the author and most readers probably already believed, such as, “If these results are replicated, programs should be implemented that will solicit the help of grandparents in addition to parents” (Liu 2016). I also include here Visser et al. (2013), who conclude that their “findings show general support for basic ecological perspectives of fear of crime and feelings of unsafety,” e.g., that reducing crime in the absence of better social protection will not improve levels of fear and feelings of unsafety. I code this one as without substantive policy contribution because that’s a big claim about the entire state policy structure, which would require much more evidence to adjudicate, much less implement, and the paper offers only a small empirical nudge in one direction (which, again, is fine!).
Several in this category offered essentially no policy implications. This includes Wang (2010) who states at the outset that, “the question of motives for private transfers is one with important policy implications” for public transfer programs like food stamps and social security, but never comes back to discuss policies relevant to the results. And Barrett and Pollack (2011), who recommend that health practitioners develop better understanding of the issues raised and that “contemporary sexual civil rights efforts” pay more attention to sexual discrimination. Finally, Lepianka (2015), reports on media depictions of poverty and related policy, but doesn’t offer any implications of the study itself for policy. So, half of these had abstracts that were overpromising in terms of policy.
The other 5 papers do include substantive policy implications, explored to varying degrees. One is hard-hitting but brief: Shwed et al. (2018), whose analysis has direct implications which they do not thoroughly discuss. Their “unequivocal” result is that “multicultural schools, with their celebration of difference, entail a cost in terms of social integration compared to assimilationist schools—they enhance ethnic segregation in friendship networks. … While empowering minorities, it enhances social closure between groups.” The empirical analysis they did could no doubt be used as part of a policy analysis on the question of cultural orientation of schools in Israel.
Three offer sustained policy discussions, including the very specific: an endorsement of prison-based dog training programs (Antonio et al. 2017); a critique of sow-housing policy in the European Union (de Krom 2015); and recommendations for environmental lending practices at the World Bank (Sommer at al. 2017). The last one qualifies, albeit at a very macro level: Gauchat et al.’s (2011) analysis of economic dependence on military spending in metropolitan areas, the implications of which surpass everyday policy debates but are of course relevant.
To summarize my reading, with percentages based on extrapolating my subsample (so, wide confidence intervals): 23% of papers promising policy implications had none, and 38% had either vague statements or general statements that did not rely on empirical findings in the paper. The remaining 40% had substantive policy discussion and/or specific recommendations.
This is a quick coding and not validated. Others might treat differently papers that report an effect and then recommend changing the prevalence of the independent variable — e.g., poverty causes poor health; policies should reduce poverty — which I coded as not substantive. For example, I coded this from Thoits (2010) as not substantive or specific: “policies and programs should target children who are at long-term health risk due to early exposure to poverty, inadequate schools, and stressful family circumstances.” You could say, “policies should attempt to make life better,” but it’s not clear you need research for that. Anyway, my own implications (below) don’t depend on a precise accounting.
Implications
I am really, really not saying these are bad papers, or wrong to do what they did. I am not criticizing them, but rather the institutional convention that classifies the attempt to make our research relevant as “policy implications,” even when we have nothing specific to say about real policies, and then rewards sociologists for shoehorning their conclusions into such a frame.
Let me give an example of an interesting and valuable paper that is burdened by its policy implications. “The impact of parental migration on depression of children: new evidence from rural China,” by Yue et al. (2020) used a survey of families in China to assess the relationship between parental migration, remittances, household labor burdens, parent-child communication, and children’s symptoms of depression. After regression models with direct and indirect effects on children’s depression, including both children who were “left behind” by migrating parents and those who weren’t, they conclude: “non-material resources (parent-child communication, parental responsiveness, and self-esteem) play a much more important role in child’s depression than material resources (remittances).” Interesting result. Seems well done, with good data. The policy suggestions that follow are to encourage parent-child communication (e.g., through local government programs) and teach children in school that they are not abandoned by parents who migrate.
What is wrong with this? First, Yue et al. (2020) is an example of a common model that amounts to, “based on our regressions, more of this variable would be good.” It seems logical, but a serious approach to the question would have to be based on evidence that such programs actually have their intended effect, and that they would be better than directing the money or other resources toward something else. That would be an unreasonable burden for the authors, and slow the production of useful empirical results. So we’re left with something superficial that distracts more than it adds. Further (and here I hope to win some converts to my view), these policy implication sections are a major source of of peer review friction — reviewers demanding them, reviewers hating them, authors contorting themselves, and so on. Much better, in my view, would be to just add the knowledge produced by papers like this to the great hopper of knowledge, and let it contribute to a real policy analysis down the road.
Empirical peer-reviewed sociology articles should be shorter, removing non-essential parts of the paper that are major sources of peer review bog-down. Having different kinds of work reviewed and approved together in a single paper — a lengthy literature review, a theoretical claim, an empirical analysis, and a set of policy implications — creates inefficiencies in the peer review process. Why should a whole 60-page paper be rejected because one part of it (the policy implications, say) is rejected by one out of three reviewers? This is very wasteful. It puts reviewers in a position to review aspects of the work they aren’t qualified to judge. And it skews incentives by rewarding the less important parts of our work. Of course it’s reasonable to spend a few paragraphs stating the relevance of the question in the paper, but not a whole treatise (in the front and back) of every paper.
Advice for sociologists
1. Don’t try to pin big conclusions on a single piece of peer reviewed empirical research. That’s a sad legacy of a time when publishing was hard, sociologists had few opportunities to do so, and peer reviewed journals were the source of validation we were expected to rely on. So you devoted years of your life to a small number of “publications,” and those were the sum total of your intellectual production. We have a lot of other ways to express our social and political views now, and we should use them. The fact that you have a PhD, a job, and have published peer reviewed research, are all sources of legitimacy you can draw on to get people to pay attention to your writing.
2. Write for the right audience. If you are serious about influencing policy, write for staffers doing research for advocacy organizations, activists, or campaigns. If you want to influence the public, write in lay terms in venues that draw regular people as readers. If you want to set the agenda for funding agencies, write review pieces that synthesize research and make the case for moving in the right direction. These are all different kinds of writing, published in different venues. Crucially, none of them rely only on the empirical results of a single analysis, nor should they. The last three paragraphs of your narrow empirical research paper — excellent, important, and cutting-edge as it is — will not reach these different audiences.
3. Stop asking researchers to tack superficial policy implications sections onto the end of their papers. If you are a reviewer or an editor, stop demanding longer literature reviews and conclusions. Start rewarding the most important part of the work, the part you are qualified to evaluate.
4. If you are in an academic department, on a hiring committee, or on a promotion and tenure committee, look at the whole body of work, including the writing outside peer-reviewed journals. No one expects to get tenure from writing an op-ed, but people who work to reach different audiences may be building a successful career in which peer-reviewed research is a foundational building block. Look for the connections, and reward the people who make them.
*The full sample (metadata and abstracts) is available on Zotero. Some are open access, some I got through my library, but all are available from from Sci-Hub (which steals them so you don’t have to).
References mentioned in the text:
Andersen, Robert, and Herman G. van de Werfhorst. 2010. “Education and Occupational Status in 14 Countries: The Role of Educational Institutions and Labour Market Coordination.” British Journal of Sociology 61(2):336–55. doi: 10.1111/j.1468-4446.2010.01315.x.
Antonio, Michael E., Rosalyn G. Davis, and Susan R. Shutt. 2017. “Dog Training Programs in Pennsylvania’s Department of Corrections Perceived Effectiveness for Inmates and Staff.” Society & Animals 25(5):475–89. doi: 10.1163/15685306-12341457.
de Krom, Michiel P. M. M. 2015. “Governing Animal-Human Relations in Farming Practices: A Study of Group Housing of Sows in the EU.” Sociologia Ruralis 55(4):417–37. doi: 10.1111/soru.12070.
Gauchat, Gordon, Michael Wallace, Casey Borch, and Travis Scott Lowe. 2011. “The Military Metropolis: Defense Dependence in U.S. Metropolitan Areas.” City & Community 10(1):25–48. doi: 10.1111/j.1540-6040.2010.01359.x.
Kohn, Paul M., and G. W. Mercer. 1971. “Drug Use, Drug-Use Attitudes, and the Authoritarianism-Rebellion Dimension.” Journal of Health and Social Behavior 12(2):125–31. doi: 10.2307/2948519.
Lepianka, Dorota. 2015. “Images of Poverty in a Selection of the Polish Daily Press.” Current Sociology 63(7):999–1016. doi: 10.1177/0011392115587021.
Liu, Ruth X. 2018. “Physical Discipline and Verbal Punishment: An Assessment of Domain and Gender-Specific Effects on Delinquency Among Chinese Adolescents.” Youth & Society 50(7):871–90. doi: 10.1177/0044118X15618836.
Martin, Rebeca Ibanez. 2018. “Thinking with La Cocina: Fats in Spanish Kitchens and Dietary Recommendations.” Food Culture & Society 21(3):314–30. doi: 10.1080/15528014.2018.1451039.
Pensiero, Nicola. 2017. “In-House or Outsourced Public Services? A Social and Economic Analysis of the Impact of Spending Policy on the Private Wage Share in OECD Countries.” International Journal of Comparative Sociology 58(4):333–51. doi: 10.1177/0020715217726837.
Phelan, Jo C., Bruce G. Link, and Parisa Tehranifar. 2010. “Social Conditions as Fundamental Causes of Health Inequalities: Theory, Evidence, and Policy Implications.” Journal of Health and Social Behavior 51:S28–40. doi: 10.1177/0022146510383498.
Shwed, Uri, Yuval Kalish, and Yossi Shavit. 2018. “Multicultural or Assimilationist Education: Contact Theory and Social Identity Theory in Israeli Arab-Jewish Integrated Schools.” European Sociological Review 34(6):645–58. doi: 10.1093/esr/jcy034.
Sommer, Jamie M., John M. Shandra, and Michael Restivo. 2017. “The World Bank, Contradictory Lending, and Forests: A Cross-National Analysis of Organized Hypocrisy.” International Sociology 32(6):707–30. doi: 10.1177/0268580917722893.
Stoilova, Rumiana, Petya Ilieva-Trichkova, and Franziska Bieri. 2020. “Work-Life Balance in Europe: Institutional Contexts and Individual Factors.” International Journal of Sociology and Social Policy 40(3–4):366–81. doi: 10.1108/IJSSP-08-2019-0152.
Thoits, Peggy A. 2010. “Stress and Health: Major Findings and Policy Implications.” Journal of Health and Social Behavior 51:S41–53. doi: 10.1177/0022146510383499.
Visser, Mark, Marijn Scholte, and Peer Scheepers. 2013. “Fear of Crime and Feelings of Unsafety in European Countries: Macro and Micro Explanations in Cross-National Perspective.” Sociological Quarterly 54(2):278–301. doi: 10.1111/tsq.12020.
Wang, Jingshu. 2010. “Motives for Intergenerational Transfers: New Test for Exchange.” American Journal of Economics and Sociology 69(2):802–22. doi: 10.1111/j.1536-7150.2010.00725.x.
Wiborg, Oyvind N., and Marianne N. Hansen. 2018. “The Scandinavian Model during Increasing Inequality: Recent Trends in Educational Attainment, Earnings and Wealth among Norwegian Siblings.” Research in Social Stratification and Mobility 56:53–63. doi: 10.1016/j.rssm.2018.06.006.
Yue, Zhongshan, Zai Liang, Qian Wang, and Xinyin Chen. 2020. “The Impact of Parental Migration on Depression of Children: New Evidence from Rural China.” Chinese Sociological Review 52(4):364–88. doi: 10.1080/21620555.2020.1776601.
Twelve days ago I posted my paper on the COVID-19 epidemic in rural US counties. I put it on the blog, and on the SocArXiv paper server. At this writing the blog post has been shared on Facebook 69 times, the paper has been downloaded 149 times, and tweeted about by a handful of people. No one has told me it’s wrong yet, but not one has formally endorsed it yet, either.
Until now, that is. The paper, which I then submitted to the European Journal of Environment and Public Health, has now been peer reviewed and accepted. I’ve updated the SocArXiv version to the journal page proofs. Satisfied?
It’s a good question. We’ll come back to it.
Preprints
The other day (I think, not good at counting days anymore) a group of scholars published — or should I say posted — a paper titled, “Preprinting a pandemic: the role of preprints in the COVID-19 pandemic,” which reported that there have already been 16,000 scientific articles published about COVID-19, of which 6,000 were posted on preprint servers. That is, they weren’t peer-reviewed before being shared with the research community and the public. Some of these preprints are great and important, some are wrong and terrible, some are pretty rough, and some just aren’t important. This figure from the paper shows the preprint explosion:
All this rapid scientific response to a worldwide crisis is extremely heartening. You can see the little sliver that SocArXiv (which I direct) represents in all that — about 100 papers so far (this link takes you to a search for the covid-19 tag), on subjects ranging from political attitudes to mortality rates to traffic patterns, from many countries around the world. I’m thrilled to be contributing to that, and really enjoy my shifts on the moderation desk these days.
On the other hand some bad papers have gotten out there. Most notoriously, an erroneous paper comparing COVID-19 to HIV stoked conspiracy theories that the virus was deliberately created by evil scientists. It was quickly “withdrawn,” meaning no longer endorsed by the authors, but it remains available to read. More subtly, a study (by more famous researchers) done in Santa Clara County, California, claimed to find a very high rate of infection in the general population, implying COVID-19 has a very low death rate (good news!), but it was riddled with design and execution errors (oh well), and accusations of bias and corruption. And some others.
Less remarked upon has been the widespread reporting by major news organizations on preprints that aren’t as controversial but have become part of the knowledge base of the crisis. For example, the New York Times ran a report on this preprint on page 1, under the headline, “Lockdown Delays Cost at Least 36,000 Lives, Data Show” (which looks reasonable in my opinion, although the interpretation is debatable), and the Washington Post led with, “U.S. Deaths Soared in Early Weeks of Pandemic, Far Exceeding Number Attributed to Covid-19,” based on this preprint. These media organizations offer a kind of endorsement, too. How could you not find this credible?
Peer review
To help sort out the veracity or truthiness of rapid publications, the administrators of the bioRxiv and medRxiv preprint servers (who are working together) have added this disclaimer in red to the top of their pages:
Caution: Preprints are preliminary reports of work that have not been certified by peer review. They should not be relied on to guide clinical practice or health-related behavior and should not be reported in news media as established information.
That’s reasonable. You don’t want people jumping the gun on clinical decisions, or news reports. Unless they should, of course. And, on the other hand, lots of peer reviewed research is wrong, too. I’m not compiling examples of this, but you can always consult the Retraction Watch database, which, for example, lists 130 papers published in Elsevier journals in 2019 that have been retracted for reasons ranging from plagiarism to “fake peer review” to forged authorship to simple errors. The database lists a few peer-reviewed COVID-19 papers that have already been retracted as well.
This comparison suggests that the standard of truthiness cannot be down to the simple dichotomy of peer reviewed or not. We need signals, but they don’t have to be that crude. In real life, we use a variety of signals for credibility that help determine how much to trust a piece of research. These include:
The reputation of the authors (their degrees, awards, twitter following, media presence)
The institutions that employ them (everyone loves to refer to these when they are fancy universities reporting results they favor, e.g., “the Columbia study showed…”)
Who published it (a journal, an association, a book publisher), which implies a whole secondary layer of endorsements (e.g., the editor of the journal, the assumed expertise of the reviewers, the prestige or impact factor of the journal as a whole, etc.)
Perceived conflicts of interest among the authors or publishers
The transparency of the research (e.g., are the data and materials available for inspection and replication)
Informal endorsements, from, e.g., people we respect on social media, or people using the Plaudit button (which is great and you should definitely use if you’re a researcher)
And finally, of course, our own assessment of the quality of the work, if it’s something we believe ourselves qualified to assess
As with the debate over the SAT/GRE for admissions, the quiet indicators sometimes do a lot of the work. Call something a “Harvard study” or a “New York Times report,” and people don’t often pry into the details of the peer review process.
Analogy: People who want to eat only kosher food need something to go on in daily life, and so they have erected a set of institutional devices that deliver such a seal (in fact, there are competing seal brands, but they all offer the same service: a yes/no endorsement by an organization one decides to trust). The seals cost money, which is added to the cost of the food; if people like it, they’re willing to pay. But, as God would presumably tell you, the seal should not always substitute for your own good judgment because even rabbis or honest food producers can make mistakes. And in the absence of a good kosher inspection to rely on altogether, you still have to eat — you just have to reason things through to the best of your ability. (In a pinch, maybe follow the guy with the big hat and see what he eats.) Finally, crucially for the analogy, anyone who tells you to ignore the evidence before you and always trust the authority that’s selling the dichotomous indicator is probably serving their own interests as least as much as they’re serving yours.
In the case of peer review, giant corporations, major institutions, and millions of careers depend on people believing that peer review is what you need to decide what to trust. And they also happen to be selling peer review services.
My COVID-19 paper
So should you trust my paper? Looking back at our list, you can see that I have degrees and some minor awards, some previous publications, some twitter followers, and some journalists who trust me. I work at a public research university that has its own reputation to protect. I have no apparent way of profiting from you believing one thing or another about COVID-19 in rural areas (I declared no conflicts of interest on the SocArXiv submission form). I made my data and code available (even if no one checks it, the fact that it’s there should increase your confidence). And of course you can read it.
And then I submitted it to the European Journal of Environment and Public Health, which, after peer review, endorsed its quality and agreed to publish it. The journal is published by Veritas Publications in the UK with the support of Tsinghua University in China. It’s an open access journal that has been publishing for only three years. It’s not indexed by Web of Science or listed in the Directory of Open Access Journals. It is, in short, a low-status journal. On the plus side, it has an editorial board of real researchers, albeit mostly at lower status institutions. It publishes real papers, and (at least for now) it doesn’t charge authors any publication fee, it does a little peer review, and it is fast. My paper was accepted in four days with essentially no revisions, after one reviewer read it (based on the summary, I believe they did read it). It’s open access, and I kept my copyright. I chose it partly because one of the papers I found on Google Scholar during my literature search was published there and it seemed OK.
So, now it’s peer reviewed.
Here’s a lesson: when you set a dichotomous standard like peer-reviewed yes/no and tell the public to trust it, you create the incentive for people to do the least they can to just barely get over that bar. This is why we have a giant industry of tens of thousands of academic journals producing products all branded as peer reviewed. Half a century ago, some academics declared themselves the gatekeepers of quality, and called their system peer review. To protect the authority of their expertise (and probably because they believed they knew best), they insisted it was the standard that mattered. But they couldn’t prevent other people from doing it, too. And so we have a constant struggle over what gets to be counted, and an effort to disqualify some journals with labels like “predatory,” even though it’s the billion-dollar corporations at the top of this system that are preying on us the most (along with lots of smaller scam purveyors).
In the case of my paper, I wouldn’t tell you to trust it much more because it’s in EJEPH, although I don’t think the journal is a scam. It’s just one indicator. But I can say it’s peer reviewed now and you can’t stop me.
Aside on service and reciprocity: Immediately after I submitted my paper, the EJEPH editors sent me a paper to review, which I respect. I declined because it wasn’t qualified, and then they sent me another. This assignment I accepted. The paper was definitely outside my areas of expertise, but it was a small study quite transparently done, in Nigeria. I was able to verify important details — like the relevance of the question asked (from cited literature), the nature of the study site (from Google maps and directories), the standards of measurement used (from other studies), the type of the instruments used (widely available), and the statistical analysis. I suggested some improvements to the contextualization of the write-up and recommended publication. I see no reason why this paper shouldn’t be published with the peer review seal of approval. If it turns out to be important, great. If not, fine. Like my paper, honestly. I have to say, it was a refreshing peer review experience on both ends.
As a faculty sociologist who works in the area of family demography and inequality, my interest in open scholarship falls into the category of “service” among my academic obligations, essentially unrecognized and unremunerated by my employer, and competing with research and teaching responsibilities for my time. In that capacity I founded SocArXiv in 2016 (supported by several small grants) and serve as its director, organized two conferences at the University of Maryland under the title O3S: Open Scholarship for the Social Sciences, and I was elected to the Committee on Publications of the American Sociological Association. While continuing that work during a sabbatical leave, I was extremely fortunate to land a half-time position as visiting scholar at the MIT Libraries in the fall 2018, which helped me integrate that service agenda with an emerging research agenda around scholarly communication.
The semester was framed by the MIT Grand Challenges Summit in the spring, which I attended, and the report that emerged from that meeting: A Grand Challenges-Based Research Agenda for Scholarly Communication and Information Science, on which I was a collaborator. The report, published in December, describes a vision for a more inclusive, open, equitable, and sustainable future for scholarship; it also characterizes the barriers to this future, and identifies the research needed to bring it to fruition.
Sociology and SocArXiv
Furthering my commitments to sociology and SocArXiv, I continued to work on the service. SocArXiv is growing, with increased participation in sociology and other social sciences. In the fall the Center for Open Science, our host, opened discussions with its paper serving communities about weaning the system off its core foundation financial support and using contributions from each service to make it sustainable (thus far have not paid COS for its develop and hosting). This was an expected challenge, which will require some creative and difficult work in the coming months.
Finally, at the start of the semester I noted that most sociologists — even those interested in open access issues — were not familiar with current patterns, trends, and debates in the scholarly communications ecosystem. This has hampered our efforts to build SocArXiv, as well as our ability to press our associations and institutions for policy changes in the direction of openness, equity, and sustainability. In response to this need, especially among graduate students and junior scholars, I drafted a scholarly communication primer for sociology, which reviews major scholarly communication media, policies, economic actors, and recent innovations. I posted a long draft (~13,000 words) for comment in January, and received a very positive response. It appears that a number of programs will incorporate the revised primer into their training, and many individuals are already reading and sharing it with their networks.
Peer review
One of the chief barriers identified in the Grand Challenges report is the lack of systematic theory and empirical evidence to design and guide legal, economic, policy and organizational interventions in scholarly publishing and in the knowledge ecosystem generally. As social scientists, Micah and I drew on this insight, and used the case of peer-review in sociology as an entry point. We presented our formative analysis of this case in the CREOS Research Talk, “Can Fix Peer Review.” Here is the summary of this talk:
Contemporary journal peer review is beset by a range of problems. These include (a) long delay times to publication, during which time research is inaccessible; (b) weak incentives to conduct reviews, resulting in high refusal rates as the pace of journal publication increases; (c) quality control problems that produce both errors of commission (accepting erroneous work) and omission (passing over important work, especially null findings); (d) unknown levels of bias, affecting both who is asked to perform peer review and how reviewers treat authors, and; (e) opacity in the process that impedes error correction and more systematic learning, and enables conflicts of interest to pass undetected. Proposed alternative practices attempt to address these concerns — especially open peer review, and post-publication peer review. However, systemic solutions will require revisiting the functions of peer review in its institutional context.
The full slides, with embedded video of the talk (minus the first few minutes) is embedded below:
Research design and intervention
Mapping out the various interventions and proposed alternatives in the peer review space raised a number of questions about how to design and evaluate interventions in a complex system with interdependent parts and actors embedded in different institutional logics — for example, university researchers (some working under state policy), research libraries, for-profit publishers, and academic societies. Working with Jessica Polka, Director of ASAPbio, we are expanding this analysis to consider a range of innovations open science. This analysis highlights the need for systematic research design that can guide the design of initiatives aimed at altering the scholarly knowledge ecosystem.
Applying the ecosystem approach in the Grand Challenges report, we consider large-scale interventions in public health and safety, and their unintended consequences, to build a model for designing projects with the intention of identifying and assessing such consequences across the system. Addressing problems at scale may have such unintended effects as leading vulnerable populations to adapt to new technology in harmful ways (mosquito nets used for fishing); providing new opportunities for harmful competitors (the pesticide treadmill); the displacement of private actors by public goods (dentists driven away by public water fluoridation); and risk compensation by those who receive public protection (anti-lock brakes and riskier driving, vaccinations). Our forthcoming white paper will address such risks in light of recent open science interventions: PLOS One, bioRxiv and preprints generally, and open peer review, among others. We combine research design methods for field experiments in social science, outcomes identified in the grand challenge report, and the ecosystem theory based on an open science lifecycle model.
ARL/SSRC meeting and Next Steps
Coming out of discussions at the first O3S meeting, in December the Association of Research Libraries and the Social Science Research Council convened a meeting on open scholarship in the social sciences, which included leaders from scholarly societies, university libraries, researchers advocating for open science, funders, and staff from ARL, SSRC, and the Coalition for Networked Information. I was fortunate to participate on the planning committee for the meeting, and in that capacity I conducted a series of short video interviews with individual stakeholders from the participating organizations to help expose us all to the range of values, objectives, and concerns we bring to the questions we collectively face in the movement toward open scholarship.
For our own work on peer review, which we presented at the meeting, I was especially drawn to the interviewees’ comments on transparency, incentives, and open infrastructure. In particular, MIT Libraries Director Chris Bourg challenged social scientists to recognize what their own research implies for the peer review system:
Brian Nosek, director of the Center for Open Science, stressed to the need to consider incentives for openness in our interventions:
And Kathleen Fitzpatrick, project director for Humanities Commons, described the necessity of open infrastructure that is flexibly interoperable, allowing parallel use by actors on diverse platforms:
These insights about intervention principles for an open scholarly ecosystem helped Micah and me develop a proposal for discussion at the meeting. Our proposed program, IOTA (I Owe The Academy) aims to solve the supply-and-demand problem for quality peer review in open science interventions (the name is likely to change). We understand that most academics are willing to do peer review when it contributes to a better system of scholarship. At the same time, new peer review projects need (good) reviewers in order to launch successfully. And the community needs (good) empirical research on the peer review process itself. The solution is to match reviewers with initiatives that promote better scholarship using a virtual token system, whereby reviewers pledge review effort units, which are distributed to open peer review projects — while collecting data for use in evaluation and assessment. After receiving positive feedback at the meeting, we will develop this proposal further.
Our presentation is embedded in full below:
A report on the ARL/SSRC meeting describes the shared interests, challenges to openness, and conditions for successful action discussed by participants. And it includes five specific projects they agreed to pursue — one of which is peer review on the SocArXiv and PsyArXiv paper platforms.
What’s next…
In the coming several months we expect to produce a white paper on research design, a proposal for IOTA, and a presentation for the Coalition for Networked Information meeting in April, to spark a discussion about the ways libraries can jointly support additional targeted work to promote, inspire, and support evidence-based research. And a revised version of the scholarly communication primer for sociology is on the way.
The divorce paper I posted two weeks ago, “The Coming Divorce Decline,” suddenly took off in the media the other day (blog post | paper | data and code). I’ve now written an op-ed about the findings for The Hill, including this:
I am ambivalent about these trends. Divorce is often painful and difficult, and most people want to avoid it. The vast majority of Americans aspire to a lifelong marriage (or equivalent relationship). So even if it’s a falling slice of the population, I’m not complaining that they’re happy. Still, in an increasingly unequal society and a winner-take-all economy, two-degree couples with lasting marriages may be a buffer for the select few, but they aren’t a solution to our wider problems.
Here’s my media scrapbook, with some comment about open science process at the end.
The story was first reported by Ben Steverman at Bloomberg, who took the time to read the paper, interview me at some length, send the paper to Susan Brown (a key expert on divorce trends) for comment, and produce figures from the data I provided. I was glad that his conclusion focused on the inequality angle from my interpretation:
“One of the reasons for the decline is that the married population is getting older and more highly educated,” Cohen said. Fewer people are getting married, and those who do are the sort of people who are least likely to get divorced, he said. “Marriage is more and more an achievement of status, rather than something that people do regardless of how they’re doing.”
Many poorer and less educated Americans are opting not to get married at all. They’re living together, and often raising kids together, but deciding not to tie the knot. And studies have shown these cohabiting relationships are less stable than they used to be.
Fewer divorces, therefore, aren’t only bad news for matrimonial lawyers but a sign of America’s widening chasm of inequality. Marriage is becoming a more durable, but far more exclusive, institution.
The Bloomberg headline was, “Millennials Are Causing the U.S. Divorce Rate to Plummet.” Which proved irresistible on social media. I didn’t use the terms “millennials” (which I oppose), or “plummet,” but they don’t fundamentally misrepresent the findings.
Naturally, though, the Bloomberg headline led to other people misrepresenting the paper, like Buzzfeed, which wrote, “Well, according to a new study, millennials are now also ‘killing’ divorce.” Neither I nor Bloomberg said anyone was “killing” divorce; that was just a Twitter joke someone made, but Buzzfeed was too metameta to pick up on that. On the other hand, never complain about a Buzzfeed link, and they did link to the paper itself (generating about 800 clicks in a few days).
Then Fox 5 in New York did a Skype interview with me, and hit the bar scene to talk over the results (additional footage courtesy of my daughter, because nowadays you provide your own b-roll):
The next day Todaydid the story, with additional information and reporting from Bowling Green’s National Center for Family and Marriage Research, and Pew.
The Maryland news office saw the buzz and did their own story, which helped push it out.
An article in Atlantic featured an interview with Andrew Cherlin putting the trends in historical context. Rachelle Hampton in Slate tied the divorce trend to a Brookings report showing marriage is increasingly tied to higher education. On KPCC, AirTalk hosted a discussion with Megan Sweeney and Steven Martin. On Wisconsin Public Radio, Stephanie Coontz widened the discussion to put changes in marriage and divorce in historical perspective.
Rush Limbaugh read from the Bloomberg article, and was just outraged: “Now, who but deranged people would look at it this way?”
How anybody thinks like this… You have to work to be this illogical. I don’t know where this kind of thing comes from, that a plummeting divorce rate is a bad sign for America in the left’s crazy world of inequality and social justice and their quest to make everybody the same. So that’s just an example of the… Folks, that is not… That kind of analysis — and this is a sociology professor at the University of Maryland. This is not stable. That kind of thinking is not… It’s just not normal. Yet there it is, and it’s out there, and it’s be widely reported by the Drive-By Media, probably applauded and supported by others. So where is this coming from? Where is all of this indecency coming from? Why? Why is it so taking over the American left?
The Limbaugh statement might have been behind this voicemail I received from someone who thinks I’m trying to “promote chaos” to “upend the social order”:
I had a much more reasonable discussion about marriage, divorce, and inequality in this interview with Lauren Gilger in KJZZ (Phoenix public radio).
The Chicago Tribuneeditorial board used the news to urge parents not to rush their children toward marriage:
This waiting trend may disturb older folks who followed a more traditional (rockier?) path and may be secretly, or not so secretly, wondering if there’s something wrong with their progeny. There isn’t. Remember: Unlike previous generations, many younger people have a ready supply of candidates at their fingertips in the era of Tinder and other dating apps. They can just keep swiping right. Our advice for parents impatient to marry off a son or daughter? Relax. The older they get, the less likely you’ll be stuck paying for the wedding.
The Catholic News Agency got an expert to chime in, “If only we could convince maybe more of them to enter into marriage, we’d be doing really well.”
I don’t know how TV or local news work, but somehow this is on a lot of TV stations. Here’s a selection.
Fox Business Network did a pretty thorough job.
Some local stations added their own reporting, like this one in Las Vegas:
And this one in Buffalo:
And this one in Boise, which brought in a therapist who says young people aren’t waiting as long to start couples therapy.
Jeff Waldorf on TYT Nation did an extended commentary, blaming capitalism:
Open science process
Two things about my process here might concern some people.
The first is promoting research that hasn’t been peer reviewed. USA Today was the only report I saw that specifically mentioned the study is not peer reviewed:
The study, which has not been published in a peer-reviewed journal, has been submitted for presentation at the 2019 Population Association of America meeting, an annual conference for demographers and sociologists to present research.
But, when Steverman interviewed me I emphasized to him that it was not peer-reviewed and urged him to consult other researchers before doing the story — he told me he had already sent it to Susan Brown. Having a good reporter consult a top expert who’s read the paper is as good a quality peer review as you often get. I don’t know everything Brown told him, but the quote he used apparently showed her endorsement of the main findings:
“The change among young people is particularly striking,” Susan Brown, a sociology professor at Bowling Green State University, said of Cohen’s results. “The characteristics of young married couples today signal a sustained decline [in divorce rates] in the coming years.”
For the story to be clear enough to become a news event, the research often has to be pretty simple. That’s the case here: what I’m doing is looking at an easily-identified trend and providing my interpretation of it. If this has to be peer-reviewed, then almost anything an academic says should be. Of course, I provided the publicly verifiable data and code, and there are a lot of people with the skills to check this if it concerned them.
On the other hand, there is a lot of research that is impossible to verify that gets reported. Prominent examples include the Alice Goffman ethnographic book and the Raj Chetty et al. analysis of confidential IRS data. These were big news events, but whether they were peer reviewed or not was irrelevant because the peer reviewers had no way to know if the studies were right. My conclusion is that sharing research is the right thing to do, and sharing it with as much supporting material as you can is the responsible way to do it.
The second concern is over the fact that I posted it while it was being considered for inclusion in the Population Association of America meetings. This is similar to posting a paper that is under review at a journal. Conference papers are not reviewed blind, however, so it’s not a problem of disclosing my identity, but maybe generating public pressure on the conference organizers to accept the paper. This happens in many forms with all kinds of open science. I think we need to see hiding research as a very costly choice, one that needs to be carefully justified — rather than the reverse. Putting this in the open is the best way to approach accountability. Now the work of the conference organizers, whose names are listed in the call for papers, can be judged fairly. And my behavior toward the organizers if they reject it can also be scrutinized and criticized.
Although I would love to have the paper in the conference, in this case I don’t need this paper to be accepted by PAA, as it has already gotten way more attention than I ever expected. PAA organizers have a tough job and often have to reject a lot of papers for reasons of thematic fit as well as quality. I won’t complain or hold any grudges if it gets rejected. There’s a lot of really good demography out there, and this paper is pretty rudimentary.
The Unknown Land, by E. Blair Leighton. Seems somehow relevant.
Update: My review was subsequently published in Men and Masculinities and is available here.
Prologue
(Bad sign when a blog post has a prologue)
When I was invited to review Mark Regnerus’s book Cheap Sex: The Transformation of Men, Marriage, and Monogamy, I agreed because it would give me a justification for reading the book. I already knew writing about it would scratch a number of my itches: research ethics, methods for studying sexuality and families, same-sex marriage (homogamy), the politics of academia, and Regnerus himself — whose work has been making me itch for years. But I had planned to set him aside after pulling the whole Regnerus Affair together for a chapter of my new book. Then I saw some people were treating the book normally (and, in the case of the one and only Anthony Giddens, who lavished 37 words of hyperbolous blurb on it, even seriously), so I thought it might actually be worth reading. And then I saw it was a terrible, awful book that no one should take seriously, much less read or buy, and that someone needed to say that, with evidence, so as not to allow its normalization. In other words, I got sucked back in.
So I took all these notes I wanted, and know I may as well share them, partly to protect myself and partly to help others for whom they might be useful. I have to be thorough when I do this, because I am afraid of making a mistake or missing something good so that my over-the-top attack boomerangs and makes me look stupid or petty or crooked. However, the journal has asked for only 1000 words (I’m not naming it in case they decide not to publish my review). So I’ll dump the notes here — which will be long — then write a formal review from these notes.
The first section is my attack on his ethics, and then I’ll get into the book itself. (Some readers may want to just read this section, then wait for the review.)
The Regnerus case
It is important to separate three problems with Regnerus’s academic work. The first is its poor quality, but that’s the least important. If it was just another pile of low quality sociological research it wouldn’t be worth getting this worked up over it. However, the fact that it’s so bad is important context for considering, for example, why Oxford University Press would publish it, or why the Wall St. Journal runs an excerpt.
The second issue is his repugnant, fanatical political and religious views. This is obviously a matter of taste, but there is no sense denying it as if I’m some sort of dispassionate methods or ethics vigilante. I care much more about taking on his work because of the bad he is trying to do with it in the world.
Those two issues will figure in the review, but the ethics issue will less so, so I’m getting into it here, with references. To get background on the story of the Regnerus Affair, you can read the chapter in my book, or read the entire Regnerus thread on this blog, or read this 2015 recap, which is the latest long piece, with links to everything else. For purposes of this discussion, these conclusions are salient: he used crudely biased survey methods to gin up harms attributable to same-sex parenting, to help stop same-sex marriage in the courts, as part of a conspiracy with other right-wing academics (principally Brad Wilcox) and institutions (Heritage Foundation, Bradley Foundation, Witherspoon Institute), which included manipulating the peer review process to plant supporters on the panel and submitting the article for publication before the data collection was even complete, and then repeatedly lying about all that to cover up the conspiracy (including in the published work itself, where he falsely denied the involvement of the funders, and in an ethics proceeding by his university).
So what do we do with all this now? All that didn’t get him fired, and he still does research in the academic system. That is galling, because there is at least one really good, honest researcher who doesn’t have a tenure-track job today because Regnerus does. But that’s the system. Meanwhile life is long, people can change. In our weak system, however, which relies almost entirely on good will and honesty by researchers, reputation matters. With his reputation, you simply can’t take his word in the way that we (naively) do with regular researchers. I think there are two options, then, if we are to take the research seriously. The first is he could come clean, admit to what he did, and make an honest attempt to re-enter respectable academia. The other (non-exclusive) option is for him to make his research open and transparent, to subject it to scrutiny and verification, and let people see that he is behaving honestly and ethically now.
He has not yet done either of those things. If he ever comes clean and admit what he did, that would be a (welcome) sight to see. On the second option, he has made noises about openness. And with the anti-gay research project, he did make the dataset public (after he published with it), which allowed it to be picked at, and then thoroughly (paywall; sci-hub) debunked (open). Now, with the data collection he did for Cheap Sex, called Relationships in America, for which he was the principal investigator (again using private money), he may have learned his lesson. The website promises, “In an effort to allow others to build upon our work, and as part of our commitment to transparency in research, we are pleased to announce that the full data set will be made available in mid-2015” (this is on the website as of January 2018). The announcement of this sort of thing brings much of the benefit in terms of reputation, because credulous readers who think he seems cool don’t know that he’s lying.
Anyway, to cover my bases, I emailed the Austin Institute for the Study of Family and Culture, the non-profit Regnerus currently uses to launder his tax-exempt contributions (it’s worth a trip through their website sometime), to ask for the data. On November 16, 2017, I wrote to the “research team“:
I am writing to inquire about the availability of the Relationships in America data. I am interested in replicating the analysis by Mark Regnerus in Cheap Sex, and conducting additional analysis. Please let me know how I can obtain the data for this purpose.
After receiving no response, I resent the message a month later (and needled them on Twitter), and got a message back from Kevin Stuart, the executive director of the institute:
When the Relationships in America report was written, Professor Regnerus estimated he would finish his work with the data by late 2015. The book project was delayed, and subsequent analyses of the data are still ongoing. When those are finished, we will release the data and announce it on social media. Whether that is in the new year [2018], or even late next year, I do not know. Should you wish to receive the data collection agency’s project report, which includes the survey instrument, we would be happy to provide it.
Obviously, “commitment to transparency in research” doesn’t mean you wait to share the data till you’re done with it. (Ideally, reviewers should have access to the data in the peer-review process, although this is often not practiced.)
As a reviewer, knowing Regnerus’s history of dishonest behavior and unethical research practices, no one can believe anything he says which can’t be independently verified. So it’s difficult to write a review of his work. Of course, it would be best if everyone’s research was open and transparent, so everyone was accountable, and we didn’t have to go through this evaluation of people’s ethical credentials. But that’s not the system we have. The approach I decided on was to not accept any of the facts he reports from his original research, but to discuss the methods he claims to have used as if they were real. I’m open to suggestions on how to handle this. I think it’s very important not to give dishonest researchers a pass as long as we’re stuck with this lousy system of unaccountability. Time passing is not sufficient to regain the public trust. (For more on the system we have, read this excellent review article by Jeremy Friese and David Peterson; and to help fix it get involved with SocArXiv.)
Notes on Cheap Sex
An interesting thing about “peer-reviewed” books in sociology is that the intensity of the review is highly variable, with some books receiving thorough reviews and some receiving virtually none at all. Hardly any, however, receive the level of scrutiny that articles in the prestigious journals (usually) get, with detailed, blow-by-blow critiques of their methods, findings, and interpretations. If the typical top-notch article receives maybe a dozen hours of reviewer time, repeated several times for revisions, how much attention should a peer-reviewed book receive? Relative to the scale of the research, it’s invariably much less.
The other thing about academic peer-reviewed book publishers is that they make their decisions to publish much more according to marketing considerations than do most academic journals. Although university (branded) presses are often non-profit, and may lose money for the institutions they serve, they need to sell books. Oxford, the biggest university book publisher, sells a lot of books and makes a lot of money. They are not non-profit, with reported profits of more than $100 million in 2017 (a down year). Prestige is their brand (they claim to have printed the very first book), but money is money, and they want books that sell, too.
Anyway, I did these notes as if I were peer reviewing the book, but more thoroughly than I normally would because I intended to make them public. (Question on the future of peer review: if people will read your reviews, and evaluate you based on their quality, would you write better reviews? I would.)
For some background, I have written previously about the “sexual economics” theory Regnerus got from Roy Baumeister, and its insane sexism, here; and about the Catholic stance on the gender binary to which Regnerus subscribes.
The notes are in sections rather than a single narrative. Here goes. From theory to methods, more or less.
Theory in a nutshell
It is common in a review to demonstrate you have read the material by summarizing it briefly. This is my summary: “Why buy the cow when you can get the milk for free,” is the theory in a nutshell. The book is an extended rant on that theme.
24-25: “So men want more sex than women do, on average. Economically speaking – at least in the heterosexual world – women have what men want. … So in the heterosexual mating market broadly understood, there is demand – interested men – and supply: women.”
So, in heterosexual exchange, “men’s sex objectively [has] no value,” while women’s does (but only to men).
Because women want marriage and men want sex, there is a “split mating market,” and women are overabundant on the wanting-marriage side so they have a hard time getting married. You would think this means there is a shortage of women on the just-wanting-sex side, which would hurt men’s ability to have no-strings sex so cheaply. But actually “the modern mating market [plays] more to men’s advantage than to women’s – that is, he gets what he wants more readily and consistently than she does”? (27-28). Why? His explanation is that women in the marriage market are in such abundance that they have no choice but to cheapen their sex in the hope of getting a man – and any that holds out is undermined by the sluts (who in the old days would protect other women through their “cartel”). So it turns out the mating market isn’t split after all, because the sex-wanting side is flooded with women who want marriage but have no choice but to be sluts if they want any chance at marriage. If women would just hold out collectively – rebuild the cartel they had in the good old days of patriarchy – they could pull men over from the no-strings side of the market (and men would have to then work harder and be more ambitious generally to get sex), but women don’t do that because they are sluts.
Definition:
“What is cheap sex? … Cheap sex is both an objective fact and a social fact, characterized by personal ease of sexual access and social perceptions of the same. Sex is cheap if women expect little in return in return for it and if men do not have to supply much time, attention, resources, recognition, or fidelity in order to experience it” (28).
Note sex is something that men buy and women sell, period.
The pill is the major technological shock that helps drive all this, with its associated “mentality.” Then porn, and online dating/sex services “created a massive slow-down” in marriage, which “put the fertility of increasing numbers of women at risk,” and “have arguably even taken a toll on men’s economic and relational productivity [I think that means marriage?], prompting fewer of them to be considered marriage material than even before” (11)
Theoretical incoherence
This was a large category of notes, into which fell claims and statements that seem theoretical, but either don’t make sense or are contradictory.
He goes from the fact that men want sex somewhat more than women do to that difference being the very definition of sex. “Heterosexual sex exhibits an exchange relationship wherein men access sex that women provide, typically in return for desired resources” (60). Question: how great an imbalance in sex drive would there have to be for it to be the defining characteristic of all sex? He doesn’t ask this question.
“Remember, sex is her resource, and in a consensual relationship she controls access to it. It doesn’t happen if she doesn’t permit it” (95). By definition (consensual) this is true of men as well. The fact that Regnerus says this only of women is very important: what is happening (society going down the tubes) is because women are opening their legs.
The theory is misnamed the “exchange model,” which implies that people exchange things in sexual relationships. But it’s actually only between men and women, heterosexually, what he calls “the supply of sex and the supply of resources” (46). “The exchange model is rooted in stable realities about male-female differences that are not socially constructed and will not disappear” (44). In other words, it doesn’t apply to same-sex sex (see below). “The exchange model can neither be reversed nor declared dead” (45). He pretends to base this on science (biology), but it is really a religious affirmation, representing the abuse of science by Catholic doctrine, the leaders of which have decided to embrace words like “natural” and “science” while imposing their preordained view of truth on them, especially with regard to gender (this is how they frame their opposition to marriage equality). Not only can it not be reversed or declared dead, but it “may bend but it won’t break,” and it “may be old fashioned but it is not faulty” (45).
For the model to fail, he lists a series of supposedly-impossible things that would have to happen: “Men would pine to stay at home longer with their infants. Women would play fantasy football. All unlikely scenarios” (45). However, in 2011, 21% of fathers were the primary caretakers for their preschool-aged children, meaning they provided more hours of care for them than any other person or arrangement (https://www.census.gov/prod/2013pubs/p70-135.pdf). Women are 29% of fantasy football players (https://fsta.org/research/industry-demographics/, 12/24/17).
Does “sexual economics” apply to homosexuality? He has a whole section about this, saying yes but answering no. The “sexual economics approach concerns the distinctive relational interests of men and women, whether they are gay or straight” (54). In his view, homosexuality is basically a behavioral malfunction that doesn’t change people’s God-given “relational interests.” There is no sexual exchange in same-sex relationships.
The definition of “cheap sex” is men getting sex for lower cost: “men have to do less wooing (fewer dates, less costly indicators of commitment, etc.) in order to access real sex. Hence, sex is cheaper” (11). It’s not clear why masturbation (whether or not with pornography) is also cheap sex. How are pornography and masturbation “the cheapest forms of sex” (107) if they don’t involve women exchanging anything with men? If a man masturbates alone, how is he getting sex from a woman? What is the definition of sex, to which a price may be attached, if it’s not being bought from anyone? E.g., he describes the increase in pornography as an increase in the “supply” of sex (11), totally against his own definition. It’s just not clear how, under his theory, masturbation and pornography consumption are sex.
“Women have plenty of agency, opportunity, and success … Women can openly pursue sex for its own sake in a manner utterly foreign to their great-grandmothers. They can try the demand side of the equation. Of course, they will succeed in their efforts” (26), by which he means men will gladly have sex with them, because they are sluts. What does “try the demand side” mean, though? He just said (24) “women never pay men for sex.” This caveat seems like a recognition that his theory is wrong, but he doesn’t incorporate it substantially.
He quotes Baumeister and Vohs: “Once women had been granted wide opportunities for education and wealth, they no longer had to hold sex hostage.” Although Regnerus says “hostage” is an overstatement, he endorses the narrative (46-47). But the economics here is incoherent. When women had no wealth or power, they completely controlled access to sex, and held it “hostage” to get marriage. Now that they have everything they need without a man, they give sex away for nothing. He says: “If women no longer need men’s resources … then sex simply becomes less consequential, easier to get or give away” (51). He returns to this: “The question to ask is why women demand so little of men in return for offering men what they want — what they are willing to sacrifice a great deal for. And the answer is economic: it is because many do not need what men can offer” (67). In reality, of course, poor women seem to “demand” marriage less than rich ones do, so this seems wrong. But further, what economics works like this? Sure, when buyers have no money sellers lower the price, but in this case why don’t they just keep it? If they can get the money they need from their jobs, and men aren’t giving them love or protection anyway, why do they “have” (give) sex for free? The only answer is they are stupid, and sluts.
Oddly, he says “Online dating’s superior efficiency works against relationship development … and positively rages against the goal of efficient marriage market ‘clearing’” (70). What is the definition of efficiency here? Usually it’s a combination of quality and quantity, but he uses “efficient clearing” to refer to the number of marriages period, regardless of quality. He acknowledges online dating could be a way of “maximizing the likelihood of locating a spouse who is more desirable,” which would seem to be “efficient,” but, “more often we are allowing ourselves to treat human being as commodities” (70). This is a non sequitur.
One good old days example he uses is his own marriage. His then girlfriend dumped him for “being distant, unpleasant, and uncertain about us,” and then she went on a date with someone else. But then because “the search costs [were] fairly tall” he decided to call her and “we were back together before the weekend was out” (70-71). Her role in the decision is not specified. (Why is this story here? Unclear how it fits his model.)
Is this an evolutionary theory or not? “Men can see more flesh in five minutes than their great-grandfathers could in a lifetime,” and “they can do that in seconds in a way unanticipated by their genetic material … In other words, humans are not evolutionarily familiar with the accessibility, affordability, and anonymity that Internet pornography offers” (107). I am pretty sure men saw more female nakedness in the environment of evolutionary adaptedness than our great-grandfathers did, too – especially in the million or so years between when we lost our fur and the development of corsets. Raising the fascinating question: what if you showed modern porn to Paleolithic men? Would they have stopped hunting and gathering if they didn’t need to demonstrate their physical prowess in order to see nakedness? Of course it’s true porn has changed sexuality, there are just a lot more useful things to read about that than this book.
Historical mythmaking. Describing respondent Carlos, who masturbates a lot even though his girlfriend wishes he wouldn’t: “There may have been an era in which Carlos would have had trouble retaining the sexual interest of a woman, but that era is no more” (111). When did it matter more that men were sexually desirable? With universal early marriage and no divorce? Is that when women were more free to dump a man they didn’t like? (No.)
Weird jag on the military and same-sex marriage (184). He is sure the military has turned away from supporting marriage because it allows same-sex marriage. How, though? All he can come up with is that because the military defines adultery only as heterosexual sex, “enforcing its own adultery codes would first require considerable revision before new prosecutions can move forward.” So they can’t police adultery with same-sex couples. OK, this would presumably only affect adultery among gay service members, though, so I don’t get it how he jumps to this: “I cannot imagine that [revision] occurring. Hence the armed forces’ recession from actively supporting marriage and generously benefiting married spouses leaves organized religion as the only obvious, active institutional supporter of marriage.” That is just nuts; the military is of course very supportive of marriage.
Deep sexism and the gender binary
In my book I organized the story of Regnerus around his determination to protect the gender binary. This is where he becomes most passionate, and irrational and religious (Catholic). There is a lot of this in Cheap Sex.
“I write, too, as a man, yet one who has concerns that are in historical alignment with women’s (and many men’s) long-standing relational interests — things like commitment, stability, monogamy, tranquility, and a family” (21). (Also file this under confusion caused by imprecision – does “many men’s” imply these are ALL women’s interests? Otherwise what is the distinction?)
Homosexuality is not real; this is a recurring, unstated but undeniable theme. Lesbian couples have less sex because they are women, and homosexuality can’t change their nature. His evidence is 52% of lesbians say they want more sex than they’re having. Why?
“Sexual economics provides an answer rooted in evolutionary psychology which suggests that just because someone self-identifies as something other than heterosexual does not mean they are able to just opt out of deeply embedded sexual differences in socio-sexual behavior…. The frequency [of lesbian sex] is lower due to the fact that the couple is comprised of women, who are historically sexual gatekeepers” (82).
Thus, they may “mimic the heterosexual exchange model. But mimicry it is” (83). So they cock block each other, I guess.
Reification of nature as juxtaposed to culture. He’s talking about contraception and ART as “a concerted accomplishment of synthetic technology undermining nature in the service of human consumption” (199). He also elevates “nature” with regard to marriage: “It will not be deconstructed, because it is not a mere social construction” (207). What’s most natural of all is the gender binary.
Everything nowadays is going against nature. And gay marriage maybe most of all. “The successful movement to ‘de-gender’ civil marriage in the West has reinvigorated efforts aimed at the general dismantling of gender and male-female distinction,” which is part of the feminist agenda “aimed at obliterating all sexual difference – that is, eradicating the truth of sexual dimorphism” (212). Thus he insists “sex is observed, not ‘assigned,’ at birth” (213). He approvingly quotes Dick Udry as saying, “A social engineering program to de-gender society would require a Maoist approach: continuous renewal of revolutionary resolve and a tolerance for conflict,” and Regnerus concludes: “And since it builds upon a theory of sex differences that is empirically groundless, it won’t work” (214).
Some weird sexist asides. E.g., holding the position that porn use is a deal-breaker for their relationships “would likely backfire on women (as many things tend to do in the domain of relationships)” (132). E.g., this is a list of “unintended consequences” of delayed marriage: “more living alone, more dual-earner families… more infertility concerns, more solitary sex” (173). Odd to include dual-earner families on that list.
Many examples of women causing problems. E.g., his slutty respondents don’t realize they are ruining it for other women. “What Nina and Sarah (and numerous others) do not realize, however, is that even wasted sex is priced – cheaply – and contributes to the socially discernible cost of sex in the surrounding mating market” (176). They are selling out other women. “In the domain of sex and relationships men will act as nobly as women collectively demand” (177). The assumptions here are that (a) men’s bad behavior is women’s fault and (b) men’s behavior used to be more noble (measured, presumably, by marriage rates).
Finally,
“Who are the winners in this new relational regime? The easiest to spot, of course, are career-minded women, for whom access to the contraception that made sex far cheaper also enabled them to finish education and commence careers… in so doing fostering new structured patterns (and a culture of expectation) of career building. … There are other winners. Sexual minorities … sexually opportunistic men … the wealthy … short-term corporate profit … [and] America’s late modern capitalist economy” (194-195).
These “career-minded women” are in a list of decadent cultural parasites.
Confusion caused by imprecision in writing
Moving from theory and argument to more mechanical critiques of the book, there are a lot of passages – a lot – where the specific meaning is literally ambiguous, impossible to discern from the text; or where poor writing and editing creates logical contradictions. These are examples.
“In fact, the relationship histories that young Americans tell us about are growing increasingly predictable: plenty of sex, starting early…” Increasingly predictable means decreasing variance in experience, but that’s not happening; rather they are increasingly conforming to the narrative that he is describing.
“I am after answers to several important questions, including… Is marriage still perceived as a key goal, or is it increasingly perceived as optional” (13). These are not mutually exclusive.
“Men, on average, are more often principally drawn to the powerful physical pleasures of sex than women are” (22). In this sentence, “on average,” “more often” and “principally” are all imprecise modifiers just creating mud. And on the next page, “I know that women can and do like sex. Rest assured, though, that men—historically, and on average—tend to want sex more and pursue it with greater abandon and single-mindedness” (23). Why is “can and do”? Why do you need “on average” and “tend to”? etc.
“The bottom line is this: women are the sexual gatekeepers within their relationships. Men infrequently function as the ‘gatekeepers’ of sex in their relationships. (If they are, they are comparatively easier to convince)” (26). The first sentence is contradicted (in different ways) by the next two.
Regnerus writes, “most young adults still pay deferential lip service to marriage,” then later on the page, “most of them [Americans] – especially women – are still invested in monogamy and marriage” (32).
“The mating market in this ‘state of nature’ [before the Pill] was populated by roughly equal numbers of men and women, whose bargaining positions – averaged together – were roughly comparable and predictable, with men valuing attractiveness more than women, and women valuing productivity and economic promise more than men” (35).
What does this literally mean? What is the “average” of attractiveness and productivity, and how are they “roughly comparable”?
He concludes a section of speculation about how cheap sex is probably causing same-sex relationships by saying, “The bottom line is that as Americans’ sexual culture becomes less heteronormative, which appears to be the case, the effect of it on mating market dynamics is almost certain, but it is not simple to predict. For that reason alone, we ought to pay attention” (60). How is something “almost certain” “not simple to predict,” and what is “that reason”?
“In reality, the phrase [lesbian bed death] indicates a process by which lesbian couples are thought to diminish the frequency of sexual activity within a relationship over time, until their baseline average is well under that of gay men’s or straight couples” (81). Incorrect use of “baseline,” and apostrophes.
“Recall that Sarah delayed first sex until well after her adolescence was over, in step with a trend that the CDC data has long noted: from 1988 through 2013, the share of teenage girls who were sexually experienced declined from 51 to 43 percent, respectively. And yet that did not seem to matter much about what happened next. Over the course of her twenties, Sarah slept with numerous men” (85).
In addition to the redundant “respectively,” the “and yet” doesn’t follow at all, because the trend is about teens, so starting to have sex in her twenties fits perfectly. He probably means “the trend shows increasing chastity, and yet she was actually slutty.”
Ironically, just after saying his interviews “found men consistently inarticulate about the subject” of pornography, he writes this sentence: “Just like the psychiatrists debating the matter, these men are seldom prepared to label it a problem, but they also clearly display enough halting conversation about it that neither are they prepared to suggest that nothing is wrong or off-kilter” (127). You spend a lot of time reading this book knowing he’s saying things wrong but sort of knowing what he’s saying. If you won’t write better than this, get an editor.
Is masturbation becoming more prevalent? “While it is impossible to say for sure, the existing evidence supports the notion that masturbation has increased in frequency – recently – and is arguably at an all-time high” (138). Four pages later, “Pornography and masturbation … are surging in popularity” (142). He was right the first time – his evidence doesn’t support that unqualified conclusion.
“While genuine demand for masturbation could have naturally increased in 20 years, there’s no reason to think it would at this point in history, unless the technological and social fostering of sexual desire (and hence demand) has increased. And it has, revealing that male desire and arousal is not fixed; it is malleable and can and is being stimulated” (139).
Question, does technology cause “natural” increase? If not, what is “unless” doing here? What does “it has” refer to? “desire and arousal” are two things, so what specifically is “malleable and can and is being stimulated”?
“When more and more men are considered less and less marriageable, the resulting sex-ratio disparity in the pool of marriageable men tends to spell greater and greater problems for women in how they conduct their relationships” (152). Try again while pretending words have meaning.
What do the various uses of “it” refer to in this passage?: “Individuals may elect not to form marriages or families … but they are not capable of socially constructing monogamy out of existence. We are simply not free to write off fertility’s debt to love, its desire for exclusivity, and its idealization of marital union. It will resist and reemerge, if even only in wounded form” (184).
Contempt for respondents (and kids these days)
The book includes text which is said to be from interviews. There is no systematic analysis of the interviews or respondents. Who are they, how were they recruited? Who did the interviews? What was the interview schedule? He says he “limited the interviews to 100 overall” (14) but no general information is presented about them; and only a small number are quoted in the book. They serve two purposes: to repeat his conclusions back to him, and to be treated with contempt. In no case does he learn from them or tell us anything new from the interviews. He doesn’t learn from their experiences. I don’t like it when interviewers use their respondents as analysts instead of using their stories as data; maybe I’m too positivist.
When respondents restate his theory directly, and you wonder if they’re real or what
Alyssa (p. 49):
“I know in my mind, and from my feminist perspective, that sex is something that people come together for, that women and men should enjoy equally, and that there shouldn’t be any work on either part, there’s no trade-offs … But culturally, there’s definitely ingrained in me something that says it’s a gift that a woman is giving a man, and that he needs to deserve it.”
A 24-year-old woman who “waxed eloquent on the dating scene,” said men don’t ask women on dates. Regnerus provides his question, “Do you have thoughts as to why that is the case?” and her answer is a direct restatement of his thesis,
“I feel like the guys don’t do it because they don’t have to. I feel like the girls don’t make them… I feel like it’s just too easy for guys just to say ‘Hey, let’s you and I hang out and see what happens … And the girls aren’t saying, ‘You need to do this.’ You know, to win my affection, you need to take me out. You know, guys aren’t gonna do it, I guess, if we’re not making them” (65-66).
Ben, “28-year-old Denver-area man,” says relationship skills “have been cheapened with the advent of uh, I guess you could call it information-age sex. … I think it’s made sexuality a commodity in a huge way” (95). Totally authentic quote, dude.
25-year-old woman: “relationships are more casual than they used to be … So I think now that may, maybe it’s because women have taken on a stronger role in both relationships and, and pretty much everywhere that that might have something to do with it” (96).
Respondent:
“People from my generation, anyone who grew up with the Internet, get a lot of their ideas about sex from porn, and I think that sex didn’t used to be the way it is now … that seems slightly unnatural and out of line with my idea of kind of purpose and function of romantic sex in a traditional relationship” (118).
Showing contempt for his respondents, and kids these days in general
“Alyssa, a 27-year-old from Milwaukee, told us she had higher libido than her live-in boyfriend. While nearly everything about her past shouts ‘cheap sex’ and the problems that often accompany it…” (49). Nice.
Alyssa started having sex at 15 and has had “almost 20” partners, she “struggles with monogamy” (his term), and may be bisexual. “Despite all the sex-related problems she has endured and, in some cases (by her own admission) provoked, Alyssa has remarkable insight on sexual influence and, at age 27, hopes the future is more stable. She can even envision marriage, something she has seldom witnessed” (119). Wait, why should someone have insight “despite” having problems (of her own making or not)? And why is it surprising that she can “envision marriage”? And who has “seldom witnessed” marriage in the US?
“Wen, a bubbly 28-year-old Asian American from Austin” (150). Don’t call your Asian women respondents bubbly. Also, he doesn’t describe any of his respondents as White, but does describe three as African American (pp. 45, 50, 95).
“While Elizabeth’s high hopes for enduring marriage seem noble, her disdain for dependence upon a husband and her knee-jerk criteria for leaving nevertheless suggest the pure relationship mentality has profoundly altered how she understands marriage.” And then he quotes her: “Maybe one day my husband will fall in love with somebody else. What am I gonna do? Or he cheats on me or he hits me. You know, then I’m gonna have to get out” (159). Her “knee-jerk criteria” are cheating and violence. Nice.
This is unverifiable second-hand aspersion: “Kendalia, a 32-year-old African American woman from Milwaukee cohabiting with an unemployed man who spends most of his days playing video games and watching pornography” (50).
A respondent who thinks she “doesn’t even need marriage to enjoy a successful life … mistakenly equates elective decision-making about sexual and reproductive health with signals of deep human flourishing” (177). In other words, she doesn’t share his values.
Unsubstantiated imposition of his preconceptions
The jumps between what Regnerus claims as evidence and the conclusions he offers are ridiculous. The missing link is his preconceptions, which are always confirmed.
As “physical risks of sex” have decreased and “economic trajectories of women have soared … this new era has been remarkable for women in terms of career options and labor force success, but more challenging on them relationally.” This is presumably as defined by lower marriage rates, as no other evidence is given, but “the route to marriage – something the vast majority still holds as a goal – is more fraught with years and failed relationships than in the past” (43). So the 1950s marriages were not challenging “relationally” because they married young after a short search. Often pregnant. Not challenging at all. He elaborates that when women no longer need men’s resources, “the relationships are far more difficult to navigate because strong commitments and emotional validation are just plain less necessary (and thus slower to emerge) from men” (51). Is there less commitment and emotional validation now that divorce is an option? I’m skeptical, but there is no evidence presented on that either way. He returns to this in expressing disagreement with Giddens: “While Giddens was on target to hold that ‘sexual freedom follows power and is an expression of it,’ it simply does not spell the power to make relationships flourish and last” (51). But does it spell the power to end bad relationships? This is not important to him.
He believes homosexuality is not natural, but is the result of “sexual malleability” made possible by the Pill, etc. As an example he recounts a story he heard while “chatting with a friend of mine” whose sister is “in a same-sex relationship” and “eventually married a woman” (he does not call her a lesbian). He then descends into a pseudo-scientific jag about her, starting with how her coming out “coincided with early twenty-something difficulty navigating the relationship world of men as a tall, athletic woman. She didn’t fit in and was seldom asked out. I am not at all suggesting,” he says, while 100% suggesting, “this experience was a key reason for trending toward relationships with women” (58). So why bring it up? He goes on to mention that this “happened for Amanda in a historic period of political change around sexuality,” then mentions that Washington, DC is “a city known for having the worst sex ratio in the country” (58). On the second page of this speculation,
“We should expect that some share of women will respond to perceived mating market constraints and struggles by experimenting with same-sex-relationships. … This need not be the case for all or most self-identified lesbians or bisexual women [note he never concedes lesbians actually exist]. No matter. In Amanda’s case, she told her sister [says her sister?] that she very well could have ended up with a man had interest from such been expressed and received at critical times. But it did not happen.”
This is a belabored way of saying “relationship difficulty” made her “trend toward relationships with women” because she is definitely not naturally a lesbian. (Also, why use long anecdotes from friends when you supposedly did 100 interviews?)
He offers a two-page description of the 2012 American Sociological Review article by Elizabeth Armstrong, Paula England, and Alison Fogarty (only the first two of which he names), which analyzes the determinants of orgasm and sexual enjoyment among college students. He criticizes them for “elect[ing] to focus mostly on sexual technique, something they note was more apt to occur when sex partners liked each other enough to be in a relationship” (105), rather than focusing on relationship status and aspirations (which also affect the outcomes). Here he seems to substitute the term “sexual technique” for “oral sex” (which is what they measure but he never mentions) as something that is “more apt to occur” in relationships. Maybe he is afraid if he mentions oral sex – which has the largest effects in their models on both odds of orgasm and sexual enjoyment – he will give people the idea (like porn does).
Does porn make men gay? He could have studied this, but he didn’t. So he just speculates and offers a “hypothesis.” He says porn is an “influential teacher,” and
“some of its lessons … are learned, liked, and repeated. As an example of the hypothesis I am posing, straight men are glimpsing other men having sex (with women) in pornography – the ‘cumshot’ scenario in porn is not just common but listed as popular and desirable by straight men when queried about their own pornography preferences. In reality, then, straight men are attracted to the sexual pleasure of other aroused men. I am not suggesting here that porn use leads straight men to ‘turn’ gay. No. What I think is a reasonable interpretation, however, is that pornography is indirectly shaping (and increasing) the sexualization of situations, what people are willing to try, and what they come to desire sexually. … Pornography, then, is blurring the lines between sexual orientations, contributing to the growth of what is sexually attractive” (123).
So it’s making them want to have sex with men, but not to be gay. This is important because he doesn’t think anyone is naturally gay, so he needs reasons for why men would have sex with men. He returns to this later, referring to “the clear interest among straight men in the depiction of male pleasure and ejaculation” as part of a social context in which “more same-sex experimentation will occur” (208). He’s really stuck on the cumshot. What he misses about it, however, is that it might reflect not straight men’s attraction to male arousal, but how it makes erotic the degradation of women. Regardless, he has no evidence for this “hypothesis” except the fact that gay men watch more porn than straight men.
He thinks that men don’t have to work hard in general now because sex is cheap, but his history is off. “Previous cohorts of men who did not make ample wages were simply not considered marriageable and hence were unable to access sex with the regularity they craved. They worked for it, and some became marriageable” (173). It’s a great story of American greatness gone by, men pulling themselves up by their erections. But what is this history? From 1950 to 1980, about 90% of men were married by age 35. Did we not have low-earning men back then?
Straw men
Regnerus argues against demons.
He calls the social change “over the last several decades”
“technology-driven social change. Recognizing this counters the simple and reductionist explanations like ‘social construction,’ ‘the right side of history,’ ‘liberation,’ ‘enlightenment,’ or ‘the triumph of rights and freedom over ignorance and bigotry’ for the new variations in social sanctioned intimate relationships” (12).
There are no references to any of these “simple and reductionist” thinkers.
“My claims have less to do with lawyers, doctors, and executives than they do with regular people farther down the socioeconomic ladder – the kinds of men and women social scientists often claim to represent but frequently overlook in their own research methods” (14).
Such a rebel.
“Though no economist, Anthony Giddens agrees that contraception altered the playing field. (I know of no serious scholar who denies it, but few discuss it)” (33). Why would one need to be an economist to see that? And “few discuss it” is hilarious. See Wu, Martin, and England 2017:
“What led to the decoupling of sex and marriage? A conclusive causal answer to this question will likely remain elusive, but many have argued, on plausible theoretical grounds, that advances in contraceptive technology and the introduction and diffusion of the birth control pill in particular were decisive factors by allowing women and couples far greater control over whether and when to become pregnant” (emphasis added).
My own textbook says, “Few innovations had a social impact to rival that of the birth control pill.”) Note, however, that his whole discussion of the Pill, which runs throughout the book, is about how it affects dating and mating for unmarried people, but one-third of people who use the Pill are married, and it’s improving their family as well as their work lives.
“As pornography increasingly saturates American private life, it is become scientifically untenable to maintain that porn doesn’t matter” (123). Who says porn doesn’t matter?
Funny / ridiculous
Before getting into methods, pause for some laughs.
“There is wisdom to the slower pace of science” (19), says the person who submitted a paper before data collection was even complete.
No shit: “I lean conservative in my own life and personal perspective” (20).
Post-hoc reasoning
In the section on gays and lesbians, which is devoted to explaining how unnatural homosexuality is, he presents a figure he claims is from his Relationships in America survey, which shows women are least likely to be “100% heterosexual” in their late twenties. We “should not be surprised” about this, he says, because women “face a fixed fertility schedule” (57), so they become straighter as they age into their thirties. If he was being honest, he would admit it’s weird for this theory that women are least heterosexual during their peak fertility years. Instead, he pretends it’s not surprising they get out of their homosexual years just in time for the end of their “fertility schedule.”
Sloppy and slapdash analysis on porn effects
Sometimes he analyzes women, sometimes men, without explanation. This smells like effect shopping, consistent with this very selective and incomplete reporting on his analyses in general. But because the data aren’t available, we can’t investigate. For example, on porn, he reports use declines with age more among women than among men (115): “Why the greater attraction of pornography among younger women? Speculation is difficult to avoid.” (It might be hard to avoid, but you don’t have to publish it.) Anyway, the speculation that follows offers nothing for why this pattern would occur for women and not men (the proposed mechanisms all would apply to men as well). From there he goes to the effects of porn – on women, ignoring that men use it more, suggesting it “undermines long-standing ideas [and values] about marriage” (120). To support that, he discusses women who “say they never watch porn” (why this category, when he has a continuous measure?) who are “least likely” to cheat in relationships and “most likely” to disagree that “traditional marriage is outdated,” with no reference to the comparison groups, adding, helpfully “(results not shown”). He concludes: “It makes sense. Porn use thus appears to constitute a liberalizing force” (120-121). No reason why this discussion is just about women.
Later (123-126), he asks, “Does heightened porn use matter for fashioning political attitudes about marriage?” and answers, “It does among men.” There is no reason for why he doesn’t include women in this discussion or analysis. He says there is a “linear association” between pornography use and men’s support for same-sex marriage. The regression table (221) says it is OLS regression but doesn’t define the independent variable beyond, “Last pornography use (behavior, 0 = most recent”; or the dependent variable, beyond “support for same sex marriage.” Are these scales? There is no information on how they are measured or coded, needed to judge whether the use of OLS is appropriate. There is also no measurement specified for five control variables in the model (education, income, social media use, religious attendance, and importance of religion). (In all his tables he practices asterisk inflation, so * = p<.10, etc.)
Regnerus doesn’t like the GSS porn question, which is reasonable. But note I did a quick analysis of GSS and find that among both men and women, those who have watched an x-rated movie in the past year are more supportive of same-sex marriage rights, since 2006 when they started asking the question, controlling for age, race/ethnicity, education, political views, and religious attendance. (This paper found porn increased support for same-sex marriage in GSS 2006-2010, but only among low-educated men.)
He finds this porn effect hugely significant. How could porn affect views on gay marriage? It’s not about men watching gay sex, but rather about porn’s “veritable fire-hose dousing of sex-act diversity … different positions, roles, genders, and varying numbers of participants — and that is basically where porn leads today: away from sex as having anything approaching a classic marital sense or structure” (125-126). He also cites evidence that porn users are more likely to identify as feminists.
Without any ability to assess the actual effect of porn on marriage equality attitudes (we can’t even judge the effect size in his model), it’s impossible to evaluate his bizarre conclusion:
“In the end, contrary to what very many people might wish to think, men’s support for redefining marriage may not be the product of actively adopting ideals about expansive freedoms, rights, liberties, and a noble commitment to fairness. It may be, at least in part, a passive byproduct of regular exposure to the diversity of sex found in contemporary porn” (126).
Quite a leap. (And again, why is this just about men?)
He also suspects that porn is encouraging some men to stay out of the marriage market, because they can just masturbate. To support this, he has a long quote from Milo Yiannopoulis (yes, really), one interview quote (“porn is one that uh, that I can’t get away from. … Sex you have to go look for, usually” [129]). Based on that evidence, he writes, “The question is not whether some men have exited the market, courtesy of porn. The question is how many” (130). (What does “courtesy of” mean?) Then he says Relationships in America shows that 29% of never-married men under 50 selected “haven’t really dated in the past year” (which he describes as “had not dated in the past year”), and of them, the majority are regular porn consumers. From this weak evidence, he concludes:
“It may be too much of an assumption to hold that such frequent pornography consumers who report no dating in the past year because they don’t want to or think they’re uninviting are off the mating market because of their pornography use, but I hold that their porn use may be undermining their participation” (131).
Since this is obviously a major question for him, why didn’t he design some research around answering it? (Also what does it mean to “hold” that something “may” be the case?)
Methods
There isn’t much for methods in the book. I don’t include the supposed 100 interviews under “methods,” but nothing about that effort, as reported, qualifies as research; he just talked to some people. He does have a few descriptive statistics and some regression tables that he says are from the Relationships in America data, inadequately described. I discuss them here.
For women, but not men, Regnerus says there is a relationship between political views and wanting more sex (77-80). Conservative women are less likely than liberal women to say they want more sex than they are having. He shows this table for percentage of women who want more sex, which he says derives from the Relationships in America survey:
Very conservative: 16%
Conservative: 30%
Moderate: 38%
Liberal: 44%
Very liberal: 53%
These are the terrible research methods he then employs to investigate the question:
He says (but does not show) that this correlation is not found among men, a discrepancy which does not raise any questions for him, and to which he never returns, although the speculative theory he introduces could just as well apply to men as to women.
He treats this as a question of liberalism’s effect rather than conservatism’s effect, as if conservatives live in a natural state, so their views need not be explained; also as if women desiring sex is the condition that needs to be explained, instead of the reverse. So he says of the table, “It is obvious that more politically liberal women are apt to say they would prefer more sex. Why?”
To try to understand the liberalism effect, Regnerus says, “I discussed this conundrum with others, and a plausible, four-part path explanation emerged.” That path is laid out as follows: (a) liberals are less religious, (b) they “therefore are more likely to have a difficult time attributing transcendent value” to life, but (c) they see sex as transcendent, so (d) “liberal women therefore desire more frequent sex because they feel poignantly the lack of sufficient transcendence in life.” Some path.
To test this post-hoc theoretical speculation, Regnerus presents a logistic regression table, with odds ratios and no confidence intervals or standard errors, but with asterisks, one of which indicates a p-value of .10 or less (no reason is given for this non-traditional alpha level). The dependent variable is dichotomized, to indicate those who want more sex than they are having. The independent variable is labeled “political liberalism,” but it is not defined in relation to the five-point political views scale he describes in the text.
The logistic table has three models and an N for each of 1,387: the first has liberalism, age, race, education, and marital status. The liberalism odds ratio is 1.39** (which means p<.05). Is that for the difference between liberals and all others, or for each point on the political views scale? No way to know. The second model adds five controls for sexual behavior, orientation, and emotional well-being. Now liberalism’s odds ratio is reduced to 1.24* (p<.10). The final model adds three religion controls: importance of religion (odds ratio: .99), religious service attendance (odds ratio: 1.03), and “less religious than 10 years ago” (odds ratio 1.68**). Now the liberalism odds ratio is… the same, but no longer has a p<.10 asterisk.
Regnerus writes triumphantly of this result: “So I added religious service attendance, importance of religion, and a unique measure of having become less religious in the past decade to the regression model predicting wanting more sex, and – as theorized – becoming less religious predicts wanting more sex. And what is more, political liberalism no longer matters for wanting more sex.” If the coefficient doesn’t change, but the standard error increases, can you say it “no longer matters,” if that change happens to push it over the p<.10 threshold? No, you cannot.
He adds, “This theory about replacing the loss of the sacred (with a quest for sex) is a plausible one.” (Good to know.) “Unfortunately,” he intones, “something so immanent as sex will not – and cannot – function in the manner in which religion can, has, and does.” (He loves these, will not, cannot, can, has, does sequences.) After a little more of this, he concludes, “Maybe that is why very liberal women are also twice as likely to report being depressed or currently in psychotherapy than very conservative women” (there are no details provided; twice as likely as whom?). (In 2016 the GSS included the CESD depression scale, which seems to be the scale Regnerus used; my quick analysis of that, with basic controls, shows that “extremely liberal” and “extremely conservative” people (male or female) are more likely than people with moderate views to be depressed, but not significantly different from each other.)
Empirically false statements
I noted a handful of just false statements.
“There are now more women than men in the paid labor force” (11). This unattributed fact, presumably from Hanna Rosin, is not true and has never been true, as I explained in this blog post.
“Women never pay men for sex” (24). They do a lot less than men, obviously, but some do nonetheless (e.g., there is research on sex tourism in which rich-country women travel to poor countries for sex with poor-country men).
“In 1992, there simply was no online pornography” (139). There already was BBS porn in 1992.
“To be sure, things did not change overnight following [the Pill’s] debut in 1960. … But change things it did. The vagaries of less-reliable contraceptive devices or condoms, which men never much appreciated, could now be avoided. Marriage plans could be stalled. Careers could be developed without fear of interruption. Women could have two children instead of five or six” (33-34). The total fertility rate was below 3.0 before the Great Depression, and never reached 4.0 even during the Baby Boom.
Sex isn’t the only thing that motivates men. “(For example, men are powerfully motivated by competition in sports and business, but seldom over women anymore)” (153). Really, men “seldom” compete over women anymore? Is that because there just too many sluts to choose from? I’m skeptical.
Grammar and editing errors
Back to basics.
“In the world of sex, men and women often display differences, and it has significant and far-reaching consequences” (24). What is “it” (and what does “often display differences” mean?)
The third paragraph on p. 28 is out of place, starting with an “It” that does not reference anything in the previous passages and ending, bizarrely, with,“But you get the point.”
Incorrectly calls a difference between two percentages a “percentile gap” (90). Similarly, a woman “reported climaxing in her current relationship about half the time, but says that the 50th percentile is not a problem” (103). That’s not what a percentile is.
On 134-135, he says in the text that 24% of women and 9% of men say they have never masturbated, but the chart shows the numbers are about 27% and 13%.
The age of respondent “Elizabeth” changes from 26 to 25 (p. 25, 103, 156).
In the abstract, the missions of science and science reporting align. But in the market arena they both have incentives to cheat, stretch, and rush. Members of the two groups sometimes have joint interests in pumping up research findings. Reporters feel pressure to get scoops on cutting edge research, research that they want to appear important as well as true — so they may want to avoid a pack of whining, jealous tweed-wearers seen as more hindrance than help. And researchers (and their press offices) want to get splashy, positive coverage of their discoveries that isn’t bogged down by the objections of all those whining, jealous tweed-wearers either.
Despite some bad incentives, the alliance between good researchers and good reporters may be growing stronger these days, with the potential to help stem the daily tide of ridiculous stories. Partly due to social media interaction, it’s become easier for researchers to ping reporters directly about their research, or about a problem with a story; and it’s become easier for reporters to find and contact researchers to cover their work, and for comment or analysis of research they’re covering. The result is an increase in research reporting that is skeptical and exploratory rather than just exuberant or exaggerated. Some of this rapid interaction between experts researchers and expert reporters, in fact, operates as a layer of improved peer review, subjecting potentially important research to more extreme vetting at just the right moment.
Those of us in these relationships who want to do the right thing really do need each other. And one way to help is to encourage the development of prosocial norms and best practices. To that end, I think we should agree on a No Paper No News pact. Let’s pledge:
If you are a researcher, or university press office, and you want your research covered, free up the paper — and insist that news coverage link to it. Make the journal open a copy, or post a preprint somewhere like SocArXiv.
If you are a reporter or editor, and you want to cover new research, insist that the researcher, university, or journal, provide open access to its content — then link to it.
If you are a consumer of science or research reporting, and you want to evaluate news coverage, look for a clear link to an open access copy of the paper. If you don’t see one, flag it with the #NoPaperNoNews tag, and pressure the news/research collaborators to comply with this basic best practice.
This is not an extremist approach. I’m not saying we must require complete open access to all research (something I would like to see, of course). And this is not dissing the peer review process, which, although seriously flawed in its application, is basically a good idea. But peer review is nothing like a guarantee that research is good, and it’s even less a guarantee that research as translated through a news release and then a reporter and an editor is reliable and responsible. #NoPaperNoNews recognizes that when research enters the public arena through the news media, it may become important in unanticipated ways, and it may be subject to more irresponsible uses, misunderstandings, and exploitation. Providing direct access to the research product itself makes it possible for concerned people to get involved and speak up if something is going wrong. It also enhances the positive impact of the research reporting, which is great when the research is good.
Plenty of reporters, editors, researchers, and universities practice some version of this, but it’s inconsistent. For example, the American Sociological Association currently has a news release up about a paper in the American Sociological Review, by Paula England, Jonathan Bearak, Michelle Budig, and Melissa Hodges. And, as is now usually the case, that paper was selected by the ASR editors to be the freebie of the month, so it’s freely available. But the news release (which also only lists England as an author) doesn’t link to the paper. Some news reports link to the free copy but some don’t. ASA could easily add boilerplate language to their news releases, firmly suggesting that coverage link to the original paper, which is freely available.
Some publishers support this kind of approach, laying out free copies of breaking news research. But some don’t. In those cases, reporters and researchers can work together to make preprint versions available. In the social sciences, you can easily and immediately put a preprint on SocArXiv and add the link to the news report (to see which version you are free to post — pre-review, post-review, pre-edit, post-edit, etc. — consult your author agreement or look up the journal in the Sherpa/Romeo database.)
This practice is easy to enforce because it’s simple and technologically easy. When a New York Times reporter says, “I’d love to cover this research. Just tell me where I can link to the paper,” most researchers, universities, and publishers will jump to accommodate them. The only people who will want to block it are bad actors: people who don’t want their research scrutinized, reporters who don’t want to be double-checked, publishers who prioritize income over the public good.
Regular readers know I have objections to the framing of teen pregnancy, as a thing generally and as a problem specifically, separate from the rising age at childbearing generally (see also, or follow the teen births tag).
In this debate, one economic analysis of the effect of the popular MTV show 16 and Pregnant has played an outsized role. Melissa Kearney and Phillip Levine showed that was more decline in teen births in places where the show was popular, and attempted to establish that the relationship was causal — that the show makes people under age 20 want to have babies less. As Kearney put it in a video promoting the study: “the portrayal of teen pregnancy, and teen childbearing, is something they took as a cautionary tale.” (The paper also showed spikes in Twitter and Google activity related to birth control after the show aired.)
This was very big news for the marriage promotion people, because it was taken as evidence that cultural intervention “works” to affect family behavior — which really matters because so far they’ve spent $1 billion+ in welfare money on promoting marriage, with no effect (none), and they want more money.
The 16 and Pregnant paper has been cited to support statements such as:
Brad Wilcox: “Campaigns against smoking and teenage and unintended pregnancy have demonstrated that sustained efforts to change behavior can work.”
Washington Post: “By working with Hollywood to develop smart story lines on popular shows such as MTV’s ’16 and Pregnant’ and using innovative videos and social media to change norms, the [National Campaign to Prevent Teen and Unplanned Pregnancy] has helped teen pregnancy rates drop by nearly 60 percent since 1991.”
Boston Globe: “As evidence of his optimism, [Brad] Wilcox points to teen pregnancy, which has dropped by more than 50 percent since the early 1990s. ‘Most people assumed you couldn’t do much around something related to sex and pregnancy and parenthood,’ he said. ‘Then a consensus emerged across right and left, and that consensus was supported by public policy and social norms. . . . We were able to move the dial.’ A 2014 paper found that the popular MTV reality show ’16 and Pregnant’ alone was responsible for a 5.7 percent decline in teen pregnancy in the 18 months after its debut.”
I think a higher age at first birth is better for women overall, health permitting, but I don’t support that as a policy goal in the U.S. now, although I expect it would be an outcome of things I do support, like better health, education, and job opportunities for people of color and people who are poor.
Anyway, this is all just preamble to a new debate from a reanalysis and critique of the 16 and Pregnant paper. I haven’t worked through it enough to reach my own conclusions, and I’d like to hear from others who have. So I’m just sharing the links in sequence.
The initial paper, posted as a (non-peer reviewed) NBER Working Paper in 2014:
This paper explores how specific media images affect adolescent attitudes and outcomes. The specific context examined is the widely viewed MTV franchise, 16 and Pregnant, a series of reality TV shows including the Teen Mom sequels, which follow the lives of pregnant teenagers during the end of their pregnancy and early days of motherhood. We investigate whether the show influenced teens’ interest in contraceptive use or abortion, and whether it ultimately altered teen childbearing outcomes. We use data from Google Trends and Twitter to document changes in searches and tweets resulting from the show, Nielsen ratings data to capture geographic variation in viewership, and Vital Statistics birth data to measure changes in teen birth rates. We find that 16 and Pregnant led to more searches and tweets regarding birth control and abortion, and ultimately led to a 5.7 percent reduction in teen births in the 18 months following its introduction. This accounts for around one-third of the overall decline in teen births in the United States during that period.
A revised version, with the same title but slightly different results, was then published in the top-ranked American Economic Review, which is peer-reviewed:
This paper explores the impact of the introduction of the widely viewed MTV reality show 16 and Pregnant on teen childbearing. Our main analysis relates geographic variation in changes in teen childbearing rates to viewership of the show. We implement an instrumental variables (IV) strategy using local area MTV ratings data from a pre-period to predict local area 16 and Pregnant ratings. The results imply that this show led to a 4.3 percent reduction in teen births. An examination of Google Trends and Twitter data suggest that the show led to increased interest in contraceptive use and abortion.
Then last month David A. Jaeger, Theodore J. Joyce, and Robert Kaestner posted a critique on the Institute for the Study of Labor working paper series, which is not peer-reviewed:
We reassess recent and widely reported evidence that the MTV program 16 and Pregnant played a major role in reducing teen birth rates in the U.S. since it began broadcasting in 2009 (Kearney and Levine, American Economic Review 2015). We find Kearney and Levine’s identification strategy to be problematic. Through a series of placebo and other tests, we show that the exclusion restriction of their instrumental variables approach is not valid and find that the assumption of common trends in birth rates between low and high MTV-watching areas is not met. We also reassess Kearney and Levine’s evidence from social media and show that it is fragile and highly sensitive to the choice of included periods and to the use of weights. We conclude that Kearney and Levine’s results are uninformative about the effect of 16 and Pregnant on teen birth rates.
And now Kearney and Levine have posted their response on the same site:
This paper presents a response to Jaeger, Joyce, and Kaestner’s (JJK) recent critique (IZA Discussion Paper No. 10317) of our 2015 paper “Media Influences on Social Outcomes: The Impact of MTV’s 16 and Pregnant on Teen Childbearing.” In terms of replication, those authors are able to confirm every result in our paper. In terms of reassessment, the substance of their critique rests on the claim that the parallel trends assumption, necessary to attribute causation to our findings, is not satisfied. We present three main responses: (1) there is no evidence of a parallel trends assumption violation during our sample window of 2005 through 2010; (2) the finding of a false placebo test result during one particular earlier window of time does not invalidate the finding of a discrete break in trend at the time of the show’s introduction; (3) the results of our analysis are robust to virtually all alternative econometric specifications and sample windows that JJK consider. We conclude that this critique does not pose a serious threat to the interpretation of our 2015 findings. We maintain the position that our earlier paper is informative about the causal effect of 16 and Pregnant on teen birth rates.
So?
There are interesting methodological questions here. It’s hard to identify the effects of interventions that are swimming with the tide of change. In fact, the creation of the show, the show’s popularity, the campaign to end teen pregnancy, and the rising age at first birth may all be outcomes of the same general historical trend. So I’m not that invested in the answer to this question, though I am very interested.
There are also questions about the publication process, which I am very invested in. That’s why I work to promote a working paper culture among sociologists (through the SocArXiv project). The original paper was posted on a working paper site without peer review, but NBER is for economists who already are somebody, so that’s a kind of indirect screening. Then it was accepted in a top peer-reviewed journal (somewhat revised), but that was after it had received major attention and accolades, including a New York Times feature before the working paper was even released and a column devoted to it by Nicholas Kristof.
So is this a success story of working paper culture gone right — driving attention to good work faster, and then also drawing the benefits of peer review through the traditional publication process? (And now continuing with open debate on non-gated sites). Or is it a case of political hype driving attention inside and outside of the academy — the kind of thing that scares researchers and makes them want to retreat behind the slower, more process-laden research flow which they hope will protect them from exposure to embarrassment and protect the public from manipulation by the credulous news media. I think the process was okay even if we do conclude the paper wasn’t all it was made out to be. There were other reputational systems at work — faculty status, NBER membership, New York Times editors and sources — that may be as reliable as traditional peer review, which itself produces plenty of errors.
So, it’s an interesting situation — research methods, research implications, and research process.
Keep that gate. (Photo by Rob Nunn, https://flic.kr/p/4DbzCG)It’s hard to describe the day I got my first acceptance to American Sociological Review. There was no social media back then so I have no record of my reaction, but I remember it as the day — actually, the moment, as the conditional acceptance slipped out of the fax machine — that I learned I was getting tenure, that I would have my dream job for the rest of my life, with a personal income in the top 10 percent of the country for a 9-month annual commitment. At that moment I was not inclined to dwell on the flaws in our publishing system, its arbitrary qualities, or the extreme status hierarchy it helps to construct.
In a recent year ASR considered more than 700 submitted articles and rejected 90% or more of them (depending on how you count). Although many people dispute the rationality of this distinction, publishing in our association’s flagship journal remains the most universally agreed-upon indicator of scholarship quality. And it is rare. I randomly sampled 50 full-time sociology faculty listed in the 2016 ASA Guide to Graduate Departments of Sociology (working in the U.S. and Canada), and found that 9, or 18%, had ever published a research article in ASR.
Not only is it rare, but publication in ASR is highly concentrated in high-status departments (and individuals). While many departments have no faculty that have published in ASR (I didn’t count these, but there are a lot), some departments are brimming with them. In my own, second-tier department, I count 16 out of 27 faculty with publications in ASR (59%), while at a top-tier, article-oriented department such as the University of North Carolina at Chapel Hill (where I used to work), 19 of the 25 regular faculty, or 76%, have published in ASR (many of them multiple times).
Without diminishing my own accomplishment (or that of my co-authors), or the privilege that got me here, I should be clear that I don’t think publication in high-status journals is a good way to identify and reward scholarly accomplishment and productivity. The reviews and publication decisions are too uneven (although obviously not completely uncorrelated with quality), and the limit on articles published is completely arbitrary in an era in which the print journal and its cost-determined page-limit is simply ridiculous.
We have a system that is hierarchical, exclusive, and often arbitrary — and the rewards it doles out are both large and highly concentrated.
I say all this to put in perspective the grief I have gotten for publicly criticizing an article published in ASR. In that post, I specifically did not invoke ethical violations or speculate on the motivations or non-public behavior of the authors, about whom I know nothing. I commented on the flaws in the product, not the process. And yet a number of academic critics responded vociferously to what they perceive as the threats this commentary posed to the academic careers and integrity of the authors whose work I discussed. Anonymous critics called my post “obnoxious, childish, time wasting, self promoting,” and urged sociologists to “shun” me. I have been accused of embarking on a “vigilante mission.” In private, a Jewish correspondent referred me to the injunction in Leviticus against malicious gossip in an implicit critique of my Jewish ethics.*
In the 2,500-word response I published on my site — immediately and unedited — I was accused of lacking “basic decency” for not giving the authors a chance to prepare a response before I posted the criticism on my blog. The “commonly accepted way” when “one scholar wishes to criticize the published work of another,” I was told, is to go through a process of submitting a “comment” to the journal that published the original work, which “solicits a response from the authors who are being criticized,” and it’s all published together, generally years later. (Never mind that journals have no obligation or particular inclination to publish such debates, as I have reported on previously, when ASR declined for reasons of “space” to publish a comment pointing out errors that were not disputed by the editors.)
This desire to maintain gatekeepers to police and moderate our discussion of public work is not only quaint, it is corrosive. Despite pointing out uncomfortable facts (which my rabbinical correspondent referred to as the “sin of true speech for wrongful purpose”), my criticism was polite, reasoned, with documentation — and within the bounds of what would be considered highly civil discourse in any arena other than academia, apparently. Why are the people whose intellectual work is most protected most afraid of intellectual criticism?
In Christian Smith’s book, The Sacred Project of American Sociology (reviewed here), which was terrible, he complains explicitly about the decline of academic civilization’s gatekeepers:
The Internet has created a whole new means by which the traditional double-blind peer-review system may be and already is in some ways, I believe, being undermined. I am referring here to the spate of new sociology blogs that have sprung up in recent years in which handfuls of sociologists publicly comment upon and often criticize published works in the discipline. The commentary published on these blogs operates outside of the gatekeeping systems of traditional peer review. All it takes to make that happen is for one or more scholars who want to amplify their opinions into the blogosphere to set up their own blogs and start writing.
Note he is complaining about people criticizing published work, yet believes such criticism undermines the blind peer-review system. This fear is not rational. The terror over public discussion and debate — perhaps especially among the high-status sociologists who happen to also be the current gatekeepers — probably goes a long way toward explaining our discipline’s pitiful response to the crisis of academic publishing. According to my (paywalled) edition of the Oxford English Dictionary, the definition of “publish” is “to make public.” And yet to hear these protests you would think the whisper of a public comment poses an existential threat to the very people who have built their entire profession around publishing (though, to be consistent, it’s mostly hidden from the public behind paywalls).
This same fear leads many academics to insist on anonymity even in normal civil debates over research and our profession. Of course there are risks, as there tend to be when people make important decisions about things that matter. But at some point, the fear of repression for expressing our views (which is legitimate in some rare circumstances) starts looking more like avoidance of the inconvenience or discomfort of having to stand behind our words. If academics are really going to lose their jobs for getting caught saying, “Hey, I think you were too harsh on that paper,” then we are definitely having the wrong argument.
“After all,” wrote Eran Shor, “this is not merely a matter of academic disagreements; people’s careers and reputations are at stake.” Of course, everyone wants to protect their reputation — and everyone’s reputation is always at stake. But let’s keep this in perspective. For those of us at or near the top of this prestige hierarchy — tenured faculty at research universities — damage to our reputations generally poses a threat only within a very narrow bound of extreme privilege. If my reputation were seriously damaged, I would certainly lose some of the perks of my job. But the penalty would also include a decline in students to advise, committees to serve on, and journals to edit — and no change in that lifetime job security with a top-10% salary for a 9-month commitment. Of course, for those of us whose research really is that important, anything that harms our ability to work in exactly the way that we want to has costs that simply cannot be measured. I wouldn’t know about that.
But if we want the high privilege of an academic career — and if we want a discipline that can survive under scrutiny from an increasingly impatient public and deepening market penetration — we’re going to have to be willing to defend it.
* I think if random Muslims have to denounce ISIS then Jews who cite Leviticus on morals should have to explain whether — despite the obvious ethical merit to some of those commands — they also support the killing of animals just because they have been raped by humans.
I have written a few times about problems with peer review and publishing.* My own experience subsequently led me to the problem of coercive self-citation, defined in one study as “a request from an editor to add more citations from the editor’s journal for reasons that were not based on content.” I asked readers to send me documentation of their experiences so we could air them out. This is the result.
Introduction
First let me mention a new editorial in the journal Research Policy about the practices editors use to inflate the Journal Impact Factors, a measure of citations that many people use to compare journal quality or prestige. One of those practices is coercive self-citation. The author of that editorial, Ben Martin, cites approvingly a statement signed by a group of management and organizational studies editors:
I will refrain from encouraging authors to cite my journal, or those of my colleagues, unless the papers suggested are pertinent to specific issues raised within the context of the review. In other words, it should never be a requirement to cite papers from a particular journal unless the work is directly relevant and germane to the scientific conversation of the paper itself. I acknowledge that any blanket request to cite a particular journal, as well as the suggestion of citations without a clear explanation of how the additions address a specific gap in the paper, is coercive and unethical.
So that’s the gist of the issue. However, it’s not that easy to define coercive self-citation. In fact, we’re not doing a very good job of policing journal ethics in general, basically relying on weak enforcement of informal community standards. I’m not an expert on norms, but it seems to me that when you have strong material interests — big corporations using journals to print money at will, people desperate for academic promotions and job security, etc. — and little public scrutiny, it’s hard to regulate unethical behavior informally through norms.
The clearest cases involve asking for self-citations (a) before final acceptance, for citations (b) within the last two years and (c) without substantive reason. But there is a lot short of that to object to as well. Martin suggests that, to answer whether a practice is ethical, we need to ask: “Would I, as editor, feel embarrassed if my activities came to light and would I therefore object if I was publicly named?” (Or, as my friend Matt Huffman used to say when the used-textbook buyers came around offering us cash for books we hadn’t paid for: how would it look in grainy hidden-camera footage?) I think that journal practices, which are generally very opaque, should be exposed to public view so that unethical or questionable practices can be held up to community standards.
Reports and responses
I received reports from about a dozen journals, but a few could not be verified or were too vague. These 10 were included under very broad criteria — I know that not everyone will agree that these practices are unethical, and I’m unsure where to draw the line myself. In each case below I asked the current editor if they would care to respond to the complaint, doing my best to give the editor enough information without exposing the identity of the informant.
Here in no particular order are the excerpts of correspondence from editors, with responses from the editors to me, if any. Some details, including dates, may have been changed to protect informants. I am grateful to the informants who wrote, and I urge anyone who knows, or thinks they know, who the informants are not to punish them for speaking up.
Journal of Social and Personal Relationships (2014-2015 period)
Congratulations on your manuscript “X” having been accepted for publication in Journal of Social and Personal Relationships. … your manuscript is now “in press” … The purpose of this message is to inform you of the production process and to clarify your role in the process …
IMPORTANT NOTICE:
As you update your manuscript:
1. CITATIONS – Remember to look for relevant and recent JSPR articles to cite. As you are probably aware, the ‘quality’ of a journal is increasingly defined by the “impact factor” reported in the Journal Citation Reports (from the Web of Science). The impact factor represents a ratio of the number of times that JSPR articles are cited divided by the number of JSPR articles published. Therefore, the 20XX ratings will focus (in part) on the number of times that JSPR articles published in 20XX and 20XX are cited during the 20XX publication year. So citing recent JSPR articles from 20XX and 20XX will improve our ranking on this particular ‘measure’ of quality (and, consequently, influence how others view the journal. Of course only cite those articles relevant to the point. You can find tables of contents for the past two years at…
Response from editor Geoff MacDonald:
Thanks for your email, and for bringing that to my attention. I agree that encouraging self-citation is inappropriate and I have just taken steps to make sure it won’t happen at JSPR again.
Sex Roles (2011-2013 period)
In addition to my own report, already posted, I received an identical report from another informant. The editor, Irene Frieze, wrote: “If possible, either in this section or later in the Introduction, note how your work builds on other studies published in our journal.”
Response from incoming editor Janice D. Yoder:
As outgoing editor of Psychology of Women Quarterly and as incoming editor of Sex Roles, I have not, and would not, as policy require that authors cite papers published in the journal to which they are submitting.
I have recommended, and likely will continue to recommend, papers to authors that I think may be relevant to their work, but without any requirement to cite those papers. I try to be clear that it is in this spirit of building on existing scholarship that I make these recommendations and to make the decision of whether or not to cite them up to the author. As an editor who has decision-making power, I know that my recommendations can be interpreted as requirements (or a wise path to follow for authors eager to publish) but I can say that I have not further pressured an author whose revision fails to cite a paper I recommended.
I also have referred to authors’ reference lists as a further indication that a paper’s content is not appropriate for the journal I edit. Although never the sole indicator and never based only on citations to the specific journal I edit, if a paper is framed without any reference to the existing literature across journals in the field then it is a sign to me that the authors should seek a different venue.
I value the concerns that have been raised here, and I certainly would be open to ideas to better guide my own practices.
European Sociological Review (2013)
In a decision letter notifying the author of a minor revise-and-resubmit, the editor wrote that the author had left out of the references some recent, unspecified, publications in ESR and elsewhere (also unspecified) and suggested the author update the references.
Response from editor Melinda Mills:
I welcome the debate about academic publishing in general, scrutiny of impact factors and specifically of editorial practices. Given the importance of publishing in our profession, I find it surprising how little is actually known about the ‘black box’ processes within academic journals and I applaud the push for more transparency and scrutiny in general about the review and publication process. Norms and practices in academic journals appear to be rapidly changing at the moment, with journals at the forefront of innovation taking radically different positions on editorial practices. The European Sociological Review (ESR) engages in rigorous peer review and most authors agree that it strengthens their work. But there are also new emerging models such as Sociological Science that give greater discretion to editors and focus on rapid publication. I agree with Cohen that this debate is necessary and would be beneficial to the field as a whole.
It is not a secret that the review and revision process can be a long (and winding) road, both at ESR and most sociology journals. If we go through the average timeline, it generally takes around 90 days for the first decision, followed by authors often taking up to six months to resubmit the revision. This is then often followed by a second (and sometimes third) round of reviews and revision, which in the end leaves us at ten to twelve months from original submission to acceptance. My own experience as an academic publishing on other journals is that it can regularly exceed one year. During the year under peer review and revisions, relevant articles have often been published. Surprisingly, few authors actually update their references or take into account new literature that was published after the initial submission. Perhaps this is understandable, since authors have no incentive to implement any changes that are not directly requested by reviewers.
When there has been a particularly protracted peer review process, I sometimes remind authors to update their literature review and take into account more recent publications, not only in ESR but also elsewhere. I believe that this benefits both authors, by giving them greater flexibility in revising their manuscripts, and readers, by providing them with more up-to-date articles. To be clear, it is certainly not the policy of the journal to coerce authors to self-cite ESR or any other outlets. It is vital to note that we have never rejected an article where the authors have not taken the advice or opportunity to update their references and this is not a formal policy of ESR or its Editors. If authors feel that nothing has happened in their field of research in the last year that is their own prerogative. As authors will note, with a good justification they can – and often do – refuse to make certain substantive revisions, which is a core fundament of academic freedom.
Perhaps a more crucial part of this debate is the use and prominence of journal impact factors themselves both within our discipline and how we compare to other disciplines. In many countries there is a move to use these metrics to distribute financing to Universities, increasing the stakes of these metrics. It is important to have some sort of metric gauge of the quality and impact of our publications and discipline. But we also know that different bibliometric tools have the tendency to produce different answers and that sociology fairs relatively worse in comparison to other disciplines. Conversely, leaving evaluation of research largely weighted by peer review can produce even more skewed interpretations if the peer evaluators do not represent an international view of the discipline. Metrics and internationally recognized peer reviewers would seem the most sensible mix.
Work and Occupations (2010-2011 period)
“I would like to accept your paper for publication on the condition that you address successfully reviewer X’s comments and the following:
2. The bibliography needs to be updated somewhat … . Consider citing, however critically, the following Work and Occupations articles on the italicized themes:
[concept: four W&O papers, three from the previous two years]
[concept: two W&O papers from the previous two years]
The current editor, Dan Cornfield, thanked me and chose not to respond for publication.
Sociological Forum (2014-2015 period)
I am pleased to inform you that your article … is going to press. …
In recent years, we published an article that is relevant to this essay and I would like to cite it here. I have worked it in as follows: [excerpt]
Most authors find this a helpful step as it links their work into an ongoing discourse, and thus, raises the visibility of their article.
Response from editor Karen Cerulo:
I have been editing Sociological Forum since 2007. I have processed close to 2500 submissions and have published close to 400 articles. During that time, I have never insisted that an author cite articles from our journal. However, during the production process–when an article has been accepted and I am preparing the manuscript for the publisher–I do sometimes point out to authors Sociological Forum pieces directly relevant to their article. I send authors the full citation along with a suggestion as to where the citation be discussed or noted. I also suggest changes to key words and article abstracts, My editorial board is fully aware of this strategy. We have discussed it at many of our editorial board meetings and I have received full support for this approach. I can say, unequivocally, that I do not insist that citations be added. And since the manuscripts are already accepted, there is no coercion involved. I think it is important that you note that on any blog post related to Sociological Forum
I cannot tell you how often an author sends me a cover letter with their submission telling me that Sociological Forum is the perfect journal for their research because of related ongoing dialogues in our pages. Yet, in many of these cases, the authors fail to reference the relevant dialogues via citations. Perhaps editors are most familiar with the debates and streams of thought currently unfolding in a journal. Thus, I believe it is my job as editor and my duty to both authors and the journal to suggest that authors consider making appropriate connections.
Unnamed journal (2014)
An article was desk-rejected — that is, rejected without being sent out for peer review — with only this explanation: “In light of the appropriateness of your manuscript for our journal, your manuscript has been denied publication in X.” When the author asked for more information, a journal staff member responded with possible reasons, including that the paper did not include any references to the articles in that journal. In my view the article was clearly within the subject area of the journal. I didn’t name the journal here because this wasn’t an official editor’s decision letter and the correspondence only suggested that might be the reason for the rejction.
Sociological Quarterly (2014-2015 period)
In a revise and resubmit decision letter:
Finally, as a favor to us, please take a few moments to review back issues of TSQ to make sure that you have cited any relevant previously published work from our journal. Since our ISI Impact Factor is determined by citations, we would like to make sure papers under consideration by the journal are referring to scholarship we have previously supported.
The current editors, Lisa Waldner and Betty Dobratz, have not yet responded.
Canadian Review of Sociology (2014-2015 period)
In a letter communicating acceptance conditional on minor changes, the editor asked the author to consider citing “additional Canadian Review of Sociology articles” to “help with the journal’s visibility.”
Response from current editor Rima Wilkes:
In the case you cite, the author got a fair review and received editorial comments at the final stages of correction. The request to add a few citations to the journal was not “coercive” because in no instance was it a condition of the paper either being reviewed or published.
Many authors are aware of, and make some attempt to cite the journal to which they are submitting prior to submission and specifically target those journals and to contribute to academic debate in them.
Major publications in the discipline, such as ASR, or academia more generally, such as Science, almost never publish articles that have no reference to debates in them.
Bigger journals are in the fortunate position of having authors submit articles that engage with debates in their own journal. Interestingly, the auto-citation patterns in those journals are seen as “natural” rather than “coerced”. Smaller journals are more likely to get submissions with no citations to that journal and this is the case for a large share of the articles that we receive.
Journals exist within a larger institutional structure that has certain demands. Perhaps the author who complained to you might want to reflect on what it says about their article and its potential future if they and other authors like them do not engage with their own work.
Social Science Research (2015)
At the end of a revise-and-resubmit memo, under “Comment from the Editor,” the author was asked to include “relevant citations from Social Science Research,” with none specified.
The current editor, Stephanie Moller, has not yet responded.
City & Community (2013)
In an acceptance letter, the author was asked to approve several changes made to the manuscript. One of the changes, made to make the paper more conversant with the “relevant literature,” added a sentence with several references, one or more of which were to City & Community papers not previously included.
One of the current co-editors, Sudhir Venkatesh, declined to comment because the correspondence occurred before the current editorial teams’ tenure began.
Discussion
The Journal Impact Factor (JIF) is an especially dysfunctional part of our status-obsessed scholarly communication system. Self-citation is only one issue, but it’s a substantial one. I looked at 116 journals classified as sociology in 2014 by Web of Science (which produces the JIF), excluding some misplaced and non-English journals. WoS helpfully also offers a list excluding self-citations, but normal JIF rankings do not make this exclusion. (I put the list here.) On average removing self-citations reduces the JIF by 14%. But there is a lot of variation. One would expect specialty journals to have high self-citation counts because the work they publish is closely related. Thus Armed Forces and Society has a 31% self-citation rate, as does Work & Occupations (25%). But others, like Gender & Society (13%) and Journal of Marriage and Family (15%) are not high. On the other hand, you would expect high-visibility journals to have high self-citation rates, if they publish better, more important work; but on this list the correlation between JIF and self-citation rate is -.25. Here is that relationship for the top 50 journals by JIF, with the top four by self-citation labeled (the three top-JIF journals at bottom-right are American Journal of Sociology, Annual Review of Sociology, and American Sociological Review).
The top four self-citers are low-JIF journals. Two of them are mentioned above, but I have no idea what role self-citation encouragement plays in that. There are other weird distortions in JIFs that may or may not be intentional. Consider the June 2015 issue of Sociological Forum, which includes a special section, “Commemorating the Fiftieth Anniversary of the Civil Rights Laws.” That issue, just a few months old, as of yesterday includes the 9 most-cited articles that the journal published in the last two years. In fact, these 9 pieces have all been cited 9 times, all by each other — and each article currently has the designation of “Highly Cited Paper” from Web of Science (with a little trophy icon). The December 2014 issue of the same journal also gave itself an immediate 24 self-citations for a special “forum” feature. I am not suggesting the journal runs these forum discussion features to pump up its JIF, and I have nothing bad to say about their content — what’s wrong with a symposium-style feature in which the authors respond to each other’s work? But these cases illustrate what’s wrong with using citation counts to rank journals. As Martin’s piece explains, the JIF is highly susceptible to manipulation beyond self-citation promotion, for example by tinkering with the pre-publication queue of online articles, publishing editorial review essays, and of course outright fraud.
Anyway, my opinion is that journal editors should never add or request additional citations without clearly stated substantive reasons related to the content of the research and unrelated to the journal in which they are published. I realize that reasonable people disagree about this — and I encourage readers to respond in the comments below. I also hope that any editor would be willing to publicly stand by their practices, and I urge editors and journal management to let authors and readers see what they’re doing as much as possible.
However, I also think our whole journal system is pretty irreparably broken, so I put limited stock in the idea of improving its operation. My preference is to (1) fire the commercial publishers, (2) make research publication open-access with a very low bar for publication; and (3) create an organized system of post-publication review to evaluate research quality, with (4) republishing or labeling by professional associations to promote what’s most important.
In light of the news on social science fraud, I thought it was a good time to report on an experiment I did. I realize my results are startling, and I welcome the bright light of scrutiny that such findings might now attract.
The following information is fake.
An employee training program in a major city promises basic job skills and as well as job search assistance for people with a high school degree and no further education, ages 23-52 in 2012. Due to an unusual staffing practice, new applications were for a period in 2012 allocated at random to one of two caseworkers. One provided the basic services promised but nothing extra. The other embellished his services with extensive coaching on such “soft skills” as “mainstream” speech patterns, appropriate dress for the workplace, and a hard work ethic, among other elements. The program surveyed the participants in 2014 to see what their earnings were in the previous 12 months. The data provided to me does not include any information on response rates, or any information about those who did not respond. And it only includes participants who were employed at least part-time in 2014. Fortunately, the program also recorded which staff member each participant was assigned to.
Since this provides such an excellent opportunity for studying the effects of soft skills training, I think it’s worth publishing despite these obvious weaknesses. To help with the data collection and analysis, I got a grant from Big Neoliberal, a non-partisan foundation.
The data includes 1040 participants, 500 of whom had the bare-bones service and 540 of whom had the soft-skills add-on, which I refer to as the “treatment.” These are the descriptive statistics:
As you can see, the treatment group had higher earnings in 2014. The difference in logged annual earnings between the two groups is significant at p
As you can see in Model 1, the Black workers in 2014 earned significantly less than the White workers. This gap of .15 logged earnings points, or about 15%, is consistent with previous research on the race wage gap among high school graduates. Model 2 shows that the treatment training apparently was effective, raising earnings about 11%. However, The interactions in Model 3 confirm that the benefits of the treatment were concentrated among the Black workers. The non-Black workers did not receive a significant benefit, and the treatment effect among Black workers basically wiped out the race gap.
The effects are illustrated, with predicted probabilities, in this figure:
What would you do if you saw this in a paper or at a conference? Would you suspect it was fake? Why or why not?
I confess I never seriously thought of faking a research study before. In my day coming up in sociology, people didn’t share code and datasets much (it was never compulsory). I always figured if someone was faking they were just changing the numbers on their tables to look better. I assumed this happens to some unknown, and unknowable, extent.
So when I heard about the Lacour & Green scandal, I thought whoever did it was tremendously clever. But when I looked into it more, I thought it was not such rocket science. So I gave it a try.
Details
I downloaded a sample of adults 25-54 from the 2014 ACS via IPUMS, with annual earnings, education, age, sex, race and Hispanic origin. I set the sample parameters to meet the conditions above, and then I applied the treatment, like this:
First, I randomly selected the treatment group:
gen temp = runiform()
gen treatment=0
replace treatment = 1 if temp >= .5
drop temp
Then I generated the basic effect, and the Black interaction effect:
gen effect = rnormal(.08,.05)
gen beffect = rnormal(.15,.05)
Starting with the logged wage variable, lnwage, I added the basic effect to all the treated subjects:
replace newlnwage = lnwage+effect if treatment==1
Then added the Black interaction effect to the treated Black subjects, and subtracted it from the non-treated ones.
replace newlnwage = newlnwage+beffect if (treatment==1 & black==1)
replace newlnwage = newlnwage-beffect if (treatment==0 & black==1)
This isn’t ideal, but when I just added the effect I didn’t have a significant Black deficit in the baseline model, so that seemed fishy.
That’s it. I spent about 20 minutes trying different parameters for the fake effects, trying to get them to seem reasonable. The whole thing took about an hour (not counting the write-up).
Would I get caught for this? What are we going to do about this?
BUSTED UPDATE:
In the comments, ssgrad notices that if you exponentiate (unlog) the incomes, you get a funny list — some are binned at whole numbers, as you would expect from a survey of incomes, and some are random-looking and go out to multiple decimal places. For example, one person reports an even $25,000, and another supposedly reports $25251.37. This wouldn’t show up in the descriptive statistics, but is kind of obvious in a list. Here is a list of people with incomes between $20000 and $26000, broken down by race and treatment status. I rounded to whole numbers because even without the decimal points you can see that the only people who report normal incomes are non-Blacks in the non-treatment group. Busted!
So, that only took a day — with a crowd-sourced team of thousands of social scientists poring over the replication file. Faith in the system restored?
