What the editors of 6000 journals tell us about gender, international diversity, open access, and research transparency

Micah Altman and I have written a paper using the new Open Editors dataset from Andreas Pacher, Tamara Heck, and Kerstin Schoch. They scraped up data on almost half a million editors (editors in chief, editors, editorial board member) at more than 6000 journals from 17 publishers (most of the big ones; they’ve since added some more). Micah and I genderized them (fuzzily), geolocated them in countries, and then coded the journals as either open access or not (using the Directory of Open Access Journals), and according to whether they practice transparency in research (using the Transparency and Openness Promotion signatories). After just basic curiosity about diversity, we wondered whether those that practice open access and research transparency have better gender and international diversity.

The results show overwhelming US and European dominance, not surprisingly. And male dominance, which is more extreme among editors in chief, across all disciplines. Open access journals are a little less gender diverse, and transparency-practicing journals a little more internationally diverse, but those relationships aren’t strong. There are other differences by discipline. A network analysis shows not much overlap between journals, outside of a few giant clusters (which might indicate questionable practices) although it’s hard to say for sure — journals should really use ORCIDs for their editors. Kudos to Micah for doing the heavy lifting on the coding, which involved multiple levels of cleaning and recoding (and for making the R markdown file for the whole thing available).

Lots of details in the draft, here. Feedback welcome!

Here are the editors, by country:

Citizen Scholar: new book under contract

PN Cohen photo

My new book, Citizen Scholar, is under contract with Columbia University Press (thanks to the support of editor, and Editorial Director, Eric Schwartz).

Some of the writing I’ve been doing here is part of the book’s development, including the piece on “policy implications,” essays on transparency and accountability in research, as well as talks and materials about preprints, open science and the pandemic, politics and science, and others. It’s time for a book (and also more talks, if you’d like to invite me!). I will post essays and excerpts as I go, here, and I welcome your critiques, suggestions, and ideas. The first post describes my ambitions, and plan, for the book.

I love Family Inequality and everyone here but it seemed awkward to repeatedly post stuff for the new book under this heading. So I set up a blog style page, and I’ll post links here, too (and I’ll figure out you can subscribe, for those who want their blog posts via email).

Wish me luck!

Basic self-promotion

Five years ago today I wrote a post called “Basic self promotion” on here. There has been a lot of work and advice on this subject in the intervening years (including books, some of which I reviewed here). So this is not as necessary as it was then. But it holds up pretty well, with some refreshing. So here is a lightly revised version. As always, happy to have your feedback and suggestions in the comments — including other things to read.


48943567406_2ccfa2b882_k
Present yourself. PN Cohen photo: https://flic.kr/p/2hyYzqs.

If you won’t make the effort to promote your research, how can you expect others to?

These are some basic thoughts for academics promoting their research. You don’t have to be a full-time self-promoter to improve your reach and impact, but the options are daunting and I often hear people say they don’t have time to do things like run a Twitter account or write for blogs and other publications. Even a relatively small effort, if well directed, can help a lot. Don’t let the perfect be the enemy of the good. It’s fine to do some things pretty well even if you can’t do everything to your ideal standard.

It’s all about making your research better — better quality, better impact. You want more people to read and appreciate your work, not just because you want fame and fortune, but because that’s what the work is for. I welcome your comments and suggestions below.

Present yourself

Make a decent personal website and keep it up to date with information about your research, including links to freely available copies of your publications (see below). It doesn’t have to be fancy. I’m often surprised at how many people are sitting behind years-old websites. (I recently engaged Brigid Barrett, who specializes in academics’ websites, to redesign mine.)

Very often people who come across your research somewhere else will want to know more about you before they share, report on, or even cite it. Your website gives your work more credibility. Has this person published other work in this area? Taught related courses? Gotten grants? These are things people look for. It’s not vain or obnoxious to present this information, it’s your job. I recommend a good quality photo, updated at least every five years.

Make your work available

Let people read the actual research. For work not yet “published” in journals, post drafts when they are ready for readers (a good time is when you are ready to send it to a conference or journal – or earlier if you are comfortable with sharing it). This helps you establish precedence (planting your flag), and allows it to generate feedback and attract readers. It’s best to use a disciplinary archive such as SocArXiv (which, as the director, I highly recommend) or your university repository, or both. This will improve how they show up in web searches (including Google Scholar) indexed for things like citation or grant analysis, and archived. You can also get a digital object identifier (DOI), which allows them to enter the great stream of research metadata. (See the SocArXiv FAQ for more answers.)

When you do publish in journals, prefer open-access journals because it’s the right thing to do and more people can read your work there. If a paper is paywalled, share a preprint or postprint version. On your website or social media feeds, please don’t just link to the pay-walled versions of your papers, that’s the click of death for someone just browsing around, plus it’s elitist and antisocial. You can almost always put up a preprint without violating your agreements (ideally you wouldn’t publish anywhere that won’t let you do this). To see the policies of different journals regarding self-archiving, check out the simple database at SHERPA/RoMEO, or, of course, the agreement you signed with the journal.

I oppose private sites like Academia.edu, ResearchGate, or SSRN. These are just private companies making a profit from doing what your university and its library, and nonprofits like SocArXiv are already doing for the public good. Your paper will not be discovered more if it is on one of these sites.

I’m not an open access purist, believe it or not. (If you got public money to develop a cure for cancer, that’s different, then I am a purist.) Not everything we write has to be open access (books, for example), but the more it is the better, especially original research. This is partly an equity issue for readers, and partly to establish trust and accountability in all of our work. Readers should be able to see our work product – our instruments, our code, our data – to evaluate its veracity (and to benefit their own work). And for the vast majority of readers who don’t want to get into those materials, the fact they are there increases our collective accountability and trustworthiness. I recommend using the Open Science Framework, a free, nonprofit platform for research sharing and collaboration.

Actively share your work

In the old days we used to order paper reprints of papers we published and literally mail them to the famous and important people we hoped would read and cite them. Nowadays you can email them a PDF. Sending a short note that says, “I thought you might be interested in this paper I wrote” is normal, reasonable, and may be considered flattering. (As long as you don’t follow up with repeated emails asking if they’ve read it yet.)

Social media

If you’re reading this, you probably use at least basic social media. If not, I recommend it. This does not require a massive time commitment and doesn’t mean you have to spend all day doomscrolling — you can always ignore them. Setting up a public profile on Twitter or a page on Facebook gives people who do use them all the time a way to link to you and share your profile. If someone wants to show their friends one of my papers on Twitter, this doesn’t require any effort on my part. They tweet, “Look at this awesome new paper @familyunequal wrote!” (I have some vague memory of this happening with my papers.) When people click on the link they go to my profile, which tells them who I am and links to my website.

Of course, a more active social media presence does help draw people into your work, which leads to exchanging information and perspectives, getting and giving feedback, supporting and learning from others, and so on. Ideally. But even low-level attention will help: posting or tweeting links to new papers, conference presentations, other writing, etc. No need to get into snarky chitchat and following hundreds of people if you don’t want to. To see how sociologists are using Twitter, you can visit the list I maintain, which has more than 1600 sociologists. This is useful for comparing profile and feed styles.

Other writing

People who write popular books go on book tours to promote them. People who write minor articles in sociology journals might send out some tweets, or share them with their friends on Facebook. In between are lots of other places you can write something to help people find and learn about your work. I still recommend a blog format, easily associated with your website, but this can be done different ways. As with publications themselves, there are public and private options, open and paywalled. Open is better, but some opportunities are too good to pass up – and it’s OK to support publications that charge subscription or access fees, if they deserve it.

There are also good organizations now that help people get their work out. In my area, for example, the Council on Contemporary Families is great (I’m a former board member), producing research briefs related to new publications, and helping to bring them to the attention of journalists and editors. Others work with the Scholars Strategy Network, which helps people place Op-Eds, or the university-affiliated site The Society Pages, or others. In addition, there are blogs run by sections of the academic associations, and various group blogs. And there is Contexts (which I used to co-edit), the general interest magazine of ASA, where they would love to hear proposals for how you can bring your research out into the open (for the magazine or their blog).


For more on the system we use to get our work evaluated, published, transmitted, and archived, I’ve written this report: Scholarly Communication in Sociology: An introduction to scholarly communication for sociology, intended to help sociologists in their careers, while advancing an inclusive, open, equitable, and sustainable scholarly knowledge ecosystem.

Data analysis shows Journal Impact Factors in sociology are pretty worthless

The impact of Impact Factors

Some of this first section is lifted from my blockbuster report, Scholarly Communication in Sociology, where you can also find the references.

When a piece of scholarship is first published it’s not possible to gauge its importance immediately unless you are already familiar with its specific research field. One of the functions of journals is to alert potential readers to good new research, and the placement of articles in prestigious journals is a key indicator.

Since at least 1927, librarians have been using the number of citations to the articles in a journal as a way to decide whether to subscribe to that journal. More recently, bibliographers introduced a standard method for comparing journals, known as the journal impact factor (JIF). This requires data for three years, and is calculated as the number of citations in the third year to articles published over the two prior years, divided by the total number of articles published in those two years.

For example, in American Sociological Review there were 86 articles published in the years 2017-18, and those articles were cited 548 times in 2019 by journals indexed in Web of Science, so the JIF of ASR is 548/86 = 6.37. This allows for a comparison of impact across journals. Thus, the comparable calculation for Social Science Research is 531/271 = 1.96, and it’s clear that ASR is a more widely-cited journal. However, comparisons of journals in different fields using JIFs is less helpful. For example, the JIF for the top medical journal, New England Journal of Medicine, is currently 75, because there are many more medical journals publishing and citing more articles at higher rates, and more quickly than do sociology journals. (Or maybe NEJM is just that much more important.)

In addition to complications in making comparisons, there are problems with JIFs (besides the obvious limitation that citations are only one possible evaluation metric). They depend on what journals and articles are in the database being used. And they mostly measure short-term impact. Most important for my purposes here, however, is that they are often misused to judge the importance of articles rather than journals. That is, if you are a librarian deciding what journal to subscribe to, JIF is a useful way of knowing which journals your users might want to access. But if you are evaluating a scholar’s research, knowing that they published in a high-JIF journal does not mean that their article will turn out to be important. It is especially wrong to look at an article that’s old enough to have citations you could count (or not) and judge its quality by the journal it’s published in — but people do that all the time.

To illustrate this, I gathered citation data from the almost 2,500 articles published in 2016-2019 in 15 sociology journals from the Web of Science category list.* In JIF these rank from #2 (American Sociological Review, 6.37) to #46 (Social Forces, 1.95). I chose these to represent a range of impact factors, and because they are either generalist journals (e.g., ASR, Sociological Science, Social Forces) or sociology-focused enough that almost any article they publish could have been published in a generalist journal as well. Here is a figure showing the distribution of citations to those articles as of December 2020, by journal, ordered from higher to lower JIF.

After ASR, Sociology of Education, and American Journal of Sociology, it’s hard to see much of a slope here. Outliers might be playing a big role (for example that very popular article in Sociology of Religion, “Make America Christian Again: Christian Nationalism and Voting for Donald Trump in the 2016 Presidential Election,” by Whitehead, Perry, and Baker in 2018). But there’s a more subtle problem, which is the timing of the measures. My collection of articles is 2016-2019. The JIFs I’m using are from 2019, based on citations to 2017-2018 articles. These journals bounce around; for example, Sociology of Religion jumped from 1.6 to 2.6 in 2019. (I address that issue in the supplemental analysis below.) So what is a lazy promotion and tenure committee, which is probably working off a mental reputation map at least a dozen years old, to do?

You can already tell where I’m going with this: In these sociology journals, there is so much noise in citation rates within the journals, compared to any stable difference between them, that outside the very top the journal ranking won’t much help you predict how much a given paper will be cited. If you assume a paper published in AJS will be more important than one published in Social Forces, you might be right, but if the odds that you’re wrong are too high, you just shouldn’t assume anything. Let’s look closer.

Sociology failure rates

I recently read this cool paper (also paywalled in the Journal of Informetrics) that estimates the odds of this “failure probability,” the odds that your guess about which paper will be more impactful based on the journal title turns out to be wrong. When JIFs are similar, the odds of an error are very high, like a coin flip. “In two journals whose JIFs are ten-fold different, the failure probability is low,” Brito and Rodríguez-Navarro conclude. “However, in most cases when two papers are compared, the JIFs of the journals are not so different. Then, the failure probability can be close to 0.5, which is equivalent to evaluating by coin flipping.”

Their formulas look pretty complicated to me, so for my sociology approach I just did it by brute force (or if you need tenure you could call it a Monte Carlo approach). I randomly sampled 100,000 times from each possible pair of journals, then calculated the percentage of times the article with more citations was from a journal with a higher impact factor. For example, in 100,000 comparisons of random pairs sampled from ASR and Social Forces (the two journals with the biggest JIF spread), 73% of the time the ASR article had more citations.

Is 73% a lot? It’s better than a coin toss, but I’d hate to have a promotion or hiring decision be influenced by an instrument that blunt. Here are results of the 10.5 million comparisons I made (I love computers). Click to enlarge:

Outside of the ASR column, these are very bad; in the ASR column they’re pretty bad. For example, a random article from AJS only has more citations than one from the 12 lower-JIF journals 59% of the time. So if you’re reading CVs, and you see one candidate with a two-year old AJS article and one with a two-year-old Work & Occupations article, what are you supposed to do? You could compare the actual citations the two articles have gotten, or you could assess their quality of impact some other way. You absolutely should not just skim the CV and assume the AJS article is or will be more influential based on the journal title alone; the failure probability of that assumption is too high.

On my table you can also see some anomalies, of the kind which plague this system. See all that brown in the BJS and Sociology of Religion columns? That’s because both of those journals had sudden increases in their JIF, so their more recent articles have more citations, and most of the comparisons in this table (like in your memory, probably) are based on data from a few years before that. People who published in these journals three years ago are today getting an undeserved JIF bounce from having these titles on their CVs. (See the supplemental analysis below for more on this.)

Conclusion

Using JIF to decide which papers in different sociology journals are likely to be more impactful is a bad idea. Of course, lots of people know JIF is imperfect, but they can’t help themselves when evaluating CVs for hiring or promotion. And when you show them evidence like this, they might say “but what is the alternative?” But as Brito & Rodríguez-Navarro write: “if something were wrong, misleading, and inequitable the lack of an alternative is not a cause for continuing using it.” These error rates are unacceptably high.

In sociology most people won’t own up to relying on impact factors, but most people (in my experience) do judge research by where it’s published all the time. If there is a very big difference in status — enough to be associated with an appreciably different acceptance rate, for example — that’s not always wrong. But it’s a bad default.

In 2015 the biologist Michael Eisen suggested that tenured faculty should remove the journal titles from their CVs and websites, and just give readers the title of the paper and a link to it. He’s done it for his lab’s website, and I urge you to look at it just to experience the weightlessness of an academic space where for a moment overt prestige and status markers aren’t telling you what to think. I don’t know how many people have taken him up on it. I did it for my website, with the explanation, “I’ve left the titles off the journals here, to prevent biasing your evaluation of the work before you read it.” Whatever status I’ve lost I’ve made up for in virtue-signaling self-satisfaction — try it! (You can still get the titles from my CV, because I feel like that’s part of the record somehow.)

Finally, I hope sociologists will become more sociological in their evaluation of research — and of the systems that disseminate, categorize, rank, and profit from it.

Supplemental analysis

The analysis thus far is, in my view, a damning indictment of real-world reliance on the Journal Impact Factor for judging articles, and thus the researchers who produce them. However, it conflates two problems with the JIF. First is the statistical problem of imputing status from an aggregate to an individual, when the aggregate measure fails to capture variation that is very wide relative to the difference between groups. Second, more specific to JIF, is the reliance on a very time-specific comparison: citations in year three to publications in years one and two. Someone could do (maybe already has) an analysis to determine the best lag structure for JIF to maximize its predictive power, but the conclusions from the first problem imply that’s a fool’s errand.

Anyway, in my sample the second problem is clearly relevant. My analysis relies strictly on the rank-ordering provided by the JIF to determine whether article comparisons succeed or fail. However, the sample I drew covers four years, 2016-2019, and counts citations to all of them through 2020. This difference in time window produces a rank ordering that differs substantially (the rank order correlation is .73), as you can see:

In particular, three journals (BJS, SOR, and SFO) moved more than five spots in the ranking. A glance at the results table above shows that these journals are dragging down the matching success rate. To pull these two problems apart, I repeated the analysis using the ranking produced within the sample itself.

The results are now much more straightforward. First, here is the same box plot but with the new ordering. Now you can see the ranking more clearly, though you still have to squint a little.

And in the match rate analysis, the result is now driven by differences in means and variances rather than by the mismatch between JIF and sample-mean rankings (click to enlarge):

This makes a more logical pattern. The most differentiated journal, ASR, has the highest success rate, and the journals closest together in the ranking fail the most. However, please don’t take from this that such a ranking becomes a legitimate way to judge articles. The overall average on this table is still only 58%, up only 4 points from the original table. Even with a ranking that more closely conforms to the sample, this confirms Brito and Rodríguez-Navarro’s conclusion: “[when rankings] of the journals are not so different … the failure probability can be close to 0.5, which is equivalent to evaluating by coin flipping.”

These match numbers are too low to responsibly use in such a way. These major sociology journals have citation rates that are too variable, and too similar at the mean, to be useful as a way to judge articles. ASR stands apart, but only because of the rest of the field. Even judging an ASR paper against its lower-ranked competitors produces a successful one-to-one ranking of papers just 72% of the time — and that only rises to 82% with the least-cited journal on the list.

The supplemental analysis is helpful for differentiating the multiple problems with JIF, but it does nothing to solve the problem of using journal citation rates to evaluate individual articles.


*The data and Stata code I used is up here: osf.io/zutws. This includes the lists of all articles in the 15 journals from 2016 to 2020 and their citation counts as of the other day (I excluded 2020 papers from the analysis, but they’re in the lists). I forgot to save the version of the 100k-case random file that I used to do this, so I guess that can never be perfectly replicated; but you can probably do it better anyway.

Sociologist, scientist? Toward transparency, accountability, and a sharing culture

With the help of the designer Brigid Barrett, I have a new website at philipncohen.com, and a redesigned blog to match (which you’re looking at now). We decided on the tagline, “Sociologist / Demographer” for the homepage photo. It’s true I am those two things, but I also like how they modify each other, a type of sociologist and a type of demographer. First some reflections, then a little data.

I shared the website on Twitter, and wrote this in a thread:

Having “sociologist” attached to your name is not going to signal scientific rigor to the public in the way that other discipline labels might (like, I think, “demographer”). A lot of sociologists, as shown by their behavior, are fine with that. Your individual behavior as a researcher can shape the impression you make, but it will not change the way the discipline is seen. Until the discipline — especially our associations but also our departments — adopts (and communicates) scientific practices, that’s how it will be. As an association, ASA has shown little interest in this, and seems unlikely to soon.

A substantial portion of sociologists rejects the norms of science. Others are afraid that adopting them will make their work “less than” within the discipline’s hierarchy. For those of us concerned about this, the practices of science are crucial: openness, transparency, reproducibility. We need to find ways at the sub-discipline level to adopt and communicate these values and build trust in our work. Building that trust may require getting certain publics to see beyond the word “sociologist,” rather than just see value in it. They will see our open practices, our shared data and code, our ability to admit mistakes, embrace uncertainty, and entertain alternative explanations.

There are other sources of trust. For example, taking positions on social issues or politics is also a way of building trust with like-minded audiences. These are important for some sociologists, and truly valuable, but they’re different from science. Maybe unreasonably, I want both. I want some people to give my work a hearing because I take antiracist or feminist positions in my public work, for example. And also because I practice science in my research, with the vulnerability and accountability that implies. Some people would say my public political pronouncements undermine not just my science, but the reputation of the discipline as a whole. I can’t prove they’re wrong. But I think the roles of citizen and scholar are ultimately compatible. Having a home in a discipline that embraced science and better communicated its value would help. A scientific brand, seal of approval, badges, etc., would help prevent my outspokenness from undermining my scientific reputation.

One reply I got, confirming my perception, was, “this pretence of natural science needs to be resisted not indulged.” Another wrote: “As a sociologist and an ethnographer ‘reproducibility’ will always be a very weak and mostly inapplicable criterion for my research. I’m not here to perform ‘science’ so the public will accept my work, I’m here to seek truth.” Lots of interesting responses. Several people shared this old review essay arguing sociology should be more like biology than like physics, in terms of epistemology. The phrase “runaway solipsism” was used.

I intended my tweets to focus on the open “science practices” which which I have been centrally concerned, centered on scholarly communication: openness, transparency, replicability. That is, I am less interested in the epistemological questions of what is meaning and truth, and solipsism, and more concerned with basic questions like, “How do we know researchers are doing good research, or even telling the truth?” And, “How can we improve our work so that it’s more conducive to advancing research overall?”

Whether or not sociology is science, we should have transparency, accountability, and a sharing culture in our work. This makes our work better, and also maybe increases our legitimacy in public.

Where is ASA?

To that end, as an elected member of the American Sociological Association Committee on Publications, two years ago I proposed that the association adopt the Transparency and Openness Promotion Guidelines from the Center for Open Science, and to start using their Open Science Badges, which recognize authors who provide open data, open materials, or use preregistration for their studies. It didn’t go over well. Some people are very concerned that rewarding openness with little badges in the table of contents, which presumably would go mostly to quantitative researchers, would be seen as penalizing qualitative researchers who can’t share their data, thus creating a hierarchy in the discipline.

So at the January 2019 meeting the committee killed that proposal so an “ad hoc committee could be established to evaluate the broader issues related to open data for ASA journals.” Eight months later, after an ad hoc committee report, the publications committee voted to “form an ad hoc committee [a different one this time] to create a statement regarding conditions for sharing data and research materials in a context of ethical and inclusive production of knowledge,” and to, “review the question about sharing data currently asked of all authors submitting manuscripts to incorporate some of the key points of the Committee on Publications discussion.” The following January (2020), the main committee was informed that the ad hoc committee had been formed, but hadn’t had time to do its work. Eight months later, the new ad hoc committee proposed a policy: ask authors who publish in ASA journals to declare whether their data and research materials are publicly available, and if not why not, with the answers to be appended in a footnote to each article. The minutes aren’t published yet, but I seem to remember us approving the proposal (minutes should appear in the spring, 2021). So, after two years, all articles are going to report whether or not materials are available. Someday. Not bad, for ASA!

To see how we’re doing in the meantime, and inspired by the Twitter exchange, I flipped through the last four issues of American Sociological Review, the flagship journal of the association, to assess the status of data and materials sharing. That is, 24 articles published in 2020. The papers and what I found are listed in the table below.

There were six qualitative papers and three mixed qualitative/quantitative papers. None of these provided access to research materials such as analysis code, interview guides, survey instruments, or transcripts — or provided an explanation for why these materials were not available. Among the 15 quantitative papers, four provided links to replication packages, with the code required to replicate the analyses in the papers. Some of these used publicly available data, or included the data in the package, while the others would require additional steps to gain access to the data. The other 11 provided neither data nor code or other materials.

That’s just from flipping through the papers, searching for “data,” “code,” “available,” reading the acknowledgments and footnotes, and so on. So I may have missed something. (One issue, which maybe the new policy will improve, is that there is no standard place on the website or in the paper for such information to be conveyed.) Many of the papers include a link on the ASR website to “Supplemental Material,” but in all cases this was just a PDF with extra results or description of methods, and did not include computer code or data. The four papers that had replication packages all linked to external sites, such as Github or Dataverse, which are great but are not within the journal’s control, so the journal can’t ensure they are correct, or that they are maintained over time. Still, those are great.

I’m not singling out papers (which, by the way, seem excellent and very interesting — good journal!), just pointing out the pattern. Let’s just say that any of these authors could have provided at least some research materials in support of the paper, if they had been personally, normatively, or formally compelled to do so.

Why does that matter?

First, providing things like interview guides, coding schemes, or statistical code, is helpful to the next researcher who comes along. It makes the article more useful in the cumulative research enterprise. Second, it helps readers identify possible errors or alternative ways of doing the analysis, which would be useful both to the original authors and to subsequent researchers who want to take up the baton or do similar work. Third, research materials can help people determine if maybe, just maybe, and very rarely, the author is actually just bullshitting. I mean literally, what do we have besides your word as a researcher that anything you’re saying is true? Fourth, the existence of such materials, and the authors’ willingness to provide them, signals to all readers a higher level of accountability, a willingness to be questioned — as well as a commitment to the collective effort of the research community as a whole. And, because it’s such an important journal, that signal might boost the reputation for reliability and trustworthiness of the field overall.

There are vast resources, and voluminous debates, about what should be shared in the research process, by whom, for whom, and when — and I’m not going to litigate it all here. But there is a growing recognition in (almost) all quarters that simply providing the “final” text of a “publication” is no longer the state of the art in scholarly communication, outside of some very literary genres of scholarship. Sociology is really very far behind other social science disciplines on this. And, partly because of our disciplinary proximity to the scholars who raise objections like those I mentioned above, even those of us who do the kind of work where openness is most normative (like the papers below that included replication packages), can’t move forward with disciplinary policies to improve the situation. ASR is paradigmatic: several communities share this flagship journal, the policies of which are serving some more than others.

What policies should ASA and its journals adopt to be less behind? Here are a few: Adopt TOP badges, like the American Psychological Association has; have their journals actually check the replication code to see that it produces the claimed results, like the American Economic Association does; publish registered reports (peer review before results known), like all experimental sciences are doing; post peer review reports, like Nature journals, PLOS, and many others do. Just a few ideas.

Change is hard. Even if we could agree on the direction of change. Brian Nosek, director of the Center for Open Science (COS), likes to share this pyramid, which illustrates their “strategy for culture and behavior change” toward transparency and reproducibility. The technology has improved so that the lowest two levels of the pyramid are pretty well taken care of. For example, you can easily put research materials on COS’s Open Science Framework (with versioning, linking to various cloud services, and collaboration tools), post your preprint on SocArXiv (which I direct), and share them with the world in a few moments, for free. Other services are similar. The next levels are harder, and that’s where we in sociology are currently stuck.

COS_Culture_and_Behavior_Change_model.width-500 1

For some how-to reading, consider, Transparent and Reproducible Social Science Research: How to Do Open Science, by Garret Christensen, Jeremy Freese, and Edward Miguel (or this Annual Review piece on replication specifically). For an introduction to Scholarly Communication in Sociology, try my report with that title. Please feel free to post other suggestions in the comments.


Four 2020 issues of American Sociological Review

ReferenceQuant/QualData typeData available?Code available?Note
Faber, Jacob W. 2020. “We Built This: Consequences of New Deal Era Intervention in America’s Racial Geography.” American Sociological Review 85 (5): 739–75.QuantCensus+NoNo
Brown, Hana E. 2020. “Who Is an Indian Child? Institutional Context, Tribal Sovereignty, and Race-Making in Fragmented States.” American Sociological Review 85 (5): 776–805. QualArchivalNoNo
Daminger, Allison. 2020. “De-Gendered Processes, Gendered Outcomes: How Egalitarian Couples Make Sense of Non-Egalitarian Household Practices.” American Sociological Review 85 (5): 806–29. QualInterviewsNoNo
Mazrekaj, Deni, Kristof De Witte, and Sofie Cabus. 2020. “School Outcomes of Children Raised by Same-Sex Parents: Evidence from Administrative Panel Data.” American Sociological Review 85 (5): 830–56. QuantAdministrativeNoUpon requestInfo on how to obtain data provided.
Becker, Sascha O., Yuan Hsiao, Steven Pfaff, and Jared Rubin. 2020. “Multiplex Network Ties and the Spatial Diffusion of Radical Innovations: Martin Luther’s Leadership in the Early Reformation.” American Sociological Review 85 (5): 857–94. QuantNetworkNoNoSays data is in the ASR online supplement but it’s not.
Smith, Chris M. 2020. “Exogenous Shocks, the Criminal Elite, and Increasing Gender Inequality in Chicago Organized Crime.” American Sociological Review 85 (5): 895–923. QuantNetworkNoNoCode described.
Storer, Adam, Daniel Schneider, and Kristen Harknett. 2020. “What Explains Racial/Ethnic Inequality in Job Quality in the Service Sector?” American Sociological Review 85 (4): 537–72. QuantSurveyNoNo
Ranganathan, Aruna, and Alan Benson. 2020. “A Numbers Game: Quantification of Work, Auto-Gamification, and Worker Productivity.” American Sociological Review 85 (4): 573–609. MixedMixedNoNo
Fong, Kelley. 2020. “Getting Eyes in the Home: Child Protective Services Investigations and State Surveillance of Family Life.” American Sociological Review 85 (4): 610–38. QualMixedNoNo
Musick, Kelly, Megan Doherty Bea, and Pilar Gonalons-Pons. 2020. “His and Her Earnings Following Parenthood in the United States, Germany, and the United Kingdom.” American Sociological Review 85 (4): 639–74. QuantSurveyYesYesOffsite replication package.
Burdick-Will, Julia, Jeffrey A. Grigg, Kiara Millay Nerenberg, and Faith Connolly. 2020. “Socially-Structured Mobility Networks and School Segregation Dynamics: The Role of Emergent Consideration Sets.” American Sociological Review 85 (4): 675–708. QuantAdministrativeNoNo
Schaefer, David R., and Derek A. Kreager. 2020. “New on the Block: Analyzing Network Selection Trajectories in a Prison Treatment Program.” American Sociological Review 85 (4): 709–37. QuantNetworkNoNo
Choi, Seongsoo, Inkwan Chung, and Richard Breen. 2020. “How Marriage Matters for the Intergenerational Mobility of Family Income: Heterogeneity by Gender, Life Course, and Birth Cohort.” American Sociological Review 85 (3): 353–80. QuantSurveyNoNo
Hook, Jennifer L., and Eunjeong Paek. 2020. “National Family Policies and Mothers’ Employment: How Earnings Inequality Shapes Policy Effects across and within Countries ,  National Family Policies and Mothers’ Employment: How Earnings Inequality Shapes Policy Effects across and within Countries.” American Sociological Review 85 (3): 381–416. QuantSurvey+YesYesOffsite replication package.
Doering, Laura B., and Kristen McNeill. 2020. “Elaborating on the Abstract: Group Meaning-Making in a Colombian Microsavings Program.” American Sociological Review 85 (3): 417–50. MixedSurvey+NoNo
Decoteau, Claire Laurier, and Meghan Daniel. 2020. “Scientific Hegemony and the Field of Autism.” American Sociological Review 85 (3): 451–76. QualArchivalNoNo“Information on the coding schema is available upon request.”
Kiley, Kevin, and Stephen Vaisey. 2020. “Measuring Stability and Change in Personal Culture Using Panel Data.” American Sociological Review 85 (3): 477–506. QuantSurveyYesYesOffsite replication package.
DellaPosta, Daniel. 2020. “Pluralistic Collapse: The ‘Oil Spill’ Model of Mass Opinion Polarization.” American Sociological Review 85 (3): 507–36. QuantSurveyYesYesOffsite replication package.
Simmons, Michaela Christy. 2020. “Becoming Wards of the State: Race, Crime, and Childhood in the Struggle for Foster Care Integration, 1920s to 1960s.” American Sociological Review 85 (2): 199–222. QualArchivalNoNo
Calarco, Jessica McCrory. 2020. “Avoiding Us versus Them: How Schools’ Dependence on Privileged ‘Helicopter’ Parents Influences Enforcement of Rules.” American Sociological Review 85 (2): 223–46. QualEthnography w/ surveyNoNo
Brewer, Alexandra, Melissa Osborne, Anna S. Mueller, Daniel M. O’Connor, Arjun Dayal, and Vineet M. Arora. 2020. “Who Gets the Benefit of the Doubt? Performance Evaluations, Medical Errors, and the Production of Gender Inequality in Emergency Medical Education.” American Sociological Review 85 (2): 247–70. MixedAdministrativeNoNo
Kristal, Tali, Yinon Cohen, and Edo Navot. 2020. “Workplace Compensation Practices and the Rise in Benefit Inequality ,  Workplace Compensation Practices and the Rise in Benefit Inequality.” American Sociological Review 85 (2): 271–97.QuantAdministrativeNoNo
Abascal, Maria. 2020. “Contraction as a Response to Group Threat: Demographic Decline and Whites’ Classification of People Who Are Ambiguously White.” American Sociological Review 85 (2): 298–322.QuantSurvey experimentNoNoPreanalysis plan registered. Data embargoed.
Friedman, Sam, and Aaron Reeves. 2020. “From Aristocratic to Ordinary: Shifting Modes of Elite Distinction.” American Sociological Review 85 (2): 323–50.QuantArchivalNoNo

Where preprints fit in, COVID-19 edition

I recorded a 16-minute talk on the scientific process, science communication, and how preprints fit in to the information ecosystem around COVID-19.

It’s called, “How we know: COVID-19, preprints, and the information ecosystem.” The video is on YouTube here, also embedded below, and the slides, with references, are up here.

Happy to have your feedback, in the comments or any other way.

COVID-19 code, data, codebooks, figures

Every day for who knows how long I’ve tinkered with COVID-19 data and made graphs using Stata. Now I’ve condensed my tools down to several elements, updated daily, which I’m sharing:

  • A program that assembles the COVID death and case data, by date, at the county, state, and country level. To this I have added some population, income, and political variables. The program is here, along with the codebook it outputs.
  • The data file is here in Stata format and CSV format. It’s in long shape, so one record for each place on each date.
  • A Stata program that makes my favorite graphs right now (currently 24 per day). The Figures are stored here in PNG format.
  • The Stata scheme I use to make them look the how I like is here.

These files are linked to my laptop so they update automatically when I revise them. Yay, Open Science Framework, which is non-profit, open source, free to use, and deserves your support.

I hope someone finds these helpful, for teaching or exploring on their own. It’s all yours.

Here are a few figures from today’s runs (click to enlarge):

counties with any cases

deaths and GDP scatter

The blog’s decade

Blogging is dead. Long live the blog!

At 268,000, visits to this blog are now down 37% from the peak year of 2015. At the same time, this year I had the fewest number of new posts, just 39. On the other hand, this year I had 25 million impressions on Twitter. Whatever that means.

decade-stats

In my case, and probably many others, the role of the blog has changed with the growth of Twitter. A lot of what the blog did was provide an immediate outlet for daily chatter and work in progress thoughts, a way to get feedback, check in with colleagues, learn new things and meet new people. That’s a lot of what I use Twitter for now, more efficiently (if more noisily).

The other squeeze on the blog is the imperative to do open science more systematically, for which I use the Open Science Framework to post data and code — in projects, which may include multiple files, and quick files for single documents. And of course I use SocArXiv for more formal working papers, reviews, and preprints (mine are here).

So what is the role of the blog? It’s the place for official news and announcements about new work — including notifications of stuff I’m publishing elsewhere — longer arguments, and informal work. It’s a way for people to subscribe to my news via email (it also goes on Facebook, which a lot of sociologists use).

In several talks I have tried to illustrate the total information strategy in something like this pentagulation:

pentagulate

For a wider perspective, I also wrote a report on Scholarly Communication in Sociology, which is intended especially for grad students and early career scholars.

I’m happy to hear suggestions (on any platform) for how to handle communication strategy.

Book aside

The tricky relationship between platforms and different media came home to roost in my book, Enduring Bonds: Inequality, Marriage, Parenting, and Everything Else That Makes Families Great and Terrible. That book was inspired by the success of this blog, which is what enticed University of California Press to consider it. Unbeknownst to the vast majority of my readers on other platforms, I worked pretty hard on it, selecting the best blog posts, and then combining, updating, and adding to them to make a collection of essays, with data. I don’t know how successful the book is compared with other academic books generally, but, with almost no marketing beyond my social media platforms, it has generated basically no buzz for me (media, invitations, etc.). That’s in contrast to working papers, tweets, and blog posts, which continue to bring in wider attention. I know other people have done amazing blog-to-book projects, but this experience definitely showed me that the successful translation is far from automatic. Live and learn! Maybe in the long run the book will be what persists from the first decade of this blog.

ASA’s letter against the public interest and our values

youdidwhat

Update 1: I submitted a resolution to the ASA Committee on Publications, for consideration at our January meeting. You can read and comment on it here.

Update 2: The Committee on Publications on January 23 voted to approve the following statement: “The ASA Committee on Publications expresses our opposition to the decision by the ASA to sign the December 18, 2019 letter.”

The American Sociological Association has signed a letter that profoundly betrays the public interest and goes against the values that many of us in the scholarly community embrace.

The letter to President Trump, signed by dozes of academic societies, voices opposition to a rumored federal policy change that would require federally funded research be made freely available upon publication, rather than according to the currently mandated 12-month embargo — which ASA similarly, bitterly, opposed in 2012. ASA has not said who made the decision to sign this letter. All I know is that, as a member of the Committee on Publications, I wasn’t consulted or notified. I don’t know what the ASA rules are for issuing such statements in our name, but this one is disgraceful.

The argument is that ASA would not be able to make money selling research generated by federal funding if it were required to be distributed for free. And because ASA would suffer, science and the public interest would suffer. Like when Trump says getting Ukraine to help him win re-election is by definition in the American interest — what helps ASA is what’s good for science.

The letter says:

Currently, free distribution of research findings is subject to a 12-month embargo, enabling American publishers to recover the investment made in curating and assuring the quality of scientific research content. … The current 12-month embargo period provides science and engineering society publishers the financial stability that enables us to support peer review that ensures the quality and integrity of the research enterprise.

That is funny, because in 2012 ASA director Sally Hillsman (since retired) said the 12-month embargo policy “could threaten the ability of scholarly societies, including the ASA, to continue publishing journals” and was “likely to seriously erode and eventually jeopardize our financial ability to perform the critical, value added peer review and editorial functions of scientific publishing.”

The current letter, at least with regard to ASA, tell this whopper: “we support open access and have a strong history of advancing open access through a broad array of operational models.” They literally oppose open access, including in this letter, and including the current, weak, open access policy.

The ASA-signed letter is very similar to one sent about the same time by a different (but overlapping) large group of publishers, including Elsevier, and the U.S. Chamber of Commerce, claiming the rumored policy would hurt ‘merica. But there are subtle differences. The ASA letter refers to “the current proven and successful model for reporting, curating and archiving scientific results and advancing the U.S. research enterprise,” which should not be tampered with. The other letter warns of the danger of “step[ing] into the private marketplace” in which they sell research. Knowledge philosopher Peter Suber offered an excellent critique of the market claims here in this Twitter thread:

ASA and the other money-making societies really want you to believe there is no way to do curation and peer review without them. If we jeopardize their business model, ASA says, the services they provide would not happen. In fact, the current subscription models and paywalls stand in the way of developing the cheaper, more efficient models we could build right now to replace them. All we need to do is take the money we currently devote to journal subscriptions and publisher profits, and redirect it to the tasks of curation and peer review without profits and paywalls — and free distribution (which is a lot cheaper to administer than paywalled distribution).

The sooner we start working on that the better. In this effort — and in the absence of leadership by scholarly societies — the university libraries are our strongest allies. This is explained by UNC Librarian Elaine Westbrooks in this Twitter thread:

Compare this forwarding thinking librarian’s statement with Elsevier. In proudly sharing the publishers’ statement, Elsevier vice president Ann Gabriel said, “Imagine a world without scientific, medical societies and publishers who support scholarship, discovery and infrastructures for peer review, data archiving and networks.” Notice two things in this statement. First, she does not mention libraries, which are the academy-owned institutions that do literally all this as well. And second, see how she bundles publishers and societies. This is the sad reality. If instead of “societies and publishers” we had “societies and libraries” maybe we’d be getting somewhere. Instead, our societies, including the American Sociological Association, are effectively captured by publishers, and represent their interests instead of the public interest, and the values of our community.

I remain very pessimistic about ASA, which is run by a professional group with allegiance to the paywall industry, along with mostly transient, naive, and/or ineffectual academics (of which I am certainly one). But I’m torn, because I want to see a model of scholarly societies that works, which is why I agreed to serve of the ASA Committee on Publications — which mostly does busy work for the association while providing the cover of legitimacy for the professional staff.

Letter of opposition

So I posted a letter expressing opposition to the ASA letter. If you are a sociologist, I hope you will consider sharing and signing it. We got 100 signatures on the first day, but it will probably take more for ASA to care. To share the letter, you can use this link: https://forms.gle/ecvYk3hUmEh2jrETA.

It reads:

In light of a rumored new White House Open Access Policy, the American Sociological Association (ASA), and other scholarly societies, signed a letter to President Trump in support of continued embargoes for federally-funded research.

We are sociologists who join with libraries and other advocates in the research community in support of federal policy to make the results of taxpayer-funded research immediately available to the public for free. We endorse a policy that would eliminate the current 12-month waiting period for open access to the outputs of taxpayer-funded scientific research. Ensuring full open access to publicly-funded research contributes to the public good by improving scientific productivity and equalizing access — including international access — to valuable knowledge that the public has already paid for. The U.S. should join the many other countries that already have strong open access policies.

We oppose the decision by ASA to sign this letter, which goes against our values as members of the research community, and urge the association to rescind its endorsement, to join the growing consensus in favor of open access to to scholarship, including our own.

Why we need open science in demography, and how we can make it happen

“Why we need open science in demography, and how we can make it happen” is the title of a talk I gave at the Max Planck Institute for Demographic Research yesterday, as part of an open science workshop they hosted in Rostock, Germany. (The talk was not nearly as definitive as the title.)

The other (excellent) keynote was by Monica Alexander. I posted the slides from my talk here. There should be a video available later. The organizing committee for the event is working to raise the prominence of open science discussions at the Institute, and consider practices and policies they might adopt. We had a great meeting.

As an aside, I also got to hear an excellent tutorial by E. F. Haghish, who has written Markdoc, a “literate programming” (markdown) package for Stata, which is very cool. These are his slides.

rostock talk 2rostock group shot