This post deals with problems in academic research publishing. It’s off the usual topic of the blog, although the publications in question do concern families and inequality. I decided to publish it here rather than try to place it somewhere else because I thought it might be controversial, and I want to take personal responsibility for it. I welcome discussion of these questions here in the comments, or in other forums where these issues are pertinent — you are welcome to repost this, with attribution.
The case here is a pair of articles by John R. Hipp, an associate professor of Criminology, Law and Society at U.C. Irvine. The two articles are:
- Hipp, John R. 2010. “The Role of Crime in Housing Unit Racial/ethnic Transition,” Criminology 48(3). DOI: 10.1111/j.1745-9125.2010.00199.x
- Hipp, John R. 2011. “Violent Crime, Mobility Decisions, and Neighborhood Racial/Ethnic Transition.” Social Problems 58(3). DOI: 10.1525/sp.2011.58.3.410
Little of substance is learned from one that could not be learned from the other, they contain many nearly-identical passages, and they both claim to make the same major original contributions. This isn’t the most extreme case like this ever published, but it’s the most obvious one I’ve noticed that involves a major sociology journal. Without attributing any cause or motivation, we can call these two “very similar publications.”
The practice of publishing VSPs:
- Wastes the time of editors, reviewers, and future researchers.
- Takes up valuable space in journals, space for which other researchers are competing for their publications and to enhance their careers.
- Misleads reviewers and administrators who are evaluating and comparing publication records.
- Misleads the research community by creating a false impression of the weight of original research on the topic (e.g., “many studies show”).
Here are the two abstracts. I don’t know an elegant way to show these side beside except a screen grab (click for higher resolution); I numbered the sentences in the abstracts.
I highlighted the major difference between the two articles: the Criminology article analyzes perceptions of crime in “microneighborhoods,” while the Social Problems piece analyzes reported violent crime rates in Census tracts (and adds a measure of short-term crime rate change). This is a substantive difference, and it involves using different versions of the American Housing Survey. But the abstracts are not written as a sequence of incremental discoveries; they claim to make a number of the same innovations and discoveries. Substantively, the difference could have been handled with one additional table, or even a hefty footnote. (The first paper reports that the violent crime rate and residents’ perceptions are correlated at about .70). In any event, the difference is not a significant part of the motivation for either article, as the abstracts make clear.
Here are the outlines of the articles, with Hipp’s headings. As you can see, the Social Problems article includes a measure of short-term crime rate change, and the Criminology article includes a section on sensitivity analyses. They are not identical (the first paper also includes a methodological appendix).
The second paper, from Social Problems, does acknowledge the existence of the first, but not in a way that would truly communicate the relationship between them. It notes (p. 411):
Recent scholarship has suggested that … residents’ perceptions of crime in the microneighborhood can differentially affect in-mobility and out-mobility for different racial/ethnic groups (Hipp 2010b). We extend this literature by using official violent crime rates within the broader neighborhood as measured by the census tract.
Later (p. 414), the Social Problems paper minimizes the findings of the Criminology paper:
One study provided suggestive evidence of disproportionate out-mobility using information on the perceptions of crime among residents living within a micro-neighborhood of the nearest 11 housing units (Hipp 2010b). Whereas white households perceiving more crime were more likely to move within four years, black and Latino households showed no such tendency (Hipp 2010b). Furthermore, whites living in microneighborhoods with a general perception of more crime were also more likely to leave the unit, whereas Latino and black households again showed no such tendency. This evidence of the importance of perceptions of crime within a small micro-environment is important, but it cannot assess whether such perceptions accurately capture the crime environment of the micro-area, nor whether the crime environment of the broader neighborhood is also important. The present study addresses these limitations.
The Criminology paper was published online in August 2010, and the Social Problems paper is dated August 2011, so I can’t tell if the reviewers for the Social Problems paper had access to a published copy of the first at the time of their review.
If the second paper constitutes a genuine additional contribution, it would be reasonable to publish it separately, making clear that it represents a methodological variation of findings already known. But instead the second paper announces the same contributions as the first. In fact, as the text from the sections titled “summary” in the introduction of both articles show, two out of three of the “important contributions to the literature” are identical:
The third contribution differs. However, by the publication of the second paper, the first two “important contributions” are no longer original (the generally understood meaning of “contribution”).
Later, in the conclusions, the assertions of originality (the “key/crucial implication” and “important takeaway”) are nearly identical. Here are some excepts from the conclusions:
The genuine relationship between the two papers is not revealed.
There is no substitute for reading the text with human eyes. However, there also are tools for displaying and analyzing similarity in documents. The U.S. Dept. of Health and Human Services Office of Research Integrity provides a reference to the Plagiarism Resource Center (once) at the University of Virginia, which distributes a program called WCopyfind, an open-source Windows-based document comparison tool.
After converting these two articles to text documents, and removing the tables, references, methodological appendix, and extraneous page numbers and other fragments, I subjected them to the WCopyfind comparison. These are the parameters I used, which were recommended for cases in which minor editing is presumed.
Shortest Phrase to Match: 6
Ignore Punctuation: No
Ignore Outer Punctuation: Yes
Ignore Letter Case: Yes
Skip Long Words: No
Most Imperfections to Allow: 0
The analysis found 2,235 words that were in duplicate 6-word strings, accounting for 20% of each article’s text. For example, here are two passages, with the strings that the algorithm flagged in red:
The present study provides an important corrective to the large volume of prior research finding a positive relationship between the size of racial/ethnic minority groups in a neighborhood and the rate of crime at one point in time and assuming that the causal direction runs from the presence of such minorities to higher rates of crime.
Prior research frequently has found a relationship between the presence of racial/ethnic minorities in a neighborhood and the rate of crime at one point in time. Although they sometimes posit different mechanisms, these studies almost always conclude that the causal direction runs from the presence of such minorities to higher rates of crime.
On the one hand, we can see that the passages are more similar than the red text implies, but on the other hand there are times when the algorithm appears to just catch phrases that occur in this line of research, such as “are more likely to move into.” If you set the shortest phrase to 5 words, the program flags 23% of the text; at 10 words it flags 12%.
The nice thing about the program is it creates a side-by-side comparison of the entire documents, with the common text strings linked, so you can click on the text in one article and see where it appears in the other, in context. I have put this side-by-side file here for your perusal. The full articles (behind paywalls) are linked above.
(Odd aside: the Criminology paper is in the first person singular, while the Social Problems paper is written in the first person plural, although both have only one author.)
What is this?
This is not a duplicate paper, although it includes a large amount of copied text, which would normally be called “self-plagiarism,” defined by Miguel Roig like this:
Whereas plagiarism involves the presentation of others’ ideas, text, data, images, etc., as the products of our own creation, self-plagiarism, occurs when we decide to reuse in whole or in part our own previously disseminated ideas, text, data, etc without any indication of their prior dissemination. Perhaps the most commonly-known form of self-plagiarism is duplicate publication, but other forms exist and include redundant publication, augmented publication, also known as meat extender, and segmented publication, also known as salami, piecemeal, or fragmented publication. The key feature in all forms of self-plagiarism is the presence of significant overlap between publications and, most importantly, the absence of a clear indication as to the relationship between the various duplicates or related papers.
Thinly-sliced “salami” articles do not necessarily include any duplication, but rather just tiny increments of scientific progress. There is some of that here, as well as elements of a “meat extender” case, in which small amounts of additional data or analysis are added — without a transparent disclaimer and rationale. However, regardless of the differences in analysis, the texts — especially the framing and concluding bread around the salami — are too similar to be justified.
That last distinction in Roig’s definition, the “key feature,” is what’s important here. Using a little boilerplate theoretical language in related work, or a very similar equation or description of variables when using the same datasets, doesn’t undermine the value of the work, as long as the original source is attributed. This distinction appears in a Nature news story:
…although the repetition of the methods section of a paper is not necessarily considered inappropriate by the scientific community, “we would expect that results, discussion and the abstract present novel results”, says Harold Garner, a bioinformatician at Virginia Polytechnic Institute and State University in Blacksburg.
There are in fact some “novel results” in the second paper, but the novelty is not as important as the basic findings, which are described nearly identically — as original in both cases.
In the normal course of pursuing a research agenda across multiple article-length publications, some repetition is justifiable and even helpful. But without a “a clear indication as to the relationship between the various duplicates or related papers,” in Roig’s words, the sandwich is not kosher.
Consider an alternative example, in an economics paper, “Booms, Busts, and Divorce,” by Judith Hellerstein and Melinda Morrill, which I wrote about recently. Their main contribution is the finding that divorce was pro-cyclical from 1976 to 2009 – that is, there was more divorce when the economy grew. The “main analysis” is this:
we combine data on state-by-year unemployment rates with state-by-year vital statistics data across the United States over the period 1976 to 2009. We assess the impact of local macroeconomic conditions on state-level divorce rates, controlling for year fixed effects, state-specific time trends, and state-specific time-invariant determinants of divorce rates.
But they then add this:
Our basic finding of pro-cyclical divorce is robust to alternative empirical specifications and is found when we allow the effect of unemployment rates on divorce to vary by the fraction of the population that is Catholic … the census region, and by time period. We also show that this finding is robust to extending our unemployment series back to 1970 … Finally, we replace the unemployment rate at the state-by-year level as our measure of macroeconomic conditions with two alternative measures: state-by-year per capita gross domestic product (GDP) and state-by-year per capita income.
Each of those different steps is a way of extending and corroborating the “main analysis,” and each required adding data from a different source and conducting a new statistical analysis. But they did not make the main finding a new discovery warranting an additional publication with the same motivation and conclusions as the first.
Look at me
To head off an inevitable question, you are welcome to look at my own publication record. There are two specific cases in which my co-author Matt Huffman and I wrote pairs of articles addressing similar questions. They were: whether variation in gender segregation across labor markets affected patterns of gender wage devaluation (here and here); and whether the presence of female managers is associated with lower levels of gender inequality (here and here).
In each case the later paper acknowledged the earlier (or, they acknowledged each other, in the first pair), and explained the differences in approach — they involved different kinds of data and statistical methods in each case as well. The results were consistent in each pair, and thus the conclusions were strengthened, the pattern corroborated. Maybe we could have combined each pair into one very dense article, but that might very well have been rejected as too long or complicated for the journals we used, and they weren’t ready at the same time. (For what it’s worth, I also checked the WCopyfind comparison between the latter pair of our articles, and found 37 words in matched strings of 6 words or more.)
What to do
There is considerable research on the problem of duplicate publication in the medical literature, as Nature reports about 0.4% of articles in MedLine are probable duplicates. I don’t know how widespread the various kinds of duplication are in the social sciences. I can’t say this is a rampant problem, or that this case is an extreme one — I don’t know.
However, based on what I have read and shown above, I’m satisfied this case is a problem the likes of which we should try to avoid.
First, however, we should consider this kind of problem in the context of the “cycle of publication overproduction,” in which the Academy finds itself. That’s the phrase from a report by Diane Harley and Sophia Krzys Acord, “Peer review in academic promotion and publishing: its meaning, locus, and future,” published by Center for Studies in Higher Education at Berkeley. They write:
…the problems we face in scholarly communication are not about publishing, per se, or the process of peer review in that system. Instead, the problems lie with the current advancement system in a multitude of higher education sectors globally that increasingly demand unrealistic publication requirements of their members in lieu of conducting thorough and context-appropriate institutional peer review, at the center of which should be a close reading and evaluation of a scholar’s overall contributions to a field in the context of each institution’s primary mission.
And one of their first recommendations is: “Encourage scholars to publish peer-reviewed work less frequently and more meaningfully.”
- Journal peer review. A large survey of peer reviewers found that, in the social sciences, about 88% believe ensuring acknowledgement of previous work should be one of the purposes of peer review, but only 59% believe the system is current able to do so. The editors and associations in this case — Criminology (the American Society of Criminology) and Social Problems (the Society for the Study of Social Problems) — or others have an incentive to prevent this waste of their resources. Editors and reviewers must be told when highly similar work exists, obviously, and the current ethics statement for American Sociological Review states that “Significant findings or contributions that have already appeared (or will appear) elsewhere must be clearly identified.” But in one concrete step to strengthen that, we could require that referenced work that has not been published — in press, under review, forthcoming, etc. — be accompanied by access to the papers, so the editors can look for redundancy. As it is, these papers are not available to reviewers or editors for verification.
- Promotion committees and administrators. Counting up articles, and weighting them according to journal prestige or impact factor, is widely practiced in promotion decisions. Really reading the work is much better, but it requires more time, and more specialized skill. The Harley and Acord report has some great systemic recommendations. But in the meantime one thing I might start doing when reviewing cases is to ask myself, “Could any of these articles have been shaved into multiple salami slices?” If they could, but weren’t, let’s give credit for that decision (even if the work wasn’t in a top journal).
- Ethics guidelines. The American Sociological Association’s Code of Ethics prohibits publishing “data or findings that they have previously published elsewhere.” However, the plagiarism provision only mentions “another person’s” work. The “self-plagiarism” (or duplication) aspect of this should be beefed up.
- Shorter articles. Thin slices of salami are not so distasteful when eaten without a full load of bread and condiments. Sociology journals tend to have long articles compared with some other social sciences (psychology, economics, public health). If one of the two articles in this case could have been published as a brief report, with a few references and a clear emphasis on what was new, that might be reasonable. In many cases, the copying is in the theory, literature review, and methods. I know I just said we publish too much as it is, but sometimes shorter articles would help with this problem.
Why would I write this and make it public? I certainly have nothing personal against John Hipp, whom I hardly know. But part of the privilege of having tenure protection is that the fear of offending people shouldn’t prevent me from speaking up on issues I think are important, and I am worked up about this issue. Academia has the unrivaled privilege of policing itself, and we have built a system that runs on trust, believe it or not.
I have done a few promotion reviews since I’ve been tenured, and reviewed many journal articles (more than 70 in the past 7 years). I have evaluated hundreds of job candidates and thousands of graduate student applications. Maybe it’s too much to hope that academia will abandon its addiction to bean-counting in the evaluation of productivity and merit any time soon. But we can increase our sensitivity to the flaws of that approach, and promote the honesty and transparency that are prerequisites for its functioning.
These are some other interesting sites and pages on these issues:
- Andrew Gellman’s blog
- Song From Myself: An Anatomy of Self-Plagiarism, by Patrick M. Scanlon
- Plagiarism and self-plagiarism: What every author should know, by Miguel Roig
- Olaf Storbeck’s collection of posts on the economist Bruno Frey
- The Deja Vu Database of Highly Similar Citations in MedLine (see also this article in PLoS One)