Our broken peer review system, in one saga

When at last Odysseus returns.

When at last Odysseus returns.

Everybody’s got a story. This is the story of publishing a peer-reviewed journal article called, “The Widening Gender Gap in Opposition to Pornography, 1975–2012.” The paper has now been published, and is available here in preprint, or here if you’re on a campus that subscribes to Social Currents through Sage.

Lucia Lykke, a graduate student in our program, and I began this project in the fall of 2012. We came up with the idea together. I did the coding and she wrote the text. Over the course of two years we sent the paper to four journals – once to Gender & Society, four times to Sex Roles, once to Social Forces, and twice to Social Currents, which finally accepted it in July 2015 and published it online on September 21.*

This story illustrates some endemic problems with our system of scholarly communication, both generally and in the discipline of sociology specifically. I discuss the problems after the story.

THE GIST

The gist of our paper is this: Opposition to pornography has declined in the U.S. since 1975, but faster for men than for women. As a result, the gender gap in opposition – with women more likely to oppose pornography – has widened.

That’s the finding. Our interpretation – which is independent of the veracity of our finding – is that opposition has declined as porn became more ubiquitous, but that women have been slower to drop their opposition because at the same time mainstream porn has become more violent and degrading to women. We see all this reflecting two trends: pornographication (more things in popular culture becoming more pornographic) and post-feminism (less acceptance of speaking up against the sexist nature of popular media, including porn). We could be wrong in our interpretation, and there is no way to test it, but the empirical analysis is pretty straightforward and we should accept it as a description of the trend in attitudes toward pornography. And for doing that empirical work we beg permission to tell you our interpretation.

The analysis is possible because the General Social Survey has, since 1975, asked a large sample of U.S. adults this question about every two years:

Which of these statements comes closest to your feelings about pornography laws: 1. There should be laws against the distribution of pornography whatever the age. 2. There should be laws against the distribution of pornography to persons under 18. 3. There should be no laws forbidding the distribution of pornography.

We tracked the rate at which people selected the first choice versus the others. It’s not very complicated (although we tried it half a dozen other ways, of course). Also of course it’s not perfect – it’s not a great question for today’s social reality, but it’s the only thing like it asked over such a long period. This is what’s great and what’s limiting about the General Social Survey. So, let’s agree to collect better data, and also use this. There, was that so hard?

Here is supporting detail on our particular saga. (We have left the typos from reviewers intact, because it makes us look smarter than they are. And these are selective excerpts to make various points – there was a lot, lot, more.)

Before and after

Just to be clear what the world gained by 13 reviews and two years of waiting, you can compare the abstract at the beginning to the one at the end. This was the original abstract:

In the last several decades pornography in the U.S. has become more mainstream, more accessible, and more phallocentric and degrading to women. Yet research has not addressed how opposition to pornography has changed over the past several decades. Here, we examine opposition to pornography and gender differences in anti-pornography attitudes, using the 1975-2012 General Social Survey. Our findings show that both men’s and women’s opposition to pornography have decreased significantly over the past 40 years, but men’s opposition has declined faster and women remain more opposed to pornography. This is consistent with both the growing normative nature of pornography consumption for men and its increasingly degrading content. We situate these trends within a cultural climate in which women are caught between postfeminism and pornographication – between cultural messages that signal the social acceptability of pornography and compel women’s acquiescence, on the one hand, and the increased presence of pornography many women consider offensive and harmful on the other.

And this was the abstract we ended up with:

In the last several decades pornography in the U.S. has become more mainstream, more accessible, more phallocentric and more degrading to women. Further, consumption of pornography remains a major difference in the sexual experiences of men and women. Yet research has not addressed how opposition to pornography has changed over the this period, despite shifts in the accessibility and visibility of pornography as well as new cultural and legal issues presented by the advent of Internet pornography. We examine gender differences in opposition to pornography from 1975 to 2012, measured by support for legal censorship of pornography in the General Social Survey. Results show that both men’s and women’s opposition to pornography have decreased significantly over the past 40 years, suggesting a cultural shift toward “pornographication” affecting attitudes. However, women remain more opposed to pornography than men, and men’s opposition has declined faster, so the gender gap in opposition to pornography has widened, indicating further divergence of men’s and women’s sexual attitudes over time. This is consistent with the increasingly normative nature of pornography consumption for men, increases over time in men’s actual consumption of pornography, and its increasingly degrading depiction of women.

The regression model we started with in 2013 had logistic regression coefficients showing a decline of .012 per year in the log odds of women favoring laws against the distribution of pornography, versus .022 for men. (That is, the decline has been almost twice as fast for men.) After all we went through with the other variables, we ended up with .012 and .023.

THE SAGA

August 6, 2013: Submitted to Gender & Society

September 23, 2013: Rejected, with four reviews

Reviewer A was concerned about framing, and about the dependent variable.

if one takes this more complex and nuanced definition of postfeminism into account, the theoretical frame of does not work well for the paper … I also thought that the authors could have gone further in discussing broader cultural changes in sexuality in the media, especially the increasing sexualization and pornification in advertising and the media. …

an analysis of a GSS question concerning laws regarding the restriction of pornography, seem limited. In particular, that GSS question does not seem to get at the historical changes that have occurred in pornography distribution and consumption given its widespread internet usage.

Reviewer B was all about framing:

[I] appreciate your analysis of anti-pornography research and the effects of post-feminism on attitudes towards pornography … [but] I think the literature review needs to spend at least some time outlining feminist pro-pornography arguments. …

doesn’t it make sense to incorporate a discussion of the history of pornography regulation since the 1970s in the U.S? [… and …] While you bring up race in the analysis of your data, the literature review is surprisingly devoid of anything having to do with pornographic representations of gender and race.

Reviewer C thought we should have included a content analysis of pornography over time – done a different study, that is — and framed it differently:

Pornography needs to be defined … Cost, images and rejection of feminist view would clearly support a content analysis on pornography … The provided discussion of pornographication seems to more support the use of images and actual study of pornography, more so than people’s attitudes toward it … more justification to the existing literature needs to be added … Some legal gender studies should be included here … Gender is not one sided and the author should consider adding some agency to [men’s] role in the study and discussion.

Reviewer D concluded that the data weren’t good enough to support our interpretation:

The author, however, does not empirically demonstrate that the found decline in opposition is the result of either postfeminism or pornographication. … The General Social Survey is convenient, easy to access, and quick to run. This, however, does not necessarily make for good empirical evidence. … If the author wanted to investigate postfeminism and pornagraphication and the relationship to pornography, a much more nuanced empirical study would have needed to have been designed.

In a world with limited space for publishing research – which is not our world – this would be a good reason to reject the article.

October 7, 2013 (approximate): Submitted to Sex Roles

October 9, 2013: Returned by the editor

The editor, Irene Frieze, returned the paper almost immediately, saying: “major revisions are needed before we can move ahead in the review process.”

Some of what she asked for reflects the competitive climate of contemporary academic journals. For example, she asked us to pad the journal’s citation count: “If possible, either in this section or later in the Introduction, note how your work builds on other studies published in our journal.”

And she tried to make the journal seem more international:

Explain why your study is important to readers from many countries with a sentence or two. … Note what country each empirical study you cite was done in and explain how any cited studies done in other countries are relevant in understanding your sample.

She also asked for what appear to be standard requirements for the journal:

Add demographic information about the sample and explain more about how they were recruited. Add a table showing the demographic characteristics of the women as compared to the men in the sample in different time periods. … Add correlations computed separately for women and men as well.

And, the dreaded memo requirement: “Assuming you do wish to submit a revision, I would need a revised manuscript and a detailed list outlining the changes you have made in response to these comments.”

November 9, 2013: Resubmitted to Sex Roles, first revision

February 17, 2014: Revise and resubmit, based on one review (“major revisions”)

The reviewer had trouble with our statistical presentation:

I see that on Table 2, the difference between the women’s and men’s regression effect for year shows both women’s and men’s significant (-.012 and -.022). This suggests that for both female and male respondents the year is significant, but it doesn’t show statistically that men’s decline in opposition is steeper than is women’s. Where is the statistic showing a significant difference in slope? [The table had a superscript b next to the men’s coefficient, with the note, “Gender difference significant at p<.05.” Although we didn’t provide the details, that test came from a separate, “fully-interacted” model in which every variable is allowed to have a separate effect by gender.]

This reviewer – who stuck with this complaint for three rounds – also had trouble with the smallness of the coefficients:

Although the coefficient is twice as large for year among men than among women, it’s a very small percentage. With such a large sample size, almost anything will be significant. I’d like to see an effect size statistic.

She might have been confused because the variable here is “year” – a continuous variable ranging from 0 in 1975 to 37 in 2012, so the coefficient reflects the size of the average one-year change, which makes it look “very small.”

A common problem for authors responding to reviewers is the simultaneous demands for less and more. Sometimes that’s good – a healthy revision process. Here is a funny example of that: “There seems to be a much longer introduction than is needed for the findings, especially since what would be interesting to me is omitted.”

However she grasped the concise nature of the findings, which she somehow took as a weakness:

I would like to see how each of these control variables interacts with the changes over years. I believe that analysis is possible using time series analyses. The reader is left with only a few main conclusions: both men and women indicate less opposition over time to pornography, and that men’s opposition declines more than female’s, and men show less opposition to pornography control overall.

Exactly. Oh well.

May 17, 2014: Resubmitted to Sex Roles, second revision

July 8, 2014: Revise and resubmit, with two reviews

The editor now told us: “We were able to find a second reviewer, this time. We won’t continue to add new reviewers for additional drafts.” (This promise, sadly, did not hold.)

The dependent variable – that three-response question about laws regulating pornography – caused continuing consternation. The editor wrote:

none of us feels that the combining of the three categories of responses for the pornography acceptance variable is appropriate. You either need to omit one of the 3 categories from the analysis, or do something like a discriminant analysis to look at differences in those responding to each of the three categories.

And then this bad signal that the editor and reviewers did not understand the basic structure of the analysis:

Another issue that all of us agree on is that you have failed to provide statistical evidence supporting your assertion of evidence of a linear trend in support over time. Either do a real trend analysis, for women and men separately, or compare the data over several specific years using something like ANOVA by year and gender. This would also allow you to see if these is really the interaction you assert is present.

As you can see in the final paper – which was the case in this revision as well – we did a “real trend analysis, for women and men separately.”

We tried to make this as clear as possible, writing in the paper:

We use logistic regression models to test for differences on this measure between men and women across the 23 administrations of the GSS since 1975. We test time effects with a continuous variable for year, which ranges from 0 in 1975 to 37 in 2012. This coding allows for an intuitive interpretation of the intercept and produces coefficients equal to the predicted change in the odds of opposing pornography associated with a one year change in the independent variable (non-linear specifications did not improve the model fit). … The first model combines men and women, while models 2 and 3 analyze men and women separately, after tests showed differences in the coefficients by gender on six of the variables (marked with superscript ‘b’). … Comparison of Model 2 and Model 3 confirms that the decline in opposition to pornography has been more pronounced for men than for women, as the coefficient for the year variable is almost twice as large.

We thought that Reviewer 1, back from the previous round, was doubling down on misunderstanding what we did, and the editor thought this as well. The reviewer wrote: “I don’t agree that the years need collapsing in the analyses. I believe it is better to see the linear trend. Also, I don’t like to see data left out, in this case data from the individual years.”

In fact, we found out in the next round of reviews we found the she meant this is a disagreement with the editor! (“The authors misread my statement about collapsing the years. I was disagreeing with the editor who suggested collapsing the years. I did not suggest myself that the years should be collapsed. I agree that the years should not be collapsed. It’s not me who misread the paper, it’s the authors who misread my statement.”)

That said, she still did not grasp the analysis:

You state that ‘This coding allows for an intuitive interpretation of the intercept and produces coefficients equal to the predicted change in the odds of opposing pornography associated with a one year change in the indepenjdent variable.’ In the results section, please describe how your data fit an ‘intuitive’ interpretation and how the coefficients that are produced explain the one year change. There is a disconnect for me from this statement and the description of the data.

And she added:

Please carefully describe the statistical analysis and statistical findings that describe the difference between the declines in opposition for women vs. men. Is the beta for gender .78 and for year -.02, and how did you test for the difference in betas of -.01 vs. -.02? Mention the test you used to assess this. This doesn’t seem like much of a difference in slope. That one is twice as large as the other is fairly meaningless when it is .01 vs. .02.

And added again later: “P. 22, agvain when you say a coefficient for the year variables is “amost twice as large,” you are talking about .01 vs .02.”

Sigh.

The editor and Reviewer 1 had a long-running dispute about how to handle all of our control variables. The editor was sticking to the policy that we needed a table showing complete correlations of all variables separately by gender. And a discussion of every variable, with references, justifying its inclusion. The editor said in the first round:

You also need to explain each of the control variables you include in your regressions in the Introduction. Add at least a sentence for each variable explaining why it is important to the issues you are testing.

In response, we included a long section beginning with, “Various social and demographic characteristics are associated with pornography use and attitudes toward pornography, and we account for these characteristics in our empirical analysis below.”

But then Reviewer 1 said of that passage: “Much of the material in “Attitudes Toward Pornography” is not relevant. … Gender and gender differences are what you are studying.”

And in response to our gigantic correlation table of all variables separately by gender, Reviewer wrote: “I … strongly recommend deletion of Table 3. This is not a study of the correlates of attitudes toward pornography, and the intercorrelations of all the control variables are outside the range of your focus.”

Never mind.

Reviewer 2, the new reviewer, had some reasonable questions and suggestions. For example, s/he recommended analyzing the outcome with a multinomial logistic regresstion, which we did but it didn’t matter; and controlling for pornography consumption (“watched an x-rated movie in the past year”), which we did and it didn’t matter (in fact, basically none of the control variables affect the basic story much, but reviewers have a hard time believing this). S/he also had lots of objections to how we characterized various feminist authors and terms in the framing, and really didn’t like “pornographication” as a term, listing as a “major” objection:

the term ‘pornographication’ is problematic and should be removed from the paper in favor of a more academic description of increased access to sexualized media.

September 10, 2014 (approximate): Resubmitted to Sex Roles, third revision

October 11, 2014: Revise and resubmit, with one review

The editor now informed us that one reviewer just recommended rejecting the paper because we didn’t address her concerns, while the other called for “major revisions.”

Given this type of feedback, I would normally reject a paper already in its third revision. However, I would like to offer one more opportunity for you to make the requested changes. If you do resubmit, I may seek new reviewers and essentially begin the review process anew, unless it is clear that my earlier concerns are fully addressed.

Despite three drafts and as many memos, the editor still did not seem to understand that our outcome variable was a single question with three options. She wrote:

One of my basic requests has been that you consider the question about exposure of pornography to those under 18 as a separate dependent variable, or omit this entirely from the study. Conceptually, I feel this is quite different from the other two survey items and cannot be combined with them. This will require major changes in the analysis and rationale for predictions relating to each of these measures.

The reviewer, however, disagreed, voicing approval for our choice. The editor clarified, “If my requests conflict with those of the reviewer, it is my requests you need to follow, not those of the reviewer.”

They had no trouble agreeing, however, that they did not understand the linear time trend we were testing: “As the reviewer explains, we do need a clearer discussion of how the linear trend is being tested.”

Reviewer 1 wrote:

Regarding the analysis of the time trend, although the authors state [in the memo] that the starring of the coefficients on Table 4 demonstrate a significant linear trend, it was not apparent to the editor and reviewers. As one of the main points of the study, it should be made very obvious that there is a significant linear trend via statistics. If this means being more explicit in the text of the results section, it would be important to do. If there’s this much confusion, the statistical analysis needs clarification.

You can look at the table in the final publication for yourself to see if this remains unclear. And then the reviewer added:

As I previously mentioned, though significant, a change of -.02 vs .-.01 is not substantial. Thus, the authors should refrain from concluding one is twice as large as the other.

We decided to take our business elsewhere rather than submit another revision.

November 4, 2014: Submitted to Social Forces

December 29, 2014: Rejected, with two reviews

Reviewer 1 only had concerns about framing, such as, “expand their discussion of the broader cultural changes in sexuality in the culture,” and discuss “changes in gay and lesbian identities and visibility during this period.”

Reviewer 2 simply thought we couldn’t answer the questions we posed with the data we had:

The paper is motivated by a largely assumed cultural ‘pornographication’ process linked to post-feminism. Neither concept seems well-suited to explain public opinion formation or change, and greater specificity about these concepts would likely outstrip the operational capacity of the GSS to model how gender and sexuality attitudes may influence shifts in beliefs about pornography.

There were some other technical issues about specific variables that aren’t very important. Again, this is very reasonable basis for making the ridiculous judgment forced by the system of publishing in the limited pages of a print journal.

January 16, 2015: Submitted to Social Currents

April 9, 2015: Revise and resubmit, based on three reviews

The editors, Toni Calasanti and Vincent Roscigno, wrote:

While stated differently in each case, the overriding sentiment across the reviewers is that the paper needs better framing. … the potential contribution of this study is not realized because the theoretical framework is lacking, limiting your ability to discuss the implications of your findings.

Reviewer 1 wanted the “post-feminism” discussion put back in the front: “It’s not until the conclusion of the manuscript that we learn about a potential contribution to ‘postfeminism’ and current work there.”

Reviewer 1 also attempted to lead us into a common trap. S/he wrote:

The hypotheses don’t necessarily derive from a particular theory in sociology or test a specific argument about gender, public opinion theories, and pornography per se. Rather, the project is descriptive (divergence of male/female support for legal control, rate of change over time, etc.). That isn’t fatal. But a project that makes a more direct connection to advancing current theoretical work in feminism and sexuality studies, or current theorizing about the importance of public opinion and values about pornography, would strengthen the overall contribution of this research.

Making the paper more theoretical is not a bad suggestion, but in this context – since the data are so limited – it’s a sure setup for a future reviewer to complain that you have asked questions you can’t sufficiently answer with your data.

The three reviewers’ other concerns by this point were quite familiar to us. For example, “perhaps a line or two to strengthen the validity of measure could be added based on some of the studies cited.” And a worry about about collapsing the dependent variable into two categories. And the need to acknowledge debates within feminism about the meaning of “pornographication.” We dutifully beefed up, clarified, and strengthened. And wrote a memo.

May 20, 2015: Resubmitted to Social Currents, first revision

July 18, 2015: Accepted

WHAT’S WRONG HERE

Some of the problems apparent in this story are common to sociology, some are more general.

Sociologists care way too much about framing. Most (or all) of the reviewers were sociologists, and most of what they suggested, complained about, or objected was about the way the paper was “framed,” that is, how we establish the importance of the question and interpret the results. Of course framing is important – it’s why you’re asking your question, and why readers should care (see Mark Granovetter’s note on the rejected version of “the Strength of Weak Ties”). But it takes on elevated importance when we’re scrapping over limited slots in academic journals, so that to get published you have to successfully “frame” your paper as more important than some other poor slob’s.

The journal system gets in the way. When journals reject you they report the low percentage of papers they accept. This is supposed to make the rejected authors feel better, but it also shows the gross inefficiency of the system: why should you bounce from journal to journal with low acceptance rates – in our case, asking our colleagues to write 13 reviews – instead of being vetted once by a centralized system with reviewers who work to a common standard? The answer is because that’s the way they did it in the Dark Ages, when physically printing research papers at high cost was the only way of distributing scholarly output.

The system is slow. As a result of these and other systemic problems, we do a terrible job of advancing knowledge. From the time of our first submission to the publication date was 776 days. For 281 of those days it was in our hands, but for the other 495 days it was in the hands of editors, reviewers, and the publisher. Despite responding to 13 reviews, with a lot of tinkering, the basic result did not change from our first submission in August 2013 to our last submission in May 2015. The new knowledge was all created two years before it was published.

The system is arbitrary I don’t want to make Social Currents look bad here, with the implication that they are a lower quality journal because they published something rejected by three journals before. After all, Granovetter’s paper was rejected by American Sociological Review before getting 35,000 citations as an American Journal of Sociology paper. I also like the example of Liana Sayer and Suzanne Bianchi’s paper on economic independence and divorce, which was rejected by the Journal of Marriage and Family, the flagship journal of the National Council on Family Relations (NCFR), before promptly winning NCFR’s best-paper award after it was published in the Journal of Family Issues. That is, one small group of reviewers deemed it unpublishable in a top journal, and the next declared it the best article of the year. That’s a very wide spread. The arbitrariness of the review system we have now creates cases like this – and who knows how many others. It’s not a systemic problem that Sex Roles has a reviewer that won’t let you say .02 is twice as large as .01. The problem is that could happen anywhere – and cost people their careers – at the same time that bad stuff gets through for arbitrary (or pernicious) reasons. There is too much noise in the current peer-review system to trust it for quality control.

WHAT TO DO

Consider an alternative system, for example, in which the paper – having passed a very low bar of basic quality – had been published after the first set of reviews and then subjected to post-publication review and discussion in the field. Another alternative is publishing it before any formal review process, and allowing post-publication review to do the whole vetting process.

Models exist. Sociology doesn’t have a central working paper system, but there are smaller systems. In my neck of the woods, the California Center for Population Research has a working paper archive, which houses papers from six population centers. Math types have arXiv, which has more than a million papers, with each new one “reviewed by expert moderators to verify that they are topical and refereeable scientific contributions that follow accepted standards of scholarly communication.” They also use a system of member endorsement to cut down on junk submissions. If papers are subsequently published the arXiv version is updated to link to the published version. Sociology should make something like this.

Another step in the right direction is rapid-response, open-access peer-review, with quick up-or-down decisions. In sociology this includes Sociological Science, run by an independent team and supported by author fees (often paid by university libraries or grants); and Socious, run by the American Sociological Association and subsidized by the for-profit publisher Sage in an attempt to pacify open-access advocates. These work more or less like PLOS One, which “accepts scientifically rigorous research, regardless of novelty.”

I’m happy to publish in such outlets, but many of us worry about the career implications for our students who risk having their CVs seen as sketchy by old-fashioned types. We need them to be institutionalized.

In the meantime, those of us in position to conduct peer review can do our part to be better reviewers (see this excellent advice). And we can make explicit decisions about which journals we will review for. The system runs off our discretionary contributions, and we shape it through our actions. That argument is for a separate post.

* We did the research together — and Lucia did most of the work — but blame me for the content of this post.

32 Comments

Filed under Me @ work

32 responses to “Our broken peer review system, in one saga

  1. Bill Bielby

    Ugh! My most recent coauthored peer-reviewed article proceeded on an almost identical timeline. On the .02 versus .01, did you offer a graphical representation of the trend lines across the entire range of years?

    Like

  2. Deirdre

    Yup…we all have a story or two or three….Thanks for posting this!

    Liked by 1 person

  3. Reblogged this on Dr Anna Tarrant and commented:
    An really though examination of the problems with the peer review system here. Worth the read – this posts builds up the frustrations inherent in this approach for authors and not to mention the time it takes to go through this back and forthing!

    Like

  4. RS

    Sounds Kafkaesque and totally disheartening. I cannot parse the theoretical implications but the finding is clearly significant and should have been published immediately. Glad I went into research consulting rather than academia.

    Liked by 1 person

  5. Did you remove the citations to _Sex Roles_ after you decided to submit somewhere else?

    Liked by 3 people

  6. A very frustrating story indeed, and I agree with some of your conclusions. However, I tend to blame the silly ways reviewers review less than the editorial gatekeepers. Thinking about the alternatives you outline (various forms of pre- and peri-review publication), I am quite concerned that these would encourage crappy and politically-motivated research to be “published” so it can be used for political purposes; then the reviews posted after publication are of no value for addressing the public-sphere uptake. (No big mystery what I’m talking about here.)

    I think much would be helped if reviewers treated reviews less like seminar discussions, where they are expected to offer interesting tidbits, and more like grading papers. Assess the paper’s goal, evidence, and claims internally (is it successful?) and its contribution externally (is it important?). Don’t suggest that the author(s) do a different study, dramatically change the framing, or otherwise engage in a dialogue as you might in a seminar or writing group.

    Liked by 1 person

    • Well, The crappy political stuff is published now as reports and briefing papers from think tanks etc. i’m not sure they would game political advantage from publishing them in a non-reviewed archive.

      Like

      • I think they’d love the opportunity to post un-reviewed junk and call it a “published article” in a “sociology journal.” Basically I think peer review is a perfectly fine system as long as (a) enough people do their duty and review, and (b) when people review they do a good job of it.

        Like

  7. dp

    I am posting this anonymously because one of the papers is still under review. But I wanted to add another bit of frustration with sociology journals. I have had two papers rejected by sociology journals for elementary mistakes in understanding statistics.
    The papers involved Time-series–cross-section analysis, and the reviewers clearly did not grasp basic issues related to stationarity and co-integration. In both papers we presented a series of tests showing the necessity of the models we used, and got back as major issues with the paper the fact that we hadn’t used precisely the now outdated models that our tests indicated we should avoid. There were even more elementary mistakes, with a reviewer not understanding the distinction between levels and changes. In one of the cases, we contacted the editors about the mistake and heard nothing back. On the other, we didn’t bother.
    Of the two papers, one of them ended up in a much higher impact factor political science journal than our original submission, while the other should come out in a similar type of publication. And while the higher impact journal was a nice venue, this was still very problematic to me for a couple of reasons. I am an assistant professor at a sociology department. Wasting so much time like this is obviously bad (major issue), and when promotion review time comes I will probably have to waste a couple of sentences on why my political science publications matter to me as a sociologist (minor issue).

    And, for the record, the political science review was so much better (though I understand it may have been just luck of the draw). Essentially, the reviewer disagreed with the causal mechanism we claimed, but said that since we had found an effect no one else had, and we had run models according to all of the state of the art specifications, the paper should be published so that others may follow up and try to confirm it or disprove it.

    Like

  8. Anon Psych

    Being anon in case I submit there again, but I’m actually heartened to see your treatment by the editor at Sex Roles was as frustrating as mine. I thought she just didn’t like me…

    Like

  9. Reblogged this on Installing (Social) Order and commented:
    We’ve discussed the peer-review system in science a number of times; this time, as a saga. I’ve had this experience many, many times. If you don’t wish to review the saga, the final section has some alternatives to business-as-usual in the peer-review system.

    Like

  10. I’m not saying that this is a solution, but there is the matter of precisely “who” owns those review materials. In a relatively recent paper, my colleague and I (writing about reflexivity) included some reviewer comments in our paper (it is an unorthodox paper overall, so do keep that in mind if you decide to take a peak at it).

    Here is a blog post about it (although the paper is explicitly not meant to be a critique of the peer-review process): http://installingorder.org/2013/10/03/writing-reflexively-lessons-from-ant/

    Also, if you cannot get over the paywall, the paper is called “Beware of Allies”: https://pennstate.academia.edu/NicholasRowland

    Like

  11. Chris W

    I suspect that the overemphasis on “framing” encountered in the review process is proportionate with the growing number of submissions that reviewers are asked to review that are primarily data-driven research studies. By data-driven I mean studies that are inspired by the availability of and/or familiarity with certain data sets, as opposed to studies inspired by genuine research problems, theory and hypotheses, and/or controversies in the research literature — in other words, what used to be considered the first few steps of the scientific method. Given the mandate “publish or perish,” I can understand why folks would work backwards or skip a few steps in the research process. But the result is that a lot of the submissions we’re asked to review are disconnected from theory or ongoing debates in the literature, and thus need more framing — in some cases, some framing, any framing. Just because you have data and you know how to manipulate it in a reasonable way, doesn’t mean you have a fully developed study.

    As for the review process, I’d like to see more transparency. E.g., what % of potential reviewers have accepted or rejected requests to review? How are reviewers found? What % are grad students, NTTs, junior faculty, senior faculty, etc. What % have reviewed at the journal in question before? How much experience do the reviewers have at reviewing in general? I remember once an editor asked me to suggest names of potential reviewers for my submission. At the time, I thought that that was problematic. But is it any more problematic than a review system that lacks transparency?

    Like

  12. Pratik

    Thanks for this brave post. Many, including me, share your frustration with the current competitive review system. My paper was reviewed by a person who clearly lacked the understanding of the method used in the study, and asked me to follow an inferior method for the purpose. One journal reviewers gave me comments in one direction, the other journal reviewers gave me completely opposite comments. Finally both the journals rejected the paper citing lack of adequate contribution as a reason, which is vague and unjust ground for rejecting something.

    I am feeling like sending your post to the journal editor.

    Regards,

    Like

  13. Pingback: Peer Pressure | Iranianredneck's Weblog

  14. Pingback: some thoughts on institutional logics, in three parts | orgtheory.net

  15. Pingback: Shine a light on journal self-citation inflation | Family Inequality

  16. Yue

    Thanks, Philip, for this post. I recently got a rejection after an R&R in a flagship journal. We waited for seven months, and got two splitting reviews and an RR decision. We took about two months to revise and submit it (mainly revising the framing). After almost three months, the paper was reviewed again only by the reviewer who did not like our paper in the first place…We got rejection…Maybe many of us likely have many similar frustrating experiences to share. The review process is getting longer but the job market is increasingly competitive. Feel very stressed to be a junior person. But as a sociologist, I know how difficult it is to change the structure.

    Like

  17. jrg

    Great to see the sausage being made!

    A couple of thoughts:

    (1) I like the idea of posting reviews, as well as responses to reviews on-line, especially after a paper has been accepted. Anyone who reads a paper should be interested in what others thought and in how the author responded. +100 for science.

    (2) My experience with journals like Plos ONE is that they are also interested in framing and in theoretical interest — so after several revise and resubmits one ends up spending a long time at such a journal as well.

    (3) The main lesson I drew from this story is that the authors made a huge mistake in re-submitting the desk reject from _Sex Roles_ . Why did they get sucked into doing 3+ revisions for a set of reviewers (and editor) that were so obviously negatively disposed to their paper? Hindsight is 20-20, but from the editor’s first response, it seems pretty clear that the treatment of this paper was not going to go the way the author’s wanted. What could be less inviting than “major revisions are needed before we can move ahead in the review process.”

    (4) The paper seems decent and it was in the end published. So this tale seems less a condemnation of scholarly publishing as it is than an unfortunate eddy of re-re-re-submission, before finally seeing the light and moving elsewhere.

    (5) Most of publishing is finding the right reviewers, not writing the right paper. My advice is always: believe in what you’re doing, and keep submitting until you find the right reviewers. Then listen to them. Otherwise one is just aiming at a moving target.

    Fiat lux!

    Like

  18. Logan

    Very thoughtful article. However, one line in particular stood out to me, seemingly unrelated to the main point:

    >Our interpretation – which is independent of the veracity of our finding

    Is it really okay to make an interpretation that doesn’t depend on the underlying data? You’re saying that, whether the data is true or not, the interpretation would be the same?

    Science consists of using data to make interpretations, not the other way around. The interpretation should always be dependent on the data. If there’s no related data, then it’s simply a baseless statement and shouldn’t be considered science at all.

    Interesting article and one that is very insightful into the publishing process. However, statements like the above make sociology and other related “sciences” difficult to take seriously.

    Like

  19. Pingback: Our broken peer review system, in one saga | Family Inequality | the neuron club

  20. Kerrick Staley

    Academia.edu is working to speed up the publishing process. They have an online tool called Sessions that’s a lot like Google Docs’s comment feature, except for PDFs. It allows very rapid back-and-forth during the review process. Then, when you actually publish, it’s just a lightweight process where you get other researchers (not necessarily the ones who participated in feedback) to endorse your paper. So the feedback process doesn’t gate actually publishing. (Although I think they’re still defining how publishing works, and you can’t actually publish on their platform yet). http://www.academia.edu/

    Like

  21. Dave Higgins

    The misunderstanding of reviewers can act as a help in encouraging better description of the work, but boy, did you have a tough row to hoe!

    Like

  22. Pingback: The broken review system | Memoirs of a SLACer

  23. Pingback: Journal self-citation practices revealed | Family Inequality

  24. Pingback: Year-end report | Family Inequality

  25. Pingback: Posting my peer reviews on Publons and a few notes about reviewing | Andrew Wheeler

  26. Pingback: How broken is our system (hit me with that figure again edition) | Family Inequality

  27. Pingback: Morning Ed: Science {2016.09.13.T} | Ordinary Times

Comments welcome (may be moderated)

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s