Journal peer review times, crowdsourced edition (with your help)

A crowd of crowdsourcing stock images.

UPDATED with results and data availability as of June 26.


First about peer review in general, and then the crowdsourcing.

Journal peer review is in crisis, and “COVID-19 has made it worse.” The system that relies on unrecognized and unrewarded labor, from a shrinking pool of appropriately employed academics, to police an exponentially increasing number of research papers, is unable to cope. People say review times are increasing, and more and more reviewers are declining invitations to review. The plaintive cry is growing louder: “Who will review our manuscripts?

I have argued that we will have to sacrifice anonymity for recognition if we want to salvage the system:

I could be wrong about the pros and cons of opening up peer review. The idea has developed much more rapidly in the natural science than in social science (see this by Jessica Polka and colleagues in Nature, 2018; and this by Wolfram and colleagues in Scientometrics in 2020). And it’s growing up alongside the idea of openly reviewing preprints outside the journal system (I was on this team that developed a taxonomy for preprint review). So there are reform efforts to get behind as well.

Anyway, that’s background.


Crowdsourcing

Crowdsourcing

How many times have you asked a friend, colleague, mentor, or random social media crowd how long it took to get reviews from a particular journal? Lets expand and deepen that circle a little.

Despite some spotty, slow reporting, most journals in my disciplines at least don’t report details of their peer review process: How long it takes, how many reviews they get, who the reviewers are, the quality of reviews, the distribution of decisions, and so on (the American Sociological Association provides a little of this information, at least turnaround times and decision rates, for their journals). But this is a case where crowdsourcing could be a great addition to practical knowledge.

Yesterday I built a short form using Qualtrics at the University of Maryland, through which anyone can enter the journal name, dates, decisions, and number of reviews, at each stage of the review process of for a single article. It collects no personal information (not even IP addresses).

The median time to complete the survey is about 3.5 minutes. If a lot of people set aside a couple minutes for each article they publish (or attempt to publish), this could be a very useful ongoing source of information for authors, reviewers, and editors.

These are the results from the first 162 responses as of June 26, just showing the first decision, grouped by journal, to give an idea of what this project will yield.

  Median wait time to first decision, by journal (with mean number of reviews)
      +------------------------------------------------------------------------------+
      |                         Journal name   Median days   Mean Reviews          N |
      |------------------------------------------------------------------------------|
   1. |                              Soc Sci            18            0.5          2 |
   2. |                  Physical Rev Letter            19            1.0          1 |
   3. |                Nature Human Behavior            26            3.0          1 |
   4. |              Society & Mental Health            26            2.0          1 |
   5. |               J Cross-Cultural Psych            31            0.0          1 |
   6. |                   European Societies            35            4.0          1 |
   7. |                         Soc Religion            38            1.0          1 |
   8. |                      Demographic Res            39            3.0          2 |
   9. |                              Soc Rev            42            4.0          1 |
  10. |                              Poetics            48            3.0          1 |
  11. |                              Ams Rev            49            3.0          1 |
  12. |                       Health Affairs            50            2.0          1 |
  13. |                             PLOS One            50            2.0          1 |
  14. |                        Soc Education            51            3.0          1 |
  15. |                          J Marketing            53            4.0          1 |
  16. |                               Socius            55            2.0          4 |
  17. |                 Feminist Criminology            57            3.0          1 |
  18. |             Work, Aging & Retirement            57            3.0          1 |
  19. |      Annals Am Academy Pol & Soc Sci            59            2.0          1 |
  20. |                                 PNAS            63            1.7          3 |
  21. |                            Soc Focus            64            3.0          1 |
  22. |                           Am Soc Rev            67            2.8          4 |
  23. |                  J Marriage & Family            68            3.0          2 |
  24. |                          Criminology            70            3.0          1 |
  25. |                           Demography            74            2.8          4 |
  26. |                          Am Ed Res J            76            3.0          1 |
  27. |                        Contraception            78            2.0          2 |
  28. |                                Soc Q            78            4.0          1 |
  29. |               Nature Scientific Data            80            1.0          1 |
  30. |                          Soc Psych Q            80            2.5          2 |
  31. |             J Res In Music Education            82            3.0          1 |
  32. |                  Rev Int Pol Economy            82            4.0          1 |
  33. |                                Mis Q            86            3.0          1 |
  34. |           Population Development Rev            88            3.0          1 |
  35. |              Ethnic & Racial Studies            96            3.0          1 |
  36. |         J Ethnic & Migration Studies            97            2.0          1 |
  37. |                         Soc Networks            98            2.0          1 |
  38. |           Studies In Family Planning            98            2.0          1 |
  39. |                                 JAMA           106            2.0          1 |
  40. |                         Soc Problems           107            3.0          1 |
  41. |                           Soc Forces           117            4.0          1 |
  42. |                    J Cognitive Psych           118            2.0          1 |
  43. |                             Ed Reser           121            3.0          1 |
  44. |                           Lancet Hiv           121            4.0          1 |
  45. |                      J Documentation           122            1.0          1 |
  46. |                     Gender & Society           123            4.0          2 |
  47. |                    Soc Methods & Res           123            3.0          1 |
  48. |                     European Soc Rev           129            3.0          1 |
  49. |                  Crime & Delinquency           133            1.0          1 |
  50. |                   J Higher Education           138            2.0          1 |
  51. |                 Nationalities Papers           139            2.0          1 |
  52. |            Frontiers In Microbiology           141            2.0          1 |
  53. |            Ed Eval & Policy Analysis           145            2.0          1 |
  54. |                   Hum & Soc Scis Com           151            1.0          1 |
  55. |                             Qual Soc           168            3.0          1 |
  56. |                European J Population           179            2.0          1 |
  57. |                             Am J Soc           202            3.0          4 |
  58. | Ieee Trans On Antennas & Propagation           207            2.0          1 |
  59. |                          Soc Sci Res           210            2.5          2 |
  60. |                    Int Migration Rev           211            3.0          1 |
  61. |    Trans On Human Robot Interactions           235            3.0          1 |
  62. |                      J Family Issues           240            2.5          2 |
  63. |                             Qual Rev           243            2.0          1 |
  64. |                            Rural Soc           288            2.0          1 |
  65. |           J Transnational Enviro Law           325            2.0          1 |
  66. |                      J Homosexuality           380            2.0          1 |
  67. |                          Soc Inquiry           426            3.0          1 |
  68. |                   Clinical Psych Sci           628            2.0          1 |
      |------------------------------------------------------------------------------|
 Mean |                                                122            2.5       1.32 |
      +------------------------------------------------------------------------------+

The form also collects information on the subsequent revisions (up to 3), and the final publication date. I suggest people fill it out for the most paper of theirs that has completed the review process. However, you can fill it out as many times as you like. I’ll commit to reporting out the data on a regular basis (with help if I need it), and to making the data public for others to analyze.

Please not this can be used for journals in any discipline.

Maybe some journal editors won’t appreciate this, and claim it’s being used by disgruntled authors to make them look bad (hello, AJS!). So I’ll also commit to linking to any rebuttal data they would like to make public, and remove any data they can disprove — and in that way maybe we’ll encourage more transparency just by doing this.

The Journal Review Time Survey form is available here. I hope you’ll use it, and share it in your networks. Thanks!


Data availability

I have put the raw data, Stata code to clean it up, a codebook, and a final .csv file on the Open Science Framework here: osf.io/swqf6/. This will update automatically when I download new data. There is also a basic report generated each time it updates, which you can read here: osf.io/r8kxb.

Comments welcome (may be moderated)

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s