UPDATED with results and data availability as of June 26.
First about peer review in general, and then the crowdsourcing.
Journal peer review is in crisis, and “COVID-19 has made it worse.” The system that relies on unrecognized and unrewarded labor, from a shrinking pool of appropriately employed academics, to police an exponentially increasing number of research papers, is unable to cope. People say review times are increasing, and more and more reviewers are declining invitations to review. The plaintive cry is growing louder: “Who will review our manuscripts?“
I have argued that we will have to sacrifice anonymity for recognition if we want to salvage the system:
I could be wrong about the pros and cons of opening up peer review. The idea has developed much more rapidly in the natural science than in social science (see this by Jessica Polka and colleagues in Nature, 2018; and this by Wolfram and colleagues in Scientometrics in 2020). And it’s growing up alongside the idea of openly reviewing preprints outside the journal system (I was on this team that developed a taxonomy for preprint review). So there are reform efforts to get behind as well.
Anyway, that’s background.
Crowdsourcing
Crowdsourcing
How many times have you asked a friend, colleague, mentor, or random social media crowd how long it took to get reviews from a particular journal? Lets expand and deepen that circle a little.
Despite some spotty, slow reporting, most journals in my disciplines at least don’t report details of their peer review process: How long it takes, how many reviews they get, who the reviewers are, the quality of reviews, the distribution of decisions, and so on (the American Sociological Association provides a little of this information, at least turnaround times and decision rates, for their journals). But this is a case where crowdsourcing could be a great addition to practical knowledge.
Yesterday I built a short form using Qualtrics at the University of Maryland, through which anyone can enter the journal name, dates, decisions, and number of reviews, at each stage of the review process of for a single article. It collects no personal information (not even IP addresses).
The median time to complete the survey is about 3.5 minutes. If a lot of people set aside a couple minutes for each article they publish (or attempt to publish), this could be a very useful ongoing source of information for authors, reviewers, and editors.
These are the results from the first 162 responses as of June 26, just showing the first decision, grouped by journal, to give an idea of what this project will yield.
Median wait time to first decision, by journal (with mean number of reviews)
+------------------------------------------------------------------------------+
| Journal name Median days Mean Reviews N |
|------------------------------------------------------------------------------|
1. | Soc Sci 18 0.5 2 |
2. | Physical Rev Letter 19 1.0 1 |
3. | Nature Human Behavior 26 3.0 1 |
4. | Society & Mental Health 26 2.0 1 |
5. | J Cross-Cultural Psych 31 0.0 1 |
6. | European Societies 35 4.0 1 |
7. | Soc Religion 38 1.0 1 |
8. | Demographic Res 39 3.0 2 |
9. | Soc Rev 42 4.0 1 |
10. | Poetics 48 3.0 1 |
11. | Ams Rev 49 3.0 1 |
12. | Health Affairs 50 2.0 1 |
13. | PLOS One 50 2.0 1 |
14. | Soc Education 51 3.0 1 |
15. | J Marketing 53 4.0 1 |
16. | Socius 55 2.0 4 |
17. | Feminist Criminology 57 3.0 1 |
18. | Work, Aging & Retirement 57 3.0 1 |
19. | Annals Am Academy Pol & Soc Sci 59 2.0 1 |
20. | PNAS 63 1.7 3 |
21. | Soc Focus 64 3.0 1 |
22. | Am Soc Rev 67 2.8 4 |
23. | J Marriage & Family 68 3.0 2 |
24. | Criminology 70 3.0 1 |
25. | Demography 74 2.8 4 |
26. | Am Ed Res J 76 3.0 1 |
27. | Contraception 78 2.0 2 |
28. | Soc Q 78 4.0 1 |
29. | Nature Scientific Data 80 1.0 1 |
30. | Soc Psych Q 80 2.5 2 |
31. | J Res In Music Education 82 3.0 1 |
32. | Rev Int Pol Economy 82 4.0 1 |
33. | Mis Q 86 3.0 1 |
34. | Population Development Rev 88 3.0 1 |
35. | Ethnic & Racial Studies 96 3.0 1 |
36. | J Ethnic & Migration Studies 97 2.0 1 |
37. | Soc Networks 98 2.0 1 |
38. | Studies In Family Planning 98 2.0 1 |
39. | JAMA 106 2.0 1 |
40. | Soc Problems 107 3.0 1 |
41. | Soc Forces 117 4.0 1 |
42. | J Cognitive Psych 118 2.0 1 |
43. | Ed Reser 121 3.0 1 |
44. | Lancet Hiv 121 4.0 1 |
45. | J Documentation 122 1.0 1 |
46. | Gender & Society 123 4.0 2 |
47. | Soc Methods & Res 123 3.0 1 |
48. | European Soc Rev 129 3.0 1 |
49. | Crime & Delinquency 133 1.0 1 |
50. | J Higher Education 138 2.0 1 |
51. | Nationalities Papers 139 2.0 1 |
52. | Frontiers In Microbiology 141 2.0 1 |
53. | Ed Eval & Policy Analysis 145 2.0 1 |
54. | Hum & Soc Scis Com 151 1.0 1 |
55. | Qual Soc 168 3.0 1 |
56. | European J Population 179 2.0 1 |
57. | Am J Soc 202 3.0 4 |
58. | Ieee Trans On Antennas & Propagation 207 2.0 1 |
59. | Soc Sci Res 210 2.5 2 |
60. | Int Migration Rev 211 3.0 1 |
61. | Trans On Human Robot Interactions 235 3.0 1 |
62. | J Family Issues 240 2.5 2 |
63. | Qual Rev 243 2.0 1 |
64. | Rural Soc 288 2.0 1 |
65. | J Transnational Enviro Law 325 2.0 1 |
66. | J Homosexuality 380 2.0 1 |
67. | Soc Inquiry 426 3.0 1 |
68. | Clinical Psych Sci 628 2.0 1 |
|------------------------------------------------------------------------------|
Mean | 122 2.5 1.32 |
+------------------------------------------------------------------------------+
The form also collects information on the subsequent revisions (up to 3), and the final publication date. I suggest people fill it out for the most paper of theirs that has completed the review process. However, you can fill it out as many times as you like. I’ll commit to reporting out the data on a regular basis (with help if I need it), and to making the data public for others to analyze.
Please not this can be used for journals in any discipline.
Maybe some journal editors won’t appreciate this, and claim it’s being used by disgruntled authors to make them look bad (hello, AJS!). So I’ll also commit to linking to any rebuttal data they would like to make public, and remove any data they can disprove — and in that way maybe we’ll encourage more transparency just by doing this.
The Journal Review Time Survey form is available here. I hope you’ll use it, and share it in your networks. Thanks!
Data availability
I have put the raw data, Stata code to clean it up, a codebook, and a final .csv file on the Open Science Framework here: osf.io/swqf6/. This will update automatically when I download new data. There is also a basic report generated each time it updates, which you can read here: osf.io/r8kxb.
Family Inequality 