Tag Archives: socarxiv

Why we need open science in demography, and how we can make it happen

“Why we need open science in demography, and how we can make it happen” is the title of a talk I gave at the Max Planck Institute for Demographic Research yesterday, as part of an open science workshop they hosted in Rostock, Germany. (The talk was not nearly as definitive as the title.)

The other (excellent) keynote was by Monica Alexander. I posted the slides from my talk here. There should be a video available later. The organizing committee for the event is working to raise the prominence of open science discussions at the Institute, and consider practices and policies they might adopt. We had a great meeting.

As an aside, I also got to hear an excellent tutorial by E. F. Haghish, who has written Markdoc, a “literate programming” (markdown) package for Stata, which is very cool. These are his slides.

rostock talk 2rostock group shot

Leave a comment

Filed under Me @ work

The Coming Divorce Decline, Socius edition

“The Coming Divorce Decline, ” which I first posted a year ago, has now been published by the journal Socius.  Three thousand people have downloaded it from SocArXiv, I presented it at the Population Association, and it’s been widely reported (media reports), but now it’s also “peer reviewed.” Since Socius is open access, I posted their PDF on SocArXiv, and now that version appears first at the same DOI or web address (paper), while the former editions are also available.

Improvement: Last time I posted about it here I had a crude measure of divorce risk with one point each for various risk factors. For the new version I fixed it up, using a divorce prediction model for people married less than 10 years in 2017 to generate a set of divorce probabilities that I apply to the newly-wed women from 2008 to 2017:

…the coefficients from this model are applied to newly married women from 2008 to 2017 to generate a predicted divorce probability based on 2017 effects. The analysis asks what proportion of the newly married women would divorce in each of their first 10 years of marriage if 2017 divorce propensities prevailed and their characteristics did not change.

The result looks like this, showing the annual probability falling from almost 2.7% to less than 2.4%:

divprobnewlyweds

The fact that this predicted probability is falling is the (now improved) basis for my prediction that divorce rates will continue to decline in the coming years: the people marrying now have fewer risk factors. (The data and code for all this is up, too).


Prediction aside: The short description of study preregistration is “specifying your plan in advance, before you gather data.” You do this with a time-stamped report so readers know you’re not rejiggering the results after you collect data to make it look like you were right all along. This doesn’t always make sense with secondary data because the data is already collected before we get there. However, in this case I was making predictions about future data not yet released. So the first version of this paper, posted last September and preserved with a time stamp on SocArXiv, is like a preregistration of the later versions, effectively predicting I would find a decline in subsequent years if I used the same models — which I did. People who use data that is released on a regular schedule, like ACS, CPS, or GSS, might consider doing this in the future.


Rejection addendumSociological Science rejected this — as they do, in about 30 days, with very brief reviews — and based on their misunderstandings I made some clarifications and explained the limitations before sending it to Socius. Since the paper was publicly available the whole time this didn’t slow down the progress of science, and then I improved it, so I’m happy about it.

Just in case you’re worried that this rejections means the paper might be wrong, I’m sharing their reviews here, as summarized by the editor. If you read the current version you’ll see how I clarified these points.

* While the analyses are generally sensible, both Consulting Editors point out the paper’s modest contribution to the literature relative to Kennedy and Ruggles (2014) and Hemez (2017). The paper cites both of these papers but does not make clear how the paper adds to our understanding derived from those papers. If the relatively modest extension in the time frame in this paper is sociologically consequential, the paper does not make the case clearly.

* There is more novelty in the paper’s estimates of the annual divorce probability for newly-married women (shown in Table 3 and Figure 3), based on estimating a divorce model for the most recent survey year, and then applying the coefficients from that model to prior years. This procedure was somewhat difficult for the readers to follow, but issues were raised, most notably the question of the sensitivity of the results to the adjustments made. As one CE noted, “Excluding those in the first year of marriage is problematic as newlyweds have a high rate of divorce. Also, why just married in the last 10 years? Consider whether married for the first time vs remarried matters. Also, investigate the merits of an age restriction given the aging of the population Kennedy and Ruggles point to.”

Leave a comment

Filed under Me @ work

I spent my semester as an MIT / CREOS Visiting Scholar and it was excellent

PNC in Cambridge in the fall.

Cambridge in the fall.

As a faculty sociologist who works in the area of family demography and inequality, my interest in open scholarship falls into the category of “service” among my academic obligations, essentially unrecognized and unremunerated by my employer, and competing with research and teaching responsibilities for my time. In that capacity I founded SocArXiv in 2016 (supported by several small grants) and serve as its director, organized two conferences at the University of Maryland under the title O3S: Open Scholarship for the Social Sciences, and I was elected to the Committee on Publications of the American Sociological Association. While continuing that work during a sabbatical leave, I was extremely fortunate to land a half-time position as visiting scholar at the MIT Libraries in the fall 2018, which helped me integrate that service agenda with an emerging research agenda around scholarly communication.

The position was sponsored by a group of libraries organized by the Association of Research Libraries — MIT, UCLA, the University of Arizona, Ohio State University, and the University of Pittsburgh — and hosted by the new Center for Research on Equitable and Open Scholarship (CREOS) at MIT. My principal collaborator has been Micah Altman, the director of research at CREOS.

The semester was framed by the MIT Grand Challenges Summit in the spring, which I attended, and the report that emerged from that meeting: A Grand Challenges-Based Research Agenda for Scholarly Communication and Information Science, on which I was a collaborator. The report, published in December, describes a vision for a more inclusive, open, equitable, and sustainable future for scholarship; it also characterizes the barriers to this future, and identifies the research needed to bring it to fruition.

Sociology and SocArXiv

Furthering my commitments to sociology and SocArXiv, I continued to work on the service. SocArXiv is growing, with increased participation in sociology and other social sciences. In the fall the Center for Open Science, our host, opened discussions with its paper serving communities about weaning the system off its core foundation financial support and using contributions from each service to make it sustainable (thus far have not paid COS for its develop and hosting). This was an expected challenge, which will require some creative and difficult work in the coming months.

Finally, at the start of the semester I noted that most sociologists — even those interested in open access issues — were not familiar with current patterns, trends, and debates in the scholarly communications ecosystem. This has hampered our efforts to build SocArXiv, as well as our ability to press our associations and institutions for policy changes in the direction of openness, equity, and sustainability. In response to this need, especially among graduate students and junior scholars, I drafted a scholarly communication primer for sociology, which reviews major scholarly communication media, policies, economic actors, and recent innovations. I posted a long draft (~13,000 words) for comment in January, and received a very positive response. It appears that a number of programs will incorporate the revised primer into their training, and many individuals are already reading and sharing it with their networks.

Peer review

One of the chief barriers identified in the Grand Challenges report is the lack of systematic theory and empirical evidence to design and guide legal, economic, policy and organizational interventions in scholarly publishing and in the knowledge ecosystem generally. As social scientists, Micah and I drew on this insight, and used the case of peer-review in sociology as an entry point. We presented our formative analysis of this case in the CREOS Research Talk, “Can Fix Peer Review.” Here is the summary of this talk:

Contemporary journal peer review is beset by a range of problems. These include (a) long delay times to publication, during which time research is inaccessible; (b) weak incentives to conduct reviews, resulting in high refusal rates as the pace of journal publication increases; (c) quality control problems that produce both errors of commission (accepting erroneous work) and omission (passing over important work, especially null findings); (d) unknown levels of bias, affecting both who is asked to perform peer review and how reviewers treat authors, and; (e) opacity in the process that impedes error correction and more systematic learning, and enables conflicts of interest to pass undetected. Proposed alternative practices attempt to address these concerns — especially open peer review, and post-publication peer review. However, systemic solutions will require revisiting the functions of peer review in its institutional context.

The full slides, with embedded video of the talk (minus the first few minutes) is embedded below:

Research design and intervention

Mapping out the various interventions and proposed alternatives in the peer review space raised a number of questions about how to design and evaluate interventions in a complex system with interdependent parts and actors embedded in different institutional logics — for example, university researchers (some working under state policy), research libraries, for-profit publishers, and academic societies. Working with Jessica Polka, Director of ASAPbio, we are expanding this analysis to consider a range of innovations open science. This analysis highlights the need for systematic research design that can guide the design of initiatives aimed at altering the scholarly knowledge ecosystem.

Applying the ecosystem approach in the Grand Challenges report, we consider large-scale interventions in public health and safety, and their unintended consequences, to build a model for designing projects with the intention of identifying and assessing such consequences across the system. Addressing problems at scale may have such unintended effects as leading vulnerable populations to adapt to new technology in harmful ways (mosquito nets used for fishing); providing new opportunities for harmful competitors (the pesticide treadmill); the displacement of private actors by public goods (dentists driven away by public water fluoridation); and risk compensation by those who receive public protection (anti-lock brakes and riskier driving, vaccinations). Our forthcoming white paper will address such risks in light of recent open science interventions: PLOS One, bioRxiv and preprints generally, and open peer review, among others. We combine research design methods for field experiments in social science, outcomes identified in the grand challenge report, and the ecosystem theory based on an open science lifecycle model.

ARL/SSRC meeting and Next Steps

Coming out of discussions at the first O3S meeting, in December the Association of Research Libraries and the Social Science Research Council convened a meeting on open scholarship in the social sciences, which included leaders from scholarly societies, university libraries, researchers advocating for open science, funders, and staff from ARL, SSRC, and the Coalition for Networked Information. I was fortunate to participate on the planning committee for the meeting, and in that capacity I conducted a series of short video interviews with individual stakeholders from the participating organizations to help expose us all to the range of values, objectives, and concerns we bring to the questions we collectively face in the movement toward open scholarship.

For our own work on peer review, which we presented at the meeting, I was especially drawn to the interviewees’ comments on transparency, incentives, and open infrastructure. In particular, MIT Libraries Director Chris Bourg challenged social scientists to recognize what their own research implies for the peer review system:

Brian Nosek, director of the Center for Open Science, stressed to the need to consider incentives for openness in our interventions:

And Kathleen Fitzpatrick, project director for Humanities Commons, described the necessity of open infrastructure that is flexibly interoperable, allowing parallel use by actors on diverse platforms:

These insights about intervention principles for an open scholarly ecosystem helped Micah and me develop a proposal for discussion at the meeting. Our proposed program, IOTA (I Owe The Academy) aims to solve the supply-and-demand problem for quality peer review in open science interventions (the name is likely to change). We understand that most academics are willing to do peer review when it contributes to a better system of scholarship. At the same time, new peer review projects need (good) reviewers in order to launch successfully. And the community needs (good) empirical research on the peer review process itself. The solution is to match reviewers with initiatives that promote better scholarship using a virtual token system, whereby reviewers pledge review effort units, which are distributed to open peer review projects — while collecting data for use in evaluation and assessment. After receiving positive feedback at the meeting, we will develop this proposal further.

Our presentation is embedded in full below:

A report on the ARL/SSRC meeting describes the shared interests, challenges to openness, and conditions for successful action discussed by participants. And it includes five specific projects they agreed to pursue — one of which is peer review on the SocArXiv and PsyArXiv paper platforms.

What’s next…

In the coming several months we expect to produce a white paper on research design, a proposal for IOTA, and a presentation for the Coalition for Networked Information meeting in April, to spark a discussion about the ways libraries can jointly support additional targeted work to promote, inspire, and support evidence-based research. And a revised version of the scholarly communication primer for sociology is on the way.

1 Comment

Filed under Me @ work

Interrogating Ethnography comments

pnc-sign-at-northwestern

Elite law schools have better views.

I’ve edited out 4 minutes of stammering and ums, and a couple of errors, from my comments on Steven Lubet’s new book, Interrogating Ethnography: Why Evidence Matters, from this conference held over the weekend. So here it is in 16 minutes:

It turns out some people saw things differently from how I did, and I had a lot to think about. Comments welcome.

(There was a court reporter and video, so a complete record of all the panels should be available at some point.)

2 Comments

Filed under Me @ work

Sociology’s culture of trust, don’t verify

Replication in sociology is a disaster. There basically isn’t any. Accountability is something a select few people opt into; as a result, mostly people with nothing to hide ever have their work verified or replicated. Even when work is easily replicable, such as that using publicly available datasets, there is no common expectation that anyone will do it, and no support for doing it; basically no one funds or publishes replications.

Peer review is good, but it’s not about replicability, because it almost always relies on the competence and good faith of the authors. Reviewers might say, “This looks funny, did you try this or that?” But if the author says, “Yes, I did that,” that’s usually the end of it. Academic sociology, in short, runs on a system of trust. That’s worth exactly what it’s worth. It doesn’t have to be this way.

I thought of this today when I read the book excerpt by Mark Regnerus in the Wall Street Journal. (I haven’t read his new book, Cheap Sex yet, although I called the basic arguments a “big ball of wrong” three years ago when he first published them.) Regnerus opens that essay with a single quote supposedly from an anonymous 24-year-old recent college graduate that absolutely perfectly represents his thesis:

If you know what girls want, then you know you should not give that to them until the proper time. If you do that strategically, then you can really have anything you want…whether it’s a relationship, sex, or whatever. You have the control.

(Regnerus argues men have recently gained control over sex because women have stopped demanding marriage in exchange for it.)

Scholars and readers in sociology don’t normally question whether specific quotes in qualitative research are real or not. We argue over the interpretation, or elements of the research design that might call the interpretation into question (such as the method of selecting respondents or a field site). But if we simply don’t trust the author, what do we do? In the case of Regnerus, we know that he has lied, a lot, about important things related to his research. So how do you read his research in a discipline with no norm of verification or replicability, a discipline naively based on trust? The fake news era is here; we have to address this. Fortunately, every other social discipline already is, so we don’t have to reinvent the wheel.

Tackling it

Of course there are complicated issues with different kinds of sociology, especially qualitative work. It’s one of the things people wrestled with in the Contexts forum Syed Ali and I organized for the American Sociological Association on how to do ethnography right.

That forum took place in the wake of all the attention Alice Goffman received for her book, and article, On the Run (my posts on that are under this tag). One person who followed that controversy closely was law professor Steven Lubet, who has written a new book titled, “Interrogating Ethnography: Why Evidence Matters,” which addresses that situation in depth. The book comes out October 20, at a conference at Northwestern University’s law school. I will be one of a number of people commenting on the book and its implications.

inteth

I hope you can come to the event in Chicago.

Finally, regardless of your opinion on recent controversies in sociology, if you haven’t read it, I urge you to read (and, if you’re in such a position, require that your students read) “Replication in Social Science,” by Jeremy Freese and David Peterson, in the latest Annual Review of Sociology (SocArXiv preprint; journal version). Freese and Peterson refer to sociology as “the most undisciplined social science,” and they write:

As sociologists, the most striking thing in reviewing recent developments in social science replication is how much all our neighbors seem to be talking and doing about improving replicability. Reading economists, it is hard not to connect their relatively strict replication culture with their sense of importance: shouldn’t a field that has the ear of policy-makers do work that is available for critical inspection by others? The potential for a gloomy circle ensues, in which sociology would be more concerned with replication and transparency if it was more influential, but unwillingness to keep current on these issues prevents it from being more influential. In any case, the integrative and interdisciplinary ambitions of many sociologists are obviously hindered by the field’s inertness on these issues despite the growing sense in nearby disciplines that they are vital to ensuring research integrity.

That paper has some great ideas for easy reforms to start out with. But we need to get the conversation moving. In addition developing replication standards and norms, we need to get the next generation of sociologists some basic training in the (jargon alert!) political economy of scholarly communication and the publishing ecosystem. The individual incentives are weak, but the need for the discipline to act is very strong. If we can at least get sociologists to be vaguely aware of the attention to this issue generated in most other social science disciplines, it would be a great step forward.

Incidentally, Freese will also present on the topic of replication at the O3S: Open Scholarship for the Social Sciences symposium SocArXiv is hosting at the University of Maryland later this month; still time to register!

3 Comments

Filed under In the news

Let’s use award incentives to promote open scholarship (at ASA this year!)

At the American Sociological Association of America meetings in Montreal next month, I will begin a one-year term as chair of the Family Section. I’m honored to have been elected to this position, and will do my best to make a positive contribution in that role. Besides doing the job in the normal ways — organizing our sessions at the conference next year, coordinating committees, and so on — I will bring a proposal to the section’s council to open our graduate student paper award. Here’s what I mean.

Steps toward solutions

Sociology has an inertia problem with regard to open scholarship. Lots of us understand that it would be better if our work was shared faster and more freely. That would be better for the generation and dissemination of new knowledge, it would promote collaboration, reduce costs to the public, and increase our capacity for engagement with each other and the public. Unfortunately, the individual steps toward that goal are unclear or daunting. Many of us need promotion and tenure, which requires prestige, which is still driven by publication in the paywalled journals that work against our open goals: they slow down dissemination, restrict access to our work, and bilk our institutions with exorbitant subscription fees.

To help overcome this inertia, a group of us have created SocArXiv, a non-profit, open access, open source archive of social science research that allows free, immediate publication of papers at any stage of the publication process. When and if the papers are published in a peer-reviewed journal, the preprint version can link to the journal version, providing a free copy of the paywalled paper. (Here’s an example of a new paper published in American Sociological Review, with a free copy on SocArXiv, which includes a link to the ASR version). In the meantime, the paper is available to our peers and the public. It provides a time-stamped record of the development of our original ideas, and is discoverable through Google Scholar and other search tools. People can still get their jobs and promotions, but the quality, efficiency, and reach of our research is improved. And part of what we are rewarding is open scholarship itself.

flipaward

Using awards

SocArXiv, of which I’m director, is trying to get the word out and encourage the use of our system, and open scholarship in general. One of our new ideas is opening paper awards. This may help people get in the habit of openness — and start to see its benefits — and also work against the negative impression that many people have of open access as a cesspool of low quality work. We hope this intervention will be especially effective coming early in the career of up-and-coming scholars.

Using its grant money and support from academic libraries, SocArXiv is offering sections of the ASA — like the Family Section — $400 to transport their paper award winner to the conference next year, if they using the archive as the submission platform for their awards. I’m bringing this proposal to the Family Section (and one just like it to the Population Section, of which I’m Secretary Treasurer).

We hope the open paper award will become a common best practice in our association — still providing the prestige and reward functions of the award, but also promoting best practices with regard to open scholarship, increasing our visibility, building the scholarly communication infrastructure of the future, and generating buzz for our conference and our research.

There are possible objections to this idea. Here are a few, with my responses:

  • Sharing unpublished work will lead to someone stealing their ideas. You protect yourself by posting it publicly.
  • We shouldn’t promote the dissemination of research that hasn’t been peer reviewed yet. We do this all the time at conferences, and SocArXiv allows posting updated versions that replace the original when it is revised.
  • This would impose a burden on people submitting papers. Being considered for an award is a privilege, not a right; it’s OK to require a short, free submission process.
  • Sharing a paper publicly will compromise its publishability later. All ASA journals, and all journals worthy of our support, allow posting preprints prior to publication. Here’s a list of 25 top journals and their policies.

Details

In the case of the Family Section, it looks like no change in the bylaws is needed, because they don’t specify the submission process for the graduate student paper award. They state:

Best Graduate Student Paper Award. The committee will be chaired by the Section Chair. Two additional members of the Section will be appointed by the Section Chair. The committee will select a best paper from among nominations submitted. Papers, dealing with a family-related topic, may be either published or unpublished and must have been writted by a graduate student (or group of graudate students) while still enrolled in a graduate program. The award, in the form of a Plaque and citation, shall be presented at a Section Reception (or, in the event no reception is held, at a Business Meeting of the Section).

Instead, I think we can just revise the call for award nominations, like this:

The Family Section Outstanding Graduate Student Paper Award

​Deadline: 3/13/2018

Graduate students are invited to submit an article-length paper on the family. The paper should represent a finished product rather than a proposal for future work. The submission can be based on a course paper, a recently published journal article, a manuscript under review at a journal, or a conference presentation. Co-authored papers are acceptable if all authors are students, although the prize will be shared. The paper must have been written when the author was enrolled in a graduate program. The paper may not exceed 30 pages or 11,000 words. Submissions are made by posting the paper on SocArXiv and sending a link to the paper to the committee chair, Philip N. Cohen, at pnc@umd.edu. To submit your paper, go to SocArXiv.org, and click “Add a preprint.” If you don’t yet have an account, you will fill out a short form — it’s free, non-profit, and won’t spam you! For assistance, contact socarxiv@gmail.com or consult the FAQ page. Please indicate whether you would like your paper to be included in a public list of submissions (this will not affect your chances of winning). The winner will receive a plaque and travel reimbursement up to $400 to attend the 2018 Family Section reception at the ASA meetings.

The Family Section Council will consider this proposal next month in Montreal. Please let us know what you think!

1 Comment

Filed under Me @ work

Update on SocArXiv and social science without walls

social science without walls

Meanwhile, over at SocArXiv, we’re working on revolutionizing the research process and how we communicate about it in the social sciences. You can follow the exploits of the SocArXiv project on our blog SocOpen. There you can read, most recently:

That’s the update!

Leave a comment

Filed under Me @ work