Tag Archives: socarxiv

Interrogating Ethnography comments

pnc-sign-at-northwestern

Elite law schools have better views.

I’ve edited out 4 minutes of stammering and ums, and a couple of errors, from my comments on Steven Lubet’s new book, Interrogating Ethnography: Why Evidence Matters, from this conference held over the weekend. So here it is in 16 minutes:

It turns out some people saw things differently from how I did, and I had a lot to think about. Comments welcome.

(There was a court reporter and video, so a complete record of all the panels should be available at some point.)

1 Comment

Filed under Me @ work

Sociology’s culture of trust, don’t verify

Replication in sociology is a disaster. There basically isn’t any. Accountability is something a select few people opt into; as a result, mostly people with nothing to hide ever have their work verified or replicated. Even when work is easily replicable, such as that using publicly available datasets, there is no common expectation that anyone will do it, and no support for doing it; basically no one funds or publishes replications.

Peer review is good, but it’s not about replicability, because it almost always relies on the competence and good faith of the authors. Reviewers might say, “This looks funny, did you try this or that?” But if the author says, “Yes, I did that,” that’s usually the end of it. Academic sociology, in short, runs on a system of trust. That’s worth exactly what it’s worth. It doesn’t have to be this way.

I thought of this today when I read the book excerpt by Mark Regnerus in the Wall Street Journal. (I haven’t read his new book, Cheap Sex yet, although I called the basic arguments a “big ball of wrong” three years ago when he first published them.) Regnerus opens that essay with a single quote supposedly from an anonymous 24-year-old recent college graduate that absolutely perfectly represents his thesis:

If you know what girls want, then you know you should not give that to them until the proper time. If you do that strategically, then you can really have anything you want…whether it’s a relationship, sex, or whatever. You have the control.

(Regnerus argues men have recently gained control over sex because women have stopped demanding marriage in exchange for it.)

Scholars and readers in sociology don’t normally question whether specific quotes in qualitative research are real or not. We argue over the interpretation, or elements of the research design that might call the interpretation into question (such as the method of selecting respondents or a field site). But if we simply don’t trust the author, what do we do? In the case of Regnerus, we know that he has lied, a lot, about important things related to his research. So how do you read his research in a discipline with no norm of verification or replicability, a discipline naively based on trust? The fake news era is here; we have to address this. Fortunately, every other social discipline already is, so we don’t have to reinvent the wheel.

Tackling it

Of course there are complicated issues with different kinds of sociology, especially qualitative work. It’s one of the things people wrestled with in the Contexts forum Syed Ali and I organized for the American Sociological Association on how to do ethnography right.

That forum took place in the wake of all the attention Alice Goffman received for her book, and article, On the Run (my posts on that are under this tag). One person who followed that controversy closely was law professor Steven Lubet, who has written a new book titled, “Interrogating Ethnography: Why Evidence Matters,” which addresses that situation in depth. The book comes out October 20, at a conference at Northwestern University’s law school. I will be one of a number of people commenting on the book and its implications.

inteth

I hope you can come to the event in Chicago.

Finally, regardless of your opinion on recent controversies in sociology, if you haven’t read it, I urge you to read (and, if you’re in such a position, require that your students read) “Replication in Social Science,” by Jeremy Freese and David Peterson, in the latest Annual Review of Sociology (SocArXiv preprint; journal version). Freese and Peterson refer to sociology as “the most undisciplined social science,” and they write:

As sociologists, the most striking thing in reviewing recent developments in social science replication is how much all our neighbors seem to be talking and doing about improving replicability. Reading economists, it is hard not to connect their relatively strict replication culture with their sense of importance: shouldn’t a field that has the ear of policy-makers do work that is available for critical inspection by others? The potential for a gloomy circle ensues, in which sociology would be more concerned with replication and transparency if it was more influential, but unwillingness to keep current on these issues prevents it from being more influential. In any case, the integrative and interdisciplinary ambitions of many sociologists are obviously hindered by the field’s inertness on these issues despite the growing sense in nearby disciplines that they are vital to ensuring research integrity.

That paper has some great ideas for easy reforms to start out with. But we need to get the conversation moving. In addition developing replication standards and norms, we need to get the next generation of sociologists some basic training in the (jargon alert!) political economy of scholarly communication and the publishing ecosystem. The individual incentives are weak, but the need for the discipline to act is very strong. If we can at least get sociologists to be vaguely aware of the attention to this issue generated in most other social science disciplines, it would be a great step forward.

Incidentally, Freese will also present on the topic of replication at the O3S: Open Scholarship for the Social Sciences symposium SocArXiv is hosting at the University of Maryland later this month; still time to register!

1 Comment

Filed under In the news

Let’s use award incentives to promote open scholarship (at ASA this year!)

At the American Sociological Association of America meetings in Montreal next month, I will begin a one-year term as chair of the Family Section. I’m honored to have been elected to this position, and will do my best to make a positive contribution in that role. Besides doing the job in the normal ways — organizing our sessions at the conference next year, coordinating committees, and so on — I will bring a proposal to the section’s council to open our graduate student paper award. Here’s what I mean.

Steps toward solutions

Sociology has an inertia problem with regard to open scholarship. Lots of us understand that it would be better if our work was shared faster and more freely. That would be better for the generation and dissemination of new knowledge, it would promote collaboration, reduce costs to the public, and increase our capacity for engagement with each other and the public. Unfortunately, the individual steps toward that goal are unclear or daunting. Many of us need promotion and tenure, which requires prestige, which is still driven by publication in the paywalled journals that work against our open goals: they slow down dissemination, restrict access to our work, and bilk our institutions with exorbitant subscription fees.

To help overcome this inertia, a group of us have created SocArXiv, a non-profit, open access, open source archive of social science research that allows free, immediate publication of papers at any stage of the publication process. When and if the papers are published in a peer-reviewed journal, the preprint version can link to the journal version, providing a free copy of the paywalled paper. (Here’s an example of a new paper published in American Sociological Review, with a free copy on SocArXiv, which includes a link to the ASR version). In the meantime, the paper is available to our peers and the public. It provides a time-stamped record of the development of our original ideas, and is discoverable through Google Scholar and other search tools. People can still get their jobs and promotions, but the quality, efficiency, and reach of our research is improved. And part of what we are rewarding is open scholarship itself.

flipaward

Using awards

SocArXiv, of which I’m director, is trying to get the word out and encourage the use of our system, and open scholarship in general. One of our new ideas is opening paper awards. This may help people get in the habit of openness — and start to see its benefits — and also work against the negative impression that many people have of open access as a cesspool of low quality work. We hope this intervention will be especially effective coming early in the career of up-and-coming scholars.

Using its grant money and support from academic libraries, SocArXiv is offering sections of the ASA — like the Family Section — $400 to transport their paper award winner to the conference next year, if they using the archive as the submission platform for their awards. I’m bringing this proposal to the Family Section (and one just like it to the Population Section, of which I’m Secretary Treasurer).

We hope the open paper award will become a common best practice in our association — still providing the prestige and reward functions of the award, but also promoting best practices with regard to open scholarship, increasing our visibility, building the scholarly communication infrastructure of the future, and generating buzz for our conference and our research.

There are possible objections to this idea. Here are a few, with my responses:

  • Sharing unpublished work will lead to someone stealing their ideas. You protect yourself by posting it publicly.
  • We shouldn’t promote the dissemination of research that hasn’t been peer reviewed yet. We do this all the time at conferences, and SocArXiv allows posting updated versions that replace the original when it is revised.
  • This would impose a burden on people submitting papers. Being considered for an award is a privilege, not a right; it’s OK to require a short, free submission process.
  • Sharing a paper publicly will compromise its publishability later. All ASA journals, and all journals worthy of our support, allow posting preprints prior to publication. Here’s a list of 25 top journals and their policies.

Details

In the case of the Family Section, it looks like no change in the bylaws is needed, because they don’t specify the submission process for the graduate student paper award. They state:

Best Graduate Student Paper Award. The committee will be chaired by the Section Chair. Two additional members of the Section will be appointed by the Section Chair. The committee will select a best paper from among nominations submitted. Papers, dealing with a family-related topic, may be either published or unpublished and must have been writted by a graduate student (or group of graudate students) while still enrolled in a graduate program. The award, in the form of a Plaque and citation, shall be presented at a Section Reception (or, in the event no reception is held, at a Business Meeting of the Section).

Instead, I think we can just revise the call for award nominations, like this:

The Family Section Outstanding Graduate Student Paper Award

​Deadline: 3/13/2018

Graduate students are invited to submit an article-length paper on the family. The paper should represent a finished product rather than a proposal for future work. The submission can be based on a course paper, a recently published journal article, a manuscript under review at a journal, or a conference presentation. Co-authored papers are acceptable if all authors are students, although the prize will be shared. The paper must have been written when the author was enrolled in a graduate program. The paper may not exceed 30 pages or 11,000 words. Submissions are made by posting the paper on SocArXiv and sending a link to the paper to the committee chair, Philip N. Cohen, at pnc@umd.edu. To submit your paper, go to SocArXiv.org, and click “Add a preprint.” If you don’t yet have an account, you will fill out a short form — it’s free, non-profit, and won’t spam you! For assistance, contact socarxiv@gmail.com or consult the FAQ page. Please indicate whether you would like your paper to be included in a public list of submissions (this will not affect your chances of winning). The winner will receive a plaque and travel reimbursement up to $400 to attend the 2018 Family Section reception at the ASA meetings.

The Family Section Council will consider this proposal next month in Montreal. Please let us know what you think!

1 Comment

Filed under Me @ work

Update on SocArXiv and social science without walls

social science without walls

Meanwhile, over at SocArXiv, we’re working on revolutionizing the research process and how we communicate about it in the social sciences. You can follow the exploits of the SocArXiv project on our blog SocOpen. There you can read, most recently:

That’s the update!

Leave a comment

Filed under Me @ work

Advice for and about ASA

Last summer the incoming American Sociological Association President, Michèle Lamont, asked me to offer some advice to ASA about open access publishing issues. It was an open-ended request, and I didn’t know how to go about it. My understanding of ASA is that it is not well outfitted as a change agent; it’s much more likely to respond to external developments in its ecosystem than to take the lead, especially when its revenue stream is at stake. Nevertheless, lots of good people work in and around the association, and it has great capacity. (I am involved myself, as co-editor of the ASA magazine Contexts, as chair-elect of the Family Section, and as secretary treasurer of the Population Section.) So I wrote a short essay on what ASA might do, or what its members might do or demand of it.

It’s not coincidental that this is posted on the SocArXiv blog, SocOpen, which is part of that changing external environment that I hope will lead to ASA adapting for the better. I believe that devoting my energy to this project is producing something tangible for research and scholarly communication, while also pressuring ASA (and maybe other associations) to move in the right direction.

I hope you’ll read it on SocOpen.

Leave a comment

Filed under Me @ work

Do we get tenure for this?

My photo. For the occasion I titled it, Openness. https://flic.kr/p/FShb6d

For the occasion I titled this photo of Utah “Openness.”

Colleen Flaherty at Inside Higher Ed has written up the American Sociological Association’s committee report, “What Counts? Evaluating Public Communication in Tenure and Promotion.”

I was once a member of the ASA Subcommittee on the Evaluation Of Social Media and Public Communication In Sociology, which was chaired by Leslie McCall when they produced the report. (It is a subcommittee of the task force on engaging sociology, convened by then-President Annette Lareau.)

It’s worth reading the whole article, which also includes comments from Sara Ovink, McCall and me, in addition to the report. Having thought about this issue a little, I was happy to respond to Flaherty’s request for comment. These are the full comments I sent her, from which she quoted in the article:

1. We don’t need credit toward promotion for every thing we do. Scholars who take a public-facing stance in their work often find that it enhances the quality and quantity of their work in the traditional fields of assessment (research, teaching, service), so that separately rewarding the public work is not always necessary. I don’t need credit for having a popular blog – that work has led to new research ideas, better feedback on my research, better grad students, teaching ideas, invitations to contribute to policy, and book contracts.

2. We’d all love to be promoted for authoring a great tweet but no one wants to be fired for a bad one. Assessment of public engagement needs to be holistic and qualitative, taking into the account quality, quantity, and impact of the work. Simplistic quantitative metrics will not be useful.

3. It is also important to value and reward openness in our routine work, such as posting working papers, publishing in open access journals, sharing replication files, and disseminating open teaching materials. Public engagement does not need to mean separate activities and products, but can mean taking a public-facing stance in our existing work.

The SocArxiv project is one outcome of these conversations (links to latest infosubmit a paper), especially relating to point #3 above. Academics who open up their work should be recognized for that contribution to the public good and for promoting the future of academia. In that spirit also I proposed a rule change for the ASA Dissertation Award, which now includes this:

To be eligible for the ASA Dissertation Award, candidates’ dissertations must be publicly available in Dissertation Abstracts International or a comparable outlet. Dissertations that are not available in this fashion will not be considered for the award.

It’s hard to change everything, but it’s not that hard to make some important changes in the right direction. Rewarding engagement and openness is an important step in the right direction.

1 Comment

Filed under Uncategorized