Interrogating Ethnography comments

Elite law schools have better views.

I’ve edited out 4 minutes of stammering and ums, and a couple of errors, from my comments on Steven Lubet’s new book, Interrogating Ethnography: Why Evidence Matters, from this conference held over the weekend. So here it is in 16 minutes:

It turns out some people saw things differently from how I did, and I had a lot to think about. Comments welcome.

(There was a court reporter and video, so a complete record of all the panels should be available at some point.)

Sources for Interrogating Ethnography conference

I’m at this conference today for the launch Steven Lubet’s new book Interrogating Ethnography. Since I’m going to mention the things I’ve written about Alice Goffman’s work in On The Run, I’m putting the links here for easy reference:

My comment to the American Sociological Review about problems with the survey in her article and book is here. The rejection I received from ASR is here.

My blog posts are:

• Sociology’s culture of trust, don’t verify
• Goffman dissertation followup
• On Goffman’s survey (most of this is better in the later piece)
• Proposed rule change for the American Sociological Association Dissertation Award
• On the ropes (Goffman review)

They say the sessions at this conference will be recorded and made available later.

Sociology’s culture of trust, don’t verify

Replication in sociology is a disaster. There basically isn’t any. Accountability is something a select few people opt into; as a result, mostly people with nothing to hide ever have their work verified or replicated. Even when work is easily replicable, such as that using publicly available datasets, there is no common expectation that anyone will do it, and no support for doing it; basically no one funds or publishes replications.

Peer review is good, but it’s not about replicability, because it almost always relies on the competence and good faith of the authors. Reviewers might say, “This looks funny, did you try this or that?” But if the author says, “Yes, I did that,” that’s usually the end of it. Academic sociology, in short, runs on a system of trust. That’s worth exactly what it’s worth. It doesn’t have to be this way.

I thought of this today when I read the book excerpt by Mark Regnerus in the Wall Street Journal. (I haven’t read his new book, Cheap Sex yet, although I called the basic arguments a “big ball of wrong” three years ago when he first published them.) Regnerus opens that essay with a single quote supposedly from an anonymous 24-year-old recent college graduate that absolutely perfectly represents his thesis:

If you know what girls want, then you know you should not give that to them until the proper time. If you do that strategically, then you can really have anything you want…whether it’s a relationship, sex, or whatever. You have the control.

(Regnerus argues men have recently gained control over sex because women have stopped demanding marriage in exchange for it.)

Scholars and readers in sociology don’t normally question whether specific quotes in qualitative research are real or not. We argue over the interpretation, or elements of the research design that might call the interpretation into question (such as the method of selecting respondents or a field site). But if we simply don’t trust the author, what do we do? In the case of Regnerus, we know that he has lied, a lot, about important things related to his research. So how do you read his research in a discipline with no norm of verification or replicability, a discipline naively based on trust? The fake news era is here; we have to address this. Fortunately, every other social discipline already is, so we don’t have to reinvent the wheel.

Tackling it

Of course there are complicated issues with different kinds of sociology, especially qualitative work. It’s one of the things people wrestled with in the Contexts forum Syed Ali and I organized for the American Sociological Association on how to do ethnography right.

That forum took place in the wake of all the attention Alice Goffman received for her book, and article, On the Run (my posts on that are under this tag). One person who followed that controversy closely was law professor Steven Lubet, who has written a new book titled, “Interrogating Ethnography: Why Evidence Matters,” which addresses that situation in depth. The book comes out October 20, at a conference at Northwestern University’s law school. I will be one of a number of people commenting on the book and its implications.

I hope you can come to the event in Chicago.

Finally, regardless of your opinion on recent controversies in sociology, if you haven’t read it, I urge you to read (and, if you’re in such a position, require that your students read) “Replication in Social Science,” by Jeremy Freese and David Peterson, in the latest Annual Review of Sociology (SocArXiv preprint; journal version). Freese and Peterson refer to sociology as “the most undisciplined social science,” and they write:

As sociologists, the most striking thing in reviewing recent developments in social science replication is how much all our neighbors seem to be talking and doing about improving replicability. Reading economists, it is hard not to connect their relatively strict replication culture with their sense of importance: shouldn’t a field that has the ear of policy-makers do work that is available for critical inspection by others? The potential for a gloomy circle ensues, in which sociology would be more concerned with replication and transparency if it was more influential, but unwillingness to keep current on these issues prevents it from being more influential. In any case, the integrative and interdisciplinary ambitions of many sociologists are obviously hindered by the field’s inertness on these issues despite the growing sense in nearby disciplines that they are vital to ensuring research integrity.

That paper has some great ideas for easy reforms to start out with. But we need to get the conversation moving. In addition developing replication standards and norms, we need to get the next generation of sociologists some basic training in the (jargon alert!) political economy of scholarly communication and the publishing ecosystem. The individual incentives are weak, but the need for the discipline to act is very strong. If we can at least get sociologists to be vaguely aware of the attention to this issue generated in most other social science disciplines, it would be a great step forward.

Incidentally, Freese will also present on the topic of replication at the O3S: Open Scholarship for the Social Sciences symposium SocArXiv is hosting at the University of Maryland later this month; still time to register!

Goffman dissertation followup

I previously reviewed Alice Goffman’s book On The Run, and wrote a critique of the survey that was part of that project (including a formal comment sent to American Sociological Review). Then I complained that her dissertation was not made public, despite being awarded the American Sociological Association’s dissertation prize. I proposed a rule change for the association, requiring that the winning dissertation be “publicly available through a suitable academic repository by the time of the ASA meeting at which the award is granted.”

Here’s a quick followup.

I was interested in Goffman’s 2010 dissertation because I thought it might have more information about the survey she conducted than the 2014 book did. When I inquired about the dissertation on June 4 of this year, Princeton’s director of media relations, Martin Mbugua, told me she “was granted an exemption from submitting her dissertation to the University Archives, so we do not have a copy of her dissertation in our collection.”

Jesse Singal at New York magazine reported yesterday that they now have the dissertation, and he’s read it. Not only does it not have more methodological information than the book, Singal reports, it actually has less, as the methodological appendix that’s in the book is not in the dissertation. In a saved-you-a-trip-to-Princeton email to me, Singal says the dissertation’s description of her survey is “basically identical” to what is in ASR. That speaks to my critique of her survey, which seems unaffected by the release of the dissertation. (I’m not in charge of dissertations at Princeton, so I’m not critiquing the dissertation anyway.)

With regard to the open-science-inspired rule change for ASA dissertation awards, Singal’s article just reinforces my desire to see the rule adopted. Mbugua told Singal that Princeton now allows up to two two-year embargo periods for PhD students who don’t want their dissertations publicly released. But why embargo it? I think most people do this because they don’t want to undermine their book deals. The need for this may be overstated, but it’s a thing. (Eric Schwartz who acquires sociology books for Columbia University Press, tweeted: “No problem. Book and dissertation are for different audiences.”)

Anyway, Singal quotes Goffman giving a quite different reason:

The dissertation contained very sensitive material about people who were vulnerable to arrest and incarceration. … I wanted to think through the ethical and human subjects issues of making it available beyond the committee members and I wanted some time to go by between the actual events and a public reading. That felt safer for the people who had granted me permission to write about their lives, and for me, than publishing right away.

Apart from the fact that this concern did not prevent Goffman from submitting her book to a reading by an awards committee — “beyond the [dissertation] committee members” — I do not find this very credible, and I don’t like that rationale. If it was wrong to release it in 2010 because it would endanger her subjects, then it was wrong to publish a book in 2014 with the same — actually, more — incriminating information. In fact, as we now know, identifying the individuals mentioned in the book was trivial using Google, and of course the police knew who they were anyway. By this rationale, I cannot understand why the dissertation would not be given to the library until 14 months after the book was published — or until three months after the commercial paperback edition was published. Oh, wait.

Look, if people want to embargo their dissertations for financial gain, and their elite private universities allow it, then so be it. But that doesn’t have to be ASA’s problem. We can add one small piece to that calculation: giving up the ASA Dissertation Award.

On Goffman’s survey

Survey methods.

Jesse Singal at New York Magazine‘s Science of Us has a piece in which he tracks down and interviews a number of Alice Goffman’s respondents. This settles the question — which never should have been a real question — about whether she actually did all that deeply embedded ethnography in Philadelphia. It leaves completely unresolved, however, the issue of the errors and possible errors in the research. This reaffirms for me the conclusion in my original review that we should take the volume down in this discussion, identify errors in the research without trying to attack Goffman personally or delegitimize her career — and then learn from the affair ways that we can improve sociology (for example, by requiring that winners of the American Sociological Association dissertation award make their work publicly available).

That said, I want to comment on a couple of issues raised in Singal’s piece, and share my draft of a formal comment on the survey research Goffman reported in American Sociological Review.

First, I want to distance myself from the description by Singal of “lawyers and journalists and rival academics who all stand to benefit in various ways if they can show that On the Run doesn’t fully hold up.” I don’t see how I (or any other sociologists) benefit if Goffman’s research does not hold up. In fact, although some people think this is worth pursuing, I am also annoying some friends and colleagues by doing this.

More importantly, although it’s a small part of the article, Singal did ask Goffman about the critique of her survey, and her response (as he paraphrased it, anyway) was not satisfying to me:

Philip Cohen, a sociologist at the University of Maryland, published a blog post in which he puzzles over the strange results of a door-to-door survey Goffman says she conducted with Chuck in 2007 in On the Run. The results are implausible in a number of ways. But Goffman explained to me that this wasn’t a regular survey; it was an ethnographic survey, which involves different sampling methods and different definitions of who is and isn’t in a household. The whole point, she said, was to capture people who are rendered invisible by traditional survey methods. (Goffman said an error in the American Sociological Review paper that became On the Run is causing some of the confusion — a reference to “the 217 households that make up the 6th Street neighborhood” that should have read “the 217 households that we interviewed … ” [emphasis mine]. It’s a fix that addresses some of Cohen’s concerns, like an implied and very unlikely 100 percent response rate, but not all of them.) “I should have included a second appendix on the survey in the book,” said Goffman. “If I could do it over again, I would.”

My responses are several. First, the error of describing the 217 households as the whole neighborhood, as well as the error in the book of saying she interviewed all 308 men (when in the ASR article she reports some unknown number were absent), both go in the direction of inflating the value and quality of the survey. Maybe they are random errors, but they didn’t have a random effect.

Second, I don’t see a difference between a “regular survey” and an “ethnographic survey.” There are different survey techniques for different applications, and the techniques used determine the data and conclusions that follow. For example, in the ASR article Goffman uses the survey (rather than Census data) to report the racial composition of the neighborhood, which is not something you can do with a convenience sample, regardless of whether you are engaged in an ethnography or not.

Finally, there are no people “rendered invisible by traditional survey methods” (presumably Singal’s phrase). There are surveys that are better or worse at including people in different situations. There are “traditional” surveys — of varying quality — of homeless people, prisoners, rape victims, and illiterate peasants. I don’t know what an “ethnographic survey” is, but I don’t see why it shouldn’t include a sampling strategy, a response rate, a survey instrument, a data sharing arrangement, and thorough documentation of procedures. That second methodological appendix can be published at any time.

ASR Comment (revised June 22)

I wrote up my relatively narrow, but serious, concerns about the survey, and posted them on SocArXiv, here.

It strikes me that Goffman’s book (either the University of Chicago Press version or the trade book version) may not be subject to the same level of scrutiny that her article in ASR should have been. In fact, presumably, the book publishers took her publication in ASR as evidence of the work’s quality. And their interests are different from those of a scientific journal run by an academic society. If ASR is going to play that gatekeeping role, and it should, then ASR (and by extension ASA) should take responsibility in print for errors in its publications.

Proposed rule change for the American Sociological Association Dissertation Award

Dear Prof. Janoski and members of the ASA Dissertation Award Committee,

I am writing to suggest a rule change for the Dissertation Award. Jordan Robison at ASA told me:

Any rule change for an award committee is usually voted on by the same award committee, presented to Committee on Awards. If Committee on Awards approves any rule change it is then brought to Council, who makes the final vote.

So it is my hope to persuade you to take this rule change up the chain to the Committee on Awards (currently chaired by Jane Sell at Texas A&M), and from there to Council (where the current liaison to the Committee on Awards is Adia Harvey Wingfield at Georgia State, moving to Washington University in St. Louis).

Background

The issue has been raised with regard to 2011 winner Alice Goffman’s dissertation at Princeton. That dissertation is not available at the Princeton library. A query from me to the library produced this response from Martin A. Mbugua, Director of Media Relations & University Spokesperson:

Alice Goffman was granted an exemption from submitting her dissertation to the University Archives, so we do not have a copy of her dissertation in our collection. The Graduate School in 2012 instituted a dissertation embargo policy under which no dissertation would be exempted from the submission requirement. Thus, a dissertation may be embargoed for a period of time to allow for publication in other forms, but it must be submitted to the University’s Seeley G. Mudd Manuscript Library.

Rationale

Goffman’s case is extreme in that her dissertation apparently may never be available, but it remains the case that people at Princeton (and at other schools) may choose to embargo their dissertations, to keep them out of public circulation, while they publish them commercially. Perhaps other schools also allow a complete exemption. If that is their privilege, and they choose to exercise it, then I believe they should be ineligible for the ASA Dissertation Award. My logic is that, if ASA holds up a dissertation as the best of the year, the purpose of that honor is partly to inspire and inform other sociologists about what constitutes the best doctoral research. If the dissertation is not publicly available, that purpose is undermined. Further, I think it would be a welcome – albeit small – signal in support of emerging norms of open science for the association to affirm the principle that dissertation research should be publicly available.

Proposal

I propose the following rule be added:

In order to be considered for the ASA Dissertation Award, a nominee must commit to making his or her dissertation publicly available through a suitable academic repository by the time of the ASA meeting at which the award is granted.

Thank you for considering this request. I welcome your response, and would be happy to work with you on getting this rule – or something like it – passed by the ASA Council. Please let me know if there is anything else you need.

Sincerely,

Philip Cohen

On the ropes (Goffman review)

First a short overview, then my comments.

Alice Goffman’s book On the Run: Fugitive Life in an American City, is one of the major events in sociology of the last few years. How unusual is it for a book based on a sociology dissertation to get this treatment?

Cornel West endorsed it as “the best treatment I know of the wretched underside of neoliberal capitalist America.”  Writing in theNew York Times Book Review, Alex Kotlowitz said it was “a remarkable feat of reporting” with an “astonishing” level of detail and honesty.  The New Yorker’s Malcolm Gladwell called it “extraordinary,” and Christopher Jencks, in the New York Review of Books, predicted that it would “become an ethnographic classic.”  Tim Newburn, a highly regarded criminologist at the London School of Economics, hailed On the Run as “gloriously readable” and “sociology at its best.”

On the other hand, the book has lots of critics — here’s James Forman in the Atlantic — who think Goffman’s research subjects aren’t representative of the poor Black communities she wants to describe. And then there’s the exploitation/privilege/outsider argument, summarized by Claude Fischer (I added the links):

A typical line of criticism charges that outsiders cannot accurately describe their subjects of study. For example, one highly circulated review of Goffman’s book, alarmed at her white privilege, describes the study as “theft,” abetting “fantasies of black pathology,” and possibly causing harm by revealing to police the tricks of hiding. “Inner city Philadelphia isn’t Alice Goffman’s home,” another reviewer writes, “and it’s not her job to turn it into a jungle that needs interpreting.” A Buzzfeed writer simply tweeted, “Ban outsider ethnographies.”

Comments

On the morals and ethics, I’m not going to draw conclusions, except to say that I agree Goffman was wrong to help try to find and kill the guy who killed her friend, if what she says is true.

On the social science, I also have a limited perspective, because I don’t really read the book as social science; I think it’s much more a sociological memoir — and I don’t mean that as a criticism of ethnography, to which I would not apply that label in general.

In fact, the book is least persuasive when she tries to be most dispassionate. On the issue of representativeness, for example, Goffman clearly is wrong when she writes:

Initially I assumed that Chuck, Mike, and their friends represented an outlying group of delinquents: the bad apples of the neighborhood. After all, some of them occasionally sold marijuana and crack cocaine to local customers, and sometimes they even got into violent gun battles. I grew to understand that many young men from 6th Street were at least intermittently earning money by selling drugs, and the criminal justice entanglements of Chuck and his friends were on a par with what many other unemployed young men in the neighborhood were experiencing.

That’s a non sequitur, with the typical slippery use of “many” (which I have probably fallen into myself). The fact is her entire project was shaped by the guy she initially fell in with, and his friends, and they were by any measure an “outlying group.” If you don’t see people who have multiple running gun battles as unusual, your perspective may be a little skewed.

Patrick Sharkey’s review puts their outlier status in the context of declining violent crime — including in Philadelphia’s most violent neighborhoods:

The decision to engage in violence can be thought of as a rational response to the pressures and threats that young men perceive in this environment. But the decision to fire a gun in public space is one that is now universally rejected by every segment of American society, and it is a decision that comes with clear, long-term consequences that are understood by all of the young men in the neighborhood that Goffman studies. … As callous as it sounds given the hardships faced by this young man and the lack of choices available to him in this situation, my sense is that most residents of the block where he chose to fire his weapon would consider attempted murder to be an appropriate charge and would have been happy if he were located by the police and sentenced to prison.

In a number of passages it’s impossible to differentiate what Goffman knows versus what she was told. When the facts are wrong, that’s unfortunate. For example, she writes, “By the time Chuck entered his senior year of high school in 2002, young women outnumbered young men in his classes by more than 2:1.” Was that his perspective? People are reliably bad at estimating group sizes. The actual ratio of women to men among Black 17-year-olds living below the poverty line and enrolled in school in Philadelphia was skewed, but only 1.65-to-1.

Nevertheless, her description of the many ways the incarceration empire impinges on the daily lives of poor Black people in Philadelphia is sometimes insightful and useful. I mostly agree with Goffman’s political description and conclusions about the injustices here, and if the book does some harm it also will do some good. I would be happy to see the volume come down in this discussion and for us to treat this is a regular book — despite the inordinate attention it gets outside of academia. It’s deeply flawed but it’s worth reading. Its systematic evidence is weak but it’s thought-provoking and offers lots of food for thought in research and policy debates. It’s well-written and its topic is important. I have nothing against her and look forward to what she will do next.

That said…

My extremely shallow expertise in qualitative research is unfortunately not balanced by a vast knowledge of survey research methods. But that is my relative strength here, and on that I’d like to register an objection to Goffman’s study — both the book and her 2009 article in American Sociological Review (official link here, Google for a free copy).

There is an understandable problem with reproducibility even in the best ethnographies. As evidence for her conclusions, I simply discount her ostensibly meticulous counting of events, like this:

In that same eighteen-month period, I watched the police break down doors, search houses, and question, arrest, or chase people through houses fifty-two times. Nine times, police helicopters circled overhead and beamed searchlights onto local streets. I noted blocks taped off and traffic redirected as police searched for evidence— or, in police language, secured a crime scene— seventeen times. Fourteen times during my first eighteen months of near daily observation, I watched the police punch, choke, kick, stomp on, or beat young men with their nightsticks.

To me this just means “a lot.” First, there is no denominator. That is, no way to gauge how prevalent these events were compared with anything else. “A lot” is a fine metric for this kind of observation, because all the insights come from the details that follow, not from the recitation of frequencies. (There also is a frustrating lack of precision in these passages. Consider: “I watched the police break down doors, search houses, and question, arrest, or chase people through houses fifty-two times.” What exactly happened 52 times? What do “and” and “or” mean in that sentence?) However, if you think of it as sociological memoir, those numbers mean something because they tell you about her evolving perspective and experience. Wow, I think, if I witnessed 14 police beatings first-hand that would really affect me.

That part of the study is not reproducible. But if you do a survey while you’re doing an ethnography, you’re still doing a survey. The survey is not an ethnography. That means you should (my opinion, not a rule) make available your instrument, methods employed, and the data. Goffman has said she destroyed her field notes so they couldn’t be subpoenaed, but I don’t see why that should apply to her survey data, which could have been stripped of identifying information and processed like any other survey with sensitive material. Unlike the recitation of event counts, the survey is potentially reproducible. She reports (a few) percentages. Other researchers could conceivably conduct similar research in a different place or time and draw useful comparisons; someone might even attempt to reproduce her survey just to see if they get the same result.

Let me back up. I can, without violating the copyright rules regarding quotation length, reproduce everything she wrote about the survey portion of her study. (The much remarked upon 50-page “methodological note” at the end of the book never mentions the survey.) Here I will compare what she said in ASR versus the book, not as a gotcha exercise but because there is so little information in either that the variation between the two may be informative.

In ASR (passages excerpted across several pages of the article):

The five blocks known as 6th Street are 93 percent Black, according to a survey of residents that Chuck and I conducted in 2007. … Of the 217 households surveyed, roughly one fourth received housing vouchers. In all but two households, members reported receiving some type of government assistance in the past three years. … In the survey that Chuck and I conducted in 2007, of the 217 households that make up the 6th Street neighborhood, we found 308 men between the ages of 18 and 30 in residence. Of these men, 144 reported that they had a warrant issued for their arrest because of either delinquencies with court fines and fees or for failure to appear for a court date within the past three years. Also within the past three years, warrants had been issued to 119 men for technical violations of their probation or parole (e.g., drinking or breaking curfew).

The footnote to that last passage clarifies that “in residence” doesn’t really mean living there:

I counted men who lived in a house for three days a week or more (by their own estimates and in some cases, my knowledge) as members of the household. I included men who were absent because they were in the military, at job training programs (like JobCorp), or away in jail, prison, drug rehab centers, or halfway houses, if they expected to return to the house and had been living in the house before they went away.

From this we learn that the survey included 217 households — and also that those 217 households make up the entire 6th Street neighborhood (not its real name). That seems to imply a 100% response rate, because the 217 households surveyed corresponds to the 217 that “make up” the neighborhood. (That is an excellent response rate.) That is also important because of what comes next. If there were 217 households in the neighborhood, but she had only done interviews with, say, half of them, it would have been very surprising to find 308 men ages 18-30 living there. As it is, it’s extremely unlikely there were 308 men 18-30 living in all 217 households. The footnote says she included men who only lived there part time, or who were away for prison or other institutional spells. Frustratingly, it doesn’t say how many in the sample this applied to.

Here’s what tripped me up about that: 308 men 18-30 in 217 households is 1.4 per household — too many, I thought. So I looked at the 45 census tracts in West Philadelphia that are 75% Black or more (based on this map) in the 2005-2009 American Community Survey, and calculated the average number of men in that age range as .57 per household. (Actually I did the estimate for 18-29 because of how the data are published.) There is one tract with 1.04 men per household, but that’s an outlier. If she’s getting almost 2.5-times more than the average in her sample (1.4 versus .57), there must be a lot of missing men living in the neighborhoods not counted by the Census Bureau. Besides wondering if that’s accurate, you also ought to wonder who answered the questions about those men who weren’t actually residing in their residences.

The book’s version, unfortunately, doesn’t help clear that up:

In 2007 Chuck and I went door to door and conducted a household survey of the 6th Street neighborhood. We interviewed 308 men between the ages of eighteen and thirty. Of these young men, 144 reported that they had a warrant issued for their arrest because of either delinquencies with court fines and fees or failure to appear for a court date within the previous three years. For that same period, 119 men reported that they had been issued warrants for technical violations of their probation or parole (for example, drinking or breaking curfew).

This says they interviewed all 308 men, which seems like just an editing mistake. But then of the 144 who “reported that they had a warrant,” was that out of 308 or out of some smaller number of men whom they actually interviewed? Did someone else answer for the men who weren’t there?

Later in the book she describes the women she interviewed in the same survey:

In our 2007 household survey of the 6th Street neighborhood, 139 of 146 women reported that in the past three years, a partner, neighbor, or close male relative was either wanted by the police, serving a probation or parole sentence, going through a trial, living in a halfway house, or on house arrest. Of the women we interviewed, 67 percent said that during that same period, the police had pressured them to provide information about that person.

If she had limited the sample of women to the same age range as the men (which there’s no indication she did) then you would expect, based on the Black West Philadelphia census tracts, about 1.2 men for every woman, or about 250 women in the sample. However, if she included women of all ages, it should have been even more. Why are there only 146 women? Without any more information — and there is no more information in the book — it’s impossible to figure this out.

The last thing you might like to know if you wanted to pursue these very interesting and potentially important results, is what the survey instrument was like, and how it was administered. This was more than 450 people, a decent size survey, with good potential to yield useful results. We know they asked race, housing and other government assistance, warrant status (including the type of warrant) for the men, and criminal justice status for the partner/neighbor/relative (asked separately?) of the women. What strikes me as challenging here is asking these very sensitive questions and getting such a high response rate. Especially given all we learn from the book, knocking on doors and asking people if they have any warrants seems like it wouldn’t always be welcome. How long were the interviews? Did they make multiple visits when people weren’t home? So knowing how they did it would be very helpful for future work.

In the end, besides what we learn from On the Run itself, I hope we learn from the debate over it how we can better balance the need for protecting research subjects — while learning a lot from them — with the imperative to conduct research that is transparent, verifiable, and (as much as possible) reproducible.

ADDENDUM: The importance of the survey

The survey plays a very small part in the book and ASR article, only mentioned a few times, but it is important because it does offer a  hint of generalizability to her research. James Forman wrote in the Atlantic:

The fact that Goffman’s subjects have serious criminal histories impairs our ability to generalize from some of her findings. For example, her central characters are all wanted on warrants at one time or another, some of them repeatedly (Mike has 10 warrants altogether). To Goffman, this indicates that Philadelphia’s criminal-justice system issues too many warrants. But it may simply indicate that Mike and his friends are unusually criminally active.

Perhaps anticipating this challenge, Goffman extends her inquiry beyond the most criminally active members of the community. When she conducts a door-to-door survey of Sixth Street, she finds that about half the men there were wanted on warrants over a three-year period. This is astounding; no previous researcher has reported such a high concentration of fugitives living in one community. This raises questions that Goffman doesn’t answer with precision, but that I hope she and others will explore in the future: How many of these warrants were for failure to pay court costs—which should rarely if ever be imposed on poor people in the first place—versus something more serious, such as skipping a court date? Does fugitive status affect the lives of less criminally involved young men in the same ways it affects the lives of Mike and his friends? If it does, and if other communities harbor equally large proportions of fugitives, Goffman has discovered a profound social problem that deserves further research and a policy response.

The survey thus represents an important avenue for the book’s impact on future research.

ADDITIONAL ADDENDUM: Goffman’s response

In a statement on her website, Goffman responds to several of the recent criticisms. The first is that she participated in a conspiracy to commit murder. Her description there doesn’t change my impression of the scene she describes. But it is interesting for what it reveals about her sense of the men at the core of her story: Chuck and Mike. She writes:

In the months before he died, Chuck was actively working to preserve a precarious peace between his friends and a rival group living nearby. His sudden death was a devastating blow not only to his friends and family, but to the whole neighborhood. After Chuck was shot and killed, people in the neighborhood were putting a lot of pressure on Mike and on Chuck’s other friends to avenge his murder. It seemed that Chuck’s friends were expected to fulfill the neighborhood’s collective desire for retribution. Many of the residents were emphatic that justice should be served, and the man who killed Chuck must pay.

This surprises me, because I did not get the sense from the book that the gang rivalry Goffman described had motivated a collective conflict between entire neighborhoods. That contradicts the comment from Patrick Sharkey above, which speculates that most people in the neighborhood would have been glad to see the police arrest and incarcerate the people who conduct running gun battles in the street. Maybe Skarkey and I are victims of do-gooder liberal attitudes, and really there are neighborhoods in Philadelphia that see themselves at war with whole other neighborhoods and clamor for more shooting.

As an aside, I recently read Ghettoside: A True Story of Murder in America, in which journalist Jill Leovy argues that poor Black communities are at once over-policed — as innocent people are harassed, violated, prosecuted, and incarcerated for minor (or no) offenses — and under-policed such that murderers in Black communities are much less likely to be arrested and convicted than are those who kill Whites. This latter neglect by police contributes to the real problem of violence because it encourages informal (violent) means of addressing conflict in the community. By Leovy’s reasoning, and Sharkey might agree, majorities of people in violent Black neighborhoods would like fewer drug users incarcerated, but more of the people like Mike and whoever killed Chuck incarcerated.

Of course, I don’t know the situation in the “6th Street” neighborhood. But it’s one thing for an activist or community leader to say, “Our neighborhood stands united!”, and another for a sociologist to speak of “the neighborhood’s collective desire.” The former may be effective politics, but the latter is likely an overly-simplified research conclusion.

Which brings me finally back to the issue of Goffman’s survey. In her response, Goffman says Chuck was actively working on neighborhood peace in the months before his death, which occurred in the summer of 2007. Remarkably, that is the same time that she and Chuck conducted their survey of all 217 households in the neighborhood, with a reported 454 interviews. The timing of the survey is not completely clear, but she wrote that it was in 2007. and in an interview she said the survey was in the summer (“Yeah, so we did this survey, Chuck and I, one summer. We interviewed the households in this four-block radius…”). So it’s either a head-scratching story or an incredibly impressive image: Chuck and Goffman conducting a door-to-door household survey at the same time that he’s negotiating a delicate gang truce, interviewing every single household in the neighborhood in the weeks leading up to Chuck’s death, gathering the information that would allow Goffman to in fact accurately speak of their “collective desire.”