On Goffman’s survey

Survey methods.
Survey methods.

Jesse Singal at New York Magazine‘s Science of Us has a piece in which he tracks down and interviews a number of Alice Goffman’s respondents. This settles the question — which never should have been a real question — about whether she actually did all that deeply embedded ethnography in Philadelphia. It leaves completely unresolved, however, the issue of the errors and possible errors in the research. This reaffirms for me the conclusion in my original review that we should take the volume down in this discussion, identify errors in the research without trying to attack Goffman personally or delegitimize her career — and then learn from the affair ways that we can improve sociology (for example, by requiring that winners of the American Sociological Association dissertation award make their work publicly available).

That said, I want to comment on a couple of issues raised in Singal’s piece, and share my draft of a formal comment on the survey research Goffman reported in American Sociological Review.

First, I want to distance myself from the description by Singal of “lawyers and journalists and rival academics who all stand to benefit in various ways if they can show that On the Run doesn’t fully hold up.” I don’t see how I (or any other sociologists) benefit if Goffman’s research does not hold up. In fact, although some people think this is worth pursuing, I am also annoying some friends and colleagues by doing this.

More importantly, although it’s a small part of the article, Singal did ask Goffman about the critique of her survey, and her response (as he paraphrased it, anyway) was not satisfying to me:

Philip Cohen, a sociologist at the University of Maryland, published a blog post in which he puzzles over the strange results of a door-to-door survey Goffman says she conducted with Chuck in 2007 in On the Run. The results are implausible in a number of ways. But Goffman explained to me that this wasn’t a regular survey; it was an ethnographic survey, which involves different sampling methods and different definitions of who is and isn’t in a household. The whole point, she said, was to capture people who are rendered invisible by traditional survey methods. (Goffman said an error in the American Sociological Review paper that became On the Run is causing some of the confusion — a reference to “the 217 households that make up the 6th Street neighborhood” that should have read “the 217 households that we interviewed … ” [emphasis mine]. It’s a fix that addresses some of Cohen’s concerns, like an implied and very unlikely 100 percent response rate, but not all of them.) “I should have included a second appendix on the survey in the book,” said Goffman. “If I could do it over again, I would.”

My responses are several. First, the error of describing the 217 households as the whole neighborhood, as well as the error in the book of saying she interviewed all 308 men (when in the ASR article she reports some unknown number were absent), both go in the direction of inflating the value and quality of the survey. Maybe they are random errors, but they didn’t have a random effect.

Second, I don’t see a difference between a “regular survey” and an “ethnographic survey.” There are different survey techniques for different applications, and the techniques used determine the data and conclusions that follow. For example, in the ASR article Goffman uses the survey (rather than Census data) to report the racial composition of the neighborhood, which is not something you can do with a convenience sample, regardless of whether you are engaged in an ethnography or not.

Finally, there are no people “rendered invisible by traditional survey methods” (presumably Singal’s phrase). There are surveys that are better or worse at including people in different situations. There are “traditional” surveys — of varying quality — of homeless people, prisoners, rape victims, and illiterate peasants. I don’t know what an “ethnographic survey” is, but I don’t see why it shouldn’t include a sampling strategy, a response rate, a survey instrument, a data sharing arrangement, and thorough documentation of procedures. That second methodological appendix can be published at any time.

ASR Comment (revised June 22)

I wrote up my relatively narrow, but serious, concerns about the survey, and posted them on SocArXiv, here.

It strikes me that Goffman’s book (either the University of Chicago Press version or the trade book version) may not be subject to the same level of scrutiny that her article in ASR should have been. In fact, presumably, the book publishers took her publication in ASR as evidence of the work’s quality. And their interests are different from those of a scientific journal run by an academic society. If ASR is going to play that gatekeeping role, and it should, then ASR (and by extension ASA) should take responsibility in print for errors in its publications.

12 thoughts on “On Goffman’s survey

  1. Maybe I’m one of those “friends you are also annoying”, but how big an issue is the survey in this research? Most of the power comes from the ethnography, right? And Jesse Singal’s piece on that is convincing.

    I’m afraid to say that this confirms my skepticism of the popularity of mixed methods. I don’t fully trust an ethnographer to do a good survey (nor a survey researcher to do a good ethnography). We need *teams* of people with complementary skills and experience to do good mixed methods research and shouldn’t be encouraging our students to try to do it all.

    Liked by 1 person

  2. I absolutely think you should write it up as a formal ASR comment (though of course it’s easy for me to tell you how to spend your time).

    Measuring hard to reach populations through survey research is a serious methodological problem and not the kind of thing that should be conducted on a winging it basis or described parenthetically and in footnotes. I suspect the managing editor didn’t even notice the survey in the initial skim to assign reviewers (and why should he/she, it’s only mentioned in a few places) and assigned it to ethnography reviewers who could be forgiven for understanding the survey as just an icing on the cake kind of thing and not noticing the sex ratio.

    It bothers me that there was no formal write-up of the methods, no citations to the literature on proxy responses or household rosters to capture itinerant men, and no comparison to baseline Census or criminal justice data. That all needs to be in a comment.

    I’m thinking it might be best to have multiple comments, like they published in response to Udry 2000, with yours being on the survey and others about qualitative discrepancies, most notably the hospital thing. For instance the ASR says Alex was arrested for a minor parole violation, but Josh told Singal “Now, mind you, his arrest was for some crimes that were very serious, so take that into consideration.” It’s theoretically possible that Josh is confusing or conflating the underlying offense and the parole violation, but as stated it sounds like Josh is saying Alex had a fresh warrant out for a felony.


    1. Thanks. No idea how ASR (outgoing or incoming) editors would respond.

      On the ASR reviewers, she identified them in the book: Steven Lopez, Philip Kasinitz, Jack Katz, and Patricia Adler. I don’t know their expertise, or what the review process was like.


      1. I can’t quite tell if you’re kidding about not “knowing their expertise”? Jack Katz is a legend in criminology and the sociology of deviance — the father of cultural criminology, really — and if you haven’t read his Seductions of Crime you should really put on your list. Kasinitz is also a famous urban ethnographer, Lopez has done some interesting participant observation, and Patricia Adler is if nothing else nationally famous for being pushed out of UC Boulder for making her Deviance students dress up as prostitutes. These are probably some of the most prominent, influential, and well known ethnographic/cultural sociologists currently working.

        Which makes it all the more embarrassing that they missed so many glaring problems in Goffman’s messy, messy method.

        I do hope you publish your comment to ASR. Your critiques don’t seem as bloodthirsty as some others. There’s no need to go for blood, but that doesn’t mean the book’s claims and methods don’t deserve much closer inspection.


        1. Thanks. I meant I don’t know their complete expertise, that is, relevant to Rossman’s comment about survey research methods. Just acknowledging that I don’t know what came up in the review process.


          1. Ah, whoops, that makes sense. Well, maybe my quick sketches of their work will help somebody else reading your blog. 🙂


  3. I still do not understand what your concern is regarding 217 households/308 men.

    Is it:

    1. she did not visit 217 households?
    2. All 217 households responded?
    3. 308 men from 217 households responded, whereas using ACS survvey, you expected less than 200 men to be present?
    4. That all 308 men responded?

    What is your issue here?


Comments welcome (may be moderated)

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s