Category Archives: Research reports

On the ropes (Goffman review)

westphilly

First a short overview, then my comments.

Alice Goffman’s book On the Run: Fugitive Life in an American City, is one of the major events in sociology of the last few years. How unusual is it for a book based on a sociology dissertation to get this treatment?

Cornel West endorsed it as “the best treatment I know of the wretched underside of neoliberal capitalist America.”  Writing in theNew York Times Book Review, Alex Kotlowitz said it was “a remarkable feat of reporting” with an “astonishing” level of detail and honesty.  The New Yorker’s Malcolm Gladwell called it “extraordinary,” and Christopher Jencks, in the New York Review of Books, predicted that it would “become an ethnographic classic.”  Tim Newburn, a highly regarded criminologist at the London School of Economics, hailed On the Run as “gloriously readable” and “sociology at its best.”

On the other hand, the book has lots of critics — here’s James Forman in the Atlantic — who think Goffman’s research subjects aren’t representative of the poor Black communities she wants to describe. And then there’s the exploitation/privilege/outsider argument, summarized by Claude Fischer (I added the links):

A typical line of criticism charges that outsiders cannot accurately describe their subjects of study. For example, one highly circulated review of Goffman’s book, alarmed at her white privilege, describes the study as “theft,” abetting “fantasies of black pathology,” and possibly causing harm by revealing to police the tricks of hiding. “Inner city Philadelphia isn’t Alice Goffman’s home,” another reviewer writes, “and it’s not her job to turn it into a jungle that needs interpreting.” A Buzzfeed writer simply tweeted, “Ban outsider ethnographies.”

Comments

On the morals and ethics, I’m not going to draw conclusions, except to say that I agree Goffman was wrong to help try to find and kill the guy who killed her friend, if what she says is true.

On the social science, I also have a limited perspective, because I don’t really read the book as social science; I think it’s much more a sociological memoir — and I don’t mean that as a criticism of ethnography, to which I would not apply that label in general.

In fact, the book is least persuasive when she tries to be most dispassionate. On the issue of representativeness, for example, Goffman clearly is wrong when she writes:

Initially I assumed that Chuck, Mike, and their friends represented an outlying group of delinquents: the bad apples of the neighborhood. After all, some of them occasionally sold marijuana and crack cocaine to local customers, and sometimes they even got into violent gun battles. I grew to understand that many young men from 6th Street were at least intermittently earning money by selling drugs, and the criminal justice entanglements of Chuck and his friends were on a par with what many other unemployed young men in the neighborhood were experiencing.

That’s a non sequitur, with the typical slippery use of “many” (which I have probably fallen into myself). The fact is her entire project was shaped by the guy she initially fell in with, and his friends, and they were by any measure an “outlying group.” If you don’t see people who have multiple running gun battles as unusual, your perspective may be a little skewed.

Patrick Sharkey’s review puts their outlier status in the context of declining violent crime — including in Philadelphia’s most violent neighborhoods:

The decision to engage in violence can be thought of as a rational response to the pressures and threats that young men perceive in this environment. But the decision to fire a gun in public space is one that is now universally rejected by every segment of American society, and it is a decision that comes with clear, long-term consequences that are understood by all of the young men in the neighborhood that Goffman studies. … As callous as it sounds given the hardships faced by this young man and the lack of choices available to him in this situation, my sense is that most residents of the block where he chose to fire his weapon would consider attempted murder to be an appropriate charge and would have been happy if he were located by the police and sentenced to prison.

In a number of passages it’s impossible to differentiate what Goffman knows versus what she was told. When the facts are wrong, that’s unfortunate. For example, she writes, “By the time Chuck entered his senior year of high school in 2002, young women outnumbered young men in his classes by more than 2:1.” Was that his perspective? People are reliably bad at estimating group sizes. The actual ratio of women to men among Black 17-year-olds living below the poverty line and enrolled in school in Philadelphia was skewed, but only 1.65-to-1.

Nevertheless, her description of the many ways the incarceration empire impinges on the daily lives of poor Black people in Philadelphia is sometimes insightful and useful. I mostly agree with Goffman’s political description and conclusions about the injustices here, and if the book does some harm it also will do some good. I would be happy to see the volume come down in this discussion and for us to treat this is a regular book — despite the inordinate attention it gets outside of academia. It’s deeply flawed but it’s worth reading. Its systematic evidence is weak but it’s thought-provoking and offers lots of food for thought in research and policy debates. It’s well-written and its topic is important. I have nothing against her and look forward to what she will do next.

That said…

My extremely shallow expertise in qualitative research is unfortunately not balanced by a vast knowledge of survey research methods. But that is my relative strength here, and on that I’d like to register an objection to Goffman’s study — both the book and her 2009 article in American Sociological Review (official link here, Google for a free copy).

There is an understandable problem with reproducibility even in the best ethnographies. As evidence for her conclusions, I simply discount her ostensibly meticulous counting of events, like this:

In that same eighteen-month period, I watched the police break down doors, search houses, and question, arrest, or chase people through houses fifty-two times. Nine times, police helicopters circled overhead and beamed searchlights onto local streets. I noted blocks taped off and traffic redirected as police searched for evidence— or, in police language, secured a crime scene— seventeen times. Fourteen times during my first eighteen months of near daily observation, I watched the police punch, choke, kick, stomp on, or beat young men with their nightsticks.

To me this just means “a lot.” First, there is no denominator. That is, no way to gauge how prevalent these events were compared with anything else. “A lot” is a fine metric for this kind of observation, because all the insights come from the details that follow, not from the recitation of frequencies. (There also is a frustrating lack of precision in these passages. Consider: “I watched the police break down doors, search houses, and question, arrest, or chase people through houses fifty-two times.” What exactly happened 52 times? What do “and” and “or” mean in that sentence?) However, if you think of it as sociological memoir, those numbers mean something because they tell you about her evolving perspective and experience. Wow, I think, if I witnessed 14 police beatings first-hand that would really affect me.

That part of the study is not reproducible. But if you do a survey while you’re doing an ethnography, you’re still doing a survey. The survey is not an ethnography. That means you should (my opinion, not a rule) make available your instrument, methods employed, and the data. Goffman has said she destroyed her field notes so they couldn’t be subpoenaed, but I don’t see why that should apply to her survey data, which could have been stripped of identifying information and processed like any other survey with sensitive material. Unlike the recitation of event counts, the survey is potentially reproducible. She reports (a few) percentages. Other researchers could conceivably conduct similar research in a different place or time and draw useful comparisons; someone might even attempt to reproduce her survey just to see if they get the same result.

Let me back up. I can, without violating the copyright rules regarding quotation length, reproduce everything she wrote about the survey portion of her study. (The much remarked upon 50-page “methodological note” at the end of the book never mentions the survey.) Here I will compare what she said in ASR versus the book, not as a gotcha exercise but because there is so little information in either that the variation between the two may be informative.

In ASR (passages excerpted across several pages of the article):

The five blocks known as 6th Street are 93 percent Black, according to a survey of residents that Chuck and I conducted in 2007. … Of the 217 households surveyed, roughly one fourth received housing vouchers. In all but two households, members reported receiving some type of government assistance in the past three years. … In the survey that Chuck and I conducted in 2007, of the 217 households that make up the 6th Street neighborhood, we found 308 men between the ages of 18 and 30 in residence. Of these men, 144 reported that they had a warrant issued for their arrest because of either delinquencies with court fines and fees or for failure to appear for a court date within the past three years. Also within the past three years, warrants had been issued to 119 men for technical violations of their probation or parole (e.g., drinking or breaking curfew).

The footnote to that last passage clarifies that “in residence” doesn’t really mean living there:

I counted men who lived in a house for three days a week or more (by their own estimates and in some cases, my knowledge) as members of the household. I included men who were absent because they were in the military, at job training programs (like JobCorp), or away in jail, prison, drug rehab centers, or halfway houses, if they expected to return to the house and had been living in the house before they went away.

From this we learn that the survey included 217 households — and also that those 217 households make up the entire 6th Street neighborhood (not its real name). That seems to imply a 100% response rate, because the 217 households surveyed corresponds to the 217 that “make up” the neighborhood. (That is an excellent response rate.) That is also important because of what comes next. If there were 217 households in the neighborhood, but she had only done interviews with, say, half of them, it would have been very surprising to find 308 men ages 18-30 living there. As it is, it’s extremely unlikely there were 308 men 18-30 living in all 217 households. The footnote says she included men who only lived there part time, or who were away for prison or other institutional spells. Frustratingly, it doesn’t say how many in the sample this applied to.

Here’s what tripped me up about that: 308 men 18-30 in 217 households is 1.4 per household — too many, I thought. So I looked at the 45 census tracts in West Philadelphia that are 75% Black or more (based on this map) in the 2005-2009 American Community Survey, and calculated the average number of men in that age range as .57 per household. (Actually I did the estimate for 18-29 because of how the data are published.) There is one tract with 1.04 men per household, but that’s an outlier. If she’s getting almost 2.5-times more than the average in her sample (1.4 versus .57), there must be a lot of missing men living in the neighborhoods not counted by the Census Bureau. Besides wondering if that’s accurate, you also ought to wonder who answered the questions about those men who weren’t actually residing in their residences.

The book’s version, unfortunately, doesn’t help clear that up:

In 2007 Chuck and I went door to door and conducted a household survey of the 6th Street neighborhood. We interviewed 308 men between the ages of eighteen and thirty. Of these young men, 144 reported that they had a warrant issued for their arrest because of either delinquencies with court fines and fees or failure to appear for a court date within the previous three years. For that same period, 119 men reported that they had been issued warrants for technical violations of their probation or parole (for example, drinking or breaking curfew).

This says they interviewed all 308 men, which seems like just an editing mistake. But then of the 144 who “reported that they had a warrant,” was that out of 308 or out of some smaller number of men whom they actually interviewed? Did someone else answer for the men who weren’t there?

Later in the book she describes the women she interviewed in the same survey:

In our 2007 household survey of the 6th Street neighborhood, 139 of 146 women reported that in the past three years, a partner, neighbor, or close male relative was either wanted by the police, serving a probation or parole sentence, going through a trial, living in a halfway house, or on house arrest. Of the women we interviewed, 67 percent said that during that same period, the police had pressured them to provide information about that person.

If she had limited the sample of women to the same age range as the men (which there’s no indication she did) then you would expect, based on the Black West Philadelphia census tracts, about 1.2 men for every woman, or about 250 women in the sample. However, if she included women of all ages, it should have been even more. Why are there only 146 women? Without any more information — and there is no more information in the book — it’s impossible to figure this out.

The last thing you might like to know if you wanted to pursue these very interesting and potentially important results, is what the survey instrument was like, and how it was administered. This was more than 450 people, a decent size survey, with good potential to yield useful results. We know they asked race, housing and other government assistance, warrant status (including the type of warrant) for the men, and criminal justice status for the partner/neighbor/relative (asked separately?) of the women. What strikes me as challenging here is asking these very sensitive questions and getting such a high response rate. Especially given all we learn from the book, knocking on doors and asking people if they have any warrants seems like it wouldn’t always be welcome. How long were the interviews? Did they make multiple visits when people weren’t home? So knowing how they did it would be very helpful for future work.

In the end, besides what we learn from On the Run itself, I hope we learn from the debate over it how we can better balance the need for protecting research subjects — while learning a lot from them — with the imperative to conduct research that is transparent, verifiable, and (as much as possible) reproducible.

ADDENDUM: The importance of the survey

The survey plays a very small part in the book and ASR article, only mentioned a few times, but it is important because it does offer a  hint of generalizability to her research. James Forman wrote in the Atlantic:

The fact that Goffman’s subjects have serious criminal histories impairs our ability to generalize from some of her findings. For example, her central characters are all wanted on warrants at one time or another, some of them repeatedly (Mike has 10 warrants altogether). To Goffman, this indicates that Philadelphia’s criminal-justice system issues too many warrants. But it may simply indicate that Mike and his friends are unusually criminally active.

Perhaps anticipating this challenge, Goffman extends her inquiry beyond the most criminally active members of the community. When she conducts a door-to-door survey of Sixth Street, she finds that about half the men there were wanted on warrants over a three-year period. This is astounding; no previous researcher has reported such a high concentration of fugitives living in one community. This raises questions that Goffman doesn’t answer with precision, but that I hope she and others will explore in the future: How many of these warrants were for failure to pay court costs—which should rarely if ever be imposed on poor people in the first place—versus something more serious, such as skipping a court date? Does fugitive status affect the lives of less criminally involved young men in the same ways it affects the lives of Mike and his friends? If it does, and if other communities harbor equally large proportions of fugitives, Goffman has discovered a profound social problem that deserves further research and a policy response.

The survey thus represents an important avenue for the book’s impact on future research.

ADDITIONAL ADDENDUM: Goffman’s response

In a statement on her website, Goffman responds to several of the recent criticisms. The first is that she participated in a conspiracy to commit murder. Her description there doesn’t change my impression of the scene she describes. But it is interesting for what it reveals about her sense of the men at the core of her story: Chuck and Mike. She writes:

In the months before he died, Chuck was actively working to preserve a precarious peace between his friends and a rival group living nearby. His sudden death was a devastating blow not only to his friends and family, but to the whole neighborhood. After Chuck was shot and killed, people in the neighborhood were putting a lot of pressure on Mike and on Chuck’s other friends to avenge his murder. It seemed that Chuck’s friends were expected to fulfill the neighborhood’s collective desire for retribution. Many of the residents were emphatic that justice should be served, and the man who killed Chuck must pay.

This surprises me, because I did not get the sense from the book that the gang rivalry Goffman described had motivated a collective conflict between entire neighborhoods. That contradicts the comment from Patrick Sharkey above, which speculates that most people in the neighborhood would have been glad to see the police arrest and incarcerate the people who conduct running gun battles in the street. Maybe Skarkey and I are victims of do-gooder liberal attitudes, and really there are neighborhoods in Philadelphia that see themselves at war with whole other neighborhoods and clamor for more shooting.

As an aside, I recently read Ghettoside: A True Story of Murder in America, in which journalist Jill Leovy argues that poor Black communities are at once over-policed — as innocent people are harassed, violated, prosecuted, and incarcerated for minor (or no) offenses — and under-policed such that murderers in Black communities are much less likely to be arrested and convicted than are those who kill Whites. This latter neglect by police contributes to the real problem of violence because it encourages informal (violent) means of addressing conflict in the community. By Leovy’s reasoning, and Sharkey might agree, majorities of people in violent Black neighborhoods would like fewer drug users incarcerated, but more of the people like Mike and whoever killed Chuck incarcerated.

Of course, I don’t know the situation in the “6th Street” neighborhood. But it’s one thing for an activist or community leader to say, “Our neighborhood stands united!”, and another for a sociologist to speak of “the neighborhood’s collective desire.” The former may be effective politics, but the latter is likely an overly-simplified research conclusion.

Which brings me finally back to the issue of Goffman’s survey. In her response, Goffman says Chuck was actively working on neighborhood peace in the months before his death, which occurred in the summer of 2007. Remarkably, that is the same time that she and Chuck conducted their survey of all 217 households in the neighborhood, with a reported 454 interviews. The timing of the survey is not completely clear, but she wrote that it was in 2007. and in an interview she said the survey was in the summer (“Yeah, so we did this survey, Chuck and I, one summer. We interviewed the households in this four-block radius…”). So it’s either a head-scratching story or an incredibly impressive image: Chuck and Goffman conducting a door-to-door household survey at the same time that he’s negotiating a delicate gang truce, interviewing every single household in the neighborhood in the weeks leading up to Chuck’s death, gathering the information that would allow Goffman to in fact accurately speak of their “collective desire.”

39 Comments

Filed under Research reports

Stop me before I fake again

In light of the news on social science fraud, I thought it was a good time to report on an experiment I did. I realize my results are startling, and I welcome the bright light of scrutiny that such findings might now attract.

The following information is fake.

An employee training program in a major city promises basic job skills and as well as job search assistance for people with a high school degree and no further education, ages 23-52 in 2012. Due to an unusual staffing practice, new applications were for a period in 2012 allocated at random to one of two caseworkers. One provided the basic services promised but nothing extra. The other embellished his services with extensive coaching on such “soft skills” as “mainstream” speech patterns, appropriate dress for the workplace, and a hard work ethic, among other elements. The program surveyed the participants in 2014 to see what their earnings were in the previous 12 months. The data provided to me does not include any information on response rates, or any information about those who did not respond. And it only includes participants who were employed at least part-time in 2014. Fortunately, the program also recorded which staff member each participant was assigned to.

Since this provides such an excellent opportunity for studying the effects of soft skills training, I think it’s worth publishing despite these obvious weaknesses. To help with the data collection and analysis, I got a grant from Big Neoliberal, a non-partisan foundation.

The data includes 1040 participants, 500 of whom had the bare-bones service and 540 of whom had the soft-skills add-on, which I refer to as the “treatment.” These are the descriptive statistics:

fake-descriptives

As you can see, the treatment group had higher earnings in 2014. The difference in logged annual earnings between the two groups is significant at p

fake-ols-results

As you can see in Model 1, the Black workers in 2014 earned significantly less than the White workers. This gap of .15 logged earnings points, or about 15%, is consistent with previous research on the race wage gap among high school graduates. Model 2 shows that the treatment training apparently was effective, raising earnings about 11%. However, The interactions in Model 3 confirm that the benefits of the treatment were concentrated among the Black workers. The non-Black workers did not receive a significant benefit, and the treatment effect among Black workers basically wiped out the race gap.

The effects are illustrated, with predicted probabilities, in this figure:

fake-marginsplot

Soft skills are awesome.

I have put the data file, in Stata format, here.

Discussion

What would you do if you saw this in a paper or at a conference? Would you suspect it was fake? Why or why not?

I confess I never seriously thought of faking a research study before. In my day coming up in sociology, people didn’t share code and datasets much (it was never compulsory). I always figured if someone was faking they were just changing the numbers on their tables to look better. I assumed this happens to some unknown, and unknowable, extent.

So when I heard about the Lacour & Green scandal, I thought whoever did it was tremendously clever. But when I looked into it more, I thought it was not such rocket science. So I gave it a try.

Details

I downloaded a sample of adults 25-54 from the 2014 ACS via IPUMS, with annual earnings, education, age, sex, race and Hispanic origin. I set the sample parameters to meet the conditions above, and then I applied the treatment, like this:

First, I randomly selected the treatment group:

gen temp = runiform()
gen treatment=0
replace treatment = 1 if temp >= .5
drop temp

Then I generated the basic effect, and the Black interaction effect:

gen effect = rnormal(.08,.05)
gen beffect = rnormal(.15,.05)

Starting with the logged wage variable, lnwage, I added the basic effect to all the treated subjects:

replace newlnwage = lnwage+effect if treatment==1

Then added the Black interaction effect to the treated Black subjects, and subtracted it from the non-treated ones.

replace newlnwage = newlnwage+beffect if (treatment==1 & black==1)
replace newlnwage = newlnwage-beffect if (treatment==0 & black==1)

This isn’t ideal, but when I just added the effect I didn’t have a significant Black deficit in the baseline model, so that seemed fishy.

That’s it. I spent about 20 minutes trying different parameters for the fake effects, trying to get them to seem reasonable. The whole thing took about an hour (not counting the write-up).

I put the complete fake files here: code, data.

Would I get caught for this? What are we going to do about this?

BUSTED UPDATE:

In the comments, ssgrad notices that if you exponentiate (unlog) the incomes, you get a funny list — some are binned at whole numbers, as you would expect from a survey of incomes, and some are random-looking and go out to multiple decimal places. For example, one person reports an even $25,000, and another supposedly reports $25251.37. This wouldn’t show up in the descriptive statistics, but is kind of obvious in a list. Here is a list of people with incomes between $20000 and $26000, broken down by race and treatment status. I rounded to whole numbers because even without the decimal points you can see that the only people who report normal incomes are non-Blacks in the non-treatment group. Busted!

fake-busted-tableSo, that only took a day — with a crowd-sourced team of thousands of social scientists poring over the replication file. Faith in the system restored?

11 Comments

Filed under In the news, Research reports

How random error and dirty data made Regnerus even wronger than we thought

The news is nothing I have to say, but the new article, available in prepublication form, by Simon Cheng and Brian Powell, which methodically flays the infamous Regnerus paper, leaving nothing but a wisp of foul-smelling ill-will trailing from its remains. (The paper is here, where it is paywalled; feel free to email me. Follow the whole story at the Regnerus tag.)

Cheng and Powell reanalyzed the Regnerus data, the New Family Structures Survey (NFSS), and see what would happen if Regnerus had done the data processing and analysis right. This goes beyond the logical flaws and biases that were inherent in the study design (discussed here), to find the coding and analysis errors. A few examples:

  • So much for “raised by…” 24 of the 236 people coded as having a “lesbian mother” or “gay father” — because they reported one of their parents ever had a same-sex romantic relationship (I’ll use LM and GF here to refer to Regnerus’s codes, not reality) — never lived with the parent in question! We had known previously that a large number (138) had never lived with the partner in the romantic relationship, but this is a whole nother level of wrong. A total of 58 of the LM/GF sample were reported to have lived with the supposedly gay or lesbian parent for a single year or less.
  • Bad cases. The most ridiculous is the “25 year-old man who reports that his father had a romantic relationship with another man, but also reports that he (the respondent) was 7-feet 8-inches tall, weighed 88 pounds, was married 8 times and had 8 children.” Another reported being arrested for the first time at age 1. Real data collectors scrutinize cases like that and throw them out or find a way to fix them. (Really good data collectors stop the person — or the data entry — right when they say something outrageous, to see if they’re sure.)
  • Illogical cases. There are a lot of these, including the person who reported “having always lived alone but also claims to have always lived with mother, father, and two grandparents.”

Then there are a series of bad analysis and modeling decisions Regnerus made, such as coding people who refused to answer a question as 0 instead of missing, or using the wrong kind of statistical model for the particular outcome.

When they get done with it, there really is no reliable, significant negative outcome associated with having lived any appreciable amount of time with a parent who might have been gay or lesbian. There’s more to it, but I don’t want to discourage you from reading the paper.

Random error, correlated outcome

Some of the “misclassified or uncertain” cases also report serious problems in adulthood, exhibiting higher-than-average rates of suicidality, depression, drinking to get drunk, and having a poor relationship with their mothers. So those could be people whose difficult lives rendered them unable to complete the life history calendar correctly. But there is also a chance that, like the 7’8″ guy, there are people just answering some of the question at random. These were people taking the survey alone on a computer, with no supervision, and getting paid to be part of the sample. Clicking at random is not out of the question (one person only took 10 minutes to complete the lengthy survey).

Contrary to what you might assume, clicking at random does not always produce random results. I’ll illustrate this with an example. First, here’s another tidbit from Regnerus, which might fit this point. Speaking to some Franciscans in 2014, Regnerus (just after 9:00 of this video) was going on about sexual fluidity as a condition of modernity, when he dropped in this fact from the NFSS:

Despite comprising a mere 1.3 percent of the population, respondents in the NFSS [New Family Structures Survey] who said that their mothers have had a same-sex sexual relationship made up 15 [50?] percent of all the asexual identifiers in the NFSS. So, 15 [50?] percent of them come from 1.3 percent of the population. [I originally transcribed those as 50%, but on second listening I think he said 15%, but I can’t be sure.]

His raised eyebrow here is to indicate the deeply depraved nature of lesbian mothers — maybe it’s genetic, or maybe it’s child abuse — but… he lets the numbers speak for themselves. Lesbian mothers, asexual children.

Here’s how this works. If you are trying to find people in two rare conditions — for example, those with lesbian mothers and those who are asexual — and a small portion of your sample answers questions at random, not only will you have a relatively large number of false positives on your conditions, your rare conditions will also falsely appear to be correlated.

I’m sure I didn’t discover this, and I don’t have a mathematical proof for it, but it’s logical. And I confirmed it with an experiment, as follows.

Say you have a sample of 1000 people, and you’re studying two conditions that occur on average in one out of every 500 cases. I’ll call them “climbing Mt. Everest” and “going to the moon.” In your thousand cases, you will on average have 2 people who did each thing. The chances that the same person did both are probably really low (you do the maths). But, if just 1% of your sample — 10 people — answer those two yes/no questions at random, look out!

I created this scenario using Excel’s random-number function. With 990 people answering truthfully — that is, given a 1/500 chance of saying yes to each question — and 10 answering them both randomly, this is what I got: 6 people who had climbed Mt. Everest, and 8 people who had gone to the moon. But shockingly, there were 4 people who had done both — that is 67% of the mountain climbers and 50% of the moonshotters. You can’t know, from looking at the data, but I can, that all of the people who went on both adventures were in the tiny group of random answerers.

Here are the 1000 cases in random order, with green showing Everest-only cases, blue showing moon-only cases, and red showing positive answers to both questions. And here’s the statistic: in the total sample — 990 serious survey takers and 10 jokers — the correlation between climbing Mt. Everest and going to the moon is .53! Click to enlarge:

rare event errors.xlsx

Maybe Regnerus is just an incredibly, irresponsibly bad researcher, who didn’t conduct the simplest data checks before rushing to publish his paper. Or maybe he is a diabolical genius, and he realized that high random error rates in both his rare independent variable and his rare dependent variables would produce results showing poor outcomes for children of gays and lesbians.

In the Cheng and Powell paper, their various procedures and corrections wipe out many of Regenerus’s negative outcomes for GF/LM respondents before they tackle the “misclassified or uncertain” cases. But when they do that, some of the last coefficients to fall to non-significance are indeed relatively rare: having suicidal thoughts (7%), not being “entirely heterosexual” (15%), having had an STI (11%), and having had forced sex (13%). Each of these becomes non-significant when the bad cases are controlled in the Cheng and Powell models. I haven’t worked out a proof (ever), but I reckon that the rarer they are, the more likely they are to be correlated with the rare independent variable (LM/GF) if some people are answering at random — which they apparently were.

Anyway, the Cheng and Powell paper speaks for itself. But I find it interesting that unchecked data error produces false positive (that is, negative) outcomes for marginal groups. Look out!

13 Comments

Filed under Research reports

Book reviews: Sex & Unisex, among others

Or, why your important editor friend should publish my book reviews

I love writing book reviews. In fact, one occupation I really aspire to is “essayist.” How do I get that job? (Wait, I think I figured it out.) Getting a book review assignment is what makes me read a whole book carefully, something I always enjoy but rarely do.

My latest is a review of the excellent Sex and Unisex: Fashion, Feminism, and the Sexual Revolution, by Jo Paoletti, published online by Boston Review. And they found this great example of unixex fashion from the 1969 Sears catalog:

sears69

Here’s a taste of the review:

But if fashion has a hierarchy, it also has a social context. In the newly released book Sex and Unisex, Jo Paoletti tries to understand that context as it gave rise to a revolution that almost was—the unisex fashion trend that, in hindsight, appears awkwardly sandwiched between the conservative, gender-conformist 1950s and the Disney princess tidal wave of the 1990s. For a brief time, little boys and girls wore the same cowboy shirts tucked into identical blue jeans, some men and women wore the same ponchos and turtlenecks, and male and female TV space travelers wore identical outfits.

To the Rick Santorums of today’s culture wars, the 1960s were, in Paoletti’s words, “self-indulgent and aimless—just a bunch of free-love hippies waving protest signs and getting high.” But the unisex moment that era begat was actually “emblematic of a very complicated—and unfinished—conversation about sex, gender, and sexuality.” That conversation encompassed freedom and individualism, yes, but also civil rights, sexual orientation, and the emerging science of gender identity. In Paoletti’s telling, the unisex movement generated unprecedented clothing options for women, men, and children as well as a fascinating series of lawsuits in which the wayward enemies of conformity—mostly men—put their feet down against the arbitrary, controlling ways of an establishment that was temporarily back on its heels.

Help an essayist out

Writing book reviews, especially as part of my job, is a real privilege. If a friend of yours is the editor of another important periodical that publishes book reviews (or if you are such an editor), I hope you’ll recommend me. Here’s a list of the ones I’ve done, to help the cause.

Magazines (or their websites)

  • Sex and Unisex: Fashion, Feminism, and the Sexual Revolution, by Jo Paoletti (Boston Reviewlink)
  • A Troublesome Inheritance: Genes, Race and Human History, by Nicholas Wade (Boston Review  | link)
  • The Richer Sex, by Liza Mundy, and The End of Men, by Hanna Rosin (Boston Reviewlink)
  • The Rise of Women: The Growing Gender Gap in Education and What it Means for American Schools, by Thomas DiPrete and Claudia Buchmann (The Atlantic | link)

On the blog

  • The Sacred Project of American Sociology, by Christian Smith | link
  • What To Expect When No One’s Expecting: America’s Coming Demographic Disaster, by Jonathan Last | link
  • The Price of Inequality: How Today’s Divided Society Endangers Our Future, by Joseph E. Stiglitz | link
  • A roundup of good books from 2011 | link

Academic journals

  • Cut Adrift: Families in Insecure Times, by Marianne Cooper (Gender & Society | preprint)
  • Documenting Desegregation: Racial and Gender Segregation in Private-Sector Employment Since the Civil Rights Act, by Kevin Stainback and Donald Tomaskovic-Devey (Work and Occupations | preprint)
  • Occupational Ghettos: The Worldwide Segregation of Women and Men, Maria Charles and David B. Grusky (Contemporary Sociology | JSTOR).
  • Glass Ceilings and Asian Americans: The New Face of Workplace Barriers, by Deborah Woo (Review of Radical Political Economics | link)
  • The Ties That Bind: Perspectives on Cohabitation and Marriage, edited by Linda J. Waite et al. (Contemporary Sociology  | link)
  • Persistent Disparity: Race and Economic Inequality since 1945, by William A. Darity, Jr. and Samuel L. Myers, Jr. (Review of Radical Political Economicslink)
  • The Racial Contract, by Charles W. Mills. (Review of Radical Political Economicslink)

3 Comments

Filed under Me @ work, Research reports

Evolved: Nicholas Wade critique trilogy complete

Photo by Philip Cohen from Flickr Creative Commons.

Photo by Philip Cohen from Flickr Creative Commons.

After writing a book review, and further critique of Nicholas Wade’s book, A Troublesome Inheritance: Genes, Race and Human History, I have completed the trilogy with a piece forthcoming in the Annals of the American Academy of Political and Social Science.

The final article includes much of what was in the earlier pieces but edited, with more sources, and with additional material on the social science context. I have posted a pre-publication version here as, “Troubling race in the social sciences.”

Here is the conclusion:

It may be the case, as Freese (2008:S1) claims, that “the vast majority of individual-level outcomes of abiding sociological interest are genetically influenced to a substantial degree.” And it may be true that the historical migration and dispersion of people around the planet has resulted in genetically identifiable clusters that sometimes follow the contours of commonly understood races. But it does not follow that genetics explains the relative status and wellbeing of today’s racially-identified groups or their societies. In fact, these two lines of inquiry – the genetics of behavior and the geographic variation in human genetics – do not depend upon each other; the strong case linking them is the contemporary expression of scientific racism. The publication of Wade’s Troublesome Inheritance serves as a potent warning of the continued resonance of racially deterministic narratives of social inequality.

I’ve learned a lot from working on this. I hope you find it helpful.

13 Comments

Filed under Research reports

Is the Moynihan-backlash chilling effect a myth?

Recently we have seen the revival of the idea that some faction of the political left (liberal, progressive, or radical) is silencing debate through “political correctness,” as retold, for example, by Jonathan Chait. Similarly, there is a push by those reviving the 1965 Moynihan Report (neo-Moynihanists?) to advance a narrative in which venomous race police attacked Moynihan with such force that liberal social scientists were scared off the topic of “cultural explanations” (especially about marriage) for Black poverty and inequality.

This Moynihan chilling effect narrative got a recent boost from Nicholas Kristof in the New York Times. As Kristof tells it, “The taboo on careful research on family structure and poverty was broken by William Julius Wilson, an eminent black sociologist.” Kristof lifted that description from this recent article by McLanahan and Jencks (which he cites elsewhere in the column). They wrote:

For the next two decades [after 1965] few scholars chose to investigate the effects of father absence, lest they too be demonized if their findings supported Moynihan’s argument. Fortunately, America’s best-known black sociologist, William Julius Wilson, broke this taboo in 1987, providing a candid assessment of the black family and its problems in The Truly Disadvantaged.

This narrative, which seems to grow more simplistic and linear with each telling, is just not true. In fact, it’s pretty bizarre.

Herbert Gans in 2011 attributed the story to William Julius Wilson’s first chapter of The Truly Disadvantaged (1987), in which he said that, after the criticism of Moynihan, “liberal scholars shied away from researching behavior construed as unflattering or stigmatizing.” Wilson told a version of the story in 2009, in which the ideology expressed by “militant black spokespersons” spread to “black academics and intellectuals,” creating an atmosphere of “racial chauvinism,” in which “poor African Americans were described as resilient and were seen as imaginatively adapting to an oppressive society” when they engaged in “self destructive” aspects of “ghetto life.” (These aren’t scare quotes, I’m just being careful to use Wilson’s words.) In this vein of research,

…this approach sidesteps the issue altogether by denying that social dislocations in the inner city represent any special problem. Researchers who emphasized these dislocations were denounced, even those who rejected the assumption of individual responsibility for poverty and welfare, and focused instead on the structure or roots of these problems.

Accordingly, in the early 1970s, unlike in the middle 1960s, there was little motivation to develop a research agenda that pursued the structural and cultural roots of ghetto social dislocations. The vitriolic attacks and acrimonious debate that characterized this controversy proved to be too intimidating to scholars, particularly to liberal scholars. Indeed, in the aftermath of this controversy and in an effort to protect their work from the charge of racism, or of blaming the victim, many liberal social scientists tended to avoid describing any behavior that could be construed as unflattering or stigmatizing to people of color. Accordingly, until the mid-1980s and well after this controversy had subsided, social problems in the inner-city ghetto did not attract serious research attention.

Wilson includes this very strong causal statement: “the controversy over the Moynihan Report resulted in a persistent taboo on cultural explanations to help explain social problems in the poor black community.” I would love to see any direct evidence — eyewitness accounts or personal testimony — of this chilling effect on researchers.

If you read it generously, Wilson is mostly saying that there was a fall-off in the kind of argument that he preferred, one that “pursued the structural and cultural roots of ghetto social dislocations,” and showed how ghetto lifestyles were harming Black fortunes. It’s one thing to say a certain perspective fell out of favor, but that’s a far cry from claiming that “few scholars chose to investigate … the black family and its problems,” the McLanhan and Jencks assertion that Kristof repeats.

What is the evidence? To make that causal story stick, you’d have to rule out other explanations for a shift in the orientation of research (if there was one). If attitudes like Moynihan’s fell out of favor after 1965, can you think of anything else happening at that time besides vicious academic critiques of Moynihan that might have provoked a new, less victim-blamey perspective? Oh, right: history was actually happening then, too.

free_breakfast

As for the idea people simply stopped researching Black poverty, “culture,” and family structure, that’s just wrong. Here, mostly drawn from Frank Furstenberg’s review, “The Making of the Black Family: Race and Class in Qualitative Studies in the Twentieth Century,” are some of the works published during this time when researchers were supposedly avoiding the topic:

  • Billingsley A. 1968. Black Families in White America. Englewood Cliffs, NJ: Prentice-Hall
  • Williams T, Kornblum W. 1985. Growing up Poor. Lexington, MA: Lexington Books
  • Chilman CS. 1966. Growing Up Poor. Washington, DC: USGPO
  • Liebow E. 1968. Tally’s Corner. Boston: Little, Brown
  • Hannerz U. 1969. Soulside: Inquiries into Ghetto Culture and Community. New York: Columbia Univ. Press
  • Stack C. 1974. All Our Kin. Chicago: Aldine
  • Schultz DA. 1969. Coming up Black: Patterns of Ghetto Socialization. Englewood Cliffs, NJ: Prentice Hall
  • Staples R. 1978. The Black Family: Essays and Studies. Belmont, CA: Wadsworth. 2nd ed.
  • Ladner JA. 1971. Tomorrow’s Tomorrow: The Black Woman. Garden City, NY: Doubleday
  • Furstenberg FF. 1976. Unplanned Parenthood: The Social Consequences of Teenage Childbearing. New York: Free Press

In Furstenberg’s account, many of the themes in these studies were reminiscent of research done earlier in the century, when social science research on poor Black families first emerged:

…the pervasive sense of fatalism among the poor, a lack of future orientation among youth, early parenthood as a response to blocked opportunity, sexual exploitation, tensions between men and women, the unswerving commitment to children regardless of their birth status among mothers, and the tenuous commitment among nonresidential fathers.

In addition, as Alice O’Connor notes in her intellectual history, Poverty Knowledge: Social Science, Social Policy, and the Poor in Twentieth-Century U.S. History, there was a shift around this time to more quantitative, technocratic research, using individual microdata. In particular, the highly influential Panel Study of Income Dynamics began producing studies at the start of the 1970s, and many scholars published research comparing social and economic outcomes across race, class, and family type using this data source. Here is a small sample of journal articles from 1971 to 1985, when the Moynihan taboo supposedly reigned:

  • Datcher, Linda. 1982. “Effects of Community and Family Background on Achievement.” Review of Economics and Statistics 64 (1): 32–41.
  • Greenberg, David, and Douglas Wolf. 1982. “The Economic Consequences of Experiencing Parental Marital Disruptions.” Children and Youth Services Review, 4 (1–2): 141–62.
  • Hampton, Robert L. 1979. “Husband’s Characteristics and Marital Disruption in Black Families.” Sociological Quarterly 20 (2): 255–66.
  • Hofferth, Sandra L. 1984. “Kin Networks, Race, and Family Structure.” Journal of Marriage and Family 46 (4): 791–806.
  • Hoffman, Saul. 1977. “Marital Instability and the Economic Status of Women.” Demography 14 (1): 67–76.
  • McLanahan, Sara. 1985. “Family Structure and the Reproduction of Poverty.” American Journal of Sociology 90 (4): 873–901.
  • Moffitt, Robert. 1983. “An Economic Model of Welfare Stigma.” American Economic Review 73 (5): 1023–35.
  • Smith, Michael J. 1980. “The Social Consequences of Single Parenthood: A Longitudinal Perspective.” Family Relations 29 (1): 75–81.

At least three of these scholars survived the experience of researching this subject and went on to become presidents of the Population Association of America.

Finally, an additional line of research pursued the question of family structure impacts on education or economic attainment, specifically aimed at assessing the impact of family structure on racial inequality. These studies were highly influential and widely cited, including:

  • Duncan, Beverly, and Otis Dudley Duncan. 1969. “Family Stability and Occupational Success.” Social Problems 16 (3): 273–85.
  • Featherman, David L., and Robert M. Hauser. 1976. “Changes in the Socioeconomic Stratification of the Races, 1962-73.” American Journal of Sociology 82 (3): 621–51.
  • Hauser, Robert M., and David L. Featherman. 1976. “Equality of Schooling: Trends and Prospects.” Sociology of Education 49 (2): 99–120.

I don’t know how you get from this rich literature to the notion that a liberal taboo was blocking progress — unless you define research progress according to the nature of the conclusions drawn, rather than the knowledge gained.

The resilience of this narrative reflects the success of conservative critics in building an image of leftist academics as ideological bullies who suppress any research that doesn’t toe their line. Such critics have a right to their own perspectives, but not to their own facts.

[Thanks to Shawn Fremstad for pointing me to some of these readings.]

Exceptions, suggested reading, and counterarguments welcome in the comments.

22 Comments

Filed under Research reports

Bogus versus extremely low-quality, Sullins edition

https://flic.kr/p/e6sfpP

Photo by CTBTO from Flickr Creative Commons (modified)

Calling a study “peer-reviewed” gives it at least some legitimacy. And if a finding is confirmed by “many peer-reviewed studies,” that’s even better. So the proliferation of bogus journals publishing hundreds of thousands of “peer-reviewed” articles of extremely low quality is bad news both for the progress of science and for public discourse that relies on academic research.

Two weeks ago I briefly reviewed some articles published by D. Paul Sullins, the anti-gay professor at Catholic University, on the hazards of being raised by gay and lesbian parents. I called the journals, published by Science Domain International (SDI), “bogus,” but said you could make an argument for extremely low quality instead.

After that Sullins sent me an email with some boilerplate from the publisher in defense of the journals, and he accused me of having a conflict of interest because his conclusions contradict one of my published articles. After correctly pointing out that a sting operation by Science failed to entrap an SDI journal with a bogus paper about cancer research, he said:

SDI is a new and emerging publisher. … While I would not say SDI is yet in the top tier, and I don’t like their journal names much either [which mimic real journal titles], for the reasons listed above I submit that this publisher is far from ‘bogus.’

How far from bogus?

Since that post, the reviews on the third of Sullins’ papers have been published by Science Domain and its journal, the (non-) British Journal of Education, Society & Behavioural Science. So we have some more information on which to judge.

The paper, “Emotional Problems among Children with Same-sex Parents: Difference by Definition,” was reviewed by three anonymous reviewers (from the USA, Brazil, Nigeria) and one identified as Paulo Verlaine Borges e Azevêdo, from Brazil. I summarize them here.

Anonymous USA

This reviewer only suggested minor revisions (nothing in the “compulsory revision” section). These were the suggestions: Avoid the first person, clarify the race of study participants, discuss the results in more detail, don’t use the word “trivial,” add citations to several statements, grammar check.

Anonymous, Brazil

This review demanded compulsory revisions: Clarify the level of statistical significance used, explain acronyms, clarify use of “biological parents” when discussing same-sex parents. And some minor revisions: one typo, one font-size change, standardize number of decimal places.

Anonymous, Nigeria

This reviewer included compulsory revisions: mention instrument used in the abstract, clarify measures used in previous studies on children’s well-being, test all four hypotheses proposed (not just three), clarify use of instrument used, shorten the discussion. Minor revisions: check for typos.

Paulo Verlaine Borges e Azevêdo, Brazil

This reviewer requested reorganizing the text, like this:

Would be better to redistribute the lengths of results (lessened), discussion (up) and conclusion (down) sections. In many moments, in the Result section the author deal with I believe would be better located in the Discussion (e. eg., between lines 345 and 355). I suggest that the subsections of Results would be reviewed by author and parts that discuss the results be transferred to the Discussion section … Strengths and Limitations would be better located in the discussion section too.

A few additional minor text modifications were included in the marked up manuscript.

Round two

Upon revision, Sullins was subjected to a punishing second round of reviews.

This included an interesting if ultimately fruitless attempt by Anonymous Brazil to object to this somewhat nutty sentence by Sullins: “biological parentage uniquely and powerfully distinguishes child outcomes between children with opposite-sex parents and those with same-sex parents.” What he meant was, when he controlled for the biological relationship between children and their parents — since hetero parents are more likely to have any biological parentage (and they’re the only ones with two bio parents) — it statistically reduced the gap in children’s mental health between married hetero versus same-sex parents. Although the exchange was meaningless in the decision whether to publish, and Sullins didn’t change it, and the reviewer dropped the objection, and the editors just said “publish it,” you would have to say this was a moment of actual review.

OK then

That’s it. None of this touched on the obvious fatal flaws in the study — that Sullins combines children in all same-sex families into one category while breaking those currently with different-sex parents into different groups (step-parents, cohabitors, single parents, etc.) — and that he has no data on how long the children currently with same-sex couples have lived with them, or how they came to live with them. So it leaves us right where we started on the question of same-sex parenting “effects” on children.

Of course, lots of individual reviews are screwed up. So, is this journal bogus or merely extremely low quality? Do we have a way of identifying these so-bad-they’re-basically-bogus journals that is meaningful to the various audiences they are reaching?

This matters is because journalists, judges, researchers, and the concerned public would like some way to evaluate the veracity of scientific claims that bear on current social controversies, such as marriage equality and the rights of gay and lesbian parents.

7 Comments

Filed under Research reports