Tag Archives: methods

Decadally-biased marriage recall in the American Community Survey

Do people forget when they got married?

In demography, there is a well-known phenomenon known as age-heaping, in which people round off their ages, or misremember them, and report them as numbers ending in 0 or 5. We have a measure, known as Whipple’s index, that estimates the extent to which this is occurring in a given dataset. To calculate this you take the number of people between ages 23 and 62 (inclusive), and compare it to five-times the number of those whose ages end in 0 or 5 (25, 30 … 60), so there are five-times as many total years as 0 and 5 years.

If the ratio of 0/5s to the total is less than 105, that’s “highly accurate” by the United Nations standard, a ratio 105 to 110 is “fairly accurate,” and in the range 110 to 125 age data should be considered “approximate.”

I previously showed that the American Community Survey’s (ACS) public use file has a Whipple index of 104, which is not so good for a major government survey in a rich country. The heaping in ACS apparently came from people who didn’t respond to email or mail questionnaires and had to be interviewed by Census Bureau staff by phone or in person. I’m not sure what you can do about that.

What about marriage?

The ACS has a great data on marriage and marital events, which I have used to analyze divorce trends, among other things. Key to the analysis of divorce patterns is the question, “When was this person last married?” (YRMARR) Recorded as a year date, this allows the analyst to take into account the duration of marriage preceding divorce or widowhood, the birth of children, and so on. It’s very important and useful information.

Unfortunately, it may also have an accuracy problem.

I used the ACS public use files made available by IPUMS.org, combining all years 2008-2017, the years they have included the variable YRMARR. The figure shows the number of people reported to have last married in each year from 1936 to 2015. The decadal years are highlighted in black. (The dropoff at the end is because I included surveys earlier than those years.)

year married in 2016.xlsx

Yikes! That looks like some decadal marriage year heaping. Note I didn’t highlight the years ending in 5, because those didn’t seem to be heaped upon.

To describe this phenomenon, I hereby invent the Decadally-Biased Marriage Recall index, or DBMR. This is 10-times the number of people married in years ending in 0, divided by the number of people married in all years (starting with a 6-year and ending with a 5-year). The ratio is multiplied by 100 to make it comparable to the Whipple index.

The DBMR for this figure (years 1936-2015) is 110.8. So there are 1.108-times as many people in those decadal years as you would expect from a continuous year function.

Maybe people really do get married more in decadal years. I was surprised to see a large heap at 2000, which is very recent so you might think there was good recall for those weddings. Maybe people got married that year because of the millennium hoopla. When you end the series at 1995, however, the DBMR is still 110.6. So maybe some people who would have gotten married at the end of 1999 waited till New Years day or something, or rushed to marry on New Year’s Eve 2000, but that’s not the issue.

Maybe this has to do with who is answering the survey. Do you know what year your parents got married? If you answered the survey for your household, and someone else lives with you, you might round off. This is worth pursuing. I restricted the sample to just those who were householders (the person in whose name the home is owned or rented), and still got a DBMR of 110.7. But that might not be the best test.

Another possibility is that people who started living together before they were married — which is most Americans these days — don’t answer YRMARR with their legal marriage date, but some rounded-off cohabitation date. I don’t know how to test that.

Anyway, something to think about.

Leave a comment

Filed under Research reports

Theology majors marry each other a lot, but business majors don’t (and other tales of BAs and marriage)

The American Community Survey collects data on the college majors of people who’ve graduated college. This excellent data has lots of untapped potential for family research, because it tells us something about people’s character and experience that we don’t have from any other variables in this massive annual dataset. (It even asks about a second major, but I’m not getting into that.)

To illustrate this, I did two data exercises that combine college major with marital events, in this case marriage. Looking at people who just married in the previous year, and college major, I ask: Which majors are most and least likely to marry each other, and which majors are most likely to marry people who aren’t college graduates?

I combined eight years of the ACS (2009-2016), which gave me a sample of 27,806 college graduates who got married in the year before they were surveyed (to someone of the other sex). Then I cross-tabbed the major of wife and major of husband, and produced a table of frequencies. To see how majors marry each other, I calculated a ratio of observed to expected frequencies in each cell on the table.

Example: With weights (rounding here), there were a total of 2,737,000 BA-BA marriages. I got 168,00 business majors marrying each other, out of 614,000 male and 462,000 female business majors marrying altogether. So I figured the expected number of business-business pairs was the proportion of all marrying men that were business majors (.22) times the number of women that were business majors (461,904), for an expected number of 103,677 pairs. Because there were 168,163 business-business pairs, the ratio is 1.6.  (When I got the same answer flipping the genders, I figured it was probably right, but if you’ve got a different or better way of doing it, I wouldn’t be surprised!)

It turns out business majors, which are the most numerous of all majors (sigh), have the lowest tendency to marry each other of any major pair. The most homophilous major is theology, where the ratio is a whopping 31. (You have to watch out for the very small cells though; I didn’t calculate confidence intervals.) You can compare them with the rest of the pairs along the diagonal in this heat map (generated with conditional formatting in Excel):

spouse major matching

Of course, not all people with college degrees marry others with college degrees. In the old days it was more common for a man with higher education to marry a woman without than the reverse. Now that more women have BAs, I find in this sample that 35% of the women with BAs married men without BAs, compared to just 22% of BA-wielding men who married “down.” But the rates of down-marriage vary a lot depending on what kind of BA people have. So I made the next figure, which shows the proportion of male and female BAs, by major, marrying people without BAs (with markers scaled to the size of each major). At the extreme, almost 60% of the female criminal justice majors who married ended up with a man without a BA (quite a bit higher than the proportion of male crim majors who did the same). On the other hand, engineering had the lowest overall rate of down-marriage. Is that a good thing about engineering? Something people should look at!

spouse matching which BAs marry down

We could do a lot with this, right? If you’re interested in this data, and the code I used, I put up data and Stata code zips for each of these analyses (including the spreadsheet): BA matching, BA’s down-marrying. Free to use!

9 Comments

Filed under Research reports

No, early marriage is not more common for college graduates

Update: IFS has taken down the report I critiqued here, and put up a revised report. They have added an editor’s note, which doesn’t mention me or link to this post:

Editor’s Note: This post is an update of a post published on March 14, 2018. The original post looked at marriage trends by education among all adults under age 25. It gave the misimpression that college graduates were more likely to be married young nowadays, compared to non-college graduates.


At the Institute for Family Studies, Director of Research Wendy Wang has a post up with the provocative title, “Early Marriage is Now More Common For College Graduates” (linking to the Internet Archive version).

She opens with this:

Getting married at a young age used to be more common among adults who didn’t go to college. But the pattern has reversed in the past decade or so. In 2016, 9.4% of college graduates ages 18 to 24 have ever been married, which is higher than the share among their peers without a college degree (7.9%), according to my analysis of the most recent Census data.

And then the dramatic conclusion:

“What this finding shows is that even at a young age, college-educated adults today are more likely than their peers without a college degree to be married. And this is new.”

That would be new, and surprising, if it were true, but it’s not.

Here’s the figure that supports the conclusion:

figure1wendyupdate-w640

It shows that 9.4% of college graduates in the age range 18-24 have been married, compared with 7.9% of those who did not graduate from college. (The drop has been faster for non-graduates, but I’m setting aside the time trend for now.) Honestly, I guess you could say, based on this, that young college graduates are more likely than non-graduates to “be married,” but not really.

The problem is there are very very few college graduates in the ages 18-19. The American Community Survey, which they used here, reports only about 12,000 in the whole country, compared with 8.7 million people without college degrees ages 18-19 (this is based on the public use files that IPUMS.org uses; which is what I use in the analysis below). Wow! There are lots and lots of non-college graduates below age 20 (including almost everyone who will one day be a college graduate!), and very few of them are married. So it looks like the marriage rate is low for the group 18-24 overall. Here is the breakdown by age and marital status for the two groups: less than BA education, and BA or higher education — on the same population scale, to help illustrate the point:

ifs1ifs2

If you pool all the years together, you get a higher marriage rate for the college graduates, mostly because there are so few college graduates in the younger ages when hardly anyone is married.

To show the whole thing in terms of marriage rates, here is the marital status for the two groups at every age from 15 (when ACS starts asking about marital status) to 54.

ifs3

Ignoring 19-21, where there are a tiny number of college graduates, you see a much more sensible pattern: college graduates delay marriage longer, but then have higher rates at older ages (starting at age 28), for all the reasons we know marriage is ultimately more common among college graduates. In fact, if you used ages 15-24 (why not?), you get an even bigger difference — with 9.4% of college graduates married and just 5.7% of non-college graduates. Why not? In fact, what about ages 0-24? It would make almost as much sense.

Another way to do this is just to look at 24-year-olds. Since we’re talking about the ever-married status, and mortality is low at these ages, this is a case where the history is implied in the cross-sectional data. At age 24, as the figure shows, 19.9% of non-college graduates have been married, compared with 12.9% of college graduates. Early marriage is not more common for college graduates.

In general, I don’t recommend comparing college graduates and non-graduates, at least in cross-sectional data, below age 25. Lots of people finishing college below age 25 (and increasingly after that age as well). There is also an important issue of endogeneity here, which always makes education and age analysis tricky. Some people (mostly women) don’t finish college because they get married and have children).

Anyway, it looks to me like someone working for a pro-marriage organization saw what seemed like a story implying marriage is good (that’s why college graduates do it, after all), and one that also fits with the do-what-I-say-not-what-I-do criticism of liberals, who are supposedly not promoting marriage among poor people while they themselves love to get married (a critique made by Charles Murray, Brad Wilcox, and others). And, before thinking it through, they published it.

Mistakes happen. Fortunately, I dislike the Institute for Family Studies (see the whole series under this tag), and so I read it and pointed out this problem within a couple hours (first on Twitter, less than two hours after Wang tweeted it). It’s a social media post-publication peer review success story! If they correct it.

3 Comments

Filed under Research reports

For social relationships outside marriage

Stephanie Coontz has a great piece in tomorrow’s New York Times titled, “For a Better Marriage, Act Like a Single Person.” From her intro:

Especially around Valentine’s Day, it’s easy to find advice about sustaining a successful marriage, with suggestions for “date nights” and romantic dinners for two. But as we spend more and more of our lives outside marriage, it’s equally important to cultivate the skills of successful singlehood. And doing that doesn’t benefit just people who never marry. It can also make for more satisfying marriages.

From there she develops the case with, as usual, a lot of the right research. Well worth a read.

Stephanie used two empirical bits from my work:

No matter how much Americans may value marriage, we now spend more time living single than ever before. In 1960, Americans were married for an average of 29 of the 37 years between the ages of 18 and 55. That’s almost 80 percent of what was then regarded as the prime of life. By 2015, the average had dropped to only 18 years.

In many ways, that’s good news for marriages and married people. Contrary to some claims, marrying at an older age generally lowers the risk of divorce. It also gives people time to acquire educational and financial assets, as well as develop a broad range of skills — from cooking to household repairs to financial management — that will stand them in good stead for the rest of their lives, including when a partner is unavailable.

The first figure, the average years spent in marriage between the ages of 18 and 55 is very easy to calculate. You just sum the proportion of people married at each age. Here’s what it looks like, comparing 1960 (from the decennial Census) and 2015 (from the American Community Survey), both from IPUMS.org (click to enlarge):

YearsMarried

I think it’s a nice, simple way to show the declining footprint of marriage in American life. (I first did this, and described in the rationale, in 2010.)

The bit about older age at marriage being associated with lower odds of divorce is from this post. Here’s the result, showing odds of divorce in one year by age at marriage, with controls for duration, education, race/ethnicity, and nativity, for women in their first marriages (click to enlarge):
Divorce by age at marriage

There’s more discussion in the post, as well as in this followup post, which has this cool figure, where red is the highest odds of divorce and green is the lowest, and the axes are years married and age at marriage (click to enlarge):

Divorce By Age And Duration


My new book is out! Enduring Bonds: Inequality, Marriage, Parenting, and Everything Else That Makes Families Great and Terrible. Available all the usual places, plus here at the University of California Press, where Chapter 1 is available as a sample, and where instructors can request a review copy.

1 Comment

Filed under Research reports

Data analysis: Are older newlyweds saving marriage?

COS open data badgeCOS Open Materials badge


Is the “institution” still in decline if the incidence of marriage rebounds, but only at older ages?

In my new book I’ve revisited old posts and produced this figure, which shows the refined marriage rate* from 1940 to 2015, with a discussion of possible futures:

f15

The crash scenario – showing marriage ending around 2050, is there to show where the 1950-2014 trajectory is headed (it’s also a warning against using linear extrapolation to predict the future). The rebound scenario is intended to show how unrealistic the “revive marriage culture” people are. The taper scenario emerges as the most likely alternative; in fact, it’s grown more likely since I first made the figure a few years ago, as you can see by the 2010-2014 jag.

So let’s consider the tapering scenario more substantively — what would it look like? One way to get a declining marriage rate is if marriage is increasingly delayed, even if it doesn’t become less common; people still marry, but later. (If everyone got married at age 99, we would have universal marriage and a very low refined marriage rate.) I give some evidence for this scenario here.

These trends are presented with minimal discussion; I’m not looking at race/ethnicity or social class, childbearing or the recession; I’m not discussing divorce and remarriage and cohabitation, and I’m not testing hypotheses. (This is a list of research suggestions!) To make the subject more enticing as a research topic (and for accountability), I’ve shared the Census data, Stata code, and spreadsheet file used to make this post in this OSF project. You can use anything there you want. You can also easily fork the project — that is, make a duplicate of its contents, which you then own, and take off on your own trajectory, by adding to or modifying them.

Trends

For some context, here is the trend in percentage of men and women ever married, by age, from 1960. (“Ever married” means currently married, separated, divorced, or widowed.) This clearly shows both life-course delay and lifetime decline, but delay is much more prominent, at least so far. Even now, almost 90% of people have been married by age 60 or so, while the marriage rates for people under 35 have plummeted.

evmar6016

People become ever-married when they get first-married. We measure ever-married prevalence from a survey question on current marital status, but first-marriage incidence requires a question like the American Community Survey asks, “In the past 12 months, did this person get married?” Because they also ask how many times each person has been married, you can calculate a first marriage rate with this ratio:

(once married & married in the past 12 months) / (never married + (once married & married in the past 12 months))

Until recently it hasn’t been easy to measure first-marriage across all ages; now that we have the ACS marital events data (since 2008) we can. This allows us to look at the timing of first marriage, which means we can use current age-specific first-marriage rates to project lifetime ever-married rates under current conditions.

Here are the first-marriage rates for men and women, by age. Each set of bars shows the trend from 2008 to 2016. The left side shows men, by age; the right side shows women, by age; the totals for men and women are in the middle. This shows that first-marriage rates have fallen for men and women under age 35, but increased for those over age 35. The total first-marriage rate has rebounded from the 2013 crater, but is still lower than 2008.

1stmarage

This is a short-range trend, 9 years. It could be recession-specific, with people delaying marriage because of hardships, or relationships falling apart under economic stress, and then hurrying to marry a few years later. But it also fits the long-term trend of delay over decline.

The overall rates for men and women show that the 2014-2016 rebound has not brought first-marriage rates back to their 2008 level. However, what about lifetime odds of marriage? The next figure uses women’s age-specific first-marriage rates to project lifetime odds of marriage for three years: 2008, the 2013 crater, and 2016. This shows, for example, that at 2008 rates 59% of women would have married by age 30, compared with 53% in both 2013 and 2016.

1stmarproj

The 2013 and 2016 lines diverge after age 30, and by age 65 the projected lifetime ever-married rates have fully recovered. This implies that marriage has been delayed, but not forgone (or denied).

Till now I’ve shown age and sex-specific rates, but haven’t addressed other things that might changed in the never-married population. Finally, I estimated logistic regressions predicting first-marriage among never married men and women. The models include race, Hispanic origin, nativity, education, and age. In addition to the year and age patterns above, the models show that all races have lower rates than Whites, Hispanics have lower rates than non-Hispanics, foreign-born people have higher rates (which explains the Hispanic result), and people with more education first-marry more (code and results in the OSF project).

To see whether changes in these other variables change the story, I used the regressions to estimate first-marriage rates at the overall mean of all variables. These show a significant rebound from the bottom, but not returning to 2008 levels, quite similar to the unadjusted trends above:

1stmaradj

This is all consistent with the taper scenario described at the top. Marriage delayed, which reduces the annual marriage rate, but with later marriage picking up much of the slack, so that the decline in lifetime marriage prevalence is modest.


* The refined marriage rate is the number of marriages as a fraction of unmarried people. This is more informative than the crude marriage rate (which the National Center for Health Statistics tracks), which is marriages as a fraction of the total population. In this post I use what I guess you would call an age-specific refined first-marriage rate, defined above.

1 Comment

Filed under Research reports

Science finds tiny things nowadays (Malia edition)

We have to get used to living in a world where science — even social science — can detect really small things. Understanding how important really small things are, and how to interpret them, is harder nowadays than just finding them.

Remember when Hanna Rosin wrote this?

One of the great crime stories of the last twenty years is the dramatic decline of sexual assault. Rates are so low in parts of the country — for white women especially — that criminologists can’t plot the numbers on a chart.

Besides being wrong about rape (it has declined a lot, but it’s still high compared with most countries), this was a funny statement about science (I’ve heard we can even plot negative numbers now!). But the point is we have problems understanding, and communicating about, small things.

So, back to names.

In 2009, the peak year for the name Malia in the U.S., 1,681 girls were given that name, according to the Social Security Administration, or .041% of the 4.14 million children born that year (there are no male Malias in the SSA’s public database, meaning they have never recorded more than 4 in one year). That year, 7.5% of women ages 18-44 had a baby. If my arithmetic is right, say you know 100 women ages 18-44, and each of them knows 100 others (and there is no overlap in your network). That would mean there is a 30% chance one of your 10,000 friends of a friend had a baby girl and named her Malia in 2009. But probably there is a lot of overlap; if your friend-of-friend network is only 1,000 women 18-44 then that chance would fall to 3%.

Here is the trend in girls named Malia, relative to the total number of girls born, from 1960 to 2016:

names.xlsx

To make it easier to see the Malias, here is the same chart with the y-axis on a log scale.

names.xlsx

This shows that Malia has been on a long upward trend, from less than 50 per year in the 1960s to more than 1,000 per year now. And it also shows a pronounced spike in 2009, the year Malia peaked .041%. In that year, the number of people naming daughters Malia jumped 75% before declining over the next three years to resume it’s previous trend. Here is the detail on the figure, just showing the Malia in 2005-2016:

names.xlsx

What happened there? We can’t know for sure. Even if you asked everyone why they named their kid what they did, I don’t know what answers you would get. But from what we know about naming patterns, and their responsiveness to names in the news (positive or negative), it’s very likely that the bump in 2009 resulted from the high profile of Barack Obama and his daughter Malia, who was 11 when Obama was elected.

What does a causal statement like that that really mean? In 2009, it looks to me like about 828 more people named their daughters Malia than would have otherwise, taking into account the upward trend before 2008. Here’s the actual trend, with a simulated trend showing no Obama effect:

names.xlsx

Of course, Obama’s election changed the world forever, which may explain why the upward trend for Malia accelerated again after 2013. But in this simple simulation, which brings the “no Obama” trend back into line with the actual trend in 2014, there were 1,275 more Malias born than there would have been without the Obama election. This implies that over the years 2008-2013, the Obama election increased the probability of someone naming their daughter Malia by .00011, or .011%.

That is a very small effect. I think it’s real, and very interesting. But what does it mean for anything else in the world? This is not a question of statistical significance, although those tools can help. (These names aren’t a probability sample, it’s a list of all names given.) So this is a question for interpreting research findings now that we have these incredibly powerful tools, and very big data to analyze with them. The number alone doesn’t tell the story.

2 Comments

Filed under Me @ work

’16 and Pregnant’ and less so

3419870216_fded1624d2_z

From Flickr/CC: https://flic.kr/p/6dcJgA

Regular readers know I have objections to the framing of teen pregnancy, as a thing generally and as a problem specifically, separate from the rising age at childbearing generally (see also, or follow the teen births tag).

In this debate, one economic analysis of the effect of the popular MTV show 16 and Pregnant has played an outsized role. Melissa Kearney and Phillip Levine showed that was more decline in teen births in places where the show was popular, and attempted to establish that the relationship was causal — that the show makes people under age 20 want to have babies less. As Kearney put it in a video promoting the study: “the portrayal of teen pregnancy, and teen childbearing, is something they took as a cautionary tale.” (The paper also showed spikes in Twitter and Google activity related to birth control after the show aired.)

This was very big news for the marriage promotion people, because it was taken as evidence that cultural intervention “works” to affect family behavior — which really matters because so far they’ve spent $1 billion+ in welfare money on promoting marriage, with no effect (none), and they want more money.

The 16 and Pregnant paper has been cited to support statements such as:

  • Brad Wilcox: “Campaigns against smoking and teenage and unintended pregnancy have demonstrated that sustained efforts to change behavior can work.”
  • Washington Post: “By working with Hollywood to develop smart story lines on popular shows such as MTV’s ’16 and Pregnant’ and using innovative videos and social media to change norms, the [National Campaign to Prevent Teen and Unplanned Pregnancy] has helped teen pregnancy rates drop by nearly 60 percent since 1991.”
  • Boston Globe: “As evidence of his optimism, [Brad] Wilcox points to teen pregnancy, which has dropped by more than 50 percent since the early 1990s. ‘Most people assumed you couldn’t do much around something related to sex and pregnancy and parenthood,’ he said. ‘Then a consensus emerged across right and left, and that consensus was supported by public policy and social norms. . . . We were able to move the dial.’ A 2014 paper found that the popular MTV reality show ’16 and Pregnant’ alone was responsible for a 5.7 percent decline in teen pregnancy in the 18 months after its debut.”

I think a higher age at first birth is better for women overall, health permitting, but I don’t support that as a policy goal in the U.S. now, although I expect it would be an outcome of things I do support, like better health, education, and job opportunities for people of color and people who are poor.

Anyway, this is all just preamble to a new debate from a reanalysis and critique of the 16 and Pregnant paper. I haven’t worked through it enough to reach my own conclusions, and I’d like to hear from others who have. So I’m just sharing the links in sequence.

The initial paper, posted as a (non-peer reviewed) NBER Working Paper in 2014:

Media Influences on Social Outcomes: The Impact of MTV’s 16 and Pregnant on Teen Childbearing, by Melissa S. Kearney, Phillip B. Levine

This paper explores how specific media images affect adolescent attitudes and outcomes. The specific context examined is the widely viewed MTV franchise, 16 and Pregnant, a series of reality TV shows including the Teen Mom sequels, which follow the lives of pregnant teenagers during the end of their pregnancy and early days of motherhood. We investigate whether the show influenced teens’ interest in contraceptive use or abortion, and whether it ultimately altered teen childbearing outcomes. We use data from Google Trends and Twitter to document changes in searches and tweets resulting from the show, Nielsen ratings data to capture geographic variation in viewership, and Vital Statistics birth data to measure changes in teen birth rates. We find that 16 and Pregnant led to more searches and tweets regarding birth control and abortion, and ultimately led to a 5.7 percent reduction in teen births in the 18 months following its introduction. This accounts for around one-third of the overall decline in teen births in the United States during that period.

A revised version, with the same title but slightly different results, was then published in the top-ranked American Economic Review, which is peer-reviewed:

This paper explores the impact of the introduction of the widely viewed MTV reality show 16 and Pregnant on teen childbearing. Our main analysis relates geographic variation in changes in teen childbearing rates to viewership of the show. We implement an instrumental variables (IV) strategy using local area MTV ratings data from a pre-period to predict local area 16 and Pregnant ratings. The results imply that this show led to a 4.3 percent reduction in teen births. An examination of Google Trends and Twitter data suggest that the show led to increased interest in contraceptive use and abortion.

Then last month David A. Jaeger, Theodore J. Joyce, and Robert Kaestner posted a critique on the Institute for the Study of Labor working paper series, which is not peer-reviewed:

Does Reality TV Induce Real Effects? On the Questionable Association Between 16 and Pregnant and Teenage Childbearing

We reassess recent and widely reported evidence that the MTV program 16 and Pregnant played a major role in reducing teen birth rates in the U.S. since it began broadcasting in 2009 (Kearney and Levine, American Economic Review 2015). We find Kearney and Levine’s identification strategy to be problematic. Through a series of placebo and other tests, we show that the exclusion restriction of their instrumental variables approach is not valid and find that the assumption of common trends in birth rates between low and high MTV-watching areas is not met. We also reassess Kearney and Levine’s evidence from social media and show that it is fragile and highly sensitive to the choice of included periods and to the use of weights. We conclude that Kearney and Levine’s results are uninformative about the effect of 16 and Pregnant on teen birth rates.

And now Kearney and Levine have posted their response on the same site:

Does Reality TV Induce Real Effects? A Response to Jaeger, Joyce, and Kaestner (2016)

This paper presents a response to Jaeger, Joyce, and Kaestner’s (JJK) recent critique (IZA Discussion Paper No. 10317) of our 2015 paper “Media Influences on Social Outcomes: The Impact of MTV’s 16 and Pregnant on Teen Childbearing.” In terms of replication, those authors are able to confirm every result in our paper. In terms of reassessment, the substance of their critique rests on the claim that the parallel trends assumption, necessary to attribute causation to our findings, is not satisfied. We present three main responses: (1) there is no evidence of a parallel trends assumption violation during our sample window of 2005 through 2010; (2) the finding of a false placebo test result during one particular earlier window of time does not invalidate the finding of a discrete break in trend at the time of the show’s introduction; (3) the results of our analysis are robust to virtually all alternative econometric specifications and sample windows that JJK consider. We conclude that this critique does not pose a serious threat to the interpretation of our 2015 findings. We maintain the position that our earlier paper is informative about the causal effect of 16 and Pregnant on teen birth rates.

So?

There are interesting methodological questions here. It’s hard to identify the effects of interventions that are swimming with the tide of change. In fact, the creation of the show, the show’s popularity, the campaign to end teen pregnancy, and the rising age at first birth may all be outcomes of the same general historical trend. So I’m not that invested in the answer to this question, though I am very interested.

There are also questions about the publication process, which I am very invested in. That’s why I work to promote a working paper culture among sociologists (through the SocArXiv project). The original paper was posted on a working paper site without peer review, but NBER is for economists who already are somebody, so that’s a kind of indirect screening. Then it was accepted in a top peer-reviewed journal (somewhat revised), but that was after it had received major attention and accolades, including a New York Times feature before the working paper was even released and a column devoted to it by Nicholas Kristof.

So is this a success story of working paper culture gone right — driving attention to good work faster, and then also drawing the benefits of peer review through the traditional publication process? (And now continuing with open debate on non-gated sites). Or is it a case of political hype driving attention inside and outside of the academy — the kind of thing that scares researchers and makes them want to retreat behind the slower, more process-laden research flow which they hope will protect them from exposure to embarrassment and protect the public from manipulation by the credulous news media. I think the process was okay even if we do conclude the paper wasn’t all it was made out to be. There were other reputational systems at work — faculty status, NBER membership, New York Times editors and sources — that may be as reliable as traditional peer review, which itself produces plenty of errors.

So, it’s an interesting situation — research methods, research implications, and research process.

2 Comments

Filed under Research reports