More bad reporting on texting and driving, and new data

The New York Times‘ problem of misrepresenting the relationship between phones and traffic fatalities, which seems to have begun with Matt Richtel, has just gotten worse.

Richtel sells books on the fear of texting and driving (which, of course, is dangerous), and the website for his book still — despite my repeated entreaties, public and private — leads with the bad, false, unsourced Internet meme, that “the texting-while-driving epidemic continues to claim 11 teen lives per day.” (As a reporter, how could you sleep one night with that BS up under your name? Mind boggling.)

Anyway, the new entrant is David Leohnardt. At the heavy risk of jeopardizing future opportunities to publish on the Times op-ed page, I tweeted that his recent column included “one of the dumbest things I’ve ever read in the NYTimes.” Washington Post WonkBlog writer Jeff Guo pointed out Leonardt’s column, which claimed that, with regard to the recent spike in traffic deaths, “The only plausible cause is the texting, calling, watching, and posting that people now do while operating a large piece of machinery.” The column contained not a piece of evidence to support that claim (though there were some awful anecdotes), which is why I said it was dumb.

Which is too bad. But even though the spike in traffic deaths is concerning, reporting should not be wrong.

Early estimates from the National Safety Council (which uses a different method than the Federal NHTSA) show a 6% increase in traffic fatalities for 2016. Leonhardt, working really hard to make that absolutely as alarming as possible, produced this graph, showing percent change in fatalities over successive two year periods going back to 1980:

Because it’s hard to add up the pluses and minuses in your head, It would be really easy — really really easy — to look at Leonhardt’s chart and think fatalities are higher now than they were in 1980. But rather than pointing out that fatalities per person have fallen by half since 1980, he instead writes, “It’s the first significant rise in a half century,” which would be true except for the significant rise in every single decade of the last half century.

This is a lot like when Richtel described the 2015 rise as, “soaring at a rate not seen in 50 years.” Not that the rate was not seen in 50 years, of course, just that the soaring of the rate hadn’t been (or so the NYT Science Desk told me when I complained).

Adding 6% to the NHTSA numbers for 2015, I get the follow graph, showing the trends in deaths per person in the population, and deaths per mile traveled, as changes since 1970. (The deaths per mile haven’t been released for the whole year yet; click to enlarge.)

That is a troubling spike, which takes us all the way back to 2009 fatality rates. We should make the roads safer, by using them less and using them more safely. But come on, NYTimes.

Read the whole, completely aggravating series, under the texting tag.

NYT’s Richtel traffic hype soars at historic rate

What if everything you learned about traffic fatalities you got from Matt Richtel at the New York Times?

On April 27, Richtel wrote:

Over the last seven years, most states have banned texting by drivers, and public service campaigns have tried an array of tactics — “It can wait,” among them — to persuade people to put down their phones when they are behind the wheel.

Yet the problem, by just about any measure, appears to be getting worse. Americans confess in surveys that they are still texting while driving, as well as using Facebook and Snapchat and taking selfies. Road fatalities, which had fallen for years, are now rising sharply, up roughly 8 percent in 2015 over the previous year, according to preliminary estimates.

I left the paragraph breaks as they were, so you can see the connection he implied. Sure seems that all that texting is driving up the rate of fatalities, although there is no evidence offed for that. Of course, since you only read Richtel, you don’t know that since 1994, cell phone subscription rates have risen 1200% while traffic fatalities have fallen 13%. (My series on this, with all these facts, is under the texting tag.)

But what about this “now rising sharply” fact? The same fact – an estimated 8% increase in one year, grew from “now rising sharply” on April 27 to “soaring at a rate not seen in 50 years” by May 22.

When I complained to the NYT Science Desk that this was a misleading representation of a traffic fatality rate that is still at historically low levels, someone checked it out and nicely informed me they had “confirmed the article accurately states the fact: preliminary estimates indicate road fatalities are rising at a rate not seen in 50 years.” Complaint denied.

Assuming you share my obsession with this problem of hyping traffic fatalities – and distracting the public with stories of bad drivers instead of paying attention to the real problem, which is rampant car culture itself – then you’ll want to make a distinction between the facts themselves and the NYT representation of them.

Facts

The fact here is actually kind of weird. Instead of using the official traffic fatality rates, which the National Highway Traffic Safety Administration hasn’t released yet for 2015, Richtel here is reporting the preliminary 2015 estimate from a private group, the National Safety Council. The weird thing is that NSC uses a different method, counting people as dead from a traffic incident if they die from any cause within a year of the accident, while NHTSA only counts them as dead from the incident if they die within 30 days. The rule is arbitrary either way, but I prefer to NHTSA method in the absence of a compelling argument. (As any Law & Order fan can tell you, people who die can have that death attributed to something that happened years earlier if the medical examiner owes the detectives a favor.) Not surprisingly, NSC produces estimates that are higher – about 8% higher, about 3,000 deaths more than the roughly 35,000 NHTSA reports. The longer death window seems bad for comparing rates over time, because the population is aging and therefore the death rate will probably rise among people who have had an accident in the previous year just because they’re older on average.

Anyway, since NSC doesn’t report their long-term trend (at least on the free part of their website), I applied their estimate of the “soaring” 2015 change – an 8.2% increase in total deaths – to the NHTSA series (helpfully recorded on this Wikipedia page), to extend the series to 2015. We also now have the Federal Highway Administration’s report on vehicle miles traveled in 2015 (+4% from 2014), so I can use that estimate of total deaths to calculate deaths per mile for 2015, as well as deaths per person (using the average of the Census Bureau’s monthly estimates for the year, which was 321.4 million, +0.8%). By these calculations, Richtel’s soaring 8.2% increase in total deaths becomes a 4% increase in deaths per mile, and a 7.3% increase in deaths per person. Here are the 50-year trends, with recessions shown:

The dramatic increase in deaths for 2015, which is quite large on a relative scale during this time period – in fact, at no time in last 50 years has the number of deaths increase by 8.2% in one year – looks kind of small. But perhaps more important, it seems in line with the cyclical nature of the trend. Death rates fall during recessions and rebound afterwards. In fact, the declines in 2008 and 2009 – which were 9.3% and 9.5% respectively – are also unprecedented during this period, so a larger rebound is not surprising.

Representations

So how should Richtel describe the trend? Keep in mind this is a reporter who is still promoting a book premised on the crisis of distracted driving – the homepage for which, despite notifications to the author and publisher, still repeats the bogus internet meme that 11 teens per day die from texting while driving.

After I complained on Twitter, Richtel tweeted to his followers:

Fatalities soaring at rate not seen in 50 years. (Friends, please ignore white noise) https://t.co/MHjOYEE554. https://t.co/uZM26zvhCa

— matt richtel (@mrichtel) May 23, 2016

So I put the description question to the test of my Twitter readers, offering them a poll. I made two figures, one from the actual death trend (with my 2015 estimate), and one with the same trend in reverse, and asked people (without comment) which one they thought was better described by Richtel’s phrase:

Here are two trends. Based only on the description given, which series do you think fits best? (poll follows) pic.twitter.com/l8ooOIETbv

— Philip N. Cohen (@familyunequal) May 24, 2016

My readers were closely divided, but gave Richtel the edge, 56% to 44%:

Here's the poll: The description given best fits…

— Philip N. Cohen (@familyunequal) May 24, 2016

I don’t think it’s very strong vindication to barely eke out a majority in a poll where people are asked to choose which is right, what you said or the exact opposite of what you said. I would think real reporting would have a higher rate of concordance than that.

Why can’t the texting-panic establishment handle the truth?

Don’t drive distracted, okay? Now for some more updated facts. (Follow the whole series under the texting tag.)

The Diane Rehm show on NPR (Washington station WAMU) did another full episode on the perils of distracted driving. The extremely misleading title of the episode was, “Distracted Driving: What It Will Take To Lower Fatalities.”

The guests were researcher David Strayer; Jeff Larason, director of highway safety for Massachusetts; Joan Claybrook, former administrator of the National Highway Traffic Safety Administration; and Ben Leiberman, the co-founder of Distracted Operators Risk Casualties (DORCs), which is trying to develop the technology (and legislation) to allow police to scan phones at the scene of an accident to determine whether they were being used at the time of the crash.

I am pretty sure that every one of these guests knows that our roads are safer now than they have ever been, and that accident and fatality rates are at historic lows. And yet the entire conversation — without explicitly stating any trend facts — was conducted as if it is self-apparent that the problem is getting worse and worse. Several callers said they see more and more drivers on their phones; someone said one-in-four drivers is using a phone; someone said texting and driving is as dangerous as driving drunk. Maybe more and more people are using their phones while they drive, but that’s not making the roads less safe than they used to be.

Why can’t they handle the truth? Texting and other distractions are dangerous, and people shouldn’t do them — and the roads are getting safer over time. Here are the fatality trends for the last 20 years, from NHTSA:

In the last 20 years, fatalities per mile have fallen 38% and fatalities per person have fallen 34%. That doesn’t make texting and driving okay, okay? But it’s true.

Further, much was made in the conversation about the special risks posed by younger drivers, who are said to be less skilled and more distractable behind the wheel. This also highly misleading. A separate data series, maintained by the Insurance Institute for Highway Safety, has fatal accidents by the age of the driver going back to 1975. This shows that the steepest decline in fatal accidents has been among teenage drivers — a stunning 71% decline in fatal accidents per person in that age group since the peak in 1978. In fact, teen drivers are now involved in fewer fatal accidents per person than 20-34-year-olds:

I can understand that for advocates a story of continuously increasing peril is attractive. That doesn’t justify their refusal to speak facts, but it’s at least predictable. The guests all spoke of the need for more money to be devoted to the problem, more legislation, more awareness — all things that (no offense) pay their personal and professional bills.*

Less forgivable are the journalists who refuse to look seriously at the issue even as they devote inordinate amounts of time to it. This is a serious disservice, because the media-consuming public may want to seriously consider how to allocate resources to address different problems. Call me crazy, but knowing the facts seems important for this process. And in this case it’s not just that the facts are a little out of line with the narrative — they absolutely and dramatically contradict it.

Now for the fact you think I would be reluctant to mention: for the first time in two decades, the rate of property-damage-only accidents has increased for three years in a row. This may be a better measure of accident risk, because the fatality numbers could be partly driven by things like improved medical response time or auto safety devices. Still, property-only accidents per mile are down 21% since 1994 (while mobile phone subscriptions have risen more than 1200%).

That is an interesting turnaround, worth looking into. Unfortunately, I don’t have much confidence in the current crop of experts to offer a credible explanation for it.

* It’s no more surprising than academic professional association staff defending journal paywalls.

Research on teen crashes confirms that reporters selling books on phone risks hype phone risks

Using phones while driving is dangerous and should stop. But the focus on this issue distracts us from other dangers in driving (which have — you’d never believe from the news — declined rapidly in recent decades). And it distracts us from the broader danger of relying on motor vehicle transportation.

I dwell on this subject because it offers lessons beyond its substantive importance (see all the posts under the texting tag). Today’s lesson is about conflicts of interest in the news media.

The AAA Foundation for Traffic Safety published a study of about 1,700 moderate or severe car crashes in which people ages 16-19 were driving. To identify possible causes of the crashes, they used cameras and motion sensors in the cars, and analyzed the seconds before each crash. The headline result probably should have been that 79% of the crashes occurred when teens were driving too fast. But that’s apparently not news, so the AAAFTS and all the news media reporting the story focused on the fact that 59% of the crashes showed distraction as a likely cause.

The report website highlighted the data on distractions, and that’s reasonable. One of their findings is that distractions in their survey account for a greater proportion of accidents than are reported officially — something we’ve assumed but have had trouble establishing empirically. So that’s useful. They used this graphic:

By this accounting, phones were involved in 12% of crashes, second only to interacting with passengers. But this is an artifact of the way the categories are binned. And a lot of smaller categories are left out of the figure, such as eating and drinking (2%), operating vehicle controls (3%), looking at another vehicle (4%), or smoking-related distractions (1%). (Note also that one crash can have multiple related distractions, but they don’t report the overlaps so you can’t do anything about them.)

So I redid the categories. I don’t see why eating and drinking should be a separate category from grooming, or why singing/dancing should be separate from adjusting the radio. So I made a new category called “physically doing something besides driving,” which includes eating or drinking, using an electronic device (besides phone), grooming, reaching for an object, smoking-related activity, operating vehicle controls, and singing/dancing to music. Also, for some reason their figure lists “looking at something outside the vehicle” but only includes “attending to unknown outside vehicle” in that category. I added two other types of distraction to that category, “attending to another vehicle” and “attending to person outside” — bumping up the outside distraction substantially.

Here’s my new version of their figure based on the same data (from table 13 in the report). It’s more comprehensive but uses fewer categories:

Now cellphones are fourth. So that’s a lesson about using arbitrary category collapsing and then ranking the categories. (This happens all the time with occupations, for example, where people say, “The top X occupations…” but the occupations reflect different levels of granularity.)

Anyway, back to cellphones

The New York Times reporter Matt Richtel won a Pulitzer Prize for his reporting on distracted driving. And he published a book — A Deadly Wandering — that tells the tragic story of a driver who killed someone while he was texting. Unfortunately, he is prone to hyping the problem of texting, which his audience is unfortunately prone to fixating on. I previously pointed out that, on the website promoting his book, his publisher uses an extremely wrong statistic, claiming that texting “continues to claim 11 teen lives per day.” He has mentioned this statistic (or its variant, that texting kills more teens than drunk driving) on Twitter, and also in media appearances. I pointed out that this number is more than all the teens killed in motor vehicle accidents, so it’s obviously baloney. I emailed Richtel about this, and he told me he would “get it fixed.” I emailed the publisher. I emailed the Times. I emailed the Diane Rehm show. No one changed anything. Cellphone crashes are like child abuse: people will believe any statistic about how bad it is and attack anyone who’s skeptical.

Of course I don’t want to minimize the problem of distracted driving, and there’s nothing wrong with telling people it’s dangerous. And it’s not my area of expertise. So I’ve only given the issue a few hours. But playing into a public hysteria about a very narrow, behaviorally-driven problem, rather than exposing the systemic problem that it reflects, is not good.

And now that Richtel is selling a book about texting, he’s got a conflict of interest — if he hypes the problem in his NYT stories, he makes more money. So here’s the NYT headline on his story:

And this is his lead paragraph:

Memo to parents: Distracted driving by teenagers is riskier than previously thought, particularly when it comes to multitasking with a cellphone.

Again, it is true the report finds cellphone distraction causes more accidents than police reports have shown — so this is not irrelevant — though, of course, even with the new accounting they still cause orders of magnitude less than Richtel’s own promotional site claims. But mentioning phones in the headline sets the NYT apart from most of the coverage of this report:

• Washington Post: “AAA: 58 percent of teens involved in traffic crashes are distracted”
• ABC News: “Distractions a Problem for Teen Drivers, AAA Study Finds”
• Houston Chronicle: “Distraction a factor in 6 in 10 teen driver crashes”
• Chicago Tribune: “Distracted driving a key contributor to teen crashes, study shows”

On my first page of Google News searches, only the LA Times also mentioned phones: “Teen drivers distracted by cellphones, talking in most crashes.”

Who cares?

Some people who are tired of me complaining about this think you can’t have too much hype about safe driving, so who cares? But the distraction matters. The evidence that phones are a fundamental cause — a social cause — of accidents and deaths is very weak, although they are certainly the proximate cause in many cases. But we don’t have randomized controlled trials to test the effects of phones. I suspect the people crashing while futzing with their phones are mostly the same people who would be crashing for some other reason if cellphones didn’t exist.

When I look at the video compilation the AAA put out to accompany their report — which mostly shows teens crashing while using their phones — I am struck by what terrible drivers they are. They look down for three seconds and drive straight off the road without noticing. In contrast, I routinely see people driving on the freeway completely absorbed in their phones — driving obnoxiously slowly but using their peripheral vision to keep going straight. They are at grave risk of an accident if something crosses their path or traffic stops, but they’re not veering all over the road. Their slow speed probably mitigates their risk of crashing. I AM NOT RECOMMENDING THIS, I’m just saying: bad drivers cause accidents, and if you give them a phone they’ll use it to cause an accident.

Did you know teen driving fatalities have fallen by more than half in the last decade? (During that time incidentally, teen suicides have risen 45%.) Did you know that, from 1994 to 2011, mobile phone subscriptions increased more than 1200% while the number of traffic fatalities per mile driven fell 36% (and property-damage-only accidents per mile fell 31%)? Don’t count on Matt Richtel to tell you about this.

And yet, of course, thousands of people die in car accidents every year in the U.S. — at rates higher than the vast majority of other rich countries. But as long as people drive, there will be bad drivers. If we really cared, we would replace individual cars with mass transit (or self-driving cars) — putting transport in the hands of computers and professionals. Nothing’s perfect, of course, but preventing car accidents isn’t rocket science, and blaming a systemic problem on the individual behavior of predictably error-prone drivers doesn’t seem likely to help.

Why you can’t understand the texting and driving problem in one chart, in one chart

The other day I argued that focus on the “texting-white-driving epidemic” diverts attention from the dangers of driving generally. Here’s a different direction.

The contemporary fascination with using data to tell stories runs up against the need to tell stories in the length of a tweet or in one chart, sometimes resulting in data-focused news that uninforms people rather than informing them.

So, I may not be able to tell the whole teen car death story in one chart, but I can show that you can’t reduce the whole teen car death story to a texting epidemic in one chart (source).

The rate at which teen drivers are involved in fatal crashes has fallen 55% in the last 10 years, faster than the rate for all other age groups (which are also falling). This is part of a long term trend, which has accelerated in the last 10 years. Between 2002 and 2008 alone, the number of text messages sent in the US increased from almost none to more than 100 million per month.* According to the Centers for Disease Control and Prevention’s 2011 national Youth Risk Behavior Survey, reported in Pediatrics, 45% of teens say they texted while driving in the past 30 days — compared with only 10% who said they drove when they had been drinking. An astonishing 12% of teens said they text while driving every day.**

Far be it from me to decide what the public pays attention to. However, we should understand that in this era of distraction there is an opportunity cost to focusing on any one thing. For example (source):

Incidentally, there is a possible clue in that Pediatrics article as to why accident rates aren’t rising due to all this texting. The teens who text while driving are much more likely to engage in other risky behaviors: driving drunk, riding with drunk drivers, and not wearing seatbelts. So texting deaths may to some extent be displacing deaths those same teens would have caused in other ways.

Follow this series of posts at the texting tag.

Notes:

*Thank linked paper argues that texting is contributing to the increase in distracted driving deaths, based on cellphone subscription rates and texts sent per month. It’s plausible but not entirely convincing, because I have doubts about the measure of distracted driving deaths (which rely on local police reports, fluctuate wildly, and include lots of labels, including “carelessness”). They don’t analyze the trend in total traffic deaths.

**This fact may be the source of the myth that 11 teens die from texting and driving every day (less than 8 die daily from all motor vehicle accidents), because someone got carried away by lab studies showing texting while driving was as dangerous as drinking and driving and just extrapolated.

 

The number one cause of traffic fatalities

Please don’t text while driving.

Note: I have updated this post to reflect a response I received from Matt Richtel.

A data illustration follows the rant.

I don’t yet have a copy of Matt Richtel’s new book, A Deadly Wandering: A Tale of Tragedy and Redemption in the Age of Attention. Based on his Pulitzer-prize winning reporting for the New York Times, however, I’m afraid it’s unlikely to do justice to the complexity of the relationship between mobile phones and motor vehicle accidents. Worse, I fear it distracts attention from the most important cause of traffic fatalities: driving.

A bad sign

The other day Richtel tweeted a link to this old news article that claims texting causes more fatal accidents for teens than alcohol. The article says some researcher estimates “more than 3,000 annual teen deaths from texting,” but there is no reference to a study or any source for the data used to make the estimate. As I previously noted, that’s not plausible.

In fact, only 2,823 teens teens died in motor vehicle accidents in 2012 (only 2,228 of whom were vehicle occupants). So, I get 7.7 teens per day dying in motor vehicle accidents, regardless of the cause. I’m no Pulitzer-prize winning New York Times journalist, but I reckon that makes this giant factoid on Richtel’s website wrong, which doesn’t bode well for the book:

In fact, I suspect the 11-per-day meme comes from Mother Jones (or someone they got it from) doing the math wrong on that Newsday number of 3,000 per year and calling it “nearly a dozen” (3,000 is 8.2 per day). And if you Google around looking for this 11-per day statistic, you find sites like textinganddrivingsafety.com, which, like Richtel does in his website video, attributes the statistic to the “Institute for Highway Safety.” I think they mean the Insurance Institute for Highway Safety, which is the source I used for the 2,823 number above. (The fact that he gets the name wrong suggests he got the statistic second-hand.) IIHS has an extensive page of facts on distracted driving, which doesn’t have any fact like this (they actually express skepticism about inflated claims of cellphone effects).

After I contacted him to complain about that 11-teens-per-day statistic, Richtel pointed out that the page I linked to is run by his publisher, not him, and that he had asked them to “deal with that stat.” I now see that the page includes a footnote that says, “Statistic taken from the Insurance Institute for Highway Safety’s Fatality Facts.” I don’t think that’s true, however, since the “Fatality Facts” page for teenagers still shows 2,228 teens (passengers and drivers) killed in 2012. Richtel added in his email to me:

As I’ve written in previous writings, the cell phone industry also takes your position that fatality rates have fallen. It’s a fair question. Many safety advocates point to air bags, anti-lock brakes and wider roads — billions spent on safety — driving down accident rates (although accidents per miles driven is more complex). These advocates say that accidents would’ve fallen far faster without mobile phones and texting. And they point out that rates have fallen far faster in other countries (deaths per 100,000 drivers) that have tougher laws. In fact, the U.S. rates, they say, have fallen less far than most other countries. Thank you for your thoughtful commentary on this. I think it’s a worthy issue for conversation.

I appreciate his response. Now I’ll read the book before complaining about him any more.

The shocking truth

I generally oppose scare-mongering manipulations of data that take advantage of common ignorance. The people selling mobile-phone panic don’t dwell on the fact that the roads are getting safer and safer, and just let you go on assuming they’re getting more and more dangerous. I reviewed all that here, showing the increase in mobile phone subscriptions relative to the decline in traffic accidents, injuries, and deaths.

That doesn’t mean texting and driving isn’t dangerous. I’m sure it is. Cell phone bans may be a good idea, although the evidence that they save lives is mixed. But the overall situation is surely more complicated than TEXTING-WHILE-DRIVING EPIDEMIC suggests. The whole story doesn’t seem right — how can phones be so dangerous, and growing more and more pervasive, while accidents and injuries fall? At the very least, a powerful part of the explanation is being left out. (I wonder if phones displace other distractions, like eating and putting on makeup; or if some people drive more cautiously while they’re using their phones, to compensate for their distraction; or if distracted phone users were simply the worst drivers already.)

Beyond the general complaint about misleading people and abusing our ignorance, however, the texting scare distracts us (I know, it’s ironic) from the giant problem staring us in the face: our addiction to private vehicles itself costs thousands of lives a year (not including the environmental effects).

To illustrate this, I went through all the trouble of getting data on mobile phone subscriptions by state, to compare with state traffic fatality rates, only to find this: nothing:

What does predict deaths? Driving. This isn’t a joke. Sometimes the obvious answer is obvious because it’s the answer:

If you’re interested, I also put both of these variables in a regression, along with age and sex composition of the states, and the percentage of employed people who drive to work. Only the miles and drive-to-work rates were correlated with vehicle deaths. Mobile phone subscriptions had no effect at all.

Also, pickups?

Failing to find a demographic predictor that accounts for any of the variation after that explained by miles driven, I tried one more thing. I calculated each state’s deviation from the line predicted by miles driven (for example Alaska, where they only drive 6.3 thousand miles per person, is predicted to have 4.5 deaths per 100,000 but they actually have 8.1, putting that state 3.6 points above the line). Taking those numbers and pouring them into the Google correlate tool, I asked what people in those states with higher-than-expected death rates are searching for. And the leading answer is large, American pickup trucks. Among the 100 searches most correlated with this variable, 10 were about Chevy, Dodge, or Ford pickup trucks, like “2008 chevy colorado” (r = .68), shown here:

I could think of several reasons why places where people are into pickup trucks have more than their predicted share of fatal accidents.

So, to sum up: texting while driving is dangerous and getting more common as driving is getting safer, but driving still kills thousands of Americans every year, making it the umbrella social problem under which texting may be one contributing factor.

I used this analogy before, and the parallel isn’t perfect, but the texting panic reminds me of the 1970s “Crying Indian” ad I used to see when I was watching Saturday morning cartoons. The ad famously pivoted from industrial pollution to littering in the climactic final seconds:

Conclusion: Keep your eye on the ball.

Deciphering a well-told data story, cars are good for kids edition

Brad Wilcox has written up his best case for how marriage protects women and girls from violence. I discussed his initial post earlier, but the blowup has prompted me to provide more general advice for the critical data citizen — reader, writer, and editor — who has to decide what to believe when someone comes at them with a data story.

I have some tips about that at the end, but first this elaborate setup.

The information in this section is true

Consider three stories:

• When Melanie Thernstrom’s toddler, Kieran, first ate cheese, he immediately had a massive allergic attack. His face swelled, his skin turned red and scaly, and he started gasping for breath. They jumped in their car and rushed to the hospital, where doctors were able to save him.
• Chicago mother Tynisha Hilliard had six children in the car when someone opened fire. “Mommy, I’m shot,” said her nine-year-old boy from the back seat. Hilliard immediately sped to the nearest hospital. “My reaction was to save my son. That’s all I can do, save my son,” she said. After emergency surgery for a gunshot wound to the chest, the boy was expected to survive.
• When Dodgers catcher A. J. Ellis’s wife, Cindy, went into labor, they hopped in the car and headed for NYU hospital, normally a 35-minute drive. Despite racing through traffic with a police escort, they didn’t make it in time – the baby was born in the back seat – but they arrived at the hospital moments later, met by an emergency crew that whisked mother and child to care and safety in the hospital.

What do these stories have in common? Children’s lives saved by cars.

Is this part of a wider phenomenon? I know what you’re thinking: The pollution from cars hurts children, the vast resources devoted to infrastructure for cars could be spent instead in ways that help children, the need for gas causes wars all the time, and the individualism promoted by car culture contributes to social isolation instead of community efficacy.

Maybe. But let’s theorize a little. Here are three ways cars might be good for children’s health:

• Kids whose families have cars can get them to doctors in an emergency. Considering that in modern societies a lot of what kills children is various kinds of accidents and medical emergencies, this could be a major advantage.
• Say what you want about individualism, but it’s emerged as a modern character trait in tandem with the cultural shift that brought us the view of children as priceless individuals. Car culture is a major prop of individualism, so it’s reasonable to hypothesize that people who drive individual cars are more totally devoted to their priceless individual children’s well-being (rather than, say, the well-being of children in general).
• Being able to transport oneself at will — any time, any place — may create a sense of self-efficacy, of mastery over one’s environment, which makes people refuse to accept failure (or illness or death), and thus devote themselves more confidently to their survival and the survival of their children.

Don’t take a theoretical word for it, though — let’s go to the data. Here are three small studies.

Cars and children’s health across countries

First we examine the relationship between the number of passenger cars per capita and the rate of child malnutrition in 110 countries (all the countries in the World Bank’s database that have measures of both variables in the last 10 years — mostly poor countries). The largest — India, China, Brazil, and the USA — are highlighted (click to enlarge).

This is a very strong relationship. This single variable, cars per capita, statistically explains no less than 67% of the variation in child malnutrition rates.

But, you liberals object, cars are surely more common in wealthier countries, so this relationship may be spurious. Sure, income and cars are positively correlated (r=.86, in fact). But when I fit a regression model with both per capita income and per capita cars, cars still have a highly significant statistical association with malnutrition (p<.001). (All the regression models are in the appendix at the end.)

Cars and child death rates across US states

Second, we take a closer look within the United States.  Here there is a lot less variation in both the number of cars and the condition of children. Still, there is a clear relationship between private cars per person and the death rate of children and teenagers: Children are substantially less likely to die in states with more privately owned passenger cars (click to enlarge).

Again, there is less variation in income between U.S. states than there is between countries of the world. But to make sure this is not just a function of state income, I fit a regression model with cars and a control for median household income. The statistical effect of private cars remains significant at the p<.05 level, confirming it is unlikely to be due to chance.

Car commuting and children’s disabilities within the US

Third, let’s go still further, not just comparing US states but comparing children according to the car-driving habits of their parents within the US. For this I got data on children’s disabilities (four kinds of disability) and the means of transportation to work for their parents using the 2010-2012 American Community Survey, with a sample of more than 700,000 children ages 5-11.

Sure enough, children who live with parents who drive to work are substantially less likely to have disabilities than those who don’t live with a parent who drives to work:

Again, could this be because richer families are more likely to include car-driving parents? The regressions (below) show that, although it is true that children in richer households are less likely to have disabilities, the statistical effect of parents’ commuting method remains highly significant in the model that includes household income.

In summary: Children are less likely to be malnourished if they live in a country with more cars per person; they are less likely to die if they live in a state with more cars per person, and they are less likely to have disabilities if they live with parents who commute to work by car. All of these relationships are statistically significant with controls for income (of the country, state, or family). These are facts.

One interpretation

Compare this analysis to the question of marriage and violence. In their piece for the Washington Post (discussed here), Brad Wilcox and Robin Fretwell Wilson wrote about #YesAllWomen:

This social media outpouring makes it clear that some men pose a real threat to the physical and psychic welfare of women and girls. But obscured in the public conversation about the violence against women is the fact that some other men are more likely to protect women, directly and indirectly, from the threat of male violence: married biological fathers. The bottom line is this: Married women are notably safer than their unmarried peers, and girls raised in a home with their married father are markedly less likely to be abused or assaulted than children living without their own father.

With the facts above I can accurately offer this parallel construction:

Some cars pose a real threat to the health and safety of children. But obscured in the public conversation about auto safety, pollution, and environmental degradation is the fact that some other cars are more likely to protect children, directly and indirectly, from threats to their health and safety: cars driven by their own, responsible, caring parents. The bottom line is this: Children in places with more cars — and in families where parents commute by car — are notably healthier than peers without cars.

At the end of his followup post, Brad concludes:

Of course, none of these studies definitively prove that marriage plays a causal role in protecting women and children. But they are certainly suggestive. What we do know is this: Intact families with married parents are typically safer for women and children. … That’s why the conversation about violence against women and girls … should incorporate the family factor into efforts to reduce the violence facing women and girls.

I am equally confident in my conclusion:

Of course, my brief studies don’t definitively prove that cars plays a causal role in protecting children’s health and safety. But they are certainly suggestive. What we do know is this: Societies and families with cars are typically safer and healthier for children. That’s why the conversation about children’s well-being should incorporate the car factor into efforts to reduce the harms too many children continue to experience.

Another interpretation

Both the marriage story and the car story are misleading data manipulations that substitute data volume for analytical power and present results in a way intended to pitch a conclusion rather than tell the truth.

When is a non-causal story “certainly suggestive”? When the person giving you the pitch wants you to believe the conclusion.

Please do not conclude from this that all data stories are equally corrupt, and everyone just picks the version that agrees with their preconception. Not all academics lie or distort their findings to fit their personal, political, or scientific conclusions. I may be more motivated to criticize Brad Wilcox because I disagree with his conclusions (and there may be people I agree with who use bad methods that I haven’t debunked), but that doesn’t mean I’m dishonest in my interpretation and presentation of evidence. Like a real climate scientist debunking climate-change deniers, I am happy that discrediting him is both morally good and scientifically correct (and I think that’s not a coincidence).

There are two main problems with both the cars story and the marriage story. First is selection into the independent variable condition (marriage and car ownership). People end up in these conditions partly because of their values on the dependent variable. For example, women in marriages are less likely to be raped on average because women don’t want to marry men who have raped them, or likely will rape them — the absence of rape causes marriage. In the case of children with disabilities, there is evidence that children’s disabilities increase the odds their parents will divorce (which means at least one of the parents isn’t in the household and so can’t be a car-commuting parent in the ACS data).

The other main problem is omitted variables. Other things cause both family violence and children’s health, and these are not adequately controlled even if researchers tell you they control for them. Controlling for household income (and other easily-measured demographics) does not capture all the benefits and privileges that married (or car-owning) people have and transfer to their children. For tricky questions of selection and omitted variables, we need to get closer to experimental conditions in order to provide causal explanations.

Tips for critical reading

So, based on Wilcox’s car story and my car story, here are practical tips to help you avoid getting hoodwinked by a propagandist with a PhD — or a data journalist looking at a mountain of data and a tight deadline. These are some things to watch out for:

Scatter plot proof

Impressive bivariate relationships; they may be presented with mention of control variables but no mention of adjusted effect size. That’s what I did with my scatter plots above. If you have adjusted results but don’t show them, it’s selling a small net effect with a big unadjusted label. (Wilcox examples here; Mark Regnerus does this, too.)

Axis truncation

A classic example is the Obama food stamp meme, but Wilcox had a great example a few years ago when he wanted to show the drop in divorce that resulted from hard times pulling families together during the recession. If you assume divorce is always going up (it fell for decades), this looks like a dramatic change (he called it “the first annual dip since 2005”):

No head-to-head comparison of alternative explanations

This is a lot to ask, but real social scientists take seriously the alternative explanations for what they observe, and try to devise ways to test them against each other. Editors often see this as a low-hanging fruit for removal, because cutting it both shortens the piece and strengthens the argument. In the rape versus marriage story, Wilcox nodded to the alternative explanation that “women in healthy, safe relationships are more likely to select into marriage” — which he called “part of the story” — but he offered nothing to help a reader or editor adjudicate the relative size of that “part” of the story. This connects to the next red flag.

Greater than zero proof

Sometimes just showing that something exists at all is offered as evidence of its importance. That’s why I included three anecdotes about children being saved by private passenger cars — it happened, it’s real. The trick is to identify whether something matters in addition to existing. Here’s a Wilcox example where he showed that a tiny number of people said they didn’t divorce because of the recession; here’s an example in which Nate Cohn at the NYTimes Upshot said that 2% of Hispanics changing their race to White was “evidence consistent with the theory that Hispanics may assimilate as white Americans.” Neither of these provide any comparison to show how important these discoveries were relative to anything else — other reasons people delay divorce? other reasons for race-code changes? — they just exist. This is reasonable if you’re discovering a new subatomic particle, but with social behavior it’s less impressive.

Piles of studies

The reason I presented the car results as the three separate “studies” was to make the point that you can have a lot of studies, but if none of them prove your point it doesn’t matter. For example, in his post Wilcox linked to a series of publications about how children whose parents weren’t married were more likely to be sexually abused, but none of them handle the problem of selection into marriage I described above. Similarly, a generation of research showed that women who have babies as teenagers suffer negative economic consequences, but those effects were all exaggerated because people didn’t take selection into account (women with poor economic prospects are more likely to have babies as teenagers).

Describing one side of inequality as a social good

Let’s say that, in street fights, the person with a gun beats the person with a knife more than 50% of the time. Do we conclude people should have more guns? Some benefits are absolute and have no zero-sum quality to them. (I can’t think of any, but I assume there are some.) Normally, however, we’re talking about relative benefits. The benefits of marriage, or the economic benefits of education, are measured relative to people who aren’t married or schooled.

The typical description of such a pattern is, “This causes a good outcome, we should have more of it.” But we should always consider whether the best thing, socially, might be to reduce the benefit — that is, solve the problems of the people who don’t have the asset in question — rather than try to increase the number of people with the asset.

The benefit of cars that comes from being able to get to the hospital quicker may only be relative to the poor suckers stuck in an ambulance while your personal cars are blocking up Manhattan.

Ambulance stuck in Manhattan, by Philip Cohen

Appendix: Regression results