In my preparation, I was putting together notes from previous posts, the critique I co-authored with Andrew Perrin and Neal Caren, the infamous paper itself, and the media coverage of the scandal. And one piece of it I never really questioned got me thinking: his insistence that his dataset was a “a random, nationally-representative sample of the American population.” The news media repeated this assertion routinely, but what does it mean?
The data, collected by Knowledge Networks, are definitely not truly random. But not much is. They have standing panel of participants who get rewards for participating in a certain number of online surveys. The recruitment of the original panel is where the randomness comes in, with dialing (more or less) random phone numbers. But who chooses to be in it is not random, of course. What the firm does, then, is apply weights to the sample. That is, you don’t count each person as one person, you count them as a certain multiple of a person, so that the weighted total sample looks like the target population — in this case all noninstitutionalized American adults ages 18-39.
In the paper, Regnerus offers an appendix which compares his New Family Structures Study to the national population as represented in better, larger samples, such as the Current Population Survey (CPS). He writes:
Appendix A presents a comparison of age-appropriate summary statistics from a variety of socio-demographic variables in the NFSS, alongside the most recent iterations of the Current Population Survey, the National Longitudinal Study of Adolescent Health (Add Health), the National Survey of Family Growth, and the National Study of Youth and Religion—all recent nationally-representative survey efforts. The estimates reported there suggest the NFSS compares very favorably with other nationally-representative datasets.
So, he eyeballs the comparisons and determines the result is “very favorable.” I had previously eyeballed the first few rows of that table and reached the same conclusion. This is the distribution of age, race/ethnicity, region and sex from that table:
So, it looks very similar to the national population as counted by the benchmark CPS. But both of these surveys are weighted on these factors. That is, after the sample is drawn, they change the counts of people to make them match what we know from Census data (which are weighted, too, incidentally). So the fact that NFSS matches CPS on this characteristics just means they did the weights right, so far.
Think about it this way. If I collect data on 6 men and 4 women, it’s easy to call my data “representative” if I weight those 6 men by .83 and the 4 women by 1.25. The more variables you try to match on the harder the math gets, but the principle is the same.
But now I looked further down the table, and Regnerus’s data don’t compare “very favorably” to the national data on some other variables. Here are household income (from CPS) and self-reported health (from the National Survey of Family Growth):
This means that, when you apply the weights to the NFSS data, which produces comparable distributions on age, sex, race/ethnicity and region, you get a sample that is quite a bit poorer and less healthy than the national average as represented by the better surveys.
I was confused by this partly because according to the Knowledge Networks documentation on the NFSS, income was one of the weighting variables.
I don’t know how big an issue this is. Do you? And do you know of a standard by which a researcher or research firm can declare data “nationally representative” in this age of small, fast, low-response, online surveys?