ECONOMIC ANALYSIS: Wireless world making research more difficult

If you’re thinking of getting a job as a researcher, you should know it is a lonely life we lead. The world outside our offices sees issues in black and white that appear to us as infinite shades of gray. And when we occasionally emerge from our cubicles to deliver our results to the public, we are told either that we have a stranglehold on the obvious or that we have no concept of the real world.

But the world demands the answers that only research can provide, so we keep on. In fact, we’re busier than ever. Yet the challenges to research, particularly survey research that collects data from individuals, households or businesses, are making the job a lot harder.

It is an irony of the information age that the same technology that makes it so simple and inexpensive to communicate and ship information from anywhere to anywhere can frustrate attempts to collect it. Yet it is doing precisely that, according to a National Center for Health Statistics study of the impact of wireless technology on the integrity and reliability of survey sampling techniques.

The problem is this-the once-insignificant number of cell-phone-only households, who are unreachable using telephone-directory-based sampling methods, has risen to levels that can seriously challenge the integrity of survey-based findings. The once nearly universal land line phone is now found in less than 90 percent of adult households who have some kind of phone service.

This means any information gathered from telephone surveys is telling you information only about what those who have landline phones think or do, not the entire population. Worse yet, those who are left out-whose voices cannot be heard because you can’t find them in a phone directory-are different kinds of people from the ones whose views and characteristics are reflected in your data.

For example, the NCHS delivers the astounding statistic that half of wirelessonly adults are less than 30 years of age. Think of that the next time you analyze your company’s marketing data on product penetration in the young adult market.

Renters are far more likely to be in households that are wireless-only, with just 5.8 percent of homeowners cutting the umbilical cord to their landline phones. Those living in poverty, living in the South, or of Hispanic origin were more likely to be wireless-only as well.

For some special populations, of interest to some health researchers, the gaps in landline phone coverage are especially alarming. Those who do research on HIV and AIDS, for example, should know that more than half of households consisting of unrelated adults living together-which overlaps with much of the gay and lesbian populations-are wireless-only.

How did NCHS discover this? By doing what few of us can usually afford to do: knocking on doors. The in-person survey still works, but with costs several orders of magnitude higher than what telephone surveys present. And if you’re thinking your information needs can all be met using Web-based surveys, you’ve opened a new can of worms that it would take another column to go into. Suffice it to say the results of Web-based surveys-of the general population, at least-are several orders of magnitude less representative than even today’s compromised telephone-based surveys deliver.

And even if we researchers could call you on your cell phones-which I’d guess few Americans want to see happen-there still would be the difficulty of knowing where you are. The younger population more likely to be wired is also more likely to be mobile, with area codes and physical location less and less connected.

If you don’t care about survey quality, you have nothing to worry about. The surveys and polls that dominate so many discussions these days aren’t going to go away. But the crisp, precise-looking numbers they generate are becoming less and less valuable, and decisions based on what they say are becoming more and more risky.