Take Polls With A Grain Of Salt

Take Polls With A Grain Of Salt

One of the more interesting exchanges during my recent testimony to a Congressional Oversight Committee was triggered by a line of questions from Congresswoman Jackie Speier. (D-CA). Congresswoman Speier’s staffers had dug up a press release from Ford citing a Maritz study that found consumers viewed fuel economy as extremely important. She was citing this study in an apparent attempt to undermine a key point of my testimony: Consumers generally weigh other vehicle attributes as more important than fuel economy.

The back and forth heated up (below) when I affirmed that people don’t always mean what they say. Congresswoman Speier pushed further, wanting to know why then we even paid attention to polls at all. I responded that this was a “very good question."

A short time later, Congressman Dennis Kucinich (D-OH) jumped in with a monologue (below) on how I was “one of the most remarkable witnesses we’ve ever had, because (he) came to a town that has totally relied on polls and definitively told us that polls are not scientific.” (Actually, what I said is that polls do not reliably predict what people will actually do.)

I found all this interesting because it appeared Congresswoman Speier and Congressman Kucinich were both sincere in their surprise that I lack confidence in polls. And I found it surprising that they didn’t.

After the hearing, the Washington insiders I talked with made it very clear: Washington is a town that marches to the drumbeat of polls.

Having spent several years earlier in my career on the periphery of the research industry, I find this disquieting. Polls certainly have their uses. One use that I would imagine holds great interest in Washington is predicting election winners. Here they are reasonably accurate. But when it comes to capturing how respondents will behave, polls often come up lacking.

Here are three reasons why:

The first reason is a bias that has been noted and studied since the 1950s. It is called social desirability response bias, which basically means that survey respondents tend to provide answers that they deem will be socially acceptable. An example I shared with Congresswoman Speier was a survey that attempts to assess how important various factors are in an individual’s choice of a career. Survey answers might indicate that quality of life, making a difference, etc. are very important. Yet more rigorous research, which correlates decision factors to actual career choices, shows clearly that money is the top factor.

As most issues before government are inherently “social,” social desirability bias comes into play often. I would have hoped that our elected officials better understood it. (The bias is certainly not a secret. Folks outside the Beltway seem well aware of the bias and it is a standard part of college-level statistics classes.)

The second reason was noted in my written testimony. It is the idea that complex decisions involve trade-offs. People making such decisions weigh various attributes against each other in terms of relative importance. When a survey or poll doesn’t require that a respondent make such trade-offs, often all attributes will be shown to be important.

So when consumers say fuel economy is extremely important, they are not lying. The survey has just not allowed (or required) them to also point out that other attributes are even more important. Yes, fuel economy is important, but passenger capacity might be more important. Or towing capacity. Or comfort. Or (pick your attribute.) And it is these higher-weighted attributes that drive most vehicle purchase decisions.

The third reason is survey design bias; where the survey questions lead a respondent to a particular answer. Most benignly this bias is accidental. But there is a darker side to survey bias as well.

Others have noted before me how Washington is led around by poll results. And they apparently have decided to do something about it. Unfortunately, the something they have decided upon is to launch organizations whose sole function looks to be to influence policy through polling. One such group might be the Small Business Majority. AutoObserver Senior Editor John O’Dell, who covers green issues, wrote a piece on their poll that claims that small businesses back 60 mpg standards. John’s piece exposed the actual questions used in their survey. Read it and you will get an idea as to the lengths some groups seem willing to go to get poll results that support their goals.

A couple years back the Small Business Majority was playing the same game with health care reform. Back then The New York Times’ Small Business Blog posted this: “the Small Business Majority has all the hallmarks of a shadowy interest group, starting with a name that conceals more than it reveals.”

Any one of these issues should be reason enough for Washington to take poll results with a grain of salt. Add all three together and you can see why I flew back to California with a profound feeling of disquiet.

Jeremy Anwyl:  Chief Executive Officer of Edmunds.com. Follow @JeremyAnwyl on Twitter.