If Your Market Research Works, It's Not Wrong

Faris detonated a bomb the other day with his "All Market Research Is Wrong" manifesto. My first thought was "Smokey, this is not 'Nam. This is bowling. There are rules."  I understand the choice of the provocative headline and its role in attracting readers and encouraging debate but it did feel a bit like a case of yelling "Fire!" in a crowded theater. The "All Market Research is Wrong" line was copiously retweeted at least partly to (and, I suspect, by) people who wouldn't bother reading past the fourth sentence with the "epistemologically specious" bit in it.

It's hard to disagree when Faris writes that survey results must be supplemented "with real behavioral data, from direct observation, or from the web - triangulating insights from as many sources as possible." But boiling down an entire "$11bn" industry to a few misguided practices and then dismissing it as useless --  "all the data it generates should be understood as wrong" -- doesn't quite seem right.

Faris discusses market research as "the systematic collection and evaluation of data regarding customers' preferences for actual and potential products and services" and goes on to argue that using online surveys and focus groups is not going to yield predictive results about customer preferences.  (It's appropriately ironic that "market research" in the dictionary he uses is also listed as "uncountable".) This is a rather narrow definition -- compare it with Wikipedia's entry for "market research" ("any organized effort to gather information about markets or customers") or even "marketing research":

"The systematic gathering, recording, and analysis of data about issues relating to marketing products and services. The goal of marketing research is to identify and assess how changing elements of the marketing mix impacts customer behavior."

But even a cartoonish definition of market research as "web based surveys done over the weekend" (as Faris adds in a comment under his post) is too broad to exclude such methodology as, say, discrete choice modeling where options can be presented to a panel via a "web based survey done over a weekend" and which has proven useful for estimating probabilities of demand for one alternative over others.

There are plenty of things that can go wrong with all market research, survey-based or observational: misinterpretation and selective presentation of results, over-reliance on some indicators at the expense of others, wrong instruments for the task, badly formulated questions, sampling errors, biases and so on. All these issues seem to be what IT help desk people call PEBKAC: Problem Exists Between Keyboard And Chair -- a user error, not a system malfunction.

But ultimately, as one particularly pragmatic PhD student commented, "Is your market research helping you in achieving your end? If yes it's not wrong."


Also, coincidentally, picked this gem up on Twitter today - "data is not plural for anecdote."