In Defense of Small Data

“How many people used our new feature last quarter?” That’s a nice and easy question. Run some analytics, and you can have a satisfyingly neat answer. “Why did users abandon our product after using this feature last quarter?” Here, you’re getting into muddy waters.

“Big data” may be sexy, and quant research is great for determining what is happening, but numbers can also be effective at predicting what will happen. However, that doesn’t tell the story of why things are happening. That’s where “thick data” and qualitative research come in, and good product managers know that individual stories can explain or contradict big data.

Quant studies are by far the largest spend among market researchers, garnering 81% of the money spent in 2017 compared to 14% for qualitative research, according to ESOMAR, a global market research association. Despite that imbalance, it is hard for one to exist without the other, and results are strongest when they are combined. So why do so many companies default to quant, first and only?

The Garden of Quant

The recent multi-billion dollar exits of SurveyMonkey and Qualtrics point to how dominant online surveys have become in the world of quant research. So much so that the layperson would easily be excused for thinking that online surveys and quant data are interchangeable.

In reality, a multitude of quant data exists that major companies are tracking in their product and beyond, whether it’s behavioral analytics, location tracking, or sentiment analysis.

In aggregate, these numbers add up to big data. And there are enough numbers that one might wonder what other information a company could possibly need in order to succeed.

The temptation to stick with quant data alone is an understandable one. Quant data is scalable, allowing you to quickly reach thousands of people and achieve statistical significance in mere hours. Writing a survey has become more doable than ever for the everyday marketers, PMs, or UX researchers, something that may have before been the sole turf of research agencies. With the advent of online panels, online surveys have also become arguably the cheapest method of gathering user feedback.

With time-to-market becoming an ever more important differentiator, particularly in the retail space, massive quant studies have become an imperative, not a luxury.

And perhaps the least important but most desirable aspect of focusing on quant data alone is that you don’t have to actually talk to anyone. I see this all the time with our clients at Discuss.io—talking to people takes preparation, and it’s nerve-wracking to be driving a conversation with a living, breathing user of your product.

Yet despite all these advantages, qual data persists, as does interest in how to do it. A quick Google search shows 318,000,000 results for “how to write a survey” and 607,000,000 results for “how to run a focus group.”

Why isn’t quant data enough?

To Get Ahead, Go Small

It’s easy to lull yourself into thinking that quant data is all you need. After all, it’s fast, easy, and cheap to acquire. It doesn’t involve any of that messy, talking-to-people stuff.

Anecdotally, we hear that when research budgets are tight, qual is the first thing to go. And yet stories abound of missed opportunities and giants humbled who ignore or don’t gather qualitative data. One of the most telling is from Tricia Wang, whose TED talk details how Nokia’s singular focus on big data in 2009 kept them from seeing and reacting to the smartphone revolution in its earliest days.

In her work as a technology ethnographer, Wang developed a deep understanding of the motivations and spending habits of emerging markets where Nokia was dominant—China, India, and Mexico. She saw that despite limited salaries, low-income Chinese were drawn to iPhones and the shiny ads that came along with them. The device was seen as an entry to a high-tech life, and consumers were willing to spend half a month’s salary to get one. Despite Wang’s deep conviction that Nokia was in dangerous territory, Nokia dismissed her findings as anecdotal. Their “big data” analysis didn’t match with her findings. “Your surveys, your methods have been designed to optimize an existing business model, and I’m looking at these emergent human dynamics that haven’t happened yet,” she said.  And, in fact, that’s the strength of small data.

Small Data, Big Gains

Nokia couldn’t afford to ignore qualitative data in 2009, and today’s product person can’t either. Household names are losing market share to faster, nimbler companies focused more on usage of their products than purchase of their products.

HBR recently outlined the importance of focusing on users over buyers. And how do you do that? How do you understand what influences someone’s perception of your product’s convenience factor, or why they sign up for a yearly license but usage across that customer’s organization remains low?

You guessed it: small data, in the form of exploratory conversations with real people. Really, though, it’s the combination of research methods that yields the most impact. Yasmine Khan advocates for a quant/qual sandwich. As timelines continue to be compressed, “Agile research” has emerged as a way to quickly iterate on concepts using a mixture of qualitative and quantitative methods.

Quant and Qual: A Perfect Pair

Quant data and online surveys, in particular, have become a necessary part of a business’s marketing and product development process. But relying on quant alone can lull you into a false sense of having all the answers and being data-driven vs data-informed.

In reality, the game changers in terms of new markets, new products, and true innovation come from human intuition and in-depth, unstructured conversations with target users. Though the sample size is much smaller, and the data is harder to analyze, success hinges on uncovering the “why” behind human behavior.