Tom H. C. Anderson - More Than Market Research

Greenfield Online’s Hugh Davis and Anderson Analytics’ Tom H. C. Anderson Discuss Online Market Research.

 
Hugh Davis - Cofounder/EVP
Greenfield Online

Today I’m posting answers to a few questions I posed to Hugh Davis, Executive Vice President and co-founder of Greenfield Online. Hugh joined Greenfield Consulting Group in September 1992, and in 1994 he co-founded Greenfield Online with Andy Greenfield. The company was a true industry pioneer at the time; one of the first to advocate using the web for all survey research.

Hugh has been part of the executive management team since 1994 and has played significant contributing roles in growing the business from a two person start-up to a successful 600+ employee publicly traded company (Nasdaq: SRVY). Although Hugh’s management responsibility has grown over the years he has been able to remain very close to managing and improving Greenfield’s Online Panel.

Tom: Some research buyers have raised concerns about data quality over the last couple years and it seems to be mentioned every other month in either Inside Research or one of RFL’s Research Business reports. Yet most clients very rarely ask me about panel quality.

What are your thoughts on this, how much of a problem is it really? Are most clients over or ‘under’ reacting? How does the issue compare B-C vs. B-B wise?

Hugh: There is no question that data quality concerns are real; if organizations don’t closely monitor data quality, their information can easily become compromised.

The best safeguard is to work with a reputable data collection/panel company. Most such firms work hard to stay ahead of the data quality curve. At Greenfield Online, for example, we are constantly instituting new measures to stay ahead of any new ploys suspect respondents might use. And we approach business and consumer respondents differently - with different incentives and safeguards - as another way of ensuring quality.

We firmly believe that it’s important not to let concerns about data quality get in the way of your important research efforts. Rather, choose a reputable partner and get on with business.

Tom: Assuming you agree that there is a data quality issue, what proportion of the data quality problem would you assign to each of the following reasons: 1. Poor quality, boring/long surveys 2. Poor panelist recruitment process 3. Poor panel management practices 4. other

Hugh: 2 and 3. It’s impossible to assign blame for poor data quality to one point of failure; when data is compromised, it usually means the system has failed on a number of levels.

The recruitment process is, of course the first factor that contributes to data quality. You have to look closely at how you recruit survey respondents - what you’ve used to motivate them. If panelists are motivated exclusively by incentives, the quality of their responses is likely to suffer. At Greenfield Online, rather than incentives, we prefer to focus on offering panelists the opportunity to share their views on an issue important to them. Then, we ensure that surveys are structured in such a way as to offer that opportunity.

That brings us to the second area of consideration: the survey itself. If surveys are long and boring, you’re sure to lose respondents’ interest and risk poor quality responses. So it’s important to spend time crafting the instrument itself, making sure it’s interesting and manageable. That can have a major impact on data quality.

It’s also critical to look at the fielding of the survey. Are there controls and proper monitoring in place? For example are respondents completing at 15 minute survey in 5 minutes. Post-survey, it’s important to look at how questions are being answered and remove any panelists who haven’t taken the survey seriously.

In order to really impact data quality, you have to evaluate the entire survey process - from beginning to end. There are also many technical techniques we use at Greenfield to monitor data quality in real-time.

Tom: I’m often surprised at the length of some of the surveys I see being fielded. Many last well over our recommended 15 minutes and approach 30, 45 or even 60 minutes in some cases! Worse most questions seem to be mandatory, while at the same time not all questions are well designed/applicable for all respondents.

What do you think is the ideal online survey length? What proportion of your surveys are over this length? What are the panel companies responsibility in these cases? Any recommendations on how to maintain response rates with longer surveys?

Hugh: We believe the ideal survey length is 18 to 22 minutes. This is usually enough time to gather the depth of information you need, while still holding a respondent’s interest.

But survey length should always be tailored to the audience. For example, a busy executive may have only a few minutes to devote to a survey, making it much more reasonable to ask her to complete three, three-minutes surveys, rather than one lengthy questionnaire.

This is one area where Internet and mobile technology are proving invaluable. Respondents can take Internet surveys at their convenience - easily fitting short, two- or three-minute surveys into lulls in their day.

Of course, there are times when clients are determined to use lengthy questionnaires. In those cases, we counsel them to:

 Be sure respondents understand the level of time-commitment. If a respondent is surprised by the length of the survey, the quality of his answers will certainly be impacted.

 Always let survey-takers know where they are in a survey, so they understand how much more time they must allocate.

 Offer appropriate compensation for the respondent’s time.

Tom: How unique are today’s online panels? What makes various Greenfield Online unique, and why?

Hugh: Greenfield Online has been building and managing online panels since 1994, with millions of completed surveys to its credit. Our singular, 100 percent focus on data collection has allowed us to become experts in the field of panel management. We have learned how to deftly balance the needs of market research firms and respondents. We have also introduced a number of key innovations, including Real-Time Sampling (RTS), which we launched in 2006.

Real-Time Sampling (RTS) enables us to expand our respondent capability by reaching out in real-time to potential survey takers who may not choose to join an online panel. We recruit these respondents via our Global Survey Network and interested respondents are screened provided with a relevant survey opportunity assigned by the Greenfield Online Survey Router. This Survey Router enables Greenfield Online to intelligently assign respondents to relevant surveys based on demographic data. Further, the Survey Router synthesizes multiple sample and recruitment sources to ensure high-quality data.

Tom: It seems to me that everyone seems to be using DIY Survey software such as Survey Monkey, even large client side companies. While I have thus far refused to use any MR product containing the word “monkey” in it, I am a big fan of Z-Pro and a few other tools. I think these software platforms are both economical and robust enough to handle 80% of research needs.

Survey software firms like Confirmit have been doing well in the past, but it seems to me Market Tools ability to cater to two markets is far more clever. Still I am concerned about the very poor surveys I see being fielded by amateur consultants using these tools.

What are your thoughts about the prevalence of DIY survey software?

Hugh: I’m not opposed to DIY survey software across-the-board. In fact, it can be a powerful tool and - because of widespread use -might actually raise awareness of the importance of market research. But there are dangers. Because someone can plunk down $20 online, doesn’t mean they understand market research and the proper way to field a survey. Some of the really poor surveys DIY survey software spawns could certainly give market research a bad name.