Tom H. C. Anderson - Next Gen Market Research™

Stop Chasing the Big Data; Mid Data makes more sense
[Re-posted from OdinText.com Blog]

After attending the American Marketing Association’s first conference on Big Data this week, I’m even more convinced of what I already suspected from speaking to hundreds of Fortune 1000 marketers the last couple of years. Extremely few are working with anything approaching what would be called “Big Data” – And I believe they don’t need to – But many should start thinking about how to work with Mid Data!

“Big Data”, “Big Data”, “Big Data”. It seems like everyone is talking about it, but I find extremely few researchers are actually doing it. Should they be?

If you’re reading this, chances are that you’re a social scientist or business analyst working in consumer insights or related area. I think it’s high time that we narrowed the definition of ‘Big Data’ a bit and introduced a new more meaningful and realistic term “MID DATA” to describe what is really the beginning of Big Data.

If we introduce this new term, it only makes sense that we refer to everything that isn’t Big or Mid data as Small Data (I hope no one gets offended).

Small Data

I’ve included a chart, and for simplicity will think of size here as number of records, or sample if you prefer.

‘Small Data’ can include anything from one individual interview in qualitative research to several thousand survey responses in longitudinal studies. At this level of size, quantitative and qualitative can technically be lumped together, as neither currently fit the generally agreed upon (and admittedly loose) definition of what is currently “Big Data”. You see, rather than a specific size, the current definition of Big Data varies depending on the capabilities of the organization in question. The general rule for what would be considered Big Data would be data which cannot be analyzed by commonly used software tools.

As you can imagine, this definition is an IT/hardware vendor’s dream, as it describes a situation where a firm does not have the resources to analyze (supposedly valuable) data without spending more on infrastructure, usually a lot more.

Mid Data

What then is Mid Data? At the beginning of Big Data, some of the same data sets we might call Small Data can quickly turn into Big Data. For instance, the 30,000-50,000 records from a customer satisfaction survey which can sometimes be analyzed in commonly available analytical software like IBM-SPSS without crashing. However, add text comments to this same data set and performance slows considerably. These same data sets will now often take too long to process or more typically crash.

If these same text comments are also coded as is the case in text mining, the additional variables added to this same dataset may increase significantly in size. This then is currently viewed as Big Data, where more powerful software will be needed. However I believe a more accurate description would be Mid Data, as it is really the beginning of Big Data, and there are many relatively affordable approaches to dealing with this size of data. But more about this in a bit…

Big Data

Now that we’ve taken a chunk out of Big Data and called it Mid Data, let’s redefine Big Data, or at least agree on where Mid Data ends and when ‘Really Big Data’ begins.

To understand the differences between Mid Data and Big Data we need to consider a few dimensions. Gartner analyst Doug Laney famously referred to Big Data as being 3-Dimensional; that is having increasing volume, variety, and velocity (now commonly referred to as the 3V model).

To understand the difference between Mid Data and Big Data though, only two variables need to be considered, namely Cost and Value. Cost (whether in time or dollars) and expected value are of course what make up ROI. This could also be referred to as the practicality of Big Data Analytics.

While we often know that some data is inherently more valuable than other data (100 customer complaints emailed to your office should be more relevant than a 1000 random tweets about your category), one thing is certain. Data that is not analyzed has absolutely no value.

As opposed to Mid Data, to the far right of Big Data or Really Big Data, is really the point beyond which an investment in analysis, due to cost (which includes risk of not finding insights worth more than the dollars invested in the Big Data) does not make sense. Somewhere after Mid Data, big data analytics will be impractical both theoretically, and for your firm in very real economic terms.

Mid Data on the other hand then can be viewed as the Sweet Spot of Big Data analysis. That which may be currently possible, worthwhile and within budget.

So What?

Mid Data is where many of us in market research have a great opportunity. It is where very real and attainable insight gains await.

Really Big Data, on the other hand, may be well past a point of diminishing returns.

On a recent business trip to Germany I had the pleasure of meeting a scientist working on a real Big Data project, the famous Large Hedron Collider project at CERN. Unlike the Large Hadron Collider, consumer goods firms will not fund the software and hardware needed to analyze this level of Big Data. Data magnitudes common at the Collider (output of 150 million sensors delivering data 40 million times per second) are not economically feasible but nor are they needed. In fact, scientists at CERN do not analyze this amount of Big Data. Instead, they filter out 99.999% of collisions focusing on just 100 of the “Collisions of Interest” per second.

The good news for us in business is that if we’re honest, customers really aren’t that difficult to understand. There are now many affordable and excellent Mid Data software available, for both data and text mining, that do not require the exabytes of data or massively parallel software running on thousands of servers. While magazines and conference presenters like to reference Amazon, Google and Facebook, even these somewhat rare examples sound more like IT sales science fiction and do not mention the sampling of data that occurs even at these companies.

As scientists at Cern have already discovered, it’s more important to properly analyze the fraction of the data that is important (“of interest”) than to process all the data.

At this point some of you may be wondering, well if Mid Data is more attractive than Big Data, then isn’t small data even better?

The difference of course is that as data increases in size we can not only be more confident in the results, but we can also find relationships and patterns that would not have surfaced in traditional small data. In marketing research this may mean the difference between discovering a new niche product opportunity or quickly countering a competitor’s move. In Pharma, it may mean discovering a link between a smaller population subgroup and certain high cancer risk, thus saving lives!

Mid Data could benefit from further definition and best practices. Ironically some C-Suite executives are currently asking their IT people to “connect and analyze all our data” (specifically the “varied” data in the 3-D model), and in the process they are attempting to create Really Big (often bigger than necessary) Data sets out of several Mid Data sets. This practice exemplifies the ROI problem I mentioned earlier. Chasing after a Big Data holy grail will not guarantee any significant advantage. Those of us who are skilled in the analysis of Small or Mid Data clearly understand that conducting the same analysis across varied data is typically fruitless.

It makes as much sense to compare apples to cows as accounting data to consumer respondent data. Comparing your customers in Japan to your customers in the US makes no sense for various reasons ranging from cultural differences to differences in very real tactical and operational options.

No, for most of us, Mid Data is where we need to be.

@TomHCAnderson

[Full Disclosure: Tom H. C. Anderson is Managing Partner of Anderson Analytics which develops and sells patent pending data mining and text analytics software platform OdinText]

Big Data Experts on Hand for Q&A

A bit last minute I know, but wanted to let Next Gen researchers out on the West Coast know about the American Marketing Association’s event next week in San Diego. It’s called Analytics with Purpose: The Human Edge of Big Data and what’s unique about this event is that it has a ‘Science Fair’ component where one leader from each area of Big Data analytics has been selected as the subject matter expert and will answer any questions that come up among attendees during the event.

I was honored to be asked to participate as the Text Analytics expert, and really look forward to being out on the West Coast. Especially, as until very recently, all the text analytics related events have taken place here on the East coast (Boston or NYC).

I understand the event may already be sold out, but if you’re on the West coast and plan to be there please feel free to ask me anything. I’ll do my best to answer any question as honestly as I can. I’m obviously a bit biased in favor of OdinText.com but, that said, I quite often refer clients to other vendors if I see there isn’t a good match.

Hope to see you Monday.

@TomHCAnderson

PS. If you can’t be at the event and have a question on text mining you can contact me directly. Always happy to help if I can.

Anderson Analytics’ New Text Analytics Platform OdinText Positioned to Challenge Status Quo

[Re-posted from OdinText.com Blog]

Anderson Analytics’ OdinText was named as a key challenger to competitors IBM, SAS, Clarabridge and Attensity last week in both the ‘Go to Market Strength’ and ‘Customer Experience Strength’ Quadrants of the 2013 Text Analytics Victory Index Report.

The Customer Experience Strength category is evaluated based on Validity (strength of product) as well as Value (strength in meeting client objectives). Go To Market Strength is based on Viability (stability of company) and Vision (strength of company strategy).

Anderson Analytics CEO and inventor of OdinText, Tom H. C. Anderson, commented

We are pleased to be recognized by industry analysts and customers so early after launch. Clients today recognize that the best innovation typically comes from newer software solutions. The challenge will be never to rest on our reputation, but to continually build it by listening to our customers. Fortunately, this is exactly what our software OdinText is extremely good at. We’re looking forward to a very exciting 2013.

Founded in 2005, Anderson Analytics was the first market research firm to leverage text analytics in consumer insights. The firm has been recognized several times in the past for their innovative methodology and leadership in the text analytics field including awards from industry organizations such as the World Market Research Association (ESOMAR), The Advertising Research Foundation (ARF), and the American Marketing Association (AMA).

The independent study on text analytics software vendors was conducted by Hurwitz & Associates, a strategy consulting, market research and analyst firm that focuses on how technology solutions solve real world customer problems. Hurwitz research concentrates on disruptive technologies, such as Big Data and Analytics, Cloud Computing, Service Management, Information Management, Application Development and Deployment, and Collaborative Computing.

2012 Most Read NGMR Blog Posts

From Text Analytics to Market Research Awards and Predictions, 2012 was a busy year within the Next Gen Market Research community. Below are the 15 most popular blog posts of the year in rank order.

1. Practical Sentiment Analysis and Lies
2. The Year of Text Analytics
3. Next Gen Market Research C-Suite Predictions
4. Text Analysis of 2012 Presidential Debates
5. Social Media Buzz Word Wiktionary
6. 10 Top Social Media Influencers
7. Top 15 Most Innovative Research Firms
8. Leveraging Social Media with Text Analytics
9. Text Analytics Summit Contest
10. 2012 Market Research and Analytics Job Predictions
11. Do You Know Where Social Data Comes From?
12. 2012 NGMR Innovation Winners
13. 2012 Research Predictions (The NGMR Twiteratti)
14. Text Analytics, The Difficult Future You Can’t Avoid
15. Do’s and Don’ts of Social Media

@TomHCAnderson
@OdinText

In 2013 NGMR Professionals Officially Retire the Profession Screener Question

Market research professionals certainly get attached to their survey questions. Much has been written about the hotly debated and supposed ‘Only Question You Ever Need to Ask’ (NPS). But rarely do market researchers think critically about the questions that are less important and asked simply out of tradition such as ethnicity.

Perhaps the Grand Daddy of all useless questions is the all too popular screener question usually worded something like “Do you or anyone in your family work in any of the following industries?”. A list which typically include everything from the various client industries of interest to marketing, pr, advertising and also market research are then listed as options. If the respondent doesn’t choose “None of the above” they are terminated from the study.

In the Next Gen Market Research group this week the unpopularity of this question became apparent. I postulated that respondents, especially panelists know what this first question is for and will usually answer “none of the above”, and that I believed not even market researchers themselves answer this question truthfully.

Of course assumptions aren’t good enough in research, so a poll was quickly created to test the hypothesis. As it turns out, the vast majority of researchers have answered “No” to this question for various reasons (see heated discussion).

What’s the take away? I think we have finally established that this question is useless. If it is important to screen out by industry it needs to be done in a more clever way.

So starting in 2013, don’t be a Last Gen Researcher. This silly question may have served some purpose years ago. But today it’s just a waste of space and sets the wrong tone to your survey (encouraging respondents to take you less seriously - gaming).

Be brief - Be smart- Be Next Gen!

@TomHCAnderson
@OdinText

NGMR Software and Forward Looking MR Clients can make it happen in 2013!

[Re-posted from OdinText blog]

There’s lots of talk about Marketing Research professionals missing the boat on Big Data.

I’m more optimistic in this area - Why?

Marketing research professionals who typically do customer segmentation work, even if it’s typically on smaller data sets (1500-4000 records) have EXACTLY the same skill sets needed for analysis of Big Data!

So what’s missing? Three things:

1. More powerful Software
   Traditional MR statistical packages like SPSS can’t handle Big Data (nor does it do a good job on unstructured/text data). But these tools will become more readily available and affordable.
2. MR Knowledge built into the Software
   For Marketing Research to play a significant role analysis of big data must also become available to more junior analysts. This can happen by involving those that have the aforementioned experience segmenting and working with actual market data in the software design.
3. Client Side Researcher Interest
   Client side researchers must drive this, and they’re not going to do it by hopping on the social media analytics bandwagon. They must seek out valuable Big Data sources that are provide a good ROI on analytics today, not what might do so tomorrow.

Marrying the knowledge of market research analysis with the more powerful software are exactly the two things we’ve been working on over the past couple of years at Anderson Analytics. I would be surprised if others weren’t at least thinking about this as well.

Sharing this knowledge (via next generation market research software) with the rest of the MR industry is the next step.

Therefore I think MR certainly has the possibility of becoming a significant player in the Big Data space. Even a JR level MR analyst has stronger data analysis skills than your average IT professional.

I was recently asked to give a prediction for something that can definitely happen in Market Research for 2013 (both within the NGMR LI group as well as to RFL Communications). Here’s what I said:

“Big Data is the big buzzword right now, but I’m not sure MR clients who would need to drive the effort towards greater use of this data, have the interest, knowledge or clout to get it done.

I’m still amazed at how researchers at fortune 1000 companies seem to focus on just specific areas within traditional MR allowing valuable silos of big data to be analyzed by others or not at all.

The exception currently is twitter and blogs which are over hyped relative to possible ROI.

What I find hardest to believe is that many of these firms are sitting on large amounts of customer service data (call center logs and email complaints and suggestions from hundreds of thousands of customers), while this data really isn’t being analyzed by anyone client side researchers seem to prefer trying to go for a wild goose chase for what Twitterers may or may not be saying.

My prediction and hope is that client side researchers will wake up and realize that customer service data is easier to get to (few barriers to the silo) and more valuable than they thought (can be analyzed through big data and text analytics).

This can certainly happen within the next year, and I plan to personally play a role in making it a reality for our clients!”

Will market researchers play a role in big data - I think the choice is ours. Curious to hear your thoughts and whether you agree.

Do you share my desire to help make it happen this year?

@TomHCAnderson

@OdinText

One more reason for text analytics

This sign at the register of the local CVS here in Ft. Lauderdale caught my eye. Luckily I had already eaten so I didn’t take them up on the offer.

I actually see this type of gaming and other techniques quite often in all kinds of customer satisfaction tracking programs especially in hospitality/travel industry and auto dealerships. We’ve actually done quite a bit of research trying to spot this type of moral hazard and it’s one of many reasons firms should not be relying solely on likert scales but also leveraging text analytics such as OdinText.

@TomHCAnderson
@OdinText

Several Impressive Nominations - Three Winners

Each year the 20,000 Next Gen Market Research group members are asked to nominate the firms and individuals they believe are most innovative. This year our eight Regional Moderators (The NGMR Board of Advisors) carefully considered nominations from 30 firms and several talented market research professionals across three categories.

This week at The Market Research Event (TMRE) in Boca Raton FL, winners where honored during an award ceremony and subsequent panel discussion where they presented their work.

Thought Leadership

Fox Broadcasting Co. in collaboration with trueAnthem was the winner in the Thought Leadership Category for their efforts in re-thinking and defining measures for evaluating social media marketing efforts which consider paid as well as earned influence.

Research Concept Deployment

While eye tracking has been around for a while, the Eye Track Shop was recognized for making the technology more actionable and importantly more accessible than ever before. By triangulating facial characteristics their software can be leveraged via respondents personal webcams rather than expensive eye tracking equipment.

Individual

Finally, for his continual disruption, challenging the quantitative side of our industry from his pioneering work in various conjoint techniques such as CBC and Max-Diff to more recent thinking regarding latent class clustering and big data, Dr. Steven Cohen of In4mation Insights received the individual award.

Please join me in congratulating all the nominees and winners for helping to move our industry forward!

Tom [Founder and Chairman - NGMR]
@TomHCAnderson
@OdinText

Text Analytics News - What NASA and OdinText Have in Common

This year I participated on the closing panel at the Text Analytics Summit in Boston. The most interesting presentation at the event was one given by NASA’s Ashok Srivastava on how they are using text analytics to make air travel safer in the US, and in fact around the world, by monitoring what pilots and ground staff report in regard to various safety and mechanical issues.

Using text analytics in this way is obviously a bit different than how we at Anderson Analytics use our software OdinText. Understanding how different consumers view products and the problems these groups experience requires a different approach with different risk tolerances. While I believe understanding VOC to create the best products and services is very important, imagine the complexity and inherent danger involved in the system NASA analyzes.

Text Analytics Objective: Make Air Traffic Safer

[24 Hour System View]

After the presentation I spoke to Ashok and found it interesting that while we obviously have different objectives, data and therefore approach text analytics somewhat differently, one of the several areas that our approaches have in common is data visualization. It has always surprised me how many ‘text analytics’ firms use simplistic word clouds as output; one of my many pet peeves in our industry. I wasn’t surprised that NASA didn’t use them either. Instead I found that we both include correlations of unstructured data and handle these with very similar statistical/visualization techniques for insights far more meaningful than word clouds.

Ahead of Text Analytics Summit West in San Francisco next month, Text Analytics News contacted us both for an interview on how we think text analytics and big data can become a default business solution. Obviously the ROI needs to be better understood by many companies before even greater adoption takes place. In the private sector, this means there is still a first mover advantage, and firms who implement text analytics early can gain an information advantage over their competition.

Again, industry and domain specifics aside I was pleased to see we are in agreement on many of these issues as well. You can read the Q&A on the Text Analytics News website here.

@TomHCAnderson
@OdinText

PS. I understand Ashok will be speaking Text Analytics Summit West as well. You can get a glance at his presentation here: