Tom H. C. Anderson - Next Gen Market Research™top binary option strategies
binary option broker with demo account
make yourself a millionaire automatically with binary options
optionsxpress quotes binary option
binary options trading journal january 2014
buy australianbasedbinaryoptionbrokersunder50 shopping
copy binary trades ben williams
options hybrid binary trading software
binary option rainbow char
trade binary options strategy
binary option trading lesso
trade rosneft mcxrosn binary stock options
options binary jobs
optionbot review the worlds first binary options trend indicator
can you make money with binary option tradi
how to cheat at binary options
binary option methods quantitative estimation of ethan
gold binary option system bb 1
continued mt4 binary options strategy tester using custom indicators
tips for improving your binary options trading methods part 2
binary option software free download top 4
blackscholes model for binary options
binary option using paypal review options trading lessons
binary option strategies jobs
free graphs for binary options
binary options broker broker top broker
binary options omni 11 vs striker 9 pro
download auto binary signals trading v20 freeapk
how i trade binary options
signal push binary options revi
binary options technical indicators 2014
dragonoptions leading binary options platform providers spotoption ltd
types of binary options what is a option
best binary options broker in austral
binary options platinum signals review
binary options robot blog
forex binary option system marketer sign in to add to watch list
tag archives best binary option broker
best indicator for binary trading
million dollar insiderbinary options trading reviewed
one hour option binary stock market trading
binary options market review for 25 october
how to trade binary option successfully
currency binary options trading success
banc de binary options
binary option methods with no arguments
binary option pro review options signals twitter
binary options 60 seconds softwa
optionbot 20 review the worlds top binary options indicator
instaforex binary option
options trading uk review binary news
binary affiliates group gt options brokers gtoptions
miami beach trader binary options trader
thread req binary option systems omni 11
handelsstrategien binary option robot
elite trading binary options
how to trade binary options stocks and
binary option robot autotrade
how to win in binary options p
60 in binary
two basic strategies used by binary options traders 2062014
bestbinaryplatform provides an impartial view on trading binary options
tag archives no touch binary option broker
operational mechanisms of binary options
best binary option brokers review
sports trading or binary
the truth about binary option trading online options
the best binary option signal servi
binary 212
binary options trading and scam software finally explained
tips for choosing binary options brokers
overview of binary options trading
binary options hourly signal service
6 reasons binary options trading is like poker
binary options auto trading software
pivot points calculator for binary options trading and forex
binary option brokers with no minimum depos
are binary options really profitable
making good investments with traderush binary option trading
how do binary options brokers make money review software download
watch live binary options tradi
buy francobinaryoptionstradingsignalsreviewpromotionalcodes shopping
my binary options free
january 2014 binary forex options trading software best
binary options trading scams and frauds
binary options trader insight november 25th 2014
binary options brokers that accept liberty reserve deposits
are binary options better than forex
descending triangle trading strategy in binary options
trade chase delights binary options traders with new game
banc de swiss binary options trading platform review bdswiss
binaryoptionsreviewscouk overview
60 second binary options trading software
trading binary options responsibly
binary options bonuses without deposit
best strategy for binary trading
one touch and no touch binary options strategy 101 free mp3 download
binary option affiliate review software download
what are binary options how to make money trading
tag archives forex binary options ultimatum trading system
binary options 60 second strategy tradequicker uk
what to look for in a good binary options broker
tradequicker review binary options 15114 strategy 2014
binary option kraken
the best binary options insigh
sanford j grossman binary option
trading gold with daily binary profits dbpv7
virtual binary option trading
binary options partnerprogramm
binary option methods equipment
money management for binary options
s bransers binary options
binary options trading a new trend in on share buying and selling
1 ranked binary options broker
police to investigate binary options broker scam
review of binary options pro signals option auto trading software
binary trading investment options in the philippines
binary options atr indicator and strategy
trading binary options option techniques
binary options method option robot latest user reviews
the global binary options review
lesson 1 the basics of binary options trading
binary options trading signals franco revi
binary options trading pro strategy with 60 second profit 1
is binary trading allowed in isl
dealing with binary options spreads
gtoptions binary options trading
top 10 binary options
bancdebinary se joint la mode des concours de trading
earn thousands of dollars in binary trading with optionsclick
regulated binary options brokers list review trading charts
binary options indicator tips
differences between traditional trading and binary options
make a living from binary options playing it smart
binary signal bot upsells review options trading on mt4
binary options trading striker9 full review
details about 241743 binary options extreme professional forex system business mt4
binary options trading bl
binary options trading basics 2014 best strategy
tag archives binary options brokers with no minimum deposit
contact binary options home
binarix binary options game
how to trade binary options pairs
the role of planning in trading binary options
binary auto profits trading system
binary spy options trading indicator
is binary options trading a scam or a legit home based business
eztrader review of binary option broker
how to win in binary options dominator
gtoptions digital binary options broker review

More Than Market Research - Gain The Information Advantage

Tom H. C. Anderson - Next Gen Market Research™ header image 6

Forget Big Data, Think Mid Data

March 7th, 2013 · 6 Comments

Stop Chasing the Big Data; Mid Data makes more sense
[Re-posted from OdinText.com Blog]

After attending the American Marketing Association’s first conference on Big Data this week, I’m even more convinced of what I already suspected from speaking to hundreds of Fortune 1000 marketers the last couple of years. Extremely few are working with anything approaching what would be called “Big Data” – And I believe they don’t need to – But many should start thinking about how to work with Mid Data!

BigDataMidDataSmallData

“Big Data”, “Big Data”, “Big Data”. It seems like everyone is talking about it, but I find extremely few researchers are actually doing it. Should they be?

If you’re reading this, chances are that you’re a social scientist or business analyst working in consumer insights or related area. I think it’s high time that we narrowed the definition of ‘Big Data’ a bit and introduced a new more meaningful and realistic term “MID DATA” to describe what is really the beginning of Big Data.

If we introduce this new term, it only makes sense that we refer to everything that isn’t Big or Mid data as Small Data (I hope no one gets offended).

Small Data

I’ve included a chart, and for simplicity will think of size here as number of records, or sample if you prefer.

‘Small Data’ can include anything from one individual interview in qualitative research to several thousand survey responses in longitudinal studies. At this level of size, quantitative and qualitative can technically be lumped together, as neither currently fit the generally agreed upon (and admittedly loose) definition of what is currently “Big Data”. You see, rather than a specific size, the current definition of Big Data varies depending on the capabilities of the organization in question. The general rule for what would be considered Big Data would be data which cannot be analyzed by commonly used software tools.

As you can imagine, this definition is an IT/hardware vendor’s dream, as it describes a situation where a firm does not have the resources to analyze (supposedly valuable) data without spending more on infrastructure, usually a lot more.

Mid Data

What then is Mid Data? At the beginning of Big Data, some of the same data sets we might call Small Data can quickly turn into Big Data. For instance, the 30,000-50,000 records from a customer satisfaction survey which can sometimes be analyzed in commonly available analytical software like IBM-SPSS without crashing. However, add text comments to this same data set and performance slows considerably. These same data sets will now often take too long to process or more typically crash.

If these same text comments are also coded as is the case in text mining, the additional variables added to this same dataset may increase significantly in size. This then is currently viewed as Big Data, where more powerful software will be needed. However I believe a more accurate description would be Mid Data, as it is really the beginning of Big Data, and there are many relatively affordable approaches to dealing with this size of data. But more about this in a bit…

Big Data

Now that we’ve taken a chunk out of Big Data and called it Mid Data, let’s redefine Big Data, or at least agree on where Mid Data ends and when ‘Really Big Data’ begins.

To understand the differences between Mid Data and Big Data we need to consider a few dimensions. Gartner analyst Doug Laney famously referred to Big Data as being 3-Dimensional; that is having increasing volume, variety, and velocity (now commonly referred to as the 3V model).

To understand the difference between Mid Data and Big Data though, only two variables need to be considered, namely Cost and Value. Cost (whether in time or dollars) and expected value are of course what make up ROI. This could also be referred to as the practicality of Big Data Analytics.

While we often know that some data is inherently more valuable than other data (100 customer complaints emailed to your office should be more relevant than a 1000 random tweets about your category), one thing is certain. Data that is not analyzed has absolutely no value.

As opposed to Mid Data, to the far right of Big Data or Really Big Data, is really the point beyond which an investment in analysis, due to cost (which includes risk of not finding insights worth more than the dollars invested in the Big Data) does not make sense. Somewhere after Mid Data, big data analytics will be impractical both theoretically, and for your firm in very real economic terms.

Mid Data on the other hand then can be viewed as the Sweet Spot of Big Data analysis. That which may be currently possible, worthwhile and within budget.

So What?

Mid Data is where many of us in market research have a great opportunity. It is where very real and attainable insight gains await.

Really Big Data, on the other hand, may be well past a point of diminishing returns.

On a recent business trip to Germany I had the pleasure of meeting a scientist working on a real Big Data project, the famous Large Hedron Collider project at CERN. Unlike the Large Hadron Collider, consumer goods firms will not fund the software and hardware needed to analyze this level of Big Data. Data magnitudes common at the Collider (output of 150 million sensors delivering data 40 million times per second) are not economically feasible but nor are they needed. In fact, scientists at CERN do not analyze this amount of Big Data. Instead, they filter out 99.999% of collisions focusing on just 100 of the “Collisions of Interest” per second.

The good news for us in business is that if we’re honest, customers really aren’t that difficult to understand. There are now many affordable and excellent Mid Data software available, for both data and text mining, that do not require the exabytes of data or massively parallel software running on thousands of servers. While magazines and conference presenters like to reference Amazon, Google and Facebook, even these somewhat rare examples sound more like IT sales science fiction and do not mention the sampling of data that occurs even at these companies.

As scientists at Cern have already discovered, it’s more important to properly analyze the fraction of the data that is important (“of interest”) than to process all the data.

At this point some of you may be wondering, well if Mid Data is more attractive than Big Data, then isn’t small data even better?

The difference of course is that as data increases in size we can not only be more confident in the results, but we can also find relationships and patterns that would not have surfaced in traditional small data. In marketing research this may mean the difference between discovering a new niche product opportunity or quickly countering a competitor’s move. In Pharma, it may mean discovering a link between a smaller population subgroup and certain high cancer risk, thus saving lives!

Mid Data could benefit from further definition and best practices. Ironically some C-Suite executives are currently asking their IT people to “connect and analyze all our data” (specifically the “varied” data in the 3-D model), and in the process they are attempting to create Really Big (often bigger than necessary) Data sets out of several Mid Data sets. This practice exemplifies the ROI problem I mentioned earlier. Chasing after a Big Data holy grail will not guarantee any significant advantage. Those of us who are skilled in the analysis of Small or Mid Data clearly understand that conducting the same analysis across varied data is typically fruitless.

It makes as much sense to compare apples to cows as accounting data to consumer respondent data. Comparing your customers in Japan to your customers in the US makes no sense for various reasons ranging from cultural differences to differences in very real tactical and operational options.

No, for most of us, Mid Data is where we need to be.

@TomHCAnderson

[Full Disclosure: Tom H. C. Anderson is Managing Partner of Anderson Analytics which develops and sells patent pending data mining and text analytics software platform OdinText]

[Post to Twitter] 

Tags: Anderson Analytics · Big Data · Datamining · Market Research · Marketing research · Mid Data · Odin Text · OdinText · Small Data · Text Analytics · ama · american marketing association · text mining

6 responses so far ↓

  • 1 Kelley Styring // Mar 7, 2013 at 4:21 pm

    On the money, Tom. And small data is going to help mid data believers understand and communicate the findings with business relevance, simplicity, synthesis and insight.

  • 2 David Rabjohns // Mar 7, 2013 at 5:35 pm

    Great article nicely done.

  • 3 Doug Wicks // Mar 7, 2013 at 8:14 pm

    Thanks for bringing a researcher’s perspective to Big Data. A number of points to add to the conversation. 1) variety – data of different types - video, images, tweets, sensor data - these challenge researchers as they go about analyzing Big Data; 2) arguably, there is a fourth v - veracity - Is this data accurate, trustworthy? Also a key factor for researchers working with Big Data. And of course the fundamental in working with ANY data, big, mid or small - are we asking the right questions, do we have the right data to answer them, and are we providing the insights our clients need when they think about and use Big Data to solve address issues.

  • 4 Tom H C Anderson // Mar 8, 2013 at 2:19 pm

    Thanks, I find analytics/methodology is left out of decision, and it’s all left to IT/Purchasing who try to get a solution that attempts to address everyone’s needs.

    I think this is fine if focused only on operational needs. But if you implement such a system for the purpose of insights, then there needs to be some serious thinking about what data makes sense to analyze together.

    If you can’t currently analyze one Mid Data source properly, perhaps it makes a lot more sense to deal with that before spending lots of time and $ to link them all up to one giant Real Big Database, only to find out it made absolutely no sense to analyze these sources together after all!?

  • 5 Christy Pogorelac // Mar 13, 2013 at 3:42 pm

    Thanks - I attended the conference and thought the panel discussion was extremely interesting.

  • 6 Georgette Asherman // Jul 10, 2014 at 7:20 am

    Tom, you are on the mark. And most importantly you point out the importance of sampling, even when there really is big, big data. I had people tell me that their 100,000 record data set is Big Data, when it is not. Most of our challenges are in Mid Data because many of our small sample approaches stop working.

Leave a Comment