Tom H. C. Anderson - Next Gen Market Research™how to download adobe after effects cs6 32 bit
vmware workstation 7.1.3 for windows
aimersoft video converter ultimate for mac 3.6.1
adobe dreamweaver cs5 tutorial pdf download
sony sound forge 10 basics
sony vegas movie studio hd platinum 11 descargar gratis
zonealarm pro 8 download
adobe creative suite 6 design standard student and teacher edition trial
adobe photoshop elements 9 oder 10
how to insert page number in adobe indesign cs3
buy adobe acrobat xi pro download
microsoft windows 8 pro pack addon
autodesk revit 2015 operating system unsupported
adobe premiere pro cs6 plug in free
eyeon fusion 6.4 manual
tuneup utilities 2008 ver.7.0.8008 download
adobe after effects cs5 download full version 32 bit
adobe dreamweaver cs5.5 manual
adobe indesign cs4 mac gratis
adobe photoshop cs3 extended effects
adobe illustrator cs4 mac 1 link
adobe illustrator cs4 video tutorials for beginners
adobe creative suite 4 master collection key
coreldraw technical suite x6 traduo
microsoft visual studio 2012 ultimate 32bit price
pixelmator mac espaol
licencia adobe after effects cs5.5
snagit 11 download link
sony dvd architect pro 6 key
pinnacle studio 16 ultimate rar
descargar adobe indesign cs6 gratis para mac
corel digital studio 2010 activator
adobe after effects cs6 initializing media core
average time to learn spanish using rosetta stone
please insert microsoft visual studio 2010 professional disk 1 now
microsoft windows 8.1 professional with update 3264bit englishunited kingdom
microsoft expression studio 4 ultimate download
microsoft office for mac home and business 2011 student discount
microsoft office home and student 2013 rt download
coreldraw graphics suite x5 demo
adobe indesign cs4 installer
adobe creative suite 6 master collection vs design and web premium
adobe indesign cc 9.0 download
corel draw 11 mac os x lion
adobe photoshop elements 10 user guide download
camtasia studio 8 key and name 2013
microsoft visual studio 2010 ultimate 32bit english download
parallels desktop 9 for mac amazon
vmware workstation 6.5 7
acdsee pro 3 espaol gratis
adobe creative suite 6 cs6 design standard for mac
microsoft office enterprise 2007 key kostenlos
boris red 5 vegas
adobe photoshop elements 8 iso password
quarkxpress 8 and mountain lion
qual o numero de serie do adobe dreamweaver cs6
valid product key for microsoft office 2007 enterprise edition
microsoft windows 8.1 64bit fi dvd oem
autodesk inventor 2015 pdf
codigo validacion quarkxpress 8 mac
microsoft windows 8.1 pro kopen
adobe captivate 5 right click
how to get parallels desktop 9 for mac free
adobe premiere pro cs4 portable softonic
microsoft windows 8 enterprise iso
adobe illustrator cs4 4sh
adobe audition cs6 mac plugins
adobe flash professional cs5 tutorial espaol
corelcad 2013 mac os x
eset smart security 6 setting protection password
adobe premiere pro cs5.5 32 bit download
microsoft visio professional 2013 sp1
sage act premium 2012 tutorial
microsoft windows 8 dvd player
microsoft office professional plus 2007 gratis download
autodesk revit 2015 product key
corel paintshop pro x5 3d
microsoft powerpoint 2013 gezginler indir
microsoft outlook 2013 download portugues
adobe premiere pro cs4 1080p
how to get adobe dreamweaver cs3 for free
quarkxpress 10 trial download
nero 8 ultra edition letlts
eset smart security 5 password username
adobe premiere pro cs4 youtube
adobe captivate 3 help
corel windvd pro 11 ul.to
descargar microsoft office 2007 professional plus gratis en espaol
word 2010 for dummies review
adobe premiere pro cc trial version
pinnacle studio 15 hd ultimate collection.rar
coreldraw graphics suite x5 trke yama indir
aurora media workshop v3.4.28
boris red 5 manual pdf
adobe indesign cs5 classroom in a book cd files
how to install adobe flash professional cs5
microsoft office for mac home and student 2011 family pack free
download a free trial of adobe flash professional cs6
quanto costa adobe premiere pro cs3
microsoft visual studio 2008 professional license
microsoft office for mac home and student 2011 download
download microsoft office 2013 professional plus 2013 32 bit
adobe acrobat 9 pro extended english franais deutsch
corel videostudio pro x6 update
adobe premiere pro cs5 classroom in a book files
vmware workstation 8 windows 8 error
microsoft office access 2007 features
adobe illustrator cs5 for sale
adobe dreamweaver cs4 para windows 7
portable microsoft office professional edition 2003 sp3 espaol 1 link
corel videostudio pro x3 review download
adobe acrobat 3d 8
fl studio 8 xxl producer edition v8.0.0 final
microsoft office word 2007 gratis download deutsch
adobe after effects cs6 optical flares
microsoft windows 8.1 language pack x86 multi dvd iso
adobe captivate 5 tutorial youtube
adobe indesign cs5 plugin update
microsoft office 2007 professional letter template
descargar adobe illustrator cs4 gratis para mac
tutorial avanzado de sony vegas pro 11
boris red 5 language
adobe premiere pro cs4 supported video formats
microsoft streets and trips 2013 release date
microsoft streets trips 2013 download
microsoft office enterprise 2007 error 1719
microsoft project professional 2013 viewer
the adobe photoshop cs6 book for digital photographers 2012
parallels desktop 7 mac lion
ashampoo clipfinder hd letlts
microsoft office professional plus 2013 custom install
adobe indesign cs5 gratis en espaol
adobe photoshop elements 6 features
adobe after effects cs5 adobe tv
nero 11 platinum full 2012
adobe creative suite 5.5 design standard student teacher version pc
photoshop cs5 allinone for dummies 2010 malestrom.pdf
nik software dfine 2.0 chomikuj
microsoft office professional edition 2003 sp3 pl portable
adobe captivate 5.5 download windows
adobe premiere elements 9 upgrade
microsoft windows 8 download demo
vmware workstation 10 unlocker osx
microsoft word 2013 show ruler
kljuc za microsoft office standard 2007
quarkxpress 8 production tricks and experts tips
adobe creative suite 5.5 master collection cs5.5 mac
how to download adobe dreamweaver cs6 for free
adobe photoshop lightroom 2.0 download
adobe photoshop elements 10 tutorial for beginners
adobe pagemaker 7.0 for mac
adobe indesign cs6 64 bit dll
microsoft office for mac home and student 2011 pacchetto italiano 1 installazione

More Than Market Research - Gain The Information Advantage

Tom H. C. Anderson - Next Gen Market Research™ header image 6

Forget Big Data, Think Mid Data

March 7th, 2013 · 6 Comments

Stop Chasing the Big Data; Mid Data makes more sense
[Re-posted from OdinText.com Blog]

After attending the American Marketing Association’s first conference on Big Data this week, I’m even more convinced of what I already suspected from speaking to hundreds of Fortune 1000 marketers the last couple of years. Extremely few are working with anything approaching what would be called “Big Data” – And I believe they don’t need to – But many should start thinking about how to work with Mid Data!

BigDataMidDataSmallData

“Big Data”, “Big Data”, “Big Data”. It seems like everyone is talking about it, but I find extremely few researchers are actually doing it. Should they be?

If you’re reading this, chances are that you’re a social scientist or business analyst working in consumer insights or related area. I think it’s high time that we narrowed the definition of ‘Big Data’ a bit and introduced a new more meaningful and realistic term “MID DATA” to describe what is really the beginning of Big Data.

If we introduce this new term, it only makes sense that we refer to everything that isn’t Big or Mid data as Small Data (I hope no one gets offended).

Small Data

I’ve included a chart, and for simplicity will think of size here as number of records, or sample if you prefer.

‘Small Data’ can include anything from one individual interview in qualitative research to several thousand survey responses in longitudinal studies. At this level of size, quantitative and qualitative can technically be lumped together, as neither currently fit the generally agreed upon (and admittedly loose) definition of what is currently “Big Data”. You see, rather than a specific size, the current definition of Big Data varies depending on the capabilities of the organization in question. The general rule for what would be considered Big Data would be data which cannot be analyzed by commonly used software tools.

As you can imagine, this definition is an IT/hardware vendor’s dream, as it describes a situation where a firm does not have the resources to analyze (supposedly valuable) data without spending more on infrastructure, usually a lot more.

Mid Data

What then is Mid Data? At the beginning of Big Data, some of the same data sets we might call Small Data can quickly turn into Big Data. For instance, the 30,000-50,000 records from a customer satisfaction survey which can sometimes be analyzed in commonly available analytical software like IBM-SPSS without crashing. However, add text comments to this same data set and performance slows considerably. These same data sets will now often take too long to process or more typically crash.

If these same text comments are also coded as is the case in text mining, the additional variables added to this same dataset may increase significantly in size. This then is currently viewed as Big Data, where more powerful software will be needed. However I believe a more accurate description would be Mid Data, as it is really the beginning of Big Data, and there are many relatively affordable approaches to dealing with this size of data. But more about this in a bit…

Big Data

Now that we’ve taken a chunk out of Big Data and called it Mid Data, let’s redefine Big Data, or at least agree on where Mid Data ends and when ‘Really Big Data’ begins.

To understand the differences between Mid Data and Big Data we need to consider a few dimensions. Gartner analyst Doug Laney famously referred to Big Data as being 3-Dimensional; that is having increasing volume, variety, and velocity (now commonly referred to as the 3V model).

To understand the difference between Mid Data and Big Data though, only two variables need to be considered, namely Cost and Value. Cost (whether in time or dollars) and expected value are of course what make up ROI. This could also be referred to as the practicality of Big Data Analytics.

While we often know that some data is inherently more valuable than other data (100 customer complaints emailed to your office should be more relevant than a 1000 random tweets about your category), one thing is certain. Data that is not analyzed has absolutely no value.

As opposed to Mid Data, to the far right of Big Data or Really Big Data, is really the point beyond which an investment in analysis, due to cost (which includes risk of not finding insights worth more than the dollars invested in the Big Data) does not make sense. Somewhere after Mid Data, big data analytics will be impractical both theoretically, and for your firm in very real economic terms.

Mid Data on the other hand then can be viewed as the Sweet Spot of Big Data analysis. That which may be currently possible, worthwhile and within budget.

So What?

Mid Data is where many of us in market research have a great opportunity. It is where very real and attainable insight gains await.

Really Big Data, on the other hand, may be well past a point of diminishing returns.

On a recent business trip to Germany I had the pleasure of meeting a scientist working on a real Big Data project, the famous Large Hedron Collider project at CERN. Unlike the Large Hadron Collider, consumer goods firms will not fund the software and hardware needed to analyze this level of Big Data. Data magnitudes common at the Collider (output of 150 million sensors delivering data 40 million times per second) are not economically feasible but nor are they needed. In fact, scientists at CERN do not analyze this amount of Big Data. Instead, they filter out 99.999% of collisions focusing on just 100 of the “Collisions of Interest” per second.

The good news for us in business is that if we’re honest, customers really aren’t that difficult to understand. There are now many affordable and excellent Mid Data software available, for both data and text mining, that do not require the exabytes of data or massively parallel software running on thousands of servers. While magazines and conference presenters like to reference Amazon, Google and Facebook, even these somewhat rare examples sound more like IT sales science fiction and do not mention the sampling of data that occurs even at these companies.

As scientists at Cern have already discovered, it’s more important to properly analyze the fraction of the data that is important (“of interest”) than to process all the data.

At this point some of you may be wondering, well if Mid Data is more attractive than Big Data, then isn’t small data even better?

The difference of course is that as data increases in size we can not only be more confident in the results, but we can also find relationships and patterns that would not have surfaced in traditional small data. In marketing research this may mean the difference between discovering a new niche product opportunity or quickly countering a competitor’s move. In Pharma, it may mean discovering a link between a smaller population subgroup and certain high cancer risk, thus saving lives!

Mid Data could benefit from further definition and best practices. Ironically some C-Suite executives are currently asking their IT people to “connect and analyze all our data” (specifically the “varied” data in the 3-D model), and in the process they are attempting to create Really Big (often bigger than necessary) Data sets out of several Mid Data sets. This practice exemplifies the ROI problem I mentioned earlier. Chasing after a Big Data holy grail will not guarantee any significant advantage. Those of us who are skilled in the analysis of Small or Mid Data clearly understand that conducting the same analysis across varied data is typically fruitless.

It makes as much sense to compare apples to cows as accounting data to consumer respondent data. Comparing your customers in Japan to your customers in the US makes no sense for various reasons ranging from cultural differences to differences in very real tactical and operational options.

No, for most of us, Mid Data is where we need to be.

@TomHCAnderson

[Full Disclosure: Tom H. C. Anderson is Managing Partner of Anderson Analytics which develops and sells patent pending data mining and text analytics software platform OdinText]

[Post to Twitter] 

Tags: Anderson Analytics · Big Data · Datamining · Market Research · Marketing research · Mid Data · Odin Text · OdinText · Small Data · Text Analytics · ama · american marketing association · text mining

6 responses so far ↓

  • 1 Kelley Styring // Mar 7, 2013 at 4:21 pm

    On the money, Tom. And small data is going to help mid data believers understand and communicate the findings with business relevance, simplicity, synthesis and insight.

  • 2 David Rabjohns // Mar 7, 2013 at 5:35 pm

    Great article nicely done.

  • 3 Doug Wicks // Mar 7, 2013 at 8:14 pm

    Thanks for bringing a researcher’s perspective to Big Data. A number of points to add to the conversation. 1) variety – data of different types - video, images, tweets, sensor data - these challenge researchers as they go about analyzing Big Data; 2) arguably, there is a fourth v - veracity - Is this data accurate, trustworthy? Also a key factor for researchers working with Big Data. And of course the fundamental in working with ANY data, big, mid or small - are we asking the right questions, do we have the right data to answer them, and are we providing the insights our clients need when they think about and use Big Data to solve address issues.

  • 4 Tom H C Anderson // Mar 8, 2013 at 2:19 pm

    Thanks, I find analytics/methodology is left out of decision, and it’s all left to IT/Purchasing who try to get a solution that attempts to address everyone’s needs.

    I think this is fine if focused only on operational needs. But if you implement such a system for the purpose of insights, then there needs to be some serious thinking about what data makes sense to analyze together.

    If you can’t currently analyze one Mid Data source properly, perhaps it makes a lot more sense to deal with that before spending lots of time and $ to link them all up to one giant Real Big Database, only to find out it made absolutely no sense to analyze these sources together after all!?

  • 5 Christy Pogorelac // Mar 13, 2013 at 3:42 pm

    Thanks - I attended the conference and thought the panel discussion was extremely interesting.

  • 6 Georgette Asherman // Jul 10, 2014 at 7:20 am

    Tom, you are on the mark. And most importantly you point out the importance of sampling, even when there really is big, big data. I had people tell me that their 100,000 record data set is Big Data, when it is not. Most of our challenges are in Mid Data because many of our small sample approaches stop working.

Leave a Comment