Tom H. C. Anderson - Next Gen Market Research™Cash until payday
Cash in account today
Payday lenders direct loans personal finance
Wonga fast loans
Same day payday loans derect lender
Payday lenders only no credit check
310 payday loans
Small direct payday lender
Payday loan bad credit no fees
80 payday loan direct lenders only
Quick loans st helens merseyside
New payday loans company
Cash bail bonds loan new hampshire
Same day free direct payday loans
Online loans wha are not payday loans
Payday loans for bad credit and ccjs
Payday dosh
Payday loans first time lender
Payday loans in hastings minnesota
How payday loans with no credit check work
Fast cash loans for people on benefits
15 minute payday loans
New paperless creditless payday loans
Redwallet payday loan
Quick canadian online payday loans
Payday loan lenders no brokers
Short term loans cape coral fl
Legit payday loans maryville tn
Paydayloans in hazleton pa
Cash in minutes online quick desion
Pay day loan for poor credit
I need a quick cash advance
Request 1000 small loan now
Good pay day loans on tv
Loans online no telecheck or check systems
Payday loans instant decision no admin fees
Fast loans in 15 mins
Payday loans in lincoln nebraska
Fast payday loans no credit checks
Cash x
100 percent payday loan
Direct lender online payday advance
Other loans similar to mobile loans
Payday loans to suit adverst credit
Joint cash advance
Can you get loans deposited into your account
Quick payday advance online
Trust pilot loans
Bad credit payday loans no fee
Online loans tennessee
Cash in 15 minutes
Medford ma payday loans
Payday loan go compare
Payday loans bad credit instant decision
Ninja loans
Pay day loans installment
Instant easy loans
Payday loans toronto bad credit
Loans for civil rights cases
Can you have 2 payday loans at once
Loans with direct deposit
Payday loan online with no credit check
Payday loan no broker fee
Qick loan online
Instant payday loans uk no brokers
Edmonton payday loans
How to get a cash loan in wigan same day
Quick and easy loans in 15 minutes
Quick loan until payday
Snow pay day
On line lones
Today loans fast
Help finding cash advances
New payday lenders not brokers
Instant pay day lans
Payday loans without fees
Guaranteed personal loan approval bad credit
Newest payday lenders
Pay loan direct broker
Get me a loan now
Future pay now loans
Loansforpaydayusa
Low cost online loans
Pay day loans estero florida
Quick quid loans in minutes
Fast loan cheetah
Loan installment loans
Loanb4payday
Same day benefits loan no credit check direct lender
Two or more loans
Instant online payday loans new companies
Online payday loan mobile
Loans no broker no fee payday
Loans in ct danbury
Wisconsin payday loan regulations
Paydayloansforbadcreditlendersonly
Payday loan in toronto
Mobileloan company in virginia
Long term loans on line
Need online loan fast
Cash transfer loans
Payday same day loan online
Pawn shop vs payday loans
Redleaf loans
Advance payday loans dickson
Short term month loans
Www simple loan com
Calgary payday loans
Payday loans network
I ned cash today help
Cash loans on my va benefits
Cash advance no faxing required
Cash wonga
Loan services in springfield mo
Self employed online loans
Online cash advance pr
Ezcashloan
Can i get a loan without any credit
On benefits need a payday loan no fees
Online payday loans california
Loan star financial
Need urgent cash
Borrow 100 bucks
Cash cow now payday loan
Get a loan in 30 minutes
Fast pay day loans uk
Lenders search 1000 com
Moneymart payday loan
Round rock payday loans
Loans for poor credit direct lenders chicago il
Processing payday loans
Cash advance money today
Bad credit loans payday
Easy pay day loan company
Quick payday loans lenders only
Payday loans online for 1000 or more
Loan poor credit history
Payday loans shut down
Quick bad credit payday loans no fees
Uncle buck payday loan
Pounds till payday loan
Payday loans gretna louisiana
Illinois installment loans fixed income d
Iggy loans
Loan companies palmer with no credit check
I keep getting rejected for payday loans
Mobile loans loans by text payday
Lend money quick
Calgary easy loans no credit check
Cash advance emporia va
Opportunity financial loans
Chase2hrloan
Legit money loan websites
Loans that accept bancorpsouth pay day loans
Fast cash loans in okc
Payday loa s
Fast loan uk
Payday direct lenders not brokers
Payday loans in 15 mins
Pay day king
Payday loans lenders not brokers no credit check
No credit check paydayloan lenders uk only
Langley payday loans
100 loan two installment
Pay day loans when you have no money in bank
New payday loan lenders ontario
Quick loan toronto
Instant money no lenders
Instant approval payday loans no credit
Edmonton payday loans online
Personal loan with out pay stub verification
Canadian online payday loans
100 loan online payday
Loan online decision money in 5 minutes
Easy loand no faxing same day deposit
Cash loan up 10000
Bc online payday loans
Payday loans cambridge ontario
Alberta payday loans
Cash advances cambridge
Loan for 16 year olds
Quick loans no fees
Quackquack cash
Online loans winnipeg
Grande prairie loans
Short term loans port alberni
Fast cash ontario
New pay day loans uk
Need cash know
Nofeepaydaylenders
Mynextpayday
Pay day loans with no fees
Money mart payday loan extension
Easiest direct pay day loan lenders
Payday loan bc
Payday loans no credit check like wonga
1000 on line pay day loans
Instant late night payday loans
Paydays loans like cash geniue
Payday loans kelowna
Money to loan on internet
1000 installment loan no credit check
Next payday
Pay day loans tv
Payday loans for bad credit and unemployed
Borrow money with no credit check
Gurantee payday loans lenders
Faxlwssmno credit checkmtext laon
Cash loans in 5 minutes
Payday loans bc
15 minute payday loans
Loans edmonton
Apm faxless
Pay day lenders list
Payday loans no hassle
Instant approval low pay day loans
Bad credit find a payday loan
Payday loan with out credit checks direct lender
Payday lenders not brokers for bad credit
Quick cash independant lender
Payday loan without aloan broker
Pay loans online in ga
Social assistance payday loan
Get a cash loan no credit check today
Compare quick cash loans
How to make cash fast
Cash loan calculator british
Fast loans no fees no credit check no guarantor
Direct payday loans no fees
Instant blacklisted loans no credit
No credit check car leasing in toronto
No credit check payday loans no brokers
Trust money loans
Instant cash loans no fees
Payday loasn
Payday loans of 1000 over 12 months
Payday loans bad credit uk direct lenders
Get a payday loan on benefits with bad credit
Text loan online only
Cash advance online no credit checks
I want 550 quick loan
Loans quick bad credit
Payday loans by phone no teletrack
Online loans for people on ontario odsp
How to get out of multiple payday loans
Loans no direct deposit
Bad credit payday loans instant decision
Payday loan lenders not brokers no credit check
Advance america money loans
Payday loans no credit check no upfront fees
Bad credit payday loans for people on benefits
New paday lenders
Direct lenders payday loans no brokers
Guaranteed direct payday loan
Quick loans high approval
Payday loans over 24 months
Pay day loans england
Online payday loans ontario niagara falls
Cash advance loans ca
Payday loans victoria bc
Instant cash loan for bad credit lenders
Fast loans done online
Bad credit quick on line loans
Quick loans that payout instantly no upfront fees
Payday loans that use faxless
Payday monkey
Payday loans with no middle man 100 approval
Emergency personal loan california low income
Payday loans online nova scotia
Getting quick loans
Loans deposited on a prepaid card
Payday loans yorkton sk
Sameday funding payday loan in delaware
Payday loan in pa bad credit
Payday loans guaranteed in 15 mins
Online not teletrack payday loans
Instant payday loans low credit
Same day loan without checks san diego ca
Arkansas payday loans
Payday highest apr
Loans overnight
Small personal loans online for bad credit
Get patday loan no credit check
Instant payday loans michigan
I need a loan from a direct lender
No teletrak or clarity payday loans
Cash need
1 lender payday loans
Unitedcash loans
Bad credit loan in wilmington de
Quick real loans
Payday in jacksonville beach fl
Instant loans no fees bad credit
Top ten payday loan companies
Secured online payday loans
Online cash loans with no lenders
Direct lender payday loans guaranteed
Splas cash online payday loans url
Payday loan only having a prepaid debit card
Who offers payday loans in arkansas

More Than Market Research - Gain The Information Advantage

Tom H. C. Anderson - Next Gen Market Research™ header image 6

Forget Big Data, Think Mid Data

March 7th, 2013 · 6 Comments

Stop Chasing the Big Data; Mid Data makes more sense
[Re-posted from OdinText.com Blog]

After attending the American Marketing Association’s first conference on Big Data this week, I’m even more convinced of what I already suspected from speaking to hundreds of Fortune 1000 marketers the last couple of years. Extremely few are working with anything approaching what would be called “Big Data” – And I believe they don’t need to – But many should start thinking about how to work with Mid Data!

BigDataMidDataSmallData

“Big Data”, “Big Data”, “Big Data”. It seems like everyone is talking about it, but I find extremely few researchers are actually doing it. Should they be?

If you’re reading this, chances are that you’re a social scientist or business analyst working in consumer insights or related area. I think it’s high time that we narrowed the definition of ‘Big Data’ a bit and introduced a new more meaningful and realistic term “MID DATA” to describe what is really the beginning of Big Data.

If we introduce this new term, it only makes sense that we refer to everything that isn’t Big or Mid data as Small Data (I hope no one gets offended).

Small Data

I’ve included a chart, and for simplicity will think of size here as number of records, or sample if you prefer.

‘Small Data’ can include anything from one individual interview in qualitative research to several thousand survey responses in longitudinal studies. At this level of size, quantitative and qualitative can technically be lumped together, as neither currently fit the generally agreed upon (and admittedly loose) definition of what is currently “Big Data”. You see, rather than a specific size, the current definition of Big Data varies depending on the capabilities of the organization in question. The general rule for what would be considered Big Data would be data which cannot be analyzed by commonly used software tools.

As you can imagine, this definition is an IT/hardware vendor’s dream, as it describes a situation where a firm does not have the resources to analyze (supposedly valuable) data without spending more on infrastructure, usually a lot more.

Mid Data

What then is Mid Data? At the beginning of Big Data, some of the same data sets we might call Small Data can quickly turn into Big Data. For instance, the 30,000-50,000 records from a customer satisfaction survey which can sometimes be analyzed in commonly available analytical software like IBM-SPSS without crashing. However, add text comments to this same data set and performance slows considerably. These same data sets will now often take too long to process or more typically crash.

If these same text comments are also coded as is the case in text mining, the additional variables added to this same dataset may increase significantly in size. This then is currently viewed as Big Data, where more powerful software will be needed. However I believe a more accurate description would be Mid Data, as it is really the beginning of Big Data, and there are many relatively affordable approaches to dealing with this size of data. But more about this in a bit…

Big Data

Now that we’ve taken a chunk out of Big Data and called it Mid Data, let’s redefine Big Data, or at least agree on where Mid Data ends and when ‘Really Big Data’ begins.

To understand the differences between Mid Data and Big Data we need to consider a few dimensions. Gartner analyst Doug Laney famously referred to Big Data as being 3-Dimensional; that is having increasing volume, variety, and velocity (now commonly referred to as the 3V model).

To understand the difference between Mid Data and Big Data though, only two variables need to be considered, namely Cost and Value. Cost (whether in time or dollars) and expected value are of course what make up ROI. This could also be referred to as the practicality of Big Data Analytics.

While we often know that some data is inherently more valuable than other data (100 customer complaints emailed to your office should be more relevant than a 1000 random tweets about your category), one thing is certain. Data that is not analyzed has absolutely no value.

As opposed to Mid Data, to the far right of Big Data or Really Big Data, is really the point beyond which an investment in analysis, due to cost (which includes risk of not finding insights worth more than the dollars invested in the Big Data) does not make sense. Somewhere after Mid Data, big data analytics will be impractical both theoretically, and for your firm in very real economic terms.

Mid Data on the other hand then can be viewed as the Sweet Spot of Big Data analysis. That which may be currently possible, worthwhile and within budget.

So What?

Mid Data is where many of us in market research have a great opportunity. It is where very real and attainable insight gains await.

Really Big Data, on the other hand, may be well past a point of diminishing returns.

On a recent business trip to Germany I had the pleasure of meeting a scientist working on a real Big Data project, the famous Large Hedron Collider project at CERN. Unlike the Large Hadron Collider, consumer goods firms will not fund the software and hardware needed to analyze this level of Big Data. Data magnitudes common at the Collider (output of 150 million sensors delivering data 40 million times per second) are not economically feasible but nor are they needed. In fact, scientists at CERN do not analyze this amount of Big Data. Instead, they filter out 99.999% of collisions focusing on just 100 of the “Collisions of Interest” per second.

The good news for us in business is that if we’re honest, customers really aren’t that difficult to understand. There are now many affordable and excellent Mid Data software available, for both data and text mining, that do not require the exabytes of data or massively parallel software running on thousands of servers. While magazines and conference presenters like to reference Amazon, Google and Facebook, even these somewhat rare examples sound more like IT sales science fiction and do not mention the sampling of data that occurs even at these companies.

As scientists at Cern have already discovered, it’s more important to properly analyze the fraction of the data that is important (“of interest”) than to process all the data.

At this point some of you may be wondering, well if Mid Data is more attractive than Big Data, then isn’t small data even better?

The difference of course is that as data increases in size we can not only be more confident in the results, but we can also find relationships and patterns that would not have surfaced in traditional small data. In marketing research this may mean the difference between discovering a new niche product opportunity or quickly countering a competitor’s move. In Pharma, it may mean discovering a link between a smaller population subgroup and certain high cancer risk, thus saving lives!

Mid Data could benefit from further definition and best practices. Ironically some C-Suite executives are currently asking their IT people to “connect and analyze all our data” (specifically the “varied” data in the 3-D model), and in the process they are attempting to create Really Big (often bigger than necessary) Data sets out of several Mid Data sets. This practice exemplifies the ROI problem I mentioned earlier. Chasing after a Big Data holy grail will not guarantee any significant advantage. Those of us who are skilled in the analysis of Small or Mid Data clearly understand that conducting the same analysis across varied data is typically fruitless.

It makes as much sense to compare apples to cows as accounting data to consumer respondent data. Comparing your customers in Japan to your customers in the US makes no sense for various reasons ranging from cultural differences to differences in very real tactical and operational options.

No, for most of us, Mid Data is where we need to be.

@TomHCAnderson

[Full Disclosure: Tom H. C. Anderson is Managing Partner of Anderson Analytics which develops and sells patent pending data mining and text analytics software platform OdinText]

[Post to Twitter] 

Tags: Anderson Analytics · Big Data · Datamining · Market Research · Marketing research · Mid Data · Odin Text · OdinText · Small Data · Text Analytics · ama · american marketing association · text mining

6 responses so far ↓

  • 1 Kelley Styring // Mar 7, 2013 at 4:21 pm

    On the money, Tom. And small data is going to help mid data believers understand and communicate the findings with business relevance, simplicity, synthesis and insight.

  • 2 David Rabjohns // Mar 7, 2013 at 5:35 pm

    Great article nicely done.

  • 3 Doug Wicks // Mar 7, 2013 at 8:14 pm

    Thanks for bringing a researcher’s perspective to Big Data. A number of points to add to the conversation. 1) variety – data of different types - video, images, tweets, sensor data - these challenge researchers as they go about analyzing Big Data; 2) arguably, there is a fourth v - veracity - Is this data accurate, trustworthy? Also a key factor for researchers working with Big Data. And of course the fundamental in working with ANY data, big, mid or small - are we asking the right questions, do we have the right data to answer them, and are we providing the insights our clients need when they think about and use Big Data to solve address issues.

  • 4 Tom H C Anderson // Mar 8, 2013 at 2:19 pm

    Thanks, I find analytics/methodology is left out of decision, and it’s all left to IT/Purchasing who try to get a solution that attempts to address everyone’s needs.

    I think this is fine if focused only on operational needs. But if you implement such a system for the purpose of insights, then there needs to be some serious thinking about what data makes sense to analyze together.

    If you can’t currently analyze one Mid Data source properly, perhaps it makes a lot more sense to deal with that before spending lots of time and $ to link them all up to one giant Real Big Database, only to find out it made absolutely no sense to analyze these sources together after all!?

  • 5 Christy Pogorelac // Mar 13, 2013 at 3:42 pm

    Thanks - I attended the conference and thought the panel discussion was extremely interesting.

  • 6 Georgette Asherman // Jul 10, 2014 at 7:20 am

    Tom, you are on the mark. And most importantly you point out the importance of sampling, even when there really is big, big data. I had people tell me that their 100,000 record data set is Big Data, when it is not. Most of our challenges are in Mid Data because many of our small sample approaches stop working.

Leave a Comment