Tom H. C. Anderson - Next Gen Market Research™

Twitter Data and War Games

It seems everywhere you turn there’s talk of Big Data especially unstructured Big Data. Most of the time what they’re really talking about though is “Social Media Monitoring”. It’s a sin to call social media monitoring a bubble if you work anywhere in the analytics space.

Social media monitoring isn’t questioned because everyone likes the idea of it being free, open and easy to analyze.

No one points out that it’s basically just (RSS) Twitter data, nor do they ever discuss how useful, if at all, it is in being able to solve any real problems.

If Social Media Analysis were a game I think it would probably be Tic-Tac-Toe:

Just wondering how long it will take to discover this?

@TomHCAnderson

[Full Disclosure: Tom H. C. Anderson is Managing Partner of Anderson Analytics which develops and sells patent pending data mining and text analytics software platform OdinText]

Q&A, SAS 2013, and PayPal combines SixSigma with Text Analytcis

Just posting today briefly to mention a few things. First, on Seth Grimes’ new blog, Breakthrough Analysis, he recently asked me a couple of questions about OdinText/Text Analytics and Big Data in general. The title ‘Think Mid Data, and Triangulate’ may be a little misleading as I no longer feel triangulation is as important as I did back in 2005. I think some text analytics software and definitely OdinText is reliable enough out of the box so that in most cases triangulation is no longer necessary.

Second, I attended Sentiment Analysis Symposium yesterday, and it’s still a great event and continues to evolve as the text analytics market and types of users continues to grow. My favorite presentation was by PayPal which had used relatively basic text analytics (basic i.e. good and reliable) combined with Six Sigma process to produce a powerful and compelling case study which ended up saving the company over $3 Million.

This is exactly the kind of client side work and analysis I’ve been saying we need more of in text analytics. Many of the tools are good enough, it’s the creativity and ‘Just Do It’ attitude we need more of. I received my Six Sigma Black Belt training about ten years ago, and while Six Sigma seems to have fallen out of favor at many companies for various reasons, it was very impressive to see it linked with Text Analytics.

@TomHCAnderson
@OdinText

[Full Disclosure: Tom H. C. Anderson is Managing Partner of Anderson Analytics which develops and sells patent pending data mining and text analytics software platform OdinText]

Next Gen Market Research Oldest and Most Active Online Networking Group for Analytics Professionals

Gregory Piatetsky-Shapiro, Editor of KDnuggets just conducted an analysis of all the top Analytics, Big Data, Data Mining, and Data Science groups on LinkedIn with at least 7,000 members. He found that even though membership is strict and we have a zero spam tolerance, Next Gen Market Research group was in fact the most active group (calculated by comments per week divided by total members).

It was also interesting to see that NGMR was in fact the first networking group among the bunch founded way back in September of 2007.

I have to admit I hadn’t expected NGMR to have more comments than any other group, in part because the group membership (even though it’s now at above 20,000 professionals) is actually rather strict, and there is a zero spam tolerance.

What I would love to see additionally though is some content analysis, whether quantitative or qualitative, going beyond simple topic extraction i.e. “Big Data”, “text analytics”, “visualization” etc.”) in order to measure true interaction and sharing among experts interested in the field. I’m sure NGMR would do quite well if this type of analysis was conducted as well, and think it’s probably the best way to measure value of any online bulletin board.

I can’t believe it’s been six years and I feel very lucky to have met so many talented research colleagues during this time.

@TomHCAnderson
@OdinText

PS. I think Gregory may have discovered a new analytical tool for discussion forums.

4 quadrants from upper left moving right and then down, what might you call them?

If I may… if Y axis is “Content” and X axis is “Involvement” then:
1. Spam (Higher content lower involvement)
2. Active (Higher Content and Higher Involvement)
3. Dead (Lower content and lower involvement)
4. High Quality (Lower content Higher Involvement)

Stop Apologizing for the Size of Your Data!

This morning I was pleasantly surprised to hear that I had been included in the list of the 100 Most Influential in Big Data by Big Data Republic. It was an honor to be part of such an interesting group. Several respected analytics colleagues were also on the list including Gregory Piatetsky, Vincent Granville and Seth Grimes whom I’ve known for quite some time, as well as Carla Gentry and Kirk Borne whom I’ve gotten to know more recently.

Questioning yourself

With all the talk about Big Data these days it’s getting to the point where, even if you’re working with big data, with all the media mentions and glorification around it, you might start wondering “hey is this really me and my data they’re talking about, it doesn’t sound like it?”, and “maybe I need to be working with even bigger data sets?!”. In fact, partly because of this feeling, just a few weeks ago I had the urge to “pinch/remind myself” of what Big Data really is, defining it here on the blog.

So, now I’m wondering how unique am I - how many others working with Big Data have felt the same way, that the way It’s often referenced and revered makes the term feel, well a bit foreign or aspirational rather than real? And if some of us who regularly work with Big Data feel this way, well how do analysts working with more traditional data sets feel about the term?

You’re probably not alone

Until rather recently I didn’t think of myself as a Big Data professional at all. Back in 2005 when I started Anderson Analytics in order to leverage text analytics in market research, our average data set was usually well below 100,000 records.

Today, while our average client using OdinText probably does have about 100,000 records, and we have a few with over a million, the fact is that we have many more clients with smaller data sets. Several things lave led me to realize that ‘Big Data’ usually isn’t nearly as big as we seem to think it is (and the reason I really prefer the term ‘Mid Data’).

Really big Big Data is still a rarity

Speaking to managers at companies interested in text mining five days a week for several years I’ve realized that in actuality there are relatively few companies out there with those really large data sets. Also, though many fortune 1,000 companies do have these larger data sets somewhere, most of their research directors and analysts actually work with much smaller data sets. The analytical picture is in fact a lot different than Hollywood spy-fi movies, writers at Business Week or AdAge, and a slew of “enterprise software” vendors would have us believe.

This was confirmed yet again last month at the American Marketing Associations first conference on Big Data. I had the chance to answer questions and speak to quite a large number of the other speakers as well as attendees. Many came up to me, almost apologetic at first, saying something like “well this isn’t really a big data question, because our data is a bit on the small side…”.

There’s no doubt that data is increasing in size, and I also believe it’s important to include text/unstructured data in your analysis which further serves to increase data size and complexity. However, don’t worry if your data isn’t over a million customers in size, you’re definitely not an outlier.

While it’s true that some analytics including text analytics doesn’t really make sense until you reach a certain minimum threshold, the worst thing you can do is to combine various smaller or ‘Mid Data’ sources believing that multiple data sets somehow will increase the value of your data exponentially. Before even considering that, always make sure there is value in the independent sources first or you are likely to be disappointed.

Most importantly, Small, Big or Mid Data, it’s not the size of your data that counts, it’s the quality of that data and your analytical prowess and hard work that count!

@TomHCAnderson

@OdinText

[Full Disclosure: Tom H. C. Anderson is Managing Partner of Anderson Analytics which develops and sells patent pending data mining and text analytics software platform OdinText]

Coke Says No Boost from Social Buzz – Should You Care?

Guess what, “Coca-Cola recently learned that Social Buzz doesn’t correlate with it’s short term sales”.

No Kidding – this shouldn’t really be a surprise to anyone, yet the value of both social media and social media monitoring is being debated as a result of this “news”.

Here’s how it works for those of you who don’t understand marketing, social media or analytics.

Social Media is a tactic - a tool. If you have a great advertising campaign wherever it is, on TV and Social Media, just on TV, just on Social Media, whatever… then someone is probably likely to mention it on Twitter or blogs no matter how inconsequential to life your product is.

Conversely, if your advertising campaign sucks they are not likely to mention it much. Well unless it really sucks then a few might say that.

But so what, when it comes to predicting sales, monitoring tweets is at best looking for a trailing indicator - not the cause. Surely your sales numbers are a better indicator of your advertising success?

Social media certainly can and should play a critical role in most marketing campaigns today.

However, please, please realize that “Social Media Listening”, constitutes listening to mainly spam from the 8% of our population who tweets and blogs (while technically far more representative no one can analyze Facebook pages as they remain private/accessible only to Facebook).

Researchers, let’s keep things in perspective shall we. Way before we start discussing which approach to text analytics is most powerful we have to first decide what data is worthy of analysis. 140 character tweets from 8% of the population mainly trying to sell their “expertise” is just about the poorest form of data out there. Garbage In – Garbage Out!

The fact that Twitter even scores as many mentions as it does for products like “Coca-Cola”, which most regular consumers would be unlikely to ever think about any given week, is that there are so many want to be social media marketing guru’s on Twitter and blogs trying to analyze others marketing campaigns – further proving what a peculiar sample blogs and twitter is.

If you actually want to predict something like sales, you will need to first have a long serious think about what data, whether unstructured or structured, makes most sense in your model!

@TomHCAnderson

@OdinText

[Full Disclosure: Tom H. C. Anderson is Managing Partner of Anderson Analytics which develops and sells patent pending data mining and text analytics software platform OdinText]

Are there multiple answers to every question?

There are several simple ‘order of operations’ problems (memes) circulating on Facebook currently. This past weekend I saw this one “6-1×0+2÷2=?”. Over 2,500 people had answered it, 624 had liked it and 243 had shared it. Interestingly I noticed at least half of people have forgotten their order of math operations and therefore answered the problem incorrectly.

From observing people over the years it seems no matter what question you pose, or what great thing you offer - even if obviously in their benefit, there will always be a group who disagrees or don’t feel it’s a good thing. I think perhaps it’s some sort of innate survival mechanism left over from ancient days so that we wouldn’t all run over and eat those same delicious red berries.

I was curious how this question would be answered in a more professional setting, and so I posted the same problem to the Next Gen Market Research Group (NGMR) which I moderate on LinkedIn. I understood I would be far less likely to get a “1″ and more likely to get a “7″ as an answer.

In fact everyone who answered in NGMR answered 7, though they did so carefully at first, some probably thinking it was a trick question. I guess market research professionals are either much better at math than others, and/or more likely, folks in a professional group on LinkedIn are more hesitant to speak their mind quickly or take a guess and be incorrect than they are on Facebook.

As I said at least half the answers on Facebook to these questions are incorrect. To this one question in particular at least half answered with a “1″ but there were also several “3.5″ and “5’s”.

Now in my mind, there’s nothing wrong with being wrong if afterwards you can admit your error and learn from it. What surprised me most were the subsequent quite angry arguments adamantly defending their incorrect answers.
To give you an idea, here are just a few comments of the more than 2,500:

does anyone really ever get the answer….. ?????

the answer is 1 that is if you solve from left to right.

I change my answer to 5 (I suck, I had to write it down instead of doing it in my head:)

(6-1)X(0+2)/2=5 actually it’s begining algebra = 5×2=10 divided by 2. I tought Algebra ay Tampa Catholic High School in Tampa 1973- trust me. I know what I’m talking about here

too many people try to make things look difficult so they feel smarter. lets just do the math the way it’s written. first we have the #6then we’re asked to -1 which is 5 then we’re ask to multiply it by 0 which then the answer is 0 then we’re asked to add 2 now the answer is 2 and finally we’re asked to divide by 2 which now gives us 1.

I got “1″ but I can see how one could also get “7″

The answer is clearly 1. I aced the math regents. If you don’t see that you don’t deserve a high school diploma!

Why does this silly Facebook post matter at all? Well it’s basic math - there’s just one answer, it’s not up for debate.

Sometimes I wonder if this debating what is true, and what is the correct point of view on the problem is what’s the main problem currently in US politics. Even within our industry I hear ‘methodological’ arguments even within the NGMR group where some feel story is more important than data. Analyzing 40 tweets or a focus group is just as important as findings from a n=1000 US Census Rep survey.

Not sure what the answer is, but I think I agree with Mark Twain who once said “It’s easier to fool people than to convince them that they have been fooled”

@TomHCAnderson

Text Analytics World Q&A

Those of you also interested in Text Analytics may want to check out this Q&A just posted by Tom Reamy of Text Analytics World. Tom asked me and Jeremy Bentley, CEO of SmartLogic, 5 questions about the future of text analytics:

http://www.textanalyticsworld.com/press-room/interviews

Also, while traditionally most Text Analytics events have been here on the East Coast, West Coast practitioners especially may be interested in Text Analytics World’s upcoming San Francisco event in just two weeks, April 17-19. I will again be taking part in the east coast event in Boston held in conjunction with Predictive Analytics World. Those interested in either of these events are free to use my event code [OdinText].

Happy Text Mining!

@TomHCAnderson
@OdinText

PS. Curious to hear others thoughts on the future of TA.

Just writing today to thank RLF for the kind words and thoughtful Ginny Valentine Award ceremony last week in New York. I was both honored and humbled to be recognized together with several great research colleagues. I’m pleased that so many fellow researchers have found the Next Gen Market Research Group as useful as I have. It continues to be an interesting and rewarding hobby for me and a great way to get to know others passionate about our profession.

I also want to congratulate all the other great researchers who received the award last week, a few whom I already know and respect, and the rest whom I hope to get to know better soon.

@TomHCAnderson

And what it says about our profession

There’s a question I hear quite often whether at industry conferences, on blogs, or in professional groups on LinkedIn. Researchers, both supplier and even client side, are asking each other for recommendations on free software. Yes, it seems it’s more often for “free” software, once in a while “low cost” and sometimes even “the best free software”, but hardly ever for “the best software”.

Whether for survey, data mining, visualization or text analytics software, it seems that the most important variable is cost or more precisely, that it has no cost whatsoever.

I have to say it’s made me wonder a bit about the health of our profession. If software is very low cost or free, it’s usually because it has relatively little value. There are some rare exceptions of course, SurveyMonkey for instance. I’ll have to admit, for about 60% of surveys, it’s probably sufficient. While my company no longer programs surveys, as we now focus exclusively on our text analytics software, when we did we never used SurveyMonkey — Why?

Well for two reasons. First, as a professional research firm, we needed software that would work in all cases, not just 60%. So even if conjoint, advanced rotations and logic weren’t necessary for a certain project, since we already had the more robust software, there was no reason to use SurveyMonkey. The second reason, and somewhat less significant I admit, as a professional research firm I felt it was unfair to charge clients professional prices while using less professional/free tools. [Note, I don't see any problems with using Survey monkey if you are a client side researcher or a non professional researcher doing an occasional simple survey.]

Now, as for other research related software, I’ve seen very little value in any of the “free” stuff. It’s usually more difficult to use and far less powerful. This means you will typically invest more time using it and get less results in return, a lose-lose situation, especially in a profession that bills mainly for their time.

What I really don’t understand is when a supplier side researcher asks me about free software for use on a project that a client is paying for. When I probe, they usually explain, “my client is on a tight budget”. I don’t say anything but I usually wonder… “how can they then afford to pay you to waste your time?”

Ironically, I think about other occupations including carpenters, electricians, etc. who charge less per hour for their time. They usually take the greatest pride in selecting only tools of the highest quality as they know it makes them more productive and reflects on their professionalism. I’ve NEVER heard a carpenter say, “hey, I love these hammers cause I can get them free!”
Time is valuable and what we do is valuable, let’s treat it that way.

Now, switching topics a bit, I’m in need of some extra analytical help. Can anyone recommend any great free market researchers?

@TomHCAnderson

[Full Disclosure: Tom H. C. Anderson is Managing Partner of Anderson Analytics which develops and sells patent pending data mining and text analytics software platform OdinText]