Tom H. C. Anderson - Next Gen Market Research

Does Convergence of Ad Media Require Convergence of Research?

For the past several years now my firm’s media industry clients have been asking us to help measure ad effectiveness of more complex campaigns. Many media sales today go beyond traditional TV ads and also frequently include a combination of various banner ads across online networks, social media engagement tactics, apps or games, custom client web portals to name just a few. Thus measuring the campaign efficacy is also becoming more complex.

Last year I became aware of the Advertising Research Foundation’s Audience Measurement event series and look forward to the event this year as well.

I’ll be at AM 7.0 on June 11, and hope to see some fellow Next Gen Researchers there. I’m particularly interested in the best way to measure ad effectiveness of these hybrid media campaigns. Personally I think the key to measuring the convergence of traditional and digital advertising lies in the convergence of traditional and Next Gen market research techniques. While survey research remains a good way to measure pre-post lift etc. I believe new techniques such as tracking and text mining sentiment have become a critical component necessary to truly understand the efficacy of an entire campaign. Yet few if any traditional research firms have yet to develop expertise across these areas.

I’m hoping this topic will be discussed more in the future within the NGMR LI group. Please let me know if you’ll be at the event, I always look forward to learning what others are doing in this important area of consumer research.

@TomHCAnderson
@OdinText

[Note: If you're also planning to attend the event feel free to use the OdinText sponsor code (ODIN) for a discount]

Looking at Text Analytics by Area of Expertise
(Thoughts from Sentiment Analytics Symposium 2012)

Sentiment Analysis is a terribly difficult problem. The problem is in defining the problem - the input is the problem - but you can’t avoid it, IT IS the future, and I’m very optimistic about it!

Professor Bing Liu started off the Sentiment Analytics Symposium yesterday with the statement above and I couldn’t agree more. Subsequently he gave the pre-workshop audience a detailed 3.5 hour overview of the state of text analytics. It was not surprising to me that almost a quarter of the audience were young developers (MacBook Pro in hand), with the hopes to learn how to incorporate their own sentiment analysis engines into their business applications.

What was a surprise to me though, given that text analytics is so clearly “the future”, was that only three ‘traditional’ marketing research industry firms were represented at the conference (Toluna, Survey Analytics, and my firm Anderson Analytics‘ - OdinText).

I’ll be honest and tell you that I attend/speak at different conferences for different reasons. University events can be a fun way to give back to the next generation of researchers. Marketing Research industry events are an enjoyable and effective way to meet up and network with colleagues and potential clients. I attend Text Analytics events such as the Sentiment Analysis Symposium to keep an eye on the potential competition.

In a growing and competitive field like text analytics I don’t expect to learn too much about what other vendors are doing. In fact I would be very disappointed to learn that someone had gotten further than OdinText in our specific niche (market research). Most wise suppliers are careful what they choose to share, especially if they do not yet have sufficient patents in place. Still it’s always surprising to see what some are willing to divulge.

Of course one thing most text analytics vendors are willing to share are case studies. However getting client permission is often difficult even for those of us who work in the private sector (never mind those doing primarily military/defense work). That is why academics like Bing are such a valuable resource; they often do have relatively broad and deep experience and are more willing to share it.

Don’t get me wrong, there are certain non-critical yet important things suppliers will share with each other. Best practices regarding whether or not to use machine translation before analysis for instance was one of several interesting presentation shared earlier today. Conference Chair, Seth Grimes usually does a great job vetting the various speakers, which makes SAS12 one of the better conferences on text analytics.

So, what was my overall takeaway from this year’s event?
I think it’s becoming clearer and clearer to everyone that domain expertise, data source, and objective of research all benefit from some level of customization and expertise:

• Domain Expertise - Using the same approach to sentiment across brands or industries, while possible, certainly isn’t as good as customization. But customization can be expensive and time consuming
• Data Source Expertise - A marketing research open ended survey response is very direct, and so is a tweet believe it or not (because of its short concise 140 character limit). Blogs or news articles on the other hand can be very indirect and require a very different approach
• Objective Expertise - An analytical approach to understanding how to gain actionable consumer insights is very different than detecting fraud/terrorism or selecting the best resumes out of a bunch

So do you need to find a text analytics software firm that specializes only on your industry, on your specific data source of interest, and on your specific use case?

Some approaches will lend themselves to being useful across different similar data sources (such as survey data and twitter data I mentioned earlier), also many industries can benefit from the same or only slightly customized sentiment algorithms. However, be wary of any firm that claims to do be able to handle any kind of data for any industry, for any use case. No one tool is a panacea for all use cases. Luckily there are likely to be many smart folks who have been working on your specific use case for some time.

So while I was surprised that there weren’t any Honomichl Top 5 firms represented at this year’s event, I can’t say I’m not happy about it. There’s enough competition in the field, and I’m more than happy to license OdinText to these firms

@TomHCAnderson
@OdinText

Today’s Most Popular and Least Understood Research Tool Explained…Sort Of

Those of you interested in the Next Gen Market Research area of Text Analytics may find this series on ‘Text Analytics for (Very Smart) Dummies’ of interest. IIR’s Marc Dresner, ahead of their Tech Driven Market Research Event has posed the question “what really is text analytics?’ and reached out to get three different perspectives which he is posting on the TMRE blog over the next three days.

I was honored two weeks ago to take part in the first interview in the series (a supplier’s perspective), link via image below.

The second perspective last week came from IBM, a company whose text analytics software I’m quite familiar with (Anderson Analytics had an early partnership with IBM PASW). This is a rather interesting use case as they are leveraging text analytics to understand their own employees communication around IBM’s products and services.

The series was concluded today with another interesting (client side) use case that I’m more familiar with. Kodak Gallery has been using OdinText for two tracking studies for some time now and more recently even for larger ad-hoc survey research studies as well.

My thoughts in the area of whom should be using text analytics, in terms of where the ROI is greatest, continues to evolve as we work with clients with different needs (from survey comments and CRM data to call center logs and social media). I’m very interested in where others have found text analytics most useful, and equally interested in honest discussion around where it still seems to have a somewhat lower ROI such as focus groups. I’d welcome your thoughts and comments either here on the blog privately if you wish.

@TomHCAnderson
@OdinText

Test your text analytics knowledge to win a Text Analytics Summit pass

The 8th Annual Text Analytics Summit is just around the corner, June 12-13 in Boston. As usual I’ll be speaking at the summit and Anderson Analytics’ OdinText is a proud sponsor of the event. Last year we decided to give away a free event pass to the first person who answered a few questions about text analytics correctly here on the blog. It seemed the quiz was quite popular so I’ve decided to do it again this year.

Here goes:

Q1. Text Analytics is most closely associated with which of the following terms

• A. Text Mining
• B. Natural Language Processing (NLP)
• C. Both - They’re all basically used the same way more or less these days

Q2. Which of the following is not an approach to text analytics

• A. Machine Learning
• B. Statistical (Bayseian)
• C. Linguistic
• D. Neuro Linguistic Programming (NLP)
• E. Bag of words

Q3. Which of these is not a popular aggregator/reseller of unstructured social media data?

• A. Gnip
• B. Boardreader/Efisys Inc.
• C. Klout
• D. DataSift

Q4. Which of the following is true of word clouds

• A. They’re currently very popular
• B. They’re too simplistic to be of much value
• C. It’s hard to tell one word cloud from any other
• D. All of the above

Q.Bonus (Tie breaker). How many unique views (not submissions) do you think we got on last years text analytics pass give away quiz here on the blog? [closest answer wins]

[Same rules as last time. Please post your answers as a reply to this post. I'll wait to publish these answers until we close the contest and pick a winner next week.]

Good luck and hope to see you at the Summit!

@TomHCAnderson
@OdinText

Are you a marketing researcher half way between NYC and Boston?

Fairfield County CT and Westchester County NY is the gold coast of marketing research, yet there are few if any marketing research events in our area.

Last year we held the first NGMR Meetup as a way for the research community in our area to get together and network informally. Even with very little promotion we had a great turn out and so we’ve planned another meetup for Wednesday April 25th at 6:30PM.

This time we’ll be meeting at Tiernan’s Bar and Restaurant in Stamford CT.

Please come and enjoy casual networking with fellow market researchers.

Hors d’oeuvres and discounted cash bar courtesy of our sponsor for this second meetup, The NY MRA

Special thanks also to co-hosts A.J. Keirans of PureProfile and Ginger DeStefano of RTi Research.

Please RSVP on the NGMR MeetUp page so we can get an accurate count of attendees.

We look forward to seeing you there!
@TomHCAnderson
@OdinText

Q&A with Prof. Bing Liu ahead of the Sentiment Analysis Symposium and Pre Symposium Tutorial

The Sentiment Analysis Symposium in NYC is just a month away (May 8th), so I thought I’d check out who was teaching the pre conference sentiment analysis tutorial this year. For those of us working with text analytics and in the New York area, Seth Grimes Sentiment Symposium has definitely made our annual must attend list. However, what most seem to miss is the half day workshop the day before the event each year. I started attending this component last year when researchers from Amazon.com were teaching it and decided it was definitely well worth half a day in the city to get a more tactical POV on Sentiment from someone who might have a slightly different use case or experience.

This year, data mining expert Bing Liu, a Professor at University of Illinois at Chicago’s Computer Science Department, will be teaching the workshop. Some of his work on text analytics and detecting fraud in online ratings was recently published in the NY Times and as I noticed we were connected on LinkedIn from a previous text analytics event, I called him up for a quick chat to learn a bit more about his work and what I might expect to learn at his pre Symposium workshop. We had an interesting talk and subsequently I sent him a few questions as I thought others would be interested as well.

I plan on being at both the Symposium and Pre Workshop again this year. Anyone else who is interested in attending feel free to use my discount code (OdinText). Do let me know if you’ll be attending so we can meet up, it’s a relatively small and informal group.

Now on to the Q&A…

Tom: Bing, how did you get into text analytics, and sentiment analysis?

Bing: My earlier research interests were in the areas of data mining and machine learning. In about year 2000, I started to get interested in Web mining and machine learning using text data. These two topics led me to the text on the Web. Reviews naturally come to mind because they are focused and well organized, which is great for data mining. I also quickly realized that sentiment analysis was a perfect research problem on its own (I called it opinion mining then due to my data mining background). It had so many applications as every individual and organization needs opinions for decision making. There was also a whole range of challenging research problems that had not been addressed by the natural language processing or the linguistics communities. We started to work on it in 2003 and published our first paper in KDD-2004 (ACM SIGKDD International Conference on Knowledge Discovery and Data Mining). The paper basically defined the framework of feature or aspect-based sentiment analysis and opinion summarization, which is now widely used in the industry and in research.

Tom: False website reviews are an interesting application, and one that I’ve been keeping my eye on. I noticed the New York Times recently covered some of your work in this area. This type of text analytics research seems to be much more difficult than most people think. Can you tell us a bit about this problem from the text analytics perspective, and how it is different from simpler use cases like identifying spam email for instance?

Bing: Indeed, this is a very difficult problem. My group began to work on it in around 2006 or 2007 as we realized this was an important problem and would become more and more important. When we started to do it, we realized it was really hard. The main difficulty lies in the fact that it is very hard, if not impossible, to recognize fake reviews manually as it is fairly easy to craft a fake review and pose it as a genuine one. Email spam detection is a much easier problem because you will immediately recognize a spam mail when you see one. This means that spam and non-spam emails have clear differences, and that it is easy to produce training data for machine learning algorithms in order to produce predictive models and to evaluate the models.

However, for fake reviews, if one writes them very carefully, it is hard to recognize them just by reading the review text. In the extreme case, this is an impossible task logically. For example, one can write a genuine review for a good restaurant and post it as a fake review for a bad restaurant in order to promote the bad restaurant. There is no way to detect this fake review without considering information beyond the review text itself simply because one review cannot be both truthful and fake at the same time.

Tom: What do you see as some of the applications of this type of research?

Bing: Review hosting sites or any general social media sites all want their reviews and user comments to be trustworthy. They are thus interested in fake review detection algorithms. All text analytics systems that use reviews or any opinion data need to worry about this problem too. Social media is here to stay. Its content is also being used more and more in applications.

Something has to be done to ensure the integrity of this valuable source of information before it becomes full of fake opinions, lies and deceptive information. After all, there are strong motivations for businesses and individuals to post fake reviews for profit and fame. It is also easy and cheap to do so. Writing fake reviews has already become a very cheap way of marketing and product promotion.

Tom: Have you found there are certain approaches that work better than others?

Bing: It is still too early to tell. Researchers currently use both linguistic features and atypical behaviors of reviewers to detect fakes. I feel that algorithms that mine atypical behaviors of reviewers and reviews tend to produce more interpretable and trustworthy results. For example, if all 5-star reviews for a hotel were posted only by people from the surrounding area of the hotel, these reviews are clearly suspicious. This is a simple example. More sophisticated fake reviews need more involved modeling and algorithms to detect them.

Tom: It’s been my observation and experience that we as an industry are moving away from linguistic approach to text (sure, some of the basics are useful), but machine learning and statistical approaches seem more powerful. What are your thoughts on this?

Bing: For most tasks, machine learning and statistical approaches are indeed more effective than pure linguistic based approaches. Linguistic approaches are mostly based on heuristic rules and patterns (including grammar information). For those tasks that can be performed based on words, it is very hard for a linguistics based approach to beat a statistical machine learning algorithm simply because the signals used by a machine learning algorithm are far more numerous than the rules or patterns that a human person can design. Plus, machine learning algorithms optimize the performances. However, that being said, in many tasks, linguistics based signals and clues are used as features by machine learning algorithms.

Statistical approaches are not without their limits. Going forward, I believe that both linguistic knowledge and statistical modeling are important. We are working on integrating more linguistic knowledge into statistical modeling.

Tom: It seems to me a lot of folks get a little too caught up in differences between languages. My firm for instance has found it rather easy to add other European languages to our approach, and of course machine translation is always a possibility. What are your thoughts on this?

Yes, I agree. Although every language is different, different languages are still similar as they all consist of words and grammar. European languages have even more similarities due to their common roots. A learning algorithm can capture many types of grammar regularities from any language if there is a sufficient amount of training data. For those tasks that need only word or lexical information, the same algorithm can be used for any language with almost no modification because an algorithm treats words are symbols. In that sense, it does not matter what language it is.

Tom: What will you be covering during the tutorial at the sentiment symposium?

Bing: Sentiment analysis has been studied extensively for the past decade. A huge number of research papers have been published on it (probably more than 1000). It is impossible to cover them all. Therefore, I will try to cover the main threads of research that also contain aspects which can be of immediate use in practice.

In the tutorial, I will start with a short motivation and then go on to define the problem. This will provide an abstraction or statement of the problem, which will naturally introduce the key sub-problems. I will then discuss the current state-of-the-art approaches to solving these problems. Since this is a practical sentiment analysis tutorial, I will also describe how to build a practical sentiment analysis system based on my previous experience in building one. In the final part of the tutorial, I will introduce the problem of fake review detection.

A big thanks to Bing for our talk and the subsequent Q&A. Looking forward to meeting up at the Symposium.

@TomHCAnderson
@OdinText

[For those interested in more info about the sentiment tutorial a syllabus and outline is available here]

It’s that time again when the Census releases another file of raw data. Tomorrow the 1940 US Census raw data will be released, and for the first time it will all be easily available online for download at the natonal archives website here.

Below is a short video from the National Archives explaining the data release.

The questions are few, but the research geek in me is still very curious both from a general data level, but also from a personal one. I should be able to locate my father and grandparents within the 72 year old data!

These census releases are especially cherished among geneoloists. If you’re interested in finding your families census record you will need to locate their enumeration district here before finding them in the data tomorrow.

@TomHCAnderson
@OdinText

Random thoughts from this year’s ARF Re:think 2012

The Advertising Research Foundation’s annual Rethink conference just ended. As the biggest annual research event here in the north east I always enjoy catching up there with both new and old colleagues.

Buzzworthy Presentations

Two of Monday’s presentations seemed to catch a lot of attention (and offend some). One of these which attendees were still buzzing about yesterday was a presentation by Karen Nelson-Field of Ehrenberg-Bass Institute for Marketing Science. She challenged researchers on the importance and uniqueness of social media and asked whether the current investments really have been worthwhile.

Facebook Likes for instance certainly seem to have little correlation with engagement or increased sales, and even those among us who are more active on Facebook, even with the micro targeting possible, find it difficult to remember a compelling ad there. Few many of us return ever to a brand site.

Personally I think the problem lies not in the value of social media as a marketing tool, but more in how marketers are approaching it. More on this later.

The second source of buzz carried over from Monday was a panel entitled “Getting Ahead of Change” in which CEO’s of Nielsen, Millward Brown, GfK, Kantar, and Ipsos tried to dissuade the audience from the common belief that research innovation isn’t coming from them directly, but only from startups which later are bought up by the “big guys”.

While I’m obviously biased in this area having spent many years with some of the larger firms before starting Anderson Analytics (OdinText), most of those who attended seemed to continue to share my bias after the presentation. While the ‘big guys’ have traditionally had some advantage in scaling certain metrics, they certainly don’t seem too innovative and most of their acquisitions eventually just end up competing against each other ( It mainly serves to decrease margins and is more of a case of 1 + 1 = 1/2). I would also argue that with the rise of API’s and web data becoming equally available to all, this one benefit of scale will also become less of an advantage.

Digital Research and Tracking

I made a point to attend two consecutive Unilever case studies focused on digital research yesterday. The first was a collaboration between TNS’s Cymfony and Compete.com, and attempted to draw insights between what people say online and what they actually do. An interesting idea for sure, but I was rather disappointed that the data had been analyzed in isolation. The research would have been far more compelling had the data sets been linked, something that we’ve looked at in the recent past and I l know is already possible, privacy issues aide.

In the study TNS Cymfony had provided social media comment analysis, I’m assuming from the standard mix of primarily Twitter and Blog data. Compete had analyzed website traffic data. Unilever’s Bob Bowan commented that “engagement” (comments), “solutions” (sites where information was consumed), and “purchase” were occurring in completely separate online locations. The consolidation of which obviously represent a great opportunity for those who can do a better job creating credible and engaging content sites within their category. Unfortunately, as of today I rarely see this happening in practice.

At the GRIT party later that evening my friend Larry Friedman, CRO at TNS, said it well “Most are just focusing on the ‘Media’ component of ‘Social Media’, they are missing the ‘social’”. I do think brands can get there, but it will take some time to change the corporate mindset. Companies must be willing to yield more control to the customer for it to happen.

The second case study was presented by Pattii Wakeling of Unilever and John Burbank, CEO of Nielsen Online. I noticed Facebook was in attendance as well, and it’s nice to see that one social network is interested in at least the advertising related side of the market research industry (Google was also present).

Nielsen, through their relationship with Facebook, has been able to leverage users Facebook cookies allowing for a much broader level of tracking ever previously possible. This is exciting as it allows for research far more representative than traditional panel related methodologies.

By leveraging other options available today I believe it’s possible to get similar insights beyond facebook, but doing it via facebook is obviously an attractive option for Nielsen as the privacy concerns are then in facebook’s court.

Discussing the issue later with Mark Michelson of the newly formed Mobile Marketing Research Association (MMRA), we both agreed that the old research privacy standards that most of us have followed for years are becoming less and less relevant. If consumers know we are tracking them online, then whether or not our clients get user level survey information also becomes less important.

Scope of Social Listening

A lighter presentation I attended was one by Netbase in which they had looked for online comments using search topic criteria of comments containing “I like/need…” in social media, and had then compared men vs. women. In regard to food/snack mentions for instance they had found that the most popular comment by women was “chocolate” while men were most likely to mention “bacon”.

These social listening results were compared to short open end comments from a survey that seemed to find relatively similar answers. Both genders for instance were likely to say “money”, and women were more likely to say “a nice house”. These were extremely similar to data we’ve seen in our own annual research among college students where we conduct an icebreaking exercise to encourage fast top-of-mind unstructured responses.

Audience comments afterwards were rather interesting. One researcher commented “these seem to be such as basic level of needs. Is the discussion really all driven by brands online? There seems to be very little thinking… Did you just look at Twitter or also blogs with deeper thinking?”

I think this is a common issue and reflective of the problem inherent with casting a wide net in social media listening rather than more carefully scoping out a research problem and identifying specific sites for a deeper dive. The latter which I have always found more actionable for clients.

The problem lies in the fact that most social media listening data constitute RSS data from just a few large sources such as Twitter and Wordpress. If a client knows of a specific site of interest where an important customer group is active it usually requires custom scraping and text analysis.

Subsequent discussion on the topic focused on the value of visceral answers in both social media and survey research. I think everything has its place, but in my opinion the main lesson as usual is that the more specific we get in study design, whether survey or social media, the more actionable the insights will be.

@TomHCAnderson
@OdinText

What is your research type?

Greenbook’s 2012 GRIT report was released yesterday including this year’s list of the top 50 companies mentioned as being most innovative by researchers. I have to admit it was nice to see Anderson Analytics in the top 15 even though NGMR did not officially participate in the study. We dropped a bit from last year, possibly also because our focus is now more and more on OdinText, our patent pending text analytics platform.

That said, what interests me most about this research, and any research really, is less about who is #1 or #2 …, as that can be a bit of a numbers game and hard to gauge accurately depending on sample source etc. Instead what I find most interesting are relative differences within any given sample. The Quad Maps below showing what kind of researchers mentioned a specific company are an example of this kind of insight. I compared this year vs. last year to see what kind of movement, if any, there was.

It was a bit difficult at first as the X and Y axis were in different positions from last year, but quadrants are basically the same, and there are also 15 firms rather than 10 listed in this year. So in order to make visual year to year comparison I’ve just inverted the 2012 image below for quick comparison.

Interestingly, almost all firms in 2012 are still in the same quadrants as they were in 2011. I suppose this is to be expected. Firms can’t change their positioning too quickly, and arguably why would they want to.

There is only one exception, Ipsos has moved into the more “experimental” side of the chart. Not sure why this is, though I think they’ve been doing some relatively innovative things recently (they did receive an NGMR Award this year for their ‘WAR’ methodology, and were joint winners for their joint work with Mobile/GPS the year before).

Other than this insight I also found it curious that none of the new firms were in the “More Quant/Established-Traditional” Quadrant. All the new firms are in one of the other three quadrants. Can it be that is what customers want now, either more qualitative or more experimental research?

Curious to hear which of these four quadrants you think you as a researcher would fall into?

I’m guessing you can guess which one I’m in

@TomHCAnderson
@OdinText