The Non-Dummies Guide for Selecting Text Analytics

June 26th, 2014 · No Comments

[Reposted from OdinText Text Analytics Blog]

Selecting a Text Analytics (or any other) Partner

Text Analytics is a Process, Not and End!

What would you say should be the goal of good text analytics software?

Based on the questions we get from clients investigating text analytics solutions there seems to be no small amount of confusion. The fault isn’t theirs, it’s the fault of the early text analytics and social media monitoring vendors who overpromised and under delivered.

Rather than explaining to clients what kind of analysis and insights they should rightfully expect they choose instead to hide the fact that they know very little themselves about how text analytics can and should actually be applied, instead most text analytics sales staff preferred to talk theoretically using as many technical buzzwords like “natural language processing” as possible.

Here are questions you can safely set aside when investigating the right text analytics solution. They have next to no meaning whatsoever in terms of efficacy for your use case:

-How do you handle xyz stemming, semantic ABC, Ontologies and ______?
[Insert other favorite buzz word you’ve heard but don’t really understand]

-What does the output look like, do you have a pretty dashboard?
[If you buy text analytics software for pie charts and word clouds you’ll be in trouble. Dashboards, even if you find they make sense need serious customization]

-Do you have a cool black sci-fi looking background with neon colored maps?
[If you plan to put a bunch of monitors up and pretend you or on the bridge of starship enterprise I suppose this may make sense?!?!]

Instead, these kinds of questions are what you should be asking:

-Tell me about a client with the same kind of data that I have. How have they benefited from the tool?
(They better be darn specific)

-Show me how it works with my own data!?
(It’s easy to give a demo of poorly working software with canned data. Always make them use your data and never give them more than a day or two max to set it up)

Even better Text Analytics tools are becoming easier to use, and I admit, keeping OdinText intuitive as we add more features is challenging. However, one of the biggest single misconceptions about text analytics software is that they somehow have this magical “artificial intelligence” power. Some sort of power to discern everything and automatically write the report for you. I’m really not exaggerating.

Text analytics is not an end, it is a process. Find a vendor who understands this and whose software is not black box. Here simple is better. If how the software does its coding is hidden in a black box, and the sales person throws buzz words at you to make you feel safe/confused about the fact you have no idea about how the sausage is made, it’s not because they have valuable “linguistic” or “machine learning” rules (more buzz words) -those can only be developed after carefully studying your own data, it’s because their software doesn’t actually work too well and will require a lot of expensive and time consuming customization for unproven performance.

After choosing a text analytics software tool that is powerful and intuitive, a software that you can trust, then the fun begins. You or your analyst should be able to learn how to use the tool relatively quickly, but as with anything, you should expect to get better with experience.

Remember the early statistical software tools like SPSS and SAS. They worked very well on smaller data and you could trust that they actually did what you expected them to. However you still needed to know what clustering and factor analysis was, and why to look at a mean VS. a median. Just like these tools text analytics software also requires an analyst who can think about the data and how to get the most valuable insights for management.

Unfortunately, people who have never analyzed big data or conducted text analytics for real clients are building text analytics and “social listening” software. Find a vendor who understands your business. Their products will make you a data scientist. You’ll have to do a little more than press one button to understand the data, but since when has anything worthwhile been that easy?

To answer the question I posed earlier - what should be the goal of good text analytics software? – the answer depends on what field you’re in…

If you’re a marketer, then the main question you should be asking is how will this text analytics software help me sell more product to more customers less expensively?

@TomHCAnderson

[Full Disclosure: Tom H. C. Anderson is Managing Partner of Anderson Analytics, developers of patented Next Generation Text Analytics™ software platform OdinText. For more information and to inquire about software licensing visit ODINTEXT INFO REQUEST]

[Above also posted on the Next Gen Market Research blog]