Thoughts from Text Analytics Summit VII AND the time Anderson Analytics got a B+ from a Top Business School (2 for 1 post)
Last week Anderson Analytics and Clarabridge sponsored the 7th Annual Text Analytics Summit in Boston. The event was well attended; in fact it was very difficult to get a hotel room at the venue. I’ve been meaning to blog about some of the highlights of the event, including the Technology Insiders’ Panel I participated in and a great on point talk by Chris Jones of Zynga.
However, partly because of the work Anderson Analytics has done with Starwood, one presentation in particular, the one by Hilton Hotels on selecting a Text Analytics vendor has really had me thinking. Thinking back actually to potential client engagement three years ago…
Anderson Analytics gets a B+ from a Top Business School
A few years ago my firm was asked to bid on a project for a well known business school. Anderson Analytics has a hand full of University clients on both East and West Coast. The RFP looked very interesting, right up our alley in fact, so I allocated significant time from three of our team members to scope out and develop the proposal.
Though we had done work for this client before, management had changed, as had the process for vendors responding to RFP’s. We met with and spoke in detail to the point person for the selection committee. Two weeks later, after proposals from no less than seven vendors had been reviewed; we again met with the committee point person.
“I loved your proposal, and I think it was best overall, but as you know this was a committee decision and we evaluated seven vendors. The committee thought you did a great job too though, in fact your proposal came in second out of seven!”
I realized it was too late to do anything about the decision at this point and asked how exactly it had been made. The client committee point person explained “Well, you’re a market researcher, so I’m sure you’ll understand. We evaluated each firm on a number of key attributes on a five point scale, and then took the average. Your firm scored best in terms of uniqueness and overall strength of proposal, but we also had other variables such as how long each firm had been in business, whether or not the firm specialized in academia…’
I thanked the client and realized I had made a terrible mistake by not insisting to present our proposal in front of the entire committee. As a result, these days we hardly ever bid on university, government or proposals that are handled by a procurement department rather than research experts.
Hilton’s Approach to Selecting Text Analytics
Skip forward to the 7th Annual Text Analytics Summit…
Christine Hight, Director of Customer Research at Hilton Hotels presented their approach to selecting a text analytics solution. Similarly to Starwood Hotels and many other companies, Hilton has a large amount of structured and unstructured data coming into the organization. This includes 1.5 million guest satisfaction surveys, customer service call centers, email, social media etc. Hilton assumed that given their various unstructured (text) data sources, an enterprise solution would be most appropriate. Hilton then formed a steering committee with members from not only Marketing Research but several other departments, including IT, Application Development, etc. This steering committee then conducted a “literature review” in order to create a list of at least 10 vendors, one of whom would be selected for a six month Pilot Test.
The Steering Committee, then created a Task Force with interdisciplinary members including members from Legal and Finance and Accounting (17 members in all) in order to rate vendors across several weighted key variables so that the vendors could be rank ordered.
The process apparently took almost five months to complete, and Hilton has now started the six month Pilot with the vendor selected.
I hope Hilton will also share average satisfaction scores across all the departments after the pilot is complete…
The Future of Text Analytics
While the approach above would seem fair and make sense to some, it’s a bit hard for me to accept as best practices based on what we have learned about text analytics during the past six years.
There seems to be a desire for some companies to want to dump all their unstructured (text) data into one giant pot and boil the Ocean so to speak. We don’t seem to do that with structured data. Marketing research would not be likely to be asked for input into a new accounting software system (other than possibly as a user).
Without a doubt the text analytics industry has been changing quickly during the past six years. There are now many vendors entering the game with very specific use cases. Full disclosure, Anderson Analytics is one of these vendors, developing an application specifically for marketing research professionals (OdinText).
At the recent Sentiment Symposium in New York, consensus was clear that domain (industry) expertise is extremely important when designing and customizing text analytics software for clients.
It makes perfect sense that the next wave clearly involves use case expertise as well. While certain software like Radian6 might work very well for Public Relations professionals, their needs are very different from what customer service, marketing research, advertising, BI, IT, etc. need.
Theoretically, the benefit to an enterprise system of course would be that the problem with data silos might be solved. In practice, however, it seems clear to me that the enterprise approach is a bit naive. The use cases are really too different for any one software to meet expectations of the various department experts involved.
Luckily in the near future, thanks to wider availability of many more customized text analytics solutions, I don’t think anyone will need to hope for their ‘average’ needs to be met.
@TomHCAnderson
![[Post to Twitter]](../../../../wp-content/plugins/tweet-this/icons/tt-twitter-big1.png){kind=icon}
8 responses so far ↓
1 Seth Grmes // May 24, 2011 at 3:10 pm
Tom, frankly, I share your skepticism about the Hilton evaluation approach. Evaluation best practices say that an organization should do an initial, focused project that can deliver results quickly. Get some (successful!) experience under your belt and build out from there.
I’ll also relate that when the Hilton speaker mentioned looking at 17 vendors and coming up with a short-list of 4, I said to myself, I bet the 4 are among 5 vendors I had in mind… and I was right. Best practices also call for not spending 6 months on research and screening when expert guidance is available.
Seth, http://twitter.com/sethgrimes
2 Tom H C Anderson // May 24, 2011 at 3:18 pm
Thanks Seth.
Though I hope that your vendor suggestions would differ considerably depending on whether someone from marketing, marketing research, customer service, PR or someone from ’social media’ came to you?
That certainly would affect my suggestion set even now, and much more so in the very near future…
3 Stephen (Steve) Rappaport // May 24, 2011 at 3:41 pm
Thanks for these notes Tom. It’s sad that the selection procedures you describe have not changed for years: companies “evaluate” software based on comparative checklists, which seems deliberate, comprehensive and capable of reducing decision risk, but they don’t start where they should - with a clear statement or two of what they specifically want to accomplish with text analytics and why. There are a lot of aspirational objectives out there, which sound great but aren’t that helpful. As you point out, these beauty contests take time and resources on the buyer and seller sides. A practical, valuable and fast approach for buyers is to skip product demos 2-n, develop 2 or 3 sample problems, specify the type of output required, put a corpus of data together, ask vendors to run it through their system, learn what’s involved, and evaluate their effort. It’s remarkable how the proof (quality and utility of results) doesn’t always relate to the rankings.
As you say, the business has changed. It used to be that text analytics vendors wanted to make enterprise-wide sales. But the newer ones have realized that many projects are workgroup and departmental in scope and are developing targeted applications. Eventually, text analytics as a standalone application will go away and become embedded.
4 Shlomo Argamon // May 24, 2011 at 5:21 pm
Very important points, Tom. I’d just add that, from a purely technical perspective, text analytics systems need to be tuned for the specific kinds of texts and target data to be analyzed; they cannot be just purely general purpose, if they are to be accurate. This underlines the need for text analytics providers to have domain-specific expertise in the verticals that they target, or a well-defined process for tuning and customizing their systems. As Stephen notes, text analytics is and will move from being a general-purpose, enterprise-wide engine to being smaller-scope engines embedded in larger data management systems.
5 Rick Hobbs // May 24, 2011 at 7:15 pm
Hi Tom.
I share your pain in going through what seems like wrong headed evaluations. I am involved in writing proposals for committees and sometimes they are nothing more than cattle calls, where the lowest price automatically wins, or some sort of crazy price per point scenario where regardless of what is known about some or all of the suppliers, buyers are “forced” to select a supplier that will not do the best job. Equally troublesome is the “we already know who we want to select but we need proposals from two other vendors so it seems fair” proposal. My only comment about Hilton is, what was the value of the project. IF it was large, then it makes sense to take a while and get lots of people involved (6 MONTHS???) - but it drives me crazy when the amount of work necessary to prepare a propsal approaches the value of the contract if won. Cheers
6 Suresh // May 24, 2011 at 11:30 pm
Hi Tom,
I certainly agree that today the Text Analytics Market/Vendors are not mature to provide enterprise wide solution which cater to most of the business use cases. It is but natural that the customer would expect the vendors to build solution which attends to most of the use case at least pertaining to a particular industry. I hope the Text Analytics are listening to the VOCs and drafting thier product roadmap accordingly. It happened in the structured world, it is bound to happen in unstructured world…. exceptions noted.
7 Sid // May 25, 2011 at 7:15 am
Tom,
As usual, a provocative post. I do take issue with this point made above in your blog:
“There seems to be a desire for some companies to want to dump all their unstructured (text) data into one giant pot and boil the Ocean so to speak. We don’t seem to do that with structured data”
In fact we do seem to do that with structured data, all the time in fact. It’s called a data warehouse. The data warehousing marketplace developed and grew into a multi-billion dollare market precisely because while it’s fine to have financial reporting, and supply chain reporting, and sales reporting, and CRM operational reporting, in fact if you want to achieve demand/sales insights, or marketing/sales insights, or Promotion/Merchandising/SupplyChain/Financial Profitability views of your business, you MUST go to the data warehouse, there’s no other source for the integrated insight.
I can appreciate the market research view that you hold, as you have a deep and successful history of performance in that sector. But just as the world recognized the need for integrated structured data collection, transformation, and reporting in mid 90s and into the 2000s, believing that such a need does not exist for unstructured “voice of the customer” analytics, and equally importantly for fused structured/unstructured data solutions across the enterprise is short sighted at best, myopic at worst.
The markers of unstructured/structured data convergence in the marketplace are everywhere - certainly with many of our customers, and enterprise customer experience warehouses which integrate, score, tag, and analyze unstructured data from multiple touch points and source systems into holisitic solutions that meet needs across the enterprise (sales, marketing, insights, operations, etc) are being built today.
Back to the structured data argument — today there are still financial reporting, supply chain reporting, sales reporting solutionss that have been embedded by operational vendors. And there will certainly continue to be market research text analytics, social media text analytics, even CRM text analytics solutions developed, tuned, and embedded into those operational platforms. But make no mistake - there is, and will continue to be explosive growth in the market for enterprise text, and text/structured solutions and analytics. It’s happening today.
8 Tom H C Anderson // May 25, 2011 at 9:33 am
Thanks Sid, appreciate your input, and it was a real pleasure finally meeting you last week.
I’m aware of data warehousing business, and the frenzy to create dashboards at the upper management level. In fact, I’ve previously been involved in building some very large VOC reporting systems. While costly, they do have a place in helping to manage large business. However, because of their scope and scale they seem to do a better job catering to the needs of the general user than the expert. In the case at the conference, assuming research was looking for an ideal solution for them, one created specifically for researchers needs would surely be better.
My main point is that as text analytics continues to mature, buyers don’t need to pick the ‘black Ford’ anymore. More and more firms with a lot of domain experience will continue to enter this field. Structured data will certainly be incorporated with unstructured when possible, especially in my field we’ve been doing this in various degrees for years. That experience, as well as sector experience will come in handy and be less important than the mechanics such as ’sentiment’ that is so often discussed at text analytics industry conferences currently.
Leave a Comment