Tom H. C. Anderson - Next Gen Market Research

3 responses so far ↓

• 1 Beau Martin // Apr 23, 2009 at 4:02 am

  Despite both using experimental designs, conjoint and discrete choice models should not be confused. The difference really matters, as the types of variables they produce as outputs, as well as the quality of the outputs themselves, are extremely different. To answer the question, however, in my opinion, and if only for practical reasons, neither belongs in a segmentation.

  Let me start with the difference between discrete choice and conjoint models. The difference between discrete choice models and conjoint models is that discrete choice models present experimental replications of the market with the focus on making accurate predictions regarding the market, while conjoint models do not, using product profiles to estimate underlying utilities (or partworths) instead.

  As a result of their focus on market replication, discrete choice models can be counted on for accurate predictions. In a study I presented last year to the AMA’s Advanced Research Techniques ART Forum on pricing techniques the average error in my discrete choice model was 1.1%, with a correlation with actual market share of 0.99. The same cannot be said of conjoint models.

  Conjoint models - even Choice Based Conjoint models - simply do not seek to replicate the market in the same way. Instead conjoint models decompose the products in the market into attributes and levels, and those attributes and levels are then combined to create ‘product profiles’ that seem to respondents to be ‘random.’ The goal with conjoint models is to estimate the underlying utilities. Unfortunately, when those utilities, or more technically the partworths, are recombined, they do not produce accurate estimates of share. In the ART Forum paper on pricing techniques referenced earlier, the average error for conjoint models when predicting a real world market was 17% (the average share was 25%). Worse, the Nintendo Wii, the runaway best seller in the modeled market, was predicted to come in third out of four products in the market, and third out of three products for the current generation home gaming consoles.

  While a lot of people have written or presented on how to correct for poor conjoint estimates, usually referring to the process with its pseudo-scientific term ‘calibration,’ the fact is that these accuracy problems for conjoint models are rampant and well known, especially among those that know this field well.

  Bryan Orme, President of Sawtooth Software, might have said it best: “Divorcing oneself from the idea that conjoint simulations predict market shares is one of the most important steps to getting value from a conjoint analysis study and the resulting simulator. While external factors can be built into the simulation model to tune conjoint shares of preference to match market shares, we suggest avoiding this temptation if at all possible. No matter how carefully conjoint predictions are calibrated to the market, the researcher may one day be embarrassed by differences that remain.”

  So, in the end, my argument against including conjoint and discrete choice parameters in a segmentation is twofold.

  The first is that discrete choice parameters, while they are designed to lead to extremely accurate market predictions (and are from an extraordinarily powerful and useful technique) will often include, from a technical perspective, parameters that are not informed for a particular respondent, for instance parameters that relate to products the respondent never selected (and as a result never provided information for through their choices), and conjoint parameters cannot be trusted to predict choices (in fact, when predicting ‘holdouts’ they only have hit rates in the neighborhood of 65% - i.e. error rates of 35%). So, in either case, whether dealing with discrete choice or conjoint, the parameters are unreliable and should not be used in segmentation.

  My second argument for excluding conjoint and discrete choice parameters is that doing so makes it nearly impossible to create a scoring tool - especially if these parameters are important to the solution. Scoring tools are frequently important outputs from the segmentation process. Not having one, or having one that is inaccurate and makes poor predictions, would lead to a complaint I’ve seen detractors of segmentation write about frequently when criticizing segmentation, that in practice ‘we can’t find these people.’

  My own preference, in B2B especially, would be that you focus on readily identifiable nominal and process-level variables, especially those that might be referred to as ‘capability maturity level’ variables, and use those as the basis variables for the segmentation. Most everything you need is contained within them, including differences in current behavior and beliefs - at least the stated kind. The problem of course is that standard segmentation and clustering techniques do not handle nominal variables, especially when these variables are mixed with variables of other types, but that is a post for a different day.

• 2 Tom H C Anderson // Apr 25, 2009 at 2:50 pm

  Thanks for the detailed response Beau

  Multivariate analytics are half science half art. There are ways to leverage even nominal variables.

  Anyway I agree with you. Next time I get a client asking about a discrete choice segmentation, rather than getting into a debate I’m just going to ask them to figure out which of the two types of studies they want and call me back when they decide, LOL

• 3 Beau Martin // Sep 17, 2009 at 3:16 pm

  Hi Tom,

  I agree there are ways to leverage nominal variables in segmentation, even to mix them with scales of other types, but standard segmentation techniques don’t and knowing how to do this takes some special types of knowledge. Good luck with your clients!

  Beau

Leave a Comment