Predictions in Market Research White Paper 2

7/28/2019 Predictions in Market Research White Paper 2

http://slidepdf.com/reader/full/predictions-in-market-research-white-paper-2 1/11



In this second note in the ‘prediction’ series, I analyze the aggregate andrespondent level associations of traditional loyalty measres with varios marketbehaviors. I rely heavily on the work of Keiningham et al (2007), Morwitz et al(2007), and Hofmeyr et al (2008). I also draw attention to the indirect inflencethat the Fishbein-Ajzen eqation (1975) seems to have had on the development

of what’s called the ‘weighted additive model’ of preference measrement in orindstry.

Headlines

Here’s what you’ll learn:

The aggregate correlation between cstomer satisfaction and varios measresof real behavior varies between R = 0.60 and R = 0.80+. Respondent levelcorrelations average abot R = 0.20.

The aggregate correlation between prchase intention and what people go on

to se is R = 0.52. Respondent level correlations average abot R = 0.27.The aggregate correlation reported by Reichheld between NPS and past/present firm growth rates is R = 0.83. He notes that respondent levelcorrelations are ‘less accrate’ than other loyalty measres.

The correlations reported by Reichheld (2003) are based on a selection of 18ot of 50+ firms in 3 ot of 12 indstries. One wonders why he doesn’t reportthe others.

There is a big difference between the aggregate and respondent level validity of the above measres. Becase of the importance of respondent level validity, classicloyalty measres like cstomer satisfaction, prchase intention, and Net Promoter

shold not be sed as dependent variables for profiling or driver analysis if bettermeasres are available.

•

•

•

•

Prediction in Marketing Research (2):

The Validity of Customer Satisfaction, Purchase Intention, and Net Promoter Score

Prediction in Marketing Research (2): The Validity of Cstomer Satisfaction,Prchase Intention, and Net Promoter Score 1



An Important Old Eqation

A good place to start is with an old eqation that seems to nderpin a lot of

marketing research thinking (See, for example, Ryan and Bonrfield (1980) andFishbein and Ajzen, 1975 – don’t be intimidated by the notation. Yo’ll find yonderstand it, so read on):

The eqation says that what a person does, i.e., behavior, will be consistent with(‘Σ’), the strength of their intention to do it. This, in trn, will be a fnction of their beliefs abot what will happen (‘ B

i’), their attitdes – whether positive or

negative – to those otcomes (‘ai’); and the overall importance of each of thoseotcomes (‘ω

0’) relative to the importance of social norms (‘ω

1’). Add the reslts

to get a total. The second term bilds social norms into the eqation, bt we don’tneed to be concerned with that here. The basic idea is simple: the stronger a person’s behavioral intention, the more likely it shold be that the behavior willfollow.

In marketing research, what Bettman et al (1998) refer to as the weighted additivemodel of the consmer choice process has a similar strctre (see below). Thelogic goes as follows: what a person ses or bys shold be consistent withthe strength of their intention to by it. We can measre the strength of their

intention simply by asking them, e.g., with a prchase intention qestion. Next, wetry to nderstand why a person has that intention by measring their beliefs abotthe prodcts, services, or brands they might se/by. This reqires the followingsteps:

Identify the attribtes yo believe may inflence a person’s intentions (indexedby i, i = 1, … , m);

Ask the person to rate the services/brands in terms of how they perform oneach attribte (a

i);

Establish the importance of each attribte to the person ( Bi);

Mltiply the ratings by the weights and add them p.

•

•

•

•




The reslt is a weighted additive model of prchase intention. In marketingresearch we often have a series of weighted additive measres on the right handside of the eqation, e.g., different toch point ratings or attribte associations.We se measred intention (the left hand side of the eqation) as the dependentvariable and then estimate the importance weights:

The model is ‘non-comparative’ becase it doesn’t measre what a personthinks abot competitors, and it’s ‘comptationally intensive’ becase it impliesa complex mental process, i.e. draw p a list of relevant characteristics; ratethe prodct, service, or brand on each characteristic; give each characteristican ‘importance’ weight; and mltiply the rating by the weight (in yor head).

Then add the nmbers p – in yor head. It’s worth noting that cognitiveneropsychologists don’t think the brain works like this.

In the classic model, marketing researchers ‘know’ the strength of intention (theleft hand side of the eqation) and the attribte ratings. Importance weights aresally derived sing prchase intention as the dependent variable.

How well does ‘prchase intention’ correlate with real behavior?

i). The aggregate correlation of ‘purchase intention’ with behaviour

A key aspect of the model is the left hand side of the eqation which links thestrength of a person’s declared intention to their actal behavior. In a wide-ranging literatre search covering social psychology, economics, statistics, andmarketing from 1940 to 2006, Morwitz et al (2007) fond some 40 articles thatlinked a srvey of prchase intention to real world otcomes. Twenty-five of thearticles reported correlations.

The average correlation in the academic literatre at an aggregate level is R = 0.52.This is the correlation between penetration or actal sales on the one hand anda smmary measre of prchase intention on the other. Varios smmaries weresed inclding weighted scales, mean or median vales, and top boxes. Scalesvaried from 5- to 21-points.

A nmber of factors inflenced the qality of the correlation. In general, prchaseintention correlates better with penetration than actal sales. It also correlatesbetter for existing rather than new prodcts, and for drables rather than fast-moving consmer goods. The natre of the smmary measre (i.e. mean, median,top box or weighted scale) made no difference. Nor did the nmber of scalepoints.




ii). Respondent level correlation of ‘purchase intention’ with behaviour

On the Synovate website nder ‘Research on Research’ yo will find Report 23.

It stdies the relationship between what people say they will do (i.e., by or notby a microwave in the next 18 months) and what they actally go on to do in thenext 18 months. As I pointed ot in the first note in this series, aggregate resltswere relatively good: 9% of the respondents said that they wold by a microwavein the next 18 months and 7% actally did. At the respondent level however, thereslts were not good: only 26.5% of the people who said that they wold by a microwave actally did.

The fact that prchase intention may be wrong abot who actally bys and whyhas long been known in market research. In a recent analysis of prchase intentionscales, Wright and MacRae (2007, p. 617) write:

“Purchase intention scales suffer from severe theoretical and empirical problems. It has long been known that the majority of purchasers are ‘non-

intenders’ and that actual compliance with stated intention is low.”

An important qestion therefore is: what degree of respondent level validity canone expect from prchase intention qestions? Morwitz et al don’t report onrespondent level correlations. Bt in a search of the marketing research literatrefrom 1980 to 2005, I fond 5 articles that reported respondent level correlations:

Sewall (1981) measred women’s intention to by clothing in a mall intercept.The correlation between prchase intention and actal prchases was R = 0.27.

LaBarbera and Mazrsky (1983) tested the relationship of ‘prchase intention’to ‘next brand boght’ in six prodct categories. The average correlation was R = 0.24.

Chandon, Morwitz, and Reinhardt (2005) measred prchase intention inthree prodct categories then observed behavior for six months. The averagecorrelation was R = 0.24.

Seiders, Voss, Grewal, and Godfrey (2005) tested the relationship of prchaseintention to high end retailer behavior in terms of ‘no of visits/share of wallet’. The correlation was R = 0.10.

Perkins-Mnn, Aksoy, Keiningham, and Estrin (2005) tested the relationship

of prchase intention to share of wallet for trcks and pharmaceticals. Theaverage correlation was R = 0.46.

According to these reslts, the average correlation between prchase intentionand varios actal behaviors at respondent level in 13 prodct categories is abotR = 0.27.

•

•

•

•

•




The American Cstomer Satisfaction Index (ACSI) and Net Promoter Score (NPS)

i). The aggregate correlation of the ACSI and NPS with behaviour

In an award winning paper, Keiningham et al (2007) compare the aggregatepredictive performance of nmeros loyalty measres sing two data sets. Themeasres they compare are: NPS, the ACSI, and varios smmary measres of ‘satisfaction’, ‘reprchase intention’, and ‘recommendation’. The two data setsare from the ACSI data-base (See Fornell et al, 1996) and its eqivalent in Norway.The comparison standard is based on claims for NPS made by Reichheld (2003,2006) and Satmetrix (2004).

Keiningham’s analysis of the Norwegian data is difficlt to smmarize. So I’ll

not attempt a smmary here (I’ll jst recommend that yo read it). Bt it’s worthqoting his conclsion (Keiningham et al, p 42):

“There is no real indication that average levels of any of the satisfaction/

loyalty metrics … are signifcantly correlated with the relative change in

revenue within the respective industry.”

Reichheld’s claim that “NPS is the single most reliable indicator of a company’sability to grow” isn’t spported and the NPS itself doesn’t correlate consistentlywith average growth rates.

Keiningham’s comparison between the NPS and ACSI is easier to smmarize thanhis analysis of the Norwegian data (see table 1). The reslts for both measresare good: R = 0.83 (NPS) and R = 0.82 (ACSI), bt they’re aggregate correlations.What wold happen if we were able to look at these reslts at respondent level?

TABLE 1: CORRELATIONS OF TWO LOYALTY MEASuRES WITH FIRM GROWTH RATES




Neither Reichheld nor the ACSI report a respondent level analysis. BtKeiningham et al (2007, p 41) qote Reichheld as follows:

“Net Promoter yields slightly less accurate predictions for the behaviour of individual customers than models consisting of data from multiple survey

items…”

The specifics behind this essentially negative evalation of NPS metrics atrespondent level aren’t made clear. It might have been raw scores, or perhaps itwas the Net Promoter classification of people into segments: promoters, netrals,detractors. Whatever the case, the NPS approach yields reslts that aren’t as goodas cstomer satisfaction or prchase intention.

Trning to ‘cstomer satisfaction’, as with ‘prchase intention’, there are fewarticles that test its validity at the respondent level. Here is a list I fond in articles

pblished from 1983 to 2005 (See Hofmeyr et al, 2008):LaBarbera and Mazrsky (1983) tested the relationship between ‘satisfaction’and ‘next brand boght’ in six prodct categories. The average correlation wasR = 0.20.

Gstaffson, Johnson, and Roos (2005) test the relationship between cstomersatisfaction and chrn in the telecommnications market. The correlation of ‘satisfaction’→ ‘chrn’ was R = 0.13.

Seiders, Voss, Grewal, and Godfrey (2005) test the relationship betweencstomer satisfaction with high end retailers and ‘no of visits/share of wallet’.The correlation is R = 0.07.

Perkins-Mnn, Aksoy, Keiningham, and Estrin (2005) test the relationshipbetween cstomer satisfaction and share of wallet for trcks andpharmaceticals. The average correlation is R = 0.33.

Hofmeyr (2010) tests the correlation between cstomer satisfaction and shareof wallet for retailers in Italy in 2007 and again in 2008. The correlations are R = 0.27 (2007) and R = 0.22 (2008).

The average correlation between cstomer satisfaction and varios actalbehaviors at respondent level is R = 0.20. This compares with correlations thatvary between R = 0.60 to R = 0.80+ at aggregate level.

A frther note abot NPS ‘validations’

As Keiningham et al (2007) note, NPS has achieved widespread corporateacceptance. It’s therefore worth pasing to consider the natre of the ‘validations’that Reichheld sed as the basis for his original claim that NPS is “the one nmberyo need to grow.”

•

•

•

•

•




Here are some facts.

First, Reichheld’s validations look at the relationship between average NPS from

2001/2 and average corporate growth rates from 1999/2002. In other words,the correlations are partly backward looking . He is not showing that a good NPSpredicts ftre growth.

Second, as Keiningham points ot, Reichheld himself notes that NPS metricsdon’t correlate as well with individal behavior as other classical loyalty metrics.In this note I’ve shown that the correlation between classical loyalty metricsand individal behavior of varios kinds is qite poor – averaging R = 0.27for ‘prchase intention’, and abot R = 0.20 for ‘cstomer satisfaction’. We cantherefore conclde that NPS metrics are poor at respondent level.

Third, Reichheld says that he has the data for some 50 companies in twelve

indstries. Yet his HBR article only reports the correlations for 18 companies inthree indstries. Satmetrix (2004) say that their correlations are significant “inthe majority” of the indstries they looked at. That means they have indstries forwhich their reslts aren’t significant – the reporting appears to be selective.

Reicheld notes the “common sense” idea that high levels of recommendationshold lead to adoption and therefore growth. As reported by Keiningham, hethen notes that there are ‘imperfections’ in the analytics:

“All we did was quantify this common sense in a way that made sense to

business leaders... These practical leaders have little interest in advanced

statistical methods.”

However, the reason Reichheld shows any correlations at all is, I’m sre, becasehe realizes that bsiness leaders do care abot whether or not the measres theyse correspond with reality. One has to ask abot the indstries for which thereare no significant correlations, and abot the reslts that are nreported.

Why respondent level validity matters

Or indstry compensates in varios ways for the limitations of classic loyaltymetrics. For instance, we se norms and databases to help pt the reslts intocontext. At an aggregate level, the reslting models are good and achieve theirprpose. Bt we’ve become careless in failing to recognize the importance of the

difference between aggregate and respondent level models.




I can frther illstrate the problem by flipping arond the Fishbein/Ajzeneqation:

On the left we’re trying to establish a) What people believe abot prodcts,services, or brands; b) How they rate those prodcts, services, or brands; and c)How important each of the potential characteristics/motivators is.

In this hypothetical illstration of what we do, I’ve spposed that we got anotstanding driver model from or srvey – R = 0.90. The problem is: it doesn’tmatter how good the driver model is - the link with the real world is broken at therespondent level. Let’s se ‘prchase intention’ to illstrate this. We’ve seen thatthe aggregate correlation between prchase intention and real behavior is R =0.52. As in the Market Facts paper, the percent who se or by a prodct/service/brand correlates well with the percent who say they will se or by it.

Bt at the respondent level, the relationship breaks down: many people who saythey will se or by a prodct/service/brand, don’t. A similar nmber who saythey will not se or by it, do. As a reslt, the respondent level correlation is only

abot R = 0.27. The problem is: yo’ve arrived at conclsions abot why peopleintend to se or by it and who they are by looking at those who say they willse or by it. Bt most of them don’t. In the meantime, yo’ve also arrived atconclsions abot who will not se or by and why – bt many of those do. Yocannot have any confidence in yor driver analysis or profiling.

There is no problem with aggregate modeling as long as one keeps the model tothe aggregate level, bt as long as or measres identify the wrong people as loyal,we can’t be confident abot or answers to qestions abot ‘who’ or ‘why’.

Smmary

The relationship between what individal people say abot how satisfied theyare, or what they intend to do and what they actally do, is poor. We may asksatisfaction or prchase intention qestions and get weights from a driver analysis.We may then work ot how to increase people’s satisfaction or prchase intention.We may think that when we apply the reslting strategies, we’ll increase cstomerloyalty. Bt we need to recognize the extent of the slippage between these modelsand actal individal behavior. When we apply sch strategies, we need toappreciate that we may be plling marketing levers whose cogs are broken.Prediction in Marketing Research (2): The Validity of Cstomer Satisfaction,Prchase Intention, and Net Promoter Score 8



Bibliography

Anonymos Athor, Date unknown: ‘Measring bying intention: How valid is the estimate’,

Research on Research, 12, Market Facts Inc.Bettman, James R., Mary Frances Lce, John W. Payne: ‘Constrctive Consmer Choice Processes’, Jornal of Consmer Research, 25 (December), 187 - 217

Chandon, Pierre, Vicki G. Morwitz, and Werner Reinartz (2005): ‘Do intentions really predictbehavior? Self-generated validity effects in srvey research’, Jornal of Marketing, 69, 1 – 14

Fishbein, Marting and Icek Ajzen (1975): Belief, attitde, intention, and behavior: Anintrodction to theory and research, Addison-Wesley, Reading

Fornell, Claes, Egene W. Anderson, Jaesng Cha, and Barbara E. Bryant (1996): ‘The AmericanCstomer Satisfaction Index: Natre, prpose, and findings’, Jornal of Marketing, 60(October), 7 - 18

Hofmeyr, Jan, Victoria Goodall, Martin Bongers, and Pal Holtzman (2008): ‘A new measreof brand attitdinal eqity based on the Zipf distribtion’, International Jornal of MarketResearch, 50:2

Gstaffson, Anders, Michael D. Johnson, and Inger Roos (2005): ‘The effects of cstomersatisfaction, relationship commitment dimensions, and triggers on cstomer retention,’ Jornal of Marketing, 69:4 (October), 1 – 9.

Keiningham, Timothy L., Brce Cooil, Tor Wallin Andreassen, and Lerzan Aksoy (2007): ‘Alongitdinal examination of net promoter and firm revene growth,’ Jornal of Marketing, 71(Jly) 39 - 51

LaBarbera, Priscilla A. and David Mazrsky (1983): ‘A longitdinal assessment of consmersatisfaction/dissatisfaction: The dynamic aspect of the cognitive process’, Jornal of MarketingResearch, 20, 393 - 404

Morwitz, Vicki G., Joel H. Steckel, and Alok Gpta (2007): ‘When do prchase intentions predictsales’, International Jornal of Forecasting, 23, 347 – 364

Perkins-Mnn, Tiffany, Lerzan Aksoy, Timothy L. Keiningham, and Demitry Estrin (2005), “Actal

Prchase as a Proxy for Share of Wallet,” Jornal of Service Research, 7:3 (Febrary), 245 – 256Reichheld, Frederick F. (2003): ‘The one nmber yo need to grow,’ Harvard Bsiness Review, 81

(December), 46 – 54

…. (2006): The ultimate Qestion: Driving Good Profits and Tre Growth, Boston: HarvardBsiness School Press

Ryan, Michael J. and E. H. Bonfield (1980): ‘Fishbein’s Intention Model: A Test of External andPragmatic Validity’, Jornal of Marketing, 44 (Spring), 80 - 95

Satmetrix (2004): ‘The power behind a single nmber: Growing yor bsiness with Net Promoter,’Satmetrix Systems white paper available from www.satmetrix.com.

Seiders, Kathleen, Glenn B. Voss, Dhrv Grewal, and Andrea L. Godfrey (2005) ‘Do satisfiedcstomers by more? Examining moderating inflences in a retailing context’, Jornal of Marketing, 69, 26 - 43

Sewall, M. A. (1981): ‘Relative information contribtions of consmer prchase intentions andmanagement jdgments as explanators of sales’, Jornal of Marketing Research, 18. 249 – 253

Wright, Malcolm and Mrray MacRae (2007): ‘Bias and variability in prchase intention scales’, Jornal of the Academy of Marketing Science, 35, 617 – 624




Date post:	03-Apr-2018
Category:	Documents
Upload:	quentin3
View:	212 times
Download:	0 times

Predictions in Market Research White Paper 2

Documents