Can Social Media MeasureCustomer Satisfaction?
Elliot Bricker, Director, Product Management, NetBase MARCH 201 1
Reviewed by an independent team from the Walter A. Haas School of Business at the University of California at Berkeley. The team was led by Vito A. Sciaraffia – PhD candidate in business economics and strategy.
Can Social Media Measure Customer Satisfaction?
2
Executive SummaryExecutive SummaryExecutive SummaryExecutive Summary
Because customer satisfaction is a key predictor of financial performance, businesses have
invested a lot of money and resources in tracking the satisfaction of customers through
surveys and benchmarking. Social media moved into the mainstream in 2010; savvy
executives soon recognized that the millions of blog entries, micro-blogs, status updates, and
comments that consumers post every day could become a new source of customer
satisfaction data – one that is significantly faster and less expensive than traditional survey
methods. The question is, does sentiment expressed in social media—that is, whether online
posts are positive, negative, or neutral—correlate with established customer satisfaction
metrics?
To answer that question, NetBase compared the Net Sentiment Score in the NetBase Insight
Scorecard to published scores from the American Consumer Satisfaction Index (ACSI) and
found a high correlation (Pearson Product Moment Coefficient r=.773). In this white paper, we
discuss our analysis and also share research findings that will help you benchmark your online
sentiment scores with industry peers. We also discuss the roles that Passion Intensity and
Share of Buzz play as social metrics that measure distinct facets of the customer experience.
Can Social Media Measure Customer Satisfaction?
3
Table of Contents
Customer Satisfaction Matters – at the Bottom Line .....................................................................................4
Social Media: A New Source of Customer Satisfaction Data ......................................................................4
Does Online Sentiment Correlate With Customer Satisfaction? ...............................................................4
NetBase Insight Scorecards Track Three Key Metrics of Online Brand Equity ................................... 5
Measuring Sentiment and Passion With the Science of Language .......................................................... 5
NetBase’s Net Sentiment Score ............................................................................................................................... 7
Correlating Net Sentiment to ACSI ........................................................................................................................ 8
NetBase’s Passion Intensity Score ........................................................................................................................ 10
Is Passion Intensity Really Different From Net Sentiment and Buzz? .................................................... 11
Bringing it All Together: The Brand Passion Index ........................................................................................ 12
Conclusion ....................................................................................................................................................................... 13
Methodology .................................................................................................................................................................. 13
Appendix ......................................................................................................................................................................... 14
Related Reading ........................................................................................................................................................... 15
About the Author ......................................................................................................................................................... 15
Can Social Media Measure Customer Satisfaction?
4
Customer Satisfaction Matters – at the Bottom Line
Businesses know that customer satisfaction is a predictor of financial performance. Satisfied
customers generate more revenue. They are also more profitable because
it costs much less to market to existing customers than to acquire new
ones.
As a result, businesses have invested a lot of money in tracking the
satisfaction of their customers through a variety of customer satisfaction
(CSAT) and voice-of-customer (VoC) surveys. Email surveys are the most common data
source, followed by telephone incident follow-up surveys, and finally self-service follow-up
using VoC tools that “intercept” users during visits to a web site.
There are a variety of established measurement methodologies such as the Net Promoter
Score (NPS®) or SERPVAL/SERVQUAL and benchmarking services such as the American
Customer Satisfaction Index that serve to make CSAT survey data more meaningful and
actionable. However, all survey-based tracking methods have a common set of shortcomings:
• They rely on limited samples of customers
• Data collection takes time, slowing down responses to potential customer satisfaction
issues
• Because of the costs involved, most companies can only afford to track their own
brands
Social Media: A New Source of Customer Satisfaction Data
Forward-looking businesses have recognized that a new source of customer satisfaction has
emerged: social media. Every hour, consumers make more than 500,000 new blog and
micro-blog entries, status updates, and comments in social media. They are raving about
companies and brands that they like and spreading the word about the bad experiences they
have had. In fact, social media is where new impressions propagate first and fastest today.
Does Online Sentiment Correlate With Customer
Satisfaction?
Social media offers a fast, inexpensive way to measure customer satisfaction for both your
brand and your competitors’ brands. As executives evaluate it as a strategic data source,
many are asking the key question: How can social media help us to measure customer
satisfaction? In theory, an aggregate measure of online sentiment—that is, whether online
posts are positive, negative, or neutral—should correlate with established customer
satisfaction metrics.
In this white paper, we will explore that question. We will look at how the Net Sentiment
Score in the NetBase Insight Scorecard compares to scores calculated with popular CSAT
Can Social Media Measure Customer Satisfaction?
5
methodologies. Our findings demonstrate a strong correlation between the American
Consumer Satisfaction Index (ACSI) and NetBase Net Sentiment Scores derived from social
media opinions. We will also share research findings that will help you benchmark your CSAT
scores with industry peers. In addition, we will look at the roles that Passion Intensity and
Share of Buzz play in understanding customer satisfaction. Finally, we will discuss the role
that social media can play in helping executives to understand the underlying root causes of
customer satisfaction issues, so that they can make smarter business decisions faster.
NetBase Insight Scorecards Track Three Key Metrics of
Online Brand Equity
NetBase Insight Scorecards were designed to give
businesses a reliable way to measure, enhance, and
protect brand equity in social media. They deliver
up-to-date metrics on three key aspects of your
customers’ experience:
• Share of Buzz - How much people are talking about your brand
• Net Sentiment - How positively they perceive your brand
• Passion Intensity - How emotionally charged their feelings are
These three metrics, along with measurements of
key conversation drivers that you specify, are
displayed with 12 month historical trending and
benchmarked against your key competitors.
Because they do not require expensive consulting
fees or time-consuming data collection and
assembly, NetBase Insight Scorecards give you a
fast way to stay on top of your consumers’
experiences with your brand.
Measuring Sentiment and Passion With the Science of
Language
ConsumerBase, the data source accessed via Scorecard, is a social intelligence warehouse
containing a full year’s worth of social media commentary across more than 95 million
Can Social Media Measure Customer Satisfaction?
6
sources, from forums and blogs to Facebook and Twitter. We use the most advanced natural
language processing (NLP) engine to read, understand, and categorize every posting in
ConsumerBase according to the sentiments, emotions, and key ideas that your consumers
have expressed. Then, we aggregate these sentiments, emotions, and ideas into the metrics
that you see in your Scorecard: Share of Buzz, Net Sentiment, and Passion Intensity.
NetBase’s NLP engine represents a big leap in accurately analyzing content from the social media universe. Unlike tools that infer sentiment based on statistical keyword matching, NetBase understands sentence grammar at a deep level and delivers over 80 percent accuracy.
This approach does not involve counting words or analyzing text; NetBase reads sentences, evaluates grammatical sentence patterns, and organizes results to be fully searchable on a wide variety of attributes. We analyze social media content in two steps: parsing and normalization.
First, the NLP engine parses each sentence it captures from social media at a very deep level. This process is similar to the sentence diagramming that students do in a high school English class—it identifies and links the subjects, objects, verbs, adjectives, and other linguistic patterns in the sentence to extract deep and accurate understanding of what is being said. By analyzing this “connective tissue” within each sentence, our NLP engine can account for the complexities in language that make keyword-matching algorithms inaccurate.
Anaphora resolution is a key part of the parsing process that ensures that sentiment-rich associations are not missed.
Anaphora are grammatical substitutes that refer back to your brand name in sentences that may otherwise be missed in most social media analysis tools. Typically, a pronoun (e.g. it, her, him, their, they), anaphora refer back to another unit, as the use of "they're" and "they", and "them" refer to M&Ms in the sentences: M&Ms are sweet and delicious. I remember always hearing that they[M&Ms] melt in your mouth and not in your hands. We're going to buy more[M&Ms] today down at the store because we love them[M&Ms]. So we see that the anaphors obey what linguistics call binding conditions. Anaphora resolution requires semantic understanding that is even today undergoing active academic and business research in the field of natural language processing.
Can Social Media Measure Customer Satisfaction?
7
Next, our NLP engine normalizes all the parsed sentences to make them easy to aggregate into the metrics shown in our Scorecards. It takes sentences (what we call “sound bites”) and stores them, based on the type of insight they reveal, in a single, consistent format, regardless of the structure of the underlying sentences. Normalization is a fundamental part of NetBase’s unique value because it allows our solutions to expose not just positives and negatives but also deeper-level insights such as passion.
Working with many of the most sophisticated brands in the world, we have optimized the
NetBase NLP engine specifically for understanding social media and the Web. In addition to
standard English, our extensive lexicon includes a wide variety of “urban words” and phrases,
alternative spellings, and abbreviations common in social media, as well as common
misspellings. We are constantly incorporating new rules into this lexicon based on the work of
our internal linguistics experts, ongoing testing that we do using “crowdsourced” human
evaluators, and feedback from customers.
NetBase’s Net Sentiment Score
Automated sentiment analysis focuses on analyzing the content of online posts, determining
whether they are positive, negative, or neutral, and aggregating the sentiments detected into
a single generic score. The Net Sentiment Score computes a ratio of positive and negative
mentions of a topic. Our analysis is localized to individual sentences, our “sound bites;” we do
not view an entire post as a single sentiment.
The formula for Net Sentiment is:
A Net Sentiment of 100 means that all mentions of the topic are positive, and -100 means all
mentions of the topic are negative.
The average Net Sentiment Score across the thousands of brand names, companies, people,
and other product names mentioned in ConsumerBase is 32323232. This indicates that overall, the
chatter about brands and experiences is somewhat positive.
The NetBase Insight Scorecard charts the Net Sentiment Score in a time series for the brands
that you track. This view makes sentiment data more actionable because you can easily see
its directionality. You can also see a count of positivepositivepositivepositive sound bites (green area in chart on the
next page) and negativenegativenegativenegative sound bites (red area). The y-axis secondary scale (far right)
displays a scale for the volume of sentiment polar sound bites.
Can Social Media Measure Customer Satisfaction?
8
Below are some examples of a few industry- or category-specific Net Sentiment Scores. They
have been computed over the course of a single year and represent a comprehensive
selection of companies, their related products, and brands for each industry.
Additional analysis was conducted to prove that the distribution of Net Sentiment Scores
does not cluster for a specific industry. In other words, Net Sentiment Scores follow a typical
“bell curve” distribution where values are distributed fairly evenly around their mean and tail
off at the extremes of high and low scores. Please see the Appendix for more details.
Correlating Net Sentiment to ACSI
In order to look at how Net Sentiment compares
to established measures of customer
satisfaction, we computed Net Sentiment Scores
for 12 retail businesses (retail, wholesale, or
department stores) and compared them against
the academic methodology of the ACSI,
developed at the University of Michigan. These
scores are the aggregate of 12 months of data
that span February 1, 2010 until mid-February,
2011. The ACSI releases industry results
monthly, and the scores reflect the most recent
installment – February 2011.
The American Customer Satisfaction Index (ACSI) is a uniform, national, cross-industry measure of satisfaction. The distinguishing feature of the ACSI methodology is its patented cause-and-effect approach to customer satisfaction measurement. Three questions comprise the assessment of satisfaction:
• What is your overall satisfaction with our product or service?
• To what extent has our product or service met your expectations?
• How well did our product or service compare with the ideal?
Organizations normalize (weight) and average the three ratings to produce a score from 0 to 100.
Can Social Media Measure Customer Satisfaction?
9
The comparison between the NetBase Net Sentiment Score and the scores reported by the
ACSI organization for the same time period showed a high (r=.773) Pearson Product Moment
Coefficient –meaning there is a good correlation between the two scores.
Store
Net Sentiment Score
Target 78 51.3
Kohl's 81 70.1
J.C. Penney 80 65.3
Sam's Club 78 75.0
Lowe's 77 59.4
Walgreens 77 52.5
Best Buy 77 66.6
Macy's 76 66.2
Home Depot 75 58.0
Rite Aid 75 51.8
CVS 74 50.5
Wal-Mart 73 40.6
To assure that the correlations of Net Sentiment and other customer satisfaction
methodologies didn’t reflect the state of one industry, we also looked at a cross-industry
spectrum of profiles. This included companies from the automotive, airline, financial, Internet
retail, Internet travel, CPG, and grocery store verticals.1 The correlation approached a similar
coefficient to that measured for the single retail store industry. As can be seen in the chart
on the next page, that number was .714. This again implies that NetBase’s Net Sentiment
Score serves as a strong indicator of CSAT.
1 The companies included: Bank of America, BMW, Charles Schwab, Delta Airlines, eBay,E*TRADE, Expedia, Hershey, JP Morgan Chase, Kellogg's, Kia, Kraft Foods, Kroger, Lincoln Mercury (Ford), Nestlé, Netflix, Newegg Orbitz, Publix, Safeway, Southwest Airlines, TD Ameritrade, US Airways, Wells Fargo, and Whole Foods.
Can Social Media Measure Customer Satisfaction?
10
NetBase’s Passion Intensity Score
Marketers know that passion intensity matters. It drives word-of-mouth – both good and bad.
It generates brand loyalty and repurchase. But until now, companies did not have a way to
quantifyquantifyquantifyquantify how strongly consumers feel about particular brands.
The NetBase Insight Scorecards include a Passion Intensity ScorePassion Intensity ScorePassion Intensity ScorePassion Intensity Score, a number that ranges
from 0 to 100. This score complements the Net Sentiment Score by adding another
dimension of the customer experience: emotion. The formula is:
To calculate the number of mentions in each category, the NetBase NLP engine identifies
comments and posts that use a specific set of emotions and qualities. Measurements of the
"Love" emotion aggregate linguistic associations to feelings such as love, adore, fan, luv, love, adore, fan, luv, love, adore, fan, luv, love, adore, fan, luv,
thrilled thrilled thrilled thrilled and many more. Additionally, the "Love" emotion accounts for positive subjective
qualities such as brand associations to terms like incredible, fantastic, tremendous, incredible, fantastic, tremendous, incredible, fantastic, tremendous, incredible, fantastic, tremendous,
amazingamazingamazingamazing,,,, and so on.
Measurements of the "Hate" emotion cover expressions such as hate, despise, loathe, hate, despise, loathe, hate, despise, loathe, hate, despise, loathe,
detestdetestdetestdetest,,,, and disgustdisgustdisgustdisgust, , , , with the added complement of negative subjective qualities that include
horrific, appalling, shockihorrific, appalling, shockihorrific, appalling, shockihorrific, appalling, shocking, horrible, awful, terrible, suck,ng, horrible, awful, terrible, suck,ng, horrible, awful, terrible, suck,ng, horrible, awful, terrible, suck, stinkstinkstinkstink,,,, and so on.
The denominator of the formula combines the total number of these “emotion-laden” love
and hate mentions with a measure of other non-emotional but subjective qualities such as
best, worst, suck, rocks, good, best, worst, suck, rocks, good, best, worst, suck, rocks, good, best, worst, suck, rocks, good, difficult, difficult, difficult, difficult, or awesomeawesomeawesomeawesome.
Can Social Media Measure Customer Satisfaction?
11
As with Net Sentiment, we calculated an average Passion Intensity Score of 30303030 across
thousands of brands, products, people names, and issues. To give you an idea of the range of
passion intensity, here are some measurements that show scores at both ends of the
spectrum:
Lower PassioLower PassioLower PassioLower Passion Intensityn Intensityn Intensityn Intensity Higher Passion IntensityHigher Passion IntensityHigher Passion IntensityHigher Passion Intensity J.P. Morgan Chase 8.4 Windows XP 62 iShares 6.2 Volkswagen 63 Nexium 14 Grand Theft Auto 70 Marmite 81
For the 12 stores we looked at in the ACSI comparison, we calculated the Passion Intensity
Scores seen below. The scores aggregate over 12 months of analysis for individual mentions
expressing emotions and subjective-quality insights:
StoreStoreStoreStore Passion Passion Passion Passion IntensityIntensityIntensityIntensity
Target 26.2
Kohl's 41.9
J.C. Penney 15.0
Sam's Club 41.3
Lowe's 31.8
Walgreens 37.9
Best Buy 90.0
Macy's 43.6
Home Depot 30.9
Rite Aid 46.0
CVS 34.4
Wal-Mart 35.6
Is Passion Intensity Really Different From Net Sentiment
and Buzz?
NetBase believes that there is no single score that can capture all facets of online brand
equity. That’s why our Scorecards contain three metrics—Share of Buzz, Net Sentiment, and
Passion Intensity—with in-line comparisons to competitors and historical values.
Our research has shown that each of the metrics we calculate and display differs greatly from
the metric of sentiment. For example, the Passion intensity metric is very low in correlation
(r=0.10048 Pearson Product Moment Coefficient) to the Net Sentiment Score.
Can Social Media Measure Customer Satisfaction?
12
Affective discussion in social media is not always an associative measure of sentiment. For example, where sentiment can be very positive, ardor and fervency can be said to be rather blasé.
The low levels of correlation between our Net Sentiment and Passion Intensity Scores attests
to the fact that each metric stands on its own measuring an aspect of your brand’s consumer
satisfaction, with its own distinctive implication in your analysis.
Similarly, Buzz Volume—the number of occurrences of sound bites for the topic of interest—is
a facet of the customer experience that is distinct from Net Sentiment. Consumers may be
very chatty or relatively silent about a brand, but the data still proves that no correlation
exists in the volume of chatter to their overall expressions of polarity in sentiment. A
correlation analysis showed that the direction and degree (closeness) of linear relations
between buzz and sentiment was quite low at 0.01159.
Bringing it All Together: The Brand Passion Index
In order to have one crisp
visualization of the three metrics
of Buzz, Passion Intensity, and
Sentiment, NetBase provides a
Brand Passion Index.
The Brand Passion Index is an
intuitive visualization that lets
you quickly analyze consumer
sentiment, buzz, and passion
intensity across multiple brands
and view historical change in a
single chart. The map has four
quadrants: Like, Dislike, Hate,
and Love. You and your
competitors are placed on one
of these quadrants based on the
Can Social Media Measure Customer Satisfaction?
13
Passion Intensity that consumers express (Like versus Love) and their Net Sentiments (Like
versus Dislike). The size of the competitor’s bubble indicates the volume of posts for that
competitor.
Conclusion
Social media is the next strategic source of consumer insights – and of competitive
advantage. In this white paper, we have shown that consumer satisfaction expressed in social
media is both important and measurable. We have looked at how NetBase’s Net Sentiment
Score correlates with established measures of customer satisfaction (CSAT). We have also
examined Passion Intensity and Share of Buzz as distinct facets of a consumer’s experience
with a brand.
Social media insight and analysis can play an important role in the next wave of customer-
centric businesses. The businesses that will get ahead of the competition are the ones that are
starting down this path today.
Methodology
The data and statistical methodologies have been reviewed by an independent team from the
Walter A. Haas School of Business at the University of California at Berkeley. The team was
led by Vito A. Sciaraffia, PhD candidate in business economics and strategy. Mr. Sciaraffia
holds an MS in business administration from UC Berkeley, an MBA with concentration in
finance and statistics, an MA in finance from the University of Chile, and a BS in economics
and management from the Pontifical Catholic University of Chile. Additionally, he holds
several academic and professional certifications in statistics. If you would like additional
information about this white paper’s supporting data, please contact [email protected].
Can Social Media Measure Customer Satisfaction?
14
Appendix
In evaluating Net Sentiment Scores as indicators of customer satisfaction, we were concerned
about “clustering” in specific industries, where a majority of scores fall close to the mean
rather than following a typical “bell curve” distribution.
To test our hypothesis that the distribution of Net Sentiment Scores does not cluster for a
specific industry, we took over 125 women’s and men’s fashion store brands and computed
the normal distribution for each of their scores. We found that the variance, which describes
the spread of the distribution about the mean value, indicates that there is good scattering.
Additionally, there is enough mass (observations) to both sides of the mean. The lower and
upper tails of the distribution are as expected (gradually more unlikely to observe low scores
and high scores) with values nicely distributed between Net Sentiment scores of 28 to 80.
This says that the Net Sentiment Scores for each of the stores is not closely clustered around
the mean value for all brands. The 2D Histogram of the mass function show other “good”
distribution characteristics: A mean of 57.99 for Net Sentiment Score and a standard
deviation of 13.25 shows that one standard deviation on either side of the mean approaches
68%, two standard deviations - 95%, and three - 99%.
Some representative Net Sentiment Scores for both the low end and high end of the
spectrum of the 125 data points are:
Guess Jeans and Accessories 28.4
Rainbow Shops 29.0
Tous 29.9
Champs Sports 31.5
Armani A/X 80.0
GapMaternity 81.5
Heritage 1981 81.8
Tag Heuer Boutique 82.2
Boss Hugo Boss 82.8
Can Social Media Measure Customer Satisfaction?
15
Related Reading
“Net Promoter® Industry Report 2010,” Satmetrix, 2010.
“Social Media: Are You Listening to the Voice of the Customer?” A Joint Research Paper from
Verint Systems and TSIA, 2011.
“How Does NetBase Achieve the Best Accuracy for Understanding Consumers Online?”
NetBase, September 2010.
“The World’s Most Valuable Brands. Who’s Most Engaged?” Wetpaint and Altimeter Group,
July 2009.
“ACSI Score & Its Calculation,” Jeffrey Henning Mar. 11, 2009,
<http://blog.vovici.com/blog/bid/18135/ACSI-American-Customer-Satisfaction-Index-Score-
Its-Calculation>.
About the Author
Elliot Bricker participates in new feature design, pre- and post-sale
customer support, and other product success areas at NetBase. Elliot
began his career at Information Builders working in the areas of business
intelligence, data connectivity, expert systems software, and data
warehousing. Within numerous start-up companies, Elliot has focused on
applying machine learning as well as semantic search and categorization
of enterprise content to solve business problems. Most recently, his
interests have centered on opinion mining, specifically bringing together
social intelligence with business intelligence. He also has expertise in text
mining, analytics, data visualization, data warehousing, and optimization
algorithms. Elliot has contributed to numerous patent applications, both in the United States
and internationally. He holds BS and MS degrees in Computer Science.
Additional Acknowledgments
NetBase would also like to thank Linda Sonne-Harrison of Giant Stride Marketing Group for
her helpful review, feedback, and editing in regards to the content of this paper.
NetBase Social Media Insight & Analysis helps marketing teams make smarter business decisions faster. We
deliver tools and Scorecards that give market researchers and brand managers a reliable way to understand
online brand equity, analyze and compare consumer passion, and generate deep insights that answer their “why”
questions. Serving hundreds of corporate customers, our products were developed in partnership with five of the
top 10 CPG companies, including Coca-Cola and Kraft, and are used by four of the top 10 market research firms,
including J. D. Power & Associates. Based in the heart of Silicon Valley, NetBase is a privately held company.
For more information, visit: www.netbase.com @Net_Base NetBaseInc
©2011 NetBase Solutions, Inc.
NetBase Solutions, Inc.
2087 Landings Drive
Mountain View | CA 94043
P 650.810.2100
F 650.968.4872
www.netbase.com
About NetBase