Online Communities: Metrics and Reporting 2009

1

RESEARCH COORDINATION: FORUM ONE NETWORKS SAN FRANCISCO / WASHINGTON D.C.

Online Communities: Metrics and Reporting

Current Practice Research / August 2009

2Online Communities: Metrics and Reporting 2009

RESEARCH SPONSORS

HiveLive http://www.hivelive.com

HiveLive's on-demand software enables companies to rapidly deploy custom-fit enterprise communities built around people, their relationships, and their shared knowledge.

Lithium http://www.lithium.com

Lithium Technologies provides market-leading solutions for deploying and managing successful customer communities, including proven forums, blogs, and ideas.

Online Community Research Network http://www.onlinecommunityresearch.com

A research-based network of online community professionals working to understand key community building issues and best practices.

http://www.hivelive.com/

http://www.lithium.com/

http://www.onlinecommunityresearch.com/


EXECUTIVE SUMMARY

The Online Communities: Metrics and Reporting research study was initiated in late July 2009, and

ran until the second week of August 2009. The intention of the study was to get a broad look at what

online community metrics organizations are tracking, how organizations determine and report on the

ongoing value of their online community initiatives, and the reporting and metrics tools that help

companies in their efforts.

We received approximately 175 responses to our survey. Participants represent a healthy swath of the

types of organizations participating in online community culture. Participating industry categories

include: software companies, hardware companies, consumer goods firms, non-profit organizations,

independent consultants and media companies, among others.

A sample of the 175 organizations that participated include (with their permission):

Autodesk, BusinessWeek, Electronic Arts, EMC, Environmental Defense Fund, Google, Great Schools,

IBM, Juniper Networks, Microsoft, Quest Software, Sony Computer Entertainment, Sun Microsystems.

Viacom, and VMware.

Overall, we were very encouraged by the level and quality of response to the research questions.

Several key issues pertaining to online community and social media metrics surfaced during this report,

including:

In general, organizations need to do a more thorough job of defining their business objectives

for online community engagement, assessing ways to measure progress towards these

objectives, reaching beyond their native platform metrics capabilities, and finding ways to

measure the more qualitative components of community member engagement.

The Role of the Community Manager is increasingly important to developing and refining

business process, and measuring performance in these new “social spaces.”

There is a growing need for community metric standards that are platform and vendor-

independent.


TABLE OF CONTENTS

Key Insights: ...........................................................................................................................................5

01 Organization Type..............................................................................................................................8

02 Platform ..............................................................................................................................................9

03 Community Age ...............................................................................................................................10

04 Total Employees ..............................................................................................................................11

05 Annual Budget .................................................................................................................................12

06 Clarity of Expectations....................................................................................................................13

07 Purpose of Community ...................................................................................................................14

08 Alignment Metrics and Business Goals ........................................................................................20

09 Responsibility ..................................................................................................................................21

10 Determining What to Measure ........................................................................................................22

11 Technology for Tracking.................................................................................................................23

12 Metrics Currently Being Tracked....................................................................................................25

13 Determining ROI...............................................................................................................................30

14 Top 3 Key Performance Indicators.................................................................................................34

15 Satisfaction ......................................................................................................................................39

16 Additional Metrics............................................................................................................................41

17 Additional Advice.............................................................................................................................45

Final Comments....................................................................................................................................51


Key Insights: 20% of the respondents have a custom-built community platform. The three most common primary community platforms are:

Web: Facebook, Twitter, Blog

Clearspace Forums – Social Business Software

Lithium – Social CRM

9

Over a third of the participants belong to an online community that is younger than one year (38%), while over a fifth of the respondents belong to a community that has existed for more than 5 years (21%).

10

Over half of the respondents indicated that their company employs less than 500 employees (62%), the majority of those companies employ less than 50 employees (39%).

11

Over half of the respondents (51%) indicated that their online community platform annual budget is less than $50,000 and (10%) indicated it is over $500,000.

12

Just under half of the respondents (45%) felt that their management team has loosely defined the expectations regarding ROI and (38%) indicated that expectations are clearly defined.

13

The three most common primary purposes of online communities are:

Promote Services / Products

Collaborate / Gather Information

Specialty Group Network

14

Almost half of the respondents communicated that their community’s metrics goals were created in support of the organization’s business goals (47%). (31%) indicated that they had been created independently but were helping to redefine existing business goals

20

The three most common roles that are responsible for setting metrics and reporting goals are:

Senior Management Community Manager Marketing Team

21

61% of the participants said that they determine what is important to measure and report by working from a strategy based on business goals and finding solutions to help measure what is needed.

22


73% of respondents indicated that they either used the technology that their platform provided for tracking and reporting or developed in-house measurement solutions.

Google Analytics (40%) and Omniture (26%) were the two most common supporting technology tools that online communities use for tracking and reporting.

23

The top 3 items that online communities currently measure for tracking and reporting are the same for non-profit and profit related organizations, and are as follows:

Unique Visitors New Member Registrations Page Views

The top 3 items that online communities don’t measure, but want to are:

Member Satisfaction Influencer / Evangelism Member Life Cycle

25

The five most mentioned “other” metrics that respondents tracked within their online communities are:

Time On Site Posts by Type Member Demographics User Types User Engagement

30

Almost a third of respondents (34%) indicated that User Activity / Engagement is the most important item for their online community to track to determine ROI. This was considered the number one item for both profit and non-profit organizations

31

The top 3 key performance indicators in the reports respondents sent to management are:

User Activity / Engagement Membership Count [New Registrations, Active] Number of Posts / Comments

35

Over half of the participants felt that reports are either just satisfactory or that they need improvement.

40

The most common metrics that stakeholders would like to see, that are not currently reported, are:

Activity Levels / Engagement ROI: Revenue / Sales (Up sells / Renewals) User Conversion Metrics (Social Network Analysis)

42


The three most common pieces of advice that respondents wanted to offer to peers regarding metrics and reporting are:

Measure Metrics that Supports Community Goals Define Key Business Objectives First Monitor Quality of Interactions, not Numbers

46


01 Organization Type Derived from Question 5: In which industry or market segment does your company do business?

SUMMARY:

Respondents represent a healthy sampling of the types of industries that participate in online community culture:

18% (31) work in Software 14% (24) work for a Non-Profit 11% (18) work in Online Community/ Social Media Destination 9% (16) work in Media Company 6% (10) work in Online Retail 6% (10) work for an Agency 6% (10) work as a Consultant 6% (10) work in Education 4% (7) work for a Technology Manufacturer 3% (6) work for a Community Platform Provider 2% (4) work for the Government 25% (25) Other

“Other” organizations include; Research Organizations, Retail Organizations and Financial Institutions among others.


02 Platform From Question 6: What is your PRIMARY community platform? (Vendor, Product and Version, please)

SUMMARY:

20% (30) of the respondents said that their company had a custom built community platform. A large percentage of other respondents, 16% (24), indicated that they used web-based social media platforms such as Facebook and Twitter as their community platform. Other popular community platforms / software are as follows:

9% (14) Clearspace Forums 8% (12) Lithium 6% (9) Telligent Enterprise 5% (8) Sharepoint 5% (8) Drupal 5% (7) vBulletin 5% (7) Ning


03 Community Age From Question 7: How long has your online community existed?

SUMMARY:

13% (22) 0 – 3 Months 7% (12) 4 – 6 Months 18% (31) 7 – 12 Months 10% (17) 13 – 18 Months 10% (18) 19 – 24 Months 13% (23) 3 Years 5% (9) 4 Years 3% (6) 5 Years 21% (36) More than 5 Years

TAKEAWAY:

More than one third of respondents, 38% (65), have a community that is younger than one year, while 21% (36) have a community that is at least 5 years old.


04 Total Employees From Question 8: How many people does your company employ in total?

SUMMARY:

39% (68) Less than 50 23% (40) 50 – 499 8% (15) 500 – 999 13% (23) 1,000 – 4,999 4% (7) 5,000 – 9,999 13% (22) 10,000 or more

TAKEAWAY:

Over half of the respondents, 62% (108), indicated that their company employs less than 500 persons, while 39% (68) of those belong to a company that employs less than 50.


05 Annual Budget From Question 9: What is the annual budget for your online community platform and services?

SUMMARY:

33% (40) Less than $50,000 18% (22) $10,000 – 49,000 11% (14) $50,000 – 99,000 28% (34) $100,000 – 500,000 6% (8) $500,000 – 999,000 2% (3) $1 million - $4,999,000 2% (2) Over 5 million

TAKEAWAY:

Of the 123 respondents that selected an annual budget, over half (51%), indicated that their online community platform annual budget is less than $50,000 and (10%) indicated it is over $500,000. 52 respondents chose not to disclose their annual community platform budget.


06 Clarity of Expectations From Question 10: How clearly has your management team defined their expectations about value and return on investment (ROI) for your community?

SUMMARY:

10% (18) Very clearly defined value and targets 28% (48) Important milestones and benchmarks are defined 45% (78) Value is loosely defined 7% (12) Community team decides without input from management 10% (17) We do not have value and ROI

TAKEAWAY:

Most participants felt that their management team had some definition of expectations about ROI, although just under half, (45%), expressed that the value was loosely defined. As noted in later sections, the concept of value is made up of quantitative and qualitative measures, and is often not a hard dollar measurement of return. It is encouraging to see that most participants have at least a loose agreement about community value and ROI. Lack of agreement between management and staff on community value is one of the biggest risks in the sustained development and support of community programs.


07 Purpose of Community From Question 11: What is the primary purpose of your community?

SUMMARY:

Most of the respondents indicated the primary purpose of their online community was to share and gather information, support clients / customers, or to promote their company’s services / products. The following list encompasses the top five primary purposes for online communities:

24% (38) Promote Services / Products 24% (37) Collaborate / Gather Information 20% (32) Specialty Group Network 17% (26) Share Information 15% (24) Customer Support / Satisfaction

TAKEAWAY:

24% (38) of the respondents felt that a primary purpose of their online community was to promote services / products, including: driving leads, building brand advocacy and product evangelism, and encouraging product / service sales and renewals. 7% (15) of the respondents ranked product awareness and support as another related primary purpose of some communities.

Other respondents indicated that the primary purpose of their community was to collaborate / gather information, 24% (37), or simply to share information, 17% (26). Some respondents used their online community for team members, development or consumers to share and gather information, including sharing information on brands, ideas, issues and trends.

Some participant’s online communities had the primary purpose of bringing people together who belong to specialty groups, 20% (32), (minority groups, mothers, game players, travel enthusiasts) or those that support a common cause 2% (5) (spiritual, cultural, global concerns).

Customer support / satisfaction, 15% (24), was a common primary purpose of some online communities. Providing a platform for clients to network, share best practices and provide product support to one other ranked at 8% (16).


WRITE-IN RESPONSES:

Below are just a few of the responses from participants that describe the primary purpose of their online community:

“Promote Service / Product” related responses include:

Drive leads and some market research.

To breed product advocates and evangelists by offering direct contact, support, and conversation with the company.

We run several communities on behalf of our clients – all share one primary purpose, which is to turn loyal fans into brand advocates.

Developer community outreach and evangelizing.

Generating buzz about our work, moving the one-time visitor to become online advocate for our org.

Increase engagement among Enterprise customers – increase renewals, up-sell, and cross-sell.


To communicate and promote product updates.

Generate revenue and retain customers.

Provide community based support and to build brand advocacy.

Build the brand and spread our work externally.

Drive conversion.

Build visibility & generate dialogue.

“Collaborate / Gather Information” related responses include:

Collaborate on a new way to measure and manage sales and marketing by measuring and managing the relationship tied to profit.

Knowledge Management, Quality Management, Collaboration.

Knowledge sharing, peer-to-peer support, collaboration on content in wiki, learn from bloggers.

To assist developers with the platform, and to promote collaborative development.

Discuss emerging trends, technologies and business issues.

Collaboration and open dialogue between team members.

Team and subject-matter collaboration.

To gather ongoing in-depth consumer insights.

Sharing & gathering news and information on our focus areas, tracking the conversation about our brand and issues.

Internal collaboration community for our Staff and Contractors.

Share findings and best practices, ask questions, have conversations about industry trends.

Share information between employees, surface new business ideas, replace old corporate intranet.


“Specialty Group Network” related responses include:

To network with others who focus their time and energy on sustainable businesses (renewable energy, green building, organic food, etc).

To provide a means for our members and subscribers to receive advice, support and feedback from each other relative to their online and offline dating and romantic relationships.

Support the growth and development of knowledge focused networks of professionals and others interested in all the fields related to science, engineering and technology in order to boost our membership.

Educational community.

Create collaborate space for international health researchers in the fields of mental health and education.

An online community of teachers to bridge their isolation and develop 21st century skills, improve practice, and increase professional engagement.

“Share Information” related responses include:

In places with poor connectivity, we connect health workers with each other and to relevant clinical and public health information using technology.

Engage members to share experiences, which will lead to better choices by other members.

Provide members with information they need to be successful in their work.

Knowledge sharing and practice development in local government.

Educate, entertain, and interact with women and people interested in marketing to women.

Enable peers to leverage one another and to share best practices.

“Customer Support / Satisfaction” related responses include:

To build customer loyalty by providing our existing customers with a direct communication channel to our development teams to ask questions, take part in our betas, and have an influence on future product direction.


Our primary goal is facilitate the awareness and understanding of our clients and customers.

To increase our authority in our subject area and give our clients confidence in our ability and our knowledge.

The platform allows brands to include the customer voice in the decision making process.

360 degree Client Satisfaction.

Internally for increased collaboration and communication regarding products, troubleshooting, engineering, etc. Our forum is for customer support, tutorials, etc for increased customer care.

“Platform for Clients to Network” related responses include:

Enable customer-to-customer discussions regarding issues and solutions.

Increase open communication amongst global customers in the marine industry.

Enabling customers to help each other, reducing load on support and increasing sense of community.

Peer support, customer co-collaboration, and generally, help customers get more engaged and use the solution better.

Allow customers to share best practices and network.

Give customers a place to talk with other customers... connect, ask questions, share best practices.

Provide an environment for product users to communicate with other and help on technical problems.

“Product Awareness / Support” related responses include:

Product support, brand awareness.

Help customers complete and file their income tax returns; and help customers use their Quicken products. Moving toward discussions on personal finance.

Provide an environment for product users to communicate with other and help on technical problems.


To increase product utilization.

To provide a place for our customers and partners to learn about and discuss our product, ask questions and get answers.


08 Alignment Metrics and Business Goals From Question 12: Were your community’s metrics created in support of your organization’s broader business goals or were they created independent of a corporate business?

SUMMARY:

47% (81) Created to support existing business goals 31% (53) Created independently but helping redefine existing business goals 5% (9) Not in support of existing business goals at all 7% (12) No corporate business goals yet, metrics goals come first 10% (18) No corporate business goals or metrics goals yet

TAKEAWAY:

Almost half of the respondents, 47% (81), indicated that their community’s metrics goals were created in support of the organization’s business goals. 31% (53) indicated that they had been created independently but were helping to redefine existing business goals. It is worth highlighting the relatively high percentage of respondents, 31% (53), whose community programs are having a transformative effect on the organization.


09 Responsibility From Question 13: Who is responsible for setting metrics and reporting goals? Check all that apply.

SUMMARY:

22% (88) Senior Management 20% (80) Community Manager 16% (63) Marketing Team 14% (54) CEO / President 9% (36) Product Team 5% (21) Development Team 5% (18) Research / Data 2% (7) Sales 1% (4) System / Network administrator 6% (24) Other

“Other” employees responsible for setting metrics and reporting goals include: Program Manager, COO, Business Analyst, Social Media Strategist, User Groups, Product Marketing Team, Web Team, Board of Directors, and Internal Communications. TAKEAWAY:

Respondents stated that Senior Management was primarily responsible for setting community metrics and reporting goals, followed closely by Community Managers and Marketing teams. This data speaks to the increasingly important role the Community Manager (and team) play in the organization.


10 Determining What to Measure From Question 14: How does your organization determine what is important to measure and report?

SUMMARY:

20% (34) We stick with what the platform can provide 61% (100) We work from a strategy based on business goals and find solution to help us

measure what we need 19% (31) We try to measure everything, will develop more of a strategy later

TAKEAWAY:

Respondents are primarily shaping metrics strategies based on business goals, (61%), even if their platform doesn’t support gathering or tracking the desired metrics. Platform metrics are generally speaking, not comprehensive or extensible enough to create a meaningful dashboard to see overall community health, get an accurate visualization of the community’s social graph, or to understand the ongoing insight created by the sentiments of the community population. The risk in relying only on data that a platform can provide, as (20%) of the respondents indicated, is that the data sets aren’t comprehensive or contextual to the organizations’ needs. Measuring everything, as (19%) of respondents indicated, would likely overwhelm the community team and stakeholders, and is unlikely to yield meaningful performance data or insight without some rigor in the analysis.


11 Technology for Tracking From Question 15: What type of technology do you use for tracking and reporting? Please provide detail of the solution.

SUMMARY:

42% (69) We use what the platform provides 31% (51) We have developed in-house measurement solutions 19% (31) We use outside vendors for measurement 8% (14) We have purchased software


NON PLATFORM TECHNOLOGY:

The most common technologies that respondents used for tracking and reporting in their online communities (apart from reporting available from their software platform) are as follows:

40% (28) Google Analytics 26% (18) Omniture 4% (3) Social Media Tools: Twitter, Facebook 3% (2) Business Objects 3% (2) SQL Databases 24% (17) Other tools

TAKEAWAY:

The most common supporting technology tools that respondents use for tracking and reporting in their online communities are Google Analytics (40%) and Omniture (26%). Some respondents use Google Analytics and Omniture exclusively, while others use them in addition to in-house solutions, outside vendors, or in addition to the reporting available from their software platform.


12 Metrics Currently Being Tracked From Question 16: What do you currently measure?

SUMMARY:

The top five items that online communities measure for tracking and reporting are as follows:

152 Responses - Unique Visitors 150 Responses - New Member Registrations 143 Responses - Page Views 126 Responses - Visitors 116 Responses - Message Posts

The top five items that online communities don’t measure, but want to, are as follows:

90 Responses - Member Satisfaction 90 Responses - Influencer / Evangelism 84 Responses - Member Life Cycle 83 Responses - Member Loyalty 73 Responses - Referrals to Community

TAKEAWAY:

The top five items that online communities currently measure for tracking and reporting are the same for both profit and non-profit organizations and include Unique Visitors, New Member Registrations, Page Views, Visitors and Message Posts. Non-profit organizations concentrate on measuring Podcasts & Video Links and Member Satisfaction, more often than other organizations. Commercial organizations place more attention on measuring Retention / Attrition, Member Loyalty, Member Blog Posts and Conversion than non-profit organizations. As organization’s community strategies mature, the trend to primarily report on basic web metrics (page views, registrations) will be replaced by metrics that speak to the health of the community, the strength of members’ networks, the quality and type of member participation, and more robust measurements of member engagement. The data suggest that we are on the cusp of the evolution from “basic” community metrics to more robust and contextual reporting.

CURRENTLY MEASURING – PROFIT AND NON PROFIT Note: Graph is detailed on the next page. A large .PDF of the graph can be found at: http://www.onlinecommunityreport.com/images/metrics_tracked_2009_graph.png

http://www.onlinecommunityreport.com/images/metrics_tracked_2009_graph.png


TOP 20:


TOP 20: CURRENTLY MEASURING – PROFIT AND NON PROFIT

BOTTOM 20: CURRENTLY MEASURING – PROFIT AND NON PROFIT


TOP 20: CURRENTLY MEASURING – FOR PROFIT

TOP 20: CURRENTLY MEASURING - NON PROFIT


TOP 20: WANT TO MEASURE – PROFIT AND NON PROFIT

TOP 20: DON’T MEASURE – PROFIT AND NON PROFIT


13 Determining ROI From Question 18: Of the items you track, what do you consider the most important for determining ROI?

SUMMARY:

34% (49) User Activity / Engagement 13 Number of Page Views / Clicks 11 General Participation / Engagement 10 Number of Unique Visits 9 Number of Site Visits 7 Number of Active Users / Members 3 Number of Threads 3 Number of Returning Visitors 2 Amount of Time Users Spend On Site

10% (15) Sales Revenue – Renewals, Up Sells, Cross Sells… 9% (13) Number of Evangelism / Referrals 8% (12) Number of Posts / Comments 8% (11) New Member Registrations 7% (10) Member Satisfaction / Loyalty 6% (8) Number of Conversions 5% (7) Member-to-Member support 5% (7) Retention 4% (6) Reduction of Technical Support Calls 3% (5) Number of Downloads 1% (2) Fundraising Generated

TAKEAWAY:

Over a third, (34%), of respondents indicated that User Activity / Engagement was the most important item for their online community to track to determine ROI, this included Number of Page Views / Clicks, General Participation and Number of Unique Visits. Other participants felt that Sales Revenue, (10%), including renewals, up sells and cross sells, as well as Number of Conversions, (6%), and New Member Registrations, (8%), were the most important items to track. Member-to-Member support, (5%), and the resulting Reduction of Technical Support Calls, (4%) were considered the most important factors for determining ROI for other participants. User Activity / Engagement, (34%), was the number one item to track for both profit and non-profit organizations, although within this category, profit based organizations had a significantly higher percentage response rate for tracking Number of Page Views, whereas non-profit organizations had a significantly higher percentage response rate for tracking Number of Active Users / Members. Another common response from profit based organizations was related to tracking Sales Revenue, while non-profit organizations had a higher percentage response rate for tracking New Member Registrations and Member Satisfaction / Loyalty.


USER ACTIVITY / ENGAGEMENT 34% (BROKEN OUT):


MOST IMPORTANT ITEM FOR TRACKING ROI – PROFIT

MOST IMPORTANT ITEM FOR TRACKING ROI – NON PROFIT


USER ACTIVITY / ENGAGEMENT – PROFIT

USER ACTIVITY / ENGAGEMENT – NON-PROFIT


14 Top 3 Key Performance Indicators From Question 19: What are the 3 most important community key performance indicators in the reports you send to management?

SUMMARY:

32% (74) User Activity / Engagement 21% (49) Membership Count (New Registrations, Active) 18% (42) Number of Posts / Comments 5% (12) Member Satisfaction / Loyalty 4% (10) Number of Questions Answered 4% (10) Sales Revenue - Up Sell, Cross Sell, Renewals 4% (8) Leeds / Referrals Generated 3% (6) Number of Downloads 3% (6) Number of Influencers / Evangelists 2% (5) Visitor Retention 2% (5) Number of Conversions 1% (2) Donations Received 1% (2) Visitor Geographic Dispersal

TAKEAWAY:

Almost a third (32%) of respondents indicated that User Activity / Engagement is one of the most important key performance indicators in the reports that they sent to their management. The following three key performance indicators were the most commonly reported in the User Activity / Engagement category:

33 – Number of Page Views / Clicks 22 – Number of Site Visits 19 – Number of Unique Visits

The other two key performance indicators that many respondents input into management reports are Membership Count, (21%), (including new membership and total membership count) and the Number of Posts / Comments, (18%), received on their site. Both profit and non-profit organizations list User Activity / Engagement as the number one item to track. Within this category the specific key performance indicators were dispersed similarly, with the profit based organizations having a slightly higher percentage ratio on key performances such as the Number of Threads reported and General Participation. Non-profit organizations, on the other hand, have a slightly higher percentage ratio on reporting metrics such as Number of Returning Visitors and the Number of Site Visits. Another common response from profit based organizations was related to reporting key performances such as Sales Revenue and the Number of Conversions, while non-profit organizations had a higher percentage response rate for reporting the amount of Donations Received.


USER ACTIVITY / ENGAGEMENT 34% (BROKEN OUT):



USER ACTIVITY / ENGAGEMENT – NON-PROFIT



USER ACTIVITY / ENGAGEMENT – NON PROFIT


PROFIT & NON-PROFIT COMPARISON CHART


15 Satisfaction From Question 20: How satisfied are executive stakeholders with current metrics reports?

SUMMARY:

22% (37) Current reports are excellent 27% (45) Reports are just satisfactory 29% (47) Reports need improvement 9% (15) Reports are extremely poor 13% (21) We don’t currently track and report metrics

TAKEAWAY:

Over half of the participants, (58%), felt that reports are either just satisfactory or that they need improvement. 13% of the respondents indicated that they do not currently track and report metrics, although several commented that while they do not track metrics for management purposes, they track metrics for their own use. It is a relatively straight line from unclear management expectations (Section 06) through poor alignment of management and community goals (Section 08) to get to low stakeholder satisfaction with the metrics reports being produced.


WRITE-IN RESPONSES:

Below are key write-in responses from participants, outlining the level of satisfaction that executive stakeholders have with their current metrics reports listed below:

User responses include:

Unfortunately the executive stakeholders see curves of large numbers all up and to the right, and have been satisfied. Only now are they beginning to ask what the real business value is. It is really the team that is responsible for trying to unify and coordinate cross-divisional strategy that is frustrated by a lack of focus on KPI's.

We have many measurements but no real analysis of the data. I am working with one system that will measure a number of factors regularly that I and others can analyze.

They don't really know what they need, they rely on us to tell them.

We always wish we could glean more understanding from the activity and content, but find number crunching takes us away from that goal.

Show greater correlation to sales and ROI than traditional media channels.

We have received excellent data from member surveys.

We get feedback periodically, but our reports are used mainly to confirm we're on track – management isn't really that engaged.

Metric reports are NEVER good enough.

Stakeholders want real-time, dynamic metrics delivered at will. Our systems cannot provide that at this point.

The value we see on our level is excellent, executive stakeholders are accustomed to seeing other data (traditional marketing values which more clearly demonstrate ROI than community engagement) which makes it more difficult for that value to be relayed properly in a way that makes sense for executive stakeholders.

We have not yet reported anything other than # members and # discussions. The community is very new.

We do not send metrics reports to executives, but we do track them for our own internal use.


16 Additional Metrics From Question 21: What metrics would stakeholders like to see that are not currently reported?

SUMMARY:

31% (18) Activity Levels / Engagement 17% (10) ROI: Revenue / Sales (Up sells / Renewals) 16% (9) User Conversion Metrics (Social Network Analysis) 12% (7) Content Evaluation / Tracking 7% (4) Members directed from Social Media Tools 7% (4) Qualitative Engagement Measuring 5% (3) Retention 5% (3) ROI: Reduces Costs

TAKEAWAY:

Almost a third (31%) of the participants indicated that their stakeholders would like to see metrics that are related to Activity Levels and User Engagement. The engagement category consisted of responses such as the number of active members, number of site visits, the number of page views, time spent on site and content views. In addition to these behavioral metrics, a proper engagement metric would also include attitudinal data that illustrates member satisfaction, affinity, and likelihood to recommend. Return on Investment was another metric that stakeholders wanted to see in their reports, including Sales, renewals and up sells, (17%), and how the online community affects Reduction in costs, (5%). Another common response from participants was the need for stakeholders to see data related to User Conversions, (16%), in their reports, including the number of conversions and how, when, and where the conversions are taking place. Other metrics that stakeholders felt that they would like to start seeing in their reports, is Content Evaluation, (12%), which includes quality and usefulness of the content posted and a record of key distributors. A small percentage of participants indicated that they would like to see a record of Members Directed from Social Media Tools, (7%), Qualitative Engagement Measuring, (7%), and Retention, (5%).


WRITE-IN RESPONSES:

Below are key write-in responses from participants outlining the metrics that stakeholders would like to see that are not currently being reported:

“Activity Levels / Engagement” related responses include:

Active members, customer journeys.

Life cycle of the user. And the seasonal visiting, as we face a problem in identifying our growth, we have seasonal visitors, so we need to compare our growth with a similar season not just the previous month.

Unfortunately they are not yet aware of the power of the community, thus they have not asked for so many.... But, account managers and consultants want to see the activity of their customers.

We would like to see a Time Spent on Site metric. Page views/member vs. non-member would be very useful. Number of visits before initiating registration would be useful.

Specific content views. We track this, but our content is not tagged and it is difficult to organize for


reporting.

Time spent in the community (average per user, min, max, etc.).

“ROI Revenue / Sales” related responses include:

Attrition, cross-sell / up-sell, member lifecycle (all the real ROI stats!).

What percentage of the users are product owners? What level of purchase have they made?

They'd like to see more hard evidence of how the community reduces costs or increases revenues. As we all know, communities tend to have a indirect impact on those metrics, but we need more case studies and metrics that can help us communicate the value of communities to our exec chain.

Correlation with sales - renewals and up-sells.

ROI to include collateral benefits, i.e., a question may be answered and clearly benefit the original poster, but subsequent viewers will also benefit, and we can only infer that impact.

“User Conversion Metrics” related responses include:

Social graph related to conversion and retention.

Where the conversations are taking place - i.e. Social Network Analysis.

Lead conversion.

More accurate monthly conversion rates, customer lifecycle.

Comprehensive social media tracking.

“Content Evaluation / Tracking” related responses include:

What are the characteristics of social media content that lead to new customers.

Objective "quality" of content produced.

Overall how useful is the content to our consumers.

Analytics on how information based on relations flow, who are the information distributors? Who are


the bottlenecks?

“Other” related responses include:

The number of members from other social media tools such as Twitter, Facebook, MySpace, YouTube, Flickr which go through to the website and donate money.

Pipeline of customers filtering from loyalty program (highest level) to company portal (mid-level) to blog (lowest level) Sentiment of comments - text analytics/mining Top ideas (from idea generation tool) - summarized by themes automatically (text mining) Customer satisfaction rating at individual level based on survey answers, comments, ideas etc.

Need more bench marking to make the current data meaningful.

More info on the members themselves.

Reach, quality of conversation, fuzzy things like that.

Click through rate from Facebook stream, engagement rate for items that appear in Facebook stream.


17 Additional Advice From Question 22: What additional advice do you have for peers about metrics and reporting?

SUMMARY:

20% (13) Measure Metrics that Supports Community Goals 20% (13) Define Key Business Objectives First 11% (7) Monitor Quality of Interactions, not Numbers 6% (4) Analyze Metrics to Find the Story In the Numbers 5% (3) Follow Standards 5% (3) Ensure Metrics Supports Qualitative Data 5% (3) Measure Everything, from the Beginning 5% (3) Don't Track Metrics Just For Management Reporting 4% (3) Don't Measure Everything 4% (3) Measuring Metrics is Crucial to Business Success 3% (2) Use Platform that can Report ROI Stats to Execs 3% (2) Share Experiences 3% (2) Measure Metrics against Past Historical Periods 3% (2) Change Goals as Necessary 3% (2) Measure Metrics Reflecting What Membership Wants

TAKEAWAY:

The two most common pieces of advice were related to online community business goals and objectives. The first was to Define Key Business Object First, (20%), thus preventing disjointed strategies and additional development work later on, and the second was to Measure Metrics that Supported Community Goals, (20%), and business objectives that are actionable. Other pieces of advice were related to the type of metrics being tracked. Some respondents gave the advice to Monitor Quality of Interactions, not just Numbers and Not to Track Metrics Just For Management Reporting, instead learn to analyze content and to Report on User Interaction and Satisfaction, instead of just reporting on a bunch of numbers. Some respondents suggested that both qualitative (content) and quantitative (data) data should be tracked, but that Quantitative Data should Support Qualitative Data. Some respondents gave conflicting advice on the importance of metrics and reporting, some participants gave advice to Measure Everything from the Beginning, whereas others gave the advice, Don’t Measure Everything. There were many other pieces advice that respondents offered, such as the importance of Following Standards, Using Platforms that can Report ROI Statistics to Management, Sharing Experiences, Measuring Metrics against Historical Periods, Being Flexible and Changing Goals as Necessary, and Measuring Metrics Reflecting What the Membership Wants.


RESPONDENT RESPONSES:

Below are just a few of the responses from participants, outlining any additional advice that participants had for peers regarding metrics and reporting.

“Measure Metrics that Supports Community Goals” related responses include:

Focus on metrics that are actionable and relevant to key business objectives.

Don't have too many metrics, use metrics that get to the core of your business goals - choose metrics that allow you to gauge your progress towards specific goals (business & member goals) - include both qualitative and quantitative goals - don't be afraid to get rid of goals and add others... wash, rinse, repeat.

Make sure your metrics correspond to your purpose for the community, and also that your purpose corresponds with your community members' purpose for being there.

Don't track it if it won't change your decisions.

Make sure that your data has significance to an overall business plan. It makes no sense to collect


nice numbers if those counting the beans only care about unique visitors or time spent on site etc…

Don't get lost in metrics that don't affect your business outcomes.

Metrics are important but more relevant are the actions based on the metrics

So much information is available you need to think through what really matters and watch it closely.

Be sure to link community metrics to the business metrics. If the business metric is to increase renewals, you should track renewal metrics. Don't get bogged down in metrics that don't relate to the ultimate business goal.

“Define Key Business Objectives First” related responses include:

We did not successfully defined the key business goals first, and then created community metrics that measure our drive to those goals. This has resulted in a 'me too' strategy where many internal stakeholder groups are wanting to build a community site for 'their' customers but have no clear answer to the reason 'why?’

What business goal are you trying to serve? How do you measure success?

Holding the line early and defining these up front will save a lot of long-term rework and will prevent fragmented and disjointed strategies.

Get involved early in development so metrics code gets inserted. Inserting this code after the fact adds months to completion think ahead about the metrics and reporting infrastructure early, so that queries and reporting can be done efficiently without additional development work.

Balance metrics with top quality surveys related to the business. Neither gives the full picture. The worst: vanity metrics that don't indicate the health of the business, and lazy surveys that don't generate insight into the customer's experience.

Invest in BI tools, only after you know what data you want to measure.

Determine what you want to measure first, then build the system with appropriate web statistical tools.

Make sure the outcomes of your community are crystal-clear and defined first ... then, put your resources (i.e. $$$ and time) against driving those outcomes. Your metrics are really just indicators of the performance of those resources. Everything else is unnecessary.

Think about how to track/measure it before you build it.

Create a baseline as you get started with any major community project and use reports to show results proactively. This way you set the parameters that you can manage or create transparency


on what is feasible and what is important to your company.

Determine what your broader goals are for what you're trying to achieve with your online community, then determine what to measure, how to measure it, and then how to analyze it.

A community can't be successful until you determine how you're measuring success.

“Monitor Quality of Interactions, not Numbers” related responses include:

Make sure to push back against management desire for defining success by # of registrations and page views. A better measurement of success is to show increases in the number of users who regularly participate.

Pay attention to quality of interactions, not necessarily numbers of them.

While it is important to show metrics our execs, as a community manager it is critical that your metrics are helping you understand what the membership wants and needs are. Remember, communities exist to benefit the membership. If the membership sees no value in the community, the community will cease to exist.

Learn to analyze the content, rather than a bunch of numbers, that often (usually?) don't mean what they're supposed to represent. For example, time spent can indicate engagement; or, it can indicate the visitor got a phone call while on the site.

Numbers never tell the whole story. The best metric is the pulse of the community.

Find measures that report interaction and satisfaction, and don't benchmark quantitative measures against other communities.

“Analyze Metrics to Find the Story in the Numbers” related responses include:

Metrics only go so far, listening to and understanding why certain trends may occur are for more important. For example, we may see a surge in login activity during large updates, this is due to our services being offline with the exception of web services (website/forums) should we see a rise in a period where normally there shouldn't be it is my responsibility to assess and investigate why this is. Metrics should always be supplementary to help support qualitative feedback, not the other way around.

Metrics alone tell you very little. You need to find someone who can both find the story in the numbers AND be able to tell that story to others. That's a rare person.

Whilst I see the importance of metrics for some stakeholders, it is important to have a dedicated community manager or team to read between the lines of figures and ascertain the immeasurable. What sort of stories do those figures tell, or belie? Are they related to content trends that can't be


repeated – e.g. members complaining about performance issues.

“Other” related responses include:

You can't properly run or grow your online business without proper metrics and reporting. If you don't know who your customers are, what they buy, when and why they buy - you won't stay in business long. MEASURE EVERYTHING from day 1! It's the best investment in your community, organization, or business.

Don't obsess too much over them - if the community is buzzing and happy, the good stuff will rise to the surface...

Many general website statistics are nice, but do your best to get a platform that can report important ROI stats...makes the growing idea of Community and Enterprise 2.0 much more palatable to management.

It's crucial if you want to have a successful community. If you aren't actively looking at your metrics and using the analysis and results to improve your website, you'll die on the vine.

Monitor and measure everything, even if managers are not asking for a particular metric. The likelihood is that they will ask for it at some point in the future.

It would be very nice to have the right resources and tools to do proper metrics. Why not come together and start sharing our experiences?

Separate out "indicators" like page views, time on site, number of comments per post from true "performance" metrics that relate to sales growth or cost savings. Ensure your performance metrics tie into an ROI model, even if your management isn't currently asking for this - community longevity depends on ROI.

Standardization is needed.

I prefer doing two versions of metrics, one much more general and top-line that people can scan and read once a month, but also a much more in depth one for the web/Public Affairs teams to delve into.

There needs to be a balance between quantity (metrics) and quality (of content, feedback, etc.). Collect anecdotes, thank you’s, and other feedback received through your collaboration tools to help complement your metrics and better represent the value of your communities.

Gather everything (as long as it is not too time-consuming), but look for opportunities for comparison and metrics that prompt action/response.


There is no standard when it comes to reporting systems as to what should be measured and reported. In fact, the same metric reported by two different sources may be calculated quite differently. An industry standard needs to be created that 1) debunks "hits" as being valuable 2) solidifies exactly what constitutes a unique visitor and over what standard time period that is measured 3) standardizes the methods of determining each metric.


Final Comments

This research was conducted with sponsorship support from HiveLive and Lithium Technologies. We would like to thank all of the participants who spent time giving us their frank and candid answers and feedback. Thank you. We really appreciate your interest and effort!

We hope that you continue to find value in the qualitative feedback that we are presenting in this format. Please let me know if you have any suggestions that would help improve our process.

Forum One will continue to explore the emerging issues around online community and social media as the primary purpose and agenda of the Online Community Research Network. http://www.onlinecommunityresearch.com

Best regards,

Bill Johnston Chief Community Officer Forum One Networks

[email protected]

About Forum One Networks & The Online Community Research Network

Forum One Networks http://www.forumonenetworks.com connects those who connect others online. We provide events, research, networking and strategic consulting for online community and social media professionals.

We firmly believe that the best source of information about online communities and social media are those who play in this space daily. All of our activities are based on this premise.

The Online Community Research Network (OCRN) http://www.onlinecommunityresearch.com is a collaborative effort of online community professionals to better understand the principal challenges of building and managing online communities. The Research Network coordinates the collection, analysis and dissemination of useful information among online community professionals.

For further information, or for sponsorship / partnership opportunities, please contact Bill Johnston [email protected]


mailto:[email protected]

http://www.forumonenetworks.com/


mailto:[email protected]

Date post:	13-Sep-2014
Category:	Technology
View:	1,742 times
Download:	0 times