Post on 14-Jan-2015
description
transcript
Questions? info@measuring-success.com #7steps
7 Steps for Data-Driven Decision-Making
Sacha Litman, Managing Director, Measuring Success
José Fernández, Director of GuideStar Exchange
Questions? info@measuring-success.com #7steps
Agenda
5 minutes Opening poll and introduction5 minutes Benefits to using data15 minutes Illustrative Case study5 minutes Q&A and concurrent poll25 minutes 7 Steps to data-driven decisions5 minutes Take-away principles & talking points 10 minutes Q&A and concurrent closing poll1 minute Conclusion
Questions? info@measuring-success.com #7steps
A. Benefits
Questions? info@measuring-success.com #7steps
Benefits of data-driven decision making to Customers
Success matters because your mission is important!If you don’t measure and use data, you are not achieving the full potential of your mission.
Questions? info@measuring-success.com #7steps
Benefits to Management & StaffFocus limited time and energies on key activities that are real issues and have impact on our mission
Set goals for staff and hold them accountable
“Cannot manage what you do not measure” – Peter Drucker
“What gets measured gets done”
“High performing organizations use data analytics roughly three times as extensively as the lower performers in their field” – Competing on Analytics, HBS Press
Questions? info@measuring-success.com #7steps
Benefits to Boards and Donors
BoardsFocus their energies away from emotions & anecdotes
Decisions not made by loudest or wealthiest voice
“Analytical projects aimed at improved outcomes had a median ROI of 55% for CRM and 139% for financial management”– Competing on Analytics, HBS Press
DonorsAbility to show “return on investment”
Attract large gifts and grants by confidence engendered by data
Data analytics was one of the top three areas of increased investment by companies during the recession.
Questions? info@measuring-success.com #7steps
Popular Press ExamplesProfessional sports: Oakland A’s (Moneyball)
Maximize team wins while minimizing payrollBy hiring players as indicated by data, not scouts’ gut
Health Care: Intermountain Health Care (NY Times Magazine Nov 9, 2009)Improve patient outcomes (survival, longevity, quality of life) while minimizing cost By selecting treatment/procedure supported by evidence not Doctor’s gut instinct or unneccessary procedures
Technology: GoogleIncrease advertising revenues ($21 billion) by targeting ads to right customersBy mining data on all users of free Google consumer software
Questions? info@measuring-success.com #7steps
B. Quick Case Study
Questions? info@measuring-success.com #7steps
1. Chapter-based community centers
Experiencing some with member drops, financial insolvency. Challenge to business model & brand consistency.
Association HQ
Local Affiliates or Regions(N=150)
Individual Members / Users(N=1000 to 10,000 per affiliate)
Questions? info@measuring-success.com #7steps
2. Association sought “early warning system” & understanding of what led to
successful outcomes
Questions? info@measuring-success.com #7steps
3. Build measurement and dashboard system as pilot experiment
Engaged 6 willing chaptersBuilt:
Customer surveyEmployee surveyFinancial analysis toolMember participation tool
Questions? info@measuring-success.com #7steps
4. Focused on rankings and metrics that were statistically valid
Regression analysis ties activities to outcomesComparison
Against peer chapters in other geographiesAgainst local competition from other organizationsAgainst own prior measures (longitudinal)Within demographic segments of member base(see following slides)
Questions? info@measuring-success.com #7steps
4a. Identify activities associated with outcomes (multiple regression analysis)
Budget Management versus Value for Membership DollarCorrelation
Perceived Budget Management & Transparency(1= strongly disagree, 5 = strongly agree)
Valu
e fo
r Mem
bers
hip
Dol
lar
Ave
rage
Sco
re 1
-5 S
cale
Questions? info@measuring-success.com #7steps
5. Like doctor, ran diagnostics on each chapter every 2 years
Rolled out pilot to all chapters; readiness factors to participateMutually agreed on a plan for improvementSome consulting support
Select Measures from Customer Survey
Rank (of 15)
Score% StrAgree
Priority Goals & Strategy
Membership Value for the Dollar 7 25% Medium Focus on quality, budget perceptions
Professionals welcoming 2 35% Low
Budget Perceived as well managed
14 16% High Double scores in 2 years.
Questions? info@measuring-success.com #7steps
2. Consumer Latent Needs(Purpose-Driven)
3. InstitutionalStrategic Vision
4. FinancialSustainability
(Surplus/Deficit)5. Resource Intensiveness
(Mgmt Time, Sqft, Foot Traffic)
6. Contribution to People Pipeline
(Intake, Attrition)
1. Consumer Expressed
Wants
Data Creates OrganizationalAlignment & Intentionality
Questions? info@measuring-success.com #7steps
6. Turnaround: focus lead to improvement from low to average in 2 years; now aiming for top
Personal Conversations with CustomersMonthly or more often
27%
Questions? info@measuring-success.com #7steps
Data encourages prioritization:80% of board & mgmt team hypotheses about what we
anecdotally “believe” is a problem is not supported by data!
Questions? info@measuring-success.com #7steps
7. Chapter ruled by anecdotes: 80% of assumptions were not supported by data
Assumed they were strongest with eldest & wealthiest portions ofparticipant base
Shock, challenged the data, acceptanceManagement team focused energies on improvement with eldest and wealthiest. 2 years later, scores significantly higher there.
50% 43% 50% 48% 45% 40%
Less than $100,000
$100,000-$199,999
$200,000-$299,999
$300,000-$399,999
$400,000-$499,999
$500,000 and over
40% 42% 67% 63% 42% 46%
25-34 35-44 45-54 55-64 65-74 Over 75
Likelihood to recommend to a friend% Strongly Agree
Questions? info@measuring-success.com #7steps
8. Rising Tide Lifts All Boats: top, average, and low performers all improved
XX
Surplus Margin for Early Childhood ProgramSurplus as % of Expenses, after allocating all overhead
Questions? info@measuring-success.com #7steps
9. New policy: value, not priceAssumed that key driver of member retention was priceAnalysis shows not price, but perceived quality and value-for-dollarResult: association stopped encouraging price subsidization, encouraged perceived quality improvementPerceived value and value for dollar are tracked carefully and promoted system wide
Highest WillingnessTo Pay (center)
A. FinancialAbility
B. CommitmentTo Issue
C. PerceivedQuality of Services Least Ability to Affect
Demand Function
Questions? info@measuring-success.com #7steps
10. Outcome improvement: participationChapters that embraced this approach outperformed others significantly in participant enrollment and financial sustainability, despite the recessionParticipant Retention Rate:
91% to 96%New Participant Rate:
5% to 10%
Net Participation: 96% to 106%
Financial sustainability increased: coverage ratio (% expenses from fees & membership) grew from 74% to 80%
Questions? info@measuring-success.com #7steps
C. 7 Steps to Data-Driven Decision Making
Questions? info@measuring-success.com #7steps
Data driven decision making is parallel to what we learned in middle school science class
Identify issueState hypothesis: “I believe…”Perceived mechanism/ causeDesign experimentExamine dataConfirm or reject hypothesis
Questions? info@measuring-success.com #7steps
7. Commun-
ication
6. Decision Making
5. Inter-
pretation
4. Data Analysis
3. Data Collection
7 Stages of Data-Driven Decision Making
2. Hypothesis Develop-
ment
1. Framing
the Problem
Questions? info@measuring-success.com #7steps
2. Hypothesis Development
Brainstorm hypotheses. Many will turn out to be incorrect, but that’s ok so long as you can articulate a plausible mechanism.
Remember, our consistent experience with organizations is that over 80% of their initial hypotheses as to what is driving a problem are not supported by the data!
During brainstorm define:Hypothesis (may need to revise several times)Mechanism (cause & effect)How could measure it
Questions? info@measuring-success.com #7steps
2. Hypothesis Development: Gann Academy
Initial Hypothesis: unevenness in students’ experience getting their individual learning needs met by the schoolPlausible Mechanism: additional academic support (learning center).
Major differentiator in students’ learning experience (for those in the same classrooms)
Final Hypothesis: students who received additional academic support from learning center felt their individual learning needs were better met by the school.
Questions? info@measuring-success.com #7steps
2. Hypothesis Development: Gann Academy
How measure? Parent surveyQuestion on individual learning needs met by schoolCut data by whether student used learning center
t
Learning Center Yes
Learning Center No
84 2403.5 4.0
Scale 10a. Teachers Understand
Needs & Employ Effective
Techniques
Strongly Disagree 3%Disagree 9%Neither Agree Nor Disagree 16%Agree 44%Strongly Agree 28%
Average 3.8
CountAverage
Questions? info@measuring-success.com #7steps
3. Data CollectionDo…
Prioritize hypotheses Work backwards: figure out how will use results of data first before collect it Consider several methods of data collection Save time and money by considering simple, pre-built tools allowing some customization (instead of building own or buying highest end product you won’t use)
Don’t…Just go after what’s easiestOvertax your resources –data collection can be very time intensiveGo after more data than you can act upon –Garbage In Garbage Out (GIGO)
Questions? info@measuring-success.com #7steps
3. Data CollectionTool Pro Con
Survey Data collection phase is automated and not resource intensive
Obtain perceptions of quality
To get quality results and high response rates, need to put lots of time into asking the right questions, making it look professional, and assuring confidentiality
Financial analysis Data your organization already has
Requires reorganizing and “fine slicing” the financials, so need a strong CFO
Tracking system (CRM)
Integrated into daily efforts Takes a lot of discipline and time to build data
Self-reported from existing databases
Already have the database Time consuming if database does not contain the right information may end up with lots of estimates
Questions? info@measuring-success.com #7steps
3. Data Collection : Gann AcademyGood: financial modeling
Clarified board’s needs up front: 5 year projections, key levers of sustainabilityUsed already-built but simple model (in Excel) instead of building ownWorked hard to gather consistent financial data (counting $ in development and CFO’s office)“What if” scenario building enabled board buy-in
Bad: admission trackingPurchased high end database systemNot trained properlyOrphan Excel spreadsheetsNever clarified what management needed for decision making, so “GIGO”
Questions? info@measuring-success.com #7steps
3. Data Collection : Gann AcademyWork backwards. Decide how you will use the data first, as this will dictate whether and how to ask for the data.
Questions? info@measuring-success.com #7steps
You don’t need to be a statistician to correctly interpret dataHowever, do need to understand basic concepts
MeanStandard deviationCounts (sample size)Benchmarking
Frequently, someone will “find” something in the data that is not valid but it supports their personal views
5. Interpretation: draw correct conclusions from data. Danger in “seeing what want to believe”
Questions? info@measuring-success.com #7steps
Board member who was convinced the most severe threat on enrollment was from charter and public schools, not area private schools Hypothesis: public schools are better at supporting individual student needs and thus pose the greatest risk of attrition for Gann Academy families. Data source: teacher attention to individual needs cut by most attractive alternative option
5. (Mis)Interpretation: Gann Academy
Private Schools
Charter or
magnet public
schools
General public
schools
50 4 513.8 3.4 3.6
CountAverage
Relative Strength of Gann Academy’s Attention to Individual Needs vs.
Most Attractive Alternative School (higher score is better)
Gann vs. Gann vs. Gann vs.
Questions? info@measuring-success.com #7steps
Contextualization is critical:Peer group (similar organizations nationally/internationally)Local competition*Own longitudinal historyAcross demographic groups*
*Depicted on a prior slide
5. Interpretation: Benchmarking Relative to What?
Questions? info@measuring-success.com #7steps
Relative to Peer Group of other similar high schools around the countryNote if had used primary schools (K-8) as peer group
5. Interpretation: Benchmarking
Questions? info@measuring-success.com #7steps
7. Communication
Sharing with board, participants, donors, employeesFear: sharing data will undermine management’s authorityReality: sharing data engenders trust
Diffuse concerns through transparencyEven if current scores are not as high as would like, donors value a roadmap toward and metrics for success
Questions? info@measuring-success.com #7steps
7. Commun-
ication
6. Decision Making
5. Inter-
pretation
4. Data Analysis
3. Data Collection
7 Stages of Data-Driven Decision Making
2. Hypothesis Develop-
ment
1. Framing
the Problem
Questions? info@measuring-success.com #7steps
Common Non-Profit Applications of Data-driven Decision Making
Outcome measurement
Identifying activities that give greatest “bang for buck” in limited resource environment
Financial sustainability
Setting program fees or membership dues levels
Identifying most susceptible demographic groups
Donor prioritization using predictive models
Board reporting
Program evaluation
Questions? info@measuring-success.com #7steps
If your organization makes data-driven decisions you will be able to…
Focus limited energies and resources on only the 20% of hypotheses that are supported by the data
Motivate staff and volunteers because they know they are investing in what works
Align efforts of staff, volunteers, board, donors, and customers
Monitor progress toward goals
Have a dashboard for your performance
Make strategic board decisions more efficiently and gain more buy-in
Distinguish yourself against and outperform your competition by developing a core competence few non-profits have
Observe improvements in key areas such as: financial sustainability, donations, participation, mission impact
Questions? info@measuring-success.com #7steps
7 Steps for Data-Driven Decision-Making
Sacha Litman, Managing Director, Measuring Success
José Fernández, Director of GuideStar Exchange