Actionable Metrics for Continuous Improvement:
Balanced Scorecard Survey ToolsAlexis Naiknimbalkar, CSU Chancellor’s Office
Angela Song, UC San Diego
Key topics for today1. Introductions 2. Quick view of the Balanced Scorecard3. The UC San Diego Surveys and Analytics Program4. Overview of the Surveys5. How the Survey results are used to drive improvements6. The CSU – UC Collaboration Journey begins!7. Next steps
The CSU-UC collaboration
• Background• The collaboration • UC San Diego’s surveys• By using these standard questions, it enables benchmarking and
sharing of best practices between Universities and departments
First… A quick history about UCSD’s Balanced Scorecard
What is the Balanced Scorecard?
• History:• Developed in 1993 by David P. Norton and Robert Kaplan
• More than half of major companies in the US, Europe and Asia are using balanced scorecard approaches
• Harvard Business Review rates the Balanced Scorecard as one of the most influential business ideas of the past 75 years.
• Four Perspectives:• Financial/Stakeholder
• Internal Processes
• Innovation/Growth
• Customer
• Purpose:• A framework for performance
measurement, communication, and management tied to strategy
• Provides a roadmap on where we should focus energies, priorities and resource based on a holistic view of the organization.
The Survey Program’s Evolution at UC San Diego
1993 –16 Business Affairs units begin using the balanced scorecard approach
UC San Diego inducted into Balanced Scorecard Hall of Fame
Customer service, department outreach, and efficient internal processes are an expectation and norm
Surveys provide actionable data for continuous improvement initiatives and in support of the campus strategic plan
Best practices and benchmarking opportunities
1994 -Customer Satisfaction Survey
1995 -Student Satisfaction Survey
1997-Staff Climate Survey
2005-20151994-2004 2016-beyond
UC San Diego – VC Business Affairs UC San Diego – All VC AreasUC San Diego, UC Irvine (OIT), UC Riverside (BAS), CSU Chancellor’s Office (BAF)
2003-Surveys go online
Key Performance Indicators identified and benchmarked
2011 -Scalable advanced analytics and reports
2012 –UC San Diego’s 1st strategic plan
2014 -Surveys are fully scalable
Mission and vision aligned with strategy
2004-Surveys expand to all campus
2007-First study on diversity and staff satisfaction
LETS TALK ABOUT SURVEYS FOR A MOMENT:
Why surveys?
Surveys are fun!
5 most bizarre survey finds
❶❷❸❹❺
From: oddee.com
5 most bizarre survey finds
❶ 23% thought “MP3” was a Star Wars robot
(– Vouchercloud.net)
5 most bizarre survey finds
❷ 51% of surveyed Americans think stormy weather "affects" Cloud Computing
(-Wakefield Research, 2012)
5 most bizarre survey finds
❸ 1 in 4 Americans thinks the Sun goes around the Earth
(-NSF, 2012)
5 most bizarre survey finds
❹ Average Americans think they're smarter than the Average American
(-YouGov, 2014)
5 most bizarre survey finds
❺ Survey found that most Americans (75%) don’t trust Survey Results
(-Kantar data investment management poll, 2013)
Surveys allow a way to understand people’s attitudes, feelings, and behaviors
Do I think this is a good place to work? Will I do my best to contribute or will I just coast? How will I talk about it to others inside and outside the university? Do I feel valued? Do I feel I am making a difference? Am I feeling engaged and inspired orbored and just clocking my time? Does X department even care if they are helping or getting the way of my work? ……
Why survey in a university setting?
• Did we help a researcher spend more time in his or her lab (i.e., rather than track down policies and fill out confusing forms)?
• Did we help a faculty member concentrate their efforts toward their student’s learning (i.e., rather than hours spent on processing administrative paperwork)?
• Did we enhance a student’s overall experience (i.e., rather than stressing them with red tape, hoops, and waiting in lines)?
• Did we help an administrator focus on their most mission critical work (i.e., rather than getting bounced around from desk to desk to get an issue resolved)?
We should know we are helping to support the mission of the university.
How would a leader know if he/she is meeting these needs?
ITS ALL ABOUT
THE DATA
Psychometrics – getting good data
• Not
METRICS
Psychometrics – getting good data from people (because we cant read their minds!)
Psychometrics is a field of study concerned with the theory and technique of psychological measurement.
Construct and validate assessments instruments (questionnaires, tests, personality tests)
Statistical measurement theory
Psychometric research involves two major tasks:
1) construct instruments that are valid2) develop procedures to measure
Methodology – Design Customer Satisfaction Survey
Standard 8 rating questions for all services with up to 5 customized questions per service area1. Overall satisfaction2. Understands my needs3. Accessible4. Responsive5. Resolves issues6. Knowledgeable/professional/courteous, etc.7. Effective use of Blink (info sharing website)8. Moving in positive direction
Stop, save, and finish laterConfidential responsesAll staff and faculty invitedMessage: Help us help you fulfill the mission of the university!
Methodology – Design Staff@Work Survey
Tested for internal reliability, conducted Factor Analysis53 questions measure 4 dimensions:
1. UC San Diego overall2. Department effectiveness (diversity, mission)3. Supervisor effectiveness4. Employee effectiveness
Equity, diversity, and inclusion questions for comparisonsMessage:
• Are you a satisfied UCSD employee?• Would you recommend others to work here?
Regressions analysis to predict what drives this outcome
Participation rate: 100% for some areas, 56% overall
“Yes, it really is Anonymous”
Annual Campus Survey OverviewStaff@Work
(ie employee engagement)
• 18 years of data
• This is a campus wide anonymous survey for staff invited by the Chancellor
• Eight VC areas - 450 units participate
• 824 Verbatim Comments
• Who made a difference this year?
• 56% responded campus wide. Participation within departments reach up to 100%
Student Customer Satisfaction
• 20 years of data
• Departments voluntarily opt-in to be rated
• 44 student service units/programs currently evaluated
• 20,351 Verbatim Comments
• One “burning question”
• 4884 undergraduate and graduate students participated
Faculty and Staff Customer Satisfaction
• 21 years of data
• Departments voluntarily opt-in to be rated
• 80 service units/programscurrently evaluated
• 3,773 Verbatim comments and suggestions for improvement
• One “burning question”
• 36% responded with varying participation per rated unit
Congratulations! Here are your survey results. Good luck!
Analytics? Sure! Here you go.
Not cool.
ITS ALL ABOUT making meaning of
THE DATA
Customer Satisfaction Survey Reports: Who, What, Where, When, Why
Trend analysis: “When” over time Heat maps for easy identification of “Where” should we dig deeper
“What” are the basic descriptive statistics
Correlational analysis to identify drivers of satisfaction and start the conversation of “Why” are the scores varying
Drill downs to know which departments to focus your outreach or study best practices: “Who” needs attention
Customer Satisfaction Survey heat map quickly identify strengths and opportunities
Staff@Work: A picture can say a thousand words…
Arrows indicate positive or negative movement and statistical analysis inform you of significant trends
Descriptive statistics
Impact analysis:These are the items where people are saying, “I am not as happy about these things and they are also very important drivers of my satisfaction here”
Interpreting the impact analysis report
Correlation means relationship. The closer to 1.0, the stronger the relationship between satisfaction and the survey item
“I am not as happy about these things AND they are also things that are important to me in impacting my level of satisfaction here”
“I am not as happy about these things BUT they are NOT things that are as important to me in impacting my level of satisfaction here”
This quadrant shows the survey items that are rated high (above the mean score of 3.73) and also found to be highly related to satisfaction (above the mean correlation of .42). “I am happy about these things AND they are important to me in impacting my satisfaction here.”
Keep it up! Don’t change anything All is well.
Focus on these things to address!
Keep an eye on these if they move intoPrimary Opps
“I am happy about these things BUT they are NOT things that are as important to me in impacting my level of satisfaction here”
Keep it up! But no need to spend too much effort here.
Now what?
ITS ALL ABOUT what you do with
THE DATA
The Survey Accountability Loop
Step IDeploy survey and obtain
feedback (ratings and comments)
(Oct)
Step II Identify themes and
opportunities(Dec)
Step IIIOpportunities to Action:
Develop and implement action plans and set goals (share with
team and Sr. Leadership)(Jan – Feb)
Step IVFollow up and assess
performance and impact of action plans(Feb-May)
Step VDid changes result in goal
attainment?Communicate impact and share results with Senior Leadership
(June)
Step VIRealign with strategic goals
and/or course correct(Jul – Sept)
IDENTIFICATION OF NEEDS AND PROGRAM
PRIORITIES
RESPONSE PLANNING, GOAL SETTING
PROJECT AND PROGRAM IMPLEMENTATION
EVALUATION AND RECOGNITION FOR
ACTIONS TAKEN
REDEFINE SURVEY QUESTIONS
Staff@Work Case Study: Health Sciences Development: We Heard You
Risk Scenario: Based on FY13 Staff@Work survey results, Health Sciences
Development identified 12 opportunities for improvement
Data-Driven Action Taken: Sr. Leadership Team developed and executed
comprehensive strategies to address each opportunity under the “We Heard You”
campaign
Outcome: FY14 survey results revealed significant score improvements AND the
largest fundraising year for Health Sciences in the history of UC San Diego!
Case Study: We Heard You – Areas of Opportunity
Case Study: We Heard You – Data-Driven Action
The Senior Leadership Team worked together to develop comprehensive strategies that addressed each of the identified areas.
These strategies were branded under the theme “We Heard You.”
Over the past year, the Senior Leadership Team executed each strategy while periodically reminding the entire department that “We Heard You.”
Case Study: We Heard You - Outcomes
Case Study: We Heard You - Outcomes
Case Study: We Heard You - Outcomes
Case Study: We Heard You - Outcomes
• Quote from the Sr. Executive Director of Health Sciences Development:
“While we know our work is not complete, the Staff@Work survey has provided a roadmap to help guide our entire team toward improved results, least of which is accomplishing the largest fundraising year for Health Sciences in the history of UC San Diego.”
ITS ALL ABOUT usingTHE DATA to make a
positive impact
A few examples of actions taken as a result of staff and customer survey data
Creation of a Professional Development and Training Program in Business and Financial Services which has resulted in career advancement and salary increases of an average or 21% for participants, 8 graduate level degrees and 20 professional certifications
Dining enhanced their menu choices to include healthier and vegan options
Housing improved lounge and shared living spaces in response to student feedback
Facilities Management instituted a client response system to more quickly address customer requests
Transportation offered specific commuting alternatives per the feedback received
Campus Shuttle brought back a shuttle route after hearing the feedback from customers
Equipment Management created a new inventory process to alleviate the burden on departments and resulted in successful inventory of approximately 80 campus buildings and 6,000 pieces of equipment with minimal intrusion into research or operational processes
Procurements created a Department Outreach program to address the specific needs of targeted customers
Career Services Center updated their Port Triton system to make their search feature more user friendly for students seeking quality internships
The BFS STRIVE Leadership Development Program was created and implemented to support development of high-potential employees in the department through mentorship. The program increased diversity, spurred career growth, encouraged professional development and fostered mentorships for the participants. It is identified as a University “best practice” in succession planning
For more examples, quotes from leaders, and impact, visit http://blink.ucsd.edu/sponsor/OSI/opa/index.html.
ITS ALL ABOUT THE DATAso you can make meaningful
interpretations of what is important to people based on their attitudes and feelings and take actions to make a positive impact the mission of your
university
Back to our collaboration with CSU…
The timeline for launching CSU Chancellor’s Office Business and Finance Staff@Work Survey
Sponsor meeting (CSU and
UCSD)
Project approval, internal
communications rollout (CSU)
Working session
and official kickoff
(CSU and UCSD)
Develop hierarchy, email lists,
marketing (CSU)Deploy survey
(UCSD)Analyze results
(UCSD) UCSD Delivers results to CSU
Build customized survey questions and application,
testing (UCSD)
6/15/156 weeks 4 weeks4 hours 4 hours8 weeks
Approximate Pre-Launch Time: 8-10 weeks
8/25Survey announcement
8/31 survey open
10/2 survey closed
10/15/15 initial results
11/15/15 final reports
4 weeks 2 weeks
4/15/15
CSU Chancellor’s Office plan
• Deploy the Customer Satisfaction Survey to customers in the Chancellor’s Office and 23 CSU campuses
• Identify metrics for the other two perspectives of the BSC• Financial/Stakeholder• Internal Business Processes
• Now have benchmark capability with UCSD on the same survey questions year over year to collaborate and partner on best practices
Appendix: Example screenshots of the
Staff@Work Survey and Faculty and Staff Customer Satisfaction Survey
Some screenshots of the Staff@Work Survey Instrument
English or Spanish
Welcome Page
Know what you are rating & where you are in the survey
Your department name pops up if you hover over
Anonymous comments
Highlight a colleague or leader that’s made a
difference!
Conduct and behavioral questions for VC Equity, Diversity, Inclusion initiatives
Choose to “opt out” from further reminders
Thank you gift for participation!
Some screenshots of the Faculty and Staff Survey Instrument
See slide 4 and 5 for snippets of the FAQ and Dept Descriptions
Let us know if you would like this to be a specific person or your general mailbox
This is the coupon that everyone receives for a gift item at the Bookstore. Each coupon is barcoded and unique
You can click out of the survey to our webpage to see recent Actions Taken and other information
Once you submit your survey, you cannot change responses. You are automatically returned to this page. You can also come back here to print you coupon.
This is the promotional banner that will be posted in three different locations on campus
Email to our VC contacts and departments in preparation for the launch…
The communication “kit” we provided departments to help encourage response participation
Note: reminders are only sent to people who have notsubmitted their survey