Boston | Geneva | San Francisco | Seattle | Washington FSG.ORG

Why Shared Measurement Matters

Srik Gopalakrishnan, FSG

April 2013

2

FSG.ORG

© 2013 FSG

About FSG and Shared Measurement

• Nonprofit consulting and research firm founded in 2000

• Staff of 120 in Boston, San Francisco, Seattle, Washington DC, Geneva

• Success in strategy and evaluation with over 400 foundations, corporations, and nonprofits

• Thought leader: articles published in HBR, SSIR, American Journal of Evaluation including Collective Impact (SSIR 2010)

• Breakthroughs in Shared Measurement and Social Impact (funded by Hewlett Foundation) published in 2009: examined 20 approaches to performance, outcome, impact measurement

FSG is driven by the same passion that drives our clients: a passion for greater social impact using data as a key lever

3

FSG.ORG

© 2013 FSG

The Premise: There Are Several Types of Problems

Source: Adapted from “Getting to Maybe”

Simple Complicated Complex

Right “recipe” essentialGives same results every time

Example: Baking a cake

“Formulas” neededExperience built over time and can be repeated with success

Example: Sending a rocket to the moon

No “right” recipes or protocols Outside factors influence

Experience helps, but doesn’t guarantees success

Example: Raising a child

Traditional approach in social sector has been to treat problems as simple or complicated

4

FSG.ORG

© 2013 FSG

Traditional Ways of Approaching Social Change Are Not Working to Address Our Toughest Challenges

Current approaches not conducive to complex problems:

• Funders select individual grantees that offer the most promising solutions

• Grantees work separately, compete to produce the greatest independent impact

• Evaluation attempts to isolate a particular grantee’s impact

• Large scale change assumed to depend on scaling a single organization

• Corporate, government, philanthropy and nonprofit sectors often disconnected from efforts of foundations and nonprofits

IsolatedImpact

5

FSG.ORG

© 2013 FSG

Imagine a Different Approach

Shift mindset from a “technical” to an “adaptive” approach:

• Understand that social problems – and their solutions – arise from interaction of many organizations within larger system

• Large scale impact depends on increasing cross-sector alignment and learning among many organizations

• Nonprofits, government, philanthropy and corporates actively coordinate their action and share lessons learned

• Progress depends on working toward the same goal and measuring the same things – SHARED MEASUREMENT

IsolatedImpact

CollectiveImpact

6

FSG.ORG

© 2013 FSG

Shared Measurement Is a Critical Piece of Pursuing a Collective Impact Approach

Identifying meaningful common metrics for tracking progress toward a common agenda across organizations, and providing scalable platforms to

share data, discuss learnings, and improve strategy and action

1. Tracking Progress Toward a Shared Goal

2. Enabling Coordination and Collaboration

3. Learning and Course Correction

4. Catalyzing Action in the Field

Definition

Benefits of Using Shared Measurement

7

FSG.ORG

© 2013 FSG

Developmental Phases in Creating a Shared Measurement System

Design Develop Deploy1 2 3

• Shared vision for the system and its relation to broader goals, theory of change or roadmap

• View of current state of knowledge and data

• Governance and organization for structured participation

• Identification of metrics, data collection approach, including confidentiality/ transparency

• Development of data collection tools and technology platform

• Refinement and testing of platform and tools

• Staffing for data management and synthesis

• Learning forums and continuous improvement

• Ongoing infrastructure support

• Improve system based on a pilot, review, refinement, and ongoing evaluation of usability and impact

8

FSG.ORG

© 2013 FSG

Examples of Shared Measurement in Use

9 © 2013 FSG

FSG.ORG

STEM Teacher Experience DRAFT

Excellence Outcomes (quality of teachers)

Candidates demonstrate:• Academic

proficiency• Leadership

potential• Appropriate

mindset to succeed in STEM classrooms and school environments*

Candidates demonstrate basic:• Instructional

practice• STEM content

knowledge• Pedagogical

content knowledge

• Preparedness for the context in which they will teach

Qualified STEM teachers are: • Placed/hired• Prepared for

the context in which they are placed/hired

New STEM teachers demonstrate improved STEM-related:• Instructional

practice• Pedagogical

content knowledge

• STEM Content knowledge

• Student engagement

STEM teachers demonstrate improved STEM-related:• Instructional

practice• Pedagogical

content knowledge

• STEM Content knowledge

• Student engagement

• Student achievement

STEM teachers:• Take on

instructional leadership roles

• Move into administrative roles

• Engage more deeply with community stakeholders

• Influence peers and colleagues

* Mindset persists throughout STEM teaching career

Shared Measures for 100Kin10 Partner Organizations

10 © 2013 FSG

FSG.ORG

Shared Measures for Portland Metro STEM Partnership

11 © 2013 FSG

FSG.ORG

Ways Network Partners Use Shared Measurement System

To Learn with and from Peer Programs and

Organizations

To Inform Internal Decision-making About

Practices

To Identify Practices of High-performing

Organizations

To Identify Other Relevant Partners

To Uncover Areas Where Further Data or Research Is Needed

12

FSG.ORG

© 2013 FSG

Challenges in Developing and Implementing Shared Measurement Systems

Difficulty in coming to agreement on common outcomes and indicators

Concerns about relative performance / comparative measurement across providers working in the same space

Limited capacity (time and skill) for measurement and data analysis within participating organizations

Aligning funders to ask for the common measures as part of their reporting requirements

Time and cost of developing and maintaining a system, both for human capital and technology

Challenges

13

FSG.ORG

© 2013 FSG

Critical Factors in the Development of Shared Measurement Systems Help Overcome These Challenges

Source: Breakthroughs in Shared Measurement and Social Impact, FSG, 2009

Effective Funding Relationships

Strong leadership and substantial funding throughout multi-year development period

Independence from funders in devising indicators and managing the system

Broad engagement in design process by many organizations in field, with clear expectations about confidentiality/ transparency

Voluntary participation open to all relevant organizations

Broad and Open Engagement

Effective use of technology

Ongoing staffing to provide training, facilitation, and to review accuracy of data

Testing and continually improving system through user feedback

Facilitated process for participants to gather periodically to share results, learn from each other, and coordinate efforts

Infrastructure for Deployment

Pathways for Learning and Improvement

14

FSG.ORG

© 2013 FSG

FSG Lessons Learned in Implementing Shared Measurement Systems

Proceed Iteratively

Focus on Learning and Use

Don’t Underestimate the Value of Partner Engagement

Ensure Long-term Funding and Sustainability

Address Both Technical and Political Challenges

A well designed and structured Shared Measurement effort will pay dividends for years to come