Learning Across Borders:
A Monitoring, Evaluation, and Learning Framework for The MasterCard Foundation Scholars Program
March 2014
2
Overview of Presentation
The MasterCard Foundation The Scholars Program Monitoring, evaluation, and learning (MEL) framework MEL challenges, approaches, and lessons learned
The MasterCard Foundation Scholars Program
3
4
The MasterCard Foundation Scholars Program: Context and Programmatic Need
Developing countries have achieved a remarkable expansion in educational enrolment, however:
Enrolment in secondary and tertiary education remains low, particularly among poor and rural populations
There are concerns about access to quality and relevant education, even as demand for an educated workforce increases
Many skilled graduates migrate to higher-income countries for better remuneration and working conditions
International scholarship programs too often contribute to brain drain
The MasterCard Foundation Scholars Program
$500 million, 10-year initiative to educate and support 15,000 young people – particularly from Africa – to act as “socially transformative leaders,” driving change and making a positive social impact in their communities.
11,000 secondary level Scholarships, 6,000 tertiary
Target population: youth who have demonstrated academic talent, social consciousness and an orientation toward leadership, and who come from highly disadvantaged socioeconomic backgrounds.
Implemented with and through a global network of 20 NGOs and secondary and tertiary education institutions, who select and directly support the Scholars.
5
6
The MasterCard Foundation Scholars Program: Program Components
Comprehensive scholarships
Transition support
21st-century skill building
Give-back component
Alumni network
The MasterCard Foundation Scholars Program: Partners
7
The MasterCard Foundation Scholars Program MEL Framework
8
Evaluation and Learning in the Scholars Program
Objectives:
Continuous program quality improvement
Evidence of outcomes, impact and best practices…
For our partners
For the Foundation’s strategic planning
For the global community of practitioners
As the basis of a proactive argument for propagating successful Program elements and the overall vision of the Program.
9
Evaluation and Learning Overview
Program-wide evaluation: primary end-point 2016 Implementation study
Impact study
Institutional change study
Longitudinal studies of outcomes and impact Outcome study (subset of Scholar cohorts and comparison groups): continued past 5-year point
Outcome tracking (all cohorts)
Anticipated outcomes (Stay in / return to Africa, Give-back, career and leadership outcomes, citizenship outcomes)
Long-term qualitative work to understand Scholars’ roles in promoting social and economic development
Partner-initiated Research Rapid studies on emergent issues, aimed at providing actionable intelligence.
10
11
The MEL Framework: The Process
The MEL Framework: Theory of Change
12
Approachand Inputs
Activities Outputs ImpactsOutcomes
Inputs
Partner expertise
To Serve Scholars
Academic and other support
Approach
Provide Scholars with a wide range of opportunities
for educational support and enrichment,
give-back, and leadership
development
At Institutions
Revise recruitment procedures
Scholars using academic and other support
services
Partners disseminate
lessons learned
Increased academic
performance
Institutionalized changes foster best practices
New cohort of leaders
supports social
transformation and economic
growth, particularly in
Africa
13
The MEL Framework: Logic Model Components
The MEL Framework: Learning Questions, Methods, and Data Sources
Broad Learning Questions Main Design/Methods Data Sources
Implementation
How was the Scholars Program implemented? Mixed methods
Qualitative data, complemented by quantitative data
Impacts
In particular, what is the impact of the Program on educational, employment, go-back, and give-back outcomes?
Comparison group design• Random assignment• Regression discontinuity• Propensity score matching
Survey data, complemented by qualitative data
Scale-up
Did the Scholars Program become a catalyst for broader change within and beyond partner institutions?
Qualitative analysisQualitative data, complemented by quantitative data
14
Challenges, Solutions, and Lessons Learned
15
16
Challenges, Solutions, and Lessons Learned: Designing a Unified MEL Framework
Challenge Develop a single unified MEL framework to evaluate The MasterCard
Foundation Scholars Program
Solutions Develop the MEL framework from the ground up through a thorough
understanding of partner projects and what they have in common Use a flexible approach that allows for different methods
for each partner: Partner-specific comparison group designs allow for a rigorous evaluation while using different methods to draw the comparison group
Random assignment Regression discontinuity Matched comparison group (via propensity score methods)
Challenges, Solutions, and Lessons Learned: Measurement
Challenge Locating applicants based on application data may be challenging Several key outcomes are difficult to measure Key outcomes, including educational attainment, career and service
outcomes, giving back and going back, occur several years in the future
Solutions To locate hard-to-find applicants, use their own and their schools’
contact information Draw survey items from existing instruments and pilot our
own instruments Establish a plan for tracking The MasterCard Foundation Scholars over
time that is appropriate to the context
17
Challenges, Solutions, and Lessons Learned: Measurement—Baseline Data Collection in Uganda
Sample: 823 applicants spread around Uganda
Applicants were located by calling the phone number provided in their application
This is the first of three rounds of data collection planned
18
92%
8%
Locating applicants
Located
Not located
82%
18%
“Give-back” attitudes:Helping people in need is
something one should do…
for anyone
for friends and family
2% 15%
83%
Contact informationNo phone number1 phone number2 phone numbers
Challenge Incorporate formative evaluation components that will generate
relevant feedback for partners throughout the evaluation Early feedback can then inform mid-course corrections as needed
Solutions Engage with partners while developing MEL activities Collect quantitative and qualitative data at regular intervals Provide partners with feedback from all data collection efforts
19
Challenges, Solutions, and Lessons Learned: Blending Formative, Summative Evaluation Approaches
Phone interviews with 18 graduating tertiary Scholars Opportunity to gain early information on Scholar experiences This early outcome information will support continuous quality
improvement
Scholars offer suggestions for Program improvement Increase flexibility in resource allocation—for example, could leftover
resources for textbooks be used for transportation? Coordinate Scholar attendance at academic conferences
to facilitate networking Provide orientation to the Program in one intensive session early on,
rather than spread over several sessions
20
Challenges, Solutions, and Lessons Learned: Regular, Relevant Feedback to Partners
Challenges, Solutions, and Lessons Learned: Regular, Relevant Feedback to Partners (continued)
Scholars also say what is going well The Program gives Scholars “the flexibility, time
and resources to actually do the work you set out to do”
“[I’m] trying my best to really use all the knowledge they’ve given to me and trying to implement it in myself first and then try to introduce it to society”
The Masters in Development program is a “dream program” that fits well with career goal of reducing poverty in her home country
21
Conclusions
We have designed MEL activities that respond to the complex nature of The MasterCard Foundation Scholars Program with an approach that is
• Flexible, yet rigorous
• Capable of measuring key outcomes now and in the future
• Formative and summative in nature
• Collaborative and inclusive, responding to needs across the partnership
22
Contacts
Barry Burciul [email protected] http://mastercardfdnscholars.org/
Matt Sloan [email protected] http://www.mathematica-mpr.com/CIPRE/Projects/
MasterCard_Scholars_program.asp
23