I M P A C T A S S E S S M E N T P I L O T S T U D Y : M E T H O D O L O G Y A N D P R E L I M I N A R Y
F I N D I N G S
B Y I M R A N O O M E R
Lend-A-Hand India: PLAN100
Background
Current measurement focuses on quantitative measures of student achievement:Student attendanceExamination pass rates Graduation rates
Lend-A-Hand India (LAHI) is developing a robust impact measurement system to track the short-, medium-, and long term
impact on the students’ lives it touches through Plan100.
New measurements will understand behavioral outcomes of its program, such as:• Increases in student confidence• Personal aspirations• Entrepreneurial spirit
Project Objectives
LAHI is at the initial stages of developing an impact measurement system. This study will lay the groundwork for it, and specifically accomplish the following:
• …a set of student outcomes expected from participation in the IBT (Introduction to Basic Technology) ProgramDefine
• …an assessment tool to test hypothesized outcomes with students through one-on-one interviewsDevelop
• …a pilot study testing the assessment tool across a sample of IBT and non-IBT students (N>100)Execute
• …preliminary findings from pilot study assessing areas of high, moderate and low impact ; make recommendations for future impact measurement activity
Review
METHODOLOGY
The Big Picture
Input OutputIndividual
Impact
Institution-Wide
Impact
System-wide
Impact
• An example of what impact assessment aims to measures and achieve
Newly revamped
teacher training program
Well-trained ,
energized and
motivated teachers
More engaged students,
resulting in higher
attendance
Improved scores on
standardized tests
Increase in the population’s
job-readiness as a result of a
higher proportion of
high school graduates
continuing on to tertiary education
Constant Feedback Loop
Impact assessment is a method of gaining constant feedback on the design of your organization’s programs with the aim of refining them in real-time to achieve the results you want
Defining Outcomes and Metrics
The following expected student outcomes (near- and long-term) as a result of participation in the IBT program were identified during the initial phase of the study (slide 1 of 2):
Educational Achievement:
• Improved school attendance• Improved performance in
regular curriculum• Improved performance in 10th
standard examination• Heightened interest in STEM-
related learning (science, technology, engineering, math)
• Heightened interest in pursuing further education
Skills Attainment:
• Writing skills• Organizational skills• Project management skills• Problem-solving skills• Basic understanding of
economics (e.g. costs, demand)• Vocational skills (e.g.
engineering, electric wiring)
Defining Outcomes and Metrics
Behavioral Changes:
• Confidence in self• Increase in self-worth / feeling of self-
reliance• Sense of accomplishment• Pride in work• Reduced stress related to school /
examinations• Increased comfort in asking questions• Increased comfort in working with
groups (including mixed gender groups)• Increase in creativity• Increase in entrepreneurial spirit
Employability / Value to Society:
• Career aspirations• Ability to apply skills outside of school• Aspirations to give back to one’s
community• Employment status after 2 /4 / 6 years• Increase in family involvement and
commitment to children’s education• Increase in community involvement and
commitment to local education institutions
• Improved local economy with better job opportunities and a well-educated community
The pilot study has illuminated preliminary findings to support several expected outcomes across the four defined categories. The analysis, however, will focus on
behavioral outcomes – the least studied outcomes of the IBT program
The Assessment
The assessment begins with each student answering the following four unaided, open-ended questions in written form, anonymously.
The purpose is to achieve: a) an unbiased view of student preferences of school subjects, specifically in favor of or against STEM-related courses, and b) students’ unbiased evaluation of their school and IBT (where applicable)
Q1: What is your favorite subject in school? Why? Q2: What is your least favorite subject in school? Why? Q3: What do you like most about your school*? Why? Q4: What would you change about your school*? Why?
*IBT students were asked specific about the IBT program in Q3 and Q4 to gain a deeper understanding of student perceptions of the program and its benefits
The Assessment
One-one-one interviews with each student and ethnographic observation are the core of the assessment. The interviews combine qualitative and quantitative measures (complete assessment tool can be found in the appendix)
The following is an excerpt from the tool where the students rate the importance of specific skills learned at school and within the IBT program. The purpose is to measure where IBT and non-IBT students differ in their perception of skills attainment, and which specific skillsets emerge as the most significant for students in the IBT program
Q1: Please read the following 14 skill areas. Which FIVE do you feel are the most important skills you are learning at school?
Study skills Creativity Reading/writing Math
Science Communication skills Computer skills Career planning skills
Time mgmt. skills Business skills Organizational skills Problem solving skills
Job-specific skills Teamwork skills
The Assessment
The tool also includes an 11-question self-esteem battery, adapted from the widely accepted self-esteem scale developed by Dr. Morris Rosenberg*. Students rate their level of agreement with each statement (below) on a 4-point scale from strongly agree to strongly disagree.
The questions aim to gain insight into key behavioral changes among IBT students.
I am proud of my projects and homework I enjoy working in groups
I am proud of my marks at school I enjoy working in mixed groups (boys and girls, combined)
I feel positive that I will pass the 10th standard examination
I like to ask questions to my teachers
I am excited about my future I feel that I make my parents / family proud
I feel that I will have great opportunities after I finish school
If there is a problem, I believe I can fix it
I feel that I am gaining many good skills at school
*Source: http://www.bsos.umd.edu/socy/research/rosenberg.htm
Sampling and Administration
During the pilot study, data was collected from 109 students across four secondary schools in rural Maharashtra. Survey counts by gender and category are below:
The surveys were administered in Marathi, the students’ local language with a duration of 5 -10 minutes.
Survey Sample: Impact Assessment Pilot Study
Total sample: 109 students
IBT: 60 students (55%)
Non-IBT (control group): 49 students (45%)
9th Standard: 57 students (52%)
10th Standard: 52 students (48%)
Male: 61 students (56%)
Female: 48 students (44%)
PRELIMINARYFINDINGS
Preliminary Findings
Most important skills learned through school / IBT
Top 5 (in order of frequency) Most Selected Skills
IBT Students Non-IBT Students
Study skills Study skills
Teamwork skills Reading/writing
Problem-solving skills Problem-solving skills
Career-planning skills Science
Business skills Computer skills
IBT students are more likely to prioritize outcomes-focused, non-academic skills such as business skills, teamwork, and career-
planning skills – all core foci of the IBT curriculum
Preliminary Findings
Effect of IBT Program on Importance Placed on Skill Attainment (slide 1 of 2)
Skill Control % Selected
IBT % Selected
% Effect
Career planning skills 20% 43% 112%
Time management skills 27% 18% (-31%)
Business skills 33% 42% 28%
Organizational skills 24% 13% (-46%)
Problem solving skills 42% 43% 1%
Job-specific skills 30% 33% 9%
Teamwork skills 41% 45% 10%
Highlighted skills are those that IBT students chose more often than non-IBT students as being a Top 5 important skill learned at school.
Preliminary Findings
Effect of IBT Program on Importance Placed on Skill Attainment (slide 2 of 2)
Skill Control % Selected
IBT % Selected
% Effect
Study skills 71% 70% (-2%)
Creativity 14% 22% 52%
Reading/Writing 53% 37% (-31%)
Math 41% 38% (-6%)
Science 43% 40% (-7%)
Communication skills 18% 15% (-18%)
Computer skills 43% 38% (-11%)
Negative effect does not indicate reduced impact on specific skill areas; instead, it is a reflection on the prioritization given by students on other skills
Preliminary Findings
By overlaying the self-esteem question set with the proposed outcomes of the IBT program, we identified 6 key behavioral success factors to assess:
Do you want to
start your own
business?
I am proud of
my projects
I am proud of
my grades
I like to ask
questions
I have learned
good skills
I feel positive
that I will pass
I am excited
about my future
I feel like I have many
great opps.
If there is a problem, I can fix it
I like working in groups (and mixed-gender
groups)
Measures and indicators to assess factors
Preliminary Findings
Entrepreneurial Spirit 65.3% 71.7%
Self-Pride / Self-Worth 3.61 3.67
Self-Confidence 3.58 3.66
Personal Aspirations 3.51 3.40
Self-Reliance 3.51 3.58
Interpersonal Skills 3.41 3.58
Sel
f-E
stee
m B
att
ery
+S
elec
t S
urv
ey Q
ue
stio
ns
Outcome Control Score
IBT Score
% Difference
9.7%
1.6%
2.1%
-3.1%
2.1%
4.9%
Fa
cto
rs M
ost
Aff
ect
ed
Am
on
g
IBT
Stu
de
nts
Measured on a 100 ptscale
Preliminary Findings
Areas of High
Impact
• Pride in one’s work• Recognition of one’s
achievements• Entrepreneurial spirit• Gender equality
Areas of Moderate
Impact
• Confidence in test taking
• Confidence in asking questions
• Self-reliance
Areas of Low/No
Impact
• Family pride in child’s accomplishments
• Perception of a bright future
Learnings / Recommendations
• Group work and practicals, “learning by doing,” has had the biggest impact on students’ confidence and interpersonal skills
• Identify how these areas can be leveraged to positively influence students’ long-term view and development
• Self-reliance (the ability to solve problems and think independently) is a core expected outcome of IBT and should be emphasized more during the program; same for asking questions, where many students are still hesitant
• Connect more with parents and families about what IBT is offering their children. This will not only positively affect the families and the students’ self-esteems, but will help in getting families to contribute fees
Preliminary Findings
Student Aspirations for Continuing Education
Q: After 10th standard, what are you interesting in doing?
Control IBT
Go to ITI or another technical school to earn my diploma
18% 22%
Attend junior college (11th and 12th) 78% 74%
Go work with my family 0% 2%
Go work somewhere else 0% 0%
I don’t know 4% 2%
While IBT students are slightly more interested than their peers in pursuing technical education after 10th standard, the majority are still interested in junior college.
Recommendation: LAHI should evaluate its position as a feeder into diploma programs and
how this incentive is being communicated to IBT students and families
WRAP-UP
Key Findings From Pilot Study
Transmitting the Importance of Life
Skills Learning
• IBT students are twice as likely as their peers to view career planning as a fundamental skill learned during their secondary education
• Business, problem-solving, and job-specific skills are also perceived to be of higher importance among IBT students
Inspiring Future Entrepreneurs
• Specifically, an IBT student is 10% more likely to be interested in starting his or her own shop or business
• IBT students’ ideas range from tailoring and welding shops to computer institutes and medical clinics
Facilitating Strong Interpersonal Skills
• IBT students are more likely than their peers to feel comfortable working in groups, particular mixed gender groups (7% effect)
• This is a significant achievement in schools where rote learning and individual achievement are the norms
LAHI has created positive impact for IBT students across a number of measures. In this study, the most notable outcomes are:
Recommendations For Lend-A-Hand India
Questions to consider:
What outcomes exceeded expectations?
Which measures were less impacted by the program than what was expected?
How can LAHI leverage areas of high impact (e.g. entrepreneurial drive) to positively affect areas of moderate (e.g. creativity) and low impact (e.g. family pride)
In areas of high impact…In areas of moderate and
low impact…
Identify drivers (e.g. teaching styles, level of practicals) of
student achievement that can be leveraged towards improving
on other measures
Pinpoint gaps between expected and actual outcomes (e.g. students’
comfort in asking questions) and determine programmatic changes
that can be tested to minimize achievement gaps
Recommendations For Further Impact Work
Be aware of measurement pitfalls*, such as:
Trying to assess a whole program instead of specific components
Aspiring for scientific-level design that is appropriate for large scale populations when striving for generalizable trends but not for context-specific learning where a finer, more customized and qualitative approach often yields richer results
Focusing on uncontrollable , long-term community impact metrics as a measure of a program’s success over tangible, measureable, near-term outcomes
*R&D: A New Form of Evaluation, TCC Group
Impact measurement is not a static, annual process to appease funders; it is a dynamic, continuous effort aimed at understanding, refining and improving
upon an organization’s programs and outcomes
Recommendations For Further Impact Work
Expanding on the pilot study, future steps should include: Communicating expected outcomes and measurement goals to
IBT schools to onboard key stakeholders (e.g. headmaster, instructors) to a more robust assessment system
Tracking outcomes by standard to understand when and how outcomes evolve over program duration
Scaling assessment across all IBT programs with the aim of identifying and replicating star school strategies for maximizing outcomes
Expanding the next phase of the study to include more control variables such as family involvement, family income level, academic achievement, etc.
APPENDIX
Student Verbatims – Positive Reflections on IBT
“I will always have the opportunity to do something because of IBT”
“Because of IBT, we can get jobs anywhere from nursing to engineering”
“I like IBT because we get knowledge of different things”
“I chose IBT because if I don’t get a proper job, I can start my own business”
“We get to learn many different and new things; IBT taught me how to make use of waste”
“I get to know the benefits of IBT because in our future, we can do wiring, prepare chikki, and first aid solutions on our own. IBT is going to help us a lot more in our future.”
“Even if I cannot take further education, I can do something or the other and make a living because of what I’ve learned through IBT”
With the help of IBT, you learn how to solve problems and start a small-scale business in the future. “
“We get to learn many things that can help our family.”
“It’s very beneficial and it’s the only place where girls and boys can work together”
Student Verbatims – Suggestions for IBT
“We should be taught about computers as well”
“Internet should be added as the 5th section of IBT”
“Each student should be required to do each practicals because some don’t do it”
“Practicals should be increased”
“IBT should double its syllabus and newer things should be taught”
“Sometimes materials are in shortage and when tools get broken, it takes long to get them fixed”
“An electronic switchboard should be provided in all four sections”
“There should be more developed machineries for the practicalsso we can learn more things”
IBT Impact Assessment Tool (slide 1 of 4)
Pre-screener questions:
School
Standard
Gender
IBT Status (IBT or Non-IBT)
Self-reported questionnaire:
Q1: What is your favorite subject in school? Why?
Q2: What is your least favorite subject in school? Why?
Q3: What do you like most about your school*? Why?
Q4: What would you change about your school*? Why?
*IBT students were asked specific about the IBT program in Q3 and Q4 to gain a deeper understanding of student perceptions of the program and its benefits
IBT Impact Assessment Tool (slide 2 of 4)
Interview questions: Q5: Please read the following 14 skill areas. Which FIVE do
you feel are the most important skills you are learning at school?
A) Study skills B) Creativity C) Reading/Writing D) Math E) Science F) Communication skills G) Computer skills H) Career planning skills I) Time management skills J) Business skills K) Organizational skills L) Problem solving skills M) Job-specific skills N) Teamwork skills
Q6: After 10th standard, what do you want to do? A) Go to ITI or other school to get a diploma B) Go to Junior College C) Go work with my family D) Go work somewhere else E) I don’t know
IBT Impact Assessment Tool (slide 3 of 4)
Q7: What is your dream job?__________________
Q8: Why?_______________________________
Q9: Do you ever want to start your own shop or business? A) Yes B) No
Q10: What kind of shop or business?________________________________
Q11: Please reach each statement carefully. For each statement, tell me if you: A) Strongly agree B) Agree C) Disagree D) Strongly Disagree
I am proud of my projects and homework
I am proud of my marks at school
I feel positive that I will pass the 10th standard examination
IBT Impact Assessment Tool (slide 4 of 4)
Q11 (continued): Please reach each statement carefully. For each statement, tell me if you: A) Strongly agree B) Agree C) Disagree D) Strongly Disagree
I am excited about my future
I feel that I can have great opportunities after I finish school
I feel that I have gained many good skills at school
I enjoy working in groups
I enjoy working with mixed groups (boys and girls)
I like to ask questions to my teachers
I feel that I make my parents / family proud
If there is a problem, I believe I can fix it