Date post: | 04-Jan-2016 |
Category: |
Documents |
Upload: | meagan-hart |
View: | 218 times |
Download: | 0 times |
MEPA Qualitative Tools
"Not everything that counts can be counted."
SESSION OBJECTIVES• At the end of the session, the participants
are expected to:a.Discuss the basics and nuances of using
qualitative tools in MEPA;b.Explain how to prepare and use appropriately
tools such as: Interview (KII & IDI), Most Significance Change (MSC) Story Technique MSC, Participatory Rapid Appraisal (PRA), and Focus Group Discussion (FGD);
c. Develop deeper understanding on the importance of .both quantitative and qualitative tools in preparing Completed Staff Work (CSW) for MEPA in order to provide story to quantitative data.
ACTIVITY: Self-Assessment
No.
Qualitative ToolsI have adequate knowledge and have already used:
Pre-test Post-test
1 Key Informant Interview (KII) 1 2 3 1 2 3
2 In-depth Interview (IDI)
3 Most Significance Change (MSC)
4 Focus Group Discussion (FGD)
5 Participatory Rapid Appraisal (PRA)
Total Score
Level of Competence
Directions: Assess how knowledgeable and competent you are in using qualitative tools in the preparation of CSW for MEPA, using the following rating scale:
1 – Developing 3 – Advanced2 - Maturing
How is your pre-test result?
Directions: 1.Sum-up the total score and divide it by 5.2. Determine your level of competence based on the table below:
Range Descriptive Rating
Interpretation
1.0-1.49 Developing You are still in the process of acquiring the skill; hence, you need Technical Assistance.
1.50-2.49 Maturing You have acquired the skill to a larger extent, but you need to enhance it further.
2.50-3.0 Advanced Congratulations! You have fully acquired the skill to a largest extent, you are very much ready to prepare CSW for MEPA.
ANALYSIS
1. How did you find the activity?2. What is your level of competence in
terms of using qualitative tools?3. What facilitating/hindering factors
contributed for having such level?4. Which specific tool do you need
more TA?5. Why do you need to have high level
of competence in all these qualitative tools?
Qualitative Monitoring
• Data-gathering techniques that
are focused on the significance
of observations made in a study
rather than the raw numbers
themselves.
Quantitative: numbers breadth generalizabilityQualitative: words depth specific
Remember, "Not everything that counts can be counted."
4 (Quantity) Happy (Quality) Kids
Quantitative methods vs Qualitative methods
Quantitative Qualitative
Surveys Focus groups
Tests Unstructured interviews and observations
Questionnaires Rapid Appraisal
Mixing Quantitative Methods & Qualitative Methods
•Both quantitative and qualitative methods have their strengths and weaknesses. •What qualitative methods lose on reliability they gain in terms of validity. They provide a more in depth and rich description of change and learning. • Quantitative methods are reliable and can be projected to a large population but are weak in terms of describing actual experience.• Combining quantitative and qualitative research provides a more holistic picture of outcomes and impacts achieved.
Mixing Quantitative Methods & Qualitative Methods
Quantitative Monitoring Mid Term Results
Region A
Physical Outputs Accomplishment 30%
Financial Funds Utilization: 18%
Funds Liquidated: 5%
Qualitative Monitoring Mid Term ResultsRegion ADuring the Orientation on DO 62 I was full of questions. Why do we have to give so much emphasis on IP education when they do not show interest in going to school. Going to the immersion in Mindoro was an eye opener, here the IPs were both teacher and student. They learned mainstream concepts together with their own indigenous knowledge. The river, land and air were all part of the learning environment. After the immersion, the Region decided to conduct a second round of orientations on DO 62. With the benefit of our experience we understood the principles behind the order. We also decided to repeat the proposal preparation training to emphasize community participation. As a result, although late the second round of proposals submissions were much better than the first.
Mixing Quantitative Methods & Qualitative Methods
Use QUANTITATIVE Research To:
•find whether there is consensus on a particular issue
•projects results to a larger population
•test specific hypotheses and examine specific relationships
Use QUALITATIVE Research To:• look for a range of ideas and feelings about something•understand different perspectives between groups and categories of people•uncover underlying motivations and factors that influence decision making and opinions
Qualitative Data Gathering
• Qualitative research takes place in the real world, as opposed to the laboratory, and deals with how people give meaning to their own experience.
• These often involve face-to-face interactions between researcher and participant
• The researchers need to be flexible and sensitive to the needs of the social context within which the data is obtained.
Qualitative Data Analysis
• Then it is followed by an attempt to interpret the behaviour and the meanings that people have given to their experience.
• The data is then analyzed and interpreted. To look for themes rather than than trying to confirm a hypothesis.
• The objective of qualitative research is to describe and possibly explain events and experiences.
Participatory Rapid Appraisal (PRA)
Source: Cana & Tucit (2012). Participatory Rapid Appraisal. Retrieved on Dec. 7, 2014 from http://www.slideshare.net/ronelcana/participatory-rapid-appraisal
Participatory Rapid Appraisal (PRA)
Source: Cana & Tucit (2012). Participatory Rapid Appraisal. Retrieved on Dec. 7, 2014 from http://www.slideshare.net/ronelcana/participatory-rapid-appraisal
Participatory Rapid Appraisal (PRA)
Source: Cana & Tucit (2012). Participatory Rapid Appraisal. Retrieved on Dec. 7, 2014 from http://www.slideshare.net/ronelcana/participatory-rapid-appraisal
Participatory Rapid Appraisal (PRA)
Source: Cana & Tucit (2012). Participatory Rapid Appraisal. Retrieved on Dec. 7, 2014 from http://www.slideshare.net/ronelcana/participatory-rapid-appraisal
Participatory Rapid Appraisal (PRA)
Source: Cana & Tucit (2012). Participatory Rapid Appraisal. Retrieved on Dec. 7, 2014 from http://www.slideshare.net/ronelcana/participatory-rapid-appraisal
Participatory Rapid Appraisal (PRA)
Source: Cana & Tucit (2012). Participatory Rapid Appraisal. Retrieved on Dec. 7, 2014 from http://www.slideshare.net/ronelcana/participatory-rapid-appraisal
Participatory Rapid Appraisal (PRA)
Source: Cana & Tucit (2012). Participatory Rapid Appraisal. Retrieved on Dec. 7, 2014 from http://www.slideshare.net/ronelcana/participatory-rapid-appraisal
Participatory Rapid Appraisal (PRA)
Source: Cana & Tucit (2012). Participatory Rapid Appraisal. Retrieved on Dec. 7, 2014 from http://www.slideshare.net/ronelcana/participatory-rapid-appraisal
Participatory Rapid Appraisal (PRA)
Source: Cana & Tucit (2012). Participatory Rapid Appraisal. Retrieved on Dec. 7, 2014 from http://www.slideshare.net/ronelcana/participatory-rapid-appraisal
THE KEYS TO KEY INFORMANT INTERVIEW (KII)
• Who is a Key Informant? - Someone who can ‘unlock’ key
information for you. • What is KII?- Discussion between an individual
respondent and an interviewer which can be done via telephone or face-to-face ranging from informal to highly structured chats.
Source: Performance M&E, USAID Center for Dev’t Info. And Eval; http://pdf.dec.org/pdf _docs/PNABS541.PDF
THE KEYS TO KEY INFORMANT INTERVIEW (KII)
• Purpose of KII?- Collect qualitative, in-depth information
from a wide range of people about a pressing issue to understand beliefs, motivations and sensitive topics
• Types of KII:- Informal, conversational interview- General, guided interview- Standardized, open-ended interview- Closed, fixed-response interview
Source: Performance M&E, USAID Center for Dev’t Info. And Eval; http://pdf.dec.org/pdf _docs/PNABS541.PDF
THE KEYS TO KEY INFORMANT INTERVIEW (KII)
• Key to Success?- Diversity is important.- If saturation is reached, it’s time to stop.- Interviewer skills- Location: private, no distractions and non-
threatening• Tips for Writing Questions:- Open-ended, neutral, one at a time, clear,
use probing questions (can you give me examples, what changes have you noticed)
Source: Performance M&E, USAID Center for Dev’t Info. And Eval; http://pdf.dec.org/pdf _docs/PNABS541.PDF
THE KEYS TO KEY INFORMANT INTERVIEW (KII)
• Analyzing the Data:- Create interview summary sheet- Descriptive codes: themes, concepts,
questions and/or ideas
• KII is perfect when:- Descriptive information is needed- Subject matter is complex.- Respondents are of high status.- Subject matter is highly sensitive.
Source: Performance M&E, USAID Center for Dev’t Info. And Eval; http://pdf.dec.org/pdf _docs/PNABS541.PDF
IN-DEPTH INTERVIEW (IDI)
• What is IDI?• Questions are based upon the
responses collected• Objectives of IDI:- Discover the interviewee’s own
framework of meanings- Obtain rich, contextualized, in-depth
information
Source: Crystal Graphics, Inc. (2014) Retrieved on Dec. 7, 2014 from http://www.powershow.com/view3/50df42-MTE1M/Observations_and_in-
depth_interviews_powerpoint_ppt_presentation
IN-DEPTH INTERVIEW (IDI)
• IDI Guide Questionnaire- List of core questions: Topics- Variations: Order, terminology (adapt to
the terminology of the respondent)- Additional questions may be introduced
• Types of Questions for IDI:- Behavior, opinion/belief, feelings,
knowledge, sensory, background
Source: Crystal Graphics, Inc. (2014) Retrieved on Dec. 7, 2014 from http://www.powershow.com/view3/50df42-MTE1M/Observations_and_in-depth_interviews_powerpoint_ppt_presentation
IN-DEPTH INTERVIEW (IDI)
• Qualities of Questions in IDI- Open-ended, neutral, sensitive, clear to the
interviewee
• Practical Aspects of IDI:- Number of interviewees, status of informant,
recording process, dynamics of interview, leading role of informant, questions, sequential interviews, probing, relationship
Source: Crystal Graphics, Inc. (2014) Retrieved on Dec. 7, 2014 from http://www.powershow.com/view3/50df42-MTE1M/Observations_and_in-depth_interviews_powerpoint_ppt_presentation
FOCUS GROUP DISCUSSION
• What is FGD?- Involves a whole group answering
questions together (Bailey, 1994: 192).- ‘Guided group discussions’ designed to
provide information on a certain topic from a certain population (Ward et al., 1991: 267).
- Are groups of unrelated individuals that are formed by a researcher and then led in group discussion of a topic (Schutt, 1999: 308).
FOCUS GROUP DISCUSSION
• When to use FGD?- To familiarize a research topic- To polish research objectives- To deepen understanding of survey
data- To validate survey data- When interested in capturing diverse
and ‘detailed’ views and/or opinions
FOCUS GROUP DISCUSSION
• Characteristics of FGD- Most focus groups involve 7 to 10
people- May be semi-structured or
unstructured- Discussion is guided/facilitated by an
FGD facilitator/leader- FGD proceedings are
recorded/documented by an FGD documenter
FOCUS GROUP DISCUSSION
• Pre-FGD Activities- Defining the objectives- Construction of the FGD guide especially for
semi-structured FGD- Familiarizing the FGD guide/questions - Preparation of recording devices/materials- Identifying and inviting FGD participants- Confirming the attendance of FGD
participants
FOCUS GROUP DISCUSSION
• Stages in the conduct of FGD
I. Orientation of the FGD participants—setting discussion rules
II.Getting to know (especially if participants are unrelated)
III.Discussion properIV.Synthesizing and closing the discussionV.Distribution of tokens/snacks
FOCUS GROUP DISCUSSION
• Post-FGD Activities- Checking the
documentation/recorded data- Transcribing the recorded
discussion- Encoding the finalized
transcriptions- Data Analysis
MOST SIGNIFICANCE CHANGE (MSC)
• Most Significant Change (MSC) involves the collection and systematic participatory interpretation of stories of significant change.
• Unlike conventional approaches to monitoring, MSC does not employ quantitative indicators, but is a qualitative approach.
MOST SIGNIFICANCE CHANGE (MSC)
• It is a form of participatory monitoring and evaluation.
• It is participatory because many project stakeholders are involved both in deciding the sorts of change to be recorded and in analyzing the data.
HISTORY OF MSC
• The MSC approach was originally developed by Rick Davies through his work with a participatory rural development project in Bangladesh in 1994.
• It has since been adapted and widely promoted in many developing countries.
HISTORY OF MSC
• It is a form of monitoring because it occurs throughout the program cycle and provides information to help people manage the program.
• It contributes to evaluation because it provides data on impact and outcomes that can be used to help assess the performance of the program as a whole.
MSC AS M&E TOOL
• It is a form of monitoring because it occurs throughout the program cycle and provides information to help people manage the program.
• It contributes to evaluation because it provides data on impact and outcomes that can be used to help assess the performance of the program as a whole.
MSC PROCESS
• Essentially, the process involves:
• the collection of significant change (SC) stories emanating from the field level, and
• the systematic selection of the most significant of these stories by panels of designated stakeholders or staff.
MSC QUESTION
• SC stories are collected from those most directly involved, such as participants and field staff. The stories are collected by asking a simple question such as:
‘During the last quarter, in your opinion, what was the most significant change that took place for participants in the program?’
MSC PROCESS• The designated staff and stakeholders are
initially involved by ‘searching’ for project benefits.
• Once changes have been captured, various people sit down together, read the stories aloud and have regular and often in-depth discussions about the value of these reported changes.
• When the technique is implemented successfully, whole teams of people begin to focus their attention on program impact.
WHY STORIES?
• Storytelling is the vivid description of ideals, beliefs, personal experiences, and life-lessons through stories or narratives that evoke powerful emotions and insights.
It enables expression of emotional aspects, factual content, and tacit knowledge.
It increases the potential of meaningful knowledge-sharing.
It augments the likelihood that learning will take place and be passed on.
Stories are the creative conversion of life itself into a more powerful, clearer, more meaningful experience. They are the currency of human contact. — Robert McKee
Everyone is necessarily
the hero of his own life story.
JOHN BARTH
If you don’t know the trees you may be lost in the forest, but if you don’t
know the stories you may be lost in life.
— Siberian Elder
There is no agony like
bearing an untold story inside of you.
- Maya Angelou
STEPS IN IMPLEMENTING MSC TECHNIQUE
1. Raise interest in MSC2. Defining the domains of change 3. Determine the process and
responsibilities4. Collect significant change stories5. Select the most significant of the stories6. Verify stories7. Analyze the top stories for program
learning
Sample of a Promising Story
I had my first PRIME experience when I attended the immersion at the IP Center on February 10, 2012.
In my encounter with the IP students, teachers, folks and the community, I had developed in myself pride of one’s culture, love and care of one’s identity with rich tradition and diverse cultural heritage. It was an eye-opener to me that no one could certainly preserve one’s culture other than ourselves.
Sample MSC Questionnaire
Sample 1: MSC Form for Teachers
Sample 2: MSC Form on Drop-outs for School Head
Sample 3: MSC Template on New Systems, Technology
MSC SCATTERGRAMDimensions Stages of M & E
InstitutionalizationPolicy Formulation * *Managing & Monitoring
* * * * * * * * *
Initial Gain: Improving Teacher’s Competence
* * * * * * * *
Technical Assistance * * * * * * * *Innovation * * * * * * *Implementation of the Program
* * * * * * * * * * * *
New Process/ System/ Technology Experience or Encountered
* * * * * * * * * *
Awareness * * * * * * * * * ** *
ProgressM & E
Initial Gain Evaluation
Outcome Evaluation
Impact Evaluatio
nNote: * This Indicator shows that such program component is embedded explicitly / implicitly in the MSC story of the respondents.
APPLICATION: MSC
1. Tell us about your involvement in program/projects addressing dropout in your school/district/division.
2. What is the most significant change/experience of the school/community as a result of the dropout reduction program/project?
3. Can you cite observable changes among the children who have availed of the dropout reduction programs/schemes? Relate a personal account of a particular student/s.
4. How were the parents & community leaders involved in bringing about these changes?
5. Why is this story significant to you? What lessons can we learned from this experience and what could be improved?
Answer the following questions based on your experience: