1 PROJECT MANAGEMENT TRAINING SERIES PROJECT MANAGEMENT TRAINING SERIES Project Monitoring & Evaluation Project Monitoring & Evaluation By By Samuel Obino Mokaya Samuel Obino Mokaya Jomo Kenyatta University of Agriculture & Technology Jomo Kenyatta University of Agriculture & Technology P. O. Box 62000-00200, Nairobi, Kenya P. O. Box 62000-00200, Nairobi, Kenya Tel: 0722845562 / 0734615008 Tel: 0722845562 / 0734615008 E-mail: E-mail: [email protected]/ o’[email protected]/ o’[email protected]September 2009 September 2009
Transcript
11
PROJECT MANAGEMENT TRAINING SERIESPROJECT MANAGEMENT TRAINING SERIES
The most commonly used terms in monitoring and evaluation are: Feedback and Lesson learned
1. Feedback: A process through information and knowledge is disseminated and used to assess overall progress towards results or confirm the achievement of results
2. Lesson learned: An instructive example based on experience that is applicable to a situation – learning from experience.
Why M and EWhy M and E
88
Project MonitoringProject Monitoring Monitoring is the continuous day-to-day Monitoring is the continuous day-to-day
assessment of project implementation in relation assessment of project implementation in relation to agreed schedules, and of the use of inputs, to agreed schedules, and of the use of inputs, infrastructure, and services by project infrastructure, and services by project beneficiaries. beneficiaries.
Provides managers & other stakeholders with Provides managers & other stakeholders with continuous feedback on implementation. continuous feedback on implementation.
Identifies actual or potential successes & Identifies actual or potential successes & problems as early as possible to facilitate timely problems as early as possible to facilitate timely adjustments to project operation.adjustments to project operation.
It is performed while a project is being It is performed while a project is being implemented, with the aim of improving the implemented, with the aim of improving the project design & functioning while in action. project design & functioning while in action.
Designed to provide constant feedback on the Designed to provide constant feedback on the progress of a project, the problems it is facing, progress of a project, the problems it is facing, and the efficiency with which it is being and the efficiency with which it is being implemented.implemented.
Monitoring information is collected at specific Monitoring information is collected at specific times - daily, monthly or quarterly. times - daily, monthly or quarterly.
At some point this information needs to be At some point this information needs to be collated, brought together and analyzed so that it collated, brought together and analyzed so that it can answer questions such as:can answer questions such as:
How well are we doing? How well are we doing? Are we doing the right things? Are we doing the right things? What difference are we making? What difference are we making? Does the approach need to be modified, and if so Does the approach need to be modified, and if so
how?how?
It can indicate as early as possible any It can indicate as early as possible any shortcoming with regard to delivery of inputs, shortcoming with regard to delivery of inputs, execution of activities, or production of outputs in execution of activities, or production of outputs in order that corrective action can be undertaken in order that corrective action can be undertaken in time.time.
In summary monitoring addresses issues such as:In summary monitoring addresses issues such as:• ProcurementProcurement• Delivery of inputsDelivery of inputs• Schedules of activitiesSchedules of activities• Progress made in Production of outputsProgress made in Production of outputs
1111
Principles for MonitoringPrinciples for Monitoring
In terms of scope, monitoring involves:In terms of scope, monitoring involves: Progress towards outcomes.Progress towards outcomes. Factors contributing to or impending achievement of the Factors contributing to or impending achievement of the
outcome.outcome. The organization’s contribution to the outcome through The organization’s contribution to the outcome through
outputs.outputs. The partnership strategy.The partnership strategy.
Monitoring ToolsMonitoring Tools Project managers have a challenge of determining Project managers have a challenge of determining
the correct mix of monitoring tools & approaches the correct mix of monitoring tools & approaches for each project.for each project.
They include; reporting & analysis, validation and They include; reporting & analysis, validation and participation:participation:
Outcome groupsOutcome groupsSteering committeeSteering committeeStakeholder meetingsStakeholder meetingsFocus group Focus group meetingsmeetingsAnnual reviewAnnual review
1414
Project EvaluationProject Evaluation Evaluation is the periodic assessment of a Evaluation is the periodic assessment of a
project's relevance, performance, efficiency, & project's relevance, performance, efficiency, & impact (both expected and unexpected) in relation impact (both expected and unexpected) in relation to stated objectives: to stated objectives:
Studies the outcome of a project i.e. changes in income, Studies the outcome of a project i.e. changes in income, housing quality, benefits distribution, cost-effectiveness, housing quality, benefits distribution, cost-effectiveness, etc.etc.
• with the aim of informing the design of future projects. with the aim of informing the design of future projects. In summary, evaluation is carried out to:In summary, evaluation is carried out to:
Determine the Determine the EFFECTIVESSEFFECTIVESS: Did the project achieve its planned : Did the project achieve its planned purpose?purpose?
Determine Determine SIGNIFICANCESIGNIFICANCE: Did the project make a substantial : Did the project make a substantial contribution to development?contribution to development?
Measure Measure EFFICIENCYEFFICIENCY: Did we achieve a satisfactory cost/benefit : Did we achieve a satisfactory cost/benefit ratio; could we have accomplished our purpose at a lower cost?ratio; could we have accomplished our purpose at a lower cost?
Learn Learn LESSONLESSON: Which can be applied to similar activities : Which can be applied to similar activities elsewhere.elsewhere.
Types of EvaluationTypes of Evaluation
Evaluation ProcessEvaluation Process
Designing Terms of ReferenceDesigning Terms of Reference
Monitoring and EvaluationMonitoring and Evaluation systems can be an systems can be an effective way to:effective way to: Provide constant feedback on the extent to which the Provide constant feedback on the extent to which the
projects are achieving their goals.projects are achieving their goals. Identify potential problems at an early stage & propose Identify potential problems at an early stage & propose
possible solutions.possible solutions. Monitor the accessibility of the project to all sectors of Monitor the accessibility of the project to all sectors of
the target beneficiaries.the target beneficiaries. Monitor the efficiency with which different components Monitor the efficiency with which different components
of the project are being implemented and suggest of the project are being implemented and suggest improvements.improvements.
Evaluate the extent to which the project is able to Evaluate the extent to which the project is able to achieve its general objectives.achieve its general objectives.
Provide guidelines for the planning of future projects.Provide guidelines for the planning of future projects.
1919
Monitoring Vs EvaluationMonitoring Vs EvaluationMonitoringMonitoring EvaluationEvaluation
Keeps track of daily Keeps track of daily activitiesactivities
Accepts rules, policies and Accepts rules, policies and proceduresprocedures
Work towards targetsWork towards targets
Stresses the conversion of Stresses the conversion of inputs to outputsinputs to outputs
Concentrates on planned Concentrates on planned project elementsproject elements
Reports progressReports progress
Takes a long range viewTakes a long range view
Questions pertinence of Questions pertinence of policies, rules and policies, rules and proceduresprocedures
Measures progress and asks Measures progress and asks whether targets are adequatewhether targets are adequate
Emphasizes achievement of Emphasizes achievement of purposepurpose
Assesses planned elements Assesses planned elements & looks for unplanned & looks for unplanned changes, searches for causes changes, searches for causes & challenges assumptions& challenges assumptions
Records lessons learntRecords lessons learnt
2020
Components of Good M & E DesignComponents of Good M & E Design
M & E IndicatorsM & E Indicators Measures of performanceMeasures of performance Signs that something tangible has happenedSigns that something tangible has happened Standards of measurementStandards of measurement They tell us “when we get there”They tell us “when we get there”
Note that the goal of Good Project Management Practice Note that the goal of Good Project Management Practice (GPMP) is to deliver every component of the project to the (GPMP) is to deliver every component of the project to the
required quality, within the agreed budget and on time.required quality, within the agreed budget and on time.
Input indicators Input indicators •Describe the Describe the material, financial, technological and human material, financial, technological and human resources investedresources invested in a product, service, or program. in a product, service, or program. Examples might include staff time, supplies and equipment, Examples might include staff time, supplies and equipment, funds, or in-kind contributions to an interventionfunds, or in-kind contributions to an intervention
Process indicatorsProcess indicators Describe Describe activities carried outactivities carried out to achieve the desired results to achieve the desired results or objectives of an intervention; they show what is done and or objectives of an intervention; they show what is done and how well it is done. Examples of process indicators include how well it is done. Examples of process indicators include training sessions or educational programs prepared and training sessions or educational programs prepared and presented to transfer knowledge or skills.presented to transfer knowledge or skills.
Output indicatorsOutput indicators
Describe the immediate results of an interventionDescribe the immediate results of an intervention•Jobs createdJobs created•No of people trainedNo of people trained•No of condoms dispensedNo of condoms dispensed•No of toilets constructedNo of toilets constructed
Methods of Data CollectionMethods of Data Collection1.1. Interviews with beneficiaries and other stakeholdersInterviews with beneficiaries and other stakeholders2.2. Direct observation of situations / events as they happenDirect observation of situations / events as they happen3.3. Key informant interviews: interviewing leaders or Key informant interviews: interviewing leaders or
specialists in a subject specialists in a subject 4.4. Questionnaires: written questions used to get written Questionnaires: written questions used to get written
responsesresponses5.5. Focus group discussions: a group of about 6 to 12 people Focus group discussions: a group of about 6 to 12 people
interviewed together on a specific subjectinterviewed together on a specific subject6.6. Community meetings: a gathering of a fairly large number Community meetings: a gathering of a fairly large number
of community members (beneficiaries) to whom of community members (beneficiaries) to whom questions, problems and situation are put so as to elicit questions, problems and situation are put so as to elicit their responsestheir responses
7.7. Fieldwork reports: structured report forms that are used Fieldwork reports: structured report forms that are used for recording observations by the field workersfor recording observations by the field workers
Designing an M and E ToolDesigning an M and E Tool Entails formulation of a workable system design Entails formulation of a workable system design
and planning operational and resource and planning operational and resource implications:implications: What are the norms against which to compare M & E What are the norms against which to compare M & E
results to project performance and outcomes?results to project performance and outcomes? What information has to be obtained in order to make What information has to be obtained in order to make
decisions on the project?decisions on the project? How is information to be collected?How is information to be collected? How should the data be analyzed?How should the data be analyzed? To whom should the M & E findings be reported?To whom should the M & E findings be reported? Who will be responsible for carrying out the M & E?Who will be responsible for carrying out the M & E? How much should be invested in M and E?How much should be invested in M and E?
Most common M & E instruments are Most common M & E instruments are questionnaire, interviews, observation and questionnaire, interviews, observation and document analysis.document analysis.
Literature search Gather background information on methods and results of evaluation methods used by others.
Economic and efficient way of obtaining information.
Difficult to assess validity and reliability of secondary data.
Questionnaires /surveys
Oral interviews or written questionnaires of a representative sample of respondents. Most appropriate when need to quickly and/or easily get lots of information from people in a non-threatening way.
* Produce reliable information.* Can be completed anonymously.* Easy to compare and analyze.* Can be administered easily to a large number of people.* Collect a lot of data in an organized manner.* Many sample questionnaires already exist.
* Demanding and could be costly.* Might not get careful feedback.* Wording can bias client's responses.* Data is analyzed for groups and are impersonal.* Surveys may need sampling expert.* Provide numbers but do not get the full story.* Open-ended data may be difficult to analyze.
Interviews To fully understand someone's impressions or experiences, or learn more about their answers to questionnaires.Individual or group interviews could be organized to assess perceptions, views and satisfaction of beneficiaries.
* Give full range and depth of information and yield rich data, details and new insights.* Can be flexible with the client.* Permit face-to-face contact with respondents and provide opportunity to explore topics in depth.* Allow interviewer to probe, explain or help clarify questions, increasing the likelihood of useful responses.* Allow interviewer to be flexible in administering interview to particular individuals or circumstances.
* Can be hard to analyze and compare.* Interviewer can bias responses.* Can be expensive and time-consuming.* Need well-qualified and highly trained interviewers.* Interviewee may distort information through recall errors, selective perceptions and desire to please interviewer. * Flexibility can result in inconsistencies across interviews.* Volume of information too large and may be difficult to reduce data.
Documentation review
Impression of how program operates without interrupting the program by review of applications, finances, memos, minutes, etc.
* Give comprehensive and historical information* Doesn't interrupt program or client's routine in program* Information already exists.* Few biases about information.
* Often takes a lot of time* Information may be incomplete. Quality of documentation might be poor.* Need to be clear about purpose.* Not a flexible means to get data. Data restricted to what already exists.
Observation Involves inspection, field visits and observation to understand processes, infrastructure/services and their utilization.Gathers accurate information about how a program actually operates, particularly about processes.
* Well-suited for understanding processes, views, operations of a program while they are actually occurring.* Can adapt to events as they occur and exist in natural, unstructured and flexible setting.* Provides direct information about behavior of individuals and groups. * Permits evaluator to enter into and understand situation/ context. * Provides good opportunities for identifying unanticipated outcomes.
* Dependent on observer’s understanding and interpretation.* Has limited potential for generalization.* Can be difficult to interpret exhibited behaviors.* Can be complex to categorize observations.* Can influence behavior of program participants.* Can be expensive and time-consuming. * Needs well-qualified, highly trained observers and/or content experts. * Investigator has little control over situation.
Focus groups A focus group brings together a representative group of 8 to 10 people, who are asked a series of questions related to the task at hand. Used for analysis of specific, complex problems, in order to identify attitudes and priorities in sample groups. Explore a topic in depth through group discussion, e.g., about reactions to an experience or suggestion, understanding common complaints, etc.
* Efficient and reasonable in terms of cost. * Stimulate the generation of new ideas. * Quickly and reliably gets common impressions * Can be an efficient way to get a wide range and depth of information in a short time.* Can convey key information about programs.* Useful in project design and in assessing the impact of a project on a given set of stakeholders.
* Can be hard to analyze responses.* Need good facilitators. * Difficult to schedule 8-10 people together.
Case studies In-depth review of one or a small number of selected cases. To fully understand or depict beneficiaries’ experiences in a program, and conduct comprehensive examination through cross comparison of cases.
* Well-suited for understanding processes and for formulating hypotheses to be tested later.* Fully depicts client's experience in program input, process and results.* Powerful means to portray program to outsiders.
* Usually time consuming to collect, organize and describe. * Represents depth of information, rather than breadth.
Key informant interviews
Interviews with persons who are knowledgeable about the community targeted by the project.A key informant is a person (or group) who has unique skills or professional background related to the issue/intervention being evaluated, is knowledgeable about the project participants and/or has access to other information of interest to the evaluator.
* Flexible, in-depth approach. * Easy to implement.* Provides information concerning causes, reasons and/or best approaches from an "insider" point of view. * Advice/feedback increases credibility of study.* May have side benefit to solidify relationships between evaluators, beneficiaries and other stakeholders.
* Risk of biased presentation/interpretation from informants/interviewer.* Time required to select and get commitment may be substantial. * Relationship between evaluator and informants may influence type of data obtained. * Informants may interject own biases and impressions.
Direct measurement
Registration of quantifiable or classifiable data by means of an analytical instrument.
* Precise.* Reliable and often requiring few resources.
Registers only facts, not explanations.
Executive SummaryExecutive Summary•Brief terms of referenceBrief terms of reference•Main findingsMain findings•Conclusions and recommendationsConclusions and recommendations•Lessons learntLessons learnt
IntroductionIntroduction•Project descriptionProject description•Purpose of evaluationPurpose of evaluation•Who requested for the evaluationWho requested for the evaluation
Terms of ReferenceTerms of Reference•Outline of main tasksOutline of main tasks•Duration of assignmentDuration of assignment•Submission of draft and final reportsSubmission of draft and final reports
M and E ReportM and E Report
The M and E TeamThe M and E Team•Names of team membersNames of team members•Qualifications and experiencesQualifications and experiences•Roles to be playedRoles to be played
Methodology UsedMethodology Used•How M and E was carried out – methods and justificationHow M and E was carried out – methods and justification•Period to be taken for each methodology chosenPeriod to be taken for each methodology chosen•Limitations of the methodology usedLimitations of the methodology used•How problems were solved and authenticity of the reportHow problems were solved and authenticity of the report•Unresolved issues and their implicationsUnresolved issues and their implications
Main FindingsMain Findings•List FindingsList Findings•Collate and process dataCollate and process data•Analyze and interpret findingsAnalyze and interpret findings
Conclusions and RecommendationsConclusions and Recommendations•Major conclusions and justificationsMajor conclusions and justifications•Recommendations to improve future projectsRecommendations to improve future projects
Lessons LearntLessons Learnt•What worked well that should be repeated in future projectsWhat worked well that should be repeated in future projects•What did not work well that should be avoided or minimized What did not work well that should be avoided or minimized in future projectsin future projects•Learning experiences to make future projects more Learning experiences to make future projects more successfulsuccessful
AnnexesAnnexes•Detailed terms of referencesDetailed terms of references•Brief resumes for team membersBrief resumes for team members•Photos of sample products / outputsPhotos of sample products / outputs
ConclusionConclusion Both monitoring & evaluation are crucial Both monitoring & evaluation are crucial
components in effective project managementcomponents in effective project management Project management is about information gathered Project management is about information gathered
& required at different stages of the project& required at different stages of the project Expertise & clear understanding of the project is Expertise & clear understanding of the project is
crucialcrucial The focus is a process & system that yields useful The focus is a process & system that yields useful
& reliable information for decision making.& reliable information for decision making.
THANK YOU THANK YOU
Samuel Obino MokayaSamuel Obino Mokaya
Jomo Kenyatta University of Agriculture & Technology/, KENYAJomo Kenyatta University of Agriculture & Technology/, KENYA
![OT Survey I 2 Samuel. English Location of Samuel in Canon Hebrew [ Historical Books] Joshua Joshua Judges Judges Ruth Ruth Samuel Samuel Kings Kings Chronicles.](https://static.documents.pub/doc/80x56/56649dc95503460f94abf8bb/ot-survey-i-2-samuel-english-location-of-samuel-in-canon-hebrew-historical.jpg)
