DISCUSSION PAPER

PREPARED FOR THE WHO’S DEPARTMENT OF HUMAN RESOURCES FOR

HEALTH

ON BEHALF OF THE UN TASK FORCE ON IMPACT ASSESSMENT OF FELLOWSHIPS

  BY ARIE ROTEM WITH THE SUPPORT OF MICHAEL

A. ZINOVIEFF

Impact Assessment of United Nations System Fellowship Programmes:

An Evaluation Framework

Fellowships in the United Nations System

“A Fellowship in the United Nations System is a specially tailored or selected training activity that provides a monetary grant to a qualified individual for the purpose of fulfilling special learning objectives; such training, ......., should be in response to nationally approved human resources policies and plans, and should aim at impact and relevance for all stakeholders involved.”

Observations concerning fellowship programs

Variability in design and implementation Large investment of efforts and resourcesInsufficient evidence to support

accountability Need to ascertain contribution beyond

benefit to individuals Need to coordinate approaches Potential for strengthening fellowship system

Evaluation assumptions

Evaluation aims to support decision making

An integral part of all phases of planning and implementation

Main purpose to improve rather than to prove

Engagement of stakeholders essential for success

Key Challenges

Conceptual logic: Why do we expect fellowship program to contribute?

Attribution: What else may have caused observed outcomes?

Fidelity: Did we follow the necessary steps?

Modalities of fellowships: What kind of impact could we reasonably expect?

Time frame: How long is it likely to take before impact could be observed?

Mapping the pathway

A reliable “map” of the fellowship pathway , required to:Monitor progress towards the desired

destinationManage road blocks and unintended

consequencesReplicate successes and prevent

mistakes

Theory based evaluation

“Theory or program based approaches map out the channels through which the activities, inputs, and outputs are expected to result in the expected outcomes. It also allows for the identification of unintended effects”.

Inder Jit Ruprah, Office of Evaluation and Oversight, IADB.

Logic Pathway and Benefit Chain

INPUT ACTIVITY OUTPUTS

USAGE IMPACT

Improved

Outcomes

For Clients

Improved Capacity

Improved

Services

•Policies•Plans•Agreements •Fellows•Funds

•Selection•Placement•Training•Mentoring•M&E

•Satisfaction •Gained knowledge and skills•Behaviour change

•Use of gained skills• Contribution toperformance•Bridging gaps

Contribution analysis

An analysis of activities and interim outcomes that could reasonably be expected to contribute to a positive impact for the beneficiary institution.

Confirming the logical link among these outcomes and excluding other known influences (Conceptual logic )

Reducing uncertainty

Identifying outside factors not related to the fellowship that may have an effect ( Attribution)

Ascertaining that the fellowship program has been implemented properly within a logical conceptual framework

( Fidelity)

Verification of fidelity

Clear objectives for fellowship addressing agreed priorities

Selection of appropriate fellows and training programs

Appropriate support during training Opportunity to apply knowledge and skillUtilisation and support after return home Opportunity to contribute and to “echo”

learning

Assess the existing evidence on results

Set indicators and obtain evidence to verify achievement of the desired results at each level ( Monitoring key milestones)

Identify weak links between elements of the results chain and obtain further evidence to enable adjustments.

World Bank Indicators

Relevance: consistent with organizational objectives and HRD plans

Efficiency: value for money and adequate administrative and financial procedures

Effectiveness: major objectives were achieved.

Sustainability: continuation of benefit, once fellowships investment completed.

Impact: improved performance of organization and benefit to community

Extended Kirkpatrick’s framework

Planning, design and implementation- evidence of the appropriateness, relevance, efficiency and sustainability of the program.

Reaction – evidence of satisfaction (what the trainees/fellows thought and felt about the training);

Learning – evidence of learning (the resulting increase in knowledge or capability as reflected in end of course assessment);

Extended Kirkpatrick measures of training

Behaviour – evidence of behaviour change or capability improvement as reflected in job performance;

Results – evidence of contribution to the institution resulting from the fellows’ performance.

Mega Impact- evidence of the long term benefit of the improved performance of the institution and the services it provides to specific communities or target groups.

Evidence of Contribution

Self perception of enhanced capacity\contribution

Others’ perceptions of enhanced capacity\contribution (360 degrees)

Continuing to study and develop professionally

Recognition by professional bodies of gained qualification (eligibility for higher studies and duties)

Change in behaviour\attitude\performance

Evidence of Contribution

Change of work practice\procedures \ways of doing things associated with learning

Initiation of a new program or aspects of a program

Attainment of specific institutional objectives related to added capacity

Evidence of bridging a performance gap in the institution\program related to added capacity

Evidence of contribution to institution key success factors

Evidence of Contribution

Utilization of knowledge\skills by home institution

Passing on knowledge\skills to others (dissemination)

Assignment to higher or different duties that reflect use of learning

Improved career path \progression Demonstration of leadership Improved prospect of retention due to

higher motivation\morale

Use of all plausible evidence

Use any plausible evidence regardless of the design, method or source used to obtain it.

Use both qualitative and quantitative evidence (triangulation of findings)

Use information obtained through current monitoring and evaluation approaches and techniques. (Less costly and less bias)

Assemble the performance story

Document a Performance Story based on the evidence available showing the attainment of intermediate outcomes (critical milestones) and the logical link among them.

Performance story

The performance story describes the journey from the inception of a fellowship program to the attainment of its immediate and long term goals.

The important events and experiences along the way are identified as mile stones that could be monitored in order to ascertain that we are moving in the right direction and ultimately that we have reached our destination.

Performance story

The performance story should enable ‘reasonable’ observers to determine whether it is plausible that particular interventions led to certain results.

“Plausible Association"

“Plausible Association“ implies that reasonable people with adequate information concerning what has occurred or is occurring in the programme, agree that the programme contributed or is likely to contribute to the desired outcomes.

Seek out additional evidence.

Where an alternative explanation cannot be discounted, or the program cannot be shown to be a more likely contributor, the program logic should be reviewed and/or additional data gathered and evaluated.

Where this can’t be done, further evaluation is required or it should be recognised that the program is not the key contributor to the outcomes.

Work in progress

The approach is well suited to development programs where data is likely to reflect ‘progress toward results’, rather than a definitive statement of final outcomes.

A credible performance story includes;

The program context (including the results chain),

Planned and actual accomplishments, Lessons learnt, Approach for assuring the quality of informationThe main alternative explanations for the

outcomes occurring including reasons why they had limited or no influence.

Mayne (2003)

Engagement of stakeholders

Requires collaboration of stakeholders

Provides better contextual understanding and engenders ownership

Partnership and collective responsibility for success

Next Steps

Construct benefit chain for selected programs

Review existing evidence in relation to mile stones

Identify weak linksSeek additional evidenceConstruct performance storyConfirm conceptual logic and fidelity of

programs

Next steps

Assess “plausibility” of contribution to impact

Derive lessons and strengthen fellowship programs

Disseminate findings and establish benchmarks and guidelines

Engage stakeholders in all steps ( Particularly, fellowship authorities,

training institutions, beneficiary institutions, and fellows)

Recommendation

Recommend to the 17th Meeting of Senior Fellowship Officers to allocate existing resources and/or seek additional support for the conduct of pilot studies in selected countries and/or sectors, to test the appropriateness of the contribution analysis approach in the context of impact evaluation of UN system fellowship programmes.