Stakeholder data validation workshop July 2, 2012 Windhoek

Namibia looking forward:Effectiveness of capacity developmentsupport to local partners

Why we are here today

• To share Pact Namibia’s internal evaluation findings with stakeholders

• To get stakeholder validation of the evaluation recommendations

Agenda

• Click to edit Master text styles• Second level

• Third level• Fourth level

• Fifth level

Time Activity

9:30–10:45 Presentations

10:45–11:00

Tea break

11:00–11:30

Q&A

11:30–noon Small group work: Discuss findings & review recommendations

Group 1. Capacity development impactGroup 2. Graduation to direct USAID fundingGroup 3. Lessons learned for capacity development organizations

Noon–12:45 Report back

12:45–1:00 Conclusions & next steps

1:00–2:00 Lunch

Background

• USAID Namibia began scaling up HIV programs in 2003.

• In 2007 USAID awarded Pact a 5-year project which included grants, organizational and technical assistance to CSOs, predominantly in areas of: • HIV/AIDS prevention

• Care and support to orphans, HIV-vulnerable children and home-based clients

• Strengthening capacity of the Ministry of Gender Equality and Child Welfare

Background

• In 2010 – award changed to focus on two objectives:1. Strengthening capacity of the Ministry of Gender

Equality and Child Welfare to effectively coordinate activities relating to gender equality and orphans and vulnerable children

2. Strengthening six civil-society organizations to be eligible for U.S. direct funding while maintaining high standards of existing services

Tools to strengthen civil society

• Organizational Development (OD) Roadmaps measures CSO partner organizational systems and structures objectively• Basis for identifying organizational efficiency gaps

and prioritizing interventions• Progression on a scale from 1 (nascent) to 5 (mature)

• Comprehensive Institutional Strengthening Plans to identify, schedule and monitor all capacity-building activities

Tools to strengthen civil society

• Comprehensive Institutional Strengthening Plans to identify, schedule and monitor all capacity-building activities

• Organizational Development (OD) Roadmaps• 10 capacity areas: Purpose &

planningHR

managementMonitoring & evaluation

Networking

Governance

Organizational sustainability

Financial management

Grants & compliance

Operations management

Projects & services

Organizational Performance Index (OPI)Scored on a scale of 1 to 4on each of the following dimensions:

• Effectiveness.Achieving outcomes, meeting standards

• Efficiency.Delivering services, increasing reach

• Relevance.Engaging target populations, promoting learning

• Sustainability.Mobilizing resources, increasing legitimacy

Why do this evaluation?

1. Assess the sustainability of graduated partners’ capacity development

2. Review the quality and quantity of services offered to beneficiaries of former and current CSO partners (partially done)

3. Assess the effectiveness of the Pact Namibia capacity building assistance to the partners that have graduated or plan to graduate

4. Draw lessons for other Pact programs that are facing similar challenges of graduating partners

Evaluation questions

• To what extent was Pact Namibia’s capacity development support effective?

• To what extent were partners ready for and able to effectively manage direct funding?

• To what extent did service quality and coverage change over the last two years?

Process

3 phases:1. Designed terms of reference and data

collection tools (virtual group)2. Collected data (Pact staff from 5

country offices, for neutrality)3. Analyzed data, drafted and

disseminated report (current group of Pact staff)

Data collection tools

1. Partner survey (Mobenzi) • 17 current & former partner organizations interviewed• Quantitative & qualitative

2. Organizational Performance Index (OPI)• 12 of 17 partners

3. Pact staff survey (Mobenzi)• 4 core Pact Staff from each technical area

4. USAID tool5. Historical OD Roadmap scores

Data analysis

• Quantitative data: • Merged datasets and did selected frequencies with

statistics and cross-tabulations using SPSS and Excel.

• Qualitative data: thematic analysis.

Partner breakdown

OD Roadmap support, graduated to direct funding

OD Roadmap support, not graduated

No OD Roadmap support, graduated

No OD Roadmap support, not graduated

Total

Number of orgs

3(18%)

3(18%)

2(12%)

9(53%)

17

Number of partner respondents

11(23%)

12(25%)

3(6%)

22 (46%)

48

Study limitations

Sample size• Only 17 organizations• Limited number of respondents in some

organizations; some interviewed were previous staff

Time • Could not conduct survey of beneficiaries

Presentation of the findings

• Capacity development impact• Graduation to direct USAID

Funding• Lessons learned for capacity

development organizations• Preliminary recommendations

Capacity development impact

Key areas of analysis

1. How did length of support influence partners’ OPI scores and perceptions of usefulness of support?

2. How do Pact and partner perceptions of support compare?

3. How do OD Roadmap supported partners differ from other partners?

4. Which areas of Pact support did partners rate as most useful and of highest quality?

5. What were the most significant changes resulting from Pact’s support?

1. Partners scored longer support less useful

3.03.23.43.63.84.04.24.44.64.8

Usefu

lness r

ati

ng

StrategicFinancial

Program thematic

Monitoring & evaluation

2. But Pact scored longer support more useful

Partners score longer support less useful

3.03.64.24.8

Part

ner

rati

ng

Strategic

FinancialProgramthematic

M&E

Pact scores longer support more useful (except financial)

3.0

4.0

5.0

Pact

rati

ng

Strategic

Financial

Programthematic

M&E

3. Longer support – higher OPI scores

0%

20%

40%

60%

80%%

of

part

ners

scor

-

ing o

ver

3

StrategicFinancial

Program thematicMonitoring & evaluation

4. OD Roadmap partners scored higher on OPI

OD Roadmap Non-OD Roadmap

0%

50%

100%

67%

17%

OPI sc

ore

above

3

Tech

nica

l Sup

port

Finan

cial m

anag

emen

t sup

port

M a

nd E

repo

rting

supp

ort

Advoc

acy Su

ppor

t

Stra

tegic plan

ning

supp

ort

HR Man

agem

ent

Other

0%

10%20%30%40%50%60%

% o

f all

part

ner

staff

Non-graduated partners

1a. But non-graduates more keen on financial management

5. Non-OD Roadmap partners rated Pact support more useful

Strategic Support

Financial Support

Program Thematic

M&E3.50

3.75

4.00

4.25

4.50

Use

fuln

ess

rati

ng

Non-OD Roadmap

OD Roadmap

6. Partners rated M&E support highest

Program Technical

Financial Planning

Strategic Planning

M&E3.50

3.75

4.00

4.25

4.50

Quality rating

Usefulness rating

7. Most significant changes

M&E and Financial management the “cornerstones of programs” (partner, Pact and USAID responses)

• M&E• Timely reporting• Capturing the right and required data• Use data for decision making

• Finance• Proper management of organization finance & USAID funds • Complete and accurate finance reports• Proper budget management

7. Most significant changes

• Moved from no system to very good system in all areas (partner responses)

• Buy-in from organizations (Pact responses)

• How do we sustain the change?• Experience handover and staff retention• Resources

Summing up

• More years of support = lower usefulness rating, but higher OPI scores

• Pact and partners see usefulness differently over the duration of support

• OD Roadmap support = lower usefulness rating, but higher OPI scores

• M&E support rated most useful and of highest quality

• Most significant change in M&E and financial management

Graduation to direct USAID funding

Key areas of analysis

1. Capacity-building support that partners would like more of before graduation

2. Graduate vs. non-graduate performance on OD Roadmap and OPI

3. Prerequisites for graduation4. Key capacity-building support for

graduated partners

Tech

nica

l Sup

port

Finan

cial m

anag

emen

t sup

port

M a

nd E

repo

rting

supp

ort

Advoc

acy Su

ppor

t

Stra

tegic plan

ning

supp

ort

HR Man

agem

ent

Other

0%

10%20%30%40%50%60%

% o

f all

part

ner

staff

1. Partners want more technical support before graduation

-10%0%

10%20%30%40%50%60%

% o

f all

part

ner

staff

Non-graduated partners

Graduated partners

1b. And graduates far more interested in technical support

0 1 2 3 4 5 6 7 8 9 10 111

1.5

2

2.5

3

3.5

4

HR

MM

ER

L

Finan

cial

Netw

orki

ng

Gov

ern

ance

Sust

ainab

ilit y

Com

plia

nce

Opera

tion

sPr

ogra

ms

Purp

ose &

pla

nnin

g

2a. Graduates scored higher on OD Roadmap

Graduates

Non-graduates

0.5 1 1.5 2 2.5 3 3.5 4 4.51

1.52

2.53

3.54

Rele

vanc

eSu

stai

nabi

lit y

Effec

tive

ness

Efficien

cy

2b. But OPI not different

Graduates

Non-graduates

1. Strategic plan2. Governance3. Strong financial systems and internal

controls4. Administrative systems, including human

resource management5. Skilled staff and technical capacity6. Strong monitoring and evaluation

3. Graduation prerequisites from Pact and partners

4. Key post-graduation capacity development

Priority• Resource mobilization support• Technical/Program support

As needed • Monitoring and evaluation

• Database development• Evaluations• Routine data quality assessments

• Financial management

Summing up

1. Graduated partners would have liked more technical support before graduation

2. Graduated partners averaged 3.4 on Roadmap, up from 2.5 for non-graduated partners (no difference on OPI)

3. Six graduation prerequisites: strategic plan, governance, finance, admin, technical capacity, M&E

4. Key post-graduation needs: technical and resource mobilization support

Lessons learned for capacity development organizations

Key areas of analysis

1. Retention of systems after capacity building,and impact of who decided to engage in it

2. OPI scores and trends across partner subgroups

3. Different capacity needs identified by partners and donors

4. Challenges and advantages of graduating to direct funding

Stra

tegi

c Pl

anni

ng

Fina

ncia

l Man

agem

ent

Prog

ram

The

mat

ic A

rea

Mon

itorin

g & E

valu

atio

n20%40%60%80%

100%

% o

f org

s st

ill u

sing

syst

em

1a. OD Roadmap-supported partners more likely to

Non-OD Roadmap Support

OD Roadmap Support

retain systems

Who decides to acquire system

% retaining system

N

Partner alone 95% 60

Both partnerand Pact

83% 87

Pact alone 75% 24

1b. Retention likelier if system is partner-driven

20%

40%

60%

80%

100%

% r

eta

inin

g s

yst

em

1c. Retention likelier if system is partner-driven

Program thematic

Financial

StrategicMonitoring & evaluation

2. OPI scores

• Average OPI scores show lower resource mobilization but high social capital

• Graduated partners generally score higher

• Current partners score low on resource mobilization and service delivery

RelevanceEffectiveness

2a. Average OPI shows lower resource mobilization, high social capital

Efficiency Sustainability0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

Average OPI

RelevanceEffectiveness

Efficiency Sustainability

2b. Graduates need to focus on resultsand external evaluation of standards

Resul

ts

Stan

dard

s

Deliver

y

Reach

Targ

et

Lear

ning

Resou

rces

Socia

l Cap

ital

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0 OD Roadmap graduates

Average OPI

2b. Current partners scored lower on

RelevanceEffectiveness

Efficiency Sustainability

Average OPI

Non-graduated OD Roadmap partners

Resul

ts

Stan

dard

s

Deliver

y

Reach

Targ

et

Lear

ning

Resou

rces

Socia

l Cap

ital

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

resource mobilization and service delivery

3a. Different stakeholders, different top-ranked capacity needs

OD Roadmap partners1. Monitoring & evaluation

2. Technical support

3. Financial management

4. Program development

Pact1. Resource mobilization

2. Financial management

3. Institutional support

4. Technical support

USAID1. Resource mobilization

2. Financial management

3. Administration

4. Monitoring & evaluation

Non-OD Roadmap partners1. Technical support

2. Financial management

3. Advocacy

4. Monitoring & evaluation

3b. Graduated partners & USAID: contrasting expectations

Partners expect:1. Same relationship/treatment with USAID as partners had with Pact

2. Continued technical assistance, but often lack adequate budget

3. Increased USAID capacity-building support after graduation

USAID expects:1. Strong organizational capacity and governance systems, ability to work independently and maintain capacity even after staff turnover

2. USAID support to partners, but also partner ability to address own needs

3. Performance like long-term international USAID partners, responsive and accountable

4. Graduates found both challenges & benefits

Challenges1. Increased accountability in line with U.S. Government rules & regulations

2. Less frequent capacity-building support

3. Would like clearer guidance from USAID on budgets, funding & programs

4. Would like more regular feedback on financial reports

Benefits1. Shorter reporting channels give more time to prepare reports

2. Improved interaction with USAID & understanding of USAID programs

3. Less cumbersome financial reporting requirements

4. International exposure & recognition

5. Good working relations

Summing up

1. Current partners most likely to retain M&E, financial and technical systems

2. Sustainability of capacity building depends on whose decision it was to engage in capacity development

3. On average partners tend to score lower in resource mobilization on OPI scoring

4. Partners still need to develop capacity, but are struggling to understand how to access support under direct funding.

Preliminary recommendations

Preliminary recommendations

•Capacity development impact•Graduation to direct USAID

funding•Lessons learned for capacity

development (CD) organizations•General

Capacity development impact

• Capacity development support should be provided for three or more years (more research is needed).

• Capacity development organizations need to keep their support “fresh” for organizations served for longer periods.

• OD roadmap support is an important approach for preparing effective partners and should continue to be used.

• Capacity development support should increase focus on knowledge transfer and resource mobilization for increased sustainability.

Graduation to direct USAID funding

• The OD Roadmap should be considered as the tool to measure preparedness for an organization’s readiness for graduation.

• The prerequisites for transition to direct funding should include:• Strong strategic planning capacity• Governance structures• Strong financial systems and internal controls• Strong administration (including HR) systems• Skilled staff and technical capacity• Strong M&E• An ability to mobilize resources• An ability to advocate with donors to meet (internal and external)

constituent needs.

Lesson learned for CD organizations

• CD organizations should ensure ownership of interventions for maximum impact in the area of system retention.

• CD organizations need to work with partners to diversify their funding base.

• CD service providers need to help partners to remain relevant and vibrant by planning to incorporate internal capacity development support on an on-going basis.

• Increased communication on expectations will improve the transition to direct funding for both partners and USAID.

General

• Pact should consider expanding this dataset to include more Namibian respondents and more international graduates (33 worldwide) and partners in order to make the findings more robust.

• Pact should assess the impact of the capacity development approach (and the transition to direct funding) at the beneficiary level.

Thank you!