John Gaventa

World Bank Institute Seminar November 22, 2010

So what difference does it make? Assessing the impact of participation, transparency and accountability

Taking a Citizen – Led Approach: 10 Years of DFID – funded research on Citizenship, Participation and Accountability (www.drc-citizenship.org).

First phase, much attention was on dynamics of state-society relationship

Second phase, more on how citizens mobilise and empower themselves, often outside of the state

In this presentation draw from this work, but also highlight two recent projects and their implications for voice and accountability– Mapping outcomes of citizen engagement – The impact of transparency and accountability

Active citizens build effective states - not (only) the other way around

• Much has been learned about citizens view the state, and about the state-society relationship

• Citizens can help to build democratic institutions, legitimacy, responsiveness, capability, accountability

• ‘societal opportunities’ create possibilities of political reform

• But we need more focus on the society side: • how to ‘build’ active, empowered citizens• how active citizens mobilise to change

development policies, build responsive states, and do things for themselves

What difference does citizen engagement make? The plea for evidence

The idea that good governance cannot be achieved without the active involvement of citizens and civil society actors has gained growing consensus in recent years. Many donors and NGOs now support "participatory governance", "social accountability" or "demand for good governance" programmes aimed at promoting the active involvement of citizens/CSOs in public decision-making and holding government accountable [...] I'm currently involved in a research project to gather evidence of the results and/or impact of such initiatives.  

E-mail to author from World Bank consultant, 2009 

. 

‘Our number one challenge is to demonstrate what difference citizen engagement makes.’

- Representative of large donor agency 2008.

What difference does citizen engagement make? Results from a meta-synthesis of 100 case studies in 20 countries

Drawing from widely accepted approaches of systematic review, qualitative case study analysis and synthesis

Coding of 800 ‘outcomes’

Evidence points to largely positive contributions

With risks of negative outcomes

Distribution of positive and negative outcomes

Construction of cit-izenship

Practices of cit-izen participa-

tion

Responsive and accountable

states

Inclusive and cohesive soci-

eties

Total0

10

20

30

40

50

60

70

80

90

Positive

Negative

Pe

rce

nta

ge

Positive Negative

Increased civic and political knowledge

Greater sense of empowerment and agency

Increased knowledge dependencies

Disempowerment and reduced sense of agency

Outcome 1

Better Citizens

Outcome 2

More Effective Participation

Positive Negative

Increased capacities for collective action

New forms of participation

Deepening of networks and solidarities

New capacities used for ‘negative’ purposes

Tokenistic or ‘captured’ forms of participation

Lack of accountability and representation in networks

Positive Negative Greater access to state services and resources Greater realisation of rightsEnhanced state responsiveness and accountability

Denial of state services and resourcesSocial, economic and political reprisalsViolent or coercive state response

Outcome 3 More Responsive and Accountable States

Positive Negative Inclusion of new actors and issues in public spaces

Greater cohesion across social groups

Reinforcement of social hierarchies and social exclusionIncreased horizontal conflict and violence

Outcome 4 More inclusive and cohesive societies

Types of outcomes and types of engagement

Citizens engage in multiple ways – Local associations– Social movements and campaigns– Formal governance spaces– Multiple forms of engagement

Local associations and movements are particularly important for positive outcomes

Multiple forms of engagement are particularly important for realising responsive and accountable states

Beyond Putnam – not just the density but the nature of the quality and quantity of the association that counts

Distribution of positive and negative outcomes across type of citizen engagement

Outcomes sorted by type of citizen engagement(n=828)

Outcome type

Local associations

(n=324)

Social movements

and campaigns

(n=233)

Formal participatory governance

spaces (n=153)

Multiple (n=118)

Positive 90% 71% 55% 68%

Negative 10% 29% 45% 32%

Total 100% 100% 100% 100%

Types of citizen engagement (n=100)

Positive outcomes sorted by outcome categories

Local association

s(n=29)

Social movements

and campaigns

(n=29)

Formal participator

y governance

spaces(n=19)

Multiple(n=23)

Construction of citizenship

36% 35% 33% 29%

Practices of citizen participation

26% 24% 30% 22%

Responsive and accountable states

29% 33% 25% 44%

Inclusive and cohesive societies

9% 8% 12% 5%

Total 100% 100% 100% 100%

Distribution of positive outcomes across types of citizen engagement

The relationships of outcomes to context

Grouping of countries across regime (Polity IV, Freedom House, Economist Intelligence Unit|)

Positive outcomes are not linearly associated with level of democratisation – highest proportion of positive outcomes are found in most and least democratic countries

Associations are particularly strong for least democratic settings

Distribution of positive and negative outcomes across country types

Tier 1 Tier 2 Tier 3 Total0

10

20

30

40

50

60

70

80

90

100

Positive

Negative

Pe

rce

nta

ge

Types of citizen engagement

Positive Outcomes in Tier Three countries(n=273)

Construction of citizenship

(n=96)

Practices of citizen

participation

(n=66)

Responsive and

accountable states(n=83)

Inclusive and

cohesive societies

(n=28)

Local associations 89% 92% 83% 78%

Social movements and campaigns 0% 2% 2% 0%

Formal participatory governance spaces 6% 0% 4% 11%

Multiple 5% 6% 11% 11%

Implications

1. Citizen engagement makes a difference, but not always. We need to understand more the factors that lead to positive vs negative change

2. These gains do not emerge automatically: pay more attention must be paid to building citizenship e.g. of empowerment, as a component of other action

3. Only through empowered citizens will accountability relationships likely occur.

Key Findings

4. Citizen engagement makes a difference across regimes, not just in more democratic settings.

5. The role of associations in fragile settings is particularly important for building cultures of citizenship

6. Citizen engagement is often met by reprisals: it is critical to protect the democratic space for engagement if developmental outcomes are to be achieved.

Part II: What is the impact of citizen-led accountability initiatives?

Over a decade of rapid growth and spread of transparency and accountability work in development and aid circles and development academia

Apparent promise of T&A as the cures for many ‘evils’

As yet, little clarity about what is being achieved, what works, how it works, and how best to fulfil that promise…

... Particular focus:

Citizen-led, demand-side and ‘social accountability’ activities and their connection to state actors, institutions and processes

Connections between T, A and participation

Methodological challenges of impact assessment in the T&A field

‘effectiveness’ Vs ‘impact’

Service delivery, budget processes, FoI, natural resource governance, aid

... Particular focus:

Citizen-led, demand-side and ‘social accountability’ activities and their connection to state actors, institutions and processes

Connections between T, A and participation

Methodological challenges of impact assessment in the T&A field

‘effectiveness’ Vs ‘impact’

Service delivery, budget processes, FoI, natural resource governance, aid

Some evidence of impact, but highly uneven….

The positive story - in some conditions TAIs demonstrably contribute to: – Greater state responsiveness– Lower corruption – Building spaces for engagement and empowering

local voices – Better budget utilization and delivery of services

But not always: – Evidence is uneven and scattered– Initiatives are new and impacts unknown– Much focus on effectiveness rather than impact– Positive evidence in one case not corroborated by

studies in another

How do we enhance demonstrable impact?

Challenge 1:

Aims, claims, assumptions and expectations (or, against what are we assessing impact and effectiveness?)

Aims vary: Developmental outcomes, democratic outcomes, voice empowerment outcomes

Assumptions vary: eg on links between T, A and participation; about ‘citizens’; hierarchies of objectives; how explicit/implicit; etc

The need for sharper theories of change

Challenge 2:

Methodological issues: How do we know what we know?

We found some:

Quantitative surveys

Analysis of aggregated survey data, multivariate analysis

Experimental approaches (RCTs)

Qualitative case studies and case study analysis

Stakeholder interviews

Indices and rankings

We found a lack of:

Comparative studies

Ex-post long-term evaluations

Appropriate use of baselines

Rigorous participatory approaches

Complexity-aware approaches eg Most Significant Change; Outcome Mapping; narrative-based

Methodological mixes

Methodological issues (cont.)

Limited evidence, few comparators, difficulty of valid counterfactuals

Untested assumptions and poorly articulated theories of change: blurry goalposts

Correlation Vs causality; attribution Vs contribution

Indicators – what we want to measure Vs what we can realistically measure

Whose perspectives, which impacts count?

Upward or downward accountability in the impact assessment process itself

Complexity, contingency, uncontrollability

Challenge 3: Factors that make a difference

Little evidence that supports generalisations of the kind ‘initiative x produces outcome y’....

A more useful question to ask:

Which factors (enabling and disabling) shape the possibility that TAIs will achieve their stated goals in a particular context?

Challenge 3 (cont.): Factors

State (supply side)

 

Level of democratisation

Level of political will

Broader enabling legal frameworks, incentives and sanctions

 

Citizen voice (demand side)

Capabilities of citizens and civil society

Interaction of TAIs with other mobilisation and collective action

Embeddedness of TAIs in broader policy processes

Linking Mechanisms

Challenge 3 (cont.): FactorsThinking beyond the dichotomies (state-society; voice–response; supply-demand)

New thinking in governance would urge us to pay more attention to:

Multiple actors, accountability coalitions, networked approaches to governance

Changing norms and cultures of accountability in state, private sector and civil society simultaneously

Looking across levels and scales: Linking the local, national, regional, international

Bringing politics back in: – Power – The black box of political will and political economy– Links to parties, elections and political regimes

Key Lessons

The evidence base (+/-) is weak - but that doesn’t mean that TAIs are not significant. The challenge is to deepen the evidence and the methods for developing it.

On the state of the evidence:

• Develop new approaches to assessment, with complexity perspective, that combine methods and approaches

• Explore further user-centred and participatory approaches• Support comparative in-depth work across contexts and

TAIs, multi-case and other synthetic analysis• Strengthen capacities of researchers and practitioners to

develop and build on innovative approaches• Build into new TAIs ToCs, baselines, comparators, etc

Key lessons (cont.)

On factors for greater impact:• Deepen understandings of synergies between T, A,

participation and voice• Move beyond dichotomies to build new knowledge on

cross-cutting accountability coalitions• Apply to T&A field the cutting-edge thinking on

governance, especially networked governance; interaction of global – national – local; and private sector

• Explore whether initiatives can travel across context, method and issue

For links to these studies

www. drc-citizenship.org

www.ids.ac.uk

• Gaventa and Barrett, ‘ So what difference does it make? Mapping the outcomes of Citizen Engagement, ’ IDS Working paper 247

• Review of Impact and Effectiveness of Transparency and Accountability Initiatives,

Rosemary McGee & John Gaventa with contributions from Greg Barrett, Richard Calland, Ruth Carlitz, Anuradha Joshi and Andrés Mejía Acosta, IDS, October 2010