Monitoring and Evaluation in Perspective

Lessons from International Experience

Delivering on the Promises of e-government

PREM WeekWashington DC – 26 April 2005

Bruno LANVINCITPO/GICT

The World Bank

Four points

• e-gov, e-strat, k-strat, d-strat• M&E as a strategic tool (WB approach)• Some lessons from experience • Avenues for action

Point 1

• e-gov, e-strat, k-strat, d-strat• M&E as a strategic tool (WB approach)• Some lessons from experience • Avenues for action

Global Information Infrastructure

Applications

Digital Divide

Content

Knowledge

Empowerment

1985 1988 1991 1994 1997 2000 2003

Shedding different lights at a complex set of issues

2005

Missing Link(Maitland Report)

Infrastructure

Regulatory aspects

E-Strat, D-Strat and M&E StratRenewed ‘IS’ context(Information Society approach)

WSIS (Declaration and Plan of Action)

MDGs (Goals and specific

targets under Goal 8)

E-STRATEGY

Monitoring and Evaluation

M&E

OUTCOME(e-readiness, connectivity,

usage .. etc..)

Changes(PRSP, CAS, ..)

New ‘E’ context

Is ICT important

for the success of the country’s

D-Strategy ?

Evidence from ICT4D experienceDOT Force, etc ..

NO

YES

E-strategies and Development :from D to E to IS

Development StrategyD-STRATEGY

(PRSP, CAS, ..)

Monitoring and Evaluation

M&E

OUTCOME(growth, competitiveness,poverty, health, education,

etc ..)

Traditional ‘D’ context

Point 2

• e-gov, e-strat, k-strat, d-strat• M&E as a strategic tool (WB approach)• Some lessons from experience • Avenues for action

Monitoring & Evaluation Toolkit for E-strategies Results

METER: background and objectives

• Consideration of 50+ e-strategies• Identification of goals pursued and ways to

assess progress • Identification of linkages between ICT objectives

and sectoral/strategic needs (WSIS, MDGs e.g. )• Proposal for an M&E methodology and specific

implementation tools

Assumptions and risks

Resources

Actions

Key initiatives

Strategic priorities

Policy goals

From LogFrame to M&E

Indicators

Impact

Outcomes

Outputs

Deliverables

Background and rationaleBackground and rationale

MethodologyMethodology

Framework

• E-readiness• Security• Digital Divide

Infrastructureand ICT sector

• Rationale• Definition• Overview• M&E Tables• Assumptions and risks

Indicators and quantificationIndicators and quantificationAnnexes

Module 2

Module 1

Toolkit Structure

E-government

E-business

E-learning

E-health

• Rationale• Definition• Overview

• M&E Tables• Assumptions and risks

• Rationale• Definition• Overview

• M&E Tables• Assumptions and risks

• Rationale• Definition• Overview

• M&E Tables• Assumptions and risks

• Rationale• Definition• Overview

• M&E Tables• Assumptions and risks

Module 3 Modules 4 & 5

SectoralModules(6 & 7)

Create efficient, responsive, transparent

government

Publish: Provide valuable

on-line information

Interact: Engage society to

improve government

Contract: Offer cost-effective

online services

M&E : the 3 levels of E-government

e-g

overn

men

tG

enera

l fr

am

ew

ork

to incl

ud

e e

-govern

men

t in

e-s

trate

gie

s

Prerequisite Concern Activity (typical indicators)

Access Infrastructure, costs, competition/ regulation (hence includes proper regulatory and competition frameworks)

          Equipment (PCs, kiosks, community centers)          Teledensity          Rule of law          Pro-competitive ICT regulation (tariff and non-tariff barriers, competition in the ICT sector)          Cost (fixed line calls and Internet access)          Access for disadvantaged or excluded 

Basic Skills Basic education, vocational training, ICT awareness

          Literacy (alphabetization rates)          E-literacy ratios per age/group/sex/region          Vocational training

Content Value to government and citizens           Questionnaires on value to users/citizens and government          Content in local languages

Desire Political leadership and will to reform

          Public statements/decisions          Laws & regulations (perceptions of quality of legal system)

Engagement Commitment of all components of civil society

          Broad involvement of civil society (questionnaire/survey)          Local awareness of ICT potential for development (questionnaire/survey)

E-g

overn

men

tExam

ple

: “

inte

ract

” le

vel

Pyramid Layer

Objective Indicator Data source

Policy goals Create an efficient, responsive and transparent government

Perception of overall administrative burden Perception of government effectiveness

Office of Government and Ministry of Local Government

Strategic priorities  

Bring valuable information online to the public, anytime anywhere a

          Perception of government online presence          % pop. using govt. sites          Usage growth rate

E-government CIO’s Office

Key initiatives

          Roll-out of online information services b          Raise public awareness through online and offline channels

          No. of agencies with web sites          % of agencies with web sites          % of information services rolled-out on time

E-government CIO’s Office

Actions          Establish an independent central e-government group and M&E unit c Establish selection guidelines for information/content to be posted, including nature and volumeAssess technology and organizational needs/requirementsDevelop online information platformsOffer information services on line, including local language contentDevelop publicity campaign to promote new e-government initiativesSolicit feedback on usability and usefulness of online government services

          Central e-government team and M&E unit established by month A          Guidelines for M&E established by month B          Relevant information sources identified by month C          System functional requirements completed by month D          Mid-term implementation review conducted by month E          Public awareness survey results

Project team or Central M&E unit

 (a) There should be a clear linkage between the type of content that is brought on-line and larger development goals, such as economic development, anti-corruption, and attracting FDI.(b) Criteria used for information publication should be related to cost and time savings and envisaged productivity gains.(c) Central e-government group responsible for interoperability and inter-agency consistency of e-government services, security, consolidation of records, and M&E.

METER: next steps

• Gather comments on the toolkit• Establish and launch METER website• Develop new modules (in partnership)• Launch revised version of toolkit at WSIS• Offer toolkit in different languages• Continue to contribute to continuous exchange

of best practices in the area of e-strategies M&E• Reflect outcome in WB practices• Identify, disseminate and promote best practices

Point 3

• e-gov, e-strat, k-strat, d-strat• M&E as a strategic tool (WB approach)• Some lessons from experience • Avenues for action

ICT &Knowledge Economy,Innovation

ICT & sectoral improvement

(e.g. educationhealth, etc..)

ICT access(infrastructure,

equipment,connectivity)

Measurable

Strategic

ICT’s Impact on DevelopmentMeasurable or strategic ?

ICT &Job creation,Productivity,

Competitiveness,Poverty

reduction

Readiness Access Usage Value

- Enhance produc- tive uses of ICTs in economy and ICT awareness in society- Foster SME com- petitiveness at home and abroad

- Promote visible E-government services with measurable impact on SMEs- Foster regional integration through on-line services to rural areas

- Establish a one- stop portal for SME registration- Allow and promote on line request/issuance of land titles and birth certificates through commu-nity access points

PolicyD- Strategy

E- Strategy

- Adapt legal syst. for on-line business registration (OLBR)- Create/promote portal for OLBR - Establish CAPs- Create/promote Adm. Doc. portal

Initiatives

Actions

Typical M&E indicators/ time horizon

# of SMEs regis- tered on-line# of CAP created

Example of a possible ‘e-gov’ implementation chain

Relevant M&E indicators

- Pursue a ‘knowledge-based’ competitiveness strategy-Improve image and efficiency of government- Enhance involvement of private sector in key areas- Redress local imbalances (social, geographic, eg)

SME rating of e-govRegional disp.index

ICT awarenessSME

competitiv.

GDP growth% P.S. in GDP

Inequality index

# of hits to site (awareness) for SMEs, titles,..

KOR

A pre-requisite for e-government: Internet Access

60

50

40

30

20

10

0

15 20 25 30 35 40 45 50 55 60 65 70

Internet subscribers

per 100 inhabitants

Average cost for 20 hours

(day + evening) in $ PPP

FIN

TUR

ICL

DNK

FRA

GERAUT

CDN UK

USA

SWECH

NORNL

OECD

IRL

SPA

POR

MEXGRE

POL

LUX

BEL

SLK

HUNCZE

The ‘Nordic’ cluster

The ‘mature markets’ cluster

The ‘Emerging Europe’ cluster

Source : World Bank, based on OECD data

Internet Costs and Diffusion(OECD – 2002)

Governments’ functions

Governments’ roles

•E-government- Customs

- Taxes- Procurement

•Statistics/macro-ec managmnt•E-democracy/local government

•Fiscal and monetary policy•Trade/FDI policy•Diplomacy/defense•Education•Legal/regul/compet. policies•Justice/order etc. (“regalian”)

Informatization +Decentralization

Globalization +Market efficiency

CivilSociety +

IT

Demand/Supply for IT purchases

Technology supply, markets for IT

Limits to IT imports/exports

Labor supply for IT sector

Incentives/limits for IT market

Leadership +

Facilitation

E-strategy, national ambition/social project pursued through IT

Demand for IT solutions/leading through example

OLD (diminishing role)

NEW (increasing role)

Pro

du

cer

Fac

ilit

ato

rL

ead

er

Environment Readiness Usage

Provide and promote vision (KISS)• Address ‘Digital Divides’

• domestically• internationally

• Give signals to markets• ICT as a national priority• large projects or objectives

• Promote and defend national interests in international and global forums

G4

G3

Education policy• curricula/life-long learning• ICT training facilities• Wiring/networking of schools

New roles for governmentsG5

E-government• services on line• procurement• trade facilitation• civil society participation• good governance

G2Provide proper environment• Macro-economic environment• Fiscal policies (cost, innovation,

investment, VC, PPP)• Legal/regulatory environment for

ICT (competition, independent regulator, rule of law)

G1

Provide access (univ serv)• Lay out ICT infrastructure• Produce ICT equipment• Finance Public R&D

8.00

Leaders

Fac

ilit

ato

rs

y = 0.6839x + 1.93992

R = 0.3991

GUA

ZIM

PAR

BOL

VEN

ARG

HON

ECU

PAN

EGY

ELS

PHI

PER

INO

TUR

ROM

NICBAN

NIA

MAU

UKR

RUS

URU

GRE

BUL

DOM

JOR

SLK

POL

COS

COL

TRI

MEX

SAFSLV

LIT

LAT

VIE

JAM

THA

INI

CHL

ITA

BEL

GER

AUT

NOR

SRICZE

LUX

HUN

CHN

IRE

MLT

POR

BRA

NWZ

JAPEST

NETAUL

FRA

SPADEN

KOR

SWIICE

SWEUSA

HKG

UK

CAN

ISR

TAI

FIN

SIN

2.00

3.00

4.00

5.00

6.00

7.00

2.00 3.00 4.00 5.00 6.00 7.00

Note : the ‘Government’ variable of the GITR index has been used as proxy for Government leadership (x), whereas ‘Competition in telecoms’ has been used as an indicator of the effectiveness of Governments as facilitators (y)

First circle (top performers)

Second circle (the contestants)

Third circle (ready or not)

e-government for ‘new’ government

Point 4

• e-gov, e-strat, k-strat, d-strat• M&E as a strategic tool (WB approach)• Some lessons from experience • Avenues for action

The A,B,C,D,E of the Digital Divide

Access

Basic skills

Content

Desire

Engagement

- Infrastructure, costs, competition/regulation

-Basic education, vocational training, entrepreneurship- Local value, languages

- Local will to reform

-Commitment of all compo- nents of civil society

Direct effets

• Increased public sector efficiency– Savings for governemnt (lower administrative costs)– Better management of public resources

• Better access to public services by firms and citizens– Savings for users (time and money) – Extended coverage (geographic, social, timewise: 24/7)– New services offered (e.g. itineraries, doc search & comparison,.)

• Improved economic governance– Data on traffic (flows of goods & services, payments, tracking, ..)

is more precise, quasi-instantaneous and cross-referentiable– Public service staff can find renewed motivation– Cooperation improved between public sector, private sector and

citizens

Indirect effects

• Transparency/governance– e-procurement (on-line tendering)– Selectivity systems in customs and other inspection-based services– Involvement of citizens in policy debates and decisions

• Business competitiveness– Faster services (registrations, licenses, authorizations,..)– Timely access to strategic data (prices, markets, laws/regulations,,..)

• IT knowledge and litteracy– learning-by-doing, incremental improvements to software, etc…– Breaking psychological barriers (public access points, ..)

• Building information societies– Turning local knowledge into value and competitiveness (culture, modus

operandi,..)– Attracting external partners (trade, investment, ..)– Contributing to global development efforts (MDGs)

e-government vs better government

• Consider and promote e-government as a government-centered effort

• Consider and promote e-government as a technology-driven effort

• Replace every paper-based process by an electronic process

• Offer ‘ministry-specific’ e-solutions

• Launch, measure, punish/reward

• Focus the e-government process on users (citizens & businesses)

• Focus e-government strategy on people (both on the government and on the users side); favor technology-neutral choices

• Use e-government as a tool to foster changes in attitudes and thinking, and as an instrument for reforms

• Rally government-wide energies and competence around common procedures and standards (back office), and an ‘all-of-government’ approach (front office)

• Adopt an early common approach to monitoring and evaluation (accountability, ownership, results)

Ways to fail Ways to go

Thanks for your attention

blanvin@worldbank.org