Misra, D.C.(2009): E-government Monitoring and Evaluation_MDI-12.2.2009

transcript

E-government Evaluation

E-government Monitoring and Evaluation:

Implementing E-business Plan

byDr D.C.MISRA

E-government M&E© Dr D.C.Misra 2009 2

Thursday, February 12, 200911-15 a.m. to 1-15. p.m

3rd Post Graduate Diploma in Public Policy and Management Programmme

(2007-09)

School of Public Policy and Governance

Management Development Institute

P.B.60, Mehrauli Road, Sukhrali,

Gurgaon 122 001

E-government Monitoring and Evaluation:

A Presentationby

Dr D.C.Misra, I.A.S. (Retd.)E-government Researcher and Consultant

New Delhi, India

Email: dc_misra@hotmail.com Web: http://in.geocities.com/drdcmisra Think Tank: http://tech.groups.yahoo.com/group/cyber_quiz Tel: 91-11- 2245 2431 Fax: 91-11- 4244 5183

I CONTENTSI CONTENTSE-government Monitoring and E-government Monitoring and

EvaluationEvaluation

Part A: Monitoring

I. Historical Background of MonitoringII. Ministry of Statistics and Programme

ImplementationIII. Reasons for Delay in Project ImplementationIV. E-government Project Failures - An Indian

ExampleV. Causes of E-government Project FailuresVI. What can monitoring do?VII. What is not monitoring?VIII. What is then e-government monitoring?

Part A: Monitoring

IX. What is E-government Project? X. E-government Project Life Cycle-Five

Models XI. Logical Framework Approach (LFA)XII. Who will do e-government monitoring?XIII. The e-government monitoring unitXIV. E-government monitoring methodologyXV. Guiding Principles of E-government

Monitoring

ContentsPart B: Evaluation

I. Historical Background of EvaluationII. Programme Evaluation Organisation

(PEO)III. Functions of PEOIV. Four Generations of EvaluationV. What is not Evaluation?VI. What is then Evaluation?VII. Four Senses of Term EvaluationVIII. What is E-government Evaluation?

Part B: EvaluationIX. Types of EvaluationX. Evaluation Timing and Diffusion-

Adoption of CurvesXI. What is to be evaluated?XII. How E-government Domains are

evaluated?XIII. Approaches to EvaluationXIV. Who will evaluate? XV. Which Type of Evaluation is Suited

Most to E-government?

Part C: E-government M&E

FrameworkI. E-government Monitoring and

Evaluation (M&E)II. E-government Monitoring and

Evaluation (M&E) UnitIII. Relative Weights to Monitoring and

EvaluationIIV. E-government M&E Framework:

Components

E-government M&E Framework

(a) E-government Management

(b) E-government M&E Unit

(c) Information Needs Matrix

(d) E-government M&E Cycle

(e) Citizens

V. A Framework for E-government Monitoring and Evaluation

Part D and Part E

Part D:

My Questions

End of Presentation

Thank you

Part E:

Your Questions Now

Part A: MonitoringI. Historical Background of Monitoring

Monitoring in loose form has always been part of Indian administration Centralised monitoring is a recent phenomenon It came into existence at the Centre in 1985 when Ministry of Programme Implementation (MOPI) was formed MOPI is now part of Ministry of Statistics and Programme Implementation (MOSPI)

II. Ministry of Statistics and Programme Implementation

MOSPI tracks implementation of Central Sector Projects > Rs 20 Crore Report for January-March, 2007 covers 882 projects: -- Mega (Greater than Rs 1000 Crore) 69 -- Major (Between Rs 100 & 1000 Crore) 432 -- Medium (Between Rs 20 Crore & Rs 100 Crore) 381 Cost Over-run (33%) 287 Time Over-run (1 to 196 months)(35%) 301

III. Reasons for Delayin Project

Implementation1. Fund Constraints2. Land Acquisition3. Environment Clearance4. Slow Progress5. Delay in Supply of Equipments6. Law and Order7. Others (Technology selection and agreement,

Award of contract, Delay in civil works and government clearance, Geomining, Court case, Inadequate Infrastructure and bad weather)

(Source: MOSPI)

IV E-government Project Failure:

An Indian Example

Reinventing EPF (Employees’ Provident Fund) India

The largest reform project in terms of complexity Touches 40 million citizens Rs 250 million already spent Time over-run: 2001-06: 66 months: Target: 22 months Scrapped: January 2008: To be started all over again Contractor: Siemens Information Systems Limited

(SISL)Source: Dhoot (2008)

V Causes of E-government Project Failure

1. Complexity

2. Commitment Failure

3. Planning Failure

4. Vision Failure

Causes of E-government Project Failure

5. Inappropriate Methods6. Short Time Horizon7. Turbulent Environments8. Failure to Support End

UsersSource: Chiles (2001)

VI What can Monitoring do?

1. It can prevent E-government Project Failures

2. It can prevent cost and time over-runs of E-government Projects

3. It can keep a track of progress of E-government Project implementation

4. It can ensure that resources are expended as planned

5. Above all, it can ensure that the benefits of e-government project reach the target group, that is, citizens and non-citizens

VII What is not Monitoring?

Reporting ≠ Monitoring Inspection ≠ Monitoring Supervision ≠ Monitoring Audit* ≠ Monitoring Surveillance ≠ Monitoring Review ≠ Monitoring* Audit: 1. Financial, 2. Performance, 3. Development, 4.

Social, 5. Citizen (Through RTI Act)

What is not Monitoring?

SN Tool Focus

1 Report Routine reporting

2 Inspection Fault-finding in detail

3 Supervision Overseeing implementation

4 Audit Examination of accounts

5 Surveillance Scanning of environment

6 Review Comprehensive feedback

What is not Monitoring?SN Tool Focus

1 Financial Audit

Conformance to financial rules

2 Performance Audit

Project experience

3 Development Audit

Quality and durability of assets (IRDP)

4 Social Audit Scrutiny from social point

5 Citizen / Media Audit

RTI Act ?

VIII What is then E-government Monitoring?

E-government Monitoring is a specialised, systematic, dynamic, and semi-autonomous management tool to ensure that the E-government Project serves the target group- Citizens and Non-Citizens- in accordance with e-business plan taking into account the interests of various stakeholders and the emerging challenges being faced by E-government

Elements of Definition1 Specialisation 6 E-government

Project

2 Systematic 7 Service to Citizens

3 Dynamic 8 E-business plan

4 Semi-autonomous 9 Stakeholders

5 Management Tool 10 Emerging Challenges

IX What is E-government Project?

An E-government Project is a development project which aims to transform an inward-looking government to a citizen centric government making best use of information and communication technologies (ICTs) through a carefully designed e-business plan

X. E-government Project Lifecycles

Five Models

1. Generic Model (Tasmania, Australia)

2. Technocratic Model (NIC, India)

3. Audit Model (Lea’s Model)

4. Systems Model (Heeks’ Model)

5. E-government Project Model

1. Generic Model(Government of Tasmania, Ausralia)

Source: Government of Tasmania 2002

2. E-government Technocratic Model

(National Informatics Centre, New Delhi)

A. Project Initiation and Planning

B. Software Development

C. ICT Infrastructure Creation

D. Service Provision

E. System Integration and Testing

F. Project Commissioning

G. Project Completion and Sign Off

H. Maintenance

I. Retirement

Source: Mishra 2005

3. Lea’s E-government Project Lifecycle

(Audit Model)

Initiation

Planning andImplementation

Monitoring

Operations

Source: Lea 2003

4. Heeks’ System Lifecycle

Project assessment

SystemConstruction

Design of the proposed

new system

Implementation and beyond

Analysis of current reality

Source: Heeks 2006

5. E-government Project Cycle

4 Evaluate

E-business Plan

5Review

E-business Plan

3Monitor

E-business Plan

2Implement

E-business Plan

1 Prepare

E-business Plan

Modify Modify

Modify

M&EUnit

XI. Logical Framework Approach (LFA)

Developed by Leon J. Rosenberg for USAID in 1969

It logically connects project activities to resultsThe logic is: If x is done (input), y will follow (output) under asumptions z

(outcome)It presents a concise picture of the project

in a page or two

Logical Framework Approach (LFA)

It is an analytical tool

LFA (approach) should not be

confused with logframe (document)

It is 4x4 Matrix (that is, it is a matrix

of four rows and four columns)

giving rise to 16 cells

Logical Framework Approach (LFA)

S.N. Structure(Type of Information)

Indicators(of Progress)

Measurement(Means of Verification)

Assumptions and Risks(Principal Methods)

Goal(Wider objectives)

C11 C12 C13 C14

(Hypothesis)

Objectives(Short-term

Objectives)

C21 C22 C23 C24

(Project Assumptions)

Outputs(Activities)

C31 C32 C33 C34

(Implementation Assumptions)

Inputs(Resources)

C41 C42 C43 C44

(Critical Conditions)

XII. Who will do E-government Monitoring?

The E-government Monitoring Unit

Set up a monitoring unit

It will be part of organisation but function independently

It will report directly to Top Management

It will be a specialised unit

XIII. The E-government Monitoring Unit

It will consist of:

1. Head of Monitoring Unit

2. Database Administrator/System Analyst

3. Statistician

4. Economist

5. Sociologist/Political Scientist

XIV. E-government Monitoring Methodology

I. Automated

1. Online Survey

(SurveyMonkey)

2. Virtual Focus Groups (E-groups)

3. E-mail Surveys

4. Blog (Comments)

5. Wiki (Comments)

6. Online Feedback, etc.

II. Manual

1. Desk Research

2. Sample Survey

3. Focus Groups

4. Case Studies

5. Individual/Group Interviews/Discussions

6. Participatory Appraisals, etc.

XV. Guiding Principles of E-government Monitoring

E-government monitoring must be

1. Citizen-centric

2. Simple

3. Timely

4. Relevant

5. Accurate

6. Flexible

7. Action-oriented

8. Web-based

9. Top management-oriented

10. Specialised

Part B: EvaluationI. Historical Background of Evaluation

Systematic Evaluation is older than Centralised Monitoring Evaluation came into being in 1952 when Programme Evaluation Organisation (PEO), an independent organisation was set up in Planning Commission to evaluate India’s Community Development (CD) Programme PEO survives till date (2008)

EvaluationII Programme Evaluation Organisation (PEO) Headed by Adviser (Evaluation), it has 3-tier Structure 1. Headquarters- Planning Commission 2. Regional Evaluation Offices (7) (Kolkata, Chandigarh, Chennai, Hyderabad,Jaipur, Lucknow and Mumbai) 3. Project Evaluation Offices (8) (State Capitals- Guwahati, Bhubaneswar, Shimla, Bangalore,Bhopal, Patna, Thiruvananthapuram and Ahmedabad)

III. Functions of PEO

Undertakes evaluation of selected programmes/schemes under implementation Evaluation studies, assess the performance, process of implementation, effectiveness of the delivery systems and impact of programmes.

IV Four Generations of Evaluation

SN Generation Name Focus Role of Evaluator

1 First Generation

Measurement Measuring Instruments

Technical

2 Second Generation

Description Strengths and Weaknesses

Describer

3 Third Generation

Judgment Reaching Judgements

4 Fourth Generation

An Alternative Approach

Response (Interaction) & Construction (Methodology )

Mediator (among conflicting stakeholders)

Source: Guba and Lincoln 1989

V What is not Evaluation?

Analysis ≠ Evaluation Measurement ≠ Evaluation Assessment ≠ Evaluation Appraisal ≠ Evaluation Audit ≠ Evaluation Monitoring ≠ Evaluation

What is not Evaluation?

S.N. Tool Focus

1 Analysis Breaking into parts

2 Measurement Metrics

3 Assessment Cost and Benefits

4 Appraisal Investment

5 Audit Rules and regulations

6 Monitoring Implementation

VI. Four Senses of Term Evaluation

S.N. Sense Definition

1 Sense 1 Merit, Value or Worth of Something

2 Sense 2 An autonomous discipline: Study and application of procedures for doing objective and systematic evaluation

3 Sense 3 Work done by professional evaluators

4 Sense 4 Calculation of the value of an expression (in Mathematics)

VII. What is then Evaluation?

“…a robust arena of activity directed at collecting, analyzing, and interpreting information on the need for, implementation of, and effectiveness and efficiency of intervention efforts to better the lot of humankind.”

---Rossi and Freeman (1989, p-13)

VIII. What is E-government Evaluation

E-government evaluation is a systematic, objective, planned and participatory exercise undertaken during the design, implementation and after completion of a project for determining the worth of e-government to citizens against pre-set objectives and criteria for improving e-government services to citizens.

IX. Types of Evaluation

1. Formal and informal evaluation

2. Insider and outsider evaluation

3. Compliance, effectiveness, significance and efficiency evaluation

4. On-going, process or concurrent evaluation

5. Ex ante and ex post evaluation

6. Ad hoc evaluation

7. Evaluation in vivo

8. Pilot project evaluation

9. Terminal evaluation

10. Formative and summative evaluation

11. Meta evaluation (evaluation of evaluations) (Stufflebeam 1981/ Scriven 1991)

Types of Evaluation12. Scientific evaluation

13. Auto or self evaluation (ACCORD 1993)

14. Thematic evaluation

15. Individual, group and institutional evaluation

16. Casual everyday evaluation (Frutchey 1959)

17. Self-checking evaluation (ibid.)

18. Do-it-yourself evaluation (ibid.)

19. Systematic evaluation (Rossi and Freeman )

20. Fourth generation evaluation (Guba and Lincoln 1989)

Types of Evaluation21. Adversary

evaluation

22. Empowerment

evaluation (Fetterman et al.(eds.)(1996)

23. Utilisation-focused evaluation (Patton 1997)

24. Transparency

evaluation

25. Citizen evaluation

26. Right to information evaluation (being undertaken by Adam Smith International/ Administrative Staff College of India) (2007)

27. Evaluation research

28. Online and offline

evaluation

29. E-government evaluation

30. E-government special studies,etc.

X. Evaluation Timing and Diffusion-Adoption

CurvesHypothetical Curves

Time (months/years) 1 2 3 4 5 60

XI. What is to be evaluated?

I External Environment

III Service DeliveryII Organisation

Domains of E-government Evaluation

XII. How e-government domains are evaluated?

Indicators

1. Input indicators

2. Output indicators

3. Usage/Intensity indicators

4. Impact/Effect indicators

5. Environmental/Readiness indicators(Source: Jenssen 2005)

Indicators1. Input Indicators

– Amount of financial resources devoted to eGovernment. Absolute figures, per capita figures.

– IT/e-Government spending as % of GDP.– Amount of resources devoted to Research

and Development.– Amount of public resources devoted to

internet infrastructure.

Indicators

2. Output Indicators

– Number of online services for citizens;

– Number of online services for businesses;

– Percentage of government departments that have a website;

– Percentage of government websites that offer electronic services.

Indicators3. Usage Indicators

– Number of individuals that have made use of electronic services offered;

– Number of businesses that have made use of electronic services offered;

– Percentage of citizens that has visited government websites to search for information;

– Number of businesses that have made payments online;

– Percentage of internet traffic that pertains to electronic service delivery.

Indicators4. Impact Indicators

– reduction of waiting time at government counter x by y %;

– decrease in case processing time at government organisation x by y %;

– citizen/business satisfaction levels concerning eGovernment;

– survey-type questions, e.g.: ‘do you feel more positive to your government, now that you can contact it by email?’ ‘has your government become more efficient, now that you can perform services online?’

Indicators5. Environment Indicators

– ICT penetration rates (pc, internet, mobile phone) private households, work, schools;

– Indicator that measures ‘fear of invasion of privacy’;

– Online shopping rates as an indicator of trust in online environments;

– Indicator that measures ‘quality of legislation concerning the information society’;

– Telephone tariffs, GSM tariffs, Internet access tariffs.

XIII. Approaches to Evaluation Studies

1. Cross-Sectional Studies (Comparison of a group with treatment with another group without treatment)

2. Longitudinal Studies (Comparison of a group before and after treatment)

3. Benchmarking Studies (Comparison with best practices)

4. “Value Addition” Studies (Accenture)

Approaches to Evaluation Studies

5. Citizen Satisfaction Studies (American Consumer Satisfaction Index-ACSI)/ Citizen’s Report Card -Public Affairs Centre (PAC), Bangalore6. Department of Information Technology (DIT)’s Evaluation Assessment Framework (EAF) 7. U.S. Office of Management and Budget

(OMB)’s Program Assessment Rating Tool (PART)

1. DIT’s Evaluation Assessment Framework (EAF)

EAF Version 2.0 (2004)

1. Service Orientation

2. Technology

3. Sustainability

4. Cost Effectiveness

5. Replicability

Weights for Attributes

Rating of E-government Projects in India 2005-06

SN Rating Number of

Projects

Percentage

1 Extremely Good (EG)

2 Good (G) 17 46

3 Satisfactory (S) 1 3

4 Poor (P) 3 8

Total 37 100Source: DIT

3. U.S. Program Assessment Rating Tool

(PART)1. Clarity of Purpose and Well-Designed

2. Strategic Planning (valid annual and long-term goals)

3. Management (program, financial oversight and program improvement efforts)

4. Results (accuracy, consistency)

(Source: ExpectMore.gov)

Assessmet of U.S.Federal Programs by “PART”

(2008)

Number of Programs Assessed

Effective

Moderately Effective

Adequate

Ineffective

Results Not Demonstrated

DISTRIBUTION OF PROGRAM RATINGS

Source: ExpectMore.gov

XIV. Who will evaluate?

First decide whether evaluation will be done in-house or by an outside agency

In-house evaluation is preferable as it builds evaluation capability in-house

No separate in-house evaluation unit is required Entrust the evaluation function to the in-house

monitoring unit, suggested earlier And call it monitoring and evaluation (M&E) unit Let outside agencies also undertake evaluation

and special studies after a gap of 3/5 years

XV Which Type of Evaluation is Suited Most to E-government?

The type of evaluation will depend upon the specific requirements of an e-government project.

There are two key stakeholders in e-government: 1.E-government Management, and 2. Citizens

There are four standards for evaluation: 1.Utility 2.Feasibility 3.Propriety 4.Accuracy

E-government evaluation must meet the following two criteria: (a) Utility (to stakeholders-Management and Citizens) and (b) Actual Use, both geared to serve citizens

Which Type of Evaluation is Suited Most to E-government? Utilisation-focused Evaluation (Patton 1997) meets our

criteria of selection of type of evaluation Mere provision of government services online is not e-

government The online services must be utilised by the target group- the

citizens The issue of impact of e-government will arise only when the

following is satisfied:

PROVISION UTILISATION IMPACT (of e-gov (by citizens) (on general well- services) being of citizens)

Part C: E-government M&E

I. E-government Monitoring and Evaluation (M&E)

M&E is an under-developed aspect of E-government

It has so far not found any systematic application in e-government project implementation

M&E findings, where available, are not widely diffused

Its neglect hampers e-government development M&E is a tool for development of E-government

II.E-government Monitoring and Evaluation

(M&E) Unit E-government M&E unit will have the same

staff as the monitoring unit A part of the organisation, it will report

directly to top management It will undertake regular monthly

monitoring of e-government project and evaluation six-monthly or annually

It will give 80% weightage to monitoring and 20% to evaluation

III. Relative Weights to Monitoring and

EvaluationRelative Importance of M&E

Initial Stage

After 4/5 years

IV. E-government M&E Framework: Components

The E-government M&E Framework consists of five components:

(a) E-government Management

(b) E-government M&E Unit

(c) Information Needs Matrix

(d) E-government M&E Cycle

(e) Citizens

IV(a). E-government Management

Supporting Staff

Top Management

Middle Management

Information Technology (IT) Department

IV(b). E-government M&E Unit

Head of M&E Unit

Database Administrator/

System Analyst

Statistician Economist

Political Scientist/

Sociologist

Top Management

IV(c). Information Needs Matrix

Information Needs of Management

Information needs differ in three levels of management hierarchy: 1.Top Management 2.Middle Management 3. Supporting Staff

Undertake information needs analysis of various levels of management hierarchy

Prepare Information Needs Matrix

Information Needs Matrix

Information Needs

Department

Citize

eds In

isatinal N

s Info

Infrastru

cture In

Citize

n Acce

ss Info

Utilisa

tion In

Inform

Top Management

Middle Management

Supporting Staff

IV(d). E-government M&E Cycle

4. Evaluate it

2.Implement it

3.Monitor it

5. Review it

1. Prepare

E-business Plan

M&ECycle

IV(e) Citizens Citizens interact with

government in following four ways: As

(a) Information

Seekers

(of government

activities)

(b) Service Users (of

public services)

We want serviceWe want service

Citizens(c) Beneficiaries

(of public programmes like NREGP)

(d) Compliers

(with laws, rules and regulations

like payment of taxes)

(e) Stakeholders

(in public policies and programmes)

Their needs in these capacities have to be identified and met by E-government M&E

V. A Framework for E-government

Monitoring and Evaluation We are now in position to link these

components and presentA Framework for E-government

Monitoring and EvaluationThe Framework conceptualises the

complex reality of e-government and provides a Roadmap for E-government M&E Unit

Here then is the Framework.

A Framework for E-government

Monitoring and Evaluation

Information Needs

Department

Citizen

s Info

rmatio

anisatin

al Need

Infrastru

cture In

Citizen

Access In

Utilisatio

Review

rmatio

Top Mgt

Middle Mgt

Supporting Staff

Evaluate

Review

PrepareE-business Plan

Implement

Monitor

Formal Sources of Information

ICT Indicators Forecasting Monitoring & EvaluationCensus & Surveys Audit

Discussion Group

Social Sites

Legislature RTI Act

M&EUnit

IT Deptt

E-business Plan Implementation Sustainability Citizens

M&E Unit Information Matrix M&E Cycle(d)(a) (c)(b)

SupportingStaff

Top Management

M&ECycle

Citizens

We want service

Contribution of M&E to Good Governance

Monitoring information and evaluation findings can contribute to sound governance in a number of ways:

1. Evidence-based policy making (including budget decision making),

2. Policy development, management, and accountability.

3. Many governments around the world have realized much of this potential, including most OECD countries and a small but growing number of developing countries. (Source: Mackay 2007)

Contribution of M&E to Good Governance

Examples of well-functioning government M&E system: (Source: Mackay 2007)

1. Australia (by 1994, almost 80 percent of new spending proposals relied on evaluation findings)

2. Colombia (which has about 500 performance indicators)

3. United Kingdom (Public Sector Agreements between the Treasury and each of the 18 main departments)

4. U.S.A. (PART (Program Assessment Rating Tool), created in 2002, rates all 1,000 government programmes), and

5. Chile (whose Finance Ministry collects 1,550 performance indicators).

Part D: My Questions

With this I end my presentation but ask the following questions

1. What is monitoring? How does it differ from other sources of management information?

2. What is evaluation? Describe different types of evaluation. Which type of evaluation is most suited to e-government? Give reasons for your answer.

My Questions3. What role can an e-government

monitoring and evaluation (m&e) unit play in successful implementation of an e-government project?

4. Do you agree that development of a framework for e-government monitoring and evaluation (m&e) can provide a useful roadmap for m&e unit? Give reasons for your answer.

Your questions now!

Thank you for your attention.

Have a nice day.--Dr D.C.Misra