1

SOCIAL RESEARCH ASSOCIATION IRELAND/IRISH EVALUATION NETWORK

SEMINAR

GOOD PRACTICE IN COMMISSIONING RESEARCH AND EVALUATION

10TH JANUARY 2006

Contents of Presentation

2

a “supplier” perspective

nature of evaluation

typical stages in procurement process

common challenges, needs

some issues for future

Aspects of “Supplier” Perspective

3

evaluation more than research

external supplier perspective (not internal procurement)

a commercial consultancy perspective (not research grants)

multiple types, contexts

a small country perspective

emphasising common issues

Multi-dimensional Nature of Evaluation

4

Managing the team

Stakeholder Involvement

Research, Analysis

Time, resources, budget

Client, user relations

EVALUATION

A Balancing Act

Typical Stages in Procurement

5

2. Terms of Reference,

Brief

6. Reporting

1. Policy issue,

topic need

3. Invitations, Tendering

4. Selection, Contracting,

Inception

5. Managing, Undertaking,

Analysing

Common Challenges, Good Practice

6

1. Policy issue, topic, need lack of clarity lack of client consensus

clarity, consensus spell out objectives

2. Terms of Reference, Brief

problems of no.1 flow in here over-specifying methodology “piece of string” problem

don’t rush the ToR focus on objectives

3. Procurement, Tendering poor procedures overly formalised competitive

procedures which prevent dialogue

4. Selection, Contracting, Inception

disproportional procedures rush to get started

proportionality don’t rush the Inception

Challenges Good PracticeStage

Common Needs, Challenges and Good Practice (Cont’d)

7

5. Managing, Undertaking, Analysing

unrealistic deadlines absence of data data collection v analysis multiple stakeholders “scope creep”

good project management role of Steering Committees? avoid surprises

6. Reporting emphasis on written “tomes” difficulty finalising reports “reversing” compromises into

reports

separation of consultants’ recommendations from policy decision

Challenges Good PracticeStage

Procurement Procedures, Tendering (balancing competition and the need for dialogue)

8

need for better procurement skills, e.g. training, guidelines

inviting too many bidders/overuse of e-tenders website?

need for guidance on scale

circulation of all replies to all bidders!

ability, availability of commissioner representatives

see also www.fitzpatrick-associates.com (tendering)

Wider Issues for the Future

9

extent of evidence-informed, evaluation culture

the need for research, evaluation to “speak” to policy makers

need for more basic, neutral data collection

balance between “independence” and “relevance”

emphasis on costs of research/evaluation v costs of poor policy decision

more inter-disciplinary research, evaluation (e.g. “economic” v “social”)

over-evaluation of some areas, under-evaluation of others