Evaluating ‘Social’ & Health Projects Presentation by Elliot Stern International Seminar on...

Evaluating ‘Social’ & Health Projects

Presentation by Elliot Stern

International Seminar on Professional Education & Health

Salvador, Bahia July 2006

Evaluating Social & Health Projects

Address three topics

• How evaluations in ‘social’ arenas such as vocational training in health are designed

and implemented• How we ensure evaluations are used or

institutionalised• What evaluation questions should be asked

in evaluations of professionalisation and social/healthcare


Evaluation strategies build on

two foundations:

• The rationale, purpose and types of evaluation

• The characteristics of what is being evaluated


Evaluation purposes include:

• Better programme planning and design• Improving programme management & delivery

• Demonstrating success in terms of results• Explaining and predicting what works

• Strengthening social networks and institutions

Some evaluations are about learning and some about accountability – much of which is about timing and

programme stage


Main types of evaluation -

• Allocative / Economic• Management / Performance

• Causal / Explanatory• Formative/Developmental

• Participatory


Underlying these types of purposes there are of Underlying these types of purposes there are of course value choices and prioritiescourse value choices and priorities

• According to whose priorities are evaluations conducted?

• Who benefits and can use the results?

There is always the question of who are the stakeholders and legitimacy of different groups

and interests


Matters of values are essentially political, moral and ethical questions, but it also

depends on what we are evaluating

Social, health and pedagogic and training programmes are not the same as social

security, transport or information technology programmes


‘Social’ has many implications for evaluation – ethical and pragmatic

• Social emphasises that the ultimate beneficiary of programmes are citizens and communities as well as officials and professional groups

• Social highlights that the perspectives of service users or beneficiaries of development programmes have knowledge and rights

• It makes it likely that programmes will be better designed, targeted and implemented if the voices of ‘social’ stakeholders are included

• Social policies and programmes are often multi-dimensional requiring a combination of interventions


This has implications for evaluation approaches and methods

• Need to use participative and consultative methods

• Expect ‘social’ stakeholders to have a voice in programme design stages as well as to

provide ‘user’ judgements• The need to monitor social aspects of the

programme implementation process • The need to use multiple methods and

synthesise data from many sources


Yet the term social in the world of social programming and policy making has still

wider meanings. It encompasses for example service provision and social development

And the problem with social development is that it cannot be predicted, it is not static – it develops, which also leads to problems for

evaluation


Evaluators who will at some point be asked to judge ‘success’, need to criteria to make

their judgements. These criteria are usually derived from programme objectives, but if

these objectives change or are unstable…….

Which is why specific approaches are needed….


• In social development programmes, process evaluations that reveal why objectives

change or mutate are often more useful than objectives driven evaluations

• Evaluation can also help to build capacity by empowering participants in social

programmes making it easier for them to participate, manage their role and clarify

their priorities and strategies


This introduces not only evaluation purpose but also questions the methods, style and

role of the evaluator

• Is the evaluator an observer? Technician? Participant/protagonist? Referee? Advocate?

In social programmes the technician/observer role is often difficult to maintain nor is it

effective


If evaluations are to be used and useful they need to be built in to the institutional

arrangements that manage and coordinate programmes

The growth of the (new) public management has seen a growth also in internal evaluation functions that are closely tied to programme

and policy management


Public management today is characterized by:

• Multi-level governance and adjustment to local as well as national conditions

• Interagency & public private partnerships• Temporal/organisational instability

• Active engagement with civil society• New consultative, dialogical and consensus

oriented arrangements• A pre-occupation with risks, efficiency and

performance


This creates demands for new skills and competencies from evaluators. Being able to:

• Translate evaluation data into policy advice• Build generalised conclusions from very

diverse implementation contexts• Speak directly to different civil society

groups• Synthesise the experience of different

stakeholders and ‘partners’


So evaluations have to look in two directions:

• Inwards to government & public managers enabling them to manage and adjust during

implementation and improve performance on a continuous basis

• Outwards to officials, decentralised administrations, partnerships and citizens so as to integrate their experience and maintain

a dialogue and consensus


I have spoken of some of the characteristics of social programmes. One of these is that they

are multi-dimensional, using many policy instruments in combination. This has

implications for where we get our evaluation questions from. I distinguish here between evaluation and monitoring. Evaluation tries to answer evaluation questions – and these are usually derived from formal bodies of

knowledge and from theory.


Two examples of evaluation questions I have about PROFAE and its successors that I have

not yet heard addressed at this conference

One concerns professionalisation and the other the evolution of the service society


Professionalisation implies:

• Self regulation rather than control by education or certification; the market;

managers; or by government regulation• Authority defined by ‘special’ knowledge and

a willingness of consumers to accept this authority

• Exclusive rights to undertake certain work


Professionalisation’ strategies for occupational control exist:

• ‘where the tensions inherent in the producer-consumer relationship are controlled by means of an institutional framework based upon occupational authority. This form of control occurs only where certain conditions exist ……’ (Johnson 1972)

These conditions include a large market of diverse consumers with ‘diverse’ interests, who are not

organised and are largely dependent. Where there is ‘a single client or a small group of powerful clients’ professional strategies of control are less likely to

be successful.


Irrespective of whether the objective conditions for professional strategies for occupational control exist, the evaluation

question remain: What are the consequences of increasing the status, authority and

capacity for self-regulation of any occupational group? What are the risks of

rigidities, demarcations and boundary disputes occurring between newly

strengthened professionalized groups?


A second example of evaluation questions for this and future evaluations of PROFAE type

programmes concerns the evolution of service economies and risks that this

evolution sometimes undermines autonomy and self-regulation among citizens and in

communities


Health innovation in Brazil attempt to combine two tendencies:

• Greater competence, training and skill among ‘professionals’, and

• Continued democratic, participatory and inclusive involvement of civil society

Good example the role of community agents in Atenção Básica alongside nurses with ever-greater

skills


So the evaluation questions that need to be asked are:

• What are the consequences of these two tendencies – professionalisation and

participation?• Is it likely that the long-term consequences of

‘professionalisation’ will be to undermine autonomy and self regulation within communities and introduce a ‘dependency’ culture that will be

costly and socially divisive? and,• If so how can this be avoided or reconciled?


In the last two examples I have been deliberately provocative. This is to illustrate the importance of theory as well as methods

in evaluation. That we must not only use appropriate methods to gather information

but that we must also ask the right questions


In a programme and policy arena as complex as PROFAE and its successors this requires

that we draw on a wide range of relevant theories: in public management, pedagogy,

public health, welfare and dependency theory, labour markets, professionalisation and social development to name but a few!


If evaluation is to become more than monitoring, contribute to programme management and social development

it must:

• Use diverse theory & methods matching the complexity of what is being evaluated

• Be inclusive and open to stakeholder voices from across civil society as well as the public sector

• Relate to processes as well as outcomes.• Address the needs of policy makers and public

managers whilst not undermining social inclusion and the voice of civil society


Good evaluation is not easy!

But innovative and exciting programmes of the kind that exist in the social and health

sectors in Brazil deserve equally sophisticated evaluation approaches.


Date post:	11-Jan-2016
Category:	Documents
Upload:	benjamin-potter
View:	212 times
Download:	0 times

Evaluating ‘Social’ & Health Projects Presentation by Elliot Stern International Seminar on...

Documents