Article: Problems of Policy Evaluation in a Multi-Level ...Problems of policy evaluation in a...

Problems of Policy Evaluation in a

European Multi-Level Governance Context:

The case of Active Labour Policy in Belgium

Paper for the UKES / EES Joint International Conference

Evaluation in Society: Critical Connections

London, UK, 04-06 October 2006

Stream 3:

Evaluation and the European Project

Session 5.07

Bart De Peuter

Joris De Smedt

Public Management Institute

University of Leuven

Belgium

Contact:

Joris De Smedt [email protected]

Bart De Peuter [email protected]: +32 16 32 32 73

[email protected]: +32 16 32 31 80 Fax: +32 16 32 32 67 www.publicmanagement.be

Public Management Institute Faculty of Social Sciences University of Leuven E. Van Evenstraat 2A 3000 Leuven Belgium

mailto:[email protected]

http://www.publicmanagement.be/



Problems of policy evaluation in a European Multi-Level Governance context: The case of Active Labour Policy in Belgium

Abstract

Within the European multi-layered governance system each of the increasingly intertwined governmental levels is subject of structural change and/or policy-related and financial modernization. However, these simultaneous reforms do not end the existing horizontal variance within each sub-European level concerning structures, powers and resources. Structural complexity, horizontal diversity within the lower levels and scattered powers over the different tiers of government are the main characteristics. Seen from a European perspective horizontal differences between countries remain, and within most of the EU member countries differentiation also exists between states, between regions and between local governments. This differentiated reform context entails in particular challenges in terms of capacity and coordination across governmental boundaries with regard to vertical policy evaluation. This paper addresses these issues in the field of active labour policy, within the federal country Belgium. Here, in light of the European objective to obtain an employment rate of 70% by 2010, the Flemish state level is the pivotal tier of government in this policy field, operating within a federal context while steering itself local governments. This case presents a typical snapshot of the structural complexity and intertwined powers between different governmental levels which entail difficulties for policy evaluation. The paper addresses the necessity, definition, possible modes and challenges of intergovernmental evaluation. It examines how these apply to the field of active labour policy in the above mentioned context. To conclude, the paper will identify critical success factors for creating synergies between the evaluation capacities of different governmental levels for vertical cross-border policy evaluation.

Bart De Peuter & Joris De Smedt


1. The European Union: crossing borders and boundaries toward increased cooperative policy making through diverse methods - the case of active labour policy Policy-making and implementation in the EU context is a continuous interplay between the European institutions (Council, Commission and Parliament), the member state governments, regional actors and interest groups. Over the years the European level expanded its role in a number of policy areas and became involved in new policy areas, introducing again new alternative forms and instruments of intergovernmental policy-making in addition to the “community method”, e.g. by The Maastricht Treaty of 1992. The creation of diverse and alternative methods of European governance across the wide range of policy areas stems from the fundamental mechanism underpinning the integration process. This mechanism can be visualized by a pendulum swinging between two magnetic fields, the one country-based and the other trans-national (Wallace & Wallace, 1999). Depending on the varying strengths of these two fields, there will sometimes be a propensity for trans-national policies to be adopted and sometimes national ones. How far the pendulum turns in one direction, or the other, depends on the policy area. Also, the movement of the pendulum varies over time: sometimes regular, sometimes erratic, sometimes sustained, and sometimes stationary. Perceived inadequacies of the member state as the governmental level to solve problems, globalization and the (historically grown) specificity of the European continent are external pressures for crossing the borders and boundaries of the Member States and establishing supranational policy cooperation. In this paper we focus on one of the new policy fields in which the European level has gradually built up its own role: employment policy and more in particular active labour policy.

• Development of a European dimension in the employment and active labour policy With regard to employment policy a European wide policy has been recommended since challenges cross the borders and boundaries of Member States. In this policy field the European Union evolved from a player participating from the sideline to the driving force behind the employment policy process (Wallace & Wallace, 1999). The European dimension in employment policy is the result of two major initiatives separately introduced: the European Employment Strategy and the Lisbon Agenda. Describing the key moments in the simultaneous developments of both initiatives (Table 1), this paragraph outlines the growing role of the European level in this policy field.

European Employment Strategy (EES)

The Lisbon Agenda

1997 The launch of the EES 2000 Mid-term review The launch of the Lisbon Agenda 2002 Impact evaluation Reinforcement of the Lisbon Strategy 2003 Revision of the EES 2004 Enlargement of the EU Mid-term review 2005 Gear up of the EES New start of the Lisbon process

Source: http://europa.eu.int/

Table 1: Key moments in the development of the European employment policy

In 1997 the Heads of State and Government realized the need to undertake common action at the European level. The development of a common strategy was explicitly incorporated in the European legislation, more specifically in the Amsterdam Treaty (European Union, 1997). At the Luxemburg Jobs Summit they officially launched the European Employment Strategy (EES), based on previous disjointed actions and separate plans. From the outset, the clear

Bart De Peuter & Joris De Smedt 1

http://europa.eu.int/


ambition of this initiative was to achieve considerable progress within five years through the creation of more and better jobs and lower unemployment. With the EES a new form of inter-governmental cooperation was introduced: the open method of coordination relying on a common policy framework of so-called “Guidelines” developed by the Commission and the voluntary engagement of the Member States which are free to choose the implementation instruments.

In the year 2000 the moment had come for a mid-term evaluation of the EES. Some strengths and weaknesses were defined. The largest contribution of the strategy was the development of a general framework by the Commission which created several synergies across the different aspects of the employment policy. In turn this led to an increased involvement of actors both at European and national levels. It also increased transparency of employment policies (European Commission, 2000a). In spite of these positive results, some weaker points were also identified. For example, increasing differences between Member States and Regions as to labour market performance were noticed. Moreover the Regional and local governments needed more opportunities to participate in the implementation of the Guidelines. Several initiatives were undertaken by the EU to improve its strategy.

Simultaneously with the evaluation of the EES, the European Council launched additional general objectives. In March 2000, simultaneously with the EES evaluation, the EU Heads of States and Governments met in Lisbon and launched the Lisbon Agenda. With this comprehensive agenda they aspired to make the EU "the most competitive and dynamic knowledge-driven economy by 2010" (European Council, 2000). Both the protection of the current social model and maintaining the EU as one of the leading economies in the world were aimed for. One of the main priorities of the comprehensive agenda is to achieve an EU-wide employment rate of 70% in 2010. Finally this approach aimed to develop the necessary peer pressure between the Member States to achieve the objectives.

In 2002 the Commission carried out an impact evaluation of the EES in order to analyze the impact of 5 years of the EES on the initial objectives. The evaluation was rather critical on the level of achievements made (European Commission, 2002b). In the same year, at the Barcelona Summit, the European Council reinforced the Lisbon Agenda to improve the coherency of the policy cycle.

In response to the evaluation results of the EES, the Union reconsidered its strategy in 2003 (European Commission, 2003). The renewed EES has three central objectives; complete employment, quality and productivity at work and the implementation of an inclusive labour market.

The year 2004 was totally dominated by the enlargement of the Union and the mid-term review of the Lisbon Agenda (European Commission, 2004b). The conclusions indicated a number of large shortcomings. In general the evaluation revealed that the Member States are moving too slow: some European leaders have still not been convinced of this general approach.

At the beginning of 2005 the European Union became aware of the very difficult situation: an aging population and a decline in economic performance (European Commission, 2005b). To withstand these difficulties, the European Commission launched the proposal to reactivate the Lisbon strategy and to reorient the EES. Both initiatives, the EES and the objectives of the Lisbon Agenda concerning employment, were brought together. In addition, the EES has incorporated the employment objectives of the Lisbon Agenda. Both programs must reinforce each other in order for the Union to achieve the objectives.



• The Belgian employment policy: the division of powers at work Once the EU has pronounced the EES and the Lisbon Agenda, the Member States need to take action. The Belgian Government has engaged itself in the European objectives. This paragraph addresses the complex Belgian state structure with three levels resulting from several constitutional reforms. Finding solutions to activate the labour market requires a whole range of measures and programs. In the Belgian case three governmental levels are involved in the employment policy implementation. Belgium consists of a federal level, and a regional level with three “Communities” and three “Regions”, and a local level, each with its own powers. For the rest of this paper, we only discus the powers of two governmental levels: the federal and regional level.

The Federal State

The Federal Government retained a considerable quantity of competences. It represents Belgium and the federal institutions in the European Union and the NATO. In the case of employment policy, the Federal Minister of Work represents Belgium in the EU institutions. The Federal Government still has very important levers in the employment policy. Moreover the Federal level designs the policy framework and develops the labour legislation. The Federal Minister of Work is responsible for the employment legislation, the collective labour agreements, the unemployment benefits and the determination of labour cost levels. To implement the federal legislation the Federal Government has to cooperate with the Regions and Communities since most of the policy implementation falls under the authority of the Regions and Communities.

The actual implementation of the European employment policy, i.e. the “Guidelines”, within the Member States starts with the development of the national action plans (NAP). In Belgium, the Regions design their plans to address specific needs and concerns. The Federal action plan is the addition of the Regional action plans. Bringing the different plans together is a real challenge. The situation and the problems in the Regions and Communities differ strongly. Each government wants to design a plan in order to solve the problems they face. The Federal Government organizes an intergovernmental conference in order to develop the federal plan based on the regional plans. It is also the task of the Federal level to assemble regional and federal efforts and results into one report. As mentioned earlier, the Federal Government has to report on the Belgian efforts and results to the European Commission.

The Flemish Region and Community

The Belgian state consists of three Communities and three Regions (www.fgov.be). The Communities are language based, resulting in, a French-, a Dutch- and a German-speaking Community. The Communities’ powers are based on the language and the language is personal. The territories of Regions and Communities do not correspond perfectly. For example, the Dutch Community exercises its powers in the Flemish Region and in Brussels; the French Community in the Walloon Region and in Brussels, with exception of the German-speaking part in the province of Liege, which forms the German-speaking Community.

The Regions instead are based on territory: the Flemish Region, the Walloon Region and the Brussels-Capital Region. The Regions and Communities have legislative and executive organs, a council and a government. To prevent an increase of institutions and members of parliament the Flemish Community and Region were institutionally merged; one Flemish parliament and one Flemish Government exercise both the Regional and Community authorities in Flanders.



Although the Flemish administration has to play a key role in preparing, applying and assessing the Flemish labour market policy, the Flemish Government has almost no legislative powers. The most important responsibilities of the Flemish Minister of Work are: managing the Flemish employment and work programs, granting working papers to non-Belgian workers, offering incentives to part-time work and recognizing employment agencies. This scope demonstrates the important role of the Flemish Government in the coordination of the Flemish employment policy. By developing the Regional action plans, the Regions contribute in considerable measure to the Federal action plan. The Flemish contributions to the Federal employment action plan are further elaborated in so-called territorial agreements, e.g. the Vilvoorde Pact, expressing the engagement of the Flemish Government and the social partners to the Lisbon process (Flemish Government, 2004).

One of the main objectives of the Lisbon agenda is full employment by 2010 (European Council, 2000). The Flemish Government is fully engaged to obtain this objective. It develops a whole range of measures to reach the objectives, most of them implementing Federal legislation, e.g. the cooperation agreement designed by the Flemish and Federal Governments concerning the ‘service vouchers’ for personal services. The Federal Government creates the legal framework after which the Flemish Government designs and implements the appropriate measures adapted to the Regional context. Although Flanders performs well in comparison with the other Belgian Regions, it is characterized by a low activity rate of older employees and some weak points in the field of education. The employment rate in the population aged 55 to 64 is only 25% (Flemish Government, 2004). Various measures need to be taken by the Flemish Government to improve the employability of this age group. Encouraging older people to keep on working is one of the main priorities of the Flemish Government.

The Social Partners

The different Governmental levels are not the only players in the Belgian labour policy area. Belgium has a strong tradition to design and implement the policy in cooperation with the social partners. These are the representatives of employers and labour unions. They participate both in the policy-making and the implementation process. The federal social partners and institutionalized consultation organs have an important consultative role in the federal employment policy design (Federal Government, 2003). The different Regional joint committees have the same function with regard to the Regions. The social partners have a substantial impact on the policy process; their support to ‘activate’ the labour market is an important success factor. They are important stakeholders; obviously they should play an important role in the cross-level evaluation.

• Challenge Actions at two distinct levels are important to investigate: the European Union in relation to the Member States on the one hand, and the relations between the governmental levels within the Member States on the other hand. Processes at both levels need to function optimally in order to implement the objectives successfully. In addition, evaluating policy across all these levels of government is a real challenge. The following part of this paper will investigate evaluation processes across both levels.



2. Intergovernmental evaluation: necessity, definition, modes and challenges In this part of the paper, we will elaborate the concept of intergovernmental evaluation (IGE) from a theoretical point of view. First we turn to the question why IGE is relevant. Secondly, we will define IGE. The third question addressed here is how IGE can look in practice. Therefore we will outline a scheme with theoretical profiles or modes of evaluation in an intergovernmental context. In connection to the scheme, we will explore particular challenges and issues with regard to carrying out an IGE. In the subsequent section of the paper, we shall examine the practice of IGE in the context of active labour policy within the European Union and Belgium and position it within the scheme.

• The necessity of IGE in the light of increased intergovernmental relations It is an understatement to say that the literature on intergovernmental evaluation is rather scarce. It seems the issue has systematically missed the attention of the evaluation community for the past forty years since it is not addressed in major text books. For example, it is not mentioned in the Encyclopedia of Evaluation, (Mathison, 2005), Evaluation, A Systematic Approach (Rossi, Lipsey and Freeman, 2004) nor in the International Atlas of Evaluation (Furubo, 2002).

This almost absence is remarkable for two reasons. First, the literature on evaluation has extended considerably in the period mentioned with a broad range of topics and aspects covered from different perspectives and approaches. Secondly, the literature has also matured in response to changing or new emerging challenges the public sector is confronted with.

One of these challenges is the search for efficient and effective coordination mechanisms in the face of the increased complexity of policy-making. On the one hand different policy fields become more and more tied up with one another, involving many actors which have to cooperate in dealing with complex societal problems. Next to this horizontal interlacement of policy fields, in many countries the different layers of government have also become increasingly intertwined in the policy making process. And this pattern does not exclusively concern federal countries. Rhodes (1999) for example pointed at a tendency in Western Europe toward intergovernmental policy-making between the central and local levels. In recent studies several authors concluded more specifically on a general trend toward partnership as the prevalent pattern of central-local relations in Europe (Baldersheim, 2002; Banner, 2002, John, 2000; Wayenberg, 2004). However, the division of tasks between the levels of government may differ across policy fields and will rarely seamlessly correspond with a clear distinction between (upper level) policy formulation and (lower level) policy implementation. In fact, in a multi-level governance context different levels also share responsibilities throughout the classical scheme of the policy cycle (preparation – decision – implementation – evaluation).

From the widespread tendencies mentioned, we conclude that the intergovernmental perspective on evaluation is both relevant and necessary, if not unavoidable in many circumstances. In this regard the slight attention the issue of IGE has received in the research literature constitutes a real gap.

• Defining intergovernmental evaluation Having established that intergovernmental evaluation is relevant both from a practical and theoretical point of view, we now need to define IGE more precisely. Along with Rieper and Toulemonde (1997: 2) we define an evaluation as intergovernmental if two levels of government interact during at least one stage of the evaluation process. The authors divide the



evaluation process into three general stages: the decision to evaluate, the implementation of the evaluation and the use of the evaluation conclusions. However, Scriven (1980) discerns 4 standard steps which have to be taken in each evaluation: (a) structuring, i.e. clarifying the evaluative questions and criteria, (b) observing (c) analyzing and (d) judging. Interaction between governmental levels can take place in one or more of these steps.

Thus, we determine an evaluation as intergovernmental when at least two levels of government interact during at least one of the six following stages:

1) deciding to evaluate

2) structuring the evaluation

3) observing

4) analyzing

5) judging

6) utilization

• Modes of evaluation in an intergovernmental context Leaving aside a detailed listing of all possible combinations depending on the presence or absence of interaction in each stage, we can discern the following ‘modes’ of evaluation within an intergovernmental context. The typology is based on observations in a number of case studies in Europe (Rieper and Toulemonde, 1997).

(1) Partnership mode

In this mode different levels of government are co-operating on an equal footing in more than one stage of the evaluation process. The intergovernmental character is most marked in this mode. The evaluation is structured on the basis of a common perspective which is negotiated and which fits with the agenda of both levels. Representatives of each level consult and jointly select the evaluation questions and criteria. The same interaction applies to the judging and utilization stages.

(2) Co-operative mode

Under this variant one level has the lead in several stages of the evaluation process. Likely the pivotal tier of government in terms of authority will take the initiative to evaluate and will set out the broad lines. Other levels of government will play a minor role in structuring the evaluation but can be considerably involved during the data gathering. Possibly an external evaluator is asked to carry out the evaluation. In the latter case, interaction between levels of government is only indirect.

(3) Asymmetric capacity mode

Often certain levels may lack personnel resources, financial means and/or knowledge to evaluate. When another (mostly upper) level of government feels there is a need for the lower level to have its policy evaluated or that this could strengthen the lower level, it may take the initiative to carry out the evaluation for the sake of the lower level which lacks adequate evaluation capacity and will be primarily involved during the judgment and utilization stages.

(4) Avoiding interaction

Despite the fact that different levels of government do interact with each other during policy planning and/or policy implementation, a particular level may deliberately avoid the cooperation or involvement of other levels at any evaluation stage. For example, a lower level



of government takes the initiative for its own purposes, but is not willing to trigger debate with upper levels on more fundamental aspects like the division of powers in that particular policy field.

(5) Competition through evaluation

This mode also does not meet the above definition of intergovernmental evaluation in the sense that there is no direct interaction between levels of government during one or more stages of a specific evaluation. Rather different levels will use their own evaluations on the same matter when they conflict on important aspects of the policy (e.g. choice of targets, instruments) or when one level does not rely on the findings of an evaluation commissioned by the other level.

The different modes present a wide variety of possible scenarios of evaluation in an intergovernmental setting. We conclude that only the first three modes of evaluation in an intergovernmental context comply with the definition of intergovernmental evaluation (IGE) outlined above.

• Challenges for the practice of intergovernmental evaluations As described above, six stages can be discerned with regard to an evaluation: decision, structuring, observing, analyzing, judging and use. Throughout the stages difficulties can arise which are characteristic of evaluating in an intergovernmental context. They can be grouped into two categories: political and technical challenges. Political challenges stem from the inherent characteristics of any governmental level. Technical challenges are less typical of the intergovernmental evaluation context, since multi-site evaluations can also encounter the same difficulties and some of the challenges can even be generalized to a single community evaluation. However, in an intergovernmental context some of the technical challenges can strongly relate to capacity problems of a particular level which impacts on the outlook of the intergovernmental evaluation, i.e. determining to an important extent the ‘mode’ of intergovernmental evaluation. Table 2 displays an overview of the types of challenges at each stage and examples of issues at stake which are discussed further below.

Stage Type of Challenges

Issues

Decision Political Powers and responsibilities, agenda setting, funding Structuring Political Evaluative questions, criteria Observing Technical Target group and area, data collection Analyzing Technical Data pooling, aggregation Judging Political Standards and norms, targets

Formulation of synthetic conclusions Use Political

and technical

Evaluation purpose (planning / implementation management / democratic accountability / policy learning) Reliability of results and (external) validity of conclusions

Table 2: Challenges and issues across the process of intergovernmental evaluation

During the stages of decision, structuring and judging the challenges will be primarily political in nature, whereas in the stages of observing and analyzing more technical problems can come up. The last stage of utilization feeds back to the decision and structuring stages, since the purpose of an evaluation conditions the use that can be expected of it (Chelimsky, 1997). However, both political and technical issues can constrain the use of evaluation findings.



A first inherent characteristic of each governmental level potentially causing difficulties for intergovernmental evaluations is the own sphere of powers. Depending on the constitutional setting, upper level legislation may have precedence over legislation of lower levels, and principles like subsidiarity may delineate the borders between the different spheres of authority.

However, governmental levels may want to influence and alter the existing balance of responsibilities, towards a larger amount of their own exclusive responsibilities or a larger part in so-called shared responsibilities. Evaluations might be perceived as either a threat or leverage in this balance of political power. In the most extreme cases, this may lead to the mode of competition through evaluations and counter evaluations. In more moderate situations of partnership or cooperation, the level of government most likely to embark on an evaluation may be the upper level of government. In such cases we can assume this level will also become the leading level in structuring the evaluation. The tendency to set the evaluation agenda often come from a higher weight in the distribution of powers, better financial conditions to fund an evaluation or scale effects to build own evaluation capacity in terms of personnel and knowledge.

Even when there is a joint initiative to evaluate, the fact that different levels of government are politically accountable to their own electorate can pose difficulties to the evaluation. These electorates may differ in terms of geographical space and volume. Each level may have legitimate but contradictory objectives relating to the same policy field. Problems may rise during the structuring stage. This is a crucial stage; the evaluation questions and criteria are defined which constitute the focus of the evaluation. Also the evaluation techniques to be used in the following stages are selected at this point.

Even when the same criterion is used, there can still be difficulties in fine-tuning the evaluation focus. If, for example, effectiveness is chosen to be the most important criterion, the evaluation will focus on outputs and outcomes and will assess to what extent the outcomes can be attributed to the outputs. However, in the case of intergovernmental evaluation different levels of government may have a different understanding of what the outputs and outcomes at stake are. The reason for this is found in the governmental chain of policy-making. Governmental levels with a higher ‘altitude above street level’ may consider a change in the behaviour of a lower governmental level as the outcome of its own intervention, whereas the lower level looks at changes in society and the target group when defining outcome.

When, for example a higher level of government develops a grant system for local authorities in order to steer them towards better quality in a particular service to the citizens, it can consider the number of applying municipalities or their quality of service as the outcome. However, the local authority will see its service delivery as an output and will assess outcome in terms of target group conditions, e.g. better living standards. The more intermediate levels are involved, the more the evaluation might focus on bureaucratic processes at the expense of social outcomes.

Applying the same criterion will not produce the same results when evaluated from different standpoints. An evaluator might be asked to gauge the added value of each governmental level to the ultimate outcomes and policy effectiveness. However to disentangle such complicated cause and effect chains is no easy task and will be generally more difficult to do than in the case of evaluations with only one level responsible. It is not a mere technical problem since it relates to the variety in role of the layers of government involved in the policy process. Each role is shaped from a distinct rationale.

When there is consensus about the questions and a common understanding of criteria and measures, there may still be diverging views on the level of ambition or target figures. This



will cause specific problems in the judgment stage of the evaluation when, based on a number of sub-evaluations, e.g. from all entities on the lower level of government, synthetic conclusions are to be made by the upper level on the overall results of a policy.

The extent to which the structuring of the evaluation is problematic will also depend on the general political and administrative climate. When there is a general pattern of partnership in the intergovernmental relations, it could be easier to include more sensitive issues in the evaluation. Otherwise, when there is an adverse climate, a way to avoid conflict would be to focus on routine problems, but as a consequence, perhaps leaving out the most urgent questions.

These kinds of choices on the focus of an evaluation will of course condition the possible use of the evaluation results. We can discern four different purposes of evaluation: underpinning of policy planning (ex ante), managerial use in policy implementation, accountability and policy learning (Chelimsky and Shadish, 1997; Worthen, Sanders and Fitzpatrick, 1997). When it is difficult to include sensitive issues in the evaluation, the focus may be limited to the question of whether the goals are reached or not. In that case, the evaluation results can only be used for management purposes and to a lesser extent for accountability. When policy learning is the purpose, the evaluation must include questions and criteria relating to policy effectiveness. This will only be possible when the governmental levels share important assumptions underlying the policy to be evaluated. With regard to accountability it can be said that competing evaluations are likely to gain less credibility. After all, the same effects are attributed twice to interventions of separate tiers of government, while in reality these levels need each other to trigger the effects.

The second category of problems is of a technical nature and can arise during the observation and analyzing stages of the evaluation process. In an intergovernmental context they are often related to general capacity problems of a particular tier of government.

With regard to data there are a number of possible hurdles to overcome when evaluating. The mere absence of data and the lack of monitoring systems may make it difficult to answer the chosen evaluation questions and to assess on every selected criterion.

When one level needs data collected by another level, there may arise problems with regard to data management. Different data processing and formats may hamper the exchange of data between levels. Problems of data compatibility may also cause problems during the analyzing stage of the evaluation when data have to be pooled or aggregated. The lack of common definitions of indicators prevents useful comparisons from being made.

Quality of data can also be an issue. Competing governmental levels distrust each other on this aspect for political reasons, even when technically the data meet the quality standards. However, data quality may also be problematic under the partnership and cooperative modes of intergovernmental evaluation. Reliability is concerned with whether the data gathering techniques produce the same results when repeated. Construct validity concerns whether the indicators provide a representative picture of the measured object. These aspects are important in every evaluation but in the context of intergovernmental relations they may be related to the evaluation capacity of a particular governmental level. For example, lower levels may not have the resources (human, financial, knowledge) to produce reliable and valid data for its own area. In that case, a higher level with more apt capacity should take on that task. Lastly, external validity of results can also be problematic because different lower level authorities may face different problems and require different policies to solve their regional or local problems.



3. Intergovernmental evaluation of active labour policy in Europe and Belgium: feasibility and limits In this section we will examine which of the above stated challenges occur when evaluating active labour policy in the EU multi-level governance context. First we will answer this question with a closer look at the interplay of the European and member state levels, secondly we will focus on the interaction between the federal and regional level within one member state, Belgium. We will run through the entire evaluation process identifying for each stage the political and technical challenges and issues at stake.

(1) EU - Member States

Decision stage

The decision to set up an evaluation and reporting mechanism in the field of active labour policy across levels of government was embedded in the development of a common policy framework within the European Employment Strategy (EES). The Commission deployed the EES after the European level received the responsibility from the Member States at the Luxemburg Summit in 1997. Evaluation within the EES framework follows a two step process. Member States evaluate on demand of the Commission and report evaluation results in their so-called National Action Plans (NAP’s), since 2005 called National Reform Plans (NRP’s). These are both retrospective and prospective, combining a review the implementation of the Commission’s EES policy guidelines with a statement of future actions to be taken. Secondly, based on the NAP’s / NRP’s, the Commission produces a Joint Employment Report (European Commission, 2005b and c, 2006) with an overall evaluation.

Structuring

Under the open method of coordination to implement the EES, Member States’ policy priorities are steered by the guidelines and recommendations underpinning the common policy framework developed by the Commission. However, the Member States preserve a free choice of measures to implement these European wide policy priorities. With regard to an intergovernmental evaluation the EES common policy framework has the obvious advantage of determining a common focus of the evaluation across the Member States, i.e. addressing the same evaluation questions and criteria. With regard to active labour policy the principal evaluation question relates to the outcomes of this policy. Key outcomes defined with regard to active labour policy are: the overall employment rate, the employment rate of women and the employment rate of older workers.

It should be noted that the principal evaluation criterion related to this question is goal attainment: the evaluation concerns the progress made on the levels of the three stated outcome variables. Of course, for their national evaluation exercise Member States may add other criteria, for example in order to evaluate on effectiveness as well. The latter criterion has been used once on a European scale for the mid-term review of the EES in 2002, which involved an evaluation of the impact of the EES on the national employment policies of each member state (HIVA, 2002).

Observing and Analyzing

Since the launch of the EES, indicators are used to monitor the progress made on the implementation of the European Commission’s guidelines. The Lisbon Strategy has defined clear targets for some of these indicators (see further) which underpin the analysis in the



NAP’s and the Commission’s Joint Employment Report. An advantage with regard to intergovernmental evaluation is the fact that these indicators are established and agreed at the European level. A working group on indicators assists the Employment Committee within the European Commission on the selection and development of these indicators (The Employment Committee, 2004). Selection is based on a number of criteria, e.g. policy relevance and clarity. Also the quality of procedures to gather and analyze the data is considered to be of major concern. Indicators must be based on reliable statistical data and preferably from European sources like EUROSTAT, the Statistical Office of the EU. The latter cooperates with the statistical offices in the Member States according to European regulation with provisions on the design of data gathering methods, and the receiving, processing and dissemination of data.

Concerning the overall employment rate for example, aggregates on the European level are compiled, based on absolute values from the figures of all Member States. This means no weighting is applied. Data are provided on a quarterly basis and annual averages are based on a simple average of the quarterly values. When country data are missing, interpolations for quarterly missing data are not published but only used for the compilation of annual averages and European aggregates (EUROSTAT, 2005).

The coordination of data management on the European level strengthens the necessary comparability between Member States. However, in some cases problems and debate have arisen with regard to indicator definitions. For example, the fact that not all Member States have a system of compulsory education till 18 years distorts especially the comparability of the figures for the activity rates of younger people (Ministerie van de Vlaamse Regering (2004).

Judging

Constructive to the judging stage is the use of common targets across all levels of government on the indicators relating to active labour policy. The pursued level of overall employment rate is 67% by 2005 and 70% by 2010. The target for the employment rate of older persons is set at 50% by 2010. These ambition levels apply for each Member State as well as for European Union as a whole (European Council, 2000).

Nevertheless, to compare Member States and to derive synthetic judgments on the European level appears to be difficult for two reasons. First, despite the common policy framework with regard to labour policy, the Member States face different problems on the field. For example with regard to the overall employment rate, one Member State may have relatively high rates of unemployment of younger people, while another member state, like Belgium, faces the challenge to raise the employment rate of older persons. For both target groups different solutions and reforms are required. Secondly, in order to assess and compare reform trajectories in different countries it is important to bear in mind that there are different starting positions (Pollit & Bouckaert, 2005). Two countries with an equal outcome level score may differ a lot with regard to the reforms already implemented, the rate of progression and the profile of progress (growing or slowing down). In fact, this diversity among Member States is acknowledged by the use of the open method of coordination. Nevertheless, diversity in problems across Member States limits considerably the evaluation scope from a European perspective. The Commission also tries to accommodate the diversity by developing specific recommendations and feedback to each separate Member State, in addition to its Joint Employment Report.



Use

From a European point of view, yearly results can be used to assess progress on the implementation of the Commission’s guidelines of the past period and to adjust priorities within the common policy framework of the European Employment Strategy and the Lisbon Agenda. On the other hand the Member States can be compared with each other in order to incite them to continuous improvement. In this perspective the Member States have to give account of their efforts made in the light of the European targets.

However this accountability mechanism has been less effective than expected. The mid-term review of the Lisbon Strategy revealed that the overall targets probably will not be reached by 2010 due to a lack of persuasive reform efforts and commitment from the Member States. Based on this review the Commission proposed a new partnership to re-launch the strategy, focusing the EU’s efforts on two principal tasks: delivering stronger, lasting growth and more and better jobs (European Commission, 2005a).

The introduction of a 3-year cycle for new guidelines also changed the role of the NAP’s –from 2005 on called National Reform Plans (NRP’s): in the first year of the new cycle the report is conceived as a planning document, after the second year the Member States must submit an intermediate report, while the third year will focus in particular on the evaluation of the implementation of the guidelines, and the progress against the Lisbon targets (Federale Regering, 2005).

(2) Belgium: federal - regional - local levels

Decision stage

Officially the EU only corresponds with Member States and thus, for Belgium, the federal level has to deliver the NAP / NRP in the framework of the intergovernmental evaluation of the European Employment Strategy.

From the start of the EES, the practice has been in Belgium to involve the regional level of government in drawing up the NAP. In the early years regional action plans were developed and added as attachments to the national plan. Later on the regional action plans were integrated into the national plan, in response to a demand from the Commission to gear up the responsibilities to each other and to obtain better coherence between the different levels of government which are responsible for employment policy in Belgium (Flemish Government, 2003).

Besides this two-step reporting upwards, each level of government is free to launch and fund additional evaluations, with an alternative or wider focus adapted to its own sphere of authorities. However, to date such complementing evaluations are very scarce, illustrating the absence of a real evaluation culture at either the federal, regional or local level in Belgium.

Structuring

With respect to reaching a unified focus in the evaluation of this policy field the large degree of distribution and cascading down of powers to the lower levels in Belgium is partly offset by the common policy framework developed by the European level. The domestic structuring of evaluation through the NAP / NRP is also backed up by efforts to gear up policies across the levels of government. Not only are regional reports integrated in the national document, but inter-ministerial conferences and cooperation agreements also try to mutually adjust measures between the federal level and the Regions and the federal level and the municipalities, enabling them to gear up evaluation questions and criteria. The European



policy framework provides both a drive towards intergovernmental cooperation and a minimal common focus in the planning and evaluation of employment policy in Belgium.

Observing and analyzing

The core of data to be used in the Belgian report is provided by the data gathering which is coordinated at the European level. Validity and reliability of data are not an issue. Within Belgium, these data are further broken down to the three Regions.

Nevertheless, there is an uneven monitoring and evaluation capacity between levels: the Regions are better equipped in this regard compared to the federal level. However, the Belgian impact study in the framework of the EES mid-term review, conducted by external evaluators, revealed that there is a persistent lack of data and only partial evaluations with regard to the policy field of employment. Therefore, it is very difficult to undertake evaluations on efficiency and effectiveness of the measures taken at the different levels of government (HIVA 2002).

Judging

The Federal Government faces the same difficulties as the European Commission when it comes down to formulating synthetic judgments. It can not go beyond general assessments of the efforts and general comments on the progress against the targets set out by Europe. Despite efforts to gear up policies to one another, political debate sharpens from time to time between the Regions on their respective efforts and policy measures. When one Region makes progress but another Region stays behind, the effect on the ‘score’ for Belgium is reduced. On the other hand, the Regions differ in starting conditions and sub-problems as the Member States do.

In order to display its distinct problems and efforts, the Flemish Region submits its own regional action plan to the Commission. Potentially, the Regions could go further in the analysis and judgment based on an elaborate monitoring of the effects of their own employment policy measures. However, as already stated, in the absence of a real evaluation culture the lack of particular data prevents sound judgments on advanced evaluation criteria like e.g. effectiveness.

Use

From the forgoing we conclude that within Belgium the federal and regional levels face similar limits with regard to the utilization of the evaluation results compared to the European Commission. First, progress of the implementation of measures can be gauged. Secondly, progress on the European-wide targets on key indicators with regard to active labour policy can be seen as an element of accountability; both to the own electorate and to other governmental levels, since cooperation is a necessary condition to make progress on the member state level.

The Regions can use data from other monitoring instruments on more and other aspects of their own employment policies to follow-up the implementation of tailor-made measures, which they developed either to implement federal priorities or within their own sphere of competence.



• Which mode of intergovernmental evaluation applies to the case? To conclude our exploration of the feasibility and limits for intergovernmental evaluation in the context of labour policy in the EU, we will now zoom out again answering which mode of intergovernmental evaluation applies.

Based on the analysis above, it becomes clear that the overall judgment on which mode applies, should not be considered as an exclusive label, but rather as the prevalence of a pattern since interaction may vary across the stages of the evaluation process. The interaction in the decision stage is cooperative with European Commission in a leading role. The structuring stage follows a similar interaction both between the European and Member State level and between the governmental levels within Belgium which do not compete by evaluations. Also the observation and analysis stages show cooperation albeit less political in nature compared to the previous stages. The judgment stage follows a two-step process of national and European assessments, followed by European feedback to each member state. With regard to utilization of evaluation results, there tends to be more interaction between the European and Member State levels compared to the interaction between the different levels within Belgium. The latter is focused on aspects related to shared responsibilities. The Regions themselves tend to avoid interaction as far as their exclusive powers in the active labour policy are concerned. Consequently, policy learning gets hardly a chance.

In general we can conclude that the practice of evaluation in the multilevel governance setting in active labour policy meets the definition of intergovernmental evaluation rather well: in more than one stage of the evaluation process, at least two levels of government interact with each other. The overall mode of intergovernmental evaluation is this of cooperation with the European level as the leading actor. Within Belgium the drive to evaluate coming from Europe leads to cascading decisions and incites to intergovernmental cooperation during the evaluation process. Europe also determines the outlines of the focus of the evaluation, coordinates the gathering of data on the common set of indicators but relies on the capacity within the Member States’ statistical offices, provides judgments and feedback on the overall progress and formulates recommendations for each member state.

4. Critical success factors to create synergies of evaluation capacity This section addresses a number of critical success factors which are relevant for the evaluation of policy to which several governmental levels contribute, as in the domain of active labour policy. These factors can be examined at two levels: on the one hand the relations between the European Union and the Member States, and on the other hand the relations between the different governmental levels in one Member State. The Belgian case illustrates the difficulties and challenges which occur within the evaluation process of employment policy in one Member State.

• Homogenous division of powers A first critical success factor concerns the division of powers among the governmental levels involved. First, a clear division of authority is required. It has been well defined in employment policy as far as the responsibilities of the European institutions and the Member States are concerned. The Commission takes initiative, designs the policy framework and Member States are charged with implementation. The Commission reports policy results of Members States to the European Parliament and the European Council.



The analysis of the distribution of authority between the different governmental levels in the federal state of Belgium reveals, however, immediately some major challenges. This division of powers has an enormous impact on the design and evaluation of the employment policy. Yet, both the Federal and the Regional governments have several powers in this policy field which are not always clearly defined or delimited. Two major problems can be distinguished. Separate initiatives often appear to fail. The federal legal initiatives should be based on and adapted to the Walloon, Flemish and the Brussels’ problems and priorities. But within the federal framework the Flemish government has hardly levers to design and implement a real Flemish employment policy adapted to the specific concerns and priorities. The second problem relates to the implementation of the European obligation to evaluate the employment policy. Each government carries out its own evaluation. The Belgian evaluation report is an aggregation of the evaluation of the Regions. The Belgian results are too generally formulated, in spite of the several initiatives undertaken in the employment policy by the Regions.

Therefore, not only a clear division of powers is needed, but also a division of powers into homogenous packages of responsibilities. In addition, each governmental level should contribute to the evaluation, whilst ensuring that the scope of the contribution is consistent with the level’s respective powers within the field of active labour policy.

• Common policy framework A second critical success factor for creating synergies of evaluation capacity among different governmental levels, is the existence of common policy framework. The European Employment Strategy (EES) and the Lisbon Agenda are common European initiatives, based on the open method of coordination. The Commission does provide for such a common framework, but Member States participate voluntarily. Therefore, this working method needs some peer pressure among the Member States, which often need to be convinced of the surplus value of these common initiatives. Some Member States are fully convinced and will take action with complete dedication. Others will do everything to slow down the process. Evaluating such a divergent arena of efforts and reached effects at the aggregated European level is not self-evident.

The Belgian level faces the same challenges as the European level. Coordinating the employment agendas and priorities of the Federal state, the Regions and the Communities experience some severe difficulties. It is almost impossible to harmonize everything because of the enormous differences between the given starting points of the governments. Nevertheless, the harmony between the measures and initiatives of the Regions, Communities and the Federal Government is required to attain the planned objectives. Instruments must be developed to streamline the consultation process between the governments. Bringing together the regional and federal agendas and policy formulation is also a critical element for the success of future evaluations.

• Evaluation culture and capacity In recent years the European Union and its institutions have worked hard to establish a ‘European evaluation culture’. The Union itself pays a lot of attention to this process. In 2004 the Union completed 143 European evaluation studies (Member States’ evaluations and evaluations of projects not included). The number of evaluation studies is increasing; about 140 persons are working in 30 evaluation functions (The European Commission; October 2005d). These are indicators of the growing attention of the Union to evaluation processes.



The European level itself is convinced of the surplus value of policy evaluations; the challenge now is to persuade all the Member States.

At this moment there exists no Belgian evaluation culture. (Varone, F., Jacob S., De Winter, L.; 2005). This absence has important consequences for the way evaluations are applied. Closely related to this problem is the absence of sufficient expertise. Both the Federal and Regional governments do not evaluate their policies systematically.

Nevertheless an evaluation culture strongly contributes to the quality and the use of evaluation results. Establishing an evaluation culture throughout a federal country, with diverging beliefs and attitudes towards evidence-based policy-making, is important in light of evaluations of common policies. In Belgium the federal level possesses the legal instruments, but it fails to create peer pressure among the Regions and the Communities. As a consequence, it does not contribute much in spreading an evaluation culture.

• Methodological coordination Methodological problems between the governmental levels are important factors of failure. At the European level, the collection of the data and the definition of concepts differs strongly between the Member States. For example, Member States define unemployment each in a different way. In Belgium someone will enter the unemployment statistics after an interruption period of three months. In other Member States this takes place after one month, one week or even one day.

In order to carry out methodologically proper evaluations which satisfy the quality standards, the European Commission has to further develop the common methodological framework. The design of common methodological requirements and comparable problem definitions are important success factors for cross-border evaluations amongst the Member States. At a lower level this problem also arises within each Member State. Coordinating the different indicators and problem definitions is a major challenge to obtain a European evaluation report.

Member States are obliged to collect information on indicators for the EU. In Belgium each governmental level within its powers has the responsibility to collect the required information. The major methodological problems are related to a lack of consultation between the governments and a lack of expertise. Each Region collects its own information. To manage the needs and problems some governments collect more information than asked for, others do not even collect the required information. The lack of (quality of) information must be solved by methodological coordination.

• Tailor-made feedback The European Union only recognizes countries as partners in the design, implementation and evaluation of the employment policy objectives. As a consequence for Belgium, European feedback is almost all directed to the Federal Government. But in Belgium the Regions contribute to a large extent to the policy planning, implementation and evaluation of the active labour policy. The Belgian report to the Commission contains some information about the Regions but is not balanced enough. Nor do the Regions receive tailor-made feedback from the Commission on their policy efforts. Yet, this is important to create and sustain ownership and commitment of the sub-national levels in light of the evaluation process.



• Consistency between the utilization and the responsibilities The European open method of coordination has important consequences for the evaluation process. The European Commission emphasizes the comparison of the results between the Member States. This approach has an undisputable impact and has a real ‘discipline effect’ on the Member States. The political success appears from the imitation of the method in other policy domains. But the evaluation and follow up of the objectives by the National Action and Reform Plans imposed by the Commission may not lead to ‘a beauty contest’ between Member States. Comparing the indicator scores is not the essence: it is more useful to look at the evolution of these indicators.

To guarantee the success of evaluations, the Commission has to guard its role as the driving force behind the employment policy. But it can only compare the results reported by the Member States and make conclusions on goal attainment vis-à-vis the common objectives and targets. However, it is difficult for the Commission to make synthetic conclusions on policy effectiveness. The limited possibility for international comparison is a bottleneck in the development of a more detailed common European employment policy and a true European employment policy evaluation. It is also almost impossible to solve. As mentioned earlier, the differences between the Member States are tremendous in terms of problems and suitable solutions. But the Commission could generate a database with information about good practices and relevant context information. This could stimulate policy learning across Member States.



Bibliography

Administratie werkgelegenheid (2002). Voorbereiding OESO bezoek ‘actief ouder worden’ op 23.04.02,

Administratie Werkgelegenheid (2003). Mededeling aan de Vlaamse regering, opvolging van het Lissabon proces: voortgangsrapport 2003.

Baldersheim, H. (2002). ‘Subsidiarity’ at work: modes of multi-level governance in European countries. In: Caulfield, J. and H. Larsen (reds.) Local government at the Millennium, pp. 203-213. Opladen/Germany: Leske and Budrich Publishers.

Banner, G. (2002). Community governance and the new central-local relationship. International Social Science Journal, 54(2), pp. 217-231.

Chelimsky, E. and Shadish W.R. (1997). Evaluation for the 21st century: a handbook Thousand Oaks (Ca): Sage.

Council of the European Union (2005). Draft Joint Employment Report 2004/2005 Brussels.

Euractiv (2005a). Labour Market Reforms.

Euractiv (2005b). Lisbon Agenda .

European Commission (2000a). Mid – term review EES 2000.

European Commission (2000b). Joint employment report 2000.

European Commission (2002a). Joint employment report 2001-2002.

European Commission (2002b). Impact evaluation EES 2002.

European Commission (2003). Joint employment report 2002-2003.

European Commission (2004a). Joint employment report 2003-2004.

European Commission (2004b). Delivering Lisbon, Reforms for the enlarged Union.

European Commission (2005a). Working together for growth and jobs A new start for the Lisbon Strategy.

European Commission (2005b). Joint employment report 2004-2005.

European Commission (2005c). Joint employment report 2004-2005 Addendum.

European Commission (2005d), Conference: Evaluations in the European Commission Demand and Supply: A debate. Bruxelles, Charlemagne Building, 17 October 2005

European Commission (2006). Joint employment report 2005-2006.

European Council (2000). Conclusions of the Presidency - European Council of Lisbon 23-24 March 2000.

European Union (1997). Treaty of Amsterdam.

EUROSTAT (2005). Harmonised unemployment - Eurostat Metadata in SDDS format: Summary Methodology. http://epp.eurostat.cec.eu.int

Federale Regering (2003). Europese Werkgelegenheidsstrategie. Nationaal actieplan voor de werkgelegenheid België. Brussel.

Federale Regering (2004). Europese Werkgelegenheidsstrategie. Nationaal actieplan voor de werkgelegenheid België. Brussel.

18


Federale Regering (2005). Lissabon Strategie. Nationaal Hervormingsprogramma 2005-2008 - Meer groei en meer werk. Brussel

Flemish Employment Administration (2002). An introduction to the Flemish labour market policy 2002 based on the Flemish Regional Action Plan for the European employment guidelines 2002. Brussels.

Flemish Employment Administration (2002). Flemish labour market policy 2002.

Flemish Government (2003). Flemish Region Action Plan, European Employment guidelines, 2003. Brussels.

Flemish Government (2004). Het pact van Vilvoorde in concrete cijfers. Brussels.

Furubo J.-E., Rist R.C. and Sandahl R. (eds.) (2002). International Atlas of Evaluation, New Jersy: Transaction Publishers.

HIVA (2002). Evaluatie van de impact van de Europese werkgelegenheidstrategie in België, Leuven: HIVA.

John, P. (2001). Local Governance in Western Europe. London: Sage.

Mathison S. (ed.) (2005). Encyclopedia of Evaluation, Thousand Oaks (Ca): Sage.

Ministerie van de Vlaamse Regering (2002). Vlaams actieplan, Europese werkgelegenheidsrichtsnoeren 2002, een Vlaamse bijdrage aan het Belgisch nationaal actieplan 2002. Brussel.

Ministerie van de Vlaamse Regering (2004). Vlaams actieplan, Europese werkgelegenheidsrichtsnoeren 2004, een Vlaamse bijdrage aan de Europese werkgelegenheidsstrategie 2004. Brussel.

Pollit C., and Bouckaert G. (2004). Public Managment Reform, New York: Oxford University Press.

Rhodes R. A. W. (1999). Control and power in central-local relations - second edition. Aldershot: Ashgate.

Rieper O. and Toulemonde J. (1997). Politics and Practices of Intergovernmental Evaluation, New Brunswick: Transaction Publishers.

Rossi P.H., Lipsey M.W. and Freeman H.E. (eds.) (2004). Evaluation - A Systematic Approach, Thousand Oaks (Ca): Sage.

Scriven M. (1980). The Logic of Evaluation, Inverness CA: Edgepress.

The Employment Committee (2004). Conclusions of the Employement Committee on the Indicators Group’s report on the indicators to be used in the Joint Employment Report. Brussels.

Van Wichelen L. (2001). Van nap naar rap en terug. De Europese werkgelegenheidsstrategie en de federaal - regionale samenwerking in België. Over Werk 3. pp. 15-22.

Van Wichelen L. (s.d.), Presentation, Preventive and active approach for the unemployed, Flemish Employment Administration.

Varone, F., Jacob S. and De Winter, L. (2005). “Polity, Politics and Policy Evaluation in Belgium”, in: Evaluation. 11, pp. 253-273.

Vlaamse Vereniging van steden en gemeenten. (2005) Partnerschapverdrag omtrent het lokaal werkgelegenheidsbeleid.

19


Wallace H. and Wallace W. (1999). Policy-Making in the European Union, New York: Oxford University Press.

Wayenberg E. (2004). Vernieuwingen in de Vlaamse centrale-lokale verhoudingen: op weg naar partnerschap? Een kwalitatieve studie van de totstandkoming en uitvoering van het sociale impulsbeleid. Doctoral thesis, Leuven: K.U. Leuven, departement Politieke Wetenschappen.

Worthen B.R. , Sanders J.R. and Fitzpatrick J.L. (1997). Program Evaluation: alternative approaches and practical guidelines, Reading (Mass.): Addison-Wesley.

20

Date post:	26-Aug-2020
Category:	Documents
Upload:	others
View:	0 times
Download:	0 times

Article: Problems of Policy Evaluation in a Multi-Level ...Problems of policy evaluation in a...

Documents