ASSESSMENT OF TRIPLE BOTTOM LINE IN THE ACT GOVERNMENT

Evaluation of pilot

June 2012

© Australian Capital Territory, Canberra, June 2012

This work is copyright. Apart from any use permitted under the Copyright Act 1968, no part may be reproduced by any process without written permission from the Territory Records Office, Directorate Services, Territory and Municipal Services Directorate, ACT Government. GPO Box 158, Canberra City, ACT, 2601

Produced by: Policy and Cabinet Division, Chief Minister and Cabinet Directorate

Contact: Director, Economic, Regional and PlanningPolicy and Cabinet DivisionChief Minister and Cabinet DirectorateACT GovernmentGPO Box 158Canberra City ACT 2601phone: 13 22 81

This publication has been prepared by officers of the Chief Minister and Cabinet Directorate, ACT Government. It is believed that the information is correct and reliable, but neither the authors nor the Directorate give warranty in relation hereto and no liability is accepted by the authors or the Directorate, or any other person who assisted in the preparation of the publication, for errors and omissions, loss or damage suffered as a result of any person acting in reliance thereon.

Page ii

Table of contents

1. Background/purpose of the evaluation 1

2. Observations 1Positive findings 1

Negative findings 33. Analysis 6

Objectives 6Risks 8

Implementation dilemmas 9Other issues 10

4. Mitigating implementation dilemmas 135. Towards an inclusive, integrated and effective TBL assessment — conclusions 16

Key recommendations 16References 19

List of figures and tables

Figure 1: Decision points in an iterative policy cycle where TBL could be deployed 6Figure 2: Critical implementation dilemmas in the TBL process and their interaction 10

Table 1: ‘Risk hazards’ and triple bottom line assessment 9Table 2: Assessing policy values 14

List of acronyms

ANZSOG Australian New Zealand School of GovernmentCMCD Chief Minister and Cabinet DirectorateCSD Community Services DirectorateEIS Environmental Impact StatementESDD Environment and Sustainable Development DirectorateNPM New Public ManagementRIS Regulatory Impact StatementsTBL Triple Bottom Line framework

iii

1. Background/purpose of the evaluation

The TBL Assessment Framework was released by the Chief Minister and Cabinet Directorate (CMCD) as a Discussion Paper for public comment in June 2011 (CMCD 2011). The Discussion Paper sets out a proposed standard framework and approach to Triple Bottom Line (TBL) assessment. It is worth noting that the ACT Greens provided a submission in response to the Discussion Paper, and have maintained an interest in the progress of the issue.1

The ACT Government initiated a pilot study of the use of TBL in its Cabinet Submission processes in September 2011. A key output of the pilot is to evaluate the experience and findings of the pilot study in the context of the ACT Government’s Cabinet processes and identify potential improve -ments to the approach adopted.

In all, fourteen case study assessments of Cabinet Submissions were reviewed and discussions were held with authors and associates across eight directorates. The directorates themselves prepared the assessment for each submission and lodged these with CMCD.

CMCD commissioned the ANZSOG Institute for Governance at the University of Canberra (an affiliated member of the Australian New Zealand School of Government (ANZSOG)) to assist in the evaluation of the pilot study. The key personnel involved in the evaluation were Professor Mark Evans, Professorial Fellow Bill Burmester from ANZSOG and Mr Richard Bontjer from CMCD.

2. Observations

The evaluation identified a range of both positive and negative issues from the pilot study.

Positive findings

Appreciation of value

Those who had been involved in the pilot believed that there was great potential for a TBL approach of some kind to add value to public administration and decision-making in the ACT. There was consensus that a properly executed TBL system would enrich the quality of advice to government and lead to better decision-making. This observation was based on the argument that a full TBL assessment provides a more rigorous evidence base to inform Cabinet decision-making than is currently the case.

It was noted that the greatest value from TBL was likely to emerge from its use early in the policy development cycle, where negative impacts could be identified and solutions to mitigate those impacts could be incorporated into the policy proposal. However, clarification as to the purpose of TBL was seen as necessary so that the approach could be designed to best achieve that outcome and overcome the limitations evident from the pilot examples.

1 A submission was also received from the ACT Council of Social Service Inc. (ACTCOSS).

1

It was also noted, both by our interviewees and in our investigation of national and international practice, that TBL could be utilised at other decision points in the policy cycle (for example, monitoring and evaluation, horizon scanning, etc.) or as a key policy instrument in the application of a public value management approach (Moore 1995, Stoker 2006, Evans 2012). Background and discussion of national and international perspectives on enhancing strategic policy capability is provided in a compendium background document “National and international perspectives on enhancing strategic policy capability”.

Identifying ways to ensure a more consistent approach and deliver better quality assessments were also seen as important, although views on how this could be achieved varied. One approach was to centralise assessments into a single whole-of-government area expert in TBL, with an alternative being for policy developers to work collaboratively across all directorates to assemble comprehensive assessments across the full range of impacts. The latter is consistent with the suggested approach presented in the Hawke review (Hawke 2011).

The objectives of the TBL framework are consistent with and complement the government’s existing Performance and Accountability Framework (CMD 2011a). TBL furthers the objectives, including guiding continuous improvement and working towards enhanced government performance and accountability. As an ex ante evaluative framework, it informs policy and decision-making as well as strengthening and embedding a culture of evaluation in ACT Government operations, as outlined in the ACT Government Evaluation Policy and Guidelines (CMD 2011b).

Rigorous policy analysis

Officers appreciated the range of impact dimensions in the draft framework. The 31 cells were seen as providing a very useful and expansive ’set of lenses’ through which a policy proposal could be better illuminated for Ministers. It provided officers with a menu or checklist to work through and determine impacts. It was seen as providing greater specificity to the assessment and a useful guide to newcomers to the TBL process.

In a number of cases, it was felt that it was not clear where a particular anticipated impact should be recorded. For example, human rights impacts were seen as having their own status and including them under ’justice and rights’ was not seen as appropriate. This could be addressed through improved guidance and additional impact criteria.

There were some initial concerns as to the value of, and how officers were expected to provide, a three-way assessment of the particular policy proposal where there was one particularly dominant and obvious impact (economic, environmental or social) and little apparent impact in the other two. For example, one assessment on land acquisition was viewed as essentially having only social impacts and was treated as such in the assessment provided (three positive impacts were noted in the social cluster and only one in the economic group (‘greater private investment’), but apparently no impact on the ACT Budget). In discussion about this assessment, however, it was recognised that wider impacts were likely and that, had the assessment been undertaken more rigorously, those other aspects would have been identified and included in the assessment. It was also agreed that, had the assessment been undertaken as an integral part of the policy formulation stage of the process (rather than after the event as it had been for the pilot), those other aspects could have

2

been drawn upon to provide better policy advice to government. This was by no means an isolated example and it was considered that most assessments were, to some degree, narrower than they perhaps could, or even should, have been.

Integration of poverty and gender analysis

It was generally agreed that it would be beneficial if the final approach to TBL integrated all impact assessments (inclusive of gender and poverty impacts) into the primary assessment, as distinct from the stand-alone assessments. This would provide a strong evidence base worthy of mandatory inclusion in the cabinet process. This approach is supportive of a broader and more dynamic approach to assessment of impacts (see Accenture 2010, Friedman 2005, Hills and Sullivan 2006, Nicholls et al. 2012) that could incorporate a wider range of issues such as child impacts, indigenous, aged persons, etc. In cases where the purpose of the policy is to address poverty or gender issues, this could be subject to a more detailed gender or poverty assessment.

Negative findings

Expertise and quality of assessment

It was recognised by most officers that their own expertise was limited to a particular area of policy and did not extend to the full range of perspectives invoked by the pilot TBL framework.

Officers from the Environment and Sustainable Development Directorate (ESDD) doubted whether others knew enough about climate change to make an assessment and the human rights experts felt few could comment adequately on the impact on human rights.

Most authors acknowledged that they only drew on their own limited experience and field of expertise to complete assessments and did not draw on evidence and expertise from across government.

In some cases, the quality of those assessments could be considered perfunctory. Some were limited to very short sentences, or dismissed as ‘not applicable’ despite apparent impacts (for example, introducing greater regulation of boarding houses to reduce occupancy numbers was not assessed as having a cost of living impact on those residents having to find other, less crowded accommodation).

Mobilisation of bias

Many assessments reported only positive impacts, but did not include possible negative impacts, leading to the mobilisation of bias in assessments. In some instances, officers saw TBL as a way of adding to the arguments in support of the proposition and indicated that they would not include negative impacts that ran counter to the preferred policy position. In part, this view emerged from those undertaking retrospective assessments for a completed Cabinet Submission, where a negative impact could be seen as undermining the submission or requiring additional work and amendment to ameliorate the negative impact. In some cases, however, there was a perception that TBL was

3

intended to support proposals and not include negatives — ‘policy based policy-making rather than evidence based policy-making’ (Evans and Edwards 2011).

Such a mindset would need to be overcome in the full application of TBL. While this attitude may be somewhat reduced if TBL was instigated early in the policy framework, and therefore influenced the design of the policy, it is by no means certain that a positive bias would be overcome entirely. One only has to look at the role any Treasury or Finance Department takes in settling true and accurate Budget costing figures (where Treasury has the final word) to see that it is in a line agency’s interest to overstate the positives and understate the negatives. This applies to all impacts, not just the financial ones.

The pilot study approach did not provide for any external or independent vetting of claimed impacts and this is clearly an area of risk to sound advice being presented to Ministers.

Limitations of the pilot process

A number of issues emerged from the fact that the pilot was overwhelmingly viewed and treated as a test of concept, not the initial introductory stage of the full implementation of TBL. For example, TBL assessments in most cases were completed post hoc, undertaken late in the submission process or even after the Cabinet Submissions had been completed. This meant assessment evidence was drawn largely from the material already prepared for the submission (in one case the assessment was compiled entirely from a completed submission’s material by a central parliamentary coordin-ation section, not the original policy development team). Most interviewees indicated that the process was treated as an additional task against a range of competing priorities and not as being integral to the development of the policy.

Little external input was sought in preparing assessments as they were seen as dependent on the original submission, not as a whole-of-government exercise or requiring expert advice. This dependence on existing material and views expressed in the submission limited the depth and credibility of the pilot study assessments and some officers conceded that, given the circum-stances of their creation, the assessments were not of a high quality.

Most assessors indicated that they sought or accessed limited support in completing the process, often on the basis that they were unaware of what was available. This led to mixed interpretations of the objectives, requirements and depth of analysis sought in the pilot.

Uncertainties in current TBL design

A number of issues were identified by those involved in the pilot that would need to be resolved should a full TBL process be adopted.

Scope of coverage was identified as an issue as it was not considered essential for every type of submission to be subject to a TBL assessment, particularly where it was considered that little benefit would arise (for example, appointment decisions and some technical legislative submissions were seen as potential candidates for exclusion). A range of views were expressed with the majority favouring focusing on substantial policy change and excluding appointments, legislation and technical matters. However, all recognised that boundary issues would emerge

4

and some agreed that TBL assessment of simple policy change would not take that long to complete. In some instances, there would be little or no impact on many dimensions in any case, so it might be better to include more submissions in the scope of TBL than less.

Dealing with contradictory impacts where both positive and negative impacts in a particular area could be identified needed to be better handled and guidance needed to be given to those making the assessment.

Many impacts would be unquantifiable, even if it was possible to determine their direction (positive/negative). Similarly, there was no ready mechanism for changing value over time where, for example, up-front cost in the short term would be offset by the accrual of longer-term benefits or some sections of society benefitted at the expense of others.

Some proposals have impacts across a long lifespan, or only well into the future, which made it difficult to quantify them or assess their real impact. Some directorates simply ignored or excluded long term impacts while others appeared to apply an implicit/intuitive New Public Management (NPM) approach — discounting, but not excluding, long-term effects. Consistency of treatment is required.

Classification of impacts

Some potential impacts did not seem to have an obvious single ‘home’ in the template (for example, the cost of participation in community activities could fall either within ‘participation in community activities’ or ‘consumption’ or ‘cost of living’). Others areas of impact were seen to be missing (for example, ‘impact on crime’). Refinement and guidance of the framework would be needed to avoid this ambiguity. There was not ready identification of likely ‘public/interest group sensitivity’ (for example, in the assessment provided for the tree management submission) — although, pre-sumably, a well executed assessment would have identified this variable across a number of the 31 categories.

Guidance

Most saw the need for greater guidance and training if the benefits from TBL were to be realised and a consistent standard was to be achieved across government. The learning and development needs for TBL would need to be treated as a significant cultural shift from existing processes and be supported as such.

Detailed analysis on poverty and gender

In the pilot exercise, only one detailed analysis of gender was undertaken and no detailed analyses were completed on poverty impacts. This may have been due to lack of an understandable trigger or the view that detailed analysis was not warranted.

Climate change impacts

One submission in the pilot applied the climate change impact assessment framework to assess the mitigation and adaptation implications of increasing the extent of nature reserve. While this was

5

considered a useful exercise, it did require technical skills that would not be available to policy officers across the ACT.

3. Analysis

The utility and success of adopting TBL needs to be judged against whether the process leads to better decision-making as a result of decision makers having access to better, more comprehensive and more balanced advice. TBL can be viewed as fulfilling many purposes and can be assessed against both objectives and risks along with arising implementation dilemmas.

Objectives

Various purposes of TBL can be identified by the objectives that are intended to be served by the TBL process. Figure 1 provides examples of decision points in strategic direction, policy formulation, monitoring and evaluation, and horizon scanning where TBL could be deployed.

Figure 1: Decision points in an iterative policy cycle where TBL could be deployed

Policy design

Quality assurance

TBL can perform as a ‘check of last resort’ with which to test a completed submission and so expose issues and any unintended consequences that could be considered in Cabinet discussion. This would be completed after the submission is almost completed and, therefore, does not readily afford the opportunity to identify alternative solutions or approaches that could mitigate or overcome identified negative impacts. This approach could also be applied to the approval of draft legislation

6

following an earlier Cabinet decision as a way to check that the Bill did not introduce any unintended or unforeseen consequences in implementing the Cabinet decision.

Clarifier of issues

If applied as a scanning or scoping process early in preparation of a proposal, the TBL Assessment Framework could highlight areas that needed to be considered in developing the proposal. This would see the framework as a checklist aimed at identifying impacts, but not extending to iterative policy development aimed at overcoming negative consequences. One approach would be for each directorate to review proposals and identify impacts in their area of responsibility that should be subjected to an assessment by the authoring agency.

Rational decision-making

Some proponents see the TBL Assessment Framework as a rigorous decision-making tool that would provide a rational basis to determine the merits of a proposal. This is an unrealistic expectation as, in most cases, both positive and negative impacts will be identified and it requires a political process to weigh the costs and benefits of competing impacts. In a similar way, doing a TBL across the whole of a Budget does not assist decision-making, it merely provides a summary of the competing positive and negative impacts. It would even be difficult to determine the overall or ‘net’ impact from opposite impacts as many will be unquantifiable.

Pro-active policy-making

If a TBL is done early enough in the policy development cycle, many issues and potential negative impacts can be identified and creative solutions posited to overcome them. This would improve the positive balance of the eventual proposal, which could include strategies to offset potential nega-tives and so lead to better propositions being submitted for consideration. Internal processes of reconciling differences could be facilitated by applying the framework and finding accommodation for opposing impacts. This is the recommended approach.

Strategic direction

Public accountability in decision-making

Public release of a TBL assessment of a policy decision could provide an enhanced degree of accountability for decisions taken by Cabinet. From the government’s point of view this could be seen as a ‘public defensibility’ test of the decision, wherein they would have a basis on which to explain their decision, including acknowledging any negative impacts and strategies or measures taken to mitigate them. This needs to be balanced against the fact that TBL assessments do not provide a definitive rational basis to determine the merits of a proposal, as discussed above.

Drive collaboration across government

The TBL Assessment Framework could be used to draw input into a proposal from across government. This would be best achieved if specialists in each area of the framework were drawn into the design of the policy early on in the process. Such an approach is in keeping with the post-Hawke drive to find a joined-up approach to problems requiring a joined-up solution. However, this adds time to the process.

7

Delivery and learning

Monitoring and evaluation of existing programs

Post hoc evaluations could benefit from using a standard, broad framework that evaluates the success or otherwise of an existing policy or program across the full range of potential impacts. Evaluation preparedness would suggest the need for designing monitoring systems that allowed for the ongoing collection of data for each identified impact of the TBL Assessment Framework.

Contribution to citizen-centric governance

As the ACT Government deepens its commitment to citizen-centric governance, it is evident that TBL could be used to inform deliberative events with citizens on service performance and options, priority setting in budgetary allocation, the development of ideas for reform or horizon scanning on ACT futures.

Risks

From the findings of the pilot evaluation, there is evidence of a number of systemic risks to the completion of high quality, robust and evidence-based TBL assessments. Those identified include the following.

Perfunctory completion

The lack of priority attributed to or understanding of the value of TBL could lead to poor assess-ments. In addition, while a proforma approach provides ease of completion and some consistency of approach, it can also lead to a tick and flick mentality by assessors. Time pressures in the final stages of a submission process would also be likely to feed a cursory use of the template. Some examples from the trial showed minimal assessment of issues and reliance on generalisations.

Lack of evidence/knowledge

The full TBL assessment requires evidence from across a wide range of policy domains. Individual policy officers, or in fact a whole directorate, may not have adequate evidence or knowledge to provide adequate assessments across that range. The ESDD officers doubted whether others knew enough about climate change to make an assessment; the human rights officers felt that few could comment adequately on the impact on rights.

Mobilisation of bias

When the policy area responsible for a submission was also responsible for the TBL Assessment, there was a clear incentive to look for and include positive impacts, while down-playing or ignoring any negative impacts that might lead to changes to or rejection of the policy proposal. In the financial area, this risk is overcome by all costings having to be signed off by Treasury.

8

Capability constraints

The ability of policy officers to identify, and then provide sound advice on, all areas of impact from a policy proposal was likely to be compromised by the time constraints usually applying to Cabinet Submissions and the lack of adequate knowledge by individual directorates across all dimensions of the TBL Assessment Framework. A range of solutions are needed to address this constraint — from improving the ability of policy officers themselves to fostering greater collaboration across government in preparing assessments, as well as the design of the framework and its application.

Critically, particular risks became more relevant depending on the approach taken in applying the TBL process, and when and by whom it was completed in the policy cycle. The matrix outlined in table 1 sets out the ‘risk hazards’ that arise with implementation of TBL.

Table 1: ‘Risk hazards’ and triple bottom line assessment

Purposes of TBL RISKS

Favourable bias

Perfunctory completion

Lack of evidence/ knowledge

Capability constraint

Quality assurance L L P P

Clarifier of issues L L P P

Deterministic decision-making L U P P

Public accountability in decision-making

L P P P

Drive collaboration across government U P U P

Evaluation of existing programs U/P U U P

Assist in developing sound policy U/P U U PKey: U=Unlikely, P=Possible, L=Likely.

Implementation dilemmas

It should be recognised that the purpose of a pilot project is to expose critical dilemmas in the implementation process and propose mitigating strategies. We have, therefore, been deliberately provocative in our criticisms to ensure the design of the best possible TBL process, which can deliver genuine public value for the ACT citizenry.

The key delivery dilemmas for the TBL pilot project can be organised around conceptual (how the process is understood by public servants and elected members), institutional (project design and resources) and environmental constraints and triggers, many of which are beyond the control of public servants. It is observed that these sets of factors can constrain or facilitate progressive outcomes, sometimes interact in unintended ways and need to be carefully managed (figure 2). The key issues requiring attention are highlighted in red (‘urgent attention’) and amber (‘some attention’). By ‘some attention’ we refer to the need to establish consistent quality standards across the ACT Public Service. We did find instances of high quality practice, which should be shared across government, but in the main the TBL tool was not utilised effectively. Why was this the case and what can be done to improve outcomes?

9

Figure 2: Critical implementation dilemmas in the TBL process and their interactionInstitutional constraints

and triggersConceptual constraints

and triggers1. Common understanding of the

purpose of TBL amongst elected members

2. Common understanding of the purpose of TBL amongst implementing agencies

Environmental constraints and triggers affecting implementation

Project design

1. Clear purpose and consistent objectives

Project resources and instruments

2. Effective coordination including collaboration

3. Clear roles and rules for implementing agencies

4. Effective project instruments (project support documents, training, etc.)

5. Adequate allocation of resources6. Effective baseline data and access

to knowledge7. Recruitment of project officers

with adequate skills/training8. Commitment and leadership skills

of implementing officials

1. Socioeconomic conditions 2. Public support3. Attitudes and resources of

constituency groups4. Support from politicians5. Alignment with ACT Government

agendas 6. Media perceptions2

The implementation processProject outputs of implementing agencies

Compliance with project outputs by target groups

Actual impacts of project outputs

Perceived impacts of project outputs

Revisions in project

Note: Green = trigger. Amber = some attention required. Red = significant attention required. Black = not a salient variable at present.

Other issues

Scope of coverage of the climate change impact assessment

The Parliamentary Agreement with the ACT Greens provides that a Climate Change Impact Analysis is required for all Government Bills and major policy proposals. Based on feedback received in the pilot, completing the template required significant technical capacity and had time implications. The climate change impact analysis required a degree of technical understanding of climate change that is currently beyond the skills and knowledge of an average policy officer. Further, completing the analysis was time consuming, despite efforts to simplify the templates to the core information requirements. 2 A key component of any risk strategy for TBL would involve an action plan for dealing with media leaks of potentially inflammatory material into the public domain.

10

Assessment of climate change impacts will need to be conducted for Government Bills or for major policy proposals. Like the TBL Assessment Framework, the Climate Change Impact Assessment Framework is intended to be scalable to cater for Bills or major proposals from simple through to ‘with significant’ climate change impacts. The following approaches should be adopted to cater for differences across Government Bills and major policy proposals.

Where there is no anticipated climate change impacts of a Government Bill or major policy proposal (based on the TBL assessment), the TBL assessment should reflect, and,a statement should be included in the explanatory statement of Government Bills that climate change impacts have been considered and no impacts have been identified.

Where only minor climate change impacts are identified in the TBL assessment, part A of the Climate Change Impact Assessment Framework should be completed and included as part of the TBL assessment and explanatory statement of the Government Bill.

Where major climate change impacts are identified in the TBL assessment, parts A, B and C of the Climate Change Impact Assessment Framework should be completed and included as part of the TBL assessment and explanatory statement of the Government Bill.The precise thresholds for minor and major climate change impacts will need to be assessed on a case-by-case basis, recognising that the impacts will vary significantly depending on the nature of the major policy proposal or Government Bill.

The broad requirement to conduct a climate change impact assessment for all Government Bills will add an administrative cost. It will also introduce capability and capacity risks for directorates to complete the assessments adequately. It is, therefore, proposed that the application and approach to assessment of climate change impacts be piloted for 12 months and then be reviewed to consider the costs and benefits of the framework.

The broader TBL Assessment Framework would continue to apply to all proposals to describe potential aggregate impacts. Additional guidance will be provided within the ‘greenhouse gas emissions’ impact assessment to assist in gauging the broad mitigation and adaptation impacts. Where significant climate change impacts are identified (mitigation and/or adaptation) that would not otherwise be picked up by the above criteria, an optional assessment could be conducted in support of the proposal.

Relationship between TBL and Regulatory Impact Statements (RIS)

A RIS is part of a process for analysing the options available, including the possibility of regulation, to determine the greatest net benefit to society and inform government decision-making. Publication of a RIS can also promote transparency and accountability. A limited number are currently produced each year. A RIS may be required to be prepared for subordinate legislation under the Legislation Act 2001 and policy and legislative proposals presented to Cabinet as stated in the Legislation Handbook.

The TBL Assessment Framework can perform a useful complementary role to the RIS requirements by providing a template for a broad initial scan of impacts. It can also satisfy one of the RIS’s objectives (improve policy development), by providing an assessment framework that can be used more generally. The TBL Assessment Framework does not replace the requirement for a RIS.

11

How should the TBL be included in the Budget Process?

Limited TBL information is presently captured in the Budget process, with some information requested as part of the business case for capital projects, but little information is included in relation to operational proposals.3

Doing a TBL assessment across the whole of a Budget does not assist decision-making, it merely provides a summary of the competing positive and negative impacts. It would even be difficult to determine the overall or ‘net’ impact from opposite impacts as many will be unquantifiable. Notwithstanding, the requirements for operational and capital projects could be enhanced so that the preliminary assessment matrix is applied to all proposals to identify key impacts. Where impacts are identified as being significant, explanation of the impacts should be provided including how they relate to the purpose of the policy.

Should TBL assessments be made public?

The Cabinet outcomes summary is a key mechanism to improve transparency and public account-ability for the decisions of Cabinet. The summary provides a list of decisions made by Cabinet. However, is does not provide an explanation for those decisions, nor information provided to assist in the decision-making process.

Public release of a TBL assessment of a policy decision could provide an enhanced degree of accountability for decisions taken by Cabinet. From the government’s point of view this could be seen as a ‘public defensibility’ test of the decision, wherein they would have a basis on which to explain their decision, including acknowledging any negative impacts and strategies or measures taken to mitigate them.

This needs to be balanced against the fact that TBL assessments do not provide a definitive rational basis to determine the merits of a proposal. In most cases, both positive and negative impacts will be identified and it requires a political process to weigh the costs and benefits of competing impacts.

It is proposed that there should be no requirement for the full TBL assessment to be included in the Cabinet outcomes summary. However, the summary could include identification of key positive and negative impacts and that (optionally) Cabinet may decide to release the full TBL assessment in special cases (similar to approach currently adopted for a RIS).

This approach could be reviewed in the future as the TBL Assessment Framework is embedded into Cabinet decision-making processes.

4. Mitigating implementation dilemmas

Two constituent elements of an effective approach to TBL emerge from the above analysis — the need for common purpose and the design of effective TBL resources and instruments.

3 All Cabinet Submissions currently require a high level summary of social, environmental, economic, intergovernmental and regulatory impacts.

12

Common purpose

There is a need for a decision to be made on the purpose of TBL, including where and when it would be best deployed. These decisions will need to be made when considered by government.

Any decision would need to be implemented through building a consensus across ACT Government senior management on the purpose of TBL, recognising it as a priority and agreeing how best it should be deployed. This could be achieved through an independently facilitated workshop with key public servants to establish common ground. It would also involve joining up existing reporting processes to ensure a sustainable approach that is not a drain on resources and does not distract attention from the achievement of government priorities.

The overwhelming view of participants is that TBL could become a critical tool for enhancing strategic policy capability, but only if it is used appropriately, works with the grain of the organis -ational work plan and helps to distil a whole-of-government approach to solving critical governance problems.

The first issue to consider is the purpose of the tool. If the purpose of the tool is to support the development of good policy and better policy advice, we need to include some questions at the outset that allow us to assess the rigour of the policy process. These could include the following.

What is the problem to be addressed?

Does the policy have a clearly stated objective?

Is government intervention necessary and can we be sure it will not make things worse in the long term?

What options have been considered?

Does the policy align with the ACT Government’s strategic priorities?

Does the domestic and international evidence support intervention?

There is also the potential here to include an assessment of the quality of the policy intervention in contrast with other interventions, as illustrated in table 2.

We would also require a small number of indicators to allow us to measure each policy value.

The second issue to consider is: what are the desired outcomes of the policy? While clearly the framework should not be judgemental, completing the self-assessment requires some notion of ‘desirable’ impact as opposed to negative impact. For example, at present it just says under the first item:

13

Community and individual health

Will the proposal lead to a change in the community’s or individual health?

Discuss predicted impacts on human health and ACT community health.

Table 2: Assessing policy values

Policy value Current policy Option 1 Option 2EquityEffectivenessEfficiencyFlexibilityImplementableLegitimacySustainability – fiscalSustainability – environmentalSustainability – socialOverall rating

What is missing in completing a response to these questions is the identification of the overarching intended policy goal. In some cases this might be considered obvious, like improving health, so comments could say it will improve health by X or decrease it by X. But it is not stated and some other areas are not so obvious. For example, under income level and distribution, what is a positive outcome? Do both aspects have to increase? Or is increasing the income of the wealthiest a desirable outcome even if the distribution is less equal? In some areas a clue is actually given. Access to housing, for example, includes the words ‘affordable and adequate’ so there is some notion of desired outcomes. But this is not consistent in every cell. A desirable outcome needs to be articulated in every cell — expressed, for example, by: ‘Will this proposal increase total income in the community? Who will the increase in income benefit most? What will income distribution look like across the community?’

The third issue to be considered is whether the policy proposal is technically feasible. What risks to successful implementation have been identified and how they will be mitigated? And, what inform-ation needs to be gathered to measure success or failure? The existing framework does not assess the likelihood that the expected outcome will be realised. At present, there is no way of knowing whether the expected outcome is almost certain, or just a possible and improbable spinoff. We have often seen Commonwealth Cabinet Submissions in the past where the positives were clearly overstated and the negatives underplayed. Indeed, this is where the Department of Finance and Deregulation was useful in providing balance to the assessment. So we would suggest a simple mechanism of ranking (and grouping) impacts as ‘inevitable’, ‘likely’, ‘possible’ — or some similar scale (for example, a traffic light or red flag system).

14

The key questions to address here could include the following.

Does the proposal have a clear intervention logic and does it identify weak links in the chain of causation?

Have implementation risks or barriers been clearly identified?

Is there a clear strategy for mitigating such risks?

What are the consequences of action?

Is the cost of the mitigation strategy proportionate?

Has an appropriate monitoring system been identified for evaluating change?

Have robust key performance indicators been identified for evaluating change?

Have the costs for performance measurement been properly identified?

TBL resources and instruments

The lack of common understanding among directorates on the purpose of TBL is partly a function of unclear and inconsistent project objectives, partly a function of misinterpretation of project docu-mentation, partly a reflection of limited capacity and competing work priorities. There is an identi-fied need to develop a Learning and Development Framework to underpin and enhance the TBL process. This would involve:

developing clearer TBL guidance notes, including identification of the policy analysis com-petencies necessary to complete the task;

ensuring that the ACT’s Learning and Development Framework includes competency training in this area;

providing examples of better practice in various forms (for example, cases, ‘Talking Heads’, role play, etc.);

designing whole-of-government experiential workshops to train staff, and disseminate and share better practice; and

encouraging and facilitating the development of communities of practice to provide a more rigorous knowledge base to inform decision-making.

This strategy would require enhancing the coordination and oversight capability of CMCD. Indeed, several of our interviewees in participating agencies demanded it.

‘If you leave this to us it won’t get done. This is one area where we require strategic coordination.’

‘It struck me when I was filling in the TBL assessment that I needed some advice on the (x) dimensions. Luckily I had a friend who could give me some tips but if I didn't I would have struggled. We still tend to operate in our own silos. We need to be able to meet and discuss and share the good things that we do. On our own we don't have all the answers but if we help each other we can get more right than wrong.’

‘There is an ACT service delivery strategy but each area does it in its own way. There is often little connection between the two. We need a place that encourages and supports new thinking. Business as usual is no longer a way forward.’

15

‘Do we really know what the new ideas that work are? Do we have a place that we can go to that gives us a sense of what is working nationally and internationally?’

‘We are doing so many things on the fly. We have to face up to the fact that if we don't have time; we need to get the support elsewhere.’

5. Towards an inclusive, integrated and effective TBL assessment — conclusions

The evaluation confirmed the potential for a well designed TBL process to add greatly to the quality of Cabinet policy deliberations by bringing a far broader range of perspectives and richer evidence to the attention of Ministers. It is recommended that a TBL process become a standing requirement of preparing substantial policy submissions for Cabinet consideration. However, the pilot study identified a range of matters that needed to be addressed for such potential to be realised. What would the ingredients of a more effective approach to TBL look like?

Importantly, the draft TBL Assessment Framework that was applied by directorates in the pilot utilised 31 dimensions (grouped into the three TBL clusters of social, economic and environmental). These 31 dimensions provided far deeper analysis than the simple three-way impact clustering currently employed in submissions could ever hope to achieve and were seen by the officers who undertook assessments as a powerful tool for assembling the information to improve Cabinet decision-making. Nevertheless, there were a number of specific ‘areas of impact’ that were underplayed in the draft framework and clarification or refinement is recommended in these cases.

It is also considered important that, in refining the approach for future use, the objective in adopting a TBL approach must be clearly understood and specified. The evaluation identified mixed views of the TBL’s intended purpose and, consequently, a number of potential approaches that would best meet that purpose. One theme to emerge from the feedback from those undertaking assessment was the lack of clarity and purpose underpinning the TBL assessment.

The pilot also identified a capability constraint if TBL assessments became a mandatory requirement for all Cabinet business. First, the ability of policy officers to identify, and then provide sound advice on, all areas of impact from a policy proposal was likely to be compromised both by time constraints applying to Cabinet Submissions and the lack of adequate knowledge by individual directorates across all dimensions of the TBL Assessment Framework. A range of solutions are needed to address this constraint — from improving the policy analysis capabilities of policy officers themselves to fostering greater collaboration across government in preparing assessments — as well as the design of the framework and its application. It is, therefore, recommended that a Learning and Development Framework is designed for these purposes.

Key recommendations

1. The objective of adopting the proposed approach is to assist in developing sound policy through iterative policy development aimed at mitigating negative consequences of action and fostering collaboration across government by drawing on expert input into assessments from across all Directorates.

16

– Use of the framework — from the initiation of a policy proposal through to the Cabinet Submission and subsequent monitoring, evaluation and strategic learning — is essential in this approach.

– Mitigation strategies to cope with unintended consequences of action should also be incorporated into submissions where potentially negative impacts are identified.

2. Keep the scope of coverage of the TBL Assessment Framework broad.

– TBL assessments should be conducted for all policy proposals or changes to be considered by Cabinet.

– A broader approach for inclusion will reduce ambiguity regarding coverage of the framework.

– While minor policy proposals are unlikely to gain significantly from the assessment, they will not take long to complete (given the scalability of the framework) and, if there is any doubt over the significance of the policy, preference should be for an assessment to be completed.

– Adopt a scalable approach to the TBL Assessment Framework to ensure that it is equally applicable to both relatively simple proposals as well as complex policy proposals with significant impacts and complex interactions.

– Exemption could be given to technical submissions and specified in the Cabinet Handbook — such as appointment decisions and some technical legislative instruments where a TBL assessment will not add value.

3. The framework should be modified to:

– incorporate elements to support good policy development — such as ‘what is the problem to be addressed?’ and ‘what are the desired outcomes of the policy?’;

– include a focus on the technical feasibility of the implementation of the proposal; and

– incorporate additional elements to the 31 impact dimensions — such as human rights and guidance to incorporate gender, poverty, aged persons, children, climate change and Indigenous issues.

4. Integrate gender and poverty impact assessments into the primary assessment, without detailed stand-alone assessments being required.

5. Assessment of climate change impacts for Government Bills or major policy proposals should apply the Climate Change Impact Assessment Framework. The framework should be scalable to cater for Bills or major proposals from simple through to ‘with significant’ climate change impacts.

– The precise thresholds for minor and major climate change impacts will need to be assessed on a case-by-case basis, recognising that the impacts will vary significantly depending on the nature of the major policy proposal or Government Bill.

6. Establish formal centres of expertise (communities of practice) across the ACT Government in key areas including gender, poverty, aged persons, children, climate change and Indigenous issues.

17

– The centres/contacts would provide assistance in aspects of policy development across the ACT Government and provide a mechanism for more effective collaboration.

– The number and specialisation of centres should be flexible and reactive to the key issues being confronted by the ACT Government.

7. Allow CMCD to oversee the application of the framework.

– This could include providing training and support to directorates, managing collaboration across directorates when preparing assessments and establishing a small expert cell to review the adequacy of the assessments.

– CMCD already oversee the process for presenting submissions to Cabinet. This would extend to review the TBL assessments conducted in support of submissions.

8. The full TBL assessment should not be included in the Cabinet outcomes summary.

– The Cabinet outcomes summary could identify the key positive and negative impacts and include a statement that Cabinet may decide to release the full TBL assessment in special cases (similar to the approach currently adopted for a RIS).

9. While the TBL Assessment Framework should not apply across the whole of a Budget, the TBL requirements for operational and capital projects should be enhanced.

10. Develop and roll out a change management strategy to introduce this major reform across government and communicate how Cabinet will work.

18

References

Accenture 2010, The Accenture Public Sector Value Model. http://www.issa.int/pdf/cracow04/2Annex-Younger.pdf.

Alford J and O’Flynn J 2008, Public value: a stocktake of a concept, paper presented at the Twelfth Annual Conference of the International Research Society of Public Management, Faculty of Business, Queensland University of Technology, Brisbane, 26–28 March 2008.

Coats D and Passmore E 2008, Public Value: The Next Steps in Public Service Reform, The Work Foundation, London, http://www.theworkfoundation.com/assets/docs/publications/201_PV_public_service_reform_final.pdf.

Coyle D and Woolard C 2009, Public Value in Practice: Restoring the Ethos of Public Service, BBC Trust, http://www.bbc.co.uk/bbctrust/assets/files/pdf/regulatory_framework/pvt/public_value_practice.pdf.

CMCD (Chief Minister & Cabinet Directorate) 2011, ‘Triple bottom line assessment for the ACT government: discussion paper’, 2008 Parliamentary Agreement for the 7th Legislative Assembly for the ACT.

CMD (Chief Minister’s Department) 2011a, Strengthening Performance and Accountability: A Framework for the ACT Government, ACT Government, Canberra.

—— 2011b, ACT Government: Evaluation Policy and Guidelines, ACT Government, Canberra.

Evans M 2012, Innovation in Australian Local Government, Australian Centre of Excellence for Local Government and ANZSOG Institute for Governance, Canberra.

—— and Edwards M 2011, Getting Evidence into Policy-Making, ANZSOG Institute for Governance, Canberra.

Friedman M 2005, Trying Hard Is Not Good Enough: How to Produce Measurable Improvements for Customers and Communities, FPSI Publishing, www.resultsleadership.org.

Gains F and Stoker G 2008, How Politics Works: Understanding the New Realities of the Political Process in Public Management, The Work Foundation. http://www.theworkfoundation.com/assets/docs/how_politics_works.pdf.

Hawke, A 2011, Governing the City State: One ACT Government – One ACT Public Service, ACTPS Final Review, ACT Government, Canberra.

Hills D and Sullivan F 2006, Measuring Public Value 2: Practical Approaches, The Work Foundation, London, http://www.theworkfoundation.com/assets/docs/publications/171_measuring_practical3.pdf.

Horner L and Hazel L 2005, Adding Public Value, The Work Foundation, London, http://www.workfoundation.com/assets/docs/publications/86_public_value.pdf.

Horner L, Lekhi R and Blaug R 2006, Deliberative Democracy and the Role of Public Managers, Final

19

Report of the Work Foundation’s Public Value Consortium, The Work Foundation, London, http://www.theworkfoundation.co.uk/assets/docs/publications/107_Deliberative%20democracy%20and%20the%20role%20of%20public%20managers.pdf.

International Association for Public Participation Australasia 2004, IAP2 Public Participation Spectrum, http://www.iap2.org.au/sitebuilder/resources/knowledge/asset/files/36/iap2spectrum.pdf.

Kelly G, Mulgan G and Muers S 2002, Creating Public Value: An Analytical Framework for Public Services Reform, Cabinet Office, London, http://www.cabinetoffice.gov.uk/media/cabinetoffice/strategy/assets/public_value2.pdf.

Moore M 1995, Creating Public Value: Strategic Management in Government, Harvard University Press, Cambridge, MA.

—— 2003, The Public Value Scorecard: A Rejoinder and an Alternative to ‘Strategic Performance Measurement and Management in Non-Profit Organizations’ by Robert Kaplan, The Hauser Center for Nonprofit Organizations Working Paper No. 18, Harvard University, Boston, http://papers.ssrn.com/sol3/papers.cfm?abstract_id=402880.

—— and Khagram S 2004, On Creating Public Value: What Business Might Learn from Government about Strategic Management, Corporate Social Responsibility Initiative Working Paper No. 3., John F Kennedy School of Government, Harvard University, Boston, http://www.hks.harvard.edu/m-rcbg/CSRI/publications/workingpaper_3_moore_khagram.pdf.

Nicholls J, Lawlor E, Neitzert E and Goodspeed T 2012, A Guide to Social Return on Investment, The SROI Network, http://www.thesroinetwork.org/publications/doc_details/241-a-guide-to-social-return-on-investment-2012.

Stoker G 2006, ‘Public value management: a new narrative for networked governance?’, American Review of Public Administration, vol. 36, pp. 42–4.

Talbot C 2008, Measuring Public Value: A Competing Values Approach, paper prepared for The Work Foundation, Herbert Simon Institute, Manchester Business School, http://www.theworkfoundation.com/Assets/Docs/measuring_PV_final2.pdf.

20