Evaluating Indigenous programs: a toolkit for change Sara ... › 2017 › 06 › evaluati… ·...

Sara Hudson

Evaluating Indigenous programs: a toolkit for change

Research Report | June 2017

National Library of Australia Cataloguing-in-Publication Data:

Creator: Hudson, Sara, 1974- author.

Title: Evaluating Indigenous programs: a toolkit for change / Sara

Hudson.

ISBN: 9781922184870 (paperback)

Series: CIS research report ; RR28.

Subjects: Aboriginal Australians--Services for--Evaluation.

Community development--Australia--Evaluation.

Aboriginal Australians--Government policy.

Research Report 28

Evaluating Indigenous programs:

a toolkit for change

Sara Hudson

with contributions from Carlos Andres Monteverde Salvatierra

and Eva Christensen

Related CIS publications Research Report

RR18 Sara Hudson, Mapping the Indigenous Program and Funding Maze (2016).

Policy Monograph

PM105 Sara Hudson, Closing the Accountability Gap: the first step towards better Indigenous health (2009)

Contents

Executive Summary ...............................................................................................1

Introduction ..........................................................................................................3

The case for reform ...............................................................................................4

Why evaluate? ......................................................................................................7

Analysis of program evaluations ..............................................................................9

Analysing the evaluations: A hierarchy of evidence...................................................10

Productivity Commission’s criteria for evidence of ‘what works’ .......................... 12

Our criteria for evaluating the ‘evaluations’ ......................................................12

Lessons to be learnt .....................................................................................14

Examples of successful practices ....................................................................14

Discussion and conclusion .....................................................................................20

Recommendations ...............................................................................................21

Appendix A: .......................................................................................................22

Evaluation of CBA Programs ..........................................................................22

Evaluation of SROI Programs .........................................................................24

Summary of Evaluations/Case-studies/Audits ..................................................25

Appendix B: Evaluation Toolkit ..............................................................................47

Appendix C: List of Tobacco cessation programs ......................................................50

Endnotes ............................................................................................................52

ACKNOWLEDGEMENTS This research report has been assisted by comments and suggestions from two anonymous external

reviewers; my colleagues Simon Cowan, Michael Potter, Charles Jacobs, Heidi Kiekebosch-Fitt and Gary Banks; and participants who took part in a CIS roundtable on growing the evidence base for effective social programs. I am also grateful to Karla Pincott, who edited the report and Ryan Acosta who designed and laid

out the report.

All remaining errors are my own.

Evaluating Indigenous programs: a toolkit for change | 1

Previous CIS research indicated lack of evaluation of Indigenous programs is a significant problem. Of the 1082 Indigenous programs identified, only 88 (8%) had been evaluated.1

Following the release of that research and a Productivity Commission report that also called for more rigorous evaluation of Indigenous programs, the federal government announced it would allocate $40 million over four years to strengthen the evaluation of Indigenous programs and provide $50 million for research into Indigenous policy and its implementation.

However, given the average cost of an evaluation is $382,000, the extra $10 million a year for Indigenous program evaluations will not go far. To make the most of this additional funding, the government must change the way it evaluates and monitors programs.

Although formal evaluations for large government programs are important, evaluation need not involve contractors. Government must adopt a learning and developmental approach that embeds evaluation into a program’s design as part of a continuous quality improvement process.

It is not enough just to evaluate. Government must use the findings from evaluations to improve service

Executive Summarydelivery. Unfortunately, many government agencies ignore evaluations when making funding decisions or implementing new programs. A recent audit of the NSW Evaluation strategy found the NSW Treasury and NSW Department of Premier and Cabinet were not using evaluation outcomes to inform and improve practices.

Analysis of 49 Indigenous program evaluation reports found only three used rigorous methodology, and none used what is considered the ‘gold standard’ of evidence: Randomised Control Trials (RCTs). Overall, the evaluations were characterised by a lack of data and the absence of a control group, as well as an over-reliance on anecdotal evidence.

Particular features of robust evaluations include:

• A mixed method design, which involves triangulation of qualitative and quantitative data and some economic components of the program such as the cost effectiveness/or meta-analysis

• Local input into design and implementation of the program to ensure program objectives match community needs

• Clear and measurable objectives

• Pre and post program data to measure impact

2 | Evaluating Indigenous programs: a toolkit for change

Adopting a co-accountability approach to evaluation will ensure that both the government agency funding the program, and the program provider delivering the program, are held accountable for results. An overarching evaluation framework could assist with the different levels of outcomes expected over the life of the program and the various indicators needed at each level to measure whether the program is meeting its objectives. Feedback loops and a process to escalate any concerns will help to ensure government and program providers monitor one another and program learnings are shared.

Suggestions for policy makers and program funders include:

• Embedding evaluation into program design and practice — evaluation should not be viewed as an ‘add on’ but should be built into a program’s design and presented as part of a continuous quality

improvement process with funding for self-evaluation provided to organisations.

• Developing an evidence base through an accountability framework with regular feedback loops via an online data management system — to ensure data being collected is used to inform practice and improve program outcomes and there is a process for escalating concerns.

Suggestions for program providers include:

• Embedding evaluation into program practice — evaluation should not be viewed as a negative process, but as an opportunity to learn.

• Developing an evidence base through the regular collection of data via an online data management system to not only provide a stronger evidence base for recurrent funding, but also to improve service delivery and ensure client satisfaction with the program.


The first CIS report in this series ‘Mapping the Indigenous Program and Funding Maze,’ provided quantitative evidence of the lack of evaluation of Indigenous programs. Of the 1082 Indigenous programs identified in our research, only 88 (8%) had been evaluated.2 This finding was corroborated by the Productivity Commission’s 2016 Overcoming Indigenous Disadvantage Report, which found only 24 Indigenous programs had been rigorously evaluated and that there was a “pressing need for more and better evaluation of Indigenous policies and programs nationally if we are to see improvements in outcomes for Aboriginal and Torres Strait Islander Australians.”3

Following the release of these reports, the federal government announced it would be allocating $4.5 million in the next financial year to a number of key evaluations of Indigenous programs, including an evaluation of the Community Development Programme (CDP) and RCTs to assess the impact of the Prisoner Throughcare Programme in the Northern Territory and the School Enrolment and Attendance Measure Programme. In early 2017, the federal government announced it will allocate $10 million a year over four years to strengthen the evaluation of Indigenous programs. According to the government, a formal Evidence and Evaluation Framework will be developed to strengthen the reporting and monitoring of the program evaluations.

In his 2017 Closing the Gap speech, Prime Minister Turnbull reiterated the government’s emphasis on evaluation and announced the appointment of an Indigenous commissioner at the Productivity Commission and $50 million for research into Indigenous policy and its implementation.4 These announcements suggest the government is finally looking at doing something to address the serious shortfall in evidence. At the same time, the extra $10 million per year for Indigenous program evaluations will not go far. Analysis of the

Introduction

AusTender procurement contracts found the average cost of an evaluation is $382,000.5 At this price, the additional $10 million will be enough for only 26 more evaluations of Indigenous programs per year.

The Australian government has for some time been aware of the lack of evidence on the effectiveness of Indigenous programs. However, the challenge is transitioning from awareness to action that will address the knowledge gap. For years, government has claimed to be focused on delivering evidence-based policy, but if this is to become more than just empty rhetoric, government needs to urgently change the way programs and services are funded and delivered.

Although broad scale changes to the service system are probably needed, the focus of this report is how best to measure the effectiveness of current Indigenous programs and then how to use that evidence to improve program design and implementation. Once more evidence is collected, the government will have a much better understanding of what works and what changes are necessary to ensure programs meet the needs of Indigenous people and communities.

This report starts by outlining the case for reform and Indigenous people’s frustration at the Indigenous Advancement Strategy, which saw community organisations lose funding for programs they felt were working, while programs and services communities did not want or need were introduced. Next, the report examines why it is important to evaluate programs, and the concept of co-accountability. The findings of a literature review of 111 Indigenous program evaluations/audits/reviews is analysed, including what constitutes a rigorous evaluation and a possible hierarchy of evidence. Finally, recommendations for improvements to practice for both policy makers and program providers is provided.


There is general consensus that more evidence on the effectiveness of Indigenous programs is needed to improve Indigenous outcomes. However, while there is bipartisan support to conduct evidence-based policy, in practice, polices are often based on ideology instead of practical, evidence-based measures that have been tested and proven to work. Each new government wants to put their own stamp on a particular policy or program. But new policies often recycle failed policies of the past, or throw good programs out with the ‘bathwater’.

“There is a level of frenetic chopping and changing, and policy pulsing, that comes with electoral cycles and as the political pendulum swings from left to right…decision-making in Indigenous policy feels much like a merry-go round—replete with the same old traps and reinvented wheels.”6

A case in point is the Community Development Employment Program (see Box 1 overleaf), which has suffered, perhaps more than any other Indigenous program, from political pendulum shifts. 7

The previous report, ‘Mapping the Indigenous Program and Funding Maze’ found there needs to be a much more rigorous process for allocating funding for Indigenous programs and for making decisions about which programs continue to receive funding. 14 The inquiry into the tendering process for the Indigenous Advancement Strategy (IAS) funding criticised the procedures used by

government and recommended a full internal review by the Australian National Audit Office (ANAO).

The ANAO report found the Department of Prime Minister and Cabinet (PMC) had not implemented the Strategy effectively, and the grants administration processes “…fell far short of the standard required to manage billions of dollars of funding.”15 In particular, the Department was found to have not:

• assessed applications in line with the guidelines and public information provided by the Department

• met some of its obligations under the Commonwealth Grants Rules and Guidelines;

• kept records of key decisions; and

• established performance targets for all funded projects.16

Nor did the Department advise the Minister of the risks involved in implementing the Strategy in such a short time frame. According to the Australian Public Service Commission, such timidity by public servants is reportedly becoming more common, which is a worrying sign, as a well-functioning government is reliant on the provision of free and frank advice to Ministers.17

Although a performance framework was established for the Strategy, the framework did not facilitate assessing whether program outcomes had been achieved. This therefore inhibited the Department’s ability to

The case for reform


Box 1: CDEP to CDP — an example of government failure

Initial design was a community initiative and focused on community development: The first CDEP scheme was introduced in 1977 in Bamyili, a remote Indigenous community in the Northern Territory, as an alternative to unemployment benefit payments and as an instrument of community development. Instead of individual income support payments, the money was pooled to fund community development projects and to employ people. Significantly, the scheme was a community initiative rather than a government-designed and imposed program.8

Reiteration of CDEP to expand it into urban and rural areas and for it to be a transition to work program: In the mid-1980s, CDEP became part of the Aboriginal Employment Development Policy (AEDP) and was expanded into Indigenous urban and regional communities as a transition-to-work program. However, by the late 1990s, issues with the reframing and expansion of CDEP were becoming increasingly apparent. An evaluation of CDEP in 1997 found that at least 33% of CDEP participants did no work.9 More than half, or 60% of CDEP organisations paid people for home duties and mowing their own lawns. Only about 5% of CDEP participants moved from CDEP to real jobs and more than 40% of Indigenous people on CDEP from remote communities had been on CDEP for five years or more. According to a government discussion paper, CDEP had “become a destination rather than a stepping stone towards jobs.”10 There were a number of important reasons why CDEP was not meeting its objectives.

1) There were few jobs for people to transition to in remote areas.

2) There were no incentives to transfer people to mainstream jobs, particularly in remote areas where CDEP funding was used to fund local government, health, education, and policing services.

3) There was no recognition of the need to modify the program depending upon location (ie. it may have been realistic to expect it to be a transition to employment program in mainstream areas but not in remote areas where it needed to take a more community development approach and actually create jobs).

4) There was not enough accountability of CDEP providers, with no repercussions if participants were paid for doing nothing.

Despite the problems with CDEP, some providers were actually doing a good job.11 But rather than learning from these success stories and reforming CDEP to ensure the program was meeting its objectives, or assessing whether the program’s objectives were even achievable, the government decided to abolish CDEP; replacing it with the Remote Jobs and Community Program (RJCP) in 2013.

Remote Jobs Community Program (RJCP) at odds with original intent of CDEP: Where the original CDEP program had been a community initiative aimed at avoiding the negative repercussions of welfare by pooling community members’ social welfare payments, RJCP was a top-down government-controlled program. Its emphasis was on getting Indigenous people into employment and fining those who failed to meet their activity requirements. Unlike CDEP which had large community support, RJCP failed to resonate with communities and had very burdensome administrative arrangements. The pendulum had swung too far towards a punitive model.

Rebadged RJCP to CDP: The unpopularity of RJCP and the high administration costs led the Coalition government to amend the program and change its name to the Community Development Program (CDP). Some people argue the similarity in names between CDEP and CDP was a deliberate ploy to try and get community buy-in. The then Prime Minister Tony Abbott admitted that: “Abolishing CDEP was a well- intentioned mistake and CDP is our attempt to atone for it.”12

CDP: Along with the name change, the government announced there would be more consultation with communities about what projects and activities they wanted, and less red tape. Despite this, a number of people continue to think the CDP program is too punitive and does not take into account the challenges people living in remote communities face; such as the lack of jobs. A recent report by the Australian National University found 146,000 financial penalties had been applied to 34,000 CDP participants in 2015–16, compared to 104,000 penalties to approximately 750,000 job-active participants in mainstream Australia.13 It seems the original reason CDEP was established the lack of a real economy or many job opportunities in remote Indigenous communities continues to be ignored.

Lessons to be learnt:

1) Before scaling up programs, check if the objectives need to be modified/tailored to different regions.

2) Don’t throw the baby out with the bathwater — learn from previous mistakes and successes about what does and does not work.

The pendulum swings with CDEP (and its replacements) are illustrative of the failings in going too far in either direction. Too lenient and there tends to be an absence of accountability — as evident in CDEP participants receiving money for doing nothing at all, but too far the other way and approaches tend to be excessively punitive.

To be effective, Indigenous policy initiatives need to adopt a middle ground — where there is accountability and oversight but the need for community involvement and flexibility is also recognised.


national program ‘Tackling Indigenous Smoking’ and the Victorian program ‘Yarning it Up — Don’t Smoke it Up’. The proliferation of tobacco cessation programs is probably due to the way funding is provided under the federal government’s Tackling Indigenous Smoking regional grants program, which provides grants to support locally designed anti-smoking and smoking cessation programs.

A review of Tackling Indigenous Smoking was commissioned by the Department of Health in 2014. The review found evidence that multi-level approaches to tobacco control were the most effective at reducing smoking prevalence in Indigenous Australian communities. At the same time, the review also found a lack of monitoring and evaluating of the programs. Therefore, although the review recommended retaining the flexibility of the funding approach to tailor programs at the local level, it also recommended integrating a reporting and evaluating framework into future iterations of the program to develop a stronger evidence base around effectiveness of the program.27 Following the review, the Department of Health introduced a revised Tackling Indigenous Smoking program with a budget of $116.8 million over three years ($35.3 million in 2015–16; $37.5 million in 2016–17 and $44 million in 2017–18).28

Despite the increase in the number of Indigenous programs, some communities continue to miss out on essential services. For example, Fitzroy Crossing in East Kimberley suffers from one of the highest incidents of foetal alcohol spectrum disorder (FASD) in the world, but one of the town’s most effective prevention initiatives is in danger of closing. An early learning centre that provides pre-natal and post-natal care to mothers and tuition to parents, as well as childcare, is set to close next year under changed subsidy arrangements that will see it lose $500,000 from its annual budget of $1.2 million.29 Six years ago, when alcohol restrictions were first introduced in Fitzroy Crossing, a study by Notre Dame University noted there were significant gaps in support services in the community. Most damning was the fact that while alcohol restrictions had been introduced to try and combat the epidemic of alcoholism in the town, there was no resident alcohol and drug counsellor or mental health worker. The community was serviced only twice a month by two regional mental health workers from Derby (a town several hours away).30 These are not isolated, one-off examples, they are endemic to the Indigenous program and service sector.

Mark Moran’s book Serious Whitefella Stuff illustrates through a selection of case studies how governments often make decisions without involving local Indigenous people and cut funding to programs without any assessment of their effectiveness, even though there is now widespread recognition of the importance of engaging with local Indigenous people in the design and implementation of programs.31 According to Fred Chaney: “The system under which we operate is broken, and it is the broken system that we should be evaluating.”32

“effectively verify, analyse or report on program performance.” 18 The Department had reportedly started evaluating some individual projects but had not adopted an evaluation strategy.19 A draft evaluation and performance improvement strategy had been developed, and was considered by the Indigenous Affairs Reform Implementation Project Board in July 2014, but the plan was not formally agreed to, endorsed or funded. 20

Worst of all, however, was that the Department did not document the processes they used when awarding contracts. The widespread awarding of contracts to non-Indigenous organisations meant many Aboriginal organisations had their funding reduced or missed out on funding entirely.21 Public hearings during the parliamentary inquiry into the IAS were filled with stories of organisations losing funding for programs that had run successfully for decades.22 An example was the Djarindjin domestic violence shelter on the Dampier Peninsula in Western Australia. The shelter is run by local Aboriginal women and services 50 Aboriginal communities 200 kilometres north of Broome. After their plight attracted considerable media attention, funding for the shelter was reinstated. However, there were many other organisations that were not so fortunate.23

The IAS funding process is symptomatic of a deeply flawed system that has led to gaps in programs and services in some areas and duplication and waste in others. Yet, the problems existed before the IAS, as former Northern Territory Co-ordinator General for Remote Services, Olga Havnen documented in her Remote Services Report in 2012:

“There are not only massive pre-existing service gaps but also a serious lack of high quality, evidence-based program and service development…This lack of long-term strategic vision means governments have spread resources as widely as possible in a ‘scatter-gun’ or ‘confetti’ approach. This results in partially funding community initiatives for short periods with no long term strategy for how the positions created or initiatives undertaken will be sustained.”24

Soon after the release of this report, Olga Havnen was sacked from her position as Co-ordinator General for Remote Services.25

Since there is no strategic oversight, nor a requirement for an evidence base for funding, the number of Indigenous programs has increased over time with no appreciable improvements in outcomes. When the review of programs on the Indigenous HealthInfoNet was done at the beginning of January 2016, there were 2468 programs listed on the website, of which 2024 were Indigenous-specific. Over a year, the number of programs has increased by 383 to 2851.26

The way programs are funded through multiple small grants contributes to the growing number of programs. Our research identified at least 30 different Indigenous tobacco cessation programs (see Appendix C). Of these 30 programs, only two had been evaluated: the


There are many reasons for conducting evaluations of programs. For example: to highlight what is and is not working; to inform decision making about allocation of resources; or to improve service delivery and client satisfaction with a program (see Appendix B for Evaluation Toolkit and a more detailed explanation). Ultimately, evaluation is necessary to ensure government is held accountable for monitoring how organisations are spending taxpayers’ money. Yet, there must be co-accountability — the organisation receiving the funding must be held accountable for how they have spent the money and whether the program has achieved its desired outcomes, and the government agency must be held accountable for monitoring whether the organisation is meeting its objectives and work with them to improve their practices if they have not. As Australian National University academic Will Sanders has argued: “Government must not prioritise excessive accountability to bureaucrats over accountability to communities.” Organisations are accountable to the government agency funding them, but the government is accountable to the community.

Improved accountability, however, does not mean there has to be detailed daily monitoring of the activities of both providers and participants. If there is any lesson to be learnt from the failed RJCP, it is that excessive monitoring can be a huge administrative burden for little gain.33 There needs to be an appropriate balance between maintaining program fidelity and allowing organisations

a certain degree of flexibility to tailor the program to meet community needs. This approach is different from traditional ideas of accountability, and involves moving away from simply monitoring and overseeing programs to supporting a learning and developmental approach to evaluation.34

It is also not enough to just evaluate; government must use the information from evaluations and reviews to improve service delivery.35 There is considerable evidence to suggest that even when programs have been evaluated, governments have not used the findings to inform funding decisions. For example, according to a report by Olga Havnen, the former Northern Territory Coordinator-General for Remote Services a non-government organisation (not named in report) was contracted to deliver a multi-million dollar program ($5 million over three years) in five Northern Territory communities.36 An evaluation of the program mid-term revealed “serious deficiencies” in the way the program was delivered, and the conduct of staff employed by the organisation. Despite the poor findings of the evaluation, the organisation was invited by the federal government to submit a proposal for the continuation and expansion of the program.37

Another example is a recent Indigenous health campaign – No Germs On Me – which ran three different television commercials encouraging people to use soap when they washed their hands. Although the evaluation found no

Why evaluate?


change in participants’ beliefs, behaviours or attitudes as a result of the campaign, the evaluators concluded the reach of the advertisement was satisfactory and the campaign was worth continuing.38

Every state and territory has some sort of evaluation or data monitoring guideline or strategy (see Table 1). Despite all these strategies and guidelines, a recent audit of the NSW Evaluation strategy by the Audit Office of NSW found the NSW Treasury and NSW Department of Premier and Cabinet were not using evaluation outcomes to inform and improve practices. According to the audit:

“The NSW Government’s program evaluation initiative is largely ineffective, as it is not providing sufficient information to government decision makers on the performance of programs. For program evaluation to be effective, agencies should demonstrate they are evaluating the right programs, and the outcomes from completed evaluations should inform advice to the NSW Government on investment decisions.”39

Table 1 State and Territory Evaluation Strategies

Type of documentation Key features

NSW The Centre for Program Evaluation and capability building

NSW Government Program Evaluation Guidelines(2016)40

NSW Evaluation Toolkit 2016

Guidelines are a comprehensive document with best practice principles and links to other websites with other evaluation material.

VIC Evaluation Step-by-Step Guide (2008).41

Funded Organisation Performance Monitoring Framework (2017).42

Guide is for evaluation contractors — provides four steps for managing an evaluation. Useful material.

Performance framework for monitoring funded organisations.

QLD Queensland Government Program Evaluation Guidelines (2014).43

Comprehensive document, similar advice to NSW and Victoria guidelines but better use of diagrams/tables to explain evaluation processes.

TAS Planning, evaluation and procurement guidelines (Tasmanian Government 2015).44

Guidelines are focused on communication and not as comprehensive as other evaluation guidelines. Useful link to Tasmanian Government approach to collaboration.

SA Managing a Community Organisation Evaluation (Social Inclusion, 2016).45

Guidelines directed at community organisations. Website has a series of six steps to follow when conducting or managing an evaluation.

WA Program Evaluation Unit (PEU) within the Department of Treasury.46

Program Evaluation website.47

Program Evaluation Guide, 201548

Comprehensive guide but with almost identical material as other guidelines, some useful links to other sources though and a helpful program logic table with examples.

NT Good Practice Guidelines for Funding Non-Government Organisations (2015).49

Not evaluation guidelines but rather guidelines for funding NGOs to ensure accountability and to achieve quality and value. Not as comprehensive as others states and territories.

ACT ACT Government Evaluation Policy and Guidelines (2010).50

Quite a comprehensive document, with similar material to other state/territory guidelines. Useful table on the benefits of evaluation.


Research for our previous report, ‘Mapping the Indigenous Program and Funding Maze’ identified 1082 current Indigenous-specific programs. Of these:

• 49 were federal government programs;

• 236 were state and territory programs; and

• 797 were programs delivered by non-government organisations (though many of these are funded in part or full by government).

Of the 1082 programs only 88 (8%) were found to have been (or were in the process of being) evaluated.

The largest category of programs were health related programs (n=568) followed by cultural programs (n=145) then early childhood and education programs (n=130) — see Figure 1.

The program category with the highest number of evaluations was health (n=44), followed by early childhood and education (n=16). However, percentage wise, more programs were evaluated under the jobs and economy category (15%) than the other program categories.

Of the 490 programs delivered by Aboriginal organisations, only 20 were evaluated (4%). The small number of businesses delivering a program (n=6) meant that while there were only two evaluations of Indigenous programs provided by a business, this category had the highest percentage of programs evaluated (33%). Similarly, while only six of the 33 programs delivered by schools and universities were evaluated, this category had the second highest percentage of programs evaluated (23%). Conversely, government and non-Indigenous NGO delivered programs had the highest number of evaluations, n=36 and n=24, but much lower percentages of evaluations as the number of overall programs was higher, n=278 and n=276.

Analysis of program evaluations

Figure 2: Number and percentage of evaluations by category

Source: Authors’ calculation based on a review of government, major philanthropic and NGO websites, and programs listed on the Australian Indigenous HealthInfoNet.

Figure 3: percentage of Indigenous programs evaluated by provider

Source: Government websites, major philanthropic and NGO websites, and programs listed on the Australian Indigenous HealthInfoNet.

Figure 1: Number of programs by category and number of evaluations by category

Source: Government websites, major philanthropic and NGO websites, and analysis of IAS funding recipients and programs listed on the Australian Indigenous HealthInfoNet.


Not all evaluations are equal. Many evaluations are akin to a ‘tick box’ exercise, with limited data available to measure impact. The primary focus of these types of evaluations appears to be participation in the program, or throughputs, rather than outcomes. A number of program providers seem reluctant to admit the failings of their programs and their evaluation reports read more like exercises in public relations than independent and rigorous analysis. The purpose of conducting an evaluation should be to look at what is and is not working, what some term a ‘warts and all’ evaluation.51 However, for many not-for-profits, the pressure not to publish negative evaluations is high, with specific concerns ranging from whether negative publicity will affect funding, to how staff working on the ground may perceive any criticism of the project.52 Similarly, if the findings of a government evaluation are particularly negative, it is not uncommon for government to insist that the results are not published.53 Evaluations of government programs are often conducted by the department responsible for funding or delivering the program, and even if an external evaluator is used, their ‘independence’ is compromised by the client relationship.54 How much independence can a consultant claim to have when they are reliant on their clients for business?55

Consultants can sometimes be pressured to frame the results of evaluations in a certain way and to downplay any negative findings. For example, a recent evaluation of the cashless debit card trial, came to some surprising conclusions about the effectiveness of the trial, given the weight of evidence to the contrary.56

In determining what constitutes a rigorous evaluation, state and territory evaluation guidelines provide examples of principles of ‘best practice’. The NSW Program Evaluation Guidelines contain nine principles of

best practice, these are:

1. Build evaluation into your program design.

2. Base your evaluation on sound methodology.

3. Include resources and time to evaluate.

4. Use the right mix of expertise and independence.

5. Ensure proper governance and oversight.

6. Be ethical in design and conduct.

7. Be informed and guided by relevant stakeholders.

8. Consider and use evaluation data meaningfully.

9. Be transparent and open to scrutiny.57

However, having principles and actually applying them are two different things. For instance, although evaluations should be built into the program design, in practice this does not always happen. Often evaluators are asked to evaluate a program after it has been running for a while but when there is no pre-program data or even any uniform collection of administrative data. As a result, the evaluation is not as useful as it could have been if the evaluation and implementation of the program had occurred concurrently.

The second principle, basing your evaluation on sound methodology, also sounds like common sense. Yet although there is generally agreement on a hierarchy of evidence, with meta-analyses of multiple randomised trials at the top (see Box 2), in practice, RCTs of Indigenous programs are very rare. In fact, none of the evaluation of Indigenous programs reviewed in this report used RCT. However as mentioned earlier, the Australian government is starting to invest in the method, with funding for two RCTs of Indigenous programs recently announced.58

Analysing the evaluations: A hierarchy of evidence


Box 2. Proposed Hierarchy of Evidence

Shadow Assistant Treasurer Andrew Leigh’s hierarchy of evidence involves six levels, ranging from systemic reviews at the top to expert opinion and theoretical conjecture at the bottom.

1. Systemic review (meta-analyses) of multiple randomised trials

2. High quality randomised trials

3. Systematic reviews (meta-analyses) of natural experiments and before-after studies

4. Natural experiments (quasi-experiments) using techniques such as differences-in-difference, regression discontinuity, matching or multiple regression

5. Before and after (pre-post) studies

6. Expert opinion and theoretical conjecture.

Box 3. Evaluation of ACT Extended Throughcare Pilot Program

A recent evaluation of the ACT’s pilot Throughcare program, conducted by Social Research Policy Centre, has revealed issues with establishing a satisfactory RCT. The evaluation sought to rely on a RCT sample of participants who did not take part in the program, for the period June 2013– June 2016, as the control group. This sample group was ‘insufficient’ as the number of participants in the program was cited as being ‘very high’ and therefore there were very few non-participants.

In an attempt to rectify this issue, a sample group was developed from the period 2010–2013, prior to the implementation of the program. This data had differing baseline characteristics and was supplemented with ‘before and after custodial episode data’ to attempt to account for this.

Another issue with the evaluation was that there was little data on outcomes for Indigenous people. The study highlights that, of the Indigenous male study group, 57.4% returned to custody compared to 38.3% of the control group. For Indigenous females, the figures were 28.6% returning to custody compared with 33.3% of the control group. Figures for recidivism rates were provided by ACT Corrective Services in Productivity Commission’s Report on Governments Services (ROGS), but these do not explicitly identify the rates for Indigenous people. This highlights that despite Indigenous people being significantly overrepresented in the prison population, data is lacking on the Indigenous experience and outcomes in the program.63

While there is general agreement that RCTs are the gold standard of research evidence, there are some dissenting voices on the exact order of Leigh’s hierarchy; for example, whether systematic reviews are a more rigorous methodology than genuine quasi-experimental work.59 University of Wollongong academic Peter Siminski argues that: “studies relying only on matching or multiple regression are a lower grade of evidence than genuine quasi-experimental work.”60 Quasi-experimental impact techniques are gaining in popularity as they are typically much cheaper, and face less practical barriers to implementation, than RCTs (see Box 3 for an example of some of the challenges in implementing RCTs), though, only one of the evaluations of Indigenous programs reviewed for this report adopted this type of approach. The issue in Australia is that there are few people who have the training required to conduct high quality quasi-experimental work. The fact that RCTs and quasi-experimental evaluations require highly trained practitioners to carry out the evaluations restricts their usage and arguably is a reason why alternative methods of evaluating Indigenous programs should be considered.

It is also important to note that there is a difference between a health or early-childhood intervention and a program. There may be evidence for the benefit of

the intervention but not evidence on how best to deliver that intervention as part of a program. For example, a review of Indigenous health projects in WA found there was a ‘disconnect’ between the strong scientific evidence for the health interventions and the way the service sector was delivering the health intervention.61 The success of the program was strongly influenced by the staff’s knowledge and familiarity with the interventions they were promoting or delivering. Research on ‘implementation science’ (how to implement evidence-based research into practice) has found it can take about 17 years for research evidence to be incorporated into health care practices.61 Program evaluations are also more challenging than measuring the benefit of a particular intervention, as programs to address complex social problems are likely to have multiple objectives.62

The underlying reason for conducting evaluations is to improve the delivery of programs and to achieve better outcomes. There is no point in evaluating programs and interrogating the standard of evidence if programs are not designed to use the evidence from evaluations to improve practice. As a result, it may be necessary to reach a compromise between what is considered the ‘gold standard’ in terms of research evidence and what is practical and achievable given limited resources.


Productivity Commission’s criteria for evidence of ‘what works’

In the Overcoming Indigenous Disadvantage Report, the Productivity Commission used a set of criteria to select case studies of programs or services they considered were having a positive impact on improving outcomes for Indigenous Australians.

The criteria used to select the case studies were that the program had:

• Measurable, up to date outcomes

• A reasonable track record of success (though what this means is not defined)

• Support from local Indigenous people who had used, or were affected by, the program; and

• Where possible, include an analysis of costs and benefits.64

The rigour in the selection of case studies resulted in only 24 program evaluations being included in the report (though 10 more case studies of promising programs that had not yet been evaluated were also included).

Despite the relatively high number of evaluations of Indigenous health programs, the Productivity Commission found a lack of evidence on interventions to address a range of different health indicators measured in their Overcoming Indigenous Disadvantage report.65 For instance, they considered that there is currently no evaluated program on approaches that work to reduce smoking or alcohol consumption during pregnancy. Nor is there a published robust evaluation of interventions that contribute to a decrease in the prevalence of tobacco smoking for Aboriginal and Torres Strait Islander people, even though there has been a proliferation of tobacco cessation programs under the federal government’s Tackling Indigenous Smoking program.66 Other gaps in evidence identified by the Productivity Commission included the lack of research and program evaluation on Indigenous school engagement and the absence of evaluations of programs that work to improve home ownership for Aboriginal and Torres Strait Islander people.67

The 88 program evaluations identified in our research were compared with the Productivity Commission’s evaluations in their Overcoming Indigenous Disadvantage Report. Overall:

• 12 of the Productivity Commission’s 24 programs were included in the 88 program evaluations our research identified

• 5 of the Productivity Commission’s programs were not Indigenous-specific (a criteria for programs to be included in our research); and

• 7 evaluation reports were added to our literature review (these additional reports did not come up in our initial desk-top review of publicly available program evaluations).

Our criteria for evaluating the ‘evaluations’

In developing a method for ranking the evaluations identified in our research the following scale was used:

• Weak — limited methodology reliant on qualitative evidence or a survey with a small sample size, no pre and post data, or only a summary of full evaluation report publicly available

• Moderate — a mixture of qualitative and quantitative data, some attempt at triangulation of data (cross verification from two or more sources), some evidence of impact but no pre and post data and no control groups.

• Strong — a mixture of qualitative and quantitative data with evidence of triangulation of data. Evidence the program is having an impact through the use of pre and post data or other benchmarking data. The use of experimental design/random control trials/ or control group. Or in the absence of that, evidence the evaluation utilises in addition to triangulation of data and benchmarking one or more of the following: an economic component through either a cost benefit or cost effective analysis or some mention of the financial impact of the program and or meta-analyses — reviews of multiple evaluations.

Some flexibility had to be employed in developing this list of criteria, as none of the evaluations reviewed employed RCTs. Therefore, evaluations were considered strong if they involved triangulation of data and two or more of the following: control group; meta-analyses; and cost effectiveness. This approach to weighting the methodology was based on the Victoria’s Department of Treasury and Finance’s report ‘Guide to Evaluation: How to plan and conduct effective evaluation for policy and programs’, which ranked different evaluation data and method types by level of sophistication.68 The initial identification of Indigenous program evaluation reports was quite broad and included audits and reviews of programs. At the same time, the focus on evaluation excluded some other program reports, such as case studies, Cost Benefit Analysis (CBAs) and Social Return on Investments (SROIs) (see Appendix A for description of CBAs and SROIs). When these were included, a total of 111 reports were identified. These 111 reports were then broken down into five categories, evaluations, audits, reviews, CBAs, SROIs and others (ie case studies). In total, 75 evaluation reports were identified (though some of the programs had an evaluation report and a CBA report in which case only the evaluation report is included in the table below so the program is not double counted).


Of the 111 program reports identified in our research, only 71 were reviewed in detail (5 CBA, 6 SROI reports and 60 evaluations/audits/reviews), as the full text of the remaining 40 evaluation reports was not available — see Appendix A for summary tables of our assessment of the evaluation methodology.

In total, only 49 of the 60 program reports, were able to be assessed against the scale (weak, moderate, strong) identified above, as the other 11 were not evaluation reports, but audits or reviews.

Overall our findings identified that:

• 23 evaluation reports had weak methodology

• 23 evaluation reports had moderate methodology

• 3 evaluation reports had strong methodology

Table 2 Breakdown of Indigenous program reports by type

Category Evaluation Audit Reviews CBA SROI Other Total

Crime 8 1 1 1 10

Culture 4 1 1 1 1 8

Education 15 2 5 22

Health 41 2 5 1 3 52

Housing/ 4 4 2 1 11

Jobs 3 3 7

Transport 1 1

75 7 9 3 6 11 111

Source: Government websites, major philanthropic and NGO websites, and programs listed on the Australian Indigenous HealthInfoNet.

that could affect the impact of the program and undertake sensibility analysis that considered different scenarios and to change assumptions accordingly.

Table 4 highlights the findings of the assessment of the SROIs. Overall, only 50% of the SROIs appeared to have measurable objectives or to look at the impact of the program in context. In addition, only two of the six SROIs used a discount rate in their methodology. At the same time, while the criteria used to assess the SROI is helpful in terms of evaluating the SROIs, the checklist does not tell the complete story about the quality or depth of the analysis underpinning the measurement of outcomes. For example, there were some SROI reports that did not include all the criteria, but still demonstrated sound analysis.

Table 4 Analysis of SROI methodology

Criteria Yes No

Measurable objectives 3 3

Quantification 6 0

Sensibility analysis 4 2

Inputs/Outputs 4 2

Impact in context 3 3

Discount rate 2 4

NPV calculations 5 1

Table 3 Analysis of CBA methodology

Criteria Yes No

Measurable objectives 1 4

Identification of options 1 4

Proper quantification 4 1

Sensibility analysis 3 2

Equity implications 1 4

Discount rate 2 3

NPV calculations 4 1

Figure 4: Rating of evaluation methodology of Indigenous programs

In general, Indigenous evaluations are characterised by a lack of data and the absence of a control group, as well as an over-reliance on anecdotal evidence.

Table 3 highlights the findings of the assessment of CBAs and the criteria used to determine the effectiveness of the methodology used. Overall, few CBAs appear to have measurable objectives, or to identify a range of options and use equity implications. Conversely, good examples of CBA reports tended to consider possible constraints


Lessons to be learnt

From our assessment of the evaluation methodology the following lessons can be drawn for policy makers and program providers.

Table 5 Lessons to be learnt about evaluation

Focus area Advice

Methodology • It is important to use a mixed methodology and not just rely on qualitative evidence

• A case study or review should not be considered less rigorous than an evaluation, in fact some case studies may utilise a more robust methodology than many evaluations

• There is potential for biased samples when program participants receive benefits from taking part in the program

Data • The same standards of data collection need to be upheld in each program location in order for effective comparisons to be made

• It can be difficult to measure changes in behaviour if the right administrative data is not available or collected

• Program providers need to have strategies for recording and accessing administrative data before the program is rolled out, particularly for a small cohort of program participants where there are potential privacy concerns

Analysis and reporting

• Strong analysis can overcome some of the limitations of a small sample

• It is important to take into account the environment programs are operating in, and that some programs may have their impact minimised because they do not have certain authorities

• Evaluation reports need to be clear about whether the evaluation is on the framework/service delivery or the impacts/results the program produces, or a combination of both

Program design and delivery

• There need to be effective links between policy and program initiatives

• While the general model of a program may be transferable, much of the successful implementation of programs depends on having the right combination of people with the appropriate knowledge and skills

• People delivering programs need ongoing training to ensure they have up-to-date information on the evidence available about best practice approaches

• Participants are more likely to provide honest feedback on a program when program staff have made an effort to establish positive relationships with them; this is particularly the case when the intervention being delivered by the program is of a sensitive and private nature

Examples of successful practices

In addition to the lessons that can be learnt from problems evaluating programs, there were also some examples of good practice. Particular features that made these evaluations stand out from the rest included:

• A mixed method design, which involved triangulation of qualitative and quantitative data and some economic components of the program such as cost effectiveness

• Local input into design and implementation of the program to ensure program objectives matched community needs

• Clear and measurable objectives

• Pre and post program data to measure impact

The following case studies illustrate examples of rigorous evaluation practice and/or successful programs that are regularly monitored and evaluated.


Ganbina: evidence rating = strong

Ganbina was established in 1997 to help improve school and further education completion rates and ‘real’ job prospects among about 6000 Indigenous people in the Goulburn Valley in Victoria. The program receives no government funding, relying instead on philanthropic and corporate sponsorship for its activities on an annual budget of $1.4 million. According to Ganbina’s Chief Executive, Anthony Cavanagh, “Not seeking government funding is a choice and allows the program to be innovative…” 69

An independent evaluation by Price Waterhouse Coopers in 2014 found very high Year 12 completion rates (100% in 2014) and high retention rates (over 95%). Ganbina’s cost per participant of approximately $3500 was about half the average spend of other similar type programs. Despite costing less to run, it also had the highest retention rate, gender balance and broadest age group of other comparable programs.

An Impact Assessment was conducted by Social Ventures Australia (SVA) in 2016 to assess the cumulative impact of the program since it was first implemented in 2005.

The methodology used consisted of a desktop review of client data and previous evaluations and data collected on Ganbina and consultation with stakeholders.

The Impact Assessment found Year 11 to Year 12 retention rates increased from 62% in 2009-10 to 73% in 2015-16, which was considerably higher than the rate for Indigenous people in the Greater Shepparton area and national Indigenous rates. Ganbina achieved a 100% success rate for participants who had taken part in the program for five years or more and who were aged between 25-34 years, with all achieving a Year 12 or equivalent qualification.

University participation increased from two Ganbina participants in 2009 to 15 in 2016.

Key features of the program:

Does not receive any government funding, which has enabled it to adopt a more innovative and cost-effective approach (much cheaper than other comparable programs to run)

Complete transparency with six-monthly reports provided to investors and bi-monthly newsletters that document exactly how much funding has been used on administration and how much is left.

Aboriginal Maternity Group Practice Program (AMGPP): evidence rating = strong

The AMGPP provides free antenatal and postnatal clinical care, to pregnant Aboriginal women. Each client is supported by a team of health professionals during pregnancy and for four weeks after they have given birth. Support provided includes clinical care and cultural, social, and emotional care and support.

The evaluation involved a non-randomised intervention study using data from the Western Australian Midwives Notification System. Methodology used included regression models to analyse data from 343 women (with 350 pregnancies). The analyses included developing historical and contemporary control groups of pregnant Aboriginal women and matching them for maternal age.

Participation in the AMGPP was associated with significantly improved neonatal health outcomes. Babies born to AMGPP participants were significantly less likely to be born preterm 9.1% versus historical controls of 15.9%.

Key features of the evaluation/study:

Quasi-experimental design involving regression analysis and matched control group to show the impact of the program.


Australian Electoral Commission’s (AEC) Indigenous electoral participation program: Evidence rating = moderate

The IEPP program is aimed at empowering Aboriginal and Torres Strait Islander Australians in exercising their right to vote.

The evaluation methodology included: a literature scan and document review; semi-structured interviews with staff; focus groups; case studies with a cross section of communities and analysis of data available from the Queensland and the Northern Territory elections.

The evaluation found variation in the degree to which IEPP’s stated objectives and outcomes have been achieved. The evaluation recommended basing future changes to the program on evidence of ‘what works’, and harnessing the experiences of other government agencies and programs working in an Indigenous context. This includes the adoption of a robust monitoring and evaluation system and routine analysis of performance data.

Key features of the evaluation:

Methodology was relatively robust and included a Monitoring and Evaluation Framework and triangulation of qualitative and quantitative data. Information on the number of people who participated in interviews/focus groups was provided and interview guides were provided in appendices. However, it was difficult to identify changes in electoral behaviour as ethnicity was not recorded on the electoral roll or when people vote. Performance data for the program was also not entered uniformly or consistently by States and Territories.

Indigenous Community Volunteer (ICV) Program: Evidence rating = moderate

A case study which incorporated a social and economic impact assessment was conducted by KPMG in 2015.

The ICV program is a registered charity and non-profit community development organisation that matches volunteer’s experience and skills with different Indigenous communities needs to help address Indigenous disadvantage. In 2013/14 ICV worked with 169 communities.

Assessment of activities in two communities involved stakeholder consultations and document and data analysis, including assessing the impacts of the activities in economic terms.

The study found there was evidence ICV was invited into communities and involved in discrete, well defined projects, and that volunteers were providing a positive impact and building on existing work that had been done in the community. There was also evidence that ICV had developed positive partnerships with other organisations and were collaborating with them on activities.

Key features of the evaluation and program:

Study involved triangulation of data from multiple sources, including analysis of economic data. However study only looked at two communities so difficult to extrapolate about overall program impact.

There was evidence of good practice in program design and implementation with volunteers ensuring communities wanted their assistance and only working on discrete well-defined projects in collaboration with other organisations involved in similar activities.


As discussed above, it may be necessary to reach a compromise between what is considered the ‘gold standard’ in terms of research evidence and what is practical and achievable given limited resources. There is also no point conducting ‘rigorous’ evaluations, if the evidence is not used. As a result, instead of focusing on having the highest standard of evidence for assessing the impact of a program (such as in RCTs), it may be more practical to consider how to ensure evaluation learnings are used to inform program practice. Figure 4 shows an alternative hierarchy, where the minimum standard is evidence of learnings being applied to improve program outcomes, and the highest level is where there is evidence of the impact of the program and the benefit of the particular intervention in addition to learning and program improvement.

Government departments administering funding may conduct an evaluation to analyse funding distribution and to report on the achievements and impact of the program.70 However, these types of evaluations can make organisations feel like they have to pass a test in order to continue to receive funding and they may resist the evaluation process as a result. Resistance could be indirect or subtle, such as avoiding or delaying entering program data into databases. There is evidence to suggest organisations are more likely to engage with the evaluation process when it is presented as a learning tool to improve program delivery than when it is presented as a review or audit of their performance.71 This is particularly the case if they are given the opportunity to provide input into the evaluation plan or framework, so they can see the benefit of the evaluation activities in documenting the impact of the program and contributing to evidence about what works. Evaluation as a learning tool could be considered similar to continuous quality improvement processes in the health sector and usually involves ‘reflective practice’ to help identify and address issues with program design or delivery (see Appendix B for Evaluation Toolkit which explains reflective practice in more detail).

A reflective practice approach to evaluation relies on a two-way exchange, with the experiences of those on the ground delivering the program being used to inform the ongoing implementation of the program. This is different from a government top-down technocratic approach, which might have strict accountability measures in place, but fails to recognise there may be better ways of delivering the program (see Table 6).

Another way of describing this iterative approach is ‘developmental evaluation’ — a relatively recent evaluation methodology that seeks to combine the rigour of evaluation with the flexibility and innovation of developmental approaches to social problems. The primary focus of developmental evaluation is adaptive learning to inform the implementation of programs or community development initiatives.72

The following text boxes provide examples of programs that have adopted an iterative or developmental approach to evaluation and that have used evaluation findings to improve the program. Neither of these two programs was reviewed as part of the assessment of evaluations as Ability Links NSW is not an Indigenous specific program and The Martu Leadership Program (MLP) evaluation report was only released in April 2017, after the analysis of the evaluations was completed. However, they are included as they provide the best examples of programs where evaluation has been embedded into the delivery of the program and reflective/developmental approach to evaluation is used.

Figure 5: Evidence of program impact and learnings

Table 6 Differences between top-down and bottom-up approaches in program design and evaluation

Top-down Bottom-up

Approach Technocratic/evaluator as expert Participatory/community engagement/empowerment

Orientation Identifying weaknesses, problem or deficit

Strengthening capacity/improving competence

Who defines the issue/need? Outside agent (government) Community

Evaluation methodology Quantifiable outcomes and targets Pluralistic methods, documenting changes of importance to Indigenous community

Source: Adapted from Laverack, 2000


Ability Links NSW and Early Links NSW EvaluationAbility Links NSW (‘ABNSW) is a program that was developed by the NSW Department of Family and Community Services through extensive community consultation to provide greater flexibility and control in the way services are delivered to people with disability. A concurrent program was developed for children and young people called Early Links NSW (‘ELNSW’).

ALNSW is staffed by ‘Linkers’ who work alongside people with a disability or their carer and assists in life planning as well as connecting them to relevant community organisations. The program aims to empower people with a disability to make their own decisions and work towards achieving what is important for them. The program also includes community engagement where Linkers work with community organisations to assist them to improve services and support for people with disability.

ALNSW commenced as a pilot in 2013/14 and was rolled out state-wide from July 2015. ALNSW was designed with evaluation in mind from the very beginning and evaluation processes were therefore embedded in the roll-out of the program. Urbis was commissioned by both ALNSW and ELNSW to evaluate the program over three years from 2013–2016, with Interim Evaluation Reports delivered annually.

The evaluation itself was uniquely designed as a collaborative joint approach, involving extensive participation at a community level (either people with a disability or their carers), staff involved in the program (‘Linkers’ and managers), and external linked agencies that worked with the program in various ways. Extensive consultations and surveys were undertaken with these stakeholders over a three month period to allow for a comprehensive analysis of the effectiveness of ALNSWs implementation. A key feature of the program was embodying a ‘culture of learning’. The annual Interim Evaluation Reports similarly provided an ongoing opportunity to review responses and apply the lessons learnt from the evaluation to the implementation of the program along the way.73

The Martu Leadership Program (MLP) and the Developmental Evaluation Methodology:The MLP has been at the forefront of social and economic development in Indigenous Communities in the Pilbara over the past three years. A recent report by Social Ventures Australia revealed the strengths of a developmental evaluation methodology when assessing the outcomes of such programs.74

Focussing on capacity building and governance in the Martu community, the MLP’s establishment of a community Leadership Group originally aimed to enhance individualistic leadership skills so that participants could return to impart knowledge and skills to remote communities and Martu companies.

However, the developmental strategy applied by the facilitating Martu organisation, Kanyirninpa Jukurrpa, has enabled the program to evolve in an organic manner that has had wide ranging and unexpected benefits to the community.

The highly adaptive approach, co-designed with the Martu community, allowed the MLP to evolve its strategies, goals and targets based on developments over time.

The most noticeable benefit of this approach is the evolution of the MLP from an individualistic style leadership training course to the creation of a collective Leadership Group that is actively and independently leading change in the Martu community.

Acting on behalf of all Martu, the Leadership Group now works to enhance the capacity and governance capabilities of Martu society by serving as a cohesive actor that has taken on numerous responsibilities. For example, the Leadership Group now provides a platform for Martu to meet and discuss and resolve sensitive social issues in an organised and open manner. The Group has also facilitated dialogues with external stakeholders such as Newcrest Mining to ensure the best social and economic outcomes for the community.

The evolution of the Leadership Group into an empowered body that is actively and independently promoting the Martu agenda on the national stage is a clear example of the benefits of a developmental evaluation approach. Flexible outcomes and community consultation enabled the MLP to evolve in a manner that best suited Martu interests and ultimately gives them greater control of their own development.

The overwhelmingly positive growth of the Leadership Group could not have occurred if the focus had remained on achieving the fixed outcomes originally listed by the MLP. Set targets and objectives are often the cornerstone when evaluating programs, however the success of the MLP demonstrates the unexpected positives that can arise from a more flexible approach.

Key Features of the Evaluation/Program:The report assesses the importance of viewing Indigenous economic development programs from a more qualitative mindset. It emphasizes the ability of programs to extract unexpected benefits and outcomes by utilising a developmental evaluation approach that enables context based flexibility and adaptability.


Fear of failure can inhibit government from experimenting with different program approaches, but often it is only through this process of trial and error that evidence about what truly works can be collected.75 Genuine adoption of a ‘learning by doing’ approach can be a very accountable process, as evidenced by Malaysia’s National Transformation Program.76 Under the Malaysian government’s Performance Management and Delivery Unit (PEMANDU) a three-staged approach was developed that enabled initial Action Plans to be regularly updated depending on information received from those working on the ground. A distinctive feature of the PEMANDU was the way in which any implementation issues were dealt with by being ‘bumped up’ through a series of ascending steps from an email to the relevant managers, to a closed-door meeting with the Minister (see Table 7).

Table 7 Process for escalation of concerns

Frequency Action Format

Annually Annual report Report published: televised address by PM

Once-to-twice per year ‘Putrajaya Inquisition’ Meeting chaired by PM to clear any issues not solved in lower meetings

Semi-annually PM’s performance review Closed door meeting: only PM, Minister and PEMANDU CEO

Monthly to quarterly Steering Committee meeting Co-chaired by Ministers, with senior officials from all agencies: principal decision making forum

Weekly to fortnightly Meeting of technical working group

Problem solving with relevant managers: principal working session

Weekly Progress report Emailed, uploaded, available on mobile devices.

Source: Sabel and Jordan, 2015

Under this approach, 70% of the initial Action Plans were revised during implementation. However, this did not mean the initial plans were necessarily wrong, as the final plans tended to build on what was in the original Action Plans rather than starting from scratch. 77

At the same time, while these types of participatory research approaches can allow programs to be adapted to suit local conditions, it should also be recognised that increasing community control over program design and implementation will not necessarily produce a ‘perfect’ program.78 According to research conducted by the World Bank, while involving local people can have positive impacts on program outcomes, care is required, as in some instances programs can be controlled by local ‘elites’ and more disadvantaged members of the community can miss out.79


The previous report ‘Mapping the Indigenous Program and Funding Maze’ recommended that all Indigenous programs must be linked to outcomes and that all organisations must:

• formally account for how the money has been spent;

• provide evidence of the program’s impact; and

• assess and report on whether the program is meeting its intended objectives.

This recommendation still stands. However, while large government programs should be subjected to formal evaluation, preferably utilising RCT or quasi-experimental methodology, it would not be an efficient use of taxpayer funding to expect every Indigenous program to be evaluated by external contractors. The NSW Government Evaluation guidelines outline how evaluations should be prioritised based on their “size, strategic significance and degree of risk.”80 This is the correct approach to take, as it is not worthwhile formally evaluating a small program when the cost of the evaluation would outweigh the cost of actually delivering the program. Nor was it our intention in recommending more evaluation to unduly benefit evaluators.

Given that the average cost of an evaluation is $382,000,81 the extra $10 million a year for Indigenous program evaluations will not go far. In fact, it will be possible to formally evaluate only a small proportion of the 1000 or so Indigenous programs the federal government funds. Additional funding to conduct more evaluations is unlikely, given the critical budget situation. The government therefore, needs to move away from traditional evaluation practices involving expensive external evaluators, to approaches that embed evaluation and reflective practice into the delivery of programs.

Our research identified a plethora of small programs (particularly health and well-being programs) currently being delivered by Aboriginal organisations that are not being evaluated. For these small programs, a proper reporting and monitoring framework that allows for reflective practice and continuous quality improvement may be all that is required rather than a formal, independent evaluation (see Evaluation Toolkit in Appendix B for an example of an evaluation framework). At the same time, while it is not economical to evaluate multiple small and disparate programs, it is often community-initiated programs that appear to have the greatest impact.

Unfortunately few evaluations compare community-managed programs with non-Indigenous managed programs to provide evidence on the effectiveness of Indigenous community-led and designed programs.82 Therefore, there exists the paradox that small scale locally-based programs are less likely to be evaluated,

but when they are evaluated they often have the best outcomes.83 Yet, problems can arise when government or NGOs try to scale-up and replicate these types of community-initiated programs. If programs are responsive to the needs of individual communities, any metrics recorded may not be readily compiled or compared with those from other programs.

Other researchers have also struggled to find examples of best practice in Indigenous evaluation and program delivery that could be replicated. Mark Moran author of the book Serious Whitefella Stuff states he spent 12 months looking for a standard of evidence to sort through the complexity of Indigenous program delivery to find what he calls “the best performers and team players.”84 In examining the evidence base he assessed the following methodologies: “Randomised Control Trials; reverse cross-over (quasi-experimental) design; comparative case study analysis; process tracing; Bayesian analysis and fiscal ethnography.” He concluded that too many programs were being implemented for too few people and that as a result it was difficult to find people who had not been “treated” to form a control group.85

However, this does not mean government, or anyone involved in the delivery of Indigenous programs, should not evaluate Indigenous programs. Without some sort of evaluation and accountability measures to track what is happening to the money spent on these programs, it is impossible to know whether the lack of progress in improving outcomes is because there is not enough money relative to need, or whether the funding for Indigenous programs and services is being wasted.

Moreover, it is all very well to say that successful programs involve community involvement and buy-in, but how do you achieve this in communities resistant to change? Implementation science is a term increasingly being used to describe the field of study which examines the individual, organisational and community influences surrounding the implementation process of programs and the gaps between research and practice. Unfortunately rigid funding guidelines often prevent flexibility in implementation timelines and innovation in program design and delivery. People, and by extension programs, are not like an assembly line. Cookie-cutter solutions do not tend to work. So while it is vital that government sets objectives for programs, they should not be overly prescriptive in how those objectives are achieved. Where there are national or state-wide programs, there needs to be a balance between maintaining program fidelity and allowing flexibility for local contexts. In this context, a developmental evaluation approach may be helpful, as the main focus of this type of evaluation is understanding the activities of a program and how the program operates in different environments.

Discussion and conclusion


The overarching recommendation of this report is:

There must be co-accountability for government funded programs

Organisations receiving funding must be held accountable for how they have spent the money and whether the program has achieved its desired outcomes, and the government agency must be held accountable for monitoring whether the organisation is meeting its objectives and working with them to improve their practices if they have not.

This approach is different from traditional ideas of accountability and involves moving away from simply monitoring and overseeing programs to supporting a learning and developmental approach to evaluation.86 A two-way, learning by doing approach to evaluation, with regular feedback loops, will help to ensure both government and program providers keep each other honest.

In recognition of this co-accountability, the following table presents recommendations for both policy makers/program funders and program providers.

Recommendations

Table 8: Recommendations

Policy makers/program funders Program providers

Embedding evaluation into program design and practice

Evaluation should not be viewed as an ‘add on’ but should be built into a program’s design and presented as part of a continuous quality improvement process. Where funding constraints do not allow for an external evaluation, funding should be provided to organisations for self-evaluation.

Evaluation should not be viewed as a negative process but rather as an opportunity to learn. If your organisation does not have the capacity to hire an external evaluator consider hiring a professional evaluator to help with the development of an evaluation framework and for some advice/training in undertaking self-evaluations.

Developing an evidence base

Regular feedback loops with a process for escalating concerns should be part of the data and monitoring process to ensure data being collected is used to inform practice and improve program outcomes. Develop a co-accountability framework and consider providing funding for an online data management system for data collection which will make it easier for program providers to enter and share data.

Documenting how you have achieved the program’s objectives through regular collection and analysis of data is important, not only for providing a stronger evidence base for recurrent funding but also to improve service delivery and ensure client satisfaction with the program. Consider using an online data management system for data collection which will make it easier for staff to enter and share data.

Questions to ask before implementing/delivering a program

•What is the program trying to achieve?

•Is the program needed?

•Is there community support for the program?

•Is there an existing program already addressing a similar need?

•What is different about this program?

•Who will implement the program?

•Do you think the programs objectives meet the needs of the community?

•Do you think the community will support this program?

•What is different about this program?

•What staff will you need to deliver this program?

Questions to ask before evaluating a program

•What is the program trying to achieve AND how will you measure whether it is meeting this objective — are the program’s objectives measurable?

•What type of data are you able to collect to monitor the effectiveness of the program?

•Is there existing data (e.g. administrative data/ABS data) that could be used to measure change/impact?

•How will you collect the data? — what methodology will be used to collect the information — ie surveys/interviews

•Who will collect the data/undertake the evaluation?

•What is the program trying to achieve AND how will you measure whether it is meeting this objective — are the program’s objectives measurable?

•How will you collect the administration data needed to measure the impact of the program? For example, will there be an online database for staff to add data to, or will they be required to enter program data into an Excel spreadsheet? How could you make this process as streamlined as possible?

•How will you show evidence of the program’s impact; e.g. will you undertake pre-admission surveys and post-exit surveys of participants in the program?


Eva

luati

on

of

CB

A P

rog

ram

s

The

tabl

e 9

belo

w a

naly

ses

appr

opriat

enes

s of

the

met

hodo

logy

of

5 re

cent

CBAs

whi

ch s

peci

fical

ly m

easu

re p

rogr

ams

aim

ed a

t he

lpin

g In

dige

nous

peo

ple

in A

ustr

alia

.

Tab

le 9

Eva

luati

on

of

CB

A r

ep

ort

s*

Pro

gra

m

nam

eD

esc

rip

tio

nFi

nd

ing

sM

easu

rab

le

ob

ject

ives

Iden

tifi

cati

on

o

f o

pti

on

sP

rop

er

qu

an

tifi

cati

on

Sen

sib

ilit

y an

aly

sis

Eq

uit

y im

pli

cati

on

sD

isco

un

t ra

teN

PV

calc

ula

tio

ns

Yuen

dum

u M

edia

tion

an

d Ju

stic

e Com

mitte

e

An

Indi

geno

us

desi

gned

con

flict

m

edia

tion

sys

tem

The

proj

ect

retu

rned

ec

onom

ic b

enef

its

that

exce

eded

its

econ

omic

co

sts,

with

a be

nefit

cos

t ra

tio

of 4

.3

Full

list

of a

ssum

ptio

ns

not

prov

ided

No

No

Yes

Yes

Yes

Yes

Yes

— t

he

aggr

egat

e N

PV

was

est

imat

ed a

s $1

4,16

3,00

0 in

20

14 d

olla

rs.

Hea

ling

Foun

dation

Esta

blis

hmen

t of

13

Ind

igen

ous

heal

ing

cent

res

for

spiritua

l hea

ling

The

CBA is

pro

spec

tive

, th

e av

aila

ble

CBA

evid

ence

fro

m s

imila

r in

itia

tive

s in

dica

tes

the

heal

ing

cent

res

coul

d re

turn

, on

ave

rage

, a

bene

fit t

o co

st r

atio

of

over

4.1

Full

list

of a

ssum

ptio

ns

not

prov

ided

No

No

Yes

No

No

No

No

Aus

tral

ian

Indi

geno

us

Men

toring

Ex

perien

ce

(AIM

E)

A p

rogr

am

prov

idin

g ed

ucat

ion,

tra

inin

g an

d em

ploy

men

t fo

r In

dige

nous

Aus

tral

ians

as

wel

l as

pro

tect

ing

and

cons

ervi

ng t

he

envi

ronm

ent

For

each

$1

spen

t, $

7 in

be

nefit

s is

gen

erat

ed f

or

the

econ

omy

Full

list

of a

ssum

ptio

ns

not

prov

ided

Yes

No

Yes

Yes

No

Yes

Yes

— t

he n

et

bene

fit o

f th

e AIM

E pr

ogra

m

in 2

012

was

$38

m

illio

n ba

sed

on a

7%

dis

coun

t ra

te

(ben

efits

are

$58

mill

ion

whi

le c

osts

, in

clud

ing

cost

s of

ed

ucat

ion,

are

$21

m

illio

n).

Ap

pen

dix

A


Pro

gra

m

nam

eD

esc

rip

tio

nFi

nd

ing

sM

easu

rab

le

ob

ject

ives

Iden

tifi

cati

on

o

f o

pti

on

sP

rop

er

qu

an

tifi

cati

on

Sen

sib

ilit

y an

aly

sis

Eq

uit

y im

pli

cati

on

sD

isco

un

t ra

teN

PV

calc

ula

tio

ns

Opa

l Unl

eade

d Pe

trol

Str

ateg

y to

add

ress

pe

trol

sni

ffin

g Bas

e ca

se p

etro

l sni

ffin

g be

nefit

s of

$53.

7 m

illio

n pe

r an

num

an

d ba

se c

ase

Opa

l ro

llout

cos

ts o

f $2

6.6

mill

ion,

prod

ucin

g a

net

gain

of

$27.

1 m

illio

n. F

ull l

ist

of

assu

mpt

ions

pro

vide

d.

No

Yes

Yes

Yes

No

No

Yes

— s

cena

rio

resu

lts

rang

e fr

om

a ne

t ga

in o

f

$4.3

mill

ion

in t

he

wor

st c

ase

to a

ne

t ga

in o

f $5

0.8

mill

ion

in t

he b

est

case

.

Wor

king

on

Cou

ntry

A p

rogr

am

prov

idin

g ed

ucat

ion,

tra

inin

g an

d em

ploy

men

t as

w

ell a

s pr

otec

ting

th

e en

viro

nmen

t

In 2

009-

10 t

he b

udge

t or

boo

k co

st o

f W

orki

ng

on C

ount

ry w

as $

41.2

m

illio

n, h

owev

er t

he t

rue

cost

was

fou

nd t

o be

be

twee

n $3

4 m

illio

n an

d $3

2 m

illio

n.

Full

list

of a

ssum

ptio

ns

not

prov

ided

.

No

No

No

No

No

No

Yes

— t

he t

rue

cost

of th

e pr

ogra

m is

at

estim

ated

to

be a

t le

ast

17 t

o 23

%

low

er t

han

the

book

cos

t.

So

urc

e:

Acc

ess

Econ

omic

s, 2

006,

Alle

n Con

sultin

g G

roup

, 20

11,

Dal

y, A

. an

d Bar

rett

, G

., n

.d., D

eloi

tte

Acc

ess

Econ

omic

s, 2

013,

Del

oitt

e Acc

ess

Econ

omic

s, 2

014,

PW

C,

2015

, KPM

G,

2013

*Som

e of

the

pro

gram

s ha

d an

eva

luat

ion

repo

rt a

nd a

CBA r

epor

t


Eva

luati

on

of

SR

OI

Pro

gra

ms

Tabl

e 10

indi

cate

s th

e m

etho

dolo

gies

use

d in

a S

RO

I re

port

and

ana

lyse

s si

x SRO

I re

port

s th

at m

easu

re the

soc

ial i

mpa

ct o

f di

ffer

ent In

dige

nous

pro

gram

s ai

med

at he

lpin

g In

dige

nous

pe

ople

in A

ustr

alia

. Th

e ta

ble

anal

yses

whi

ch m

etho

dolo

gies

hav

e be

en in

clud

ed a

nd w

hich

hav

e be

en o

mitte

d fr

om e

ach

of t

he d

iffer

ent

repo

rts.

Ove

rall,

as

SRO

I is

still

an e

volv

ing

field

, th

ere

is s

ome

variab

ility

in t

he c

ompr

ehen

sive

ness

of

the

met

hodo

logy

use

d in

the

rep

orts.

Tab

le 1

0:

Eva

luati

on

of

SR

OI

rep

ort

s

Pro

gra

m n

am

eK

ey

featu

res

Fin

din

gs

Measu

rab

le

ob

ject

ives

Qu

an

tifi

cati

on

Sen

sib

ilit

y an

aly

sis

Inp

uts

/O

utp

uts

Imp

act

in

co

nte

xt

Dis

cou

nt

rate

NP

V

calc

ula

tio

ns

Kan

yirn

inpa

Ju

kurr

pa (

KJ)

Ran

ger

prog

ram

Prog

ram

s in

clud

e te

ams

of

rang

er e

mpl

oyee

s, (

Ret

urn

to C

ount

ry)

trip

s an

d cu

ltur

e an

d he

rita

ge p

rogr

ams

The

SRO

I ra

tio

equa

tes

to 3

:1

and

estim

ated

$55

m in

soc

ial

valu

e w

as g

ener

ated

com

pare

d w

ith

the

$20m

inve

stm

ent.

Yes

Yes

Yes

Yes

Yes

Yes

Yes

Hel

ping

Han

d an

d Li

nkin

g Yo

uth

Inte

nsiv

e ca

se m

anag

emen

t to

Abo

rigi

nal y

outh

to

addr

ess

the

unde

rlyi

ng

caus

es o

f of

fend

ing

Fore

cast

SRO

I no

t ac

tual

. Ye

sYe

sYe

sYe

sYe

sN

oYe

s

Gan

bina

Org

aniz

atio

n w

orki

ng w

ith

youn

g Abo

rigi

nal p

eopl

e th

roug

h th

eir

scho

ol y

ears

an

d be

yond

, to

mak

e su

re t

hey

get

the

righ

t ed

ucat

ion,

job

s tr

aini

ng a

nd

expe

rien

ce,

and

life

skill

s

An

inve

stm

ent

of $

1.1m

in t

he

2012

cal

enda

r ye

ar c

reat

ed

$7.5

m o

f so

cial

and

eco

nom

ic

valu

e, a

SRO

I ra

tio

of 6

.7:1

or

$6.7

0 fo

r ev

ery

$1.

Cos

t w

as

appr

ox.

$3,5

00 p

er p

artici

pant

in

201

2.

Yes

Yes

Yes

Yes

Yes

Yes

Yes

Sup

ply

Nat

ion

Cer

tifie

d Sup

plie

rs

Con

nect

s

Indi

geno

us o

wne

d bu

sine

sses

with

oppo

rtun

itie

s in

cor

pora

te

supp

ly c

hain

s.

The

SRO

I w

as e

stim

ated

to

ave

rage

$4.

41 f

or e

very

do

llar

of r

even

ue,

how

ever

qu

estion

able

met

hodo

logy

us

ed.

No

Yes

No

No

No

No

Yes

Nat

iona

l Aus

tral

ia

Ban

k:

Indi

geno

us

Mon

ey M

ento

r

Initia

tive

aim

ed a

t bu

ildin

g th

e fin

anci

al c

apac

ity

of

Indi

geno

us p

eopl

e, t

hrou

gh

face

-to-

face

pra

ctic

al

finan

cial

sup

port

and

acc

ess

to m

icro

finan

ce

From

an

annu

al in

vest

men

t of

$4

48,0

00,

it w

as f

orec

ast

that

$1

.89

mill

ion

will

be

crea

ted

in

soci

al v

alue

in o

ne y

ear.

No

Yes

Yes

Yes

No

No

Yes

NAB/I

ndig

enou

s Tr

aine

es P

rogr

amSch

ool-

base

d tr

aine

eshi

ps,

and

inte

rnsh

ips

for

Indi

geno

us A

ustr

alia

ns t

o pu

rsue

car

eers

in fin

anci

al

serv

ices

Fore

cast

SRO

I ra

tio

for

3 ye

ars

from

201

4. A

SRO

I of

1:2

.71

for

scho

ol-b

ased

tr

aine

eshi

ps a

nd 1

:3.1

4 fo

r fu

lltim

e tr

aine

eshi

ps.

No

Yes

No

No

No

No

No

Sou

rce:

Rav

i, A., 2

013,

Soc

ial V

entu

res

Aus

tral

ia,

2014

a, S

ocia

l Ven

ture

s Aus

tral

ia,

2014

b, S

ocia

l Ven

ture

s Aus

tral

ia,

2014

c, S

ocia

l Ven

ture

s Aus

tral

ia,

2014

d, B

urto

n, R

. an

d To

mki

nson

, E.

n.d

., E

Y, 2

014,

Rav

i, A.

and

Sid

diqi

, T.

, 20

13


Su

mm

ary

of

Eva

luati

on

s/C

ase

-stu

die

s/A

ud

its

Tab

le 1

1 S

um

mary

of

Eva

luati

on

s/ca

se-s

tud

ies/

au

dit

s

Typ

e o

f p

rog

ram

Nam

e a

nd

desc

rip

tio

n o

f p

rog

ram

/fo

rm o

f p

ub

lica

tio

nTyp

e o

f eva

luati

on

/d

esc

rip

tio

n o

f m

eth

od

olo

gy

Key

fin

din

gs

Rati

ng

of

evi

den

ce

1.

Cri

me

Cir

cle S

en

ten

cin

g P

rog

ram

(N

SW

A

tto

rney

Gen

era

l’s

Off

ice)

—

Eval

uation

Rep

ort

(Cul

tura

l & I

ndig

enou

s Res

earc

h Cen

tre

Aus

tral

ia,

May

200

8).

The

Circl

e Sen

tenc

ing

Prog

ram

aim

ed t

o pr

ovid

e an

alter

native

sen

tenc

ing

cour

t fo

r ad

ult

Abo

rigi

nal o

ffen

ders

tha

t m

ore

dire

ctly

invo

lved

Abo

rigi

nal p

eopl

e in

the

se

nten

cing

pro

cess

.

The

eval

uation

met

hodo

logy

prim

arily

ad

opte

d a

qual

itat

ive

appr

oach

, in

volv

ing

inte

rvie

ws

and

grou

p di

scus

sion

s w

ith

115

peop

le a

nd a

lite

ratu

re r

evie

w

of A

ustr

alia

n tr

ends

with

rega

rd t

o In

dige

nous

sen

tenc

ing

prog

ram

s an

d ex

isting

eva

luat

ions

. T

here

was

how

ever

so

me

anal

ysis

of

exis

ting

dat

a on

Circl

e Sen

tenc

ing

offe

nder

s an

d Lo

cal C

ourt

da

ta.

The

eval

uation

fou

nd t

he p

rogr

am

met

mos

t of

its

obje

ctiv

es,

and

that

th

e “g

ener

al/s

take

hold

er c

onse

nsus

” w

as t

hat

the

prog

ram

was

‘cul

tura

lly

appr

opriat

e’ a

nd a

suc

cess

. H

owev

er,

the

repo

rt a

lso

stat

es t

hat

data

fro

m

the

NSW

Bur

eau

of C

rim

e an

d Sta

tist

ics

show

s th

at t

he p

rogr

am d

id n

ot h

ave

an

impa

ct o

n ra

tes

of r

ecid

ivis

m,

whi

ch w

as

one

of t

he m

ain

goal

s. T

here

was

als

o a

lack

of

quan

tita

tive

dat

a, w

hich

was

at

trib

uted

to

the

prog

ram

tak

ing

plac

e in

diff

eren

t pl

aces

with

differ

ent

data

co

llect

ion

met

hods

.

Furt

herm

ore,

in e

valu

atin

g re

-offen

ding

, it w

as f

ound

tha

t th

e Circl

e gr

oups

wer

e m

ore

likel

y to

reo

ffen

d th

an t

he c

ontr

ol

grou

p.

Mo

dera

te

Altho

ugh

the

qual

itat

ive

data

was

of hi

gh

qual

ity,

the

re w

as a

lack

of co

nsis

tent

qu

antita

tive

dat

a.

Dat

a on

vic

tim

par

tici

pation

was

not

co

llect

ed c

onsi

sten

tly.

2.

Cri

me

Qu

een

slan

d I

nd

igen

ou

s A

lco

ho

l D

ivers

ion

Pro

gra

m (

QIA

DP

) —

a

reci

divi

sm s

tudy

(Spe

cial

ist

Cou

rts

and

Div

ersi

on,

Lega

l Ser

vice

s Bra

nch,

N

ovem

ber

2010

).

QIA

DP

was

a v

olun

tary

tre

atm

ent

prog

ram

for

Abo

rigi

nal p

eopl

e w

ho h

ave

been

arr

este

d an

d ap

pear

ed in

cou

rt f

or

alco

hol r

elat

ed o

ffen

ces.

Qua

ntitat

ive

data

was

col

lect

ed f

rom

lo

cal p

olic

e so

urce

s.

Lim

itat

ions

list

ed in

clud

ed:

smal

l sam

ple

size

s an

d a

lack

of

cont

rol g

roup

s.

Rat

es o

f re

cidi

vism

impr

oved

whi

le

offe

nder

s w

ere

part

icip

atin

g in

the

pr

ogra

m.

How

ever

, fin

ding

s al

so s

how

ed

that

the

re w

as n

o dr

op in

the

sev

erity

of r

eoff

endi

ng.

The

mix

ed r

esul

ts

rais

ed q

uest

ions

abo

ut t

he s

ucce

ss o

f th

e pr

ogra

m.

Find

ings

wer

e de

emed

in

conc

lusi

ve.

Mo

dera

te

Des

pite

the

lack

of so

me

quan

tita

tive

da

ta t

he d

ata

that

was

use

d in

the

re

port

was

ana

lyse

d th

orou

ghly

in

num

erou

s w

ays.

Ana

lysi

s in

clud

ed

differ

ent

type

s of

offen

ces

and

type

s of

cr

imes

com

mitte

d as

wel

l as

differ

ent

tim

e fr

ames

. In

con

clus

ion,

the

han

dlin

g of

the

dat

a w

as q

uite

str

ong,

but

due

to

the

sm

all s

ampl

e si

zes

and

lack

of

cont

rol g

roup

s, t

he r

atin

g ca

n on

ly b

e ju

dged

as

‘mod

erat

e’.


Typ

e o

f p

rog

ram

Nam

e a

nd

desc

rip

tio

n o

f p

rog

ram

/fo

rm o

f p

ub

lica

tio

nTyp

e o

f eva

luati

on

/d

esc

rip

tio

n o

f m

eth

od

olo

gy

Key

fin

din

gs

Rati

ng

of

evi

den

ce

3.

Cri

me

Th

e T

iwi Is

lan

ds

Yo

uth

Deve

lop

men

t an

d D

ivers

ion

Un

it (

TIY

DD

U)

(NT)

base

d in

Wu

rru

miy

an

ga —

an

eval

uation

(Ste

war

t et

al 2

014;

CTG

CH

20

14).

The

prog

ram

sta

rted

in 2

003

and

cons

ists

of a

12-w

eek

dive

rsio

n pr

ogra

m

for

first

-tim

e yo

uth

offe

nder

s fr

om t

he

Tiw

i Isl

ands

.

Eval

uation

invo

lved

qua

litat

ive

and

quan

tita

tive

dat

a an

alys

is in

clud

ing

inte

rvie

ws,

doc

umen

t an

alys

is,

obse

rvat

ions

and

ana

lysi

s of

ad

min

istr

ativ

e da

ta.

The

eval

uation

fou

nd t

he p

rogr

am

was

eff

ective

in r

educ

ing

adve

rse

cont

act

betw

een

Tiw

i you

th a

nd t

he

crim

inal

jus

tice

sys

tem

. In

divi

dual

re

-off

ence

dat

a fr

om N

T Po

lice

for

prog

ram

par

tici

pant

s sh

owed

tha

t 20

% o

f pa

rtic

ipan

ts (

13 o

f 65

you

ng

peop

le)

had

cont

act

with

the

polic

e fo

r al

lege

d of

fenc

es in

the

yea

r fo

llow

ing

com

men

cem

ent

with

the

prog

ram

—

belo

w w

hat

wou

ld b

e ex

pect

ed for

thi

s po

pula

tion

witho

ut t

he in

terv

ention

.

Mo

dera

te

Dat

a w

as n

ot a

vaila

ble

for

com

pariso

ns

with

yout

h th

at d

id n

ot p

artici

pate

in t

he

prog

ram

dur

ing

this

tim

e, a

nd fur

ther

as

sess

men

t of

the

out

com

es fro

m t

his

prog

ram

by

com

paring

with

a si

mila

r gr

oup

who

did

not

par

tici

pate

wou

ld b

e de

sira

ble.

Sch

ool a

tten

danc

e da

ta for

in

divi

dual

s w

ere

requ

este

d bu

t un

able

to

be

prov

ided

due

to

conf

iden

tial

ity

and

smal

l num

bers

4.

Cri

me

Yu

en

du

mu

Med

iati

on

an

d J

ust

ice

Co

mm

itte

e p

rog

ram

— C

BA r

epor

t (D

aly

and

Bar

rett

, n.

d).

The

Yuen

dum

u M

edia

tion

and

Jus

tice

Com

mitte

e pr

ogra

m h

as b

een

oper

atin

g si

nce

2011

and

dra

ws

upon

tr

aditio

nal W

arlp

iri d

ispu

te r

esol

utio

n an

d re

lation

ship

-sus

tain

ing

prac

tice

s to

str

engt

hen

fam

ily r

elat

ions

hips

an

d de

velo

p st

rate

gies

tha

t pr

omot

e co

mm

unity

safe

ty a

nd a

ddre

ss fam

ily

viol

ence

.

The

eval

uation

met

hodo

logy

use

d a

desk

top

revi

ew,

inte

rvie

ws

with

key

stak

ehol

ders

and

key

doc

umen

ts t

o id

entify

and

val

ue t

he e

cono

mic

cos

ts

and

bene

fits

of t

he Y

uend

umu

Med

iation

an

d

Just

ice

Com

mitte

e us

ing

an a

pplic

atio

n of

Cos

t Ben

efit A

naly

sis.

The

proj

ect

was

fou

nd t

o ha

ve r

educ

ed

and

prev

ente

d vi

olen

ce b

y en

suring

co

nflic

ts a

re s

ettled

pea

cefu

lly a

nd

quic

kly.

The

Net

Pre

sent

Val

ue in

201

4 w

as $

14.1

mill

ion

with

a be

nefit

-cos

t ra

tio

of 4

.3.

N/

A

5.

Cri

me

Mara

pai N

gart

ath

ati

Mu

rri W

om

en

s G

rou

p a

nd

Yu

rru

Ng

art

ath

ati

Men

s G

rou

p I

nd

igen

ou

s ju

stic

e p

rog

ram

—

eva

luat

ion

sum

mar

y (J

ames

Coo

k U

nive

rsity,

Nov

embe

r 20

13).

The

dive

rsio

nary

reh

abili

tative

bai

l pr

ogra

m t

arge

ts I

ndig

enou

s of

fend

ing

in

Mou

nt I

sa,

Que

ensl

and.

The

eval

uation

invo

lved

tal

king

to

peop

le

who

had

tak

en p

art

in t

he m

en’s

and

w

omen

’s g

roup

s an

d ot

her

peop

le in

the

co

mm

unity

invo

lved

with

the

prog

ram

. Th

e re

port

con

sist

ed o

f th

ree

page

s pr

ovid

ing

a su

mm

ary

of t

he e

valu

atio

n fin

ding

s.

The

findi

ngs

of t

he e

valu

atio

n br

iefly

ou

tlin

ed w

hat

was

wor

king

wel

l with

the

prog

ram

, th

e im

pact

of th

e pr

ogra

m o

n pe

ople

and

wha

t ne

eded

to

impr

ove

with

the

prog

ram

Weak

Onl

y th

e ev

alua

tion

sum

mar

y w

as a

vaila

ble,

so

diffic

ult

to d

raw

co

nclu

sion

s. A

ltho

ugh

peop

le r

epor

ted

posi

tive

ben

efits

from

tak

ing

part

in t

he

prog

ram

the

re w

as v

ery

little

evid

ence

of

the

effe

ctiv

enes

s of

the

pro

gram

bey

ond

anec

dota

l evi

denc

e.


Typ

e o

f p

rog

ram

Nam

e a

nd

desc

rip

tio

n o

f p

rog

ram

/fo

rm o

f p

ub

lica

tio

nTyp

e o

f eva

luati

on

/d

esc

rip

tio

n o

f m

eth

od

olo

gy

Key

fin

din

gs

Rati

ng

of

evi

den

ce

6.

Cri

me

On

Th

e R

oad

Lis

mo

re d

rive

r ed

uca

tio

n —

eva

luat

ion

repo

rt (

Lism

ore

Adu

lt C

omm

unity

Educ

atio

n (A

CE)

).

On

the

Roa

d is

a c

ompr

ehen

sive

dr

iver

edu

cation

pro

gram

tha

t ta

rget

s Abo

rigi

nal p

eopl

e liv

ing

in t

he F

ar

Nor

th C

oast

of N

SW

. Th

e ov

eral

l goa

l of

the

pro

gram

is t

o re

duce

the

ove

r-re

pres

enta

tion

of Abo

rigi

nal p

eopl

e liv

ing

in t

he F

ar N

orth

Coa

st o

f N

SW

in t

he

crim

inal

jus

tice

sys

tem

.

Lite

ratu

re r

evie

w.

Qua

ntitat

ive:

sta

tist

ics

from

the

ACE

enro

lmen

t an

d BO

CSAR d

atab

ases

.

Qua

litat

ive:

18

face

-to-

face

inte

rvie

ws,

5

phon

e in

terv

iew

s an

d 5

focu

s gr

oups

w

ith

55 p

artici

pant

s.

No

cont

rol g

roup

.

The

repo

rt s

tate

s th

at t

he p

rogr

am h

as

been

suc

cess

ful i

n ac

cess

ing

its

targ

et

audi

ence

and

incr

easi

ng a

war

enes

s of

its

avai

labl

e re

sour

ces.

The

repo

rt a

lso

stat

es t

hat

ther

e ha

s be

en a

dec

line

in t

he n

umbe

r of

drivi

ng

offe

nces

for

the

tar

gete

d Abo

rigi

nal

popu

lation

, ho

wev

er,

witho

ut a

n es

tabl

ishe

d co

ntro

l gro

up,

this

can

not

be

attr

ibut

ed t

o th

e pr

ogra

m.

Furt

herm

ore,

th

e la

ck o

f ro

utin

ely

colle

cted

dat

a in

the

are

a (l

ack

of id

entific

atio

n of

Abo

rigi

nalit

y in

RTA

roa

d cr

ashe

s) m

akes

it im

poss

ible

to

link

impr

ovem

ents

to

the

prog

ram

.

Weak

The

stru

ctur

e of

the

rep

ort

is g

ood

and

ther

e w

ere

clea

r ob

ject

ives

. H

owev

er,

the

repo

rt is

onl

y ab

le t

o co

nfirm

tha

t aw

aren

ess

in t

he r

egio

n ha

d be

en

impr

oved

on.

The

cor

e ob

ject

ives

(r

educ

ing

the

num

ber

of d

rivi

ng o

ffen

ces

and

indi

vidu

als

com

ing

into

con

tact

w

ith

the

crim

inal

sys

tem

) co

uld

not

be

adeq

uate

ly a

ccou

nted

for

.

7.

Cu

ltu

reV

ibe A

ust

ralia s

uit

e o

f p

rod

uct

s —

ev

alua

tion

(KPM

G,

2013

)

Vib

e Aus

tral

ia is

a N

FP o

rgan

isat

ion

whi

ch a

ims

to im

prov

e th

e w

ell-

bein

g of

Ind

igen

ous

peop

le t

hrou

gh n

atio

nal

com

mun

icat

ions

, m

edia

and

eve

nts

serv

ices

.

Eval

uation

met

hodo

logy

con

sist

ed o

f:

desk

top

rese

arch

; a

read

er s

urve

y to

re

ader

s of

Vib

e’s

Dea

dly

Vib

e m

agaz

ine

(3,5

00 w

ere

dist

ribu

ted)

; an

onl

ine

surv

ey;

part

icip

ant

surv

ey;

focu

s gr

oups

an

d te

leph

one

inte

rvie

ws.

The

maj

ority

of fun

ding

rec

eive

d by

Vib

e (8

9%)

is e

xpen

ded

on p

rogr

am

deliv

ery

rath

er t

han

adm

inis

trat

ive

fund

ing.

Vib

e al

so g

ener

ated

add

itio

nal

inco

me

from

cor

pora

te e

ntitie

s an

d as

a

resu

lt V

ibe

was

abl

e to

mat

ch t

he

fede

ral g

over

nmen

t’s c

ontr

ibut

ion

of

$2.3

mill

ion

to g

ener

ate

$4.6

for

its

suite

of p

rodu

cts.

Ove

rall,

whi

le t

here

was

so

me

variab

ility

bet

wee

n th

e pr

oduc

ts

and

inte

grat

ion

betw

een

prod

ucts

cou

ld

be im

prov

ed,

the

eval

uation

fou

nd t

hey

prov

ide

valu

e fo

r m

oney

and

a u

niqu

e se

rvic

e.

Mo

dera

te

The

met

hodo

logy

was

com

preh

ensi

ve

with

a ra

nge

of q

ualit

ativ

e an

d qu

antita

tive

met

hods

utilis

ed.

How

ever

, th

e nu

mbe

r of

peo

ple

who

res

pond

ed

to t

he s

urve

ys a

nd p

artici

pate

d in

the

in

terv

iew

s w

as n

ot p

rovi

ded.


Typ

e o

f p

rog

ram

Nam

e a

nd

desc

rip

tio

n o

f p

rog

ram

/fo

rm o

f p

ub

lica

tio

nTyp

e o

f eva

luati

on

/d

esc

rip

tio

n o

f m

eth

od

olo

gy

Key

fin

din

gs

Rati

ng

of

evi

den

ce

8.

Cu

ltu

reTh

e Y

irim

an

Pro

ject

(W

A)

—

(Pal

mer

, D

, 20

13)

The

Proj

ect

is a

cul

tura

l im

mer

sion

pr

ogra

m t

hat

cons

ists

of tr

ips

‘on

coun

try’

, w

here

eld

ers

teac

h yo

ung

peop

le a

t risk

of of

fend

ing

abou

t th

eir

cultur

al h

eritag

e.

Cas

e st

udy/

qual

itat

ive

stud

y.

Met

hodo

logy

con

sist

ed o

f:

• A li

tera

ture

rev

iew

incl

udin

g an

ap

prai

sal o

f m

edia

rev

iew

s an

d ar

ticl

es.

• W

ritt

en a

nd v

erba

l fee

dbac

k fr

om

com

mun

ity

mem

bers

• D

irec

t pa

rtic

ipat

ion

and

obse

rvat

ion

from

fie

ld t

rips

, in

clud

ing

exte

nded

vi

sits

of

two

mon

ths

during

200

9 an

d 20

10.

• In

terv

iew

s w

ith

staf

f.

The

qual

itat

ive

stud

y fo

und

that

th

e pr

ogra

m b

uild

s yo

ung

peop

le’s

co

nfid

ence

and

impr

oves

the

ir s

elf-

wor

th,

and

is c

onsi

dere

d to

hav

e he

lped

cu

rb s

uici

de,

self-

harm

and

sub

stan

ce

abus

e in

the

par

tici

pating

com

mun

itie

s.

N/

A

9.

Cu

ltu

reIn

dig

en

ou

s h

ip h

op

pro

ject

s —

Ev

alua

tion

rep

ort

of I

ndig

enou

s H

ip H

op

Proj

ects

(Kur

ongk

url K

atitjin

, Cen

tre

for

Indi

geno

us A

ustr

alia

n Ed

ucat

ion

and

Res

earc

h Fa

culty

of E

duca

tion

and

Art

s,

Edith

Cow

an U

nive

rsity

2009

).

The

IHH

P is

a p

artn

ersh

ip w

ith

Bey

ond

Blu

e to

pro

vide

wor

ksho

ps t

hat

com

bine

In

dige

nous

cul

ture

and

hip

hop

to

prom

ote

conf

iden

ce a

nd p

ositiv

e se

lf-ex

pres

sion

.

Impa

ct e

valu

atio

n of

Ind

igen

ous

Hip

H

op P

roje

cts

(IH

HP)

on

youn

g pe

ople

in

sel

ecte

d si

tes

in t

he K

imbe

rley

and

Pi

lbar

a re

gion

s of

WA.

The

eval

uation

was

con

duct

ed o

ver

thre

e st

ages

, us

ing

a co

mbi

nation

of

qua

litat

ive

met

hods

incl

udin

g qu

estion

naires

; on

e-on

-one

inte

rvie

ws

and

focu

s gr

oups

.

The

eval

uation

fou

nd t

hat

youn

g pe

ople

ap

pear

ed t

o re

spon

d w

ell t

o th

e he

alth

pr

omot

ion

mes

sage

s of

IH

HP.

The

re w

as

som

e re

call

of t

he m

essa

ges

rela

ting

to

dep

ress

ion

and

self-

resp

ect,

as

wel

l as

the

key

mes

sage

s of

look

; lis

ten;

ta

lk;

and

seek

hel

p. Y

oung

peo

ple

also

exp

ress

ed “

feel

ing

good

abo

ut

them

selv

es”

as a

res

ult

of s

ome

of t

he

IHH

P ac

tivi

ties

. H

owev

er,

the

eval

uation

re

com

men

ded

that

Bey

ond

Blu

e sh

ould

co

nsid

er p

rogr

ams

com

plim

enta

ry t

o IH

HP

that

spe

cific

ally

tar

get

educ

atio

n ab

out

depr

essi

on a

nd a

nxie

ty.

Weak

The

eva

luat

ion

soug

ht t

o us

e a

rang

e of

qua

litat

ive

met

hods

but

was

let

dow

n by

a la

ck o

f qu

antita

tive

dat

a an

d an

ov

erre

lianc

e on

ane

cdot

al e

vide

nce.

10

. C

ult

ure

Ou

r M

en

Ou

r H

ealin

g:

Cre

ati

ng

h

op

e,

resp

ect

an

d r

eco

nn

ect

ion

ex

ecut

ive

sum

mar

y of

eva

luat

ion

(Hea

ling

Foun

dation

, 20

15).

Our

Men

Our

Hea

ling

are

thre

e pi

lot

men

’s h

ealin

g pr

ojec

ts in

the

rem

ote

Nor

ther

n Te

rritor

y co

mm

unitie

s of

M

anin

grid

a, N

guku

rr a

nd W

urru

miy

anga

.

Eval

uation

foc

uses

on

impl

emen

tation

an

d ea

rly

deve

lopm

ent

of t

he p

rogr

ams.

The

exec

utiv

e su

mm

ary

did

not

desc

ribe

th

e ty

pe o

f m

etho

dolo

gy u

sed

in t

he

eval

uation

.

Key

pro

gram

ach

ieve

men

ts in

clud

e a

repo

rted

dec

reas

e in

the

inci

denc

e of

fa

mily

and

dom

estic

viol

ence

and

less

vi

olen

ce g

ener

ally

in c

omm

unitie

s. I

n on

e co

mm

unity

ther

e w

as a

rep

orte

d 50

% r

educ

tion

in t

he n

umbe

r of

men

re

gist

ered

with

the

NT

Dep

artm

ent

of

Cor

rect

iona

l Ser

vice

s an

d a

sign

ifica

nt

redu

ctio

n in

rat

es o

f re

cidi

vism

and

re

offe

ndin

g, a

s w

ell a

s in

crea

sed

leve

ls

of c

ultu

ral p

ract

ice.

Weak

Met

hodo

logy

not

des

crib

ed a

nd o

nly

exec

utiv

e su

mm

ary

avai

labl

e. P

rogr

am

achi

evem

ents

wer

e ve

ry p

ositiv

e bu

t m

ore

evid

ence

nee

ded

to s

how

cau

salit

y.

No

cont

rol g

roup

.


Typ

e o

f p

rog

ram

Nam

e a

nd

desc

rip

tio

n o

f p

rog

ram

/fo

rm o

f p

ub

lica

tio

nTyp

e o

f eva

luati

on

/d

esc

rip

tio

n o

f m

eth

od

olo

gy

Key

fin

din

gs

Rati

ng

of

evi

den

ce

8.

Cu

ltu

reTh

e Y

irim

an

Pro

ject

(W

A)

—

(Pal

mer

, D

, 20

13)

The

Proj

ect

is a

cul

tura

l im

mer

sion

pr

ogra

m t

hat

cons

ists

of tr

ips

‘on

coun

try’

, w

here

eld

ers

teac

h yo

ung

peop

le a

t risk

of of

fend

ing

abou

t th

eir

cultur

al h

eritag

e.

Cas

e st

udy/

qual

itat

ive

stud

y.

Met

hodo

logy

con

sist

ed o

f:

• A li

tera

ture

rev

iew

incl

udin

g an

ap

prai

sal o

f m

edia

rev

iew

s an

d ar

ticl

es.

• W

ritt

en a

nd v

erba

l fee

dbac

k fr

om

com

mun

ity

mem

bers

• D

irec

t pa

rtic

ipat

ion

and

obse

rvat

ion

from

fie

ld t

rips

, in

clud

ing

exte

nded

vi

sits

of

two

mon

ths

during

200

9 an

d 20

10.

• In

terv

iew

s w

ith

staf

f.

The

qual

itat

ive

stud

y fo

und

that

th

e pr

ogra

m b

uild

s yo

ung

peop

le’s

co

nfid

ence

and

impr

oves

the

ir s

elf-

wor

th,

and

is c

onsi

dere

d to

hav

e he

lped

cu

rb s

uici

de,

self-

harm

and

sub

stan

ce

abus

e in

the

par

tici

pating

com

mun

itie

s.

N/

A

9.

Cu

ltu

reIn

dig

en

ou

s h

ip h

op

pro

ject

s —

Ev

alua

tion

rep

ort

of I

ndig

enou

s H

ip H

op

Proj

ects

(Kur

ongk

url K

atitjin

, Cen

tre

for

Indi

geno

us A

ustr

alia

n Ed

ucat

ion

and

Res

earc

h Fa

culty

of E

duca

tion

and

Art

s,

Edith

Cow

an U

nive

rsity

2009

).

The

IHH

P is

a p

artn

ersh

ip w

ith

Bey

ond

Blu

e to

pro

vide

wor

ksho

ps t

hat

com

bine

In

dige

nous

cul

ture

and

hip

hop

to

prom

ote

conf

iden

ce a

nd p

ositiv

e se

lf-ex

pres

sion

.

Impa

ct e

valu

atio

n of

Ind

igen

ous

Hip

H

op P

roje

cts

(IH

HP)

on

youn

g pe

ople

in

sel

ecte

d si

tes

in t

he K

imbe

rley

and

Pi

lbar

a re

gion

s of

WA.

The

eval

uation

was

con

duct

ed o

ver

thre

e st

ages

, us

ing

a co

mbi

nation

of

qua

litat

ive

met

hods

incl

udin

g qu

estion

naires

; on

e-on

-one

inte

rvie

ws

and

focu

s gr

oups

.

The

eval

uation

fou

nd t

hat

youn

g pe

ople

ap

pear

ed t

o re

spon

d w

ell t

o th

e he

alth

pr

omot

ion

mes

sage

s of

IH

HP.

The

re w

as

som

e re

call

of t

he m

essa

ges

rela

ting

to

dep

ress

ion

and

self-

resp

ect,

as

wel

l as

the

key

mes

sage

s of

look

; lis

ten;

ta

lk;

and

seek

hel

p. Y

oung

peo

ple

also

exp

ress

ed “

feel

ing

good

abo

ut

them

selv

es”

as a

res

ult

of s

ome

of t

he

IHH

P ac

tivi

ties

. H

owev

er,

the

eval

uation

re

com

men

ded

that

Bey

ond

Blu

e sh

ould

co

nsid

er p

rogr

ams

com

plim

enta

ry t

o IH

HP

that

spe

cific

ally

tar

get

educ

atio

n ab

out

depr

essi

on a

nd a

nxie

ty.

Weak

The

eva

luat

ion

soug

ht t

o us

e a

rang

e of

qua

litat

ive

met

hods

but

was

let

dow

n by

a la

ck o

f qu

antita

tive

dat

a an

d an

ov

erre

lianc

e on

ane

cdot

al e

vide

nce.

10

. C

ult

ure

Ou

r M

en

Ou

r H

ealin

g:

Cre

ati

ng

h

op

e,

resp

ect

an

d r

eco

nn

ect

ion

ex

ecut

ive

sum

mar

y of

eva

luat

ion

(Hea

ling

Foun

dation

, 20

15).

Our

Men

Our

Hea

ling

are

thre

e pi

lot

men

’s h

ealin

g pr

ojec

ts in

the

rem

ote

Nor

ther

n Te

rritor

y co

mm

unitie

s of

M

anin

grid

a, N

guku

rr a

nd W

urru

miy

anga

.

Eval

uation

foc

uses

on

impl

emen

tation

an

d ea

rly

deve

lopm

ent

of t

he p

rogr

ams.

The

exec

utiv

e su

mm

ary

did

not

desc

ribe

th

e ty

pe o

f m

etho

dolo

gy u

sed

in t

he

eval

uation

.

Key

pro

gram

ach

ieve

men

ts in

clud

e a

repo

rted

dec

reas

e in

the

inci

denc

e of

fa

mily

and

dom

estic

viol

ence

and

less

vi

olen

ce g

ener

ally

in c

omm

unitie

s. I

n on

e co

mm

unity

ther

e w

as a

rep

orte

d 50

% r

educ

tion

in t

he n

umbe

r of

men

re

gist

ered

with

the

NT

Dep

artm

ent

of

Cor

rect

iona

l Ser

vice

s an

d a

sign

ifica

nt

redu

ctio

n in

rat

es o

f re

cidi

vism

and

re

offe

ndin

g, a

s w

ell a

s in

crea

sed

leve

ls

of c

ultu

ral p

ract

ice.

Weak

Met

hodo

logy

not

des

crib

ed a

nd o

nly

exec

utiv

e su

mm

ary

avai

labl

e. P

rogr

am

achi

evem

ents

wer

e ve

ry p

ositiv

e bu

t m

ore

evid

ence

nee

ded

to s

how

cau

salit

y.

No

cont

rol g

roup

.

Typ

e o

f p

rog

ram

Nam

e a

nd

desc

rip

tio

n o

f p

rog

ram

/fo

rm o

f p

ub

lica

tio

nTyp

e o

f eva

luati

on

/d

esc

rip

tio

n o

f m

eth

od

olo

gy

Key

fin

din

gs

Rati

ng

of

evi

den

ce

11

. C

ult

ure

A

rmy

Ab

ori

gin

al co

mm

un

ity

ass

ista

nce

pro

gra

m —

aud

it

(Aus

tral

ian

Nat

iona

l Aud

it O

ffic

e (A

NAO

),

2010

)

The

Arm

y Abo

rigi

nal C

omm

unity

Ass

ista

nce

Prog

ram

(AACAP)

aim

s to

de

velo

p an

d up

grad

e en

viro

nmen

tal

heal

th in

fras

truc

ture

in r

emot

e Abo

rigi

nal a

nd T

orre

s Str

ait

Isla

nder

co

mm

unitie

s.

Aud

it —

the

aud

it a

sses

sed

FaH

CSIA

’s

man

agem

ent

of A

ACAP

and

how

th

e D

epar

tmen

t m

onitor

s th

e co

ntribu

tion

the

pro

gram

is m

akin

g to

the

impr

ovem

ent

of p

rim

ary

and

envi

ronm

enta

l hea

lth,

and

livi

ng

cond

itio

ns,

in r

emot

e In

dige

nous

co

mm

unitie

s.

Sin

ce 1

997,

the

pro

gram

has

im

plem

ente

d pr

ojec

ts in

20 d

iscr

ete

loca

tion

s co

vering

35

com

mun

itie

s in

the

Nor

ther

n Te

rritor

y,

Wes

tern

Aus

tral

ia,

Sou

th A

ustr

alia

and

Q

ueen

slan

d. O

vera

ll th

e au

dit

foun

d th

e pr

ogra

m r

epre

sent

ed g

ood

valu

e fo

r m

oney

and

tha

t th

e pr

ogra

m w

as

cons

iste

nt w

ith

CO

AG

’s o

bjec

tive

s.

N/

A

12

. Ed

uca

tio

nA

nim

al M

an

ag

em

en

t in

Ru

ral an

d

Rem

ote

In

dig

en

ou

s C

om

mu

nit

ies

Inc.

(A

MR

ICC

) —

An

imal

Man

ag

em

en

t W

ork

er

Pro

gra

m —

ev

alua

tion

rep

ort

(Reg

ina

Hill

, Reg

ina

Hill

Effec

tive

Con

sultin

g Pt

y Lt

d, A

ugus

t 20

14).

The

proj

ect

aim

ed t

o sh

ift r

espo

nsib

ility

an

d ca

pabi

lity

back

into

the

han

ds o

f th

e lo

cal c

omm

unity

by e

mpl

oyin

g Abo

rigi

nal

anim

al w

orke

rs (

AM

Ws)

to

help

impr

ove

com

pani

on a

nim

al h

ealth

and

cont

rol i

n th

ose

regi

ons.

The

type

of

data

was

alm

ost

entire

ly

quan

tita

tive

, co

mpr

isin

g t

able

s an

d fig

ures

sho

win

g th

e nu

mbe

r of

job

s/po

sition

s cr

eate

d, n

umbe

r of

ani

mal

s tr

eate

d, h

ow lo

ng e

ach

pers

on h

ad

reta

ined

the

ir p

ositio

n/em

ploy

men

t.

The

repo

rt s

ugge

sts

som

e im

prov

emen

t in

ski

ll de

velo

pmen

t, a

ltho

ugh

long

-ter

m

empl

oym

ent

oppo

rtun

itie

s/oc

curr

ence

s ar

e la

ckin

g, m

ainl

y du

e to

per

sona

l is

sues

(as

rep

orte

d by

the

par

tici

pant

s).

It w

as n

oted

tha

t th

ere

wer

e of

ten

prob

lem

s w

ith

too

little

wor

k to

act

ually

ke

ep p

artici

pant

s bu

sy d

urin

g th

e w

eek.

Mo

dera

te

The

data

cap

ture

d se

ems

appr

opriat

e an

d ac

cura

te a

nd w

as r

epor

ted

on a

co

ntin

uous

bas

is.

13

. Ed

uca

tio

nA

IME O

utr

each

Pro

gra

m —

ev

alua

tion

(KPM

G,

2015

).

The

AIM

E (A

ustr

alia

n In

dige

nous

M

ento

ring

Exp

erie

nce)

Pro

gram

was

es

tabl

ishe

d in

200

5. T

he g

oals

of th

e pr

ogra

m a

re t

o im

prov

e re

tent

ion

rate

s of

Abo

rigi

nal a

nd T

orre

s Str

ait

Isla

nder

hi

gh s

choo

l stu

dent

s to

yea

r 12

and

po

st s

choo

l and

to

conn

ect

Abo

rigi

nal

and

Torr

es S

trai

t Is

land

er s

tude

nts

to

univ

ersi

ty a

nd e

mpl

oym

ent.

An

inde

pend

ent

eval

uation

was

un

dert

aken

in 2

012

to e

valu

ate

the

AIM

E O

utre

ach

Prog

ram

, in

com

pariso

n to

the

Cor

e Pr

ogra

m.

The

eval

uation

incl

uded

a

mix

ed-m

etho

d de

sign

inco

rpor

atin

g:

obse

rvat

ion

of p

rogr

am d

eliv

ery;

in

terv

iew

s w

ith

prog

ram

fac

ilita

tors

, m

ento

rs a

nd m

ente

es;

revi

ew o

f AIM

E do

cum

enta

tion

and

a q

uant

itat

ive

surv

ey

of m

ente

es.

The

eval

uation

fou

nd t

hat

AIM

E an

d AO

P ar

e ac

hiev

ing

posi

tive

res

ults

. In

its

first

ye

ar o

f op

erat

ion,

the

AO

P re

ache

d its

obje

ctiv

e of

enc

oura

ging

bet

ter

scho

ol

grad

e pr

ogre

ssio

n ra

tes

for

Abo

rigi

nal

and

Torr

es S

trai

t Is

land

er s

tude

nts,

co

mpa

red

with

the

nation

al a

vera

ge.

In

2015

, AIM

E co

nnec

ted

appr

oxim

atel

y 57

00 h

igh

scho

ol s

tude

nts

with

1900

vo

lunt

eer

univ

ersi

ty s

tude

nts

acro

ss 1

8 Aus

tral

ian

univ

ersi

ties

, in

all

mai

nlan

d st

ates

and

the

ACT.

Mo

dera

te

A m

ixed

-met

hod

eval

uation

des

ign

utili

sed

a ra

nge

of d

ata

sour

ces

to

enab

le fin

ding

s to

be

cros

s-ch

ecke

d an

d ve

rifie

d w

ith

othe

r da

ta.


Typ

e o

f p

rog

ram

Nam

e a

nd

desc

rip

tio

n o

f p

rog

ram

/fo

rm o

f p

ub

lica

tio

nTyp

e o

f eva

luati

on

/d

esc

rip

tio

n o

f m

eth

od

olo

gy

Key

fin

din

gs

Rati

ng

of

evi

den

ce

14

. Ed

uca

tio

nLe

t’s

Sta

rt:

Exp

lori

ng

To

geth

er

—

final

eva

luat

ion

repo

rt (

Cha

rles

Dar

win

U

nive

rsity,

200

9).

An

early

inte

rven

tion

pro

gram

whi

ch

aim

s to

hel

p pa

rent

s an

d yo

ung

child

ren

deal

with

emot

iona

l iss

ues

and

chal

leng

ing

beha

viou

r. Th

erap

eutic

supp

ort

is p

rovi

ded

to h

elp

child

ren’

s so

cial

and

em

otio

nal d

evel

opm

ent

befo

re

they

sta

rt s

choo

l.

Str

ong

and

thor

ough

ana

lysi

s of

bot

h qu

antita

tive

and

qua

litat

ive

data

. Fu

ll so

urce

s of

dat

a at

tach

ed in

the

ap

pend

ix.

No

cont

rol g

roup

.

The

repo

rt s

tate

s th

at t

here

wer

e m

arke

d im

prov

emen

ts in

the

beh

avio

ur

of b

oth

child

ren

and

pare

nts.

The

m

ost

impr

ovem

ent

was

see

n in

urb

an

indi

geno

us g

irls

and

urb

an n

on-

indi

geno

us b

oys.

Mo

dera

te

The

eval

uation

met

hodo

logy

has

a s

ound

fr

amew

ork.

How

ever

, it d

oes

not

list

the

% o

f pa

rtic

ipan

ts w

ho c

ompl

eted

th

e en

tire

12-

wee

k pr

ogra

m.

Rat

her,

it

only

list

s pa

rtic

ipan

ts w

ho c

ompl

eted

4

or m

ore

sess

ions

, w

hich

is 4

3% for

Tiw

i pe

ople

, an

d 10

% for

urb

an in

dige

nous

pe

ople

. Th

e or

igin

al s

ampl

e si

ze is

11

0 (o

f w

hich

47

are

non-

indi

geno

us)

so it

is a

ver

y sm

all s

ampl

e po

ol.

In

addi

tion

, th

e pr

ogra

m w

as d

eliv

ered

us

ing

differ

ent

met

hods

in t

he d

iffer

ent

loca

tion

s.

How

ever

, m

any

differ

ent

type

s of

an

alys

is h

ave

been

app

lied

in t

he r

epor

t,

resu

ltin

g in

the

mod

erat

e ra

ting

.

15

. Ed

uca

tio

nTh

e n

ati

on

al C

are

fo

r K

ids’

Ears

C

am

paig

n —

eva

luat

ion

repo

rt (

CIR

CA,

2013

).

This

pro

gram

is fun

ded

by t

he fed

eral

go

vern

men

t un

der

a fo

ur-y

ear

agre

emen

t to

impr

ove

eye

and

ear

heal

th s

ervi

ces

for

Indi

geno

us p

eopl

e.

The

eval

uation

use

d th

e fo

llow

ing

met

hodo

logy

:

Inte

rvie

ws

and

an o

nlin

e su

rvey

with

educ

atio

n an

d he

alth

pro

fess

iona

ls;

case

st

udie

s ba

sed

on f

ocus

gro

ups/

inte

rvie

w

with

pare

nts/

care

rs a

nd a

qua

ntitat

ive

surv

ey c

ondu

cted

with

mot

hers

and

fe

mal

e ca

rers

of

child

ren

aged

0-5

yea

rs

(bas

elin

e in

Jul

y 20

11 a

nd f

ollo

w-u

p fr

om N

ovem

ber

2012

to

Febr

uary

201

3),

with

a sa

mpl

e si

ze o

f n=

200

in e

ach

roun

d.

The

eval

uation

fou

nd t

hat

the

Cam

paig

n ha

d a

posi

tive

impa

ct o

n aw

aren

ess

of

ear

dise

ase

amon

g Abo

rigi

nal a

nd T

orre

s Str

ait

Isla

nder

com

mun

itie

s, in

clud

ing

incr

ease

d kn

owle

dge

of s

ympt

oms

and

prev

ention

, an

d in

crea

sed

help

-see

king

be

havi

ours

, as

evi

denc

ed t

hrou

gh a

fo

llow

-up

surv

ey o

f 20

0 m

othe

rs/c

arer

s 18

mon

ths

afte

r th

e ca

mpa

ign

laun

ch.

Mo

dera

te

The

met

hodo

logy

was

sou

nd,

with

a ba

selin

e su

rvey

con

duct

ed t

o m

easu

re

the

impa

ct o

f th

e na

tion

al c

ampa

ign

and

a co

ntro

l gro

up.

How

ever

, po

tent

ial

for

bias

due

to

the

non-

repr

esen

tative

na

ture

of th

e sa

mpl

ing,

in p

articu

lar

the

choi

ce o

f lo

cation

s w

as n

ot

rand

omis

ed.

As

a re

sult t

he fin

ding

s ar

e no

t ge

nera

lisab

le t

o th

e w

ider

tar

get

popu

lation

of Abo

rigi

nal a

nd/o

r To

rres

Str

ait

Isla

nder

mot

hers

and

car

ers

of

child

ren

aged

0 t

o 5

year

s in

Aus

tral

ia.


Typ

e o

f p

rog

ram

Nam

e a

nd

desc

rip

tio

n o

f p

rog

ram

/fo

rm o

f p

ub

lica

tio

nTyp

e o

f eva

luati

on

/d

esc

rip

tio

n o

f m

eth

od

olo

gy

Key

fin

din

gs

Rati

ng

of

evi

den

ce

16

. Ed

uca

tio

nA

ust

ralian

Ele

cto

ral C

om

mis

sio

n’s

(A

EC

) In

dig

en

ou

s ele

cto

ral

part

icip

ati

on

pro

gra

m —

eva

luat

ion

volu

mes

1 a

nd 2

(AEC

, 20

12).

Vol

ume

2 co

ntai

ns t

he c

ase

stud

ies

IEPP

pro

gram

is a

imed

at

empo

wer

ing

Abo

rigi

nal a

nd T

orre

s Str

ait

Isla

nder

Aus

tral

ians

in e

xerc

isin

g th

eir

righ

t to

vo

te.

Eval

uation

met

hodo

logy

incl

uded

: lit

erat

ure

scan

and

doc

umen

t re

view

; se

mi-

stru

ctur

ed in

terv

iew

s w

ith

staf

f;

focu

s gr

oups

; ca

se s

tudi

es w

ith

a c

ross

se

ctio

n of

com

mun

itie

s an

d a

naly

sis

of

data

ava

ilabl

e fr

om t

he Q

ueen

slan

d an

d th

e N

orth

ern

Terr

itor

y el

ection

s

The

eval

uation

fou

nd v

aria

tion

in t

he

degr

ee t

o w

hich

IEP

P’s

sate

d ob

ject

ives

an

d ou

tcom

es h

ave

been

ach

ieve

d. T

he

eval

uation

rec

omm

ende

d ba

sing

fut

ure

chan

ges

to t

he p

rogr

am o

n ev

iden

ce

of ‘w

hat

wor

ks’,

and

harn

essi

ng t

he

expe

rien

ces

of o

ther

gov

ernm

ent

agen

cies

and

pro

gram

s w

orki

ng in

an

Indi

geno

us c

onte

xt.

This

incl

udes

the

ad

option

of

a ro

bust

mon

itor

ing

and

eval

uation

sys

tem

and

rou

tine

ana

lysi

s of

per

form

ance

dat

a.

Mo

dera

te

Met

hodo

logy

was

rob

ust

and

incl

uded

a

Mon

itor

ing

and

Eval

uation

Fra

mew

ork

and

tria

ngul

atio

n of

qua

litat

ive

and

quan

tita

tive

dat

a. I

nfor

mat

ion

on t

he

num

ber

of p

eopl

e w

ho p

artici

pate

d in

in

terv

iew

s/fo

cus

grou

ps w

as p

rovi

ded

and

inte

rvie

w g

uide

s w

ere

prov

ided

in

appe

ndix

. H

owev

er,

diffic

ult

to id

entify

ch

ange

s in

ele

ctor

al b

ehav

iour

as

ethn

icity

not

reco

rded

on

the

elec

tora

l ro

ll or

whe

n pe

ople

vot

e. P

erfo

rman

ce

data

for

the

pro

gram

was

als

o no

t en

tere

d un

iform

ly o

r co

nsis

tent

ly

by S

tate

s an

d Te

rritor

ies.

Lac

k of

pr

epar

atio

n, c

apac

ity

and

trai

ning

im

pact

ed o

n qu

alitat

ive

anal

ysis

of Fi

eld

Offic

er jou

rnal

s.

17

. Ed

uca

tio

nA

ust

ralian

In

dig

en

ou

s Ed

uca

tio

n

Fou

nd

ati

on

(A

IEF)

— a

nnua

l rep

ort

The

AIE

F is

a p

riva

te,

non-

prof

it

orga

nisa

tion

tha

t pr

ovid

es s

chol

arsh

ips

to e

nabl

e In

dige

nous

stu

dent

s to

at

tend

priva

te b

oard

ing

scho

ols.

The

pr

ogra

ms

also

pro

vide

men

toring

and

ca

reer

sup

port

to

enco

urag

e In

dige

nous

st

uden

ts t

o at

tend

Uni

vers

ity.

The

re a

re

two

core

pro

gram

s: t

he A

IEF

Sch

olar

ship

Pr

ogra

m a

nd t

he A

IEF

Path

way

s Pr

ogra

m.

Ann

ual r

epor

ts/r

evie

ws

In 2

015

the

num

ber

of s

tude

nts

atte

ndin

g sc

hool

and

uni

vers

ity

on

AIE

F Sch

olar

ship

s ex

ceed

ed 5

00 for

the

fir

st t

ime.

Sch

ool s

tude

nts

supp

orte

d by

the

pr

ogra

m a

chie

ved

a 93

% r

eten

tion

and

Ye

ar 1

2 co

mpl

etio

n ra

te a

nd 9

6% o

f te

rtia

ry s

chol

arsh

ip s

tude

nts

cont

inue

d or

com

plet

ed t

heir u

nive

rsity

stud

ies

during

the

yea

r.

N/

A


Typ

e o

f p

rog

ram

Nam

e a

nd

desc

rip

tio

n o

f p

rog

ram

/fo

rm o

f p

ub

lica

tio

nTyp

e o

f eva

luati

on

/d

esc

rip

tio

n o

f m

eth

od

olo

gy

Key

fin

din

gs

Rati

ng

of

evi

den

ce

18

. Ed

uca

tio

nG

an

bin

a —

Im

pact

Ass

essm

ent

(SVA

, 20

16).

Gan

bina

was

est

ablis

hed

in 1

997

to h

elp

impr

ove

scho

ol a

nd fur

ther

edu

cation

co

mpl

etio

n ra

tes

and

‘rea

l’ jo

b pr

ospe

cts

amon

g ab

out

6000

Ind

igen

ous

peop

le in

th

e G

oulb

urn

Valle

y in

Vic

toria.

Gan

bina

is u

niqu

e in

tha

t it is

100

%

fund

ed b

y co

rpor

ate

and

phila

nthr

opic

fo

unda

tion

s.

The

cost

per

par

tici

pant

ave

rage

s $3

,300

a

year

.

The

purp

ose

of t

he im

pact

ass

essm

ent

was

to

asse

ss G

anbi

na’s

cum

ulat

ive

impa

ct s

ince

200

5.

The

met

hodo

logy

use

d co

nsis

ted

of s

take

hold

er c

onsu

ltat

ions

and

in

terv

iew

s, r

evie

win

g G

anbi

na’s

clie

nt

data

, de

skto

p re

sear

ch o

n re

leva

nt

cont

extu

al a

nd e

nviron

men

tal f

acto

rs,

and

data

col

lect

ed o

n pr

evio

us

eval

uation

s an

d as

sess

men

ts o

f G

anbi

na.

Year

11

to Y

ear

12 r

eten

tion

rat

es

incr

ease

d fr

om 6

2% in

200

9-10

to

73%

in

201

5-16

far

sur

pass

ing

the

Gre

ater

She

ppar

ton

Indi

geno

us a

nd n

atio

nal

Indi

geno

us r

ates

.

All

part

icip

ants

age

d 25

to

34 y

ears

who

ha

d be

en w

ith

Gan

bina

for

fiv

e ye

ars

or m

ore

had

atta

ined

Yea

r 12

or

an

equi

vale

nt q

ualif

icat

ion.

Uni

vers

ity

part

icip

atio

n in

crea

sed

from

tw

o G

anbi

na p

artici

pant

s in

200

9 to

15

in 2

016.

Str

on

g

A r

ange

of da

ta s

ourc

es w

as u

tilis

ed a

nd

a re

view

of pr

evio

us e

valu

atio

ns w

as

unde

rtak

en t

o pr

ovid

e a

long

itud

inal

as

sess

men

t of

the

pro

gram

’s im

pact

.

19

. Ed

uca

tio

nC

reati

ve R

eco

very

Pilo

t P

rog

ram

—

an e

valu

atio

n re

port

(fo

r D

epar

tmen

t of

Sci

ence

, Te

chno

logy

, In

nova

tion

and

th

e Art

s, c

ondu

cted

by

NSF

Con

sultin

g,

Dec

embe

r 20

12).

The

Cre

ativ

e re

cove

ry p

roje

ct t

each

es

arts

ski

lls t

o In

dige

nous

peo

ple

with

men

tal h

ealth

issu

es t

o he

lp im

prov

e th

eir

emot

iona

l wel

l-be

ing.

The

met

hodo

logy

was

com

pris

ed

sole

ly o

f qu

alitat

ive

data

: s

take

hold

er

cons

ulta

tion

s, in

-dep

th in

terv

iew

s,

part

icip

atio

n in

the

Cre

ativ

e Rec

over

y N

atio

nal F

orum

and

cas

e st

udie

s.

The

eval

uation

was

con

duct

ed o

ne

mon

th b

efor

e pr

ogra

m c

ompl

etio

n.

The

conc

lusi

ons

wer

e th

at t

he p

ilot

proj

ect

cont

ribu

ted

to a

“gr

owin

g bo

dy o

f ev

iden

ce”

that

art

s-le

d re

cove

ry p

roce

sses

are

ben

efic

ial f

or

the

com

mun

ity.

How

ever

, du

e to

the

ev

alua

tion

dat

a co

min

g so

lely

fro

m

prog

ram

wor

kers

and

ass

ocia

ted

artist

s,

the

conc

lusi

on is

on

shak

y gr

ound

s.

Ther

e w

as n

o da

ta g

athe

red

from

the

w

ider

com

mun

ity.

Weak

Altho

ugh

the

eval

uation

con

clud

ed t

hat

the

prog

ram

met

its

obje

ctiv

es,

ther

e w

as n

o co

st-b

enef

it a

naly

sis

prov

ided

, ot

her

than

mea

suring

suc

cess

as

prod

ucin

g th

e de

sire

d nu

mbe

r of

eve

nts

and

othe

r pr

ogra

ms.

In

othe

r w

ords

, by

co

mpl

etin

g/pr

oduc

ing

enou

gh p

rogr

ams,

it s

tate

d th

at it

was

a s

ucce

ss.

The

appe

ndic

es c

lear

ly s

how

tha

t th

e on

ly s

take

hold

ers

cons

ulte

d w

ere

inte

rnal

(w

orke

rs,

cons

ults

and

oth

ers

who

per

sona

lly b

enef

itte

d fr

om t

he

proj

ect)

.

20

. Ed

uca

tio

nIn

dig

en

ou

s Y

ou

th L

ead

ers

hip

P

rog

ram

— a

n ev

alua

tion

rep

ort

(Job

s Aus

tral

ia F

ound

atio

n, p

repa

red

by E

MS

Con

sultan

ts,

Oct

ober

201

1).

The

prog

ram

aim

s to

hel

p in

crea

se

Indi

geno

us y

oung

peo

ple’

s le

ader

ship

sk

ills

and

for

them

to

beco

me

posi

tive

ro

le m

odel

s in

the

ir c

omm

unitie

s. T

he

eval

uation

cov

ers

a pe

riod

of tw

o ye

ars,

20

10 a

nd 2

011.

Ther

e w

as a

bro

ad v

arie

ty o

f qu

antita

tive

an

d qu

alitat

ive

met

hods

use

d to

ev

alua

te.

How

ever

, th

e sa

mpl

e po

ol is

sm

all (

22 p

artici

pant

s; m

ente

es a

nd

men

tors

).

A d

raw

back

is t

hat

mos

t of

the

ev

alua

tion

is b

ased

on

“sel

f-re

port

ing”

—

aski

ng t

he p

artici

pant

s qu

estion

s at

the

en

d of

the

pro

gram

abo

ut t

heir a

ttitud

es

and

feel

ings

of

satisf

action

.

The

eval

uation

rep

ort

conc

lude

s th

at

the

prog

ram

was

a s

ucce

ss.

How

ever

, it

is s

tate

d th

at m

ore

effo

rt n

eede

d to

be

put

into

fin

ding

app

ropr

iate

men

tors

, as

man

y of

the

m w

ere

ill-e

quip

ped

to

perf

orm

the

ir d

utie

s as

out

lined

in t

he

prog

ram

.

Whi

le le

ader

ship

ski

lls w

ere

one

of

the

mai

n ob

ject

ives

of th

e pr

ogra

m,

not

all p

artici

pant

s re

port

ed t

hat

thei

r le

ader

ship

ski

lls h

ad im

prov

ed.

Weak

Due

to

a he

avy

focu

s on

sel

f-as

sess

men

t, a

nd n

o fo

llow

up,

man

y of

th

e le

arne

d ou

tcom

es c

anno

t de

finitel

y be

att

ribu

ted

to t

he a

ctiv

itie

s of

the

pr

ogra

m.

Exam

ples

of th

e qu

estion

naires

wer

e at

tach

ed a

s an

app

endi

x, b

ut n

ot t

he

answ

ers/

data

.


Typ

e o

f p

rog

ram

Nam

e a

nd

desc

rip

tio

n o

f p

rog

ram

/fo

rm o

f p

ub

lica

tio

nTyp

e o

f eva

luati

on

/d

esc

rip

tio

n o

f m

eth

od

olo

gy

Key

fin

din

gs

Rati

ng

of

evi

den

ce

21

. Ed

uca

tio

nTh

e S

po

rtin

g C

han

ce P

rog

ram

—

au

dit

repo

rt (

Offic

e of

Eva

luat

ion

and

Aud

it O

EA,

Indi

geno

us P

rogr

ams,

200

9 an

d an

eva

luat

ion

(Lon

sdal

e et

al,

2011

).

The

prog

ram

use

s sp

ort

and

recr

eation

to

hel

p im

prov

e ed

ucat

iona

l out

com

es

for

Indi

geno

us s

tude

nts

The

obje

ctiv

e of

the

per

form

ance

au

dit

was

to

asse

ss t

he p

erfo

rman

ce

of a

cade

mie

s fu

nded

und

er t

he

Spo

rtin

g Cha

nce

Prog

ram

and

DEE

WR’s

m

anag

emen

t of

the

pro

gram

.

The

audi

t di

d no

t as

sess

the

pe

rfor

man

ce o

f th

e pr

ogra

m’s

edu

cation

en

gage

men

t st

rate

gies

com

pone

nt

as t

his

part

of

the

prog

ram

was

not

in

trod

uced

unt

il 20

08.

An

eval

uation

of

the

Spo

rtin

g Cha

nce

Prog

ram

.

The

audi

t co

nclu

ded

that

the

aca

dem

ies’

ef

fort

s w

ere

dire

cted

tow

ard

achi

evin

g in

term

edia

te o

utco

mes

— t

hat

is,

help

ing

stud

ents

to

com

e to

sch

ool

and

impr

ovin

g th

eir

beha

viou

r an

d en

gage

men

t in

the

cla

ssro

om.

Ane

cdot

al

evid

ence

sug

gest

ed t

he p

rogr

am h

ad

a po

sitive

impa

ct o

n th

e ed

ucat

iona

l ex

perien

ces

of I

ndig

enou

s st

uden

ts.

Of

the

143

scho

ols

cont

acte

d as

par

t of

the

eva

luat

ion,

87

(61

per

cent

) pa

rtic

ipat

ed.

Sch

ool s

taff c

onsi

dere

d th

e pr

ogra

m t

o ha

ve a

mod

erat

e im

pact

.

Weak

As

ther

e w

as li

mited

per

form

ance

dat

a av

aila

ble,

the

aud

itor

s w

ere

unab

le t

o co

mm

ent

on t

he e

xten

t to

whi

ch t

he

acad

emie

s ha

d im

prov

ed s

tude

nts’

en

rolm

ent,

att

enda

nce,

ret

ention

and

en

gage

men

t.

The

eval

uation

did

not

incl

ude

com

pariso

n re

sults

for

scho

ols

and

stud

ents

not

in t

he p

rogr

am.

22

. Ed

uca

tio

nS

cho

ol N

utr

itio

n P

rog

ram

—

Sta

keho

lder

sur

vey

(Dep

. O

f Em

ploy

men

t, E

duca

tion

and

Wor

kpla

ce

Rel

atio

ns,

2009

).

The

prog

ram

pro

vide

s br

eakf

ast

and/

or lu

nch

to s

choo

l-ag

ed c

hild

ren

from

re

mot

e co

mm

unitie

s of

the

Nor

ther

n Te

rritor

y, t

o he

lp e

ncou

rage

the

m t

o at

tend

sch

ool a

nd t

o en

gage

in le

arni

ng.

Qua

litat

ive

stak

ehol

der

surv

ey o

nly.

No

cont

rol g

roup

.

The

conc

lusi

on o

f th

e su

rvey

is t

hat

of

over

all s

atis

fact

ion

of t

he p

rogr

am,

as

mea

sure

d by

the

sel

f-as

sess

men

t by

va

riou

s st

akeh

olde

rs.

Weak

The

data

incl

uded

in t

he r

epor

t is

pur

ely

qual

itat

ive.

It

emph

asis

es s

tate

men

ts

mad

e by

sta

keho

lder

s (p

rovi

ders

, pa

rent

s, e

tc)

such

as

‘att

enda

nce

has

impr

oved

’, ho

wev

er it

pro

vide

s no

st

atis

tica

l dat

a as

evi

denc

e.

23

. Ed

uca

tio

nTh

e M

on

eyM

ob

Talk

ab

ou

t (M

MT)

pro

gra

m —

an

eval

uation

(G

arne

r, S

and

Pryo

r, A,

2015

)

The

MM

T pr

ogra

m w

as e

stab

lishe

d in

20

12 in

the

Am

ata,

Mim

ili a

nd P

ukat

ja

com

mun

itie

s of

SA a

nd a

ssis

ts p

eopl

e in

rem

ote

Abo

rigi

nal c

omm

unitie

s to

m

anag

e th

eir

mon

ey.

Team

s al

so v

isit

othe

r re

mot

e co

mm

unitie

s in

SA,

WA

and

the

NT

prov

idin

g se

rvic

es o

n an

ou

trea

ch b

asis

.

No

met

hodo

logy

sec

tion

pro

vide

d in

re

port

, ho

wev

er m

etho

dolo

gy a

ppea

red

to c

onsi

st o

f su

rvey

s ad

min

iste

red

thro

ugh

field

trip

s an

d an

alys

is o

f pr

ogra

m d

ata

The

eval

uation

fou

nd t

hat,

whi

le m

any

com

mun

ity

mem

bers

still

had

com

plex

ba

rrie

rs t

o im

prov

ing

thei

r fin

anci

al

liter

acy,

MM

T cl

ient

s w

ere

mor

e lik

ely

than

non

-MM

T cl

ient

s to

hav

e de

velo

ped

basi

c fin

anci

al m

anag

emen

t sk

ills.

Weak

Eval

uation

wou

ld h

ave

been

impr

oved

by

a d

etai

led

met

hodo

logy

sec

tion

in

the

rep

ort

and

a di

scus

sion

on

the

limitat

ions

of th

e m

etho

dolo

gy.


Typ

e o

f p

rog

ram

Nam

e a

nd

desc

rip

tio

n o

f p

rog

ram

/fo

rm o

f p

ub

lica

tio

nTyp

e o

f eva

luati

on

/d

esc

rip

tio

n o

f m

eth

od

olo

gy

Key

fin

din

gs

Rati

ng

of

evi

den

ce

24

. H

ealt

hG

row

ing

Str

on

g —

Feed

ing

Yo

u

an

d Y

ou

r B

ab

y —

eva

luat

ion

repo

rt

(Que

ensl

and

Hea

lth,

200

9).

The

prog

ram

pro

vide

s in

form

atio

n on

he

alth

y fo

od t

o ea

t du

ring

pre

gnan

cy.

Qua

ntitat

ive:

dis

trib

utio

n an

d pr

omot

ion

of c

reat

ed c

onte

nt.

Qua

litat

ive:

sur

veys

with

heal

th w

orke

rs

and

clie

nt d

iscu

ssio

n gr

oups

.

The

goal

of

the

eval

uation

was

to

eval

uate

the

usa

ge o

f th

e G

row

ing

Str

ong

reso

urce

s by

hea

lth

wor

kers

and

ot

her

heal

th p

rofe

ssio

nals

thr

ough

out

Que

ensl

and,

and

to

obta

in c

lient

fe

edba

ck.

No

cont

rol g

roup

.

Goo

d fe

edba

ck w

as g

ener

ally

giv

en o

n th

e ut

ilisa

tion

of th

e G

S m

ater

ial,

with

9/10

hea

lth

wor

kers

res

pond

ing

that

th

ey h

ad u

sed

the

mat

eria

ls.

Posi

tive

fe

edba

ck w

as a

lso

give

n on

the

qua

lity

of t

he m

ater

ials

, an

d th

e or

gani

sation

ha

d 65

indi

vidu

al r

eque

sts

for

mor

e re

sour

ces/

mat

eria

ls,

mai

nly

from

oth

er

indi

geno

us-r

un N

GO

s.

Mo

dera

te

Hea

lth

wor

ker

surv

eys

had

a 39

%

resp

onse

rat

e an

d 69

clie

nts

part

icip

ated

in

dis

cuss

ion

grou

ps.

The

feed

back

is a

naly

sed

thor

ough

ly,

how

ever

it m

ust

be e

mph

asis

ed t

hat

the

eval

uation

is o

n th

e qu

ality

and

utili

sation

of th

e m

ater

ials

, ra

ther

tha

n w

heth

er t

he m

ater

ial m

ade

a po

sitive

di

ffer

ence

am

ongs

t re

cipi

ents

.

25

. H

ealt

hEO

N T

hri

vin

g C

om

mu

nit

ies

Pro

gra

m

in S

ix K

imb

erl

ey

Co

mm

un

itie

s —

Ev

alua

tion

rep

ort

(KPM

G,

2013

). A

The

Edge

of N

owhe

re F

ound

atio

n (E

ON

) de

liver

s th

e pr

ogra

m t

o pr

omot

e he

alth

ier

lifes

tyle

s in

rem

ote

com

mun

itie

s. Th

e Th

rivi

ng C

omm

unitie

s Pr

ogra

m is

cur

rent

ly b

eing

del

iver

ed in

16

com

mun

itie

s in

WA —

13

acro

ss t

he

Kim

berley

reg

ion

and

3 in

the

Pilb

ara.

A p

roce

ss a

nd o

utco

mes

eva

luat

ion

of

the

prog

ram

in s

ix r

emot

e In

dige

nous

co

mm

unitie

s. T

he e

valu

atio

n us

ed a

cas

e st

udy

appr

oach

to

desc

ribe

the

evo

lution

an

d im

pact

of

each

of

the

Prog

ram

co

mpo

nent

s in

all

loca

tion

s.

Lite

ratu

re a

nd d

ata

revi

ew.

Sta

keho

lder

con

sultat

ions

.

Fiel

d-vi

sits

to

five

of t

he s

ix e

valu

atio

n co

mm

unitie

s (w

ith

wet

wea

ther

pr

even

ting

the

vis

it t

o th

e si

xth

com

mun

ity)

.

The

eval

uation

fou

nd E

ON

tak

es ‘a

ge

nuin

e co

mm

unity

deve

lopm

ent

appr

oach

tha

t va

lues

long

ter

m

enga

gem

ent

over

rap

id d

eliv

ery…

’ Ev

iden

ce s

ugge

sted

the

pro

gram

ha

d in

crea

sed

peop

le’s

kno

wle

dge

of h

orticu

ltur

al a

nd h

ealthy

eat

ing.

H

owev

er,

the

eval

uation

fou

nd t

hat

poor

loca

l gov

erna

nce

and

laps

es in

the

ad

min

istr

atio

n of

cou

ncils

had

mad

e en

gage

men

t w

ith

resi

dent

s in

som

e co

mm

unitie

s di

ffic

ult.

Mo

dera

te

Eval

uation

des

ign

was

rel

ativ

ely

good

bu

t co

mm

unity

leve

l dat

a w

as n

ot

avai

labl

e fo

r th

ree

of t

he c

omm

unitie

s.

26

. H

ealt

hP

etr

ol sn

iffi

ng

str

ate

gy

— a

n ev

alua

tion

rep

ort

(Origi

n Con

sultin

g,

2013

).

The

stra

tegy

aim

ed t

o re

duce

pet

rol

sniff

ing

and

othe

r ty

pes

of s

ubst

ance

ab

use

amon

g In

dige

nous

you

ng p

eopl

e.

A W

hole

of

Str

ateg

y Ev

alua

tion

(W

OSE)

—

a h

igh-

leve

l str

ateg

ic r

evie

w o

f th

e im

plem

enta

tion

of

the

Petr

ol S

niff

ing

Str

ateg

y (P

SS)

sinc

e its

esta

blis

hmen

t in

20

04/0

5.

Ove

rall,

and

par

ticu

larly

thro

ugh

the

roll

out

of lo

w a

rom

atic

fue

l (LA

F) a

nd

yout

h se

rvic

es,

the

PSS h

as a

chie

ved

a dr

amat

ic r

educ

tion

in t

he p

reva

lenc

e of

sn

iffin

g ac

ross

muc

h of

rem

ote

Aus

tral

ia.

Mo

dera

te

The

focu

s of

the

eva

luat

ion

was

on

the

man

agem

ent

and

impl

emen

tation

of th

e PS

S a

nd d

id n

ot a

sses

s th

e cu

rren

t le

vel

of s

niffin

g to

illu

stra

te im

pact

.


Typ

e o

f p

rog

ram

Nam

e a

nd

desc

rip

tio

n o

f p

rog

ram

/fo

rm o

f p

ub

lica

tio

nTyp

e o

f eva

luati

on

/d

esc

rip

tio

n o

f m

eth

od

olo

gy

Key

fin

din

gs

Rati

ng

of

evi

den

ce

27

. H

ealt

hFi

llin

g t

he G

ap

In

dig

en

ou

s D

en

tal

Pro

gra

m —

fin

al e

valu

atio

n re

port

(L

. Pu

lver

& a

l, U

nive

rsity

of N

SW

, D

ecem

ber

2009

).

‘Fill

ing

the

Gap

’ rec

ruits

volu

ntee

r de

ntis

ts t

o ad

dres

s th

e ch

roni

c sh

orta

ge o

f de

ntal

pro

fess

iona

ls

avai

labl

e to

Ind

igen

ous

peop

le in

no

rthe

rn Q

ueen

slan

d.

Lite

ratu

re r

evie

w.

Qua

ntitat

ive

data

fro

m p

atie

nt

data

base

s.

In-p

erso

n, t

elep

hone

and

gro

up

inte

rvie

ws.

The

repo

rt is

ver

y in

-dep

th a

nd a

naly

ses

the

prog

ram

tho

roug

hly,

but

in t

erm

s of

ser

vice

del

iver

y an

d ou

tcom

es (

with

wha

t da

ta w

as a

vaila

ble)

.

The

repo

rt s

tate

s th

at t

he in

crea

se in

the

nu

mbe

rs o

f in

dige

nous

peo

ple

acce

ssin

g th

e se

rvic

es c

onfir

m t

he p

ress

ing

need

fo

r pu

blic

ly f

unde

d de

ntal

car

e in

rem

ote

loca

tion

s.

Mo

dera

te

Altho

ugh

the

repo

rt d

oes

not

utili

se

a co

ntro

l gro

up/b

ench

mar

k da

ta t

o ev

alua

te e

ffec

tive

ness

, it d

oes

prov

ide

a ve

ry g

ood

over

all,

and

in s

ome

area

s de

taile

d, a

naly

sis

of t

he p

rogr

am,

usin

g se

vera

l diff

eren

t m

etrics

.

28

. H

ealt

hTri

-Sta

te H

IV/

STI

Pro

ject

—

Eval

uation

(Aus

tral

ian

Inst

itut

e fo

r Pr

imar

y Car

e, 2

004)

The

proj

ect

aim

s to

red

uce

the

inci

dent

s an

d im

pact

of se

xual

ly t

rans

mis

sibl

e in

fect

ions

on

rem

ote

Indi

geno

us p

eopl

e liv

ing

on t

he b

orde

rs o

f Sou

th A

ustr

alia

, W

este

rn A

ustr

alia

, an

d th

e N

orth

ern

Terr

itor

y.

This

pro

ject

has

bee

n su

bjec

ted

to s

ever

al e

valu

atio

ns s

ince

its

impl

emen

tation

in 1

994.

The

mos

t re

cent

eva

luat

ion

stra

tegy

in

clud

ed a

qua

ntitat

ive

anal

ysis

of

STI

/HIV

sur

veill

ance

dat

a an

d th

e su

rvei

llanc

e sy

stem

, co

mbi

ned

with

a qu

alitat

ive

proc

ess

eval

uation

.

Key

suc

cess

es o

f th

e pr

ogra

m id

entifie

d w

ere:

enha

nced

inte

grat

ion

of s

exua

l hea

lth

into

com

preh

ensi

ve lo

cal p

rim

ary

heal

th c

are

deliv

ery;

exp

ande

d an

d co

ordi

nate

d ac

tive

cas

e fin

ding

; su

stai

ned

regi

onal

co

mm

itm

ent/

appr

oach

es t

o STI

con

trol

; an

d cr

oss-

bord

er s

tand

ardi

sation

of

sexu

al h

ealth

proc

edur

es.

How

ever

, m

ore

effe

ctiv

e lin

ks t

o po

licy

and

plan

ning

dev

elop

men

t in

the

reg

ion

was

ne

eded

.

Mo

dera

te

Altho

ugh

the

eval

uation

met

hodo

logy

w

as r

obus

t, t

he m

odel

for

the

pro

gram

w

as m

issi

ng a

set

of ke

y pe

rfor

man

ce

indi

cato

rs,

so t

hat

activi

ties

cou

ld b

e as

sess

ed a

gain

st o

bjec

tive

s.

29

. H

ealt

hW

om

en

’s d

eve

lop

men

t p

roje

ct —

ev

alua

tion

(Fr

ed H

ollo

ws

Foun

dation

, 20

12)

The

proj

ect

aim

s to

incr

ease

sel

f-de

term

inat

ion

amon

g In

dige

nous

wom

en

livin

g in

Eas

t Kat

herine

in t

he N

orth

ern

Terr

itor

y.

Str

engt

h-ba

sed

eval

uation

met

hodo

logy

.

Rev

iew

of

Doc

umen

tation

and

Liter

atur

e.

Initia

l Fie

ld T

rip

and

Sta

keho

lder

In

terv

iew

s.

Sec

ond

Fiel

d Tr

ip a

nd S

take

hold

er

Inte

rvie

ws.

The

eval

uation

fou

nd t

he p

rogr

am h

ad

‘Inc

reas

ed t

he s

elf-

dete

rmin

atio

n of

w

omen

in t

he J

awoy

n re

gion

’. Th

e m

ost

impo

rtan

t im

pact

on

the

wom

en’s

sel

f-de

term

inat

ion

was

the

est

ablis

hmen

t of

the

Ban

atja

rl W

omen

’sCou

ncil

and

the

elec

tion

of of

fice

bear

ers,

whi

ch

form

alis

ed t

he r

ole

of t

he w

omen

’s

cent

res

in s

peak

ing

up for

wom

en a

nd

taki

ng c

ontr

ol o

f is

sues

tha

t af

fect

w

omen

in t

he r

egio

n.

Mo

dera

te

Altho

ugh

the

met

hodo

logy

use

d w

as

soun

d, a

s th

is w

as t

he first

qua

litat

ive

eval

uation

the

re w

as n

o fo

rmal

bas

elin

e in

form

atio

n re

gard

ing

wom

en’s

em

pow

erm

ent

to d

raw

on.

Hig

h po

pula

tion

mob

ility

mea

nt s

ome

of t

he k

ey w

omen

wer

e aw

ay fro

m

thei

r co

mm

unitie

s du

ring

the

tim

e of

fie

ldw

ork.


Typ

e o

f p

rog

ram

Nam

e a

nd

desc

rip

tio

n o

f p

rog

ram

/fo

rm o

f p

ub

lica

tio

nTyp

e o

f eva

luati

on

/d

esc

rip

tio

n o

f m

eth

od

olo

gy

Key

fin

din

gs

Rati

ng

of

evi

den

ce

30

. H

ealt

hW

url

i-W

url

inja

ng

dia

bete

s d

ay

pro

gra

m —

an e

valu

atio

n (C

entr

e fo

r Rem

ote

Hea

lth,

201

1)

The

prog

ram

aim

s to

impr

ove

outc

omes

fo

r cl

ient

s w

ith

Type

2 D

iabe

tes

at W

urli-

Wur

linja

ng H

ealth

Ser

vice

by

enco

urag

ing

grea

ter

self-

man

agem

ent

of

diab

etes

.

A p

rogr

am lo

gic

was

use

d to

hel

p de

velo

p an

eva

luat

ion

fram

ewor

k.

This

fra

mew

ork

allo

wed

the

pro

gram

to

be e

valu

ated

in t

erm

s of

pro

cess

, im

pact

an

d ou

tcom

e.

The

prog

ram

logi

c en

able

d ef

fect

ive

eval

uation

of th

e pr

ogra

m a

nd

dem

onst

rate

d th

at t

he p

rogr

am m

odel

is

bas

ical

ly s

ound

and

gen

eral

isab

le t

o ot

her

serv

ices

. H

owev

er,

a nu

mbe

r of

is

sues

wer

e id

entifie

d w

ith

the

prog

ram

su

ch a

s a

poor

rec

all s

yste

m,

educ

atio

n se

ssio

ns t

hat

part

icip

ants

wer

e un

able

to

unde

rsta

nd,

and

med

icat

ion

com

plia

nce

and

tran

spor

t is

sues

.

Mo

dera

te

Altho

ugh

an e

valu

atio

n fr

amew

ork

was

use

d th

ere

was

no

disc

ussi

on o

n th

e lim

itat

ions

of th

e m

etho

dolo

gy.

Part

icip

ant

num

bers

wer

e al

so n

ot

incl

uded

.

31

. H

ealt

hTh

e V

icto

rian

Ab

ori

gin

al S

pect

acl

es

Su

bsi

dy

Sch

em

e —

eva

luat

ion

repo

rt

(Vic

. D

ep.

Of H

ealth

& A

ustr

alia

n Col

lege

of

Opt

omet

ry,

July

201

2).

This

pro

gram

pro

vide

s su

bsid

ies

to

Indi

geno

us V

icto

rian

s fo

r sp

ecta

cles

and

ot

her

visu

al a

ids.

Qua

ntitat

ive:

ser

vice

del

iver

y da

ta (

from

da

taba

ses)

.

Qua

litat

ive:

ser

vice

pro

vide

r in

terv

iew

s,

patien

t st

orie

s, c

omm

unity

awar

enes

s.

No

cont

rol g

roup

.

The

num

ber

of c

onsu

ltat

ions

incr

ease

d by

116

% w

ithi

n th

e fir

st y

ear

of

oper

atio

ns.

Ane

cdot

al e

vide

nce

from

the

in

terv

iew

s se

ems

to c

onfir

m t

he in

crea

se

in d

eman

d fo

r th

e se

rvic

es.

Mo

dera

te

The

eval

uation

fra

mew

ork

is s

ound

, bu

t in

dep

th c

ompa

riso

ns o

f th

e le

vel

of s

ervi

ce p

rovi

sion

pre

-and

pos

t-co

mm

ence

men

t of

the

Sch

eme

was

no

t po

ssib

le a

s ot

her

prov

ider

s ha

d no

t be

en r

equi

red

to id

entify

the

Ind

igen

ous

stat

us o

f pe

ople

acc

essi

ng t

he S

chem

e.

A k

ey li

mitat

ion

of t

he e

valu

atio

n w

as

the

amou

nt o

f se

rvic

e de

liver

y da

ta,

incl

udin

g ba

selin

e da

ta a

vaila

ble

to

anal

yse

whe

ther

the

Sch

eme

had

achi

eved

its

inte

nded

out

com

es.

32

. H

ealt

hTh

e F

am

ily

Wellb

ein

g P

rog

ram

—

an

eva

luat

ion

repo

rt (

Clo

sing

the

Gap

Cle

arin

ghou

se,

2013

).

The

prog

ram

aim

s to

hel

p Abo

rigi

nal

peop

le fin

d ne

w w

ays

to c

ope

with

grie

f an

d lo

ss.T

he p

rogr

am is

bei

ng d

eliv

ered

in

56

site

s ac

ross

Aus

tral

ia.

A s

ynth

esis

of

seve

n fo

rmat

ive

eval

uation

s of

the

pro

gram

, w

hich

in

volv

ed a

tot

al o

f 14

8 ad

ult

and

70

stud

ent

part

icip

ants

.

The

synt

hesi

s fo

und

that

the

pro

gram

ha

d in

crea

sed

the

capa

city

of

part

icip

ants

to

exer

t gr

eate

r co

ntro

l ove

r th

eir

heal

th a

nd w

ellb

eing

.

Ther

e w

as n

o ev

iden

ce p

rese

nted

of

pos

itiv

e ch

ange

s oc

curr

ing

at t

he

broa

der,

com

mun

ity

leve

l.

Mo

dera

te

Rev

iew

ing

the

seve

n ev

alua

tion

s al

low

ed

for

tren

ds o

ver

tim

e to

be

iden

tifie

d.

How

ever

, so

me

of t

he m

etho

dolo

gy o

f th

e se

ven

eval

uation

s w

as li

mited

, fo

r ex

ampl

e, for

one

eva

luat

ion

the

prim

ary

sour

ce o

f da

ta w

as a

naly

sis

of u

nedi

ted

vide

o fo

otag

e fr

om a

one

day

ref

lect

ive

wor

ksho

p fo

r FW

B p

artici

pant

s.


Typ

e o

f p

rog

ram

Nam

e a

nd

desc

rip

tio

n o

f p

rog

ram

/fo

rm o

f p

ub

lica

tio

nTyp

e o

f eva

luati

on

/d

esc

rip

tio

n o

f m

eth

od

olo

gy

Key

fin

din

gs

Rati

ng

of

evi

den

ce

33

. H

ealt

hS

tro

ng

Fath

ers

Str

on

g F

am

ilie

s p

rog

ram

(S

FSF)

— d

escr

iptive

an

alys

is o

f th

e pr

ogra

m ra

ther

tha

n an

ev

alua

tion

(U

rbis

, 20

13).

The

SFS

F pr

ogra

m a

ims

to p

rom

ote

the

role

of Abo

rigi

nal f

athe

rs a

nd o

ther

men

in

sup

port

ing

thei

r ch

ildre

n an

d fa

mili

es

with

a pa

rtic

ular

em

phas

is o

n an

tena

tal

care

and

ear

ly c

hild

hood

.

Des

crip

tive

Ana

lysi

s ra

ther

tha

n an

ev

alua

tion

.

Doc

umen

t re

view

of

prog

ram

dat

a.

Inte

rvie

ws

and

focu

s gr

oups

with

staf

f, pa

rtic

ipan

ts a

nd s

take

hold

ers.

Part

icip

atio

n in

the

prog

ram

del

iver

ed a

num

ber

of p

ositiv

e ou

tcom

es f

or m

en,

incl

udin

g in

crea

sed

self-

conf

iden

ce a

nd e

duca

tion

and

em

ploy

men

t op

port

unitie

s.

The

repo

rt r

ecom

men

ded

deve

lopi

ng

a re

port

ing

tem

plat

e to

cap

ture

dat

a ag

ains

t KPI

s re

gula

rly

and

cons

iste

ntly

. Th

is w

ould

als

o al

low

sites

and

the

D

epar

tmen

t to

mon

itor

impl

emen

tation

an

d ou

tcom

es a

cros

s si

tes,

and

mea

sure

ch

ange

in p

artici

pant

s ov

er t

ime.

N/

A

34

. H

ealt

hG

oo

d q

uic

k t

ukka:

coo

k it,

pla

te it,

sh

are

it

— e

valu

atio

n of

initia

l con

cept

(Q

AIH

C,

no d

ate)

.

The

proj

ect

teac

hes

cook

ing

skill

s w

ith

the

aim

of in

crea

sing

the

num

ber

of

mea

ls I

ndig

enou

s pe

ople

mak

e at

hom

e.

Not

a p

rope

r ev

alua

tion

as

such

, m

ore

an e

valu

atio

n pl

an.

Dat

a pr

imar

ily c

ame

from

fee

dbac

k fo

rms

from

peo

ple

who

had

par

tici

pate

d in

the

pro

ject

and

gro

up in

terv

iew

s.

From

the

qua

litat

ive

data

col

lect

ed,

ther

e w

as a

n in

crea

se in

kno

wle

dge

abou

t ot

her

cultur

al c

uisi

nes,

peo

ple’

s sk

ill le

vel i

ncre

ased

and

the

re w

as a

re

port

ed in

crea

se in

con

fiden

ce a

nd s

elf-

este

em.

One

per

son

foun

d a

job

wor

king

in a

ki

tche

n af

ter

taki

ng p

art

in t

he p

roje

ct.

Res

pond

ents

fro

m t

he g

roup

inte

rvie

w

stat

ed t

hat

they

like

d th

e re

cipe

s an

d w

ante

d m

ore

sess

ions

, 33

% o

f th

e pa

rtic

ipan

ts in

terv

iew

ed h

ad ‘c

ooke

d m

ore

ofte

n’ a

nd 7

1% h

ad c

ooke

d th

e re

cipe

s ag

ain.

N/

A

35

. H

ealt

h

Balu

nu

Fo

un

dati

on

Cu

ltu

ral H

ealin

g

pro

gra

m —

ca

se s

tudy

(M

uru

Mar

ri

Sch

ool o

f Pu

blic

Hea

lth

and

Com

mun

ity

Med

icin

e

UN

SW

, Aus

tral

ia,

2013

).

The

prog

ram

pro

vide

s yo

ung

Abo

rigi

nal

peop

le w

ith

the

oppo

rtun

ity

to r

econ

nect

w

ith

thei

r cu

ltur

al a

nd s

piritu

al id

entities

Cas

e st

udy

used

qua

litat

ive

met

hods

of

data

col

lect

ion

and

anal

ysis

, in

clud

ing

docu

men

tary

rev

iew

, in

-dep

th in

terv

iew

s an

d pa

rtic

ipan

t ob

serv

atio

n of

one

of

the

heal

ing

cam

ps.

The

prog

ram

is d

eliv

ered

by

a st

rong

te

am o

f Abo

rigi

nal p

eopl

e w

ho a

mon

gst

them

hav

e a

rang

e of

exp

erie

nces

, sk

ills

and

know

ledg

e, a

nd a

dee

p co

mm

itm

ent

to m

akin

g th

e pr

ogra

m w

ork.

The

yo

ung

part

icip

ants

rel

ate

wel

l to

the

Abo

rigi

nal s

taff w

ho h

ave

had

sim

ilar

life

expe

rien

ces

to t

hem

in t

he p

ast.

N/

A


Typ

e o

f p

rog

ram

Nam

e a

nd

desc

rip

tio

n o

f p

rog

ram

/fo

rm o

f p

ub

lica

tio

nTyp

e o

f eva

luati

on

/d

esc

rip

tio

n o

f m

eth

od

olo

gy

Key

fin

din

gs

Rati

ng

of

evi

den

ce

36

. H

ealt

hN

gala

Nan

ga M

ai p

AR

en

T G

rou

p —

Cas

e Stu

dy (

Mur

i Mar

ri S

choo

l of Pu

blic

H

ealth

and

Com

mun

ity

Med

icin

e, U

NSW

an

d

The

pARen

T G

roup

Pro

gram

, Syd

ney

Chi

ldre

n’s

Hos

pita

l, n.

d.)

The

prog

ram

aim

s to

impr

ove

the

heal

th

of y

oung

Abo

rigi

nal p

aren

ts a

nd t

heir

child

ren

by e

ncou

ragi

ng p

aren

ts t

o ta

ke

part

in e

duca

tion

al a

ctiv

itie

s.

The

case

stu

dy/r

evie

w in

volv

ed a

pa

rtic

ipat

ory

mix

ed m

etho

ds a

ppro

ach

and

quan

tita

tive

res

earc

h pr

oces

ses

incl

udin

g an

alys

is o

f ro

utin

ely

colle

cted

pr

ogra

m d

ata

and

the

colle

ctio

n an

d an

alys

is o

f su

rvey

dat

a. Q

ualit

ativ

e re

sear

ch p

roce

sses

incl

uded

foc

us

grou

ps,

sem

i-st

ruct

ured

inte

rvie

ws

and

test

imon

ial d

ata.

Bas

elin

e qu

antita

tive

dat

a w

as c

ompa

red

with

qual

itat

ive

findi

ngs.

Find

ings

sug

gest

par

ents

per

ceiv

e th

e pr

ogra

m t

o be

val

uabl

e fo

r th

eir

own

pers

onal

dev

elop

men

t as

wel

l as

thei

r ch

ild’s

.

The

prog

ram

inco

rpor

ates

impo

rtan

t el

emen

ts o

f su

cces

sful

pro

gram

s fo

r Abo

rigi

nal a

nd T

orre

s

Str

ait

Isla

nder

you

ng p

eopl

e, s

uch

as

crea

ting

a s

afe

plac

e an

d pr

ovid

ing

oppo

rtun

itie

s fo

r pe

ople

to

deve

lop

thei

r ow

n st

reng

ths

and

skill

s et

c…

The

auth

ors

argu

e m

ore

fund

ing

wou

ld

enab

le t

he p

rogr

am t

o re

ach

its

pote

ntia

l an

d al

low

its

impa

cts

to b

e su

stai

ned.

N/

A —

how

ever

, th

e m

etho

dolo

gy u

sed

com

pare

s ve

ry fav

orab

ly w

ith

thos

e us

ed

in m

any

of t

he e

valu

atio

ns.

At

the

sam

e tim

e, a

dis

prop

ortion

ally

larg

e se

ctio

n of

the

rep

ort

is s

pent

jus

tify

ing

the

met

hodo

logy

use

d ra

ther

tha

n an

alys

ing

the

findi

ngs

from

the

res

earc

h.

Stu

dy w

as a

lso

limited

due

to

the

lack

of

rob

ustn

ess

in s

uch

a sm

all s

ampl

e si

ze,

and

that

the

rev

iew

onl

y as

sess

ed

a si

ngle

poi

nt in

tim

e. I

deal

ly for

an

eval

uation

, m

easu

rem

ents

wou

ld b

e ta

ken

prio

r to

sta

rtin

g (p

re-p

rogr

am

base

line)

and

the

n at

reg

ular

inte

rval

s af

ter

com

men

cing

in o

rder

to

dete

ct

chan

ges

in r

espo

nse.

37

. H

ealt

hA

bo

rig

inal M

ate

rnit

y G

rou

p P

ract

ice

Pro

gra

m (

als

o k

no

wn

as

Mo

ort

B

oo

dja

ri M

ia)

— (

Chr

istina

Ber

tilo

ne

and

Suz

anne

McE

voy,

201

5)

The

AM

GPP

pro

gram

pro

vide

s fr

ee

ante

nata

l and

pos

tnat

al c

linic

al c

are,

to

pre

gnan

t Abo

rigi

nal w

omen

. Ea

ch

clie

nt is

sup

port

ed b

y a

team

of he

alth

pr

ofes

sion

als

during

pre

gnan

cy a

nd f

or

four

wee

ks a

fter

the

y ha

ve g

iven

birth

. Sup

port

pro

vide

d in

clud

es c

linic

al c

are

and

cultur

al,

soci

al,

and

emot

iona

l car

e an

d su

ppor

t.

Non

-ran

dom

ised

inte

rven

tion

stu

dy

usin

g da

ta f

rom

the

Wes

tern

Aus

tral

ian

Mid

wiv

es N

otifi

cation

Sys

tem

. Reg

ress

ion

mod

els

wer

e us

ed t

o an

alys

e da

ta f

rom

34

3 w

omen

(w

ith

350

preg

nanc

ies)

who

pa

rtic

ipat

ed in

the

AM

GPP

and

gav

e bi

rth

betw

een

1 Ju

ly 2

011

and

31 D

ecem

ber

2012

. M

etho

dolo

gy in

clud

ed h

isto

rica

l an

d co

ntem

pora

ry c

ontr

ol g

roup

s of

pr

egna

nt A

borigi

nal w

omen

mat

ched

for

m

ater

nal a

ge.

Part

icip

atio

n in

the

AM

GPP

ass

ocia

ted

with

sign

ifica

ntly

impr

oved

neo

nata

l he

alth

out

com

es.

Bab

ies

born

to

AM

GPP

pa

rtic

ipan

ts w

ere

sign

ifica

ntly

less

lik

ely

to b

e bo

rn p

rete

rm 9

.1%

ver

sus

hist

oric

al c

ontr

ols

of 1

5.9%

.

Str

on

g

Reg

ress

ion

anal

ysis

and

mat

ched

con

trol

gr

oup.


Typ

e o

f p

rog

ram

Nam

e a

nd

desc

rip

tio

n o

f p

rog

ram

/fo

rm o

f p

ub

lica

tio

nTyp

e o

f eva

luati

on

/d

esc

rip

tio

n o

f m

eth

od

olo

gy

Key

fin

din

gs

Rati

ng

of

evi

den

ce

38

. H

ealt

hFa

mily

Vio

len

ce P

art

ners

hip

P

rog

ram

(FV

PP

)— T

wo

eval

uation

s,

(Cou

rage

Par

tner

s, 2

005)

and

eva

luat

ion

of F

aHCSIA

Fam

ily V

iole

nce

Prog

ram

s (D

epar

tmen

t of

Fin

ance

and

Der

egul

atio

n, O

ffic

e of

Eva

luat

ion

and

Aud

it (

Indi

geno

us

Prog

ram

s),

2007

).

Prog

ram

sup

port

s pr

actica

l, gr

assr

oots

in

itia

tive

s to

add

ress

fam

ily v

iole

nce,

se

xual

ass

ault a

nd c

hild

abu

se.

Firs

t ev

alua

tion

look

ed p

rim

arily

at

FVPP

, se

cond

eva

luat

ion

look

ed a

t FV

PP a

s pa

rt

of a

sui

te o

f Fa

mily

Vio

lenc

e Pr

ogra

ms

deliv

ered

by

FaH

CSIA

.

Firs

t ev

alua

tion

invo

lved

a d

ocum

ent

and

adm

inis

trat

ive

data

rev

iew

, ke

y st

akeh

olde

r in

terv

iew

s, t

en s

ite

visi

ts

and

inte

rvie

ws/

focu

s gr

oups

with

part

icip

ants

.

Sec

ond

eval

uation

invo

lved

a li

tera

ture

re

view

(m

eta

revi

ew o

f go

vern

men

t re

port

s),

cons

ulta

tion

with

key

stak

ehol

ders

, su

rvey

of

FaCSIA

pro

ject

m

anag

ers

and

prog

ram

dat

a an

alys

is.

Firs

t ev

alua

tion

fou

nd v

ery

few

pro

ject

s w

ere

dire

ctly

bas

ed o

n kn

owle

dge

of

docu

men

ted

evid

ence

app

roac

hes

to

deal

ing

with

fam

ily v

iole

nce.

Peo

ple

did

not

appe

ar t

o kn

ow a

bout

goo

d pr

actice

and

tha

t so

me

proj

ects

wer

e co

unte

rpro

duct

ive,

ie r

isks

of a

safe

ho

use

in c

omm

unitie

s th

at a

re n

ot

yet

read

y to

man

age

the

chal

leng

es

invo

lved

.

Sec

ond

eval

uation

fou

nd t

hat

whi

le

ther

e w

ere

qual

itat

ive

and

quan

tita

tive

pe

rfor

man

ce in

dica

tors

for

FVPP

pro

ject

s th

ere

was

no

iden

tifie

d pr

ogra

m-w

ide

perf

orm

ance

indi

cato

rs for

FVPP

.

Str

on

g

Toge

ther

bot

h ev

alua

tion

s pr

ovid

e a

stro

ng e

vide

nce

base

abo

ut t

he

effe

ctiv

enes

s of

the

FVPP

pro

gram

. In

par

ticu

lar

the

met

a-an

alys

is o

f do

cum

enta

tion

, an

d tr

iang

ulat

ion

of q

ualit

ativ

e an

d qu

antita

tive

dat

a pr

ovid

ed r

evea

ling

insi

ghts

abo

ut s

ome

of t

he is

sues

with

the

FVPP

pro

gram

.

Utilis

atio

n of

thi

s in

form

atio

n w

ould

hav

e en

able

d si

gnifi

cant

impr

ovem

ents

to

the

FVPP

pro

gram

s to

be

mad

e.

39

. H

ealt

hB

um

ps

to B

ab

es

an

d B

eyo

nd —

ev

alua

tion

rep

ort

(by

Avr

ille

Bur

row

s,

Bev

erle

y Alle

n an

d Sha

ron

Gor

ton,

D

ecem

ber

2014

).

This

pro

gram

is d

esig

ned

to p

rovi

de

supp

ort

for

youn

g an

d vu

lner

able

m

othe

rs d

urin

g pr

egna

ncy

and

the

first

18

mon

ths

afte

rwar

ds.

It w

as a

2 y

ear

prog

ram

.

Lite

ratu

re r

evie

w,

inte

rvie

ws

with

part

icip

ants

and

qua

ntitat

ive

data

(d

emog

raph

ics

and

surv

eys)

.

9 m

othe

rs p

artici

pate

d in

the

pro

gram

. At

the

tim

e of

writing

the

rep

ort,

a f

ew

of t

hem

had

not

yet

com

plet

ed t

he

prog

ram

as

thei

r ch

ildre

n w

ere

unde

r th

e 18

mon

th m

ilest

one.

The

repo

rt s

tate

d m

any

posi

tive

ou

tcom

es f

or t

he m

othe

rs a

nd t

heir

babi

es (

low

er r

ates

of de

pres

sion

, al

l ba

bies

res

idin

g w

ith

thei

r m

othe

rs,

etc.

).

How

ever

, w

itho

ut a

con

trol

gro

up it

is

not

poss

ible

to

attr

ibut

e th

ese

findi

ngs

to t

he p

rogr

am.

Weak

As

a re

sult o

f lo

w p

artici

pation

, fo

cus

grou

p in

terv

iew

s w

ere

drop

ped.

Fu

rthe

rmor

e, s

ome

inte

rvie

ws

wer

e do

ne b

y te

leph

one,

due

to

tim

e an

d/or

res

ourc

e co

nstr

aint

s on

beh

alf of

the

m

othe

rs.

Ther

e is

a d

istinc

t la

ck o

f ty

ing

the

posi

tive

out

com

es s

peci

fical

ly t

o th

e st

udy.

Ther

e w

as n

o co

ntro

l gro

up.

40

. H

ealt

hD

ead

ly T

eeth

— p

roje

ct r

epor

t, P

hase

1

(Win

da-M

ara

Abo

rigi

nal C

orp.

& H

ealth

Prom

otio

n U

nit

at P

ortlan

d D

istr

ict,

May

20

12).

Prog

ram

tak

es a

hol

istic

appr

oach

to

oral

hea

lth

by foc

usin

g on

ora

l hyg

iene

an

d he

alth

lite

racy

as

wel

l as

acce

ss t

o de

ntal

ser

vice

s, for

fam

ilies

with

child

ren

aged

0-5

yea

rs.

The

repo

rt d

oes

not

incl

ude

a sp

ecifi

c m

etho

dolo

gy s

ection

(!)

.

The

repo

rt li

sts

a lit

erat

ure

revi

ew,

pre

and

post

-pro

gram

tel

epho

ne s

urve

ys.

The

emph

asis

was

on

the

fram

ewor

k itse

lf an

d re

sour

ces

utili

sed

rath

er t

han

on m

easu

ring

out

com

es.

No

cont

rol g

roup

.

The

data

col

lect

ion

is v

ery

wea

k an

d th

e su

rvey

doe

s no

t as

k sp

ecifi

c qu

estion

s re

late

d to

the

pro

gram

act

ivitie

s an

d re

sour

ces

give

n to

par

tici

pating

clin

ics.

Th

e on

ly r

ecor

ded

outc

ome

was

tha

t th

e re

sour

ces

give

n w

ere

(1)

cultur

ally

ap

prop

riat

e an

d (2

) th

at t

hey

wou

ld

use

them

aga

in.

Very

litt

le in

the

rep

ort

abou

t ac

tual

pro

gram

act

ivitie

s.

Weak

Mai

nly

data

sho

win

g w

heth

er t

he d

enta

l cl

inic

s w

ere

utili

sing

the

pro

vide

d re

sour

ces

and

anec

dota

l evi

denc

e.

Witho

ut a

con

trol

gro

up,

it is

diff

icul

t to

as

sess

the

suc

cess

of th

e pr

ogra

m.


Typ

e o

f p

rog

ram

Nam

e a

nd

desc

rip

tio

n o

f p

rog

ram

/fo

rm o

f p

ub

lica

tio

nTyp

e o

f eva

luati

on

/d

esc

rip

tio

n o

f m

eth

od

olo

gy

Key

fin

din

gs

Rati

ng

of

evi

den

ce

41

. H

ealt

hD

ead

ly E

ars

Dead

ly K

ids

Dead

ly

Co

mm

un

itie

s —

eva

luat

ion

repo

rt (

J.

Dur

ham

, L.

Sch

uber

t &

L.

Vaug

hn,

Mar

ch

2015

).

The

prog

ram

aim

s to

red

uce

high

rat

es

of c

ondu

ctiv

e he

arin

g lo

ss fro

m m

iddl

e ea

r di

seas

e am

ong

Indi

geno

us c

hild

ren.

Lite

ratu

re r

evie

w,

stak

ehol

der

inte

rvie

ws

(the

con

tent

of

whi

ch w

as in

depe

nden

tly

anal

ysed

), c

omm

unity

visi

ts a

nd

quan

tita

tive

dat

a fr

om m

edic

al r

ecor

ds.

No

cont

rol g

roup

.

The

conc

lusi

on is

tha

t st

and-

alon

e ve

ntur

es,

such

as

this

pro

gram

, is

un

likel

y to

effec

tive

ly c

omba

t CSO

M

(chr

onic

ear

infla

mm

atio

n) a

s th

e ca

uses

and

det

erm

inan

ts a

re v

arie

d an

d in

terw

oven

with

man

y ot

her

lifes

tyle

as

pect

s. I

t al

so h

ighl

ight

ed t

he n

eed

for

a st

anda

rdis

ed m

etho

d of

col

lect

ing

data

ac

ross

clin

ics.

Weak

The

repo

rt s

ays

data

col

lect

ed fro

m

the

Dea

dly

Ears

EN

T cl

inic

s “i

ndic

ated

” th

at t

he p

reva

lenc

e of

CSO

M h

ad b

een

redu

ced,

tho

ugh

seni

or o

ffic

ials

fro

m

the

clin

ics

said

the

y ha

d no

dat

a on

the

pr

eval

ence

of th

e di

seas

e. D

ata

from

ot

her

clin

ics

wer

e no

t in

clud

ed =

no

cont

rol g

roup

.

An

accu

rate

ass

essm

ent

of t

he o

vera

ll re

duct

ion

in t

he in

cide

nce

of e

ar

infla

mm

atio

n w

as n

ot p

ossi

ble,

due

to

the

lack

of po

pula

tion

leve

l dat

a.

42

. H

ealt

hH

ealt

hy

Weig

ht

Pro

gra

m (

Livi

ng

S

tro

ng

Pro

gra

m)

— e

valu

atio

n re

port

(Q

ueen

slan

d H

ealth

Prom

otio

n U

nit,

20

05).

The

prog

ram

enc

oura

ges

part

icip

ants

to

hav

e a

heal

thy

lifes

tyle

thr

ough

goo

d nu

tritio

n an

d ph

ysic

al e

xerc

ise.

Qua

ntitat

ive

part

icip

ant

data

ana

lysi

s an

d in

-dep

th in

terv

iew

s w

ith

key

stak

ehol

ders

.

Ther

e w

ere

base

line

mea

sure

men

ts

avai

labl

e ob

tain

ed p

rior

to

the

prog

ram

.

Post

-pro

gram

out

com

es w

ere

gene

rally

po

sitive

as

show

n th

roug

h th

e in

divi

dual

pa

rtic

ipan

t sc

reen

ing

data

and

qu

alitat

ive

data

. M

ost

part

icip

ants

wer

e ‘a

t risk

’ at

the

star

t of

the

HW

P; m

ost

redu

ced

thei

r bo

dy w

eigh

t an

d w

aist

/hi

p ci

rcum

fere

nces

; an

d sh

owed

som

e m

odes

t po

sitive

cha

nge

in t

erm

s of

the

ir

lifes

tyle

beh

avio

urs.

Weak

Des

pite

evi

denc

e su

gges

ting

tha

t a

tota

l of 43

2 pe

ople

par

tici

pate

d, ful

l sc

reen

ing

data

was

onl

y av

aila

ble

for

34 in

divi

dual

s. F

urth

erm

ore,

dat

a fr

om t

his

part

icip

ant

popu

lation

may

no

t be

rep

rese

ntat

ive

of p

artici

pant

s w

ho a

tten

ded

scre

enin

g se

ssio

ns o

r w

orks

hops

dur

ing

the

12-m

onth

dat

a co

llect

ion

period

for

thi

s ev

alua

tion

(!)

.

Com

para

tive

effec

tive

ness

(as

mea

sure

d ag

ains

t si

mila

r pr

ogra

ms)

was

not

in

vest

igat

ed.

The

post

-pro

gram

dat

a w

as c

olle

cted

th

roug

h qu

estion

naires

. Sel

f-re

port

ing

in r

elat

ion

to h

ealthi

er li

fest

yle

choi

ces

are

alw

ays

noto

riou

sly

over

-sta

ted,

and

th

ere

was

no

conf

irm

atio

n of

ans

wer

s by

th

e pr

ogra

m w

orke

rs.


Typ

e o

f p

rog

ram

Nam

e a

nd

desc

rip

tio

n o

f p

rog

ram

/fo

rm o

f p

ub

lica

tio

nTyp

e o

f eva

luati

on

/d

esc

rip

tio

n o

f m

eth

od

olo

gy

Key

fin

din

gs

Rati

ng

of

evi

den

ce

43

. H

ealt

hFa

mily

Vio

len

ce R

eg

ion

al A

ctiv

itie

s P

rog

ram

— fin

al e

valu

atio

n re

port

(by

Cou

rage

Par

tner

s, M

orga

n D

isne

y &

Ass

ocia

tes,

Suc

cess

Wor

ks,

May

200

5).

This

pro

gram

pro

vide

s su

ppor

t to

gr

assr

oots

pro

ject

s ai

med

at

addr

essi

ng

fam

ily v

iole

nce,

sex

ual a

ssau

lt a

nd c

hild

ab

use.

The

purp

ose

of t

he r

epor

t w

as t

o as

sess

th

e ef

fect

iven

ess

of t

he p

rogr

am.

Qua

ntitat

ive

(dat

a an

alys

is o

f ho

w t

he

prog

ram

had

run

up

until 2

005)

and

qu

alitat

ive

data

(si

te v

isits,

inte

rvie

ws)

, co

nduc

ted

in 3

pha

ses.

The

conc

lusi

on o

f th

e re

port

is t

hat

mor

e fu

ndin

g is

nee

ded

and

for

mor

e in

tegr

atio

n of

pro

gram

s ac

ross

reg

ions

.

Weak

The

repo

rt d

oes

not

offe

r m

uch

data

, it

only

sum

mar

ises

wha

t ha

s w

orke

d to

da

te a

nd m

akes

sug

gest

ions

as

to h

ow

the

prog

ram

sho

uld

be c

arried

out

in t

he

futu

re (

serv

ice

deliv

ery

mod

els,

fun

ding

m

odel

s, e

tc.)

.

44

. H

ealt

hA

bo

rig

inal To

bacc

o C

on

tro

l P

roje

ct

— Y

arn

ing

It

Up

, D

on

’t S

mo

ke I

t U

p —

fin

al e

valu

atio

n re

port

(Sou

th

Met

ropo

litan

Hea

lth

Ser

vice

, Ju

ne 2

014)

.

The

proj

ect

invo

lves

wor

ksho

ps t

o he

lp

peop

le s

top

smok

ing

and

info

rmat

ion

sess

ions

to

thos

e w

orki

ng in

the

tob

acco

ce

ssat

ion

field

.

Lite

ratu

re r

evie

w,

proj

ect

repo

rtin

g (a

t 6

mon

th in

terv

als)

, st

akeh

olde

r in

terv

iew

s an

d su

rvey

s.

No

cont

rol g

roup

.

The

evid

ence

for

pro

gram

effec

tive

ness

w

as la

rgel

y de

script

ive

with

only

a

hand

ful o

f in

terv

ention

trial

s. T

his

was

in

con

tras

t to

the

est

ablis

hed

evid

ence

of

eff

ective

tob

acco

con

trol

pro

gram

s am

ongs

t th

e ge

nera

l com

mun

ity.

Weak

Des

pite

sho

win

g at

tend

ance

s at

w

orks

hops

of al

mos

t 5,

500

peop

le,

data

ga

ther

ed (

ques

tion

naires

and

sur

veys

) sh

ows

only

num

bers

of 20

-30.

The

sm

all

sam

ple

pool

s pr

ohib

it e

ffec

tive

ana

lysi

s.

The

stat

ed o

bjec

tive

of th

e pr

ojec

t w

as

not

nece

ssar

ily t

o re

duce

the

num

ber

of s

mok

ers,

but

rat

her

to r

aise

gen

eral

aw

aren

ess

of t

he n

egat

ive

effe

cts.

In

this

sen

se,

the

eval

uation

sho

ws

that

th

e pr

ogra

m w

as a

suc

cess

, as

the

pos

t-w

orks

hop

surv

eys

show

an

incr

ease

in

awar

enes

s. H

owev

er,

?

45

. H

ealt

hH

ealin

g p

rog

ram

: h

ealt

hy

eati

ng

act

ivit

ies

(an

d)

life

styl

es

(fo

r)

Ind

igen

ou

s g

rou

ps

— a

n ev

alua

tion

(Q

LD H

ealth

for

the

Hea

lthy

Wei

ght

Prog

ram

(H

WP,

200

4).

The

HEA

LInG

pro

gram

is a

10-

wee

k he

alth

y ea

ting

and

life

styl

e pr

ogra

m,

whi

ch a

ims

to p

rovi

de p

ract

ical

and

re

alis

tic

advi

ce t

o pa

rtic

ipan

ts.

Eval

uation

met

hodo

logy

incl

uded

pre

an

d po

st p

rogr

am e

valu

atio

ns,

thou

gh

the

num

ber

of p

artici

pant

s w

ho t

ook

part

in

inte

rvie

ws

was

sm

all,

8 in

the

first

ro

und

and

11 in

the

sec

ond.

Ove

rall,

the

par

tici

pant

s lik

ed t

he

prog

ram

and

rep

orting

lear

ning

mor

e ab

out

nutr

itio

n an

d th

at t

heir p

hysi

cal

activi

ties

had

incr

ease

d. H

owev

er,

they

rep

orte

d ex

perien

cing

diff

icul

ty

mai

ntai

ning

a h

ealthi

er li

fest

yle

due

to

issu

es in

the

ir li

ves.

Weak

Altho

ugh

ther

e w

ere

pre

and

post

in

terv

iew

s w

ith

part

icip

ants

the

sam

ple

size

was

sm

all a

nd t

he e

valu

atio

n pr

imar

ily r

elie

d on

qua

litat

ive.

For

ex

ampl

e, t

here

was

no

evid

ence

th

at p

artici

pant

’s h

ealth

stat

istics

w

ere

reco

rded

to

verify

any

rep

orte

d im

prov

emen

ts in

phy

sica

l act

ivity.


Typ

e o

f p

rog

ram

Nam

e a

nd

desc

rip

tio

n o

f p

rog

ram

/fo

rm o

f p

ub

lica

tio

nTyp

e o

f eva

luati

on

/d

esc

rip

tio

n o

f m

eth

od

olo

gy

Key

fin

din

gs

Rati

ng

of

evi

den

ce

46

. H

ealt

hC

en

tral A

ust

ralian

Yo

uth

Lin

k U

p

Serv

ice (

CA

YLU

S)

— s

take

hold

er

feed

back

rep

ort

(CAY

LUS,

2013

).

This

pro

gram

’s m

issi

on is

to

supp

ort

com

mun

ity

initia

tive

s th

at im

prov

e qu

ality

of li

fe,

and

addr

ess

alco

hol a

nd

othe

r dr

ug u

se is

sues

affec

ting

you

ng

peop

le.

Tele

phon

e in

terv

iew

s w

ith

39

stak

ehol

ders

(ac

ross

7 r

emot

e co

mm

unitie

s in

NT)

; yo

uth

wor

kers

, re

gion

al s

taff,

etc

. N

o pa

rtic

ipan

ts

inte

rvie

wed

. Th

e qu

estion

s re

volv

ed

arou

nd t

he c

ondi

tion

s be

fore

a p

rogr

am

arrive

d an

d w

hat

kind

of

serv

ices

th

e pr

ogra

m d

eliv

ered

. N

o qu

estion

s or

mea

sure

men

t re

gard

ing

the

effe

ctiv

enes

s.

No

cont

rol g

roup

.

The

repo

rt g

ener

ally

sup

port

s th

e se

rvic

es p

rovi

ded

by C

AYLU

S.

A la

rge

maj

ority

of r

espo

nden

ts fel

t th

at t

he

prov

isio

n of

you

th p

rogr

ams

was

a k

ey

prog

ram

for

the

ir c

omm

unitie

s be

caus

e it p

reve

nted

crim

e, a

nd g

ave

youn

g pe

ople

mor

e po

sitive

way

s of

spe

ndin

g th

eir

tim

e.

Weak

The

stak

ehol

ders

tha

t w

ere

inte

rvie

wed

w

ere

all p

rofe

ssio

nals

wor

king

with

the

deliv

ery

of C

AYLU

S.

It w

as a

lso

pure

ly

qual

itat

ive

(and

sub

ject

ive)

dat

a. T

he

stat

emen

ts m

ade

wer

e no

t ba

cked

up

by

any

quan

tita

tive

sta

tist

ics.

47

. H

ealt

hA

bo

rig

inal P

eri

nata

l S

erv

ice

Exp

an

sio

n —

fin

al E

valu

atio

n Rep

ort

(WA P

erin

atal

Mat

erna

l Hea

lth

Uni

t,

Febr

uary

200

8).

The

proj

ect

aim

s to

dev

elop

and

trial

a

‘cul

tura

lly a

ppro

pria

te’ p

erin

atal

men

tal

heal

th s

ervi

ce fra

mew

ork.

Bot

h qu

alitat

ive

and

quan

tita

tive

m

etho

ds u

tilis

ed.

A b

asel

ine

repo

rt w

as

deve

lope

d in

200

8.

5 KPI

’s w

ere

iden

tifie

d an

d re

sults

mea

sure

d fo

r th

ese.

The

repo

rt li

sts

4 ou

t of

5 K

PIs

as

havi

ng b

een

fully

or

part

ially

met

. Th

e KPI

s m

ainl

y re

volv

e ar

ound

incr

easi

ng

awar

enes

s fo

r th

e he

alth

uni

t’s s

ervi

ces.

Sta

ff t

urno

ver

during

dat

a co

llect

ion

poin

ts h

ampe

red

prog

ress

.

Weak

The

repo

rt s

tate

s th

at d

ata

colle

ctio

n pr

oved

to

be a

cha

lleng

e. W

ith

sam

plin

g te

chni

ques

and

siz

es in

adeq

uate

for

co

nclu

sive

res

ults

(i.e

. va

lidity,

rel

iabi

lity,

tr

ansf

erab

ility

) an

d se

rvic

e pr

ovid

er d

ata

bein

g in

itia

lly s

carc

e, fin

ding

alter

native

w

ays

of m

easu

ring

and

rep

orting

ou

tcom

es o

f th

e H

PHM

Ser

vice

bec

ame

cruc

ial.

The

rese

arch

ers

had

to t

hink

ou

tsid

e th

e co

nfin

es o

f th

eir

empi

rici

st

trai

ning

and

em

ploy

met

hods

tha

t w

ere

cultur

ally

sen

sitive

and

fle

xibl

e.

Out

of ov

er 1

,000

birth

s re

gist

ered

, on

ly 5

mot

hers

com

plet

ed a

pos

t-qu

estion

naire.

48

. H

ealt

hA

keyu

lerr

e H

ealin

g C

en

ter

— fin

al

eval

uation

rep

ort

(Cha

rles

Dar

win

U

nive

rsity,

Apr

il 20

10).

The

Ang

kwer

re-i

wem

e (t

radi

tion

al

heal

ing)

pro

ject

hel

ps E

lder

s to

pra

ctis

e an

d pa

ss o

n tr

aditio

nal A

rrer

nte

heal

ing

in t

heir c

omm

unity.

Lite

ratu

re r

evie

w.

Qua

litat

ive:

inte

rvie

ws,

pho

togr

aphs

and

vi

deos

.

No

cont

rol g

roup

.

The

repo

rt is

mai

nly

desc

ript

ive

of

‘hea

ling’

and

‘hea

ling

proc

esse

s’ in

the

in

dige

nous

sen

se.

The

only

act

ual f

indi

ngs

are

anec

dota

l qu

otes

fro

m c

ompl

eted

inte

rvie

ws.

Weak

Non

e of

the

origi

nal s

ourc

es (

answ

ered

in

terv

iew

s, e

tc.)

are

att

ache

d to

the

re

port

. Th

e re

port

is m

ainl

y co

mpr

ised

of

des

crip

tion

s of

the

var

ious

opt

ions

th

at t

he H

ealin

g Cen

ter

offe

rs (

bush

tr

ips,

coo

king

cla

sses

, et

c.)

but

it fai

ls t

o ex

plai

n re

sults.


Typ

e o

f p

rog

ram

Nam

e a

nd

desc

rip

tio

n o

f p

rog

ram

/fo

rm o

f p

ub

lica

tio

nTyp

e o

f eva

luati

on

/d

esc

rip

tio

n o

f m

eth

od

olo

gy

Key

fin

din

gs

Rati

ng

of

evi

den

ce

49

. H

ealt

hA

lco

ho

l an

d O

ther

Dru

gs

—

Ind

igen

ou

s C

om

mu

nit

ies

Pro

ject

—

fin

al e

valu

atio

n re

port

(Cen

ter

for

Rem

ote

Hea

lth,

May

201

1).

The

proj

ect

seek

s to

red

uce

subs

tanc

e us

e an

d as

soci

ated

har

ms,

am

ong

Indi

geno

us c

omm

unitie

s fr

om t

he T

op

End

regi

on o

f th

e N

orth

ern

Terr

itor

y.

Qua

litat

ive:

sta

keho

lder

inte

rvie

ws.

Dat

a fr

om c

omm

unic

atio

n lo

gs a

nd r

epor

ts.

No

cont

rol g

roup

.

The

focu

s of

the

rep

ort

is la

rgel

y on

th

e “b

uild

ing

capa

city

” an

d “i

ncre

asin

g aw

aren

ess”

. U

nfor

tuna

tely

the

re is

no

evid

ence

of

actu

al e

ffec

tive

ness

in t

erm

s of

a d

rop

in s

ubst

ance

abu

se.

Weak

Altho

ugh

the

repo

rt is

gen

eral

ly o

f hi

gh

qual

ity,

the

fac

t th

at t

he e

valu

atio

n fr

amew

ork

lists

the

1st o

bjec

tive

(r

educ

ing

the

num

ber

of h

arm

s) a

s “n

ot

to b

e m

easu

red”

rai

ses

seriou

s qu

estion

s ab

out

whe

ther

the

pro

gram

was

effec

tive

or

not

.

50

. H

ealt

hJa

lari

s K

ids

Futu

re C

lub —

eva

luat

ion

repo

rt (

M.

Hav

iland

, Sid

e by

Sid

e Con

sultin

g, 2

010)

.

The

proj

ect

aim

s to

impr

ove

the

educ

atio

nal o

utco

mes

of In

dige

nous

ch

ildre

n by

hel

ping

the

m t

o pr

epar

e fo

r m

ains

trea

m s

choo

ling.

Qua

ntitat

ive:

att

enda

nce

stat

istics

(t

houg

h se

lf-re

port

ed b

y th

e ki

ds).

The

anec

dota

l evi

denc

e m

akes

up

the

bulk

of

the

findi

ngs,

with

som

e of

it

lean

ing

tow

ards

sup

port

for

the

pro

gram

(f

rom

par

ents

and

oth

er c

arer

s).

Ther

e w

as a

1%

incr

ease

in s

choo

l at

tend

ance

for

the

kid

s in

volv

ed in

the

pr

ogra

m,

but

this

is b

ased

on

a sm

all

sam

ple

size

and

was

sel

f-re

port

ed.

Weak

It is

hig

hlig

hted

tha

t on

e of

the

cor

e da

ta s

ourc

es w

as a

band

oned

in t

he

2nd

year

due

to

lack

of st

aff an

d pa

rtic

ipat

ion.

Tha

t m

eant

tha

t in

divi

dual

ch

ild o

utco

mes

cou

ld n

ot b

e m

easu

red.

51

. H

ealt

hN

ati

on

al Em

po

werm

en

t P

rog

ram

—

nation

al s

umm

ary

repo

rt (

J. M

illro

y et

al

., 2

014)

.

The

proj

ect

aim

s to

red

uce

the

rate

of

sui

cide

in A

borigi

nal a

nd T

orre

s Str

ait

Isla

nder

com

mun

itie

s th

roug

h th

e pr

omot

ion

of s

ocia

l and

em

otio

nal

wel

lbei

ng in

itia

tive

s.

Qua

litat

ive:

foc

us g

roup

, co

mm

unity

feed

back

, st

akeh

olde

r in

terv

iew

s.

No

cont

rol g

roup

.

The

repo

rt c

over

s “S

tage

1”

of t

he

NEP

, w

hich

foc

used

on

iden

tify

ing

the

caus

es f

or t

he p

oor

cond

itio

n of

the

pa

rtic

ipat

ing

com

mun

itie

s. A

ltho

ugh

the

docu

men

tation

for

tha

t is

ver

y go

od,

the

repo

rt d

oes

not

prov

ide

any

eval

uation

of

how

the

1st s

tage

impr

oved

the

co

mm

unitie

s.

Weak

The

repo

rt m

ainl

y co

ntai

ns d

escr

iption

s of

the

pas

t hi

stor

y an

d cu

rren

t co

nditio

ns

of t

he I

ndig

enou

s co

mm

unitie

s th

at

part

icip

ated

in t

he p

rogr

am.

It a

lso

cont

ains

man

y re

com

men

dation

s on

w

here

to

go fro

m h

ere.

How

ever

, th

ere

are

very

few

men

tion

s on

the

out

com

e of

the

pro

gram

.

52

. H

ou

sin

g

Nyo

on

gar

Ou

treach

Patr

ol S

erv

ice

— K

eep

ing

Peo

ple

Safe

— e

valu

atio

n re

port

(Jo

hn S

coug

all C

onsu

ltin

g Ser

vice

s, M

arch

201

2).

This

pro

gram

pro

vide

s so

cial

and

wel

fare

se

rvic

es t

o Abo

rigi

nal p

eopl

e w

ho a

re

hom

eles

s an

d or

affec

ted

by a

lcoh

ol o

r dr

ugs.

Focu

s gr

oup

mee

ting

s, s

urve

ys,

docu

men

t re

view

s, li

tera

ture

rev

iew

, a

few

mee

ting

s w

ith

key

stak

ehol

ders

. M

ainl

y qu

alitat

ive.

The

repo

rt s

tate

s th

at t

he a

vaila

ble

evid

ence

sug

gest

s th

at t

he p

rogr

am is

ha

ving

a p

ositiv

e ef

fect

on

indi

geno

us

yout

h an

d st

reet

crim

e. T

he m

ain

conc

lusi

on d

raw

n is

tha

t th

e or

gani

sation

ne

eds

mor

e fu

ndin

g in

ord

er t

o ex

pand

its

adm

inis

trat

ive

capa

bilit

ies.

The

surv

eys

wer

e ge

ared

tow

ards

st

akeh

olde

r’s

perc

eption

of th

e or

gani

sation

/pro

gram

.

Mo

dera

te

The

repo

rt s

tres

ses

that

an

evid

ence

-ba

sed

appr

oach

was

utilis

ed.

Prog

ram

pr

ovid

ers

keep

a d

atab

ase

that

rec

ords

th

e in

stan

ces.

Thi

s da

ta is

incl

uded

th

e re

port

and

is c

lear

ly la

belle

d an

d an

alys

ed.


Typ

e o

f p

rog

ram

Nam

e a

nd

desc

rip

tio

n o

f p

rog

ram

/fo

rm o

f p

ub

lica

tio

nTyp

e o

f eva

luati

on

/d

esc

rip

tio

n o

f m

eth

od

olo

gy

Key

fin

din

gs

Rati

ng

of

evi

den

ce

53

. H

ou

sin

gIn

dig

en

ou

s C

om

mu

nit

y V

olu

nte

er

Pro

gra

m —

eco

nom

ic a

nd s

ocia

l im

pact

/ca

se s

tudi

es (

KPM

G,

2015

).

ICV is

a r

egis

tere

d ch

arity

and

non-

prof

it

com

mun

ity

deve

lopm

ent

orga

nisa

tion

th

at m

atch

es a

vol

unte

er’s

exp

erie

nce

and

skill

s w

ith

differ

ent

Indi

geno

us

com

mun

itie

s ne

eds

to h

elp

addr

ess

Indi

geno

us d

isad

vant

age.

Cas

e st

udy

(but

act

ually

a s

ocia

l and

ec

onom

ic im

pact

ass

essm

ent)

.

Ass

essm

ent

of a

ctiv

itie

s in

tw

o co

mm

unitie

s in

volv

ed s

take

hold

er

cons

ulta

tion

s an

d do

cum

ent

and

data

an

alys

is,

incl

udin

g as

sess

ing

the

impa

cts

of t

he a

ctiv

itie

s in

eco

nom

ic t

erm

s.

Evid

ence

ICV w

as in

vite

d in

to

com

mun

itie

s an

d in

volv

ed in

dis

cret

e,

wel

l def

ined

pro

ject

s, a

nd t

hat

volu

ntee

rs w

ere

prov

idin

g a

posi

tive

im

pact

and

bui

ldin

g on

exi

stin

g w

ork

that

had

bee

n do

ne in

the

com

mun

ity.

Ev

iden

ce t

hat

ICV h

ad a

lso

deve

lope

d po

sitive

par

tner

ship

s w

ith

othe

r or

gani

sation

s an

d w

ere

colla

bora

ting

w

ith

them

on

activi

ties

.

Mo

dera

te

Stu

dy in

volv

ed t

rian

gula

tion

of da

ta

from

mul

tipl

e so

urce

s, in

clud

ing

anal

ysis

of

eco

nom

ic d

ata.

How

ever

stu

dy o

nly

look

ed a

t tw

o co

mm

unitie

s so

diff

icul

t to

ext

rapo

late

abo

ut o

vera

ll pr

ogra

m

impa

ct.

54

. H

ou

sin

gIn

dig

en

ou

s H

om

e O

wn

ers

hip

P

rog

ram

— a

udit r

epor

t (A

ustr

alia

n N

atio

nal A

udit O

ffic

e, D

ecem

ber

2015

).

The

IHO

P is

adm

inis

tere

d by

Ind

igen

ous

Bus

ines

s Aus

tral

ia a

nd a

ims

to h

elp

Indi

geno

us A

ustr

alia

ns in

to h

ome

owne

rshi

p.

The

obje

ctiv

e of

the

aud

it w

as t

o as

sess

th

e ef

fect

iven

ess

of I

BA’

s m

anag

emen

t an

d im

plem

enta

tion

of

the

IHO

P

The

AN

AO

fou

nd t

hat

IBA’

s m

anag

emen

t of

the

pro

gram

had

bee

n in

effic

ient

an

d le

ndin

g di

d no

t fu

lly a

lign

with

the

prog

ram

obj

ective

s fo

r w

hich

IBA

is f

unde

d. I

n pa

rtic

ular

loan

s w

ere

prov

ided

to

peop

le w

ho w

ould

hav

e be

en

able

to

acce

ss lo

ans

from

mai

nstr

eam

le

nder

s.

N/

A

55

. H

ou

sin

g

Rem

ote

are

as

ess

en

tial se

rvic

es

pro

gra

m —

aud

it

(Offic

e of

the

Aud

itor

Gen

eral

Wes

tern

Aus

tral

ia,

2015

)

Aud

it a

sses

sed

how

wel

l the

WA

Dep

artm

ent

of H

ousi

ng d

eliv

ers

pow

er,

wat

er a

nd w

aste

wat

er r

epai

r an

d m

aint

enan

ce s

ervi

ces

to s

elec

ted

rem

ote

Abo

rigi

nal c

omm

unitie

s th

roug

h th

e Rem

ote

Are

a Es

sent

ial S

ervi

ces

Prog

ram

. Th

e sc

ope

of t

his

audi

t di

d no

t in

clud

e th

e pr

ovis

ion

of a

ll se

rvic

es t

o al

l rem

ote

com

mun

itie

s or

the

ir s

usta

inab

ility

.

Qua

lity

of d

rink

ing

wat

er o

ften

fal

ls

shor

t of

Aus

tral

ian

stan

dard

s. T

esting

of

was

tew

ater

sys

tem

s w

as ir

regu

lar

or

inco

mpl

ete

betw

een

Janu

ary

2012

and

201

4, s

o H

ousi

ng

coul

d no

t be

sur

e if

they

wer

e w

orki

ng

effe

ctiv

ely.

Hou

sing

’s c

urre

nt a

rran

gem

ents

fo

r m

anag

ing

the

Prog

ram

lim

it it

s ef

fect

iven

ess

and

effic

ienc

y.

The

criter

ia t

o de

term

ine

elig

ibili

ty for

th

e Pr

ogra

m h

ave

not

been

app

lied

sinc

e 20

08 a

nd a

s a

resu

lt H

ousi

ng d

id n

ot

know

if t

he r

ight

com

mun

itie

s w

ere

in

the

Prog

ram

.

N/

A


Typ

e o

f p

rog

ram

Nam

e a

nd

desc

rip

tio

n o

f p

rog

ram

/fo

rm o

f p

ub

lica

tio

nTyp

e o

f eva

luati

on

/d

esc

rip

tio

n o

f m

eth

od

olo

gy

Key

fin

din

gs

Rati

ng

of

evi

den

ce

56

. H

ou

sin

g

Fixin

g h

ou

ses

for

bett

er

healt

h

pro

ject

s (H

ou

sin

g f

or

healt

h)

— a

udit

(Aus

tral

ian

Nat

iona

l Aud

it O

ffic

e, 2

010)

The

prog

ram

is t

arge

ted

at in

divi

dual

ho

useh

olds

and

aim

s to

pro

mot

e a

heal

thie

r liv

ing

envi

ronm

ent

by m

akin

g re

pairs

to h

omes

.

The

audi

t’s o

bjec

tive

was

to

asse

ss t

he

effe

ctiv

enes

s of

FaH

CSIA

’s m

anag

emen

t fo

r th

e Fi

xing

Hou

sing

for

Bet

ter

Hea

lth

Prog

ram

sin

ce 2

005.

Hea

lth

rela

ted

impr

ovem

ents

wer

e m

ade

to o

ver

2000

hou

ses

in 3

4 co

mm

unitie

s be

twee

n Ju

ly 2

005

and

June

200

9. H

owev

er,

FaH

CSIA

’s p

rogr

am

man

agem

ent

arra

ngem

ents

did

not

cat

er

for

the

colle

ctio

n of

dat

a th

at p

rovi

ded

a m

eans

of

linki

ng im

prov

emen

ts

mad

e to

hou

ses

in c

omm

unitie

s w

ith

impr

ovem

ents

in h

ealth

indi

cato

rs in

th

ose

sam

e co

mm

unitie

s.

N/

A

57

. Jo

bs

an

d

eco

no

my

Ab

ori

gin

al M

en

tal H

ealt

h W

ork

er

Pro

gra

m —

fin

al e

valu

atio

n re

port

(G

. Rob

inso

n &

A.

Har

ris,

Cha

rles

Dar

win

U

nive

rsity,

200

4?)

The

prog

ram

pro

mot

es a

nd s

uppo

rts

the

role

of AM

HW

s as

men

tal h

ealth

wor

kers

in

rem

ote

com

mun

itie

s.

Sta

keho

lder

inte

rvie

ws,

site

visi

ts

(pat

ient

file

aud

its,

pol

icie

s an

d pr

oced

ure

chec

ks),

hea

lth

cent

re d

ata

eval

uation

.

The

data

is c

lear

ly la

belle

d an

d ex

plai

ned.

The

repo

rt h

ighl

ight

s th

e di

ffer

ence

in

repo

rtin

g an

d da

ta c

olle

ctio

n st

anda

rds

acro

ss m

enta

l hea

lth

cam

puse

s as

a k

ey

prob

lem

for

the

eva

luat

ion.

It

also

cites

th

e hi

gh t

urno

ver

of s

taff a

nd in

secu

re

fund

ing.

The

focu

s w

as o

n de

velo

ping

a

fram

ewor

k fo

r AM

H w

orke

rs.

Mo

dera

te

The

eval

uation

rep

ort

is s

olid

, w

ith

very

th

orou

gh a

naly

sis

of b

oth

the

qual

itat

ive

and

quan

tita

tive

dat

a. T

here

are

man

y su

gges

tion

s fo

r im

prov

emen

t ba

sed

on t

he fin

ding

s. H

owev

er,

the

repo

rt

also

con

clud

es t

hat

the

emph

asis

of

the

repo

rt is

on

prog

ram

des

ign

and

serv

ice

deliv

ery,

rat

her

than

mea

suring

ef

fect

iven

ess

and

outc

omes

.

58

. Jo

bs

an

d

eco

no

my

Th

e W

ork

ing

on

Co

un

try

(Wo

C)

eval

uation

(U

rbis

, 20

12).

The

WoC

(na

tion

al)

prog

ram

pro

vide

s em

ploy

men

t an

d tr

aini

ng t

o In

dige

nous

Aus

tral

ians

livi

ng in

reg

iona

l and

re

mot

e Aus

tral

ia t

o w

ork

on I

ndig

enou

s Pr

otec

ted

Are

as (

IPA)

and

man

age

and

care

for

the

ir ‘c

ount

ry’.

Met

hodo

logy

invo

lved

18

case

stu

dies

, a

revi

ew o

f pr

ogra

m a

nd p

olic

y da

ta a

nd

docu

men

tation

and

con

sultat

ions

with

key

depa

rtm

enta

l per

sonn

el.

The

eval

uation

fou

nd t

hat

the

prog

ram

ha

d a

rang

e of

eco

nom

ic,

soci

al,

cultur

al a

nd e

nviron

men

tal b

enef

its.

Ran

gers

saw

the

job

s as

‘rea

l job

s’ t

hat

prov

ided

bet

ter

inco

me

and

cond

itio

ns,

mor

e in

tere

stin

g w

ork

and

ongo

ing

empl

oym

ent,

com

pare

d to

the

CD

EP

alte

rnat

ive.

Mo

dera

te

A r

ange

of qu

alitat

ive

and

quan

tita

tive

da

ta s

ourc

es w

ere

utili

zed

to e

nabl

e tr

iang

ulat

ion

of d

ata.


Typ

e o

f p

rog

ram

Nam

e a

nd

desc

rip

tio

n o

f p

rog

ram

/fo

rm o

f p

ub

lica

tio

nTyp

e o

f eva

luati

on

/d

esc

rip

tio

n o

f m

eth

od

olo

gy

Key

fin

din

gs

Rati

ng

of

evi

den

ce

59

. Jo

bs

an

d

eco

no

my

Cap

e Y

ork

Welf

are

Refo

rm —

ev

alua

tion

rep

ort

(Per

form

ance

and

Ev

alua

tion

Bra

nch

FaH

CSIA

, an

d in

depe

nden

t co

nsul

tant

s

from

the

Soc

ial P

olic

y Res

earc

h Cen

tre

at t

he U

nive

rsity

of N

ew S

outh

Wal

es

and

acad

emic

s fr

om t

he Aus

tral

ian

Nat

iona

l Uni

vers

ity,

201

2)

The

prog

ram

aim

s to

res

tore

soc

ial

norm

s an

d pr

omot

e co

mm

unity

lead

ersh

ip a

nd e

cono

mic

dev

elop

men

t in

fiv

e w

elfa

re r

efor

m c

omm

unitie

s:

Aur

ukun

, Coe

n, H

ope

Vale

, D

oom

adge

e an

d M

ossm

an G

orge

.

An

impl

emen

tation

eva

luat

ion

and

an

outc

omes

eva

luat

ion.

Met

hods

incl

uded

st

akeh

olde

r in

terv

iew

s in

the

fou

r co

mm

unitie

s, d

ocum

ent

anal

ysis

and

ex

amin

atio

n of

adm

inis

trat

ive

data

re

lating

to

educ

atio

n, c

hild

pro

tect

ion,

ho

usin

g, c

rim

e an

d em

ploy

men

t, a

s w

ell

as t

he F

RC’s

dat

abas

e.

The

eval

uation

did

not

incl

ude

an

econ

omic

eva

luat

ion,

altho

ugh

the

wel

fare

ref

orm

pro

gram

des

ign

repo

rt

reco

mm

ende

d th

at ‘a

n ec

onom

ic

eval

uation

sho

uld

asse

ss t

he c

ost

effe

ctiv

enes

s of

the

Inte

rven

tion

s.’

The

findi

ngs

of t

he e

valu

atio

n w

ere

very

diff

icul

t to

inte

rpre

t an

d at

trib

ute.

W

hile

the

qua

ntitat

ive

anal

ysis

sho

wed

im

prov

emen

ts o

n a

num

ber

of d

iffer

ent

dim

ensi

ons,

incl

udin

g sc

hool

att

enda

nce

and

achi

evem

ents

and

red

uction

s in

cr

ime,

the

re w

as n

o pr

ogre

ss in

oth

er

area

s.

Mo

dera

te

Whi

le t

here

was

a p

erce

ptio

n th

at t

hing

s w

ere

impr

ovin

g in

the

com

mun

itie

s, a

s ev

iden

ced

by a

soc

ial s

urve

y, t

he s

urve

y w

as n

ot c

ondu

cted

in o

ther

com

mun

itie

s,

so it

was

not

cle

ar w

heth

er t

his

was

par

t of

an

over

all t

rend

. M

any

of t

he b

enef

its

to t

he c

omm

unitie

s ap

pear

ed t

o ha

ve

aris

en fro

m t

he A

lcoh

ol M

anag

emen

t Pl

ans,

whi

ch,

alth

ough

initia

ted

by t

he

Cap

e Yo

rk I

nstitu

te w

ere

not

part

of th

e re

form

s, a

nd p

rece

ded

them

by

a co

uple

of

yea

rs.

60

. Jo

bs

an

d

eco

no

my

Jaw

un

—

exe

cutive

sum

mar

y of

rep

ort

(KPM

G,

7 N

ovem

ber

2015

).

Jaw

un is

a n

on-g

over

nmen

t or

gani

sation

w

hich

pla

ces

peop

le fro

m c

ompa

nies

and

go

vern

men

t ag

enci

es in

to I

ndig

enou

s or

gani

sation

s.

Cur

rent

ly o

pera

tes

in n

ine

Indi

geno

us

com

mun

itie

s ac

ross

Aus

tral

ia:

Cap

e Yo

rk;

Gou

lbur

n M

urra

y; E

ast

Kim

berley

; In

ner

Syd

ney;

Cen

tral

Coa

st;

Low

er

Riv

er M

urra

y; N

orth

Eas

t Arn

hem

Lan

d an

d th

e N

PY L

ands

.

Impa

ct E

valu

atio

n.

The

eval

uation

sou

ght

to v

alid

ate

Jaw

un’s

‘The

ory

of C

hang

e’ b

y co

nduc

ting

sur

veys

and

inte

rvie

ws

with

Indi

geno

us o

rgan

isat

ions

, co

rpor

ate

and

gove

rnm

ent

part

ners

and

sec

onde

es

and

by a

naly

sing

ava

ilabl

e so

cio-

dem

ogra

phic

dat

a.

The

eval

uation

fou

nd o

vera

ll as

sist

ance

of

fere

d by

Jaw

un s

econ

dees

was

ap

prop

riat

e an

d m

et I

ndig

enou

s or

gani

sation

’s n

eeds

, by

str

engt

heni

ng

thei

r ca

paci

ty a

nd b

y le

vera

ging

the

ex

pert

ise

of c

orpo

rate

and

gov

ernm

ent

part

ners

to

supp

ort

Indi

geno

us le

d pr

ojec

ts.

How

ever

, th

e ev

alua

tion

su

gges

ted

the

mod

el c

ould

be

impr

oved

by

bet

ter

asse

ssin

g or

gani

sation

s’

capa

bilit

ies

and

capa

city

to

host

se

cond

ees

effe

ctiv

ely.

Weak

Onl

y fo

ur o

f th

e ei

ght

Indi

geno

us r

egio

ns

in w

hich

Jaw

un o

pera

tes

took

par

t in

the

ev

alua

tion

(Cap

e Yo

rk,

Cen

tral

Coa

st,

East

Kim

berley

and

Inn

er S

ydne

y)

and

only

the

exe

cutive

sum

mar

y of

th

e ev

alua

tion

is p

ublic

ly a

vaila

ble

on

Jaw

un’s

web

site

.


Appendix B: Evaluation Toolkit

There are many different reasons for, and benefits in, conducting evaluations (see Table 12).

Table 12: Reasons for, and benefits in, conducting evaluations

Agency/institution

Potential benefits

Government • More efficient resource allocation• Highlights what is and is not

working• More informed decision-making• Encourages greater public trust in

government

Service providers

• Improved service delivery/client satisfaction

• Stronger basis for recurrent funding

• Opportunity for continuous improvement processes

Society • Improved government services• More open and accountable

government• Public money used more efficiently• Increased confidence in

government

Source: Adapted from ACT Government Evaluation Policy and Guidelines, 2010

There are also different types of evaluations depending on the stage of a program’s implementation and what the evaluation is seeking to measure. Generally speaking, different types of evaluation are used at different stages of a program’s implementation. These include:

• Formative evaluation — generally used at the design stage of a program and before it is implemented. Can be useful to inform decision-making about whether a program should proceed or not. Types of questions asked at this stage include, what is the problem, is government intervention appropriate, how will we measure success?

• Process evaluation — used during the program delivery process. Focuses on processes and what can be done to improve the operation of projects and programs. These types of evaluations are also known as performance evaluations. Questions asked in these evaluations tend to focus on how well an activity been executed, and inputs and output

• Summative evaluation — focuses on the outcomes and achievements of projects/or programs — also referred to as outcomes evaluation. Questions asked include: what kind of change has occurred as a result of the intervention?

• Impact evaluation — looks at how a program has affected the people participating in the program. Often not available until towards the end of the project and often relies on pre- and post-program data. Similar in many ways to summative and outcomes evaluations

• Development evaluation — a non-linear approach, not specific to a particular point in the roll out or delivery of a program. The main focus of this type of evaluation is understanding the activities of a program and how the program operates in a dynamic environment. The principle focus is on learning and feedback rather than achieving a set of predetermined outcomes. Development evaluation also recognises that positive outcomes can sometimes occur unintentionally.

Ideally evaluation should be embedded into program development and implementation. The Queensland Government Program Evaluation Guidelines have adapted the Gibbs Reflective Model to illustrate how evaluation should inform program design and implementation (see Figure 5).

The first step in undertaking an evaluation generally involves having a clear understanding of the outcomes the program is hoping to achieve and how those program outcomes will be measured — what evaluators sometimes call a program logic model or theory of change. However, although these two terms are sometimes used interchangeably they are actually two different approaches. A program logic or logic model seeks to illustrate how the needs or issues the program is seeking to address links with the intended activities outputs and outcomes of the program (see Figure 6).87

A theory of change model seeks to link outcomes and activities to explain how the desired change will occur and what factors contributed to that change. While logic models do not always identify the indicators that will be used to measure whether outcomes have been met or not, theory of change models do. For instance, the program logic for a program that seeks to improve students reading ability would identify the program as an activity and improved reading scores as an outcome, but it would not tell you that students need to attend the program at least three days a week for a minimum of x number of days and that the course material must include a focus on phonics for student’s scores to rise. As a result, a program logic model based on an underlying theory of change will have a lot more rigour than one that does not.88 An evaluation plan or framework sets out the information contained in a program logic model in more detail and generally includes a hierarchy of outcomes from inputs and process outcomes to ultimate outcomes, with key evaluation questions, indicators and potential data sources for each stage ( see Figure 7).89 A hierarchy of outcomes recognises that change can take time, and that certain outcomes need to be achieved in order to progress to a new level.

Ideally the objectives of the program should be specific, measurable, realistic and relevant to the overall objectives the program is trying to achieve. For example, a specific and measurable objective would be to increase the number of children who enrolled by 10% (from x to y) by a certain date. If a percentage increase is part of a measurable objective then it is important to have baseline data that provides a comparison for assessing program impact. However, although gathering program


Figure 6: Incorporating evaluation into program development and implementation

Figure 7: Program logic model

Program logic statements

Key evaluation questions

Indictors Potential data sources

Ultimate outcome

E.g. Indigenous people are able to achieve their goals and improve their quality of life

To what extent has the program contributed to Indigenous people achieving their goals?

Number and percentage of program participants surveyed who report improvements in the quality of their life and their ability to achieve their goals

• Interviews with people, their families and program staff

• Longitudinal case studies

• Quality of life assessment/survey

Longer term outcomes

E.g. Indigenous people are actively pursuing their goals

To what extent has the program contributed to Indigenous people being more able to determine and pursue their goals?

Level of improvement in people’s ability to set their own goals, as reported by Indigenous people, their families and staff

• Interviews with people, their families and program staff

• Longitudinal case studies

• Quality of life assessment/survey

Intermediate outcomes

E.g. Indigenous people are aware of and access the program

To what extent are Indigenous people accessing the program?

Number and reach of participants

• Program data

• Population data

Inputs and process outcomes

E.g. Support is provided to help Indigenous people identify the steps they need to take to pursue their goals, and appropriately skilled and experienced staff are recruited.

Have staff been recruited within agreed timelines?

Number and percentage of funded services that recruit staff within agreed timeframes

• Program data

Source: Queensland Treasury (2014).

Figure 8: Hierarchy of outcomes

Source: Adapted from an Urbis evaluation framework, 2014


data can be hard work, it is a myth that access to base-line data is always a problem. There are often existing administrative data sets on health, education and crime statistics that could be used to give a baseline. In addition, while ‘data is not the plural of anecdote’ a good evaluation involves a mixture of qualitative and quantitative data.90 Qualitative data assesses people’s perceptions of a program and often provide the ‘how’ for why the program has or has not achieved its objectives.

Broadly speaking there are three main areas of focus when conducting an evaluation to assess whether a program has achieved its objectives:91

• Appropriateness

• Effectiveness

• Efficiency

Appropriateness

Evaluating the appropriateness of a program involves considering whether there is a need for the program, given the social, economic and environmental context and how the program aligns with the government’s policies and priorities. Assessments of appropriateness should focus not only on the individual program, but on how the policies underpinning the program and other government policies and instruments interact with each other.92 In considering the appropriateness of a program, policy makers need to also look at whether there is a priori evidence base for the interventions. Questions related to appropriateness to consider before implementing a program are:

• Is the program needed?

• Is there community support for the program?

• Is there an evidence base for the interventions used in the program?

• Is there an existing program already addressing a similar need?

• What is different about this program?

• Who will implement the program?

Effectiveness

Evaluating the effectiveness of a program involves considering whether it is achieving the set objectives and producing worthwhile outcomes. A key challenge in illustrating the effectiveness of program is having valid measurement in place to determine whether there would have been a difference in outcomes without the program; what some term ‘estimating the counterfactual’.93 However, estimating the counterfactual when it comes to Indigenous programs can be difficult, given the myriad programs in Indigenous communities. Determining the impact of a single program in a particular Indigenous community is virtually impossible because so many programs are being delivered simultaneously. If another community is used as a counterfactual or ‘control group’ then the community is likely to be already receiving similar programs. Another factor making assessing the impact of programs difficult, is the uniqueness of Indigenous communities. For example, when the Northern Territory Emergency Response (NTER) was evaluated, it was difficult to find comparable communities that could act

as a type of control group, given that the NTER covered so many of the Indigenous communities in the Northern Territory and the varied nature of those communities.94

Efficiency

Evaluating the efficiency of a program involves identifying whether the program represents value for money, how a program’s resources are being used to achieve outputs of the desired quantity and quality, and whether the use of the resources could be improved to achieve the desired outcomes.95

Economic evaluation identifies, measures and values a program’s economic costs and benefits.96 Two methodological approaches used to measure the effectiveness and efficiency of programs are Cost Benefit Analysis (CBAs) and Social Return on Investment (SROI).

Cost Benefit Analysis expresses the costs and benefits of a program in monetary terms and focuses on community-wide rather than individual benefits.97 Values are aggregated using a discount rate that represents trade-offs between current and future consumption.98 Then, the discounted costs and benefits are compared using specific criteria. Limitations to the CBA methodology are that the benefits of some programs are very difficult to quantify due to their subjective nature. For instance, when measuring the effectiveness of Indigenous social programs, it can be difficult to place a monetary value on concepts such as social capital, wellbeing, quality of life, and cultural attachment.99 It is also difficult to quantify causal factors behind flow-on benefits, such as improved health outcomes or decreased crime rates. The CBA methodology can also be limited to ‘first round’ impacts and as a result indirect effects can be excluded.100 Another limitation is results can be skewed if an ‘improper’ rate for discounting future flows is used.101

The Social Return of Investment methodology originally began as a specialised form of cost-benefit analysis but has grown to incorporate many aspects of evaluation practice, such as qualitative interviews with stakeholders. Like CBAs, SROI methodology places a monetary value on the social impact of an activity and compares this with the costs involved in implementing that activity. However, SROIs place a greater emphasis on the social purpose for activities and how to measure the social impact.102 Although the SROI approach utilises aspects of evaluation practice it is not a comprehensive evaluation framework.103 As SROI analysis is specifically tailored to individual organisations it is not always possible to do cross-organisational comparisons. However, a SROI ratio can be used as a benchmark to enable organisations to measure changes in performance over time.104 One of the biggest issues with the SROI methodology is the tendency for people to misunderstand what the SROI ratio means. SROI is about value, rather than money. The SROI ratio represents the social value created for each $1 invested, rather than an actual financial return. As a result care needs to be taken with how the SROI ratio is communicated.105


Appendix C: List of Tobacco cessation programs

Name Provider Reach Objectives

Alcohol, tobacco and other drugs program

Tasmanian Aboriginal Centre Inc. funded by the federal government.

TAS The Alcohol, tobacco and other drugs program, run by the Tasmanian Aboriginal Centre, provides alcohol and other drug (AOD) support to the Aboriginal community across Tasmania.

Apunipima tackling smoking and healthy lifestyle program

Apunipima Cape York Health Council

QLD The program aims to raise awareness about the impacts of tobacco smoking and to help facilitate smoke-free environments.

Beyond today — it’s up to you

Australian Capital Territory Department of Health

ACT A social marketing campaign to encourage Indigenous Australians in the ACT to stop smoking.

Don’t let your dreams go up in smoke

Nunkuwarrin Yunti of South Australia

SA To encourage young Aboriginal people in Adelaide to share their ideas, stories and videos on smoking and how it is harmful.

Feet first (Thoolngoonj bowirn)

Australian Council on Smoking and Health (ACOSH)

WA Aims to reduce the amount of people smoking in Kununurra to teach Indigenous people about the harmful effects of smoking.

Good sports program Australian Drug Foundation National To address risky drinking, smoking, obesity and mental health though community sports.

Healthy lifestyle & tobacco cessation program

Central Australian Aboriginal Congress

NT The program provides services to help raise awareness of chronic disease resulting from smoking.

Heart health ‘for our people, by our people’

Derbarl Yerrigan Health Service (DYHS), the National Heart Foundation, Royal Perth Hospital (Cardiology Department)

WA A cardiac rehabilitation program, concentrating on health, medications, oral health and quitting smoking.

It’s your choice, now!

South Eastern Sydney Local Health District

NSW Encourages young Indigenous people to give up smoking by teaching them new skills and making their own films.

Kick the butt A partnership between Bunurong Health, Quitline and the Cancer Council.

VIC Aims to limit the uptake of smoking tobacco within the Southern metropolitan region of Melbourne. Provides a 24 hour hotline, social marketing campaign and advertising on SBS.

Maternal health tackling smoking program

Aboriginal Health Council of South Australia

SA Aims to reduce tobacco smoking among pregnant Aboriginal women and to increase the birth weight of babies.

No more boondah Winnunga Nimmityjah Aboriginal Health Service

ACT The program looks at what triggers people to smoke.

No more nyumree Wheatbelt Aboriginal Health Service

WA This program aims to provide ‘culturally appropriate’ support to help Aboriginal people stop smoking.

Primary Prevention Capacity Building Project

The Queensland Aboriginal and Islander Health Hub

QLD This program seeks to develop the capacity for Aboriginal organisations to offer interventions to address high rates of smoking among Indigenous people living in Queensland.

Puyu blaster (Keep it corka)

Aboriginal Health Council of South Australia (AHCSA)

SA A healthy lifestyle and anti-smoking campaign which seeks to promote local role models to encourage people to give up smoking.


Name Provider Reach Objectives

Quit for new life program

Hunter New England Local Health District

NSW This program aims to reduce the rate of smoking among pregnant women and their family or household members.

Regional tackling tobacco and healthy lifestyles program

Wuchopperen Health Service QLD The aim of this regional program is to reduce the onset and risk of chronic disease developed through tobacco use, poor nutrition and lack of physical activity.

Rewrite your story Nunkuwarrin Yunti Inc. SA This program aims to help people break the cycle of smoking and to quit for good.

Smoking cessation program

Derby Aboriginal Health Service

WA The Smoking cessation program provides information about services to help people quit smoking, including nicotine patches at no cost to participants.

Stepping Stones AOD Day Centre Ceduna

Aboriginal Drug and Alcohol Council

SA This program provides free confidential treatment, counselling and referral services for Aboriginal peoples concerned about alcohol, tobacco and other drug issues.

Substance use, social and emotional wellbeing

Katherine West Health Board Aboriginal Corporation

NT This program focuses on the harmful effects of alcohol, tobacco and cannabis use on Indigenous people in the Northern Territory.

Tackling Indigenous smoking

Australian Drug Foundation National This program aims to reduce smoking among Indigenous Australians.

Tackling smoking and healthy lifestyle program

South West Aboriginal Medical Service in Western Australia (SWAMS)

WA This program aims to tackle chronic disease risk factors including smoking, poor nutrition and lack of exercise, and to deliver community education initiatives to reduce the prevalence of these risk factors in Aboriginal and Torres Strait Islander populations.

The Gnumaries hurt program

Southern Aboriginal Corporation

WA This program was developed to reduce the uptake and prevalence of tobacco smoking among the Noongar people of the Great Southern region of Western Australia.

Time to quit Kambu Aboriginal and Torres Strait Islander Corporation for Health

QLD This program takes a holistic approach to tobacco cessation and provides people with practical suggestions to help them stop.

Tobacco and healthy lifestyles

Ngaanyatjarra Health Service

WA This program aims to reduce the risk of chronic disease from smoking and other unhealthy lifestyle choices among the Indigenous people living on the Ngaanyatjarra Lands, in Western Australia.

Tobacco cessation team

Victorian Aboriginal Community Controlled Health Organisation (VACCHO)

VIC This program provides support to the Victorian Aboriginal Community Controlled Health Organisation (VACCHO) member services to develop and implement programs and policies to reduce smoking.

Tobacco resistance and control (A-TRAC) program

Aboriginal Health and Medical Research Council of NSW

NSW This program aims to reduce smoking rates for Aboriginal people in New South Wales.

Yarning it Up — Don’t Smoke it Up

South Metropolitan Population Health Unit

VIC The project runs workshops to help people quit smoking that aim to be ‘culturally appropriate’ and non-judgmental.

Young Aboriginal drug and alcohol service (YADAS)

Young Aboriginal Drug and Alcohol Service

TAS This program aims to provide ‘culturally relevant’ anti-smoking and drug programs in partnership with other health service providers.


Endnotes

1 Hudson, S, 2016. Mapping the Indigenous Program and Funding Maze. Sydney: The Centre for Independent Studies.

2 Hudson, 2016. Mapping the Indigenous Program and Funding Maze

3 SCRGSP (Steering Committee for the Review of Government Service Provision) 2016, Overcoming Indigenous Disadvantage: Key Indicators 2016, Canberra: Productivity Commission, p.iii

4 The Prime Minister of Australia. 2017. Closing the Gap Report Statement to Parliament. Available at: https://www.pm.gov.au/media/2017-02-14/closing-gap-report-statement-parliament. Accessed 17 April 2017

5 Austender Australian Government. 2017. Entity Reports for complying with the Senate Order on Procurement Contracts and use of Confidentiality Provisions. Available at: https://www.tenders.gov.au/?event=public.senateOrder.list accessed 17 April 2017

6 Empowered Communities, 2015. Empowered Communities Design Report, Wunan Foundation

Inc, available at http://empoweredcommunities.org.au/f.ashx/EC-Report.pdf accessed 17 April 2017

7 Antonios, Z, 1997. The CDEP Scheme and Racial Discrimination: A Report by the Race Discrimination Commissioner. Sydney: Human Rights and Equal Opportunity Commission; and Hudson, S, 2012. No more dreaming of CDEP. The Centre for Independent Studies, available at: https://www.cis.org.au/commentary/articles/no-more-dreaming-of-cdep/ accessed 17 May 2017; and Morris, N and Tomllin, S, 2016. Cultural incompatibility hindering public policy in remote Western Australia, indigenous leaders say. ABC News, 22 September 2016 available at http://www.abc.net.au/news/2016-09-22/cultural-incompatibility-of-public-policy-in-remote-australia/7868810 accessed 17 May 2017

8 Senate Finance and Public Administration Committee, 2017. Commonwealth Indigenous Advancement Strategy tendering processes. Canberra: Commonwealth of Australia, available at http://www.aph.gov.au/Parliamentary_Business/Committees/Senate/Finance_and_Public_Administration/Commonwealth_Indigenous/Report accessed 17 May 2017

9 Australian National Audit Office, 2017. Indigenous Advancement Strategy, Canberra: Commonwealth of Australia available at https://www.anao.gov.au/work/performance-audit/indigenous-advancement-strategy accessed 17 May 2017

10 As above

11 Australian Public Service Commission. 2016. Providing robust advice, available at: http://www.apsc.gov.au/publications-and-media/current-publications/learning-from-failure/providing-robust-advice. accessed 17 May 2017

12 Antonios, Z, 1997. The CDEP Scheme and Racial Discrimination: A Report by the Race Discrimination Commissioner. Sydney: Human Rights and Equal Opportunity Commission; and Hudson, S, 2012. No more dreaming of CDEP. The Centre for Independent Studies, available at: https://www.cis.org.au/commentary/articles/no-more-dreaming-of-cdep/ accessed 17 May 2017; and Morris, N and Tomllin, S, 2016. Cultural incompatibility hindering public policy in remote Western Australia, indigenous leaders say. ABC News, 22 September 2016 available at http://www.abc.net.au/news/2016-09-22/cultural-incompatibility-of-public-policy-in-remote-australia/7868810 accessed 17 May 2017

13 ATSICOEA (Aboriginal and Torres Strait Island Commission Office of Evaluation and Audit), 1997 Evaluation of the Community Development Employment Projects Program: Final Report, Canberra: ATSI

14 DEWR (Australian Government Department of Employment and Workplace Relations) 2005. Building on success: CDEP discussion paper, Canberra: Australian Government

15 Hudson, S (2008) Leadership can make all the difference in a CDEP organisation, Sydney Morning Herald, 11 October, 2008, available at https://www.cis.org.au/commentary/articles/leadership-can-make-all-the-difference-in-a-cdep-organisation accessed 17 May 2017

16 Aikman, A, 2016. Widen welfare rules to all jobless, says Tony Abbott. The Australian, 6 October 2016 available at http://www.theaustralian.com.au/national-affairs/indigenous/widen-welfare-rules-to-all-jobless-says-tony-abbott/news-story/8720125387b8731d8dd6cd731cc18395 accessed 13 December 2016

17 Australian National University. 2016. Researchers call for urgent rethink of remote policy, available at: http://cass.anu.edu.au/news/research/20161202/researchers-call-urgent-rethink-remote-policy accessed 17 May 2017; and Jordan, K, and Fowkes, L, 2016. Job Creation and Income Support in Remote Indigenous Australia: Moving forward with a better system. Centre for Aboriginal Economic Policy Research, CAEPR Topical Issue NO. 2/2016, available at: http://caepr.anu.edu.au/sites/default/files/Publications/topical/CAEPR%20Topical%20Issues%202_2016.pdf accessed 17 May 2017

18 Australian National Audit Office, 2017. Indigenous Advancement Strategy, Canberra: Commonwealth of Australia

19 As above

20 Australian National Audit Office, 2017. Indigenous Advancement Strategy, Canberra: Commonwealth of Australia

21 The Senate Finance and Public Administration Committee, 2016, Commonwealth Indigenous Advancement Strategy tendering process, page 56


22 Young, E, 2016. Senator Rachel Siewert condemns Indigenous Advancement Strategy after report. WA Today, 22 March 2016 available at http://www.watoday.com.au/wa-news/senator-rachel-siewert-condemns-indigenous-advancement-strategy-after-report-20160321-gnnp74.html accessed 17 May 2017

23 As above see also Parke, E and Martin, L, 2015. Funding cut for remote Aboriginal domestic violence shelter will ‘put lives at risk’. WA Today, 18 May 2015 available at http://www.abc.net.au/news/2015-05-17/funding-withdrawal-puts-indigenous-womens-lives-at-risk/6476132 accessed 17 May 2017

24 Havnen, O, 2012. Office of the Northern Territory Coordinator-General for Remote Services Report. Darwin: The Office of the Coordinator-General for Remote Services available at www.territorystories.nt.gov.au/bitstream/handle/10070/241806/NTCGRS_fullreport_2012.pdf?sequence=1 accessed 17 May 2017

25 Gosford, R, 2012. Anderson sacks Havnen – a case of two strong hens in the NT henhouse is one too many? Crickey, available at: https://blogs.crikey.com.au/northern/2012/10/09/anderson-sacks-havnen-%E2%80%93-a-case-of-two-strong-hens-in-the-nt-henhouse-is-one-too-many/ accessed 17 May 2017

26 Australian Indigenous Healthinfonet. 2017. Programs and Projects available at: http://www.healthinfonet.ecu.edu.au/key-resources/programs-projects. accessed 17 May 2017

27 The Commonwealth Department of Health. 2014. Tackling Indigenous Smoking and Healthy Lifestyle Programme Review 2014, available at: http://www.health.gov.au/internet/main/publishing.nsf/Content/indigenous-tis-hlp-review. accessed 17 May 2017

28 The Commonwealth Department of Health. 2017. Tackling Indigenous Smoking (TIS), available at: http://www.health.gov.au/internet/main/publishing.nsf/Content/indigenous-tis-lp.accessed 17 May 2017

29 Taylor, P and Laurie, V, 2016. Fitzroy Crossing at the Crossroads. The Australian, 16 November 2016 available at http://www.theaustralian.com.au/news/inquirer/fitzroy-crossing-at-the-crossroads/news-story/c46c7b0ee66be2ac3c67354e056c7e7b accessed 17 May 2017

30 Australian Human Rights Commission, 2017. From community crisis to community control in the Fitzroy Valley. Social Justice Report 2010, available at: http://www.humanrights.gov.au/publications/chapter-3-community-crisis-community-control-fitzroy-valley-social-justice-report-2010 accessed 17 May 2017

31 Moran, M, 2016, Serious Whitefella Stuff: When solutions become the problem in Indigenous affairs, Melbourne University Press, Melbourne

32 Australian Productivity Commission, 2012, Roundtable Proceedings. In Better Indigenous Policies: The Role of Evaluation. Canberra, 22–23 October 2012. Canberra: Productivity Commission

available at http://www.pc.gov.au/research/supporting/better-indigenous-policies/better-indigenous-policies.pdf accessed 17 May 2017

33 Jordan, K, and Fowkes, L, 2016. Job Creation and Income Support in Remote Indigenous Australia: Moving forward with a better system. Centre for Aboriginal Economic Policy Research, CAEPR Topical Issue NO. 2/2016, available at: http://caepr.anu.edu.au/sites/default/files/Publications/topical/CAEPR%20Topical%20Issues%202_2016.pdf accessed 17 May 2017

34 FSG. 2016. The Case for Developmental Evaluation, available at: http://www.fsg.org/blog/case-developmental-evaluation accessed 17 May 2017

35 Neave, C, 2016. Submission by the Commonwealth Ombudsman to the Department of Prime Minister and Cabinet Consultation Paper: Changes to the Community Development Program, available at: https://www.dpmc.gov.au/sites/default/files/cdp-submissions/commonwealth-ombudsman-submission.pdf accessed 17 May 2017

36 Havnen, O, 2012. Office of the Northern Territory Coordinator-General for Remote Services Report. Darwin: The Office of the Coordinator-General for Remote Services available at www.territorystories.nt.gov.au/bitstream/handle/10070/241806/NTCGRS_fullreport_2012.pdf?sequence=1 accessed 17 May 2017, pages 57-58

37 As above

38 McDonald, E, Cunningham, T, and Slavin, N 2017. Evaluating a handwashing with soap program in Australian remote Aboriginal communities: a pre and post intervention study design. BMC Public Health 15, available at: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4662811/ accessed 14 May 2017

39 Audit Office of New South Wales, 2017. Implementation of the NSW Government’s program evaluation initiative. available at: http://www.audit.nsw.gov.au/publications/latest-reports/nsw-government-program-evaluation accessed 17 May 2017

40 New South Wales Department of Premier and Cabinet, 2016. NSW Government Program Evaluation Guidelines, available at: http://www.dpc.nsw.gov.au/programs_and_services/policy_makers_toolkit/evaluation_in_the_nsw_government accessed 17 May 2017

41 Victoria Department of Planning and Community Development, 2008. Evaluation Step-by-step Guide, available at: http://www.dhs.vic.gov.au/__data/assets/pdf_file/0011/769943/Evaluation-Step-by-Step-Guide.pdf accessed 17 May 2017

42 Victorian State Government. n.d. Funded Organisation Performance Monitoring Framework. available at: http://www.dhs.vic.gov.au/facs/bdb/fmu/service-agreement/4.departmental-policies-procedures-and-initiatives/4.10-funded-organisation-performance-monitoring-framework accessed 17 May 2017


43 Queensland Government, Queensland Treasury. 2014. Queensland Government Program Evaluation Guidelines available at: https://www.treasury.qld.gov.au/publications-resources/qld-government-program-evaluation-guidelines/. accessed 17 May 2017

44 Tasmanian Government. 2015. Planning, evaluation and procurement, available at: http://www.communications.tas.gov.au/policy/planning_and_procurement, accessed 17 May 201

45 South Australian Government. 2016. Managing a Community Organisation Evaluation, available at: https://www.sa.gov.au/topics/family-and-community/community-organisations/managing-a-community-organisation/evaluation accessed 17 May 2017

46 Western Australia Department of Treasury. 2017. Program Evaluation, available at: http://www.treasury.wa.gov.au/Treasury/Program_Evaluation/Program_Evaluation/ accessed 17 May 2017

47 The Government of Western Australia. 2017. Program Evaluation Western Australia, available at: http://www.programevaluation.wa.gov.au/ accessed 17 May 2017

48 The Government of Western Australia. 2015. Evaluation Guide, available at: http://www.treasury.wa.gov.au/uploadedFiles/Treasury/Program_Evaluation/evaluation_guide.pdf, accessed 17 May 2017

49 The Northern Territory Government. 2017. Guidelines for working with NGOs, available at: https://nt.gov.au/community/non-government-organisations-ngos/guidelines-for-working-with-ngos accessed 17 May 2017

50 ACT Government. 2010. ACT Government Evaluation Policy and Guidelines, available at: http://www.cmd.act.gov.au/__data/assets/pdf_file/0004/175432/ACT-Evaluation-Policy-Guidelines.pdf accessed 17 May 2017

51 Victorian Government Department of Primary Industries, 2017, Gathering evidence using data: the implications and advantages of undertaking evaluation research in-house. In Australian Evaluation Society Conference. Canberra, 2009 available at https://www.aes.asn.au/images/stories/files/conferences/2009/documents/Narelle%20Fitzgerald.pdf accessed 17 May 2017

52 Oxfam blogs. 2012. Getting evaluation right: a five point plan, available at: https://oxfamblogs.org/fp2p/getting-evaluation-right-a-five-point-plan/ accessed 17 May 2017

53 Cobb-Clark, D, 2012. The case for making public policy evaluations public. In Australian Productivity Commission, 2012, Roundtable Proceedings. In Better Indigenous Policies: The Role of Evaluation. Canberra, 22–23 October 2012, pages 81-91 available at http://www.pc.gov.au/research/supporting/better-indigenous-policies/07-better-indigenous-policies-chapter5.pdf accessed 17 May 2017

54 As above

55 Gruen, N, 2016. Why we accept travesties of ‘evidence-based’ policymaking. The Mandarin, 9 May 2016 available at http://www.themandarin.com.au/64557-nicholas-gruen-evidence-based-policy-part-one/ accessed 17 May 2017

56 Hunt, J, 2017. The Cashless Debit Card trial evaluation: A short review. Centre for Aboriginal Economic Policy Research, ANU College of Arts and Social Sciences, available at: http://caepr.anu.edu.au/Publications/topical/2017TI1.php

accessed 15 May 2017


58 Senator Nigel Scullion. 2016. Media Release: Overcoming Indigenous Disadvantage Report highlights progress, available at http://www.nigelscullion.com/overcoming-indigenous-disadvantage-report-highlights-progress/ accessed 17 May 2017

59 Siminski, P, 2016. How to get a better bang for the taxpayers’ buck in all sectors, not only Indigenous programs. The Conversation, 23 August 2016 available at https://theconversation.com/how-to-get-a-better-bang-for-the-taxpayers-buck-in-all-sectors-not-only-indigenous-programs-64296 accessed 17 May 2017

60 As above

61 Holman, D, 2014, A Promising Future: WA Aboriginal Health Programs Review of performance with recommendations for consolidation and advance Government of Western Australia Department of Health available at

http://ww2.health.wa.gov.au/~/media/Files/Corporate/Reports%20and%20publications/Holman%20review/a-promising-future-wa-aboriginal-health-programs.ashx accessed 17 May 2017

61 Bauer, M, Damschroder, L, Hagedorn, H, Smith, J and Kilbourne, A. 2015. An introduction to implementation science for the non-specialist. BMC Psychology, 3:32, 1-12. available at: https://bmcpsychology.biomedcentral.com/articles/10.1186/s40359-015-0089-9 accessed 14 May 2017

62 Cobb-Clark, D, 2012. The case for making public policy evaluations public. In Australian Productivity Commission, 2012, Roundtable Proceedings. In Better Indigenous Policies: The Role of Evaluation. Canberra, 22–23 October 2012, pages 81-91 available at http://www.pc.gov.au/research/supporting/better-indigenous-policies/07-better-indigenous-policies-chapter5.pdf accessed 17 May 2017


63 Griffiths A, Zmudzki F, Bates S, 2017, Evaluation of ACT Extended Throughcare Program: Final Report (SPRC Report 02/17). Sydney: Social Policy Research Centre, UNSW available at https://www.sprc.unsw.edu.au/media/SPRCFile/Evaluation_of_ACT_Extended_Throughcare_Pilot_Program.pdf accessed 17 May 2017

64 SCRGSP (Steering Committee for the Review of Government Service Provision) 2016, Overcoming

Indigenous Disadvantage: Key Indicators 2016, Productivity Commission, Canberra available at http://www.pc.gov.au/research/ongoing/overcoming-indigenous-disadvantage/2016/report-documents/oid-2016-overcoming-indigenous-disadvantage-key-indicators-2016-report.pdf accessed 17 May 2017

65 As above

66 SCRGSP (Steering Committee for the Review of Government Service Provision) 2016, Overcoming

Indigenous Disadvantage, as above

67 As above

68 Victoria Department of Treasury and Finance, 2011, Guide to Evaluation: How to plan and conduct effective evaluation for policy and programs available at www.dtf.vic.gov.au/files/41bba0b5-9ef3-4232.../DTF-Guide-to-Evaluation-2005.doc accessed 17 May 2017

69 Fitzpatrick, S, 2017. Ganbina a shining light for indigenous accountability, The Australian, 24 August 2016 available at http://www.theaustralian.com.au/national-affairs/indigenous/ganbina-a-shining-light-for-indigenous-accountability/news-story/201b36dbe8655cf83c18d2964b93ea3e accessed 17 May 2017


71 Fred Hollows Foundation, 2012. The Women’s Development Project: Indigenous Australia Program Evaluation Report available at http://www.healthinfonet.ecu.edu.au/uploads/resources/23392_23392.pdf accessed 17 May 2017


73 Urbis, 2015, Ability Links NSW Evaluation Interim Report 4,

74 Social Ventures Australia, 2017. The Martu Leadership Program Evaluation of a pilot program using the Social Return on Investment (SROI) methodology, available at https://static1.squarespace.com/static/54e3fc54e4b08f2ad4349a65/t/58fd4da9d1758eec4bd06271/1492995520775/170420+Kanyirninpa+Jukurrpa+Evaluation+of+the+Martu+Leadership+Program.pdf accessed 17 May 2017, page 20

75 James, M, 2012. Designing evaluation strategies. In Australian Productivity Commission, 2012, Roundtable Proceedings. In Better Indigenous Policies: The Role of Evaluation. Canberra, 22–23 October 2012. Canberra: Productivity Commission available at http://www.pc.gov.au/research/supporting/better-indigenous-policies/09-better-indigenous-policies-chapter7.pdf accessed 17 May 2017

76 Sabel, J, and Jordan, J, 2015. Doing, Learning, Being: Some Lessons Learned from Malaysia’s National Transformation Program. 1st ed. Washington: The World Bank Group available at http://www2.law.columbia.edu/sabel/papers/CS-LSJ--DLB%20Malaysia%20PEMANDU--Final-190115.pdf accessed 17 May 2017

77 As above

78 Jordan, K, and Fowkes, L, 2016. Job Creation and Income Support in Remote Indigenous Australia: Moving forward with a better system. Centre for Aboriginal Economic Policy Research, CAEPR Topical Issue NO. 2/2016, available at: http://caepr.anu.edu.au/sites/default/files/Publications/topical/CAEPR%20Topical%20Issues%202_2016.pdf accessed 17 May 2017



81 See Entity Reports for complying with the Senate Order on Procurement Contracts and use of Confidentiality Provisions 2015/2016 Financial Year from AusTender website

82 Morely, S , 2015 What works in effective community managed programs, Child Family Community Australia, CFCA Paper No. 32: Canberra available at https://aifs.gov.au/cfca/publications/what-works-effective-indigenous-community-managed-programs-and-organisations accessed 17 May 2017

83 As above

84 Moran, M, 2016. How community-based innovation can help Australia close the Indigenous gap. The Conversation, 4 March 2016 available at https://theconversation.com/how-community-based-innovation-can-help-australia-close-the-indigenous-gap-54907 accessed 17 May 2017

85 As above




88 Clarke, H, and Andersson, A, 2017, Theories of Change and Logic Models: Telling Them Apart. In Presentation at American Evaluation Association. Atlanta, Georgia, November 2004 available at https://www.theoryofchange.org/wp-content/uploads/toco_library/pdf/TOCs_and_Logic_Models_forAEA.pdf accessed 17 May 2017


90 Australian Productivity Commission, 2013, Summary of Roundtable discussions In Better Indigenous Policies: The Role of Evaluation,

Roundtable Proceedings, Productivity Commission, Canberra, page 3 available at http://www.pc.gov.au/research/supporting/better-indigenous-policies/02-better-indigenous-policies-summary.pdf accessed 17 May 2017

91 United States Government Accountability Office, 2012. Designing Evaluations 2012 Revision. Washington available at http://www.gao.gov/assets/590/588146.pdf accessed 17 May 2017

92 As above


94 As above

95 United States Government Accountability Office, 2012. Designing Evaluations 2012 Revision. 2nd ed. Washington: http://www.gao.gov/assets/590/588146.pdf.


97 Ackerman, F., 2008, Critique of Cost-Benefit Analysis, and Alternative Approaches to Decision-Making, Available at: http://www.ase.tufts.edu/gdae/pubs/rp/ack_uk_cbacritique.pdf, Accessed Feb 1 2016

98 Daly, A. and Barrett, G., n.d., Independent Cost Benefit Analysis of the Yuendumu Mediation and Justice Committee, University of Canberra, Available at www.centraldesert.nt.gov.au/sites/centraldesert.nt.gov.au/files/attachments/yuendumu_cba_0.pdf, accessed Jan 26 2016.

99 Daly and Barrett, n.d. as above and Access Economics, 2006, Opal Cost Benefit Analysis, Available at: http://www.npywc.org.au/wp-content/uploads/2011/09/OpalReport2006_02_23.pdf, Accessed Jan 26 2016

100 Daly and Barrett, n.d as above

101 Daly and Barrett, n.d as above and Allen Consulting Group, 2011, Assessment of the economic and employment outcomes of the Working on Country program, Available at: http://www.environment.gov.au/indigenous/workingoncountry/publications/pubs/woc-economics.pdf, Accessed Jan 26 2016

102 Nicholls, J., Lawlor,E.,Neitzert, E. and Goodspeed, T., 2009, A guide to Social Return on Investment, available at: http://socialvalueuk.org/publications/publications/doc_download/241-a-guide-to-social-return-on-investment-2012., Accessed Feb 1 2016

103 Social Ventures Australia (SVA), 2012, Social Return on Investment Lessons learned in Australia’, Available at: http://socialventures.com.au/assets/SROI-Lessons-learned-in-Australia.pdf, Accessed Feb 4 2016

104 As above

105 Social Ventures Australia (SVA), 2012, Social Return on Investment Lessons learned in Australia’, available at: http://socialventures.com.au/assets/SROI-Lessons-learned-in-Australia.pdf, Accessed Feb 4 2016

Level 1, 131 Macquarie St, Sydney NSW 2000 • phone: +61 2 9438 4377 • fax: +61 2 9439 7310 • email: [email protected]

About the Author

Research Report 28 (RR28) • ISSN: 2204-8979 (Printed) 2204-9215 (Online) • ISBN: 978-1-922184-87-0

Published June 2017 by The Centre for Independent Studies Limited. Views expressed are those of the authors and do not necessarily reflect the views of the Centre’s staff, advisors, directors or officers. © The Centre for Independent Studies (ABN 15 001 495 012), 2016This publication is available from The Centre for Independent Studies. Visit www.cis.org.au.

Sara Hudson

Sara Hudson is a Research Fellow and Manager of the Indigenous Program at the Centre for Independent Studies.

She has published widely on Indigenous policy for the CIS, with a particular focus on Indigenous programs, economic development, health and criminal justice.

Date post:	09-Jun-2020
Category:	Documents
Upload:	others
View:	1 times
Download:	0 times

Evaluating Indigenous programs: a toolkit for change Sara ... › 2017 › 06 › evaluati… ·...

Documents