QUT Digital Repository: http://eprints.qut.edu.au/
Lennie, June (2005) An evaluation capacity-building process for sustainable community IT initiatives. Evaluation, 11(4). pp. 390-414.
© Copyright 2005 Sage Publications
1
An Evaluation Capacity-Building Process for Sustainable Community IT
Initiatives: Empowering and Disempowering Impacts
June Lennie
Paper to be published in Evaluation. The International Journal of Theory,
Research and Practice, 11 (4), 390-414.
Introduction
Participatory forms of evaluation aim to produce a range of empowering outcomes and
impacts, including increased community capacities in planning and conducting evaluations and
broader stakeholder participation in decision-making. Other outcomes include improved
communication and trust among stakeholders, and constant improvement of initiatives in ways
that meet community or client needs. In her extremely positive assessment of participatory
evaluation, Diez (2001: 907) suggests that this methodology can be a useful tool to ‘mobilise
communities for regional action, empower local agents and enhance learning capacity’.
Many researchers argue that building community capacities and fostering empowerment are
more effective ways of achieving sustainable community development than programs and
success indicators imposed by outside experts (Harrison, 1998; Mobbs, 1998). They point out
that outside experts usually have limited knowledge and understanding of the particular
context, needs and issues of a community. Local solutions to achieving sustainable community
and economic development are therefore seen as important outcomes of a capacity building
approach to evaluation.
Nevertheless, evaluation methodologies which explicitly aim to be participatory raise many
complex theoretical, methodological and ethical issues, as Gregory (2000), McKie (2003),
Rebien (1996) and others point out. They include issues related to stakeholder
representativeness, the greater level of time, energy and resources required to conduct
evaluations and develop trust, the potential for dependence on the facilitator/professional
evaluation consultant, and for conflicting agendas and perspectives of various stakeholder
groups to hinder success (Gregory, 2000; Papineau and Kiely, 1996; Rebien, 1996). Rebien
(1996) also notes the conceptual weakness of the concept of participation, while the contested
2
concept of ‘empowerment’ is often used in uncritical ways. Indeed those advocating
participatory approaches to research and evaluation often make idealistic or naïve assumptions
that community participation will automatically lead to empowerment.
Thus, while power is a central issue in participatory forms of evaluation, it is often ignored
(Gregory, 2000). Vanderplaat (1995: 85) suggests that even the more critical models of
evaluation have failed ‘to deal, in any meaningful way, with the concept of relative power, or
more specifically the unequal distribution of discursive power, a central construct in
empowerment-based social programming’. As McKie highlights in her recent paper on the
concept of ‘rhetorical spaces’ (McKie, 2003), there is a need to focus on the communicative
and relational dimensions of participatory evaluations which can affect their outcomes in
unintended ways. This requires rigorous analysis of both the intended and potentially
unintended or negative impacts and effects of participatory evaluations.
This paper presents outcomes of a detailed analysis of the empowering and, at times,
disempowering impacts of the implementation of a framework known as ‘the LEARNERS
process’ (Learning, Evaluation, Action & Reflection for New technologies, Empowerment and
Rural Sustainability) for participants in two Australian rural communities. A key aim of the
LEARNERS project, which trialled this framework, was to build community capacities in
evaluating rural communication and information technology (C&IT) initiatives such as
community websites and information literacy programs. Using participatory evaluation and
participatory action research (PAR) methodologies (McTaggart, 1991; Wadsworth, 1998)
informed by praxis feminist theories and methodologies (Lather, 1991; Stanley and Wise,
1990), the project implemented and undertook an ongoing meta-evaluation and critique of the
implementation and use of the LEARNERS process in the two participating communities.
This capacity building framework was considered innovative in that it took a ‘whole of
community’ systems approach, used PAR and participatory evaluation methods, encouraged
analysis of differences such as gender, age, ethnicity and skills with C&IT, and sought to
develop ‘learning communities’ (Faris, 2001). The objectives of learning communities are
closely related to the goals of community capacity building. The process involves community
members from every sector working together to enhance the social, economic, cultural and
environmental conditions of their community and the promotion of lifelong learning (Faris,
3
2001). The implementation of the LEARNERS process also aimed to increase collaboration
and cooperation between diverse community organisations and groups and to generate new
awareness, knowledge and ideas about the use of C&IT for sustainable community and
economic development. The project was considered relatively unique in that few C&IT-based
projects in the community development field appear to have used a combination of PAR and
participatory evaluation methods.
Community members and representatives of community organisations in the Tara and
Stanthorpe Shires in Southern Queensland and representatives of five public sector
organisations participated in the project, which was conducted by a research team from
Queensland University of Technology (QUT) in Brisbane. The trial of the LEARNERS process
aimed to assist in enhancing the sustainability and the C&IT initiatives evaluated by groups in
the participating communities, and to increase community participation, empowerment and
leadership. The project also sought to increase recognition of the importance of women in
Australian rural communities in the uptake of new C&IT and in community development and
informal leadership activities (Grace and Lennie, 1998; The Rural Women and ICTs Research
Team, 1999).
After an overview of the development of the LEARNERS process, the aims of this capacity
building process and the methods involved in implementing the process in the two
participating communities are outlined. A brief summary of case studies of the implementation
of the process is then provided, followed by outcomes of a detailed analysis of the impacts of
the project for participants and their communities. While this impact assessment identified a
range of empowering effects of the LEARNERS project, a number of unintended and
disempowering impacts and other limitations of the project and the LEARNERS process were
also identified, as well as a number of barriers to participation. Drawing on these findings, the
strengths and limitations of the LEARNERS project and process are considered, as well as
lessons learned in the project for other evaluation capacity building projects.
Participatory approaches to research and evaluation
The emergence of participatory approaches to research and evaluation
Greater use of participatory and action research methodologies was made from the 1980’s
onwards as organisations started taking more inclusive, social justice-based approaches to the
4
design, implementation and assessment of community-based interventions. Evaluation
discourses shifted from those of the positivist social scientist to those of the empowerment-
oriented critical adult educator and social action researcher (Vanderplaat, 1995: 83). The
emphasis also shifted from knowledge created through social science to knowledge generated
through lived experience (Vanderplaat, 1995: 83).
Participatory action research aims to develop equal partnerships between researchers and
participants and to create knowledge that leads to action and positive social change.
Knowledge is seen as related to power and power is related to change. It is a political process
because it involves people making changes together that affect others (McTaggart, 1991: 177).
Critical reflection is a crucial step in each PAR cycle of planning, acting, observing and
reflecting. As in participatory evaluations, PAR encourages the active involvement of
participants and stakeholders in the design and conduct of projects and supports capacity
building processes. It can be a valuable method for involving a diversity of people in projects,
generating appropriate action, new ideas and long term visions, fostering ongoing change and
improvement, and regularly reflecting on outcomes (McTaggart, 1991). Thomas (2000: 112)
argues that PAR ‘challenges the hegemony of orthodox evaluation research methods’ and
therefore offers more opportunity to develop and evaluate long-term strategies for widening
participation in education and lifelong learning.
PAR has been successfully used in a large number of community development projects
conducted in developing countries (Garaway, 1995), as well as in community-based projects in
developed countries. It has been employed in a diversity of fields, including community
development, agricultural extension, education, health, and organisational management.
Participatory evaluation
Participatory evaluation methodologies emerged from the extension of PAR to evaluation
(Garaway, 1995) and the growing interest in evaluation as an action learning and capacity
building process. Collaborative or participatory forms of evaluation are seen as particularly
useful in assessing the impacts of ‘more complex system change and comprehensive
community initiatives’ (The Kellogg Foundation, 1998:5). Such initiatives include those
involving the use of new communication technologies. Participatory evaluations have been
argued to contribute to the long-term sustainability and success of community and economic
5
development programs (Brunner and Guzman, 1989; Diez, 2001; Dugan, 1996; Fetterman et
al.., 1996; Hudson, 2001; Papineau and Kiely, 1996; Rebien, 1996). They have been conducted
in a wide diversity of fields since the 1970s, including agriculture and rural development,
education, social services and health. However, this methodology has not been widely used to
evaluate C&IT projects.
Three main reasons have been proposed for increasing the involvement of participants and
stakeholders in evaluations:
(1) to increase utilisation of evaluation results; (2) to represent the values and concerns
of the multiple groups involved in decision-making; (3) to promote the empowerment
of disenfranchised stakeholder groups previously left out of the process (Papineau and
Kiely, 1996: 81).
As this quotation indicates, empowerment is often equated with participation in participatory
evaluations. However, different forms of participatory evaluation emphasise different levels of
participation. Empowerment evaluation (Fetterman et al., 1996) is notable in that it encourages
active involvement in all stages of the evaluation. Both empowerment and participation are,
however, contested concepts. Humphries (1994) points out that the concept of empowerment
can be used to justify oppressive practices; while the forms of participation identified range
from co-option to collective action (Martin, 2000: 200).
For an evaluation to be considered participatory, Rebien (1996: 160) suggests that stakeholders
must have an active role in the evaluation process, that at least representatives of stakeholders
should participate, and that stakeholders should participate in at least three evaluation phases:
‘designing terms of reference, interpreting data, and using evaluation information’. However,
Gregory (2000: 184) suggests that these criteria are insufficiently defined and ‘indeed, may
promote practices which actually have a negative impact on participation, rather than a positive
one’.
The methods and ethos of participatory evaluation thus contrast markedly with traditional
evaluation methods such as quasi-experimental impact assessments. In traditional approaches,
the key questions and methods to be used are decided by the evaluator or funding body, the
evaluator is expected to adopt an impartial and objective perspective, and program activities
6
are reduced to measurable indicators (Vanderplaat, 1995). In contrast, participant-oriented
forms of evaluation tend to use more ‘naturalistic’ inquiry methods that aim to reflect ‘the
complexities of everyday reality and the different perspectives of those engaged in providing
services’ (Worthen, Sanders and Fitzpatrick, 1997: 154). The scientific ideal of objectivity is
usually rejected in favour of a more holistic approach that openly acknowledges and takes into
account the diverse perspectives, agendas and values of participants, stakeholders and
evaluation consultants. However, evaluation rigor and the validation of findings is obtained
through using multiple methods of data collection and analysis, triangulation of data, peer
review, and engaging in critical reflexivity to enhance the ‘trustworthiness’ of results (Chess,
2000; Guba and Lincoln, 1989).
Questioning assumptions about community participation and empowerment
The research team working on the LEARNERS project adopted a critical approach which
questioned assumptions about community participation, empowerment and the sustainability of
C&IT projects. This approach recognised the often complex and contradictory nature of these
processes that can affect the outcome of participatory evaluations, as well as the importance of
paying attention to the communicative and relational dimensions of evaluation.
Taking barriers to participation into account
Research suggests that community participation methods that aim to be inclusive and
empowering and involve the use of new C&IT raise many complex issues. A range of social,
cultural, organisational, technological and economic factors and barriers have been found to
limit the effectiveness of participatory processes involving diverse community members and
groups (Boyce, 2001; Lennie, 2001, 2002b; McKie, 2003). These factors include the time,
energy and costs involved, people’s level of familiarity with participatory processes, lack of
effective access to C&IT, and the different agendas and power relations among various
stakeholder groups. Differences in knowledge about new C&IT, and concepts such as
‘community capacity building’ are other factors that need to be considered. In projects
involving academics and bureaucrats from urban areas and people and organisations in rural
communities, these factors are often highly relevant to the success or otherwise of strategies for
empowerment and inclusion in participatory evaluations (Lennie, 2001, 2002b). Analysis of
the impacts of the LEARNERS project took these complex issues and factors into account.
7
Critically evaluating claims for empowerment
Feminist critiques suggest that research projects which aim to be empowering can have
contradictory effects and that claims that research and evaluations have produced empowerment
require closer examination (Anderson, 1996; Gore, 1992; Lennie, 2002a; Lennie, Hatcher and
Morgan, 2003). This entails adopting a more critical and self-reflexive approach and using
more rigorous methods to evaluate such claims. Such an approach was taken in the meta-
evaluation and critique of the LEARNERS project, which included a rigorous analysis of the
impacts of the LEARNERS project. This analysis employed a framework of rural women’s
empowerment which was developed as part of my doctoral project (Lennie, 2001). Building on
the work of Friedmann (1992), this framework comprises four interrelated forms of power:
social, technological, political and psychological (see Figure 2 later in this paper).
Corresponding forms of disempowerment were also analysed, and gender and other differences
between participants were taken into account.
The various capacity building methods used in the LEARNERS project were seen as
underpinned by positive ‘power to’ and ‘power with’ models in which power is considered as
social and cooperative, rather than negative and related to domination and control (Deutchman,
1991). Empowerment was viewed as a process that can happen at the level of the individual,
the group, and/or the community (Claridge, 1996). Friedmann (1992: 33) suggests that
psychological empowerment can result in an increased sense of power, confidence and control,
which is often the result of successful action. Empowerment was seen as a long-term process
that people undertake for themselves, rather than something that is done to or for another
person (Lather, 1991: 4). However, outsiders, such as evaluation consultants, were considered
to play important roles in the process of empowerment by providing information and support,
and encouraging less empowered participants to gain confidence and ‘free themselves of
traditional dependency’ (Friedmann, 1992: 77).
Building community capacities in evaluation
The trial of the LEARNERS process aimed to build community and organisational awareness,
skills, confidence and capacities in participatory planning and evaluation and PAR.
Community capacity building has been defined as ‘strengthening the knowledge, skills and
attitudes of people so that they can establish and sustain their area’s development’ (Mannion,
8
1996: 2). It is seen as the ability of communities to solve their own problems, make their own
decisions and plan their own futures. Community capacity building programs aim to increase
community participation in planning and decision-making, to facilitate sustainable
development by building on existing community strengths, and to create communities that are
more inclusive, cooperative and self-reliant. In the LEARNERS project, capacity building also
referred to the process of building the capacity of organisations and community members to
undertake ongoing evaluations of local C&IT projects.
Our previous pilot research project indicated that rural community members and staff of
organisations that work closely with rural communities in Queensland often have limited skills
and experience in participatory forms of research, planning and evaluation (Lennie, Lundin and
Simpson, 2000a). The need to build community and organisational capacities in these
processes has been increasingly recognised by others working in this field (Fetterman et al.,
1996; Wadsworth, 1997; Boyle and Lemaire, 1999; O’Sullivan and O’Sullivan, 1998; Khan,
1998). PAR, empowerment evaluation, and participatory evaluation have been found to be
effective in developing these capacities.
Sustainability of C&IT initiatives
Governments and rural industry organisations in Australia have positioned new technologies
such as the Internet as being at the forefront of community and economic development in rural
and regional areas. Rural people are expected to benefit economically from new technologies
because they will provide new employment opportunities, the potential to buy and sell online,
an increase in services, and access to education and training, among other impacts (Da Rin and
Groves 1999; Groves and Da Rin 1999a, 1999b; Hearn et al., 2005).
However, many C&IT initiatives in rural Australia have failed (Geiselhart, 2004). This is due
to factors such as the small, highly scattered populations in many rural areas, limited funding
and resources, lack of access to training and technical support, and reliance on enthusiastic
‘champions’ and volunteers to successfully maintain initiatives. The long-term sustainability
of these initiatives is therefore a significant issue for rural communities.
Given that major Federal government funding programs for C&IT projects in Australia have
wound down in recent years, the continuing feasibility of these projects requires local
9
communities to find ways to make them self-sufficient and economically viable (DCITA,
2003). An approach to evaluation and impact assessment that takes a wide range of social,
cultural, economic, technological and environmental factors into account is therefore required.
This type of ‘whole of community’ systems approach was an important feature of the
LEARNERS process.
Development of the LEARNERS process
The LEARNERS process was based on a framework for planning and evaluating C&IT
projects which was designed and developed as part of a pilot research project. This project was
conducted from 1999 - 2000 by three QUT researchers in collaboration with two public sector
organisations (see Lennie et al., 2000a and 2000b).1 Development of this framework drew on
findings from substantial earlier action research projects undertaken by QUT researchers
(including the author) in partnership with people in rural, regional and remote Queensland
communities and several government and industry partners.
A multidisciplinary and iterative approach was taken in developing the original framework.
This involved the following research activities:
• A comprehensive search and critical review of relevant literature in the fields of
community development, C&IT impacts, evaluation and impact assessment methodologies,
and community participation and inclusion.
• Ongoing meetings and discussions that drew on the knowledge and expertise of the
research team and collaborating partners in diverse areas such as rural community
development, rural women’s access to and use of C&IT, and evaluating the use of C&IT in
education and training programs.
• Drawing on this literature, the outcomes of earlier research projects, and the ideas and
knowledge of personnel in the collaborating organisations, to begin collaborative
development of the framework.
• Ongoing redesign of the framework.
• Seeking feedback on the framework from diverse participants in two focus groups held in a
Queensland rural town with a total of eight women and three men and in a teleconference
that linked five women and three men in six rural and regional communities in Queensland,
10
most of whom were local coordinators of QOLN2, one of the collaborating organisations,
which provides support to distance education students.
• Making revisions to the framework based on this feedback.
This earlier research suggested a need to find new ways of making rural C&IT projects more
sustainable; a need to build community capacities in planning and evaluation; and a need to
take diversity, difference and inclusion into greater account in these processes. It also
suggested that the direct and more indirect impacts and ‘ripple effects’ of C&IT initiatives
needed to be identified, and that a ‘whole of community’ systems approach to analysis could
provide a more comprehensive understanding of these impacts. Participatory methods were
seen as valuable in undertaking these activities. This pilot research indicated that there had
been little formalised and coordinated planning, needs assessments and evaluation of C&IT
initiatives in rural and regional Queensland, and that ad hoc or informal processes were often
used.
Overview of the LEARNERS process and project
The LEARNERS project, which implemented and evaluated the trial of the framework in the
Tara and Stanthorpe Shires, was conducted from 2001 - 2004. Full details about the project are
provided in Lennie et al. (2004). The project commenced with a workshop involving the
research team and representatives of the project’s five collaborating partners which critically
assessed the original framework and recommended revisions. This revised framework, which
was subsequently labelled ‘the LEARNERS process’, is shown in Figure 1.
As discussed, the implementation and use of the LEARNERS process aimed to achieve the
following outcomes:
• the long-term sustainability of C&IT initiatives;
• increased collaboration and cooperation between people and organisations from various
sectors in rural communities;
• enhanced community capacity in participatory planning and evaluation methods;
• increased community participation and inclusion in planning and evaluation processes;
• giving value to empowering forms of leadership based on collaboration, sharing
information, networking, encouraging participation, and supporting people; and
11
• the development of learning communities.
The whole of community approach of the LEARNERS process encouraged the inclusion of a
broad diversity of people from different sectors of the community in planning and evaluating
the C&IT initiatives and collaboration and cooperation between community groups and
organisations from a diversity of sectors such as local government, education, and business.
The aim was to assess the direct and more indirect impacts of C&IT initiatives on different
groups and sectors of the community. Local project coordinators in each of the participating
communities were therefore encouraged to invite a broad diversity of community members and
representatives of a range of community organisations and groups to workshops and other
activities. They targeted people involved in or affected by the local C&IT initiatives that
community members chose to evaluate.
Figure 1 outlines the goals and outcomes of the LEARNERS process and the proposed PAR
cycles and steps involved. A document outlining possible strategies and questions to consider
at each of the nine steps involved was also prepared and distributed to participants to further
explain the process as well as a range of other relevant information and resources. This
information was placed on the LEARNERS project website: http://www.learners.qut.edu.au.
ADD FIGURE 1 ABOUT HERE
As well as using an inclusive, whole of community approach, the methods and processes
involved in implementing the ‘ideal’ LEARNERS process included:
• Identifying and building on existing community skills, knowledge and resources.
• Engaging in continuous PAR cycles of planning, acting, observing and critically reflecting
on the actions taken.
• Ensuring that all participants’ ideas, comments and feedback are included and taken into
account.
• Collaboratively planning and conducting evaluations that enable ongoing learning about
and constant improvements to C&IT initiatives.
• Gathering data that enables analysis of relevant differences such as gender, age, and levels
of skills, knowledge and access to C&IT.
12
Implementation of the LEARNERS process
Using feminist approaches to PAR (Gatenby and Humphries, 1996; Maguire, 1996), the
methodology used in the LEARNERS project involved:
• Drawing on local knowledge to identify key issues related to C&IT in the participating
communities and relevant contextual information about gender and leadership, community
networks, and other issues.
• Community participants collaboratively deciding on the C&IT initiatives which they
wanted to evaluate and, based on the outcomes of their evaluations, democratically
deciding on strategies that would improve their initiatives and increase community
participation and awareness.
• The research team and community participants collaboratively deciding on the format and
content of project activities such as workshops, and designing project activities in ways that
aimed to meet the diverse needs of participants and to be inclusive and empowering.
• Taking gender and power issues into account in designing and evaluating project activities.
• Community participants, the research team and staff of the collaborating industry partners
actively participating in project activities, including fieldwork and teleconferences, and
regularly sharing knowledge and information.
Communication processes included: holding regular meetings, workshops, teleconferences and
videoconferences involving community participants and collaborating partners; and sharing
information through two email discussion lists. Feedback on the project and suggested
revisions to the LEARNERS process were collected via email and the project’s online
discussion lists. Critical reflection workshops involving the research team, collaborating
partners, local project coordinators and interested participants were conducted both face to face
and via interactive conferencing technology in each year of the project.
While the participatory methodologies used in the project aimed to create an equal partnership
between the community participants, research team, collaborating partners and other
stakeholders, we recognised that this ideal was not easy to achieve in practice. We also
recognised that PAR requires sufficient time, energy and resources to be effective and that they
would not always be readily available to everyone involved in the project.
13
An ‘open enquiry’ approach to evaluation (Wadsworth, 1997) was used that involved regularly
seeking feedback from participants on what worked well, what did not work so well and how
various project activities could be improved, and taking this feedback into account in designing
future activities. Multiple qualitative and quantitative methods were used to evaluate project
activities and to undertake the meta evaluation and critique of the project. Evaluation methods
included: distributing short feedback forms and questionnaires at community workshops,
holding focus group discussions and semi-structured individual interviews with a diversity of
participants (ie men and women with different occupations, active and less active participants,
volunteers and paid workers etc), conducting participant observations of project activities, and
making entries in a fieldwork diary.
Community workshops used participatory methods such as small group discussions,
brainstorming and ranking of priority actions and strategies emerging from evaluations.
Information sheets about various forms of evaluation and the techniques used during the
workshops were prepared and distributed to workshop participants. They were later published
on the LEARNERS project website, along with a wide assortment of other evaluation
resources. Detailed notes from all community workshops and meetings were prepared and
distributed to all participants and collaborating partners.
Most of the qualitative data gathered during the project was entered into the QSR NVivo
program which was used to code text into broad themes and issues. The process of coding and
analysing the impacts of the project used the framework of rural women’s empowerment
described in Figure 1. The use of multiple participation, research and evaluation methods
provided a variety of rich research data and ongoing feedback. Together with the data coding
and analysis process, this enabled rigorous validation of the findings and the data analysis
results. To enhance the trustworthiness of the evaluation results, feedback and critical
comments were also sought from staff of collaborating partner organisations and community
participants on case studies, data analysis, and project reports and resources. Revisions were
made, based on this feedback.
14
Summary of the case studies
A brief summary of the case studies of the implementation of the LEARNERS process in the
Tara and Stanthorpe Shires is provided to provide some context for the analysis of project
impacts. The full case studies are found in Lennie et al. (2004).
When the project commenced, the larger Tara Shire had a small, highly scattered population of
approximately 3,800 people, while the smaller Stanthorpe Shire had a population of nearly
10,400 people. Elected leaders and senior staff of local Councils in both Shires were keen to
support and participate in the project. While the Tara Shire was considered to be a
disadvantaged community with a significant lack of services, particularly in communication,
the Stanthorpe Shire was more advantaged in terms of communication, education, housing and
other services. Both communities had already implemented new C&IT initiatives, including
community websites, computer and Internet training programs, and Learning Network
Queensland centres which provided support to distance education students.
Community leaders in the Tara Shire were interested in using the LEARNERS process to assist
the community to work together to reach its goals, and to engage in more effective planning
and evaluation. They hoped that participation in the trial might improve communication across
the Shire, as well as training and access to C&IT. Community leaders in the Stanthorpe Shire
were initially interested in finding ways to increase the sustainability of ‘GraniteNet’, a virtual
community project managed by the Shire Council that had established 80 online community
interest groups and enabled people to start their own interest group or build their own website.
In the Tara Shire, participants selected the Tara Shire Community Website, and IT training and
access across the Shire as the C&IT projects to be evaluated, while, following negotiations, the
core group in Stanthorpe elected to conduct an evaluation of the Lifelong Learning Group’s
website on GraniteNet as a pilot project.
In both communities, a larger number of people participated in initial workshops while a
smaller core group continued their involvement, including planning and conducting the
evaluations of local C&IT projects. The maximum number participating in workshops in Tara
was 23, while a maximum of 18 people took part in Stanthorpe workshops. As in the
community capacity building project reported by O’Meara, Chesters and Han (2004), involving
a wide diversity of community members and organisations was problematic. The majority of
15
active participants were women in the 40-59 age group with a white and/or Anglo-Celtic
ethnicity. Many participants worked in the areas of community development, education and
training or local government, in both paid and voluntary positions. Some participants in both
communities held formal leadership positions in local government, community or business
groups and organisations.
McKie (2003: 310) points out that the language and terms used in research and evaluation tend
to ‘exclude those who do not understand or use these terms’. Problems were initially
experienced in both communities with generating understanding about the project. The
terminology and methods used to explain the process were barriers to understanding for some
participants, particularly in the Tara community. While groups such as school teachers and
principals understood PAR and the notion of evaluation as an action learning process, it was
unfamiliar to other community members who were seeking more concrete outcomes and
greater direction from the research team.
A key outcome in the Stanthorpe Shire was the formation of a group who enthusiastically
commenced a related project that aimed to develop the Shire as a Learning Community.
However, many Stanthorpe participants became confused about the relationship between the
LEARNERS project and their local Learning Community project. This affected participation
and outcomes of the LEARNERS project in this area. Nevertheless, various positive impacts of
the project were reported in each community, including improved communication and
networking, and increased skills and knowledge in participatory planning and evaluation.
Revisions to the LEARNERS process
Feedback from community participants about their need for a simpler, easier to understand
version of the LEARNERS process and more case studies and examples, led the research team
to redesign the LEARNERS process from late 2003 to early 2004. The revised process that
emerged was a simple four step evaluation process with key questions and a comprehensive
case study of the whole process, based on outcomes from the LEARNERS project. Following
extensive discussions, the term ‘review’ was mainly used in preference to ‘evaluation’. The
four steps in the ‘EvaluateIT’ process are:
1. Plan the review.
16
2. Involve people in the review.
3. Do the review.
4. Review results and make the changes.
Details of these four steps were published with a variety of other information and evaluation
resources in the online EvaluateIT resource kit.3 Feedback on the prototype kit was sought
through questionnaires, workshops and focus groups, which were held in the participating
communities and in two other regional communities in North Queensland in July 2004 (see
Lennie et al., 2004). The final version of the kit is located at http://www.evaluateit.org.
Impacts of the LEARNERS project In addition to activities related to implementing the LEARNERS process, the research team
conducted an ongoing meta-process of evaluation, reflection, critique and analysis of the
LEARNERS project and its impacts on participants and their organisations and communities.
Impacts are understood as the changes that occur as a result of a project or intervention. They
include intended and unintended, positive and negative, empowering and disempowering
impacts. My earlier research in this field indicates that both empowering and disempowering
impacts and outcomes can be anticipated in PAR projects involving people from diverse
backgrounds who have different levels of status, power and knowledge, unequal access to
C&IT, and different goals and agendas (Lennie, 2001, 2002a; Lennie, Hatcher and Morgan,
2003). While empowerment and disempowerment were seen as separate and distinct for the
purpose of this analysis, this earlier research demonstrates that there is an inter-relationship
between these two processes.
Details of the multiple sources of qualitative and quantitative data on project impacts that were
used in this analysis and the data analysis and validation process are provided in Lennie et al.
(2004). The framework of empowerment and disempowerment used in the analysis is shown in
Figure 2. Most of the analysis reported below was obtained from feedback and information
provided by fifteen rural participants during interviews, focus group discussions and critical
reflection workshops. They comprised nine participants (eight women and one man) from
Stanthorpe Shire and six participants (four women and two men) from Tara Shire. Data from
workshop feedback questionnaires, fieldwork notes, and critical reflection workshops were
also included in the analysis.
17
ADD FIGURE 2 ABOUT HERE
Empowering impacts
The analysis indicated that participation in the project had a range of empowering impacts and
effects on participants in both communities. The social, technological, political and
psychological forms of empowerment that interviewees and other participants’ were
considered to have experienced are considered below.
Social empowerment
The analysis indicated that many participants in each community experienced various forms of
social empowerment, which is connected to the development of social capital (Woolcock,
Renton and Cavaye, 2004). The most significant forms of social empowerment identified were:
Gaining new knowledge or understanding of participatory planning and evaluation: The
majority of participants increased their knowledge and understanding in this area. Feedback on
participants’ use of the EvaluateIT kit indicated that it assisted in further developing
knowledge and understanding of participatory evaluation. The four step EvaluateIT process
was seen as simple and easy to understand for people without significant evaluation
experience, inclusive and flexible. Ongoing use of this resource is expected to continue helping
to increase community capacities in evaluation.
Increased evaluation and communication skills: Six interviewees (three women in Stanthorpe
and one woman and one man in Tara) considered that they or others had developed or
enhanced their skills in areas such as small group facilitation, brainstorming, and designing and
conducting evaluations. Some also improved their general skills in leadership, communication
and interaction.
Obtaining or sharing new information: Several participants and collaborating partners
indicated that they valued much of the information and resources provided during the project,
which was often forwarded to others via community websites and networks. This information
sharing was also seen as helping people ‘appreciate the value of other perspectives’.
Appying new knowledge, information and skills to other contexts: Nine participants (five
women and one man in Stanthorpe and two women and one man in Tara) reported that they
18
and others had applied their new or enhanced skills and knowledge in other contexts such as
their paid work in community development with youth or other community groups, or in their
voluntary positions in community organisations.
Networking with others: Seven participants (four women in Stanthorpe and two women and
one man in Tara) identified the opportunity to network and interact with others in their
community and elsewhere as a valuable outcome of the project. Networking among the
Learning Community Project Group in Stanthorpe via email and face to face meetings was
seen as a significant outcome of the project.
Participating in various group activities: Six interviewees (four women in Stanthorpe and one
woman and one man in Tara), as well as several workshop participants in both communities,
reported that they valued and often enjoyed the opportunity to take part in the various
groupwork activities such as workshops, meetings and videoconferences.
Other impacts related to social empowerment, mentioned by a smaller number of participants
and interviewees in both communities, included:
• The ‘better direction’ provided by the research team.
• The sense of collaboration and cooperation among community participants and the
‘community empowerment’.
• Feeling as if they were on an equal standing with the research team and industry partners
and other participants who were seen as having a higher status.
• The opportunity to reflect on the community and the issues raised in the project.
• Gaining a broader perspective on the community.
Technological empowerment
Eleven of the fifteen interviewees (nine women and two men) indicated that they and others in
their community had developed a new awareness or knowledge of new C&IT, or local C&IT
initiatives and issues. For example, staff and elected members of the Tara Shire Council
gained a significantly increased awareness and developed new ideas about the innovative use
of email, the Internet and other technologies. This led to the redesign of the Tara Shire website
and greater use of C&IT such as teleconferencing by the Council.
19
Eight interviewees (three Stanthorpe women, and three women and two men in Tara) also
provided examples of new uses, and greater use and confidence in using C&IT, or the
development of new skills in using various technologies. This was particularly evident in Tara.
Participants from both communities also gained useful skills, experience and knowledge in
using an interactive conferencing system to participate in three major LEARNERS project
events.
Political empowerment
While the project did not involve participants in providing a direct input into policy-making, it
produced political empowerment to some extent. Participation in workshops enabled
community members to ‘have their say’ about issues of concern, and to take collective action
on issues and the promotion or improvement of new C&IT initiatives that aimed to benefit the
community, or could help improve sustainability.
Participants in the Tara Shire saw the project as assisting in bringing the community together
to work towards a common goal. The formation of an active Learning Community Project
Group was considered as a significant outcome of the project by seven of the nine Stanthorpe
interviewees (six women and one man). This group has continued to meet on a regular basis to
plan and organise various lifelong learning activities.
Psychological empowerment
Self-confidence and self-esteem are essential first steps to empowerment (Anderson, 1996;
Claridge, 1996). The majority of participants appeared to have a fairly high existing level of
psychological empowerment. However, two less confident female participants reported an
increase in their level of confidence. Most participants also reported feeling very comfortable
participating in workshops and other activities as they were viewed as ‘non-threatening’,
‘informal, safe and respectful of the individuals participating’, and participants did not feel
pressured.
One of the critical reflection workshops that linked participants in Tara and Stanthorpe with the
research team and collaborating partners in Brisbane via interactive conferencing technology,
was seen as particularly empowering for participants in Stanthorpe. The local Stanthorpe
facilitator reported that this technology enabled participants to feel more ‘free to speak their
20
minds’ about what had not worked well in the project. This process was seen as ‘really
effective’.
Disempowering impacts
While the analysis suggested that the project resulted in a wide range of empowering and
positive impacts and effects, interviewees also reported some disempowering, negative or
unintended effects. Some of these effects appeared to have been experienced to a greater extent
by participants in Stanthorpe than in Tara. This appeared to be due to a number of factors,
particularly the initial misunderstandings and confusion about the project and its relationship to
the Stanthorpe Learning Community project.
Social disempowerment
The two most significant areas of social disempowerment identified by several participants in
both communities were:
• Not gaining sufficient knowledge or understanding about the project and the LEARNERS
process. Factors included the unfamiliar language that was used, the perceived complexity
of the diagram and other information that was used to explain the LEARNERS process, and
the purpose of the project being open to multiple interpretations.
• Initial confusion about the purpose of the project, which was greater in Stanthorpe than in
Tara.
In addition, the ongoing, cyclical nature of the LEARNERS process did not meet the needs of
some participants who were more ‘outcome driven’. Others were disappointed about the
perceived lack of immediate outcomes from the project. Understanding about the project
increased over time for those who remained active participants but (along with factors such as
lack of time, relevance and interest and living at a distance from workshop venues and lack of
initial understanding) was likely to have been a factor in several of the early participants’
decision not to continue involvement. These negative effects could have contributed to some
participants considering that the LEARNERS process did not meet their needs and/or that there
was a lack of outcomes.
21
Technological disempowerment
Participants also experienced various forms of technological disempowerment during the
project at times. However, some of these experiences were not directly related to project
activities. The main issues reported were:
• Frustration due to technological problems encountered while participating in some
workshops and meetings via conferencing technology.
• A lack of confidence with using computers, email and conferencing technologies which
were unfamiliar to some participants.
• Limited access to computers and email facilities.
• Lack of local IT training and support, which was a particular issue in the Tara Shire,
following the departure of the IT Support Officer, who was not replaced by the Council.
Political disempowerment
Some participants and project partners thought that certain project activities had been
controlled by particular participants and stakeholders, while one woman reported that she and
others in her community had experienced a lack of influence and voice. These two effects
appeared to result from different perceptions about ownership of the project and control of the
agenda, and other factors. The control of activities was considered to have impacted on the
effectiveness of the project in one of the communities in particular.
One reason for these outcomes was the pre-existing formal and informal networks that existed
between key stakeholders within one of the communities. These networks and alliances
affected community perceptions of the project, especially about who controlled its agenda and
what outcomes were expected. One of the learnings from the project was that such issues need
to be dealt with in a proactive but sympathetic manner through strategies such as choosing
local ‘champions’ and coordinators carefully and personally approaching key leaders and
stakeholders in the community (Lennie et al., 2004: 53-54).
Psychological disempowerment
A few participants reported experiencing a lack of confidence to participate in project activities
at times. Some participants also experienced frustration due to factors such as difficulties in
22
attracting new people, the confusion about the project, or having a lack of time and capacity to
participate. One participant also expressed concerns about the perceived value of her
contribution as an unpaid volunteer, compared with those in paid community development
roles.
Conclusions: The strengths and limitations of participatory evaluation
This paper has provided some critical insights into the complexity of building community
capacities in evaluating rural C&IT projects and the unintended or contradictory outcomes that
can emerge. The research team’s critical reflections on the LEARNERS project and process,
and our analysis of the impacts of the project, suggest that both the project and the
participatory evaluation process had several strengths, but also some limitations, which are
summarised in Table 1. Several barriers to community participation and inclusion are also
identified in Table 1.
Table 1: Strengths and limitations of the LEARNERS process and project
Strengths Limitations
Builds on the existing community strengths and capacities, including skills, knowledge, social capital, and human and technological infrastructure.
Greater time and resources were required to adequately identify the existing skills and knowledge of community members and to encourage their involvement. Loss of key community champions affected progress.
The process was effective in producing various forms of empowerment and building capacities in participatory planning and evaluation. Some participants transferred these skills to other contexts.
The project had some unintended and disempowering impacts on participants which affected its success. Capacity building was limited to a small core group of participants in each community.
The involvement of a broad diversity of community members and groups was encouraged. The process encouraged consideration of the needs of various groups, such as older and younger people, people in remote townships, and from non English speaking backgrounds.
Obtaining ongoing participation and involvement from the whole diversity of community members and sectors was problematic. The process attracted mainly women, people involved in community development and education, and reflective learners.
The processes used were flexible, open and transparent and could be readily adapted to meet the needs of different community groups and projects. The participatory methods used in the project were seen as applicable to other non-C&IT projects.
The LEARNERS process was initially considered complex and confusing by many participants. In addition, a paradox of the process is that it is not interventionist enough. This can result in the agenda being dominated by particular community members and groups.
The processes used in the project helped the community groups better develop and maintain a focus, define clearer objectives, and decide on priorities for action. The LEARNERS process
Participatory evaluation and PAR methods appear to fit better with some people’s values, needs and agendas than with others. Some participants preferred a more outcome-oriented process.
23
provided a useful framework for evaluation, planning, needs and impact assessments and critical reflection. All input from participants was considered valuable.
The process helped community groups to maintain interest and motivation through the positive feedback and support from other interested community leaders and members and the research team, which was provided through the evaluation process.
Planning, organising, conducting and evaluating project activities was very time consuming. The process required a commitment to regularly participate in activities and to develop new ideas for improving C&IT projects and generating community interest. There was some dependence on the research team to facilitate activities and encourage action.
Conferencing technologies and email were effectively used in the project to critically reflect on the project, maintain communication and provide information.
Poor quality technological infrastructure and lack of C&IT access in the communities affected participation and communication at times.
The project’s rigorous ongoing evaluation process provided a strong test of theories about community empowerment, inclusion and participation and the role of C&IT in these processes. The use of gender analysis highlighted the important leadership roles of rural women.
The funding, time and resources required to effectively undertake the ongoing evaluation and impact assessment process was limited. This created difficulties in effectively meeting all the project objectives and deadlines, and providing adequate training and support to participants.
The project generated good mutual learning and understanding for the community participants, industry partners and the research team.
Only the participants and industry partners who were actively involved for the duration of the project developed new learnings and understanding.
As this table indicates, the ideal LEARNERS framework and the reality of implementing the
process were somewhat different. The framework provided a flexible and inclusive whole of
community process for participation in the evaluation of rural C&IT projects. It built on
existing community capacities, resources and goals, generated mutual trust and understanding
between the diverse groups involved, and produced various forms of empowerment. The
participatory evaluation and PAR processes that were used assisted in identifying strategies to
increase the sustainability of C&IT projects in the two communities. Using a range of new
technologies to communicate with and support distant participants assisting in producing
various empowering impacts. Outcomes of the project were particularly positive in the Tara
Shire which was considered to be a disadvantaged community with a high level of need for
improved communication systems and better community networking and cohesion.
However, due to inequalities in power and knowledge, the different values and agendas of the
participants and researchers, the pre-existing relationships and networks within the
communities, and other complex issues, the project also had a number of unintended and
disempowering impacts. They included a perceived lack of ownership and control of some
24
project activities, and confusion and misunderstandings about the project and the LEARNERS
process. Several other barriers to community participation, empowerment and capacity
building were identified, such as a lack of time and/or capacity to participate, the loss of key
‘champions’ in the community, and lack of access to or limited experience with C&IT. Similar
barriers have been found in other community-based projects (Boyce, 2001; Lennie, 2002a;
O’Meara et al., 2004).
Many of these limitations and issues have also been identified in the participatory evaluation
literature. However, as Gregory (2000) points out, the central issue of power is often neglected
by those using participatory evaluation methodologies. This paper highlights the need for a
more open, honest, critical and reflexive approach to evaluating claims for the empowering
impacts of evaluation capacity building projects. This approach requires the rigorous analysis
of both the intended and unintended impacts of such projects. Gender and power issues also
need to be taken into account in such evaluations, particularly issues related to leadership,
communication and control. The framework of empowerment and disempowerment that was
used to analyse the impacts of the LEARNERS project was considered to provide a useful
means of conducting such an analysis. A more in-depth analysis of the gendered power
relations enacted in the project, using feminist discourse analysis methods, was originally
planned, but became beyond the scope of the project.
The outcomes of the LEARNERS project indicate that participatory evaluations and PAR are
political processes that can have unintended impacts and effects. These methodologies also fit
better with some people’s values and needs than with others (Lennie and Hearn, 2003). The
project mainly attracted women, people involved with community development and education,
and reflective learners. The dominance of women in the project was unsurprising, given that
women in Australian rural communities are often very active in community development
activities and have been identified as early adopters of C&IT (The Rural Women and ICTs
Research Team, 1999). However, questions about the ‘gendered’ nature of the participatory
evaluation methodology arose during the project which may need to be taken into account in
other capacity building projects that aim to involve a diversity of participants.
McKie (2003: 320) argues that ‘Whether we like it or not evaluation has created a language
and modus operandi that can be excluding’. Developing effective strategies for including a
diversity of community members in participatory evaluations clearly requires that issues
25
related to the appropriateness of the language used and the perceived value and relevance of
participation and evaluation to various groups are addressed. Otherwise some participants may
see evaluation as a judgemental process that could have negative implications for their
employment or the funding of their projects, rather than an ongoing learning process that can
help to improve projects in ways that better meet community goals and needs. Our research
suggests that the promotion and use of practical and user friendly resources such as the
EvaluateIT kit could be a valuable means of demystifying participatory evaluation and PAR
and involving a broader diversity of community groups and stakeholders in evaluations.
Notes
1. This project was a collaboration between three QUT researchers (Dr June Lennie, Dr Roy
Lundin and Lyn Simpson) and senior staff of the Queensland Open Learning Network
(QOLN) and the Queensland Department of Employment, Training and Industrial
Relations. It was funded through a QUT/Industry Collaborative Research Grant.
2. QOLN is now known as Learning Network Queensland.
3. A significant contribution to the development of the EvaluateIT resource kit was provided
by a research project funded by the State Library of Queensland and QUT through a QUT
Strategic Links with Industry grant. The design of the EvaluateIT website was funded by a
QUT Community Service grant. I acknowledge and thank Dr Mary Hanrahan for her
valuable assistance in developing the EvaluateIT kit and conducting the community
consultations.
Acknowledgements
The LEARNERS project was funded and supported by the Australian Research Council, the
Commonwealth Department of Family and Community Services, Learning Network
Queensland, the Queensland Department of Natural Resources and Mines, the Office for
Women in the Queensland Department of Local Government, Planning, Sport and Recreation,
and Legal Aid Queensland. I acknowledge and thank the other Chief Investigators in the
project, Professor Greg Hearn and Associate Professor Lyn Simpson and the project’s
Research Assistants Kitty van Vuuren and Emma Kennedy da Silva. I also acknowledge and
26
thank the people from the Tara and Stanthorpe Shires and the collaborating partners who
participated in the project and provided valuable feedback and input.
References
Anderson, J. (1996) ‘Yes, but is it empowerment? Initiation, implementation and outcomes of
community action’, in B. Humphries (ed). Critical Perspectives on Empowerment, pp 69–83.
Birmingham: Venture Press.
Boyce, W. (2001) ‘Disadvantaged persons’ participation in heath promotion projects: Some
structural dimensions’, Social Science and Medicine 52: 1551-1564.
Boyle, R. and D. Lemaire, (eds.) (1999) Building Effective Evaluation Capacity. London:
Transaction Publishers.
Brunner, I. and A. Guzman (1989) ‘Participatory evaluation: A tool to assess projects and
empower people’, in R.F. Conner and M. Hendricks (eds.), International Innovations in
Evaluation and Methodology: New Directions for Program Evaluation, pp.9-17. San
Francisco: Jossey-Bass.
Chess, C. (2000) ‘Evaluating environmental public participation: Methodological questions’,
Journal of Environmental Planning and Management 43(6): 769-784.
Claridge, C. (1996) Women, Development and the Environment: A Method to Facilitate
Women’s Empowerment, PhD thesis, Brisbane: Department of Agriculture, The University of
Queensland.
Da Rin, J. and J. Groves (1999) Demand and supply of Internet content for Australian farm
businesses, Canberra: Rural Industries Research and Development Corporation.
Department of Communications, Information Technology and the Arts (DCITA). (2003)
Maintaining the viability of online access centres in regional, rural and remote Australia,
Discussion paper.
(http://www.dcita.gov.au/tel/regional,_ rural_and_remote_communications/?a=7789)
Deutchman, I. (1991) ‘The politics of empowerment’, Women and Politics 11(2): 1–18.
27
Diez, M.A. (2001) ‘The evaluation of regional innovation and cluster policies: Towards a
participatory approach’. European Planning Studies, 9 (7): 907-923.
Dugan, M.A. (1996) ‘Participatory and empowerment evaluation: Lessons learned in training
and technical assistance’, in D. Fetterman, S. Kaftarian and A. Wandersman (eds.)
Empowerment Evaluation: Knowledge and Tools for Self-Assessment and Accountability, pp.
277-303. Thousand Oaks, California: Sage.
Faris, R. (2001) Learning Communities: Villages, Neighbourhoods, Towns, Cities and Regions
Preparing for a Knowledge-Based Society. (http://members.shaw.ca/rfaris/docs/LCdigest.pdf)
Fetterman, D., S. Kaftarian, and A. Wandersman, eds (1996) Empowerment Evaluation.
Thousand Oaks: Sage.
Friedmann, J. (1992) Empowerment: The Politics of Alternative Development. Cambridge:
Blackwell.
Garaway, G.B. (1995) ‘Participatory evaluation’, Studies in Educational Evaluation 21 (1): 85-
102.
Gatenby, B. and M. Humphries (1996) ‘Feminist commitments in organisational
communication: Participatory action research as feminist praxis’, Australian Journal of
Communication 23 (2): 73–87.
Geiselhart, K. (2004) The Electronic Canary: Sustainability Solutions for Australian
Teleservice Centres, Wangaratta, Victoria: Centre for Online Regional Research.
(http://www.teleservices.net.au/CTSA_Viability_Report%20Final.pdf)
Gore, J. (1992) ‘What we can do for you! What can “we” do for “you”?: Struggling over
empowerment in critical and feminist pedagogy’, in C. Luke and J. Gore (eds) Feminisms and
Critical Pedagogy, pp.54–73. New York: Routledge.
Grace, M. and J. Lennie (1998) ‘Constructing and reconstructing rural women in Australia: the
politics of change, diversity and identity’, Sociologia Ruralis 38(3): 351–370.
Gregory, A. (2000) ‘Problematising participation. A critical review of approaches to
participation in evaluation theory’, Evaluation 6 (2): 179-199.
28
Groves, J. and J. Da Rin (1999a) Buying and Selling Online: The Opportunities for Electronic
Commerce for Australian Farm Businesses, Canberra: Rural Industries Research and
Development Corporation.
Groves, J. and J. Da Rin (1999b) Economic and Social Impacts of Farm Internet Use,
Canberra: Rural Industries Research and Development Corporation.
Guba, E. and Y. Lincoln (1989) Fourth Generation Evaluation. Newbury Park: Sage.
Harrison, L. (1998) ‘Using community learning to identify stores of social capital: Have we
found the “right” community?’, in Learning Communities, Regional Sustainability and the
Learning Society, Vol II. Launceston: Centre for Research and Learning.
Hearn, G., M. Kimber, J. Lennie and L. Simpson (2005) ‘ICTs and regional sustainability:
Critical reflections using action research’, Journal of Community Informatics, 1 (2): 18-31.
Hudson, H.E. (2001) ‘The Acacia programme: Developing evaluation and learning systems for
African Telecentres’, in C. Latchem and D. Walker (eds.) Telecentres: Case Studies and Key
Issues. Vancouver, Canada, The Commonwealth of Learning
(http://www.col.org/telecentres/chapter%2015.pdf).
Humphries, B. (1994) ‘Empowerment and social research: Elements for an analytic
framework’, in B. Humphries and C. Truman (eds) Re-Thinking Social Research. Anti-
Discriminatory Approaches in Research Methodology, pp.185–204. Aldershot: Avebury.
Kellogg Foundation (1998) Chapter 2. Evaluation Handbook.
(http://www.wkkf.org/Publications/evalhdbk/)
Khan, M. (1998) ‘Evaluation capacity building’, Evaluation 4 (3): 310-328.
Lather, P. (1991). Getting Smart. Feminist Research and Pedagogy With/in the Postmodern.
New York: Routledge.
Lennie, J. (2001) Troubling Empowerment: An Evaluation and Critique of a Feminist Action
Research Project Involving Rural Women and Interactive Communication Technologies. PhD
thesis. Brisbane: The Communication Centre, Faculty of Business, Queensland University of
Technology.
29
Lennie, J. (2002a) ‘Rural women’s empowerment in a communication technology project:
Some contradictory effects’, Rural Society 12 (3): 224-245.
Lennie, J. (2002b) ‘Including a diversity of rural women in a communication technology
access project: Taking the macro and micro contexts into account’, in G. Johanson, and L.
Stillman (eds.), Proceedings, Electronic Networks - Building Community: 5th Community
Networking Conference, Monash University, Melbourne, 3-5 July 2002.
Lennie, J., C. Hatcher and M. Morgan (2003) ‘Feminist discourses of (dis)empowerment in an
action research project involving rural women and communication technologies’, Action
Research, 1(1): 57-80.
Lennie, J. and Hearn, G. (2003) ‘The potential of PAR and participatory evaluation for
increasing the sustainability and success of community development initiatives using new
communication technologies’. Proceedings, Action Learning, Action Research & Process
Management and Participatory Action Research Congress, University of Pretoria, South
Africa, 21-24 September, 2003.
http://www.education.up.ac.za/alarpm/PRP_pdf/Lenny&Hearn.PDF)
Lennie, J., G. Hearn, L. Simpson, E. Kennedy da Silva, M. Kimber and M. Hanrahan (2004)
Building Community Capacity in Evaluating IT Projects: Outcomes of The LEARNERS
Project, Brisbane: Creative Industries Research and Applications Centre and Service
Leadership and Innovation Research Program, Queensland University of Technology.
(http://www.learners.qut.edu.au/reports.php)
Lennie, J., R. Lundin, and L. Simpson (2000a) Development of a Participatory Evaluation,
Assessment and Planning Framework for Sustainable Rural Community Development That
Uses Interactive Communication Technologies. Brisbane: The Communication Centre,
Queensland University of Technology.
Lennie, J., R. Lundin, and L. Simpson (2000b) A Participatory Evaluation, Assessment and
Planning Framework for Sustainable Rural Community Development That Uses Interactive
Communication Technologies. Brisbane: The Communication Centre, Queensland University
of Technology.
30
Maguire, P. (1996) ‘Proposing a more feminist participatory research: Knowing and being
embraced openly’, in K. de Koning and M. Martin (eds.) Participatory Research in Health.
Issues and Experience, pp.27–39. London and New Jersey: Zed Books.
Mannion, J. (1996) ‘Partnership, participation and capacity building: Rural development based
on local bottom-up strategies’, Leader Magazine 12, (October): 6-10.
Martin, M. (2000) ‘Critical education for participatory research’, in C. Truman, D. M. Mertens
and B. Humphries (eds.) Research and Inequality, pp.191-204. New York: UCL Press.
McKie, L. (2003) ‘Rhetorical spaces: Participation and pragmatism in the evaluation of
community health work’, Evaluation 9 (3): 307-324.
McTaggart, R. (1991) ‘Principles for participatory action research’, Adult Education Quarterly
41(3): 168-187.
Mobbs, R. (1998) ‘The self empowerment of communities: A systems strategy for rural
community assessment (SSRCA)’, in Learning Communities, Regional Sustainability and the
Learning Society. Vol II. Launceston: Centre for Research and Learning.
O’Meara, P., J. Chesters and G-S Han (2004) ‘Outside - looking in: Evaluating a community
capacity building project’, Rural Society, 14(2): 126-141.
O’Sullivan, J.M. and R.G. O’Sullivan (1998) ‘Evaluation voices: Promoting evaluation from
within programs through collaboration’, Evaluation and Program Planning 21(1): 21-29.
Papineau, D. and M. Kiely (1996) ‘Participatory evaluation in a community organization:
Fostering stakeholder empowerment and utilization’, Evaluation and Program Planning 19
(1): 79-93.
Rebien, C. (1996) ‘Participatory evaluation of development assistance’, Evaluation 2(2): 151-
171.
Stanley, L and S. Wise (1990) ‘Method, methodology and epistemology in feminist research
process’, in L. Stanley (ed.) Feminist Praxis. Research, Theory and Epistemology in Feminist
Sociology, pp.20-62. London: Routledge.
31
The Rural Women and ICTs Research Team (1999) The New Pioneers: Women in Rural
Queensland Collaboratively Exploring The Potential of Communication and Information
Technologies for Personal, Business and Community Development. Brisbane: The
Communication Centre, Queensland University of Technology.
Thomas, L. (2000) ‘Bums on seats; or “Listening to Voices”: Evaluating widening
participation initiatives using participatory action research’, Studies in Continuing Education
22 (1): 95-113.
Vanderplaat, M. (1995) ‘Beyond technique. Issues in evaluating for empowerment’,
Evaluation, 1 (1): 81-96.
Wadsworth, Y. (1997) Everyday Evaluation on the Run. St Leonards, New South Wales:
Allen and Unwin.
Wadsworth, Y. (1998) What is participatory action research?
(http://www.scu.edu.au/schools/gcm/ar/ari/p-ywadsworth98.html)
Woolcock, G., D. Renton and J. Cavaye (2004) What Makes Communities Tick? Local
Government & Social Capital Action Research Project. Ipswich, Queensland: Community
Service & Research Centre, The University of Queensland and Local Government Association of
Queensland. (http://www.uq.edu.au/csrc/lcandsc/final%20report.pdf)
Worthen, B., J. Sanders and J. Fitzpatrick (1997) Program Evaluation: Alternative Approaches
and Practical Guidelines. 2nd ed. New York: Longman.
32
Key goals/ outcomes
Partnerships/
collaborations/
cooperation
Enhanced
community
capacity
Long-term
sustainability
of C&IT
initiatives
Community
empowerment
Value new
forms of
leadership
Develop
learning
communities
Increased
community involvement,
ownership participation &
inclusion
Key methods
Participatory action
research and action
learning
Participatory
evaluation and
assessment of needs
and impacts
Ongoing, in-built
evaluation and
assessment processes
Whole of community
systems approach to
analysis and
planning
Analysis of differences
(gender, age, ethnicity,
ability etc.)
Process#
*
Analyse skills development and learning needs
Identify and engage
stakeholders
Assess evaluation
learning needs
Assess strengths, resources & information
needs
Share learnings and outcomes
Design programs and processes for
evaluation & assessment
Implement evaluation &
impact assessment
Begin learning sessions to use
LEARNERS process
Plan
Reflect Act
Observe
Review and redesign
processes and initiatives
Existing community strengths and resources
33
# All of these steps do not have to be followed and do not have to be undertaken in this sequence. They represent activities that could potentially be undertaken by stakeholders and participants. * Each of the steps involves a cycle of planning, acting, observing and reflecting
Figure 1: The LEARNERS Process
34
Figure 2: The framework of empowerment used to analyse the impacts of the
LEARNERS project (from Lennie, 2001)
Social empowerment
Political empowerment
Technological empowerment
Psychological empowerment
• Having a voice and being listened to • Participating in policy making • Taking action to change your life or your
community • Networking and lobbying • Changing stereotypes about rural women
• Knowledge about ICTs • Awareness and understanding about ICTs • Skills and competence in using new ICTs • Ongoing support and advice in using ICTs • Access to high quality technologies • Confidence to use and speak about ICTs
• Self confidence and self esteem • Feeling more valued and respected • Motivation, interest and enthusiasm • Freedom to do things or express yourself • Feelings of belonging • Wellbeing and happiness
• New knowledge and information • Awareness and understanding of issues • Skills, abilities and competence • Support, friendship and inspiration • Participating in group activities with women • Networking