Final report 2016
Professor Susan Ash, external evaluator
Transforming competency-based assessment in nutrition and dietetics
2
Support for the production of this report has been provided by the
Australian Government Office for Learning and Teaching. The views
expressed in this report do not necessarily reflect the views of
the Australian Government Office for Learning and Teaching.
With the exception of the Commonwealth Coat of Arms, and where
otherwise noted, all material presented in this document is
provided under Creative Commons Attribution- ShareAlike 4.0
International License
http://creativecommons.org/licenses/by-sa/4.0/.
The details of the relevant licence conditions are available on the
Creative Commons website (accessible using the links provided) as
is the full legal code for the Creative Commons
Attribution-ShareAlike 4.0 International License
http://creativecommons.org/licenses/by-sa/4.0/legalcode.
Requests and inquiries concerning these rights should be addressed
to: Office for Learning and Teaching Department of Education and
Training
GPO Box 9880, Location code N255EL10 Sydney NSW 2001
<
[email protected]>
2016
Transforming competency-based assessment in nutrition and dietetics
3
Acknowledgements I acknowledge the support provided throughout this
Fellowship by Associate Professor Margaret Hay, Monash University,
Associate Professor Phillip Dawson, Deakin University and Professor
Kevin Whelan, Kings College London. I am thankful for the support
of my colleagues at Monash University Department of Nutrition and
Dietetics: Janeane Dart, Simone Gibson, Evelyn Volders, Andrea
Bryce, Judi Porter, Liza Barbour and Amanda Anderson, all of whom
contributed to the success of this Fellowship. I would also like to
thank Professor Helen Truby for allowing me to pursue this
endeavour and Sonia Bartoluzzi for her professional editing
service.
Transforming competency-based assessment in nutrition and dietetics
4
Executive summary Dietitians are the recognised professionals best
placed to improve the nutrition of individuals and populations. A
competent nutrition and dietetics workforce is essential to
maintain and improve the health of Australians. Quality teaching
and learning are essential to produce a competent workforce.
Dietitians, like other health professionals, undertake learning in
workplaces across the health care system and other settings. The
development of competence occurs and is assessed within the
university setting as well as within these workplaces.
Competency-based assessment is made challenging by increasing
student numbers, work pressures on assessors in the workplace,
varying expectations and lack of consistency over what constitutes
‘competency’. Improving the quality of assessment will lead to a
dietetics workforce that is better equipped to meet the nutrition
and health needs of the population and one that takes
responsibility for lifelong learning.
This Fellowship program of activities aimed to transform the way
competency-based assessment is viewed within the field of nutrition
and dietetics. Using Monash University’s Master of Dietetics
program as an exemplar, it set out to design a systems-based
approach to competency-based assessment. It also established a
Community of Practice to support dietetics educators to work
towards best-practice assessment. The Fellowship focused on
conceptualising competency-based assessment as a system with
programs of assessment. It defined ‘best-practice assessment’ as
being authentic, having defined performance criteria, involving
multiple and adequately skilled assessors, involving multiple
methods and embedding regular effective feedback. The Fellowship
was a timely investment in the discipline, given the recent
revision and release of National Competency Standards for
Dietitians (2015), led by the Fellow, and revisions to the
Australian Dietetics Council accreditation standards.
Monash University system-based assessment approach Staff (n=18) and
recent graduates (n=3) from the Department of Nutrition and
Dietetics at Monash University attended a workshop on 4 December
2014 that aimed to build participants’ skills and knowledge in
assessment and to commence discussions on how the department could
work towards ‘better’ practice in competency-based assessment. This
workshop identified the need for stakeholders to see assessment as
a whole, and as part of a system. Subsequently, the Fellow worked
with a small subset of stakeholders (2 academics and 2
practitioners) to develop a systems-based approach to assessment.
An illustration was developed that mapped all competency-based
assessment tasks within the Master of Dietetics, the key skills
developed by each task and their place within a system. Feedback
was sought through discussions with academic staff (n=7) and
placement educators (n=6) to further refine the systems map
illustration. The map provided a mechanism for all academics and
practitioners involved in assessment of competence at Monash
University to visualise how multiple and varying pieces of evidence
contribute to the final judgement of competence, and to see their
own role within the system of assessment. The map has been used at
a number of educator’s forums to build the capacity of educators to
see the role of workplace assessment and the multiple evidence that
informs judgement of competence.
Transforming competency-based assessment in nutrition and dietetics
5
Community of Practice Dietitians with a key stake in the education
and preparation of dietitians in Australia were invited to
participate in a Community of Practice. Twenty-one academics from
14 universities across Australia (of the 16 accredited at the time
of the study) and one from New Zealand participated. The Community
of Practice members initially met face-to-face in a one-day
workshop on 13 February 2015 in Brisbane. They agreed that the aims
of the network were to develop the skills, knowledge and attitudes
of individual members, to transform their methods of assessment and
to lead change. The network then met regularly every six-to-eight
weeks for approximately 1.5 hours via videoconference facilitated
by the Fellow for a period of twelve months. These sessions focused
on discussions around competency-based assessment practice,
accreditation standards, revised national competency standards,
challenges associated with changing assessment and dietetic
placement educators’ approaches. The network also designed and
developed a research project to explore students’ perceptions of
competency standards and competency-based assessment.
A pre-post participation survey, together with the ‘most
significant change’ technique, was used to evaluate the Community
of Practice. The findings of this evaluation suggest that the
Community of Practice has changed members’ thinking and knowledge
around assessment and members have been empowered to make changes
to redesign assessment systems. In particular, there was an
improvement in members’ approaches to integrating circular feedback
into the program of assessment, taking a scholarly approach to
assessment and ensuring that certification portrays students’
achievements and future learning. Participants valued the network
of like-minded people and safe space in which to share ideas and
approaches.
The Community of Practice members have partnered to design a
research project that aims to explore how students engage with and
understand competency standards and competency-based assessment.
Data collection commenced in October 2015.
This Fellowship has commenced an important dialogue on
competency-based assessment in nutrition and dietetics and has
directly changed the perspectives of at least 40 key stakeholders
in competency-based assessment. The timing of the Fellowship
alongside the release of National Competency Standards for
Dietitians has created momentum for continued development to
improve competency-based assessment for the profession. There is a
need for continued conversation to support change to assessment
practice in nutrition and dietetics and other health professions,
so that students are placed at the centre of assessment with
multiple pieces of evidence used to inform judgement of competence.
The products of this fellowship will facilitate continued dialogue.
This will support the development of health professionals better
equipped for the future with a stronger commitment and capacity for
life-long learning.
Transforming competency-based assessment in nutrition and dietetics
6
External Evaluation The intended outcomes of the project have been
achieved in the move of Monash University to integrated assessment
via a mapping process involving students and benchmarked standards,
and the development of a trans-Tasman and national Community of
Practice of university staff responsible for competency-based
assessment. Further outcomes include the Community of Practice
working on a research project which will develop new/revised
assessment methods across with work-based practice settings and in
the process will transform competency-based assessment practice in
the profession of dietetics. The Fellow has been recognised for her
outstanding leadership in this area and as such, has been engaged
by the Australian Dietetic Council of the Dietitians Association
Australia to advise on best-practice competency assessment and
transforming the process for dietetic course accreditation. Perhaps
the most significant outcome from the Fellowship has been the
Fellow’s ability to bring all parties together to consider
assessment in an evidence based fashion and to promote further
research in the area, which will have resonance beyond the
profession of dietetics. A workshop was conducted at the Dietitians
Association of Australia National conference in Perth in May 2015.
The Fellow has been invited to deliver key note presentations and
workshops the Dietitians Association of Australia National
conference in Melbourne, May 2016 and International Congress of
Dietetics in Granada, Spain, September 2016. A number of
publications have been submitted from this work.
Transforming competency-based assessment in nutrition and dietetics
7
Table of contents
Developing a systems-based approach to assessment
....................................................... 11
Creating a Community of Practice for dietetics educators
.................................................. 12
Evaluation of the Community of Practice
........................................................................
13
Creating momentum for change across the other health professions
............................... 14
Discussions with critical friends and partners
.....................................................................
14
Chapter 3 Findings and evaluation
..........................................................................................
16
Monash University Master of Dietetics systems map
......................................................... 16
Community of Practice for dietetics educators
...................................................................
19
Community of Practice participants
................................................................................
19
Community of Practice sessions
......................................................................................
19
Evaluation of the Community of Practice
........................................................................
21
Chapter 4 Discussion and conclusions
.....................................................................................
24
References
...............................................................................................................................
26
Tables and figures
Tables
Table 1: Summary of Master of Dietetics curriculum and assessment
tasks .......................... 17
Table 2: Demographics of members and attendance at Community of
Practice ................... 19
Table 3 Community of Practice members self-reported assessment
practice before and after participation
.....................................................................................................................
23
Figures
The aim of the Community of Practice is to utilise the wisdom of
dietetics educators to transform their methods of assessment and
lead change. As well as developing the skills, knowledge and
attitudes of individual members we aspire to achieve the following
outcomes from working
together:...................................................................
13
• Support revision of accreditation standards for and with
Australian Dietetics Council . 13
• Support the dissemination of revised Competency Standards
....................................... 13
• Develop a guide/resource for application of Competency Standards
to competency- based assessment
............................................................................................................
13
• Create an evidence-based approach to assess competence – examples
for pieces of assessment and link to Competency Standards
..............................................................
13
• Collection of data on assessment – credibility, dependability,
appropriateness, cost, student friendliness
.........................................................................................................
13
• Obtain feedback from students on revised Competency Standards
............................... 13
Transforming competency-based assessment in nutrition and dietetics
9
• Pilot panel (multiple stakeholders) decision as part of a system
for competency-based assessment
.......................................................................................................................
13
• Collaborative research and publications
.........................................................................
13
Figure 1 Outcomes Community of Practice members hoped to achieve.
............................... 13
Figure 2 Monash University Master of Dietetics competency-based
assessment systems map.
.................................................................................................................................
18
Transforming competency-based assessment in nutrition and dietetics
10
Chapter 1 Introduction Optimal nutrition of the population offers
perhaps the greatest potential for improving health.1 Nutrition and
dietetics professionals are the recognised experts in human
nutrition science, and are best placed to improve the nutrition of
individuals and populations. Producing a competent nutrition and
dietetics workforce is essential to maintain and improve the health
of Australians. Developing the nutrition and dietetics profession
to meet workforce needs is an ongoing challenge for Australian
universities that is mirrored internationally.
Health, disease and the health care system are complex and
ever-changing domains of practice, requiring an adequately equipped
health workforce. For dietitians, the National Competency Standards
provide a framework for the preparation of dietitians for entry
into the workforce in Australia.2 In Australia dietitians are
professionally prepared to practice medical nutrition therapy
together with food service management and community/public health
nutrition, which includes health promotion, nutrition policy and
improving the food supply, especially for vulnerable groups.
One of the most challenging areas in preparing the workforce for
practice is competency- based assessment.3 This is due to:
increasing demands for work-based learning experiences, work
pressures on assessors, varying expectations, lack of consistency
in what constitutes competency, and the need for assessors to
embrace subjectivity in making judgement on performance.4-6 In
addition, assessment of competence in dietetics has not
traditionally been viewed as a system; rather, the focus has been
on individual methods of assessment in the workplace setting.
Best-practice assessment requires clearly defined performance
criteria, multiple and adequately skilled assessors, multiple
methods, authenticity and effective feedback.7 While some allied
health profession disciplines have developed tools for reliable and
valid assessment,8, 9 few describe fully effective and functional
assessment systems. Dietetics uses a range of methods to assess
competence across the various settings, but there is a need to
build the capacity of universities and dietitians involved in
assessment and to support the profession to view assessment as a
system, and one whereby students take responsibility for their
development of competence.6 This will lead to a workforce that is
better equipped to meet the nutrition and health needs of the
population and one that takes responsibility for their lifelong
learning.
This Fellowship aimed to transform the way competency-based
assessment is viewed within nutrition and dietetics. Using Monash
University’s Master of Dietetics program as an exemplar, it set out
to design a systems-based approach to competency-based assessment
and to establish a Community of Practice for dietetics educators to
champion change.
Transforming competency-based assessment in nutrition and dietetics
11
Chapter 2 Methods for transformation
Developing a systems-based approach to assessment
A workshop was held with academic staff (n=12), placement educators
(n=6) and recent graduates (n=3) from the Department of Nutrition
and Dietetics at Monash University. The workshop aimed to build the
skills and knowledge of key staff involved in assessment, benchmark
best-practice in competency-based assessment and to commence
discussions on how the Department of Nutrition and Dietetics could
work towards ‘better’ practice in competency-based assessment in
the dietetics program. Interactive presentations were delivered by
Associate Professor Margaret Hay on systems-based approaches to
assessment, and Associate Professors Margaret Bearman and Phillip
Dawson on assessment design decisions. The workshop included groups
working on ways to enhance our assessment practices, including
taking a systems approach, placing the student more at the centre
of assessment, and including new assessment tasks around the key
professional activities of contribution to team meetings and
patient handover. The external evaluator also attended the workshop
and explained the importance of the work, the evaluation process
and collected evaluation data. The key issue identified from the
workshop was the need for our team to see assessment as a whole and
as part of a system. Workshop participants completed a survey at
the end of the workshop that asked for comments on the use and
relevance of the content and presentations, and how the content
facilitated their ability to think differently about
competency-based assessment. This data was analysed by the external
evaluator, using descriptive methods.
Based on the discussions at the workshop, a small subset of the
participants from the workshop (2 academics and 2 practitioners)
worked together with the Fellow towards the development of a
systems-based approach to assessment. This Working Group met on
four occasions over three months to attempt to illustrate the
system of assessment for the preparation of dietitians at Monash
University. The new Master of Dietetics was selected as a vehicle
to design an assessment systems map for competency-based assessment
in nutrition and dietetics at Monash University. The map was
developed based on the principles of systems theory, whereby all
elements of the system are described and interconnected.10 The
focus of the map was to combine assessment of competence across
different workplace learning settings, to illustrate the multiple
pieces of evidence that inform assessment of competence, and to
provide a means to identify gaps in assessment. The working group
examined all the assessment tasks within the units of the Master of
Dietetics. This examination questioned the role of the assessment
task in providing evidence of competence. Those assessments that
were considered competency based were included in the map, and
mapped against the newly released 2015 National Competency
Standards. Other key elements within the system of assessment, such
as the National Competency Standards, university policy and
procedures and learning and teaching strategies were also
illustrated.
Transforming competency-based assessment in nutrition and dietetics
12
To support contribution and ownership of the systems map,
discussions were conducted with academic staff (n=7), placement
educators (n=6) involved in competency-based assessment and a
convenient sample of students (n=6) who had just completed all
coursework requirements of the undergraduate dietetic degree and
were eligible to enter the workforce. These discussions explored
reactions to the assessment systems map, key elements of effective
competency-based assessment and how these elements connect within
this system, their role in the competency-based system, any gaps or
repetition within our current approaches, and any areas for
improvement. Discussions were audio-recorded and notes taken. The
data were summarised and used to inform further refinement of the
systems map and to provide feedback on the concept.
Creating a Community of Practice for dietetics educators
Dietitians with a key stake in the education and preparation of
dietitians in Australia formed the target for the Community of
Practice. The total size of this workforce across Australia is
estimated to be 147 dietitians employed in contracted roles within
university dietetics programs (excluding sessional staff).11 A
special weekly email invitation was sent via the Dietitians
Association of Australia to all its members. Dietetics educators
were asked to submit an expression of interest detailing how they
met the selection criteria, which included: being affiliated with a
university involved in teaching and learning of an accredited
nutrition and dietetics program in Australia; experience in
learning and teaching; evidence of willingness to change, and
interest in changing, assessment practice; ability to demonstrate
implementation of innovation in teaching and learning; ability to
demonstrate commitment to attending; and evidence of willingness to
disseminate own learning to others.
A total of 23 dietitians expressed interest. Two were practice
educators and 21 were academics. After individual discussions, the
practice educators elected not to participate, as it was likely
that the preponderance of academics would mean that many
discussions would not prove relevant to them. Twenty eligible
participants, from 14 universities across Australia (of the 16
accredited at the time of the study), consented to be part of the
Community of Practice. In addition, one New Zealand-based
university academic, originally canvassed as a critical friend to
the Fellow, expressed interest in being part of the Community of
Practice.
The Community of Practice members met initially face-to-face in a
one-day workshop on 13 February 2015. Discussions included what the
participants hoped to achieve (Figure 1) and guidelines for
operation so that all participants agreed over how to contribute
effectively. It also provided an opportunity for participants to
get to know one another. Content on best- practice competency-based
assessment was delivered by the Fellow together with discussions on
benchmarking best-practice assessment nationally. The external
evaluator also attended, providing a presentation on the context
and importance of the work, and explaining the evaluation process
and data that would be collected.
Transforming competency-based assessment in nutrition and dietetics
13
Following this initial session, the Community of Practice members
met every six-to-eight weeks for a period of six-months, until
September 2015 when data were collected and then continued until
February 2016 and plan to continue beyond the fellowship. Sessions
ran for approximately 1.5 hours each via videoconference (zoom
technology). The Fellow facilitated each session. A web-based
repository of notes and documents discussed at the meeting was made
available to members using Monash eResearch Virtual Research
Environment Sakai. The content of each session is described in
Chapter 3. Through the Community of Practice, the Fellow was
invited to present to the academic dietetics teams at Flinders
University and the University of Wollongong on a systems-based
approach to assessment and be the expert advisor to a project at
Edith Cowan University on revision to the placement assessment
program.
The aim of the Community of Practice is to utilise the wisdom of
dietetics educators to transform their methods of assessment and
lead change. As well as developing the skills, knowledge and
attitudes of individual members we aspire to achieve the following
outcomes from working together:
• Support revision of accreditation standards for and with
Australian Dietetics Council • Support the dissemination of revised
Competency Standards • Develop a guide/resource for application of
Competency Standards to competency-based assessment • Create an
evidence-based approach to assess competence – examples for pieces
of assessment and
link to Competency Standards • Collection of data on assessment –
credibility, dependability, appropriateness, cost, student
friendliness • Obtain feedback from students on revised Competency
Standards • Pilot panel (multiple stakeholders) decision as part of
a system for competency-based assessment • Collaborative research
and publications
Figure 1 Outcomes Community of Practice members hoped to
achieve.
Evaluation of the Community of Practice
A pre-post participation survey and the ‘most significant change’
technique were used together to evaluate the Community of Practice.
The survey was constructed based on Boud’s propositions for
assessment reform in higher education.12 These propositions
describe 18 situations, under seven main areas, when assessment is
most effective, and therefore the 18 proposition statements were
presented as principles of best-practice assessment (Table 3). All
participants were asked to complete the survey online one-week
prior to the first session (at baseline) and then again after
six-months of participation. Community of Practice members were
asked to rate their current assessment practice in applying the 18
principles statements on a 5-point Likert scale from 1 (not at all)
to 5 (completely). At baseline, members were also asked to describe
what they perceived as being barriers to changing assessment
practice. After six-months, members were asked to rate, on a scale
from 1 (not achieved at all) to 5 (achieved completely), to what
degree the Community of Practice had achieved its planned outcomes
of networking, developing new strategies and approaches to
assessment, taking an evidence-based approach to competency-based
assessment, collaboration with members and research outcomes.
Transforming competency-based assessment in nutrition and dietetics
14
Survey data was analysed using IBM SPSS Statistics, Version 20.0
(IBM Corp., Armonk, NY). The reliability of the self-rated
assessment practice survey was assessed. The change in self- rated
assessment practice after being involved in the Community of
Practice was described and the significance of the change measured
using a paired t-test. A p-value of less than 0.05 was considered
statistically significant.
The ‘most significant change’ evaluation technique entails the
creation of significant change stories, developed by the members
through a process of prioritising the most significant change to
their practice through discussion.13 Two weeks prior to the
Community of Practice videoconference held six-months after the
initial meeting, members were invited by the Fellow, through the
above-mentioned survey, to respond to the question: Looking back
over the last six months, what do you think was the most
significant change in your attitudes and practice in
competency-based assessment that resulted from involvement in this
Community of Practice, and why was this story significant for
you?
The Fellow collated the responses using a basic content-analysis
approach,14 whereby all ideas were identified and presented as a
collated summary to participants, who were then asked to discuss
whether they agreed with the summary or whether they had any detail
to add. Consensus was developed with members on the most
significant change during this final Community of Practice
discussion and verified via email following the discussion.
Creating momentum for change across the other health
professions
As part of the program of activities, the Fellow aimed to have
critical discussions with key decision makers within the Faculty.
Over the course of the Fellowship three meetings were held with the
Associate Dean (Learning and Teaching) to discuss the aims of the
program and its progress. As part of these discussions, the Fellow
was invited to join the faculty’s assessment working party, whose
terms of reference are to progress faculty procedures related to
elements of assessment and to contribute to university policy. The
Fellow was also invited to present to the faculty’s clinical
placement committee meeting (on 10 August 2015) and the Monash
University Health Professions Education and Education Research
symposia (on 9 December 2015) on the concept of a systems-based
approach to assessment and the fellowship activities.
Discussions with critical friends and partners
The Fellow utilised the external evaluator as a mentor throughout
the fellowship having regular discussions about the progress of the
program of activities and plans for evaluation. These discussions
were used to shape the design of the activities and ensure
evaluation data was collected on all elements of the work, the
process measures of quality and satisfaction as well as impact or
change to assessment practice. In addition the support of Associate
Professors Margaret Hay and Phillip Dawson was utilised by the
fellow at regular intervals over the course of the year. They
provided the fellow guidance regarding evidence in and
Transforming competency-based assessment in nutrition and dietetics
15
around assessment practice and instilled a sense of confidence in
the planned approaches to the program of activities.
The Fellow was regularly asked to provide advice and guidance to
the Australian Dietetics Council regarding assessment in the
context of revision to the accreditation standards in light of the
release of the revised National Competency Standards for
Dietitians. This consultation occurred formally through meetings
and informally through discussions with the chair and Manager
Accreditation, Recognition and Journal Service of the Dietitians
Association of Australia.
Transforming competency-based assessment in nutrition and dietetics
16
Chapter 3 Findings and evaluation
Monash University Master of Dietetics systems map
As described above, the workshop involved key staff (n=18) and
three recent graduates and aimed to build their skills and
knowledge in best practice in competency-based assessment and to
commence discussions on how the team could work towards ‘better’
practice in competency-based assessment in the dietetics program.
Participants reported that they valued the workshop and overall
found it very beneficial. It enabled them to understand how taking
a systems approach to assessment could transform the types of
assessment used. It also facilitated a greater understanding of the
importance of different stakeholders being involved in assessment,
especially students and workplace educators. Participants felt that
more time could have been devoted to discussion around the new
concepts and key professional activities that were generated in the
working groups. There remained a perceived lack of confidence to
implement change and a request for further discussion and
exploration of the themes identified.
The Working Group that formed from a subset of the workshop
participants examined all 46 individual assessment tasks within the
units of the Master of Dietetics. Seven tasks were initially deemed
to be evidence of knowledge, or application of knowledge, rather
than contributing to competency-based assessment, so these were not
mapped (Table 1). The rest of the tasks were summarised and mapped
against the Competency Standards for Dietitians (2015). Where there
were multiple similar tasks across multiple units (e.g. oral
presentation, professional behaviour appraisal), these were
summarised and mapped together as a single task and examined
vertically and horizontally across the system. The illustration was
initially developed as a single page using Microsoft Publisher and
also using Prezi presentation software which enabled the detail of
the assessment tasks, key skills developed and link to the
competency standards to be illustrated. As part of the process, the
placement portfolio was deemed a learning tool rather than an
assessment task, as it became evident that there was duplication of
assessment with this task. The professional portfolio was also
identified as needing review to ensure adequate assessment of
professional practice, with a particular emphasis on safe practice
and cultural competence.
The three group discussions provided positive feedback on the map.
The iterative process enabled refinement of the map to include
stronger links to themes within the curriculum and National
Competency Standards for Dietitians (2015) and inclusion of two
tasks as evidence of competence rather than knowledge (policy
analysis and advocacy discussion paper). The key descriptive
finding from the discussion was that the map provided a mechanism
for all academics and practitioners involved in assessment of
competence at Monash University to see all the tasks that
contributed to final judgement. The participants also felt the map
was effective in illustrating the skills demonstrated by each task
by being clearly linked to the National Competency Standards for
Dietitians (2015) (Figure 2). The map was used as a point for
discussion at a number of workshops to build the capacity of
Transforming competency-based assessment in nutrition and dietetics
17
placement educators to understand the system of assessment. The map
will continue to be used as a vehicle for discussions around
competency-based assessment into the future for practice educators
and students. Table 1: Summary of Master of Dietetics curriculum
and assessment tasks
ENTRY POINT 1: Non-cognate = for students who have completed an
eligible undergraduate degree without units of study that would
allow completion of degree in 18 months
Sem ester 2
Scientific essay (30%) Evidence-based nutritional resource (20%)
Exam (50%)
NUT3082- Public health nutrition Mid-semester test (30%) Policy
analysis assignment (40%) Advocacy discussion paper (30%)
NUT4001- Foundations of dietetic practice Professional portfolio
50% (composed of: Dietary intake assignment, older adult interview
and reflection, food culture report, nutrition assessment task,
food law task)* Group education assignment 20% (plan and
presentation) Exam 30% (hurdle)
ENTRY POINT 2: Cognate = for students who have completed the
Bachelor of Nutrition Science at Monash University
Sem ester 1
NUT5001- Introduction to dietetic practice Exam 30% (hurdle) OSCE
25% Self-developed clinical resource 15% 2 x case reports 20%
Critical incident reflection 10% Placement assessment (incl.
professional behaviour appraisal) (P
NUT5002- Practice and research in public health nutrition
Project proposal (20%) Project report (40%) Project oral
presentation (10%) Standardised oral exam (30%) (Hurdle) Peer
learning and review (Ungraded) (Hurdle)
Sem ester 2
NUT5003 Dietetic Practice 1 Mid semester exam and final exam (50%)
Case-based workbook (30%) Evidence-based education resource (10%)
Self-developed placement handbook (10%) Professional behaviour
appraisal (Ungraded) (Hurdle)
NUT5004 Food for dietetics practice Project report (30%) Oral
presentation (10%) Food therapeutic manual (20%) Oral presentation
(10%) Menu analysis executive report (2,000 words) (20%) Placement
portfolio including guided reflection (10%) Professional behaviour
appraisal form (Ungraded) (Hurdle) Competency (Practical placement)
assessment (Ungraded) (Pass / Fail)
Sem ester 1
NUT5006 Practice and research in dietetics Scientific paper (40%)
(Hurdle) Peer review of scientific paper (10%) Small grant proposal
(20%) Teamwork assignment (25%) Attendance and participation
(Hurdle) Professional behaviour appraisal (Ungraded) (Hurdle) Final
competencies sign off (Hurdle)
Italic font indicates identified competency-based assessment
Figure 2 Monash University Master of Dietetics competency-based
assessment systems map.
Transforming competency-based assessment in nutrition and dietetics
19
Community of Practice for dietetics educators
Community of Practice participants
Twenty-one academics joined the Community of Practice. Of these,
Fellowship funding supported 10 and the remaining 11 academics were
provided with financial support by their respective universities,
both to participate and to pay the costs associated with attending
a face-to-face initial meeting. Participant demographics are
summarised in Table 2. All 21 participants requested to continue to
be part of the Community of Practice after the completion of the
fellowship.
Table 2: Demographics of members and attendance at Community of
Practice
Years as a dietitian
Highest qualification
9% (n=2) Bachelor; 24% (n=5) Master Dietetics; 43% (n=9) PhD; 24%
(n=5) Grad Dip
University 1 BU; 1 CSU; 1 CU; 1 DU; 2 ECU; 1 FU; 1 GU; 1 LU; 1 MU;
1 QUT; 2 US; 2 UC; 3 USC; 2 UOW
Attendance 2015 2016
n=21
(57%)
BU=Bond University; CSU: Charles Sturt University; CU=Curtin
University; DU=Deakin University; ECU=Edith Cowan University;
FU=Flinders University; GU=Griffith University; LU=Latrobe
University; MU=Monash University; QUT=Queensland University of
Technology; the University of Sydney; UC=University of Canberra;
USC=University of the Sunshine Coast; UOW=University of
Wollongong.
Community of Practice sessions
The first Community of Practice session involved a face-to-face day
workshop in Brisbane. All 21 members attended. After that an
average of 15 members (72%) attended the seven videoconference
discussions and each member attended between three (n=3 members)
and seven (n=5 members) of the seven sessions (mean=5 out of 7
sessions).
The first session was conducted in February 2015 at Queensland
University of Technology in Brisbane, as this was the most
cost-effective venue based on the location of participants. The
session was evaluated very positively with statements such
as:
A group of very inclusive and collaborative educators who are
working towards an evidence-based approach to competency-based
assessment.
Transforming competency-based assessment in nutrition and dietetics
20
An inspiring day – an open convivial collaborative gathering of
like-minded colleagues, facilitated by Claire – keen to produce
some valuable outcomes and
progress the profession.
The participants felt that the formation of the Community of
Practice was a milestone, because it allowed collaboration to take
place between universities, despite them being in competition for
students and clinical placement resources. There was recognition
that, rather than being driven by a professional
course-accreditation process, evidence-based assessment could be
led by this academic educator community. Many acknowledged the
leadership qualities and skill of the facilitator and there was
recognition of the scholarship around competency-based assessment
that the OLT Fellowship had enabled. Some participants felt
overwhelmed by the task of trying to implement a change in culture
within their own departments and believed that the support of the
Community of Practice would assist them to implement better
assessment systems, based on evidence. There was a commitment to
ongoing research in this area.
The Community of Practice sessions covered a range of discussion
areas. Each session followed a similar format, with discussion
begun by inviting participants to share any efforts they had been
working on relating to competency-based assessment. Topical
discussions included advocacy around accreditation standards,
release of the revised national competency standards, assessment
design decisions, working within university policy to change
assessment, and challenges associated with changing dietetic
placement educators’ approaches to assessment. In addition, members
felt that an important endeavour for the group to work on was that
of research questions concerned with how students engage with and
understand competency standards and competency-based
assessment.
In response, the Fellow designed a research project and sought
collaboration from the Community of Practice members. The aim of
this research is to explore how students construct and define the
concept of ‘professional competence’ and how they view the role of
assessment in developing professional competence. It also aims to
explore whether teaching and learning approaches can shift
students’ attitudes towards assessment, to determine what a
student-led approach to competency-based assessment looks like, and
to examine how educators can engage students in the development of
a competency-based assessment system. Ethics approval has been
sought and data collection commenced in October 2015.
Professors Sandra Capra and Susan Ash, leaders in Dietetics
Education, attended the final two Community of Practice sessions
and, through a facilitated discussion led by the Fellow, reflected
on their years as educators, their key learnings and what they saw
as challenges for the future of dietetic education. The Fellow will
analyse and write up these reflections in due course.
Transforming competency-based assessment in nutrition and dietetics
21
Evaluation of the Community of Practice
Prior to participation in the Community of Practice, members
described the main barriers to change to assessment practice as
being as follows:
the 2009 competency standards and accreditation processes
a lack of time to innovate and transform
university rules and lead-in times for change
attitudes and skills of assessors (including placement educators/
practitioners), and
programs are not being seen as holistic with little control over
early-year undergraduate student assessment.
The survey measuring assessment practice against 18 principles
statements showed good internal consistency, with a Cronbach’s
alpha coefficient of 0.773. Table 3 describes Community of Practice
members’ assessment practice across 18 principles of best-practice
assessment, pre- and post-participation. Generally, participants
rated their attention to the principles as high prior to the
Community of Practice, with all principles except two achieving a
mean score of 3.4 or greater. The items where participants felt
their attention could be improved included the following:
professional and scholarly approaches to assessment by academic
staff are developed, deployed, recognised and rewarded by
institutions, and
interim assessment results used for feedback on learning and
progress do not play a significant role in determining students’
final grade.
A paired t-test was conducted to evaluate the effect of the
Community of Practice on members’ assessment practice. There was a
statistically significant increase in members’ reported practice in
three areas (Table 3):
(i) Professional and scholarly approaches to assessment by academic
staff are developed, deployed, recognised and rewarded by
institutions
(ii) Interim assessment results used for feedback on learning and
progress do not play a significant role in determining students’
final grade
(iii) Certification accurately and richly portrays graduates’ and
students’ achievements to inform future careers and learning.
The significant change in the first two principles may be explained
by the lower result in these principles prior to participation in
the Community of Practice; however the change observed in perceived
attention to assessment for certification is consistent with the
focus of discussions in the Community of Practice around
systems-based approaches to assessment.
Transforming competency-based assessment in nutrition and dietetics
22
Some principles scores decreased, but not significantly, after the
involvement in the Community of Practice. This may be explained by
increased knowledge of the factors that contribute to best-practice
assessment and thus greater insight into current abilities and
practice.
When asked to what degree the Community of Practice was able to
fulfil the planned outcomes, members reported that it had mostly
achieved outcomes of networking (mean 4.3 out of 5), collaboration
(mean 3.9 out of 5) and taking an evidence-based approach to
competency-based assessment (mean 3.8 out of 5), but only somewhat
achieved new strategies and approaches to assessment (mean 3.3 out
of 5) and research outcomes (mean 3.1 out of 5).
Twenty of the 21 members completed the survey in time for the
Community of Practice session held after a six-month period, where
there was a discussion of the summary of the answers to the
question about the most significant change. A summary of the change
was presented to members at the final Community of Practice session
and agreement was sought on the story of change. The consensus on
the most significant change statement that was brought about by
their participating in the Community of Practice was:
The Community of Practice has changed our thinking and knowledge
about assessment. The need for holistic and student-led assessment
has been brought to
the forefront of our minds and the discussions have identified the
potential deficiencies of current approaches. The discussions have
built our confidence and
encouraged change to practice. We have been empowered to make
changes or have put plans in place to make change to redesign
assessment systems into the future,
particularly in relation to placement assessment. The network of
like-minded people and safe space to share ideas and approaches, as
well the timely release of new
competency standards, has facilitated this change. The strong
connection between members is needed into the future as we all act
as ambassadors for change
within our own universities.
The Community of Practice members also requested that they continue
as a network and the Fellow agreed to continue to facilitate four
videoconference sessions per year, with a review at the conclusion
each year, to allow the work that the network had commenced to be
progressed. The Community of Practice members have planned to
facilitate a half-day workshop prior to the Dieititan’s Association
of Australia National Conference in Melbourne, May 2016 on issues
related to the education of dietitians, with a particular focus on
competency-based assessment, open to all those interested in the
preparation of dietitians for the workforce.
Transforming competency-based assessment in nutrition and dietetics
23
Table 3 Community of Practice members self-reported assessment
practice before and after participation
Item from Boud’s principles of assessment Score (max 5) Mean
change pre-post
p-value from
paired t-test
In response to the following statements, please rate your level of
attention to the propositions thinking about your assessment
practice, from 1 (not at all) to 5 (completely)
Pre-CoP n=21
Post-CoP n=20
Assessment is designed to focus students on learning 4.25 4.25 0
p=1.00
Assessment is recognised as a learning activity that requires
engagement on appropriate tasks
4.40 4.20 -0.20 p=0.330
Feedback is informative and supportive and facilitates a positive
attitude to future learning
4.30 4.60 0.30 p=0.163
Students seek and use timely feedback to improve the quality of
their learning and work
3.90 4.10 0.20 p=0.214
Students regularly receive specific information, not just marks and
grades, about how to improve the quality of their work
4.30 4.40 0.10 p=0.666
Students progressively take responsibility for assessment and
feedback processes
3.42 3.63 0.21 p=0.448
Students develop and demonstrate the ability to judge the quality
of their own work and the work of others against agreed
standards
3.68 3.63 -0.05 p=0.858
Dialogue and interaction about assessment processes and standards
are commonplace between and among staff and students
4.00 4.20 0.20 p=0.408
Assessment practices are carefully structured in early stages of
courses to ensure students make a successful transition to
university study in their chosen field
3.83 4.17 0.34 p=0.231
Assessment practices respond to the diverse expectations and
experiences of entering students
3.50 3.40 -0.10 p=0.681
Assessment design is recognised as an integral part of curriculum
planning from the earliest stages of course development
4.53 4.63 0.10 p=0.542
Assessment is organized holistically across subjects and programs
with complementary integrated tasks
3.53 3.47 -0.06 p=0.834
Professional and scholarly approaches to assessment by academic
staff are developed, deployed, recognised and rewarded by
institutions
2.95 3.74 0.79 p=0.028*
Assessment practices and the curriculum should be reviewed in the
light of graduate and employer perceptions of the preparedness of
graduates
4.15 4.35 0.20 p=0.359
Assessment of student achievements is judged against consistent
national and international standards that are subject to continuing
dialogue, review and justification within disciplinary and
professional communities
3.60 3.75 0.15 p=0.481
Interim assessment results used for feedback on learning and
progress do not play a significant role in determining students’
final grade
2.53 3.79 1.26 p=0.007*
Evidence of overall achievement to determine final grades is based
on assessment of integrated learning
3.75 3.85 0.10 p=0.629
Certification accurately and richly portrays graduates’ and
students’ achievements to inform future careers and learning
3.65 4.53 0.88 p=0.014*
*p<0.05 statistically significant.
Transforming competency-based assessment in nutrition and dietetics
24
Chapter 4 Discussion and conclusions This Fellowship program of
activities aimed to engage the profession of nutrition and
dietetics in transforming competency-based assessment. Using Monash
University’s Master of Dietetics program as an exemplar, it set out
to design a systems-based approach to competency-based assessment
and to establish a Community of Practice to support academic
dietetics educators to champion change. The Monash University
Master’s program provided an effective illustration of the system
of assessment; this facilitated important dialogue on assessment
tasks, processes and approaches and created an environment and the
need for change. The Community of Practice engaged 20 dietetics
educators from Australia and one from New Zealand. It changed their
knowledge around best-practice competency-based assessment to be
more holistic and student-led and empowered them to make change to
practice and systems of assessment within their own universities.
There remains a need to continue to build the dialogue about taking
a systems- based approach to assessment, transforming our approach
in dietetics and spreading the word of this way of thinking across
other health professions.
Communities of practice have been found to be an effective
mechanism to share knowledge and improve practice15, 16 and
advocated as a way of creating an exchange around learning and
teaching practice in higher education.17 However their ability to
change members’ practice remains unclear.15, 16 The formalised
Community of Practice conducted as part of this Fellowship
attempted to measure practice impact using mixed methods.
Triangulating the data from survey and most significant change
technique would suggest that members of the Community of Practice
have a new set of knowledge and attitudes around assessment and are
committed to changing assessment. The length of time required to
make change within higher education may have prevented change from
actually taking place during the formalised program. While
communities of practice were first described as a form of
unstructured social learning, the role of a leader in administering
the group and facilitating discussions has been acknowledged as
part of their success.15, 17 The role of the Fellow in mobilising
the group and facilitating the development of new knowledge was
vital to success. As part of future research endeavours, there is a
need to continue to monitor the change within higher education that
Community of Practice members are able to instigate. Such measures
may include: assessment of the partnerships and networks; the
members’ roles in leading change; assessment changes within
curricula; and student feedback regarding different
approaches.
Taking a systems-based approach to assessment provided a mechanism
for reform for the dietetics program at Monash University. The
workshop and systems map have begun a process for change and
highlighted the need to engage practice educators in this process.
The Fellowship demonstrated that the systems-based approach was
well received by key stakeholders. It offered a new way of thinking
to manage many of the challenges brought about by current
assessment approaches. While the fellowship engaged a number of
placement educators involved in the dietetics program at Monash
University, there is a need to continue this momentum for change
and to engage more stakeholders in this
Transforming competency-based assessment in nutrition and dietetics
25
dialogue. Doing so provides a great opportunity to transform
competency-based assessment to be more student-centred and focused
on learning. This Fellowship is only the beginning of a journey to
further this way of thinking in dietetics and then across the other
health professions. The need to draw accreditation systems into
this approach is also paramount for the future. Infiltrating
assessment change within work-based learning settings is also an
important consideration due to the fundamental role work-based
learning has in competency-based assessment. Evaluating the ability
to support change in placement educators and assessment, and the
factors that enable this, will assist other professions in attempts
to change assessment practice.
In 1997, Gale and Grant published a guide on managing change in
medical education,18 yet there are few scholarship stories of
successful change to assessment practice in higher education,19,
or, in particular, of how educators navigate university policies
and procedures to effect assessment reform. The role of leadership
in supporting change is described.18 The leadership provided by the
Fellow has been key to supporting change both within the Community
of Practice and in the dietetics team at Monash University.
Continuing this leadership and engagement, as well as identifying
elements for effective change, are important going forward.
The challenges to taking a systems-based approach to assessment in
the health professions are immense. The transformation requires
both a change to assessment approaches and a change in the way
students themselves approach assessment. Assessment drives
learning, but students’ expectations are often to see assessment as
a way of measuring learning rather than as a vehicle for learning.
Any change to assessment approaches must acknowledge the
requirement to pay attention to the need to influence students’
attitudes and approaches to assessment. While students have
reported wanting more control over the assessment process,20
teachers will need to support them with the skills and knowledge to
be able to do this.
This Fellowship program of activities has commenced the process of
transforming competency-based assessment in nutrition and
dietetics. By building the capacity of educators and showcasing an
example of how taking a systems-based approach can manage many of
the challenges in assessment, it has begun to effect change. There
is a need for continued investment to support academics, practice
educators and assessors to transform their approaches and to
measure the impact of this change on the educators and, most
importantly, on the students themselves.
This Fellowship has taught me the value of building relationships
to affect change. It has also shown me that building individual
capacity through extending knowledge and empowering individuals to
change practice can affect many more people than the individuals
you directly work with. I have also been reminded that assessment
cannot be examined in isolation from learning outcomes, content,
the people who assess and, most importantly, our students who are
being assessed.
Transforming competency-based assessment in nutrition and dietetics
26
References 1. National Health and Medical Research Council. A
review of the evidence to address targeted questions to inform the
revision of the Australian Dietary Guidelines Canberra: NHMRC
(National Health and Medical Research Council) Commonwealth of
Australia, 2011.
2. Dietitians Association of Australia. National competency
standards for dietitians. 2015.
3. Epstein R and Hundert E. Defining and assessing professional
competence. JAMA. 2002; 287: 226-35.
4. Lennie S and Juwah C. Exploring assessment for learning during
dietetic practice placements. J Hum Nutr Diet. 2010; 23:
217-23.
5. Palermo C, Capra S, Ash S, Beck E, Truby H and Jolly B. Are
clinical supervisors’ attitudes to assessment a barrier to
expanding placement opportunities? Results of a Delphi study.
Nutrition and Dietetics. 2015 (online).
6. Palermo C, Beck E, Chung A, et al. Work-based assessment:
qualitative perspectives of novice nutrition and dietetics
educators. J Hum Nutr Diet. 2013; 27: 513–52.
7. Norcini J, Anderson B, Bollela V, et al. Criteria for good
assessment: Consensus statement and recommendations from the Ottawa
2010 conference. Med Teach. 2011; 33: 206-14.
8. Dalton M, Davidson M and Keating J. The Assessment of
Physiotherapy Practice (APP) is a valid measure of professional
competence of physiotherapy students: a cross- sectional study with
Rasch analysis. J Physio. 2011; 57: 239-46.
9. Speech Pathology Australia. COMPASS® Online Assessment Tool.
2009.
10. Mennin S. Self-organisation, integration and curriculum in the
complex world of medical education. Medical Education. 2010; 44:
20-30.
11. Morgan K and Hughes R. A profile of the size and professional
attributes of the Australian academic dietetic educator workforce.
Nutrition and Dietetics. in press.
12. Boud D. Assessment 2020: Seven propositions for assessment
reform in higher education. Australian Learning and Teaching
Council, 2009.
13. Davis R and Dart J. The 'Most Significant Change' (MSC)
Technique. United Kingdom: CARE International, 2005.
Transforming competency-based assessment in nutrition and dietetics
27
14. Liamputtong P. Research methods in health. Foundations for
evidence based practice. South Melbourne: Oxford University Press,
2010.
15. Li L, Grimshaw J, Nielsen C, Judd M, Coyte P and Graham I. Use
of communities of practice in business and health care sectors: A
systematic review. Implement Sci 2009; 4: 27.
16. Ranmuthugala G, Plumb J, Cunningham F, Georgiou A, Westbrook J
and Braithwaite J. How and why are communities of pracitce
established in the healthcare sector? A systematic review of the
literature. BMC Health Services Research. 2011; 11.
17. McDonald J. Community, Domain, Practice: Facilitator catch cry
for revitalising learning and teaching through communities of
practice. Sydney: Office for Learning and Teaching, Department of
Education, 2014.
18. Gale R and Grant J. AMEE Medical Education Guide No. 10:
managing change in a medical context: guidelines for action. Med
Teach. 1997; 19: 239-49.
19. Chowthi-Williams A, Curzio J and Lerman S. Evaluation of how a
curriculum change in nurse education was managed through the
application of a business change management model: A qualitative
case study. Nurse Education Today. 2015.
20. Palermo C, Chung A, Beck E, et al. Evaluation of assessment in
the context of work- based learning. Qualitative perspectives of
new graduates. Nutr Diet. 2014; accepted ahead of print.
Transforming competency-based assessment in nutrition and dietetics
28
Appendix A Certification by Deputy Vice-Chancellor (or
equivalent)
I certify that all parts of the final report for this OLT
Fellowship provide an accurate representation of the
implementation, impact and findings of the project, and that the
report is of publishable quality.
Name: Professor Darrell Evans Date: 28 September 2015 Vice-Provost
(Learning and Teaching) Monash University
Transforming competency-based assessment in nutrition and dietetics
29
Acknowledgements
Developing a systems-based approach to assessment
Creating a Community of Practice for dietetics educators
Evaluation of the Community of Practice
Creating momentum for change across the other health
professions
Discussions with critical friends and partners
Chapter 3 Findings and evaluation
Monash University Master of Dietetics systems map
Community of Practice for dietetics educators
Community of Practice participants
Community of Practice sessions
Chapter 4 Discussion and conclusions
References
