Transforming competency-based assessment in nutrition and ...

Transforming competency-based assessment in nutrition and dietetics

Final report 2016

Dr Claire Palermo 2014 National Teaching Fellow Monash University

Professor Susan Ash, external evaluator

Transforming competency-based assessment in nutrition and dietetics 2

Support for the production of this report has been provided by the Australian Government Office for Learning and Teaching. The views expressed in this report do not necessarily reflect the views of the Australian Government Office for Learning and Teaching.

With the exception of the Commonwealth Coat of Arms, and where otherwise noted, all material presented in this document is provided under Creative Commons Attribution-ShareAlike 4.0 International License http://creativecommons.org/licenses/by-sa/4.0/.

The details of the relevant licence conditions are available on the Creative Commons website (accessible using the links provided) as is the full legal code for the Creative Commons Attribution-ShareAlike 4.0 International License http://creativecommons.org/licenses/by-sa/4.0/legalcode.

Requests and inquiries concerning these rights should be addressed to: Office for Learning and Teaching Department of Education and Training

GPO Box 9880, Location code N255EL10 Sydney NSW 2001

<[email protected]>

2016

ISBN 978-1-76028-678-1 [PDF] ISBN 978-1-76028-679-8 [DOCX] ISBN 978-1-76028-677-4 [PRINT]

http://creativecommons.org/licenses/by-sa/4.0/

http://creativecommons.org/licenses/by-sa/4.0/legalcode

mailto:[email protected]


Acknowledgements I acknowledge the support provided throughout this Fellowship by Associate Professor Margaret Hay, Monash University, Associate Professor Phillip Dawson, Deakin University and Professor Kevin Whelan, Kings College London. I am thankful for the support of my colleagues at Monash University Department of Nutrition and Dietetics: Janeane Dart, Simone Gibson, Evelyn Volders, Andrea Bryce, Judi Porter, Liza Barbour and Amanda Anderson, all of whom contributed to the success of this Fellowship. I would also like to thank Professor Helen Truby for allowing me to pursue this endeavour and Sonia Bartoluzzi for her professional editing service.


Executive summary Dietitians are the recognised professionals best placed to improve the nutrition of individuals and populations. A competent nutrition and dietetics workforce is essential to maintain and improve the health of Australians. Quality teaching and learning are essential to produce a competent workforce. Dietitians, like other health professionals, undertake learning in workplaces across the health care system and other settings. The development of competence occurs and is assessed within the university setting as well as within these workplaces. Competency-based assessment is made challenging by increasing student numbers, work pressures on assessors in the workplace, varying expectations and lack of consistency over what constitutes ‘competency’. Improving the quality of assessment will lead to a dietetics workforce that is better equipped to meet the nutrition and health needs of the population and one that takes responsibility for lifelong learning.

This Fellowship program of activities aimed to transform the way competency-based assessment is viewed within the field of nutrition and dietetics. Using Monash University’s Master of Dietetics program as an exemplar, it set out to design a systems-based approach to competency-based assessment. It also established a Community of Practice to support dietetics educators to work towards best-practice assessment. The Fellowship focused on conceptualising competency-based assessment as a system with programs of assessment. It defined ‘best-practice assessment’ as being authentic, having defined performance criteria, involving multiple and adequately skilled assessors, involving multiple methods and embedding regular effective feedback. The Fellowship was a timely investment in the discipline, given the recent revision and release of National Competency Standards for Dietitians (2015), led by the Fellow, and revisions to the Australian Dietetics Council accreditation standards.

Monash University system-based assessment approach Staff (n=18) and recent graduates (n=3) from the Department of Nutrition and Dietetics at Monash University attended a workshop on 4 December 2014 that aimed to build participants’ skills and knowledge in assessment and to commence discussions on how the department could work towards ‘better’ practice in competency-based assessment. This workshop identified the need for stakeholders to see assessment as a whole, and as part of a system. Subsequently, the Fellow worked with a small subset of stakeholders (2 academics and 2 practitioners) to develop a systems-based approach to assessment. An illustration was developed that mapped all competency-based assessment tasks within the Master of Dietetics, the key skills developed by each task and their place within a system. Feedback was sought through discussions with academic staff (n=7) and placement educators (n=6) to further refine the systems map illustration. The map provided a mechanism for all academics and practitioners involved in assessment of competence at Monash University to visualise how multiple and varying pieces of evidence contribute to the final judgement of competence, and to see their own role within the system of assessment. The map has been used at a number of educator’s forums to build the capacity of educators to see the role of workplace assessment and the multiple evidence that informs judgement of competence.


Community of Practice Dietitians with a key stake in the education and preparation of dietitians in Australia were invited to participate in a Community of Practice. Twenty-one academics from 14 universities across Australia (of the 16 accredited at the time of the study) and one from New Zealand participated. The Community of Practice members initially met face-to-face in a one-day workshop on 13 February 2015 in Brisbane. They agreed that the aims of the network were to develop the skills, knowledge and attitudes of individual members, to transform their methods of assessment and to lead change. The network then met regularly every six-to-eight weeks for approximately 1.5 hours via videoconference facilitated by the Fellow for a period of twelve months. These sessions focused on discussions around competency-based assessment practice, accreditation standards, revised national competency standards, challenges associated with changing assessment and dietetic placement educators’ approaches. The network also designed and developed a research project to explore students’ perceptions of competency standards and competency-based assessment.

A pre-post participation survey, together with the ‘most significant change’ technique, was used to evaluate the Community of Practice. The findings of this evaluation suggest that the Community of Practice has changed members’ thinking and knowledge around assessment and members have been empowered to make changes to redesign assessment systems. In particular, there was an improvement in members’ approaches to integrating circular feedback into the program of assessment, taking a scholarly approach to assessment and ensuring that certification portrays students’ achievements and future learning. Participants valued the network of like-minded people and safe space in which to share ideas and approaches.

The Community of Practice members have partnered to design a research project that aims to explore how students engage with and understand competency standards and competency-based assessment. Data collection commenced in October 2015.

This Fellowship has commenced an important dialogue on competency-based assessment in nutrition and dietetics and has directly changed the perspectives of at least 40 key stakeholders in competency-based assessment. The timing of the Fellowship alongside the release of National Competency Standards for Dietitians has created momentum for continued development to improve competency-based assessment for the profession. There is a need for continued conversation to support change to assessment practice in nutrition and dietetics and other health professions, so that students are placed at the centre of assessment with multiple pieces of evidence used to inform judgement of competence. The products of this fellowship will facilitate continued dialogue. This will support the development of health professionals better equipped for the future with a stronger commitment and capacity for life-long learning.


External Evaluation The intended outcomes of the project have been achieved in the move of Monash University to integrated assessment via a mapping process involving students and benchmarked standards, and the development of a trans-Tasman and national Community of Practice of university staff responsible for competency-based assessment. Further outcomes include the Community of Practice working on a research project which will develop new/revised assessment methods across with work-based practice settings and in the process will transform competency-based assessment practice in the profession of dietetics. The Fellow has been recognised for her outstanding leadership in this area and as such, has been engaged by the Australian Dietetic Council of the Dietitians Association Australia to advise on best-practice competency assessment and transforming the process for dietetic course accreditation. Perhaps the most significant outcome from the Fellowship has been the Fellow’s ability to bring all parties together to consider assessment in an evidence based fashion and to promote further research in the area, which will have resonance beyond the profession of dietetics. A workshop was conducted at the Dietitians Association of Australia National conference in Perth in May 2015. The Fellow has been invited to deliver key note presentations and workshops the Dietitians Association of Australia National conference in Melbourne, May 2016 and International Congress of Dietetics in Granada, Spain, September 2016. A number of publications have been submitted from this work.


Table of contents

Acknowledgements .................................................................................................................... 3

Executive summary .................................................................................................................... 4

Tables and figures ...................................................................................................................... 8

Tables ..................................................................................................................................... 8

Figures .................................................................................................................................... 8

Chapter 1 Introduction ............................................................................................................ 10

Chapter 2 Methods for transformation ................................................................................... 11

Developing a systems-based approach to assessment ....................................................... 11

Creating a Community of Practice for dietetics educators .................................................. 12

Evaluation of the Community of Practice ........................................................................ 13

Creating momentum for change across the other health professions ............................... 14

Discussions with critical friends and partners ..................................................................... 14

Chapter 3 Findings and evaluation .......................................................................................... 16

Monash University Master of Dietetics systems map ......................................................... 16

Community of Practice for dietetics educators ................................................................... 19

Community of Practice participants ................................................................................ 19

Community of Practice sessions ...................................................................................... 19

Evaluation of the Community of Practice ........................................................................ 21

Chapter 4 Discussion and conclusions ..................................................................................... 24

References ............................................................................................................................... 26

Appendix A ............................................................................................................................... 28


Tables and figures

Tables

Table 1: Summary of Master of Dietetics curriculum and assessment tasks .......................... 17

Table 2: Demographics of members and attendance at Community of Practice ................... 19

Table 3 Community of Practice members self-reported assessment practice before and after participation ..................................................................................................................... 23

Figures

The aim of the Community of Practice is to utilise the wisdom of dietetics educators to transform their methods of assessment and lead change. As well as developing the skills, knowledge and attitudes of individual members we aspire to achieve the following outcomes from working together:................................................................... 13

• Support revision of accreditation standards for and with Australian Dietetics Council . 13

• Support the dissemination of revised Competency Standards ....................................... 13

• Develop a guide/resource for application of Competency Standards to competency-based assessment ............................................................................................................ 13

• Create an evidence-based approach to assess competence – examples for pieces of assessment and link to Competency Standards .............................................................. 13

• Collection of data on assessment – credibility, dependability, appropriateness, cost, student friendliness ......................................................................................................... 13

• Obtain feedback from students on revised Competency Standards ............................... 13


• Pilot panel (multiple stakeholders) decision as part of a system for competency-based assessment ....................................................................................................................... 13

• Collaborative research and publications ......................................................................... 13

Figure 1 Outcomes Community of Practice members hoped to achieve. ............................... 13

Figure 2 Monash University Master of Dietetics competency-based assessment systems map. ................................................................................................................................. 18


Chapter 1 Introduction Optimal nutrition of the population offers perhaps the greatest potential for improving health.1 Nutrition and dietetics professionals are the recognised experts in human nutrition science, and are best placed to improve the nutrition of individuals and populations. Producing a competent nutrition and dietetics workforce is essential to maintain and improve the health of Australians. Developing the nutrition and dietetics profession to meet workforce needs is an ongoing challenge for Australian universities that is mirrored internationally.

Health, disease and the health care system are complex and ever-changing domains of practice, requiring an adequately equipped health workforce. For dietitians, the National Competency Standards provide a framework for the preparation of dietitians for entry into the workforce in Australia.2 In Australia dietitians are professionally prepared to practice medical nutrition therapy together with food service management and community/public health nutrition, which includes health promotion, nutrition policy and improving the food supply, especially for vulnerable groups.

One of the most challenging areas in preparing the workforce for practice is competency-based assessment.3 This is due to: increasing demands for work-based learning experiences, work pressures on assessors, varying expectations, lack of consistency in what constitutes competency, and the need for assessors to embrace subjectivity in making judgement on performance.4-6 In addition, assessment of competence in dietetics has not traditionally been viewed as a system; rather, the focus has been on individual methods of assessment in the workplace setting. Best-practice assessment requires clearly defined performance criteria, multiple and adequately skilled assessors, multiple methods, authenticity and effective feedback.7 While some allied health profession disciplines have developed tools for reliable and valid assessment,8, 9 few describe fully effective and functional assessment systems. Dietetics uses a range of methods to assess competence across the various settings, but there is a need to build the capacity of universities and dietitians involved in assessment and to support the profession to view assessment as a system, and one whereby students take responsibility for their development of competence.6 This will lead to a workforce that is better equipped to meet the nutrition and health needs of the population and one that takes responsibility for their lifelong learning.

This Fellowship aimed to transform the way competency-based assessment is viewed within nutrition and dietetics. Using Monash University’s Master of Dietetics program as an exemplar, it set out to design a systems-based approach to competency-based assessment and to establish a Community of Practice for dietetics educators to champion change.


Chapter 2 Methods for transformation

Developing a systems-based approach to assessment

A workshop was held with academic staff (n=12), placement educators (n=6) and recent graduates (n=3) from the Department of Nutrition and Dietetics at Monash University. The workshop aimed to build the skills and knowledge of key staff involved in assessment, benchmark best-practice in competency-based assessment and to commence discussions on how the Department of Nutrition and Dietetics could work towards ‘better’ practice in competency-based assessment in the dietetics program. Interactive presentations were delivered by Associate Professor Margaret Hay on systems-based approaches to assessment, and Associate Professors Margaret Bearman and Phillip Dawson on assessment design decisions. The workshop included groups working on ways to enhance our assessment practices, including taking a systems approach, placing the student more at the centre of assessment, and including new assessment tasks around the key professional activities of contribution to team meetings and patient handover. The external evaluator also attended the workshop and explained the importance of the work, the evaluation process and collected evaluation data. The key issue identified from the workshop was the need for our team to see assessment as a whole and as part of a system. Workshop participants completed a survey at the end of the workshop that asked for comments on the use and relevance of the content and presentations, and how the content facilitated their ability to think differently about competency-based assessment. This data was analysed by the external evaluator, using descriptive methods.

Based on the discussions at the workshop, a small subset of the participants from the workshop (2 academics and 2 practitioners) worked together with the Fellow towards the development of a systems-based approach to assessment. This Working Group met on four occasions over three months to attempt to illustrate the system of assessment for the preparation of dietitians at Monash University. The new Master of Dietetics was selected as a vehicle to design an assessment systems map for competency-based assessment in nutrition and dietetics at Monash University. The map was developed based on the principles of systems theory, whereby all elements of the system are described and interconnected.10 The focus of the map was to combine assessment of competence across different workplace learning settings, to illustrate the multiple pieces of evidence that inform assessment of competence, and to provide a means to identify gaps in assessment. The working group examined all the assessment tasks within the units of the Master of Dietetics. This examination questioned the role of the assessment task in providing evidence of competence. Those assessments that were considered competency based were included in the map, and mapped against the newly released 2015 National Competency Standards. Other key elements within the system of assessment, such as the National Competency Standards, university policy and procedures and learning and teaching strategies were also illustrated.


To support contribution and ownership of the systems map, discussions were conducted with academic staff (n=7), placement educators (n=6) involved in competency-based assessment and a convenient sample of students (n=6) who had just completed all coursework requirements of the undergraduate dietetic degree and were eligible to enter the workforce. These discussions explored reactions to the assessment systems map, key elements of effective competency-based assessment and how these elements connect within this system, their role in the competency-based system, any gaps or repetition within our current approaches, and any areas for improvement. Discussions were audio-recorded and notes taken. The data were summarised and used to inform further refinement of the systems map and to provide feedback on the concept.

Creating a Community of Practice for dietetics educators

Dietitians with a key stake in the education and preparation of dietitians in Australia formed the target for the Community of Practice. The total size of this workforce across Australia is estimated to be 147 dietitians employed in contracted roles within university dietetics programs (excluding sessional staff).11 A special weekly email invitation was sent via the Dietitians Association of Australia to all its members. Dietetics educators were asked to submit an expression of interest detailing how they met the selection criteria, which included: being affiliated with a university involved in teaching and learning of an accredited nutrition and dietetics program in Australia; experience in learning and teaching; evidence of willingness to change, and interest in changing, assessment practice; ability to demonstrate implementation of innovation in teaching and learning; ability to demonstrate commitment to attending; and evidence of willingness to disseminate own learning to others.

A total of 23 dietitians expressed interest. Two were practice educators and 21 were academics. After individual discussions, the practice educators elected not to participate, as it was likely that the preponderance of academics would mean that many discussions would not prove relevant to them. Twenty eligible participants, from 14 universities across Australia (of the 16 accredited at the time of the study), consented to be part of the Community of Practice. In addition, one New Zealand-based university academic, originally canvassed as a critical friend to the Fellow, expressed interest in being part of the Community of Practice.

The Community of Practice members met initially face-to-face in a one-day workshop on 13 February 2015. Discussions included what the participants hoped to achieve (Figure 1) and guidelines for operation so that all participants agreed over how to contribute effectively. It also provided an opportunity for participants to get to know one another. Content on best-practice competency-based assessment was delivered by the Fellow together with discussions on benchmarking best-practice assessment nationally. The external evaluator also attended, providing a presentation on the context and importance of the work, and explaining the evaluation process and data that would be collected.


Following this initial session, the Community of Practice members met every six-to-eight weeks for a period of six-months, until September 2015 when data were collected and then continued until February 2016 and plan to continue beyond the fellowship. Sessions ran for approximately 1.5 hours each via videoconference (zoom technology). The Fellow facilitated each session. A web-based repository of notes and documents discussed at the meeting was made available to members using Monash eResearch Virtual Research Environment Sakai. The content of each session is described in Chapter 3. Through the Community of Practice, the Fellow was invited to present to the academic dietetics teams at Flinders University and the University of Wollongong on a systems-based approach to assessment and be the expert advisor to a project at Edith Cowan University on revision to the placement assessment program.

The aim of the Community of Practice is to utilise the wisdom of dietetics educators to transform their methods of assessment and lead change. As well as developing the skills, knowledge and attitudes of individual members we aspire to achieve the following outcomes from working together:

• Support revision of accreditation standards for and with Australian Dietetics Council • Support the dissemination of revised Competency Standards • Develop a guide/resource for application of Competency Standards to competency-based assessment • Create an evidence-based approach to assess competence – examples for pieces of assessment and

link to Competency Standards • Collection of data on assessment – credibility, dependability, appropriateness, cost, student

friendliness • Obtain feedback from students on revised Competency Standards • Pilot panel (multiple stakeholders) decision as part of a system for competency-based assessment • Collaborative research and publications

Figure 1 Outcomes Community of Practice members hoped to achieve.

Evaluation of the Community of Practice

A pre-post participation survey and the ‘most significant change’ technique were used together to evaluate the Community of Practice. The survey was constructed based on Boud’s propositions for assessment reform in higher education.12 These propositions describe 18 situations, under seven main areas, when assessment is most effective, and therefore the 18 proposition statements were presented as principles of best-practice assessment (Table 3). All participants were asked to complete the survey online one-week prior to the first session (at baseline) and then again after six-months of participation. Community of Practice members were asked to rate their current assessment practice in applying the 18 principles statements on a 5-point Likert scale from 1 (not at all) to 5 (completely). At baseline, members were also asked to describe what they perceived as being barriers to changing assessment practice. After six-months, members were asked to rate, on a scale from 1 (not achieved at all) to 5 (achieved completely), to what degree the Community of Practice had achieved its planned outcomes of networking, developing new strategies and approaches to assessment, taking an evidence-based approach to competency-based assessment, collaboration with members and research outcomes.


Survey data was analysed using IBM SPSS Statistics, Version 20.0 (IBM Corp., Armonk, NY). The reliability of the self-rated assessment practice survey was assessed. The change in self-rated assessment practice after being involved in the Community of Practice was described and the significance of the change measured using a paired t-test. A p-value of less than 0.05 was considered statistically significant.

The ‘most significant change’ evaluation technique entails the creation of significant change stories, developed by the members through a process of prioritising the most significant change to their practice through discussion.13 Two weeks prior to the Community of Practice videoconference held six-months after the initial meeting, members were invited by the Fellow, through the above-mentioned survey, to respond to the question: Looking back over the last six months, what do you think was the most significant change in your attitudes and practice in competency-based assessment that resulted from involvement in this Community of Practice, and why was this story significant for you?

The Fellow collated the responses using a basic content-analysis approach,14 whereby all ideas were identified and presented as a collated summary to participants, who were then asked to discuss whether they agreed with the summary or whether they had any detail to add. Consensus was developed with members on the most significant change during this final Community of Practice discussion and verified via email following the discussion.

Creating momentum for change across the other health professions

As part of the program of activities, the Fellow aimed to have critical discussions with key decision makers within the Faculty. Over the course of the Fellowship three meetings were held with the Associate Dean (Learning and Teaching) to discuss the aims of the program and its progress. As part of these discussions, the Fellow was invited to join the faculty’s assessment working party, whose terms of reference are to progress faculty procedures related to elements of assessment and to contribute to university policy. The Fellow was also invited to present to the faculty’s clinical placement committee meeting (on 10 August 2015) and the Monash University Health Professions Education and Education Research symposia (on 9 December 2015) on the concept of a systems-based approach to assessment and the fellowship activities.

Discussions with critical friends and partners

The Fellow utilised the external evaluator as a mentor throughout the fellowship having regular discussions about the progress of the program of activities and plans for evaluation. These discussions were used to shape the design of the activities and ensure evaluation data was collected on all elements of the work, the process measures of quality and satisfaction as well as impact or change to assessment practice. In addition the support of Associate Professors Margaret Hay and Phillip Dawson was utilised by the fellow at regular intervals over the course of the year. They provided the fellow guidance regarding evidence in and


around assessment practice and instilled a sense of confidence in the planned approaches to the program of activities.

The Fellow was regularly asked to provide advice and guidance to the Australian Dietetics Council regarding assessment in the context of revision to the accreditation standards in light of the release of the revised National Competency Standards for Dietitians. This consultation occurred formally through meetings and informally through discussions with the chair and Manager Accreditation, Recognition and Journal Service of the Dietitians Association of Australia.


Chapter 3 Findings and evaluation

Monash University Master of Dietetics systems map

As described above, the workshop involved key staff (n=18) and three recent graduates and aimed to build their skills and knowledge in best practice in competency-based assessment and to commence discussions on how the team could work towards ‘better’ practice in competency-based assessment in the dietetics program. Participants reported that they valued the workshop and overall found it very beneficial. It enabled them to understand how taking a systems approach to assessment could transform the types of assessment used. It also facilitated a greater understanding of the importance of different stakeholders being involved in assessment, especially students and workplace educators. Participants felt that more time could have been devoted to discussion around the new concepts and key professional activities that were generated in the working groups. There remained a perceived lack of confidence to implement change and a request for further discussion and exploration of the themes identified.

The Working Group that formed from a subset of the workshop participants examined all 46 individual assessment tasks within the units of the Master of Dietetics. Seven tasks were initially deemed to be evidence of knowledge, or application of knowledge, rather than contributing to competency-based assessment, so these were not mapped (Table 1). The rest of the tasks were summarised and mapped against the Competency Standards for Dietitians (2015). Where there were multiple similar tasks across multiple units (e.g. oral presentation, professional behaviour appraisal), these were summarised and mapped together as a single task and examined vertically and horizontally across the system. The illustration was initially developed as a single page using Microsoft Publisher and also using Prezi presentation software which enabled the detail of the assessment tasks, key skills developed and link to the competency standards to be illustrated. As part of the process, the placement portfolio was deemed a learning tool rather than an assessment task, as it became evident that there was duplication of assessment with this task. The professional portfolio was also identified as needing review to ensure adequate assessment of professional practice, with a particular emphasis on safe practice and cultural competence.

The three group discussions provided positive feedback on the map. The iterative process enabled refinement of the map to include stronger links to themes within the curriculum and National Competency Standards for Dietitians (2015) and inclusion of two tasks as evidence of competence rather than knowledge (policy analysis and advocacy discussion paper). The key descriptive finding from the discussion was that the map provided a mechanism for all academics and practitioners involved in assessment of competence at Monash University to see all the tasks that contributed to final judgement. The participants also felt the map was effective in illustrating the skills demonstrated by each task by being clearly linked to the National Competency Standards for Dietitians (2015) (Figure 2). The map was used as a point for discussion at a number of workshops to build the capacity of


placement educators to understand the system of assessment. The map will continue to be used as a vehicle for discussions around competency-based assessment into the future for practice educators and students. Table 1: Summary of Master of Dietetics curriculum and assessment tasks

ENTRY POINT 1: Non-cognate = for students who have completed an eligible undergraduate degree without units of study that would allow completion of degree in 18 months

Semester 2

NUT3002-Nutrition, activity and health in chronic disease

Scientific essay (30%) Evidence-based nutritional resource (20%) Exam (50%)

NUT3082- Public health nutrition Mid-semester test (30%) Policy analysis assignment (40%) Advocacy discussion paper (30%)

NUT4001- Foundations of dietetic practice Professional portfolio 50% (composed of: Dietary intake assignment, older adult interview and reflection, food culture report, nutrition assessment task, food law task)* Group education assignment 20% (plan and presentation) Exam 30% (hurdle)

ENTRY POINT 2: Cognate = for students who have completed the Bachelor of Nutrition Science at Monash University

Semester 1

NUT5001- Introduction to dietetic practice Exam 30% (hurdle) OSCE 25% Self-developed clinical resource 15% 2 x case reports 20% Critical incident reflection 10% Placement assessment (incl. professional behaviour appraisal) (P

NUT5002- Practice and research in public health nutrition

Project proposal (20%) Project report (40%) Project oral presentation (10%) Standardised oral exam (30%) (Hurdle) Peer learning and review (Ungraded) (Hurdle)

Semester 2

NUT5003 Dietetic Practice 1 Mid semester exam and final exam (50%) Case-based workbook (30%) Evidence-based education resource (10%) Self-developed placement handbook (10%) Professional behaviour appraisal (Ungraded) (Hurdle)

NUT5004 Food for dietetics practice Project report (30%) Oral presentation (10%) Food therapeutic manual (20%) Oral presentation (10%) Menu analysis executive report (2,000 words) (20%) Placement portfolio including guided reflection (10%) Professional behaviour appraisal form (Ungraded) (Hurdle) Competency (Practical placement) assessment (Ungraded) (Pass / Fail)

Semester 1

NUT5005 Dietetic Practice 2 Placement portfolio (Pass / Fail) Case study (Pass / Fail Evidence of inter-professional learning (Ungraded) Critical incident reflection and action plan (Ungraded) Attendance (Ungraded) (Hurdle) Professional behaviour appraisal (Ungraded) (Hurdle)

NUT5006 Practice and research in dietetics Scientific paper (40%) (Hurdle) Peer review of scientific paper (10%) Small grant proposal (20%) Teamwork assignment (25%) Attendance and participation (Hurdle) Professional behaviour appraisal (Ungraded) (Hurdle) Final competencies sign off (Hurdle)

Italic font indicates identified competency-based assessment

Figure 2 Monash University Master of Dietetics competency-based assessment systems map.


Community of Practice for dietetics educators

Community of Practice participants

Twenty-one academics joined the Community of Practice. Of these, Fellowship funding supported 10 and the remaining 11 academics were provided with financial support by their respective universities, both to participate and to pay the costs associated with attending a face-to-face initial meeting. Participant demographics are summarised in Table 2. All 21 participants requested to continue to be part of the Community of Practice after the completion of the fellowship.

Table 2: Demographics of members and attendance at Community of Practice

Years as a dietitian

18.3 ± 8.3 years (min 7, max 32)

Highest qualification

9% (n=2) Bachelor; 24% (n=5) Master Dietetics; 43% (n=9) PhD; 24% (n=5) Grad Dip

University 1 BU; 1 CSU; 1 CU; 1 DU; 2 ECU; 1 FU; 1 GU; 1 LU; 1 MU; 1 QUT; 2 US; 2 UC; 3 USC; 2 UOW

Attendance 2015 2016

13-Feb 26-Mar 8-May 19-Jun 31-Jul 11-Sep 20-Nov 29-Jan

n=21

(100%)

n=17

(81%)

n=16

(76%)

n=16

(76%)

n=16

(76%)

n=16

(76%)

n=13

(62%)

n=12

(57%)

BU=Bond University; CSU: Charles Sturt University; CU=Curtin University; DU=Deakin University; ECU=Edith Cowan University; FU=Flinders University; GU=Griffith University; LU=Latrobe University; MU=Monash University; QUT=Queensland University of Technology; the University of Sydney; UC=University of Canberra; USC=University of the Sunshine Coast; UOW=University of Wollongong.

Community of Practice sessions

The first Community of Practice session involved a face-to-face day workshop in Brisbane. All 21 members attended. After that an average of 15 members (72%) attended the seven videoconference discussions and each member attended between three (n=3 members) and seven (n=5 members) of the seven sessions (mean=5 out of 7 sessions).

The first session was conducted in February 2015 at Queensland University of Technology in Brisbane, as this was the most cost-effective venue based on the location of participants. The session was evaluated very positively with statements such as:

A group of very inclusive and collaborative educators who are working towards an evidence-based approach to competency-based assessment.


An inspiring day – an open convivial collaborative gathering of like-minded colleagues, facilitated by Claire – keen to produce some valuable outcomes and

progress the profession.

The participants felt that the formation of the Community of Practice was a milestone, because it allowed collaboration to take place between universities, despite them being in competition for students and clinical placement resources. There was recognition that, rather than being driven by a professional course-accreditation process, evidence-based assessment could be led by this academic educator community. Many acknowledged the leadership qualities and skill of the facilitator and there was recognition of the scholarship around competency-based assessment that the OLT Fellowship had enabled. Some participants felt overwhelmed by the task of trying to implement a change in culture within their own departments and believed that the support of the Community of Practice would assist them to implement better assessment systems, based on evidence. There was a commitment to ongoing research in this area.

The Community of Practice sessions covered a range of discussion areas. Each session followed a similar format, with discussion begun by inviting participants to share any efforts they had been working on relating to competency-based assessment. Topical discussions included advocacy around accreditation standards, release of the revised national competency standards, assessment design decisions, working within university policy to change assessment, and challenges associated with changing dietetic placement educators’ approaches to assessment. In addition, members felt that an important endeavour for the group to work on was that of research questions concerned with how students engage with and understand competency standards and competency-based assessment.

In response, the Fellow designed a research project and sought collaboration from the Community of Practice members. The aim of this research is to explore how students construct and define the concept of ‘professional competence’ and how they view the role of assessment in developing professional competence. It also aims to explore whether teaching and learning approaches can shift students’ attitudes towards assessment, to determine what a student-led approach to competency-based assessment looks like, and to examine how educators can engage students in the development of a competency-based assessment system. Ethics approval has been sought and data collection commenced in October 2015.

Professors Sandra Capra and Susan Ash, leaders in Dietetics Education, attended the final two Community of Practice sessions and, through a facilitated discussion led by the Fellow, reflected on their years as educators, their key learnings and what they saw as challenges for the future of dietetic education. The Fellow will analyse and write up these reflections in due course.


Evaluation of the Community of Practice

Prior to participation in the Community of Practice, members described the main barriers to change to assessment practice as being as follows:

the 2009 competency standards and accreditation processes

a lack of time to innovate and transform

university rules and lead-in times for change

attitudes and skills of assessors (including placement educators/ practitioners), and

programs are not being seen as holistic with little control over early-year undergraduate student assessment.

The survey measuring assessment practice against 18 principles statements showed good internal consistency, with a Cronbach’s alpha coefficient of 0.773. Table 3 describes Community of Practice members’ assessment practice across 18 principles of best-practice assessment, pre- and post-participation. Generally, participants rated their attention to the principles as high prior to the Community of Practice, with all principles except two achieving a mean score of 3.4 or greater. The items where participants felt their attention could be improved included the following:

professional and scholarly approaches to assessment by academic staff are developed, deployed, recognised and rewarded by institutions, and

interim assessment results used for feedback on learning and progress do not play a significant role in determining students’ final grade.

A paired t-test was conducted to evaluate the effect of the Community of Practice on members’ assessment practice. There was a statistically significant increase in members’ reported practice in three areas (Table 3):

(i) Professional and scholarly approaches to assessment by academic staff are developed, deployed, recognised and rewarded by institutions

(ii) Interim assessment results used for feedback on learning and progress do not play a significant role in determining students’ final grade

(iii) Certification accurately and richly portrays graduates’ and students’ achievements to inform future careers and learning.

The significant change in the first two principles may be explained by the lower result in these principles prior to participation in the Community of Practice; however the change observed in perceived attention to assessment for certification is consistent with the focus of discussions in the Community of Practice around systems-based approaches to assessment.


Some principles scores decreased, but not significantly, after the involvement in the Community of Practice. This may be explained by increased knowledge of the factors that contribute to best-practice assessment and thus greater insight into current abilities and practice.

When asked to what degree the Community of Practice was able to fulfil the planned outcomes, members reported that it had mostly achieved outcomes of networking (mean 4.3 out of 5), collaboration (mean 3.9 out of 5) and taking an evidence-based approach to competency-based assessment (mean 3.8 out of 5), but only somewhat achieved new strategies and approaches to assessment (mean 3.3 out of 5) and research outcomes (mean 3.1 out of 5).

Twenty of the 21 members completed the survey in time for the Community of Practice session held after a six-month period, where there was a discussion of the summary of the answers to the question about the most significant change. A summary of the change was presented to members at the final Community of Practice session and agreement was sought on the story of change. The consensus on the most significant change statement that was brought about by their participating in the Community of Practice was:

The Community of Practice has changed our thinking and knowledge about assessment. The need for holistic and student-led assessment has been brought to

the forefront of our minds and the discussions have identified the potential deficiencies of current approaches. The discussions have built our confidence and

encouraged change to practice. We have been empowered to make changes or have put plans in place to make change to redesign assessment systems into the future,

particularly in relation to placement assessment. The network of like-minded people and safe space to share ideas and approaches, as well the timely release of new

competency standards, has facilitated this change. The strong connection between members is needed into the future as we all act as ambassadors for change

within our own universities.

The Community of Practice members also requested that they continue as a network and the Fellow agreed to continue to facilitate four videoconference sessions per year, with a review at the conclusion each year, to allow the work that the network had commenced to be progressed. The Community of Practice members have planned to facilitate a half-day workshop prior to the Dieititan’s Association of Australia National Conference in Melbourne, May 2016 on issues related to the education of dietitians, with a particular focus on competency-based assessment, open to all those interested in the preparation of dietitians for the workforce.


Table 3 Community of Practice members self-reported assessment practice before and after participation

Item from Boud’s principles of assessment Score (max 5) Mean

change pre-post

CoP

p-value from

paired t-test

In response to the following statements, please rate your level of attention to the propositions thinking about your assessment practice, from 1 (not at all) to 5 (completely)

Pre-CoP n=21

Post-CoP n=20

Assessment is designed to focus students on learning 4.25 4.25 0 p=1.00

Assessment is recognised as a learning activity that requires engagement on appropriate tasks

4.40 4.20 -0.20 p=0.330

Feedback is informative and supportive and facilitates a positive attitude to future learning

4.30 4.60 0.30 p=0.163

Students seek and use timely feedback to improve the quality of their learning and work

3.90 4.10 0.20 p=0.214

Students regularly receive specific information, not just marks and grades, about how to improve the quality of their work

4.30 4.40 0.10 p=0.666

Students progressively take responsibility for assessment and feedback processes

3.42 3.63 0.21 p=0.448

Students develop and demonstrate the ability to judge the quality of their own work and the work of others against agreed standards

3.68 3.63 -0.05 p=0.858

Dialogue and interaction about assessment processes and standards are commonplace between and among staff and students

4.00 4.20 0.20 p=0.408

Assessment practices are carefully structured in early stages of courses to ensure students make a successful transition to university study in their chosen field

3.83 4.17 0.34 p=0.231

Assessment practices respond to the diverse expectations and experiences of entering students

3.50 3.40 -0.10 p=0.681

Assessment design is recognised as an integral part of curriculum planning from the earliest stages of course development

4.53 4.63 0.10 p=0.542

Assessment is organized holistically across subjects and programs with complementary integrated tasks

3.53 3.47 -0.06 p=0.834

Professional and scholarly approaches to assessment by academic staff are developed, deployed, recognised and rewarded by institutions

2.95 3.74 0.79 p=0.028*

Assessment practices and the curriculum should be reviewed in the light of graduate and employer perceptions of the preparedness of graduates

4.15 4.35 0.20 p=0.359

Assessment of student achievements is judged against consistent national and international standards that are subject to continuing dialogue, review and justification within disciplinary and professional communities

3.60 3.75 0.15 p=0.481

Interim assessment results used for feedback on learning and progress do not play a significant role in determining students’ final grade

2.53 3.79 1.26 p=0.007*

Evidence of overall achievement to determine final grades is based on assessment of integrated learning

3.75 3.85 0.10 p=0.629

Certification accurately and richly portrays graduates’ and students’ achievements to inform future careers and learning

3.65 4.53 0.88 p=0.014*

*p<0.05 statistically significant.


Chapter 4 Discussion and conclusions This Fellowship program of activities aimed to engage the profession of nutrition and dietetics in transforming competency-based assessment. Using Monash University’s Master of Dietetics program as an exemplar, it set out to design a systems-based approach to competency-based assessment and to establish a Community of Practice to support academic dietetics educators to champion change. The Monash University Master’s program provided an effective illustration of the system of assessment; this facilitated important dialogue on assessment tasks, processes and approaches and created an environment and the need for change. The Community of Practice engaged 20 dietetics educators from Australia and one from New Zealand. It changed their knowledge around best-practice competency-based assessment to be more holistic and student-led and empowered them to make change to practice and systems of assessment within their own universities. There remains a need to continue to build the dialogue about taking a systems-based approach to assessment, transforming our approach in dietetics and spreading the word of this way of thinking across other health professions.

Communities of practice have been found to be an effective mechanism to share knowledge and improve practice15, 16 and advocated as a way of creating an exchange around learning and teaching practice in higher education.17 However their ability to change members’ practice remains unclear.15, 16 The formalised Community of Practice conducted as part of this Fellowship attempted to measure practice impact using mixed methods. Triangulating the data from survey and most significant change technique would suggest that members of the Community of Practice have a new set of knowledge and attitudes around assessment and are committed to changing assessment. The length of time required to make change within higher education may have prevented change from actually taking place during the formalised program. While communities of practice were first described as a form of unstructured social learning, the role of a leader in administering the group and facilitating discussions has been acknowledged as part of their success.15, 17 The role of the Fellow in mobilising the group and facilitating the development of new knowledge was vital to success. As part of future research endeavours, there is a need to continue to monitor the change within higher education that Community of Practice members are able to instigate. Such measures may include: assessment of the partnerships and networks; the members’ roles in leading change; assessment changes within curricula; and student feedback regarding different approaches.

Taking a systems-based approach to assessment provided a mechanism for reform for the dietetics program at Monash University. The workshop and systems map have begun a process for change and highlighted the need to engage practice educators in this process. The Fellowship demonstrated that the systems-based approach was well received by key stakeholders. It offered a new way of thinking to manage many of the challenges brought about by current assessment approaches. While the fellowship engaged a number of placement educators involved in the dietetics program at Monash University, there is a need to continue this momentum for change and to engage more stakeholders in this


dialogue. Doing so provides a great opportunity to transform competency-based assessment to be more student-centred and focused on learning. This Fellowship is only the beginning of a journey to further this way of thinking in dietetics and then across the other health professions. The need to draw accreditation systems into this approach is also paramount for the future. Infiltrating assessment change within work-based learning settings is also an important consideration due to the fundamental role work-based learning has in competency-based assessment. Evaluating the ability to support change in placement educators and assessment, and the factors that enable this, will assist other professions in attempts to change assessment practice.

In 1997, Gale and Grant published a guide on managing change in medical education,18 yet there are few scholarship stories of successful change to assessment practice in higher education,19, or, in particular, of how educators navigate university policies and procedures to effect assessment reform. The role of leadership in supporting change is described.18 The leadership provided by the Fellow has been key to supporting change both within the Community of Practice and in the dietetics team at Monash University. Continuing this leadership and engagement, as well as identifying elements for effective change, are important going forward.

The challenges to taking a systems-based approach to assessment in the health professions are immense. The transformation requires both a change to assessment approaches and a change in the way students themselves approach assessment. Assessment drives learning, but students’ expectations are often to see assessment as a way of measuring learning rather than as a vehicle for learning. Any change to assessment approaches must acknowledge the requirement to pay attention to the need to influence students’ attitudes and approaches to assessment. While students have reported wanting more control over the assessment process,20 teachers will need to support them with the skills and knowledge to be able to do this.

This Fellowship program of activities has commenced the process of transforming competency-based assessment in nutrition and dietetics. By building the capacity of educators and showcasing an example of how taking a systems-based approach can manage many of the challenges in assessment, it has begun to effect change. There is a need for continued investment to support academics, practice educators and assessors to transform their approaches and to measure the impact of this change on the educators and, most importantly, on the students themselves.

This Fellowship has taught me the value of building relationships to affect change. It has also shown me that building individual capacity through extending knowledge and empowering individuals to change practice can affect many more people than the individuals you directly work with. I have also been reminded that assessment cannot be examined in isolation from learning outcomes, content, the people who assess and, most importantly, our students who are being assessed.


References 1. National Health and Medical Research Council. A review of the evidence to address targeted questions to inform the revision of the Australian Dietary Guidelines Canberra: NHMRC (National Health and Medical Research Council) Commonwealth of Australia, 2011.

2. Dietitians Association of Australia. National competency standards for dietitians. 2015.

3. Epstein R and Hundert E. Defining and assessing professional competence. JAMA. 2002; 287: 226-35.

4. Lennie S and Juwah C. Exploring assessment for learning during dietetic practice placements. J Hum Nutr Diet. 2010; 23: 217-23.

5. Palermo C, Capra S, Ash S, Beck E, Truby H and Jolly B. Are clinical supervisors’ attitudes to assessment a barrier to expanding placement opportunities? Results of a Delphi study. Nutrition and Dietetics. 2015 (online).

6. Palermo C, Beck E, Chung A, et al. Work-based assessment: qualitative perspectives of novice nutrition and dietetics educators. J Hum Nutr Diet. 2013; 27: 513–52.

7. Norcini J, Anderson B, Bollela V, et al. Criteria for good assessment: Consensus statement and recommendations from the Ottawa 2010 conference. Med Teach. 2011; 33: 206-14.

8. Dalton M, Davidson M and Keating J. The Assessment of Physiotherapy Practice (APP) is a valid measure of professional competence of physiotherapy students: a cross-sectional study with Rasch analysis. J Physio. 2011; 57: 239-46.

9. Speech Pathology Australia. COMPASS® Online Assessment Tool. 2009.

10. Mennin S. Self-organisation, integration and curriculum in the complex world of medical education. Medical Education. 2010; 44: 20-30.

11. Morgan K and Hughes R. A profile of the size and professional attributes of the Australian academic dietetic educator workforce. Nutrition and Dietetics. in press.

12. Boud D. Assessment 2020: Seven propositions for assessment reform in higher education. Australian Learning and Teaching Council, 2009.

13. Davis R and Dart J. The 'Most Significant Change' (MSC) Technique. United Kingdom: CARE International, 2005.


14. Liamputtong P. Research methods in health. Foundations for evidence based practice. South Melbourne: Oxford University Press, 2010.

15. Li L, Grimshaw J, Nielsen C, Judd M, Coyte P and Graham I. Use of communities of practice in business and health care sectors: A systematic review. Implement Sci 2009; 4: 27.

16. Ranmuthugala G, Plumb J, Cunningham F, Georgiou A, Westbrook J and Braithwaite J. How and why are communities of pracitce established in the healthcare sector? A systematic review of the literature. BMC Health Services Research. 2011; 11.

17. McDonald J. Community, Domain, Practice: Facilitator catch cry for revitalising learning and teaching through communities of practice. Sydney: Office for Learning and Teaching, Department of Education, 2014.

18. Gale R and Grant J. AMEE Medical Education Guide No. 10: managing change in a medical context: guidelines for action. Med Teach. 1997; 19: 239-49.

19. Chowthi-Williams A, Curzio J and Lerman S. Evaluation of how a curriculum change in nurse education was managed through the application of a business change management model: A qualitative case study. Nurse Education Today. 2015.

20. Palermo C, Chung A, Beck E, et al. Evaluation of assessment in the context of work-based learning. Qualitative perspectives of new graduates. Nutr Diet. 2014; accepted ahead of print.


Appendix A Certification by Deputy Vice-Chancellor (or equivalent)

I certify that all parts of the final report for this OLT Fellowship provide an accurate representation of the implementation, impact and findings of the project, and that the report is of publishable quality.

Name: Professor Darrell Evans Date: 28 September 2015 Vice-Provost (Learning and Teaching) Monash University


Date post:	29-Dec-2021
Category:	Documents
Upload:	others
View:	4 times
Download:	0 times

Transforming competency-based assessment in nutrition and ...

Documents