SYMPOSIUM

ENTRUSTMENT AND LEARNING ANALYTICS IN E-PORTFOLIOS FOR WORKPLACE LEARNING AND ASSESSMENT

PRESENTERS: WATCHME-TEAMDISCUSSANT: PROF. DR. DAVID BOUD, UNIVERSITY OF TECHNOLOGY,

SYDNEY, AUSTRALIA

www.project-watchme.eu

@Project_WatchMe

Workplace-based e-Assessment Technology for Competency-based Higher Multi-professional Education

This project has received funding from the European Union’s Seventh Framework Programme for research, technological development and demonstration under grant agreement no 619349

E A R L I C O N F E R E N C E , 2 7 A U G 2 0 1 5 , L I M A S S O L C Y P R U S

E A R L I C O N F E R E N C E , 2 7 A U G 2 0 1 5 , L I M A S S O L C Y P R U S

1. Utrecht University, NL

2. University Medical Centre Utrecht, NL

3. Szent Istvan University, Hungary

4. University of Tartu, Estonia

5. Universitätsmedizin Charité Berlin, Germany

6. University of California San Francisco, USA

7. Maastricht University, NL

8. Mateum, NL

9. University of Reading, UK

10.Jayway, Denmark

11.NetRom, Romania/NL

2

E A R L I C O N F E R E N C E , 2 7 A U G 2 0 1 5 , L I M A S S O L C Y P R U S

WATCHME’S AIM

Improve efficiency and quality of workplace- based feedback and assessment by means of a mobile electronic portfolio system, that is enhanced:

Conceptually with the concept of Entrustable Professional Activities

Technically through Learning Analytics:

Student models that monitor the learners’ competency development and

inform learners and supervisors (based on data of students, supervisors

and peers - probabilistic algorithms that learn from new incoming data)

Personalized feedback and visualization of development.

3

E A R L I C O N F E R E N C E , 2 7 A U G 2 0 1 5 , L I M A S S O L C Y P R U S

OVERVIEW PRESENTATIONS

1. prof. Harm Peters, Charité - Universitätsmedizin Berlin, Germany

Delphi study into Entrustable Professional Activities

2. dr. Bert Slof, Utrecht University, the Neth. and

dr. Äli Leijen, University of Tartu, Estonia

Competences and Assessment of Student Teachers

3. dr. Marieke van der Schaaf, Utrecht University, the Neth. and

dr. Bert Slof, Utrecht University, the Neth.

Electronic Portfolios and Learning Analytics

4. Ing. Eelco Scheurs, Maastricht University, The Netherlands

Participatory Design of Learning Analytics

4

ENHANCING ELECTRONIC PORTFOLIOS FOR WORKPLACE-BASED ASSESSMENT BY LEARNING ANALYTICS

UTRECHT UNIVERSITY: MARIEKE VAN DER SCHAAF AND BERT SLOF

www.project-watchme.eu

@Project_WatchMe

E A R L I C O N F E R E N C E , 2 7 A U G 2 0 1 5 , L I M A S S O L C Y P R U S

E A R L I C O N F E R E N C E , 2 7 A U G 2 0 1 5 , L I M A S S O L C Y P R U S

Develop complex competences

Integrated in context

Demands long learning trajectories in workplace

Deliberate practice: feedback and reflection

6

E A R L I C O N F E R E N C E , 2 7 A U G 2 0 1 5 , L I M A S S O L C Y P R U S

“Well done!”“Pleasure to supervise!”“Reliable candidate”“Poor fund of content knowledge”“Needs lots of supervision”

UNFORTUNATELY, HOW MANY DAILY FEEDBACK PRACTICES LOOK LIKE

7

E A R L I C O N F E R E N C E , 2 7 A U G 2 0 1 5 , L I M A S S O L C Y P R U S

Personalized Feedback that gives (Sadler, 1989; 2010):

insight into performance

ability to evaluate and monitor own process

suggestions to fill gap between expected norm and performance

That feeds into learners’ major feedback questions (Hattie & Timperley, 2007):

Where am I going? (goals, feedup)

How am I going? (feedback)

Where to next? (feedforward)

WHAT IS NEEDED

8

E A R L I C O N F E R E N C E , 2 7 A U G 2 0 1 5 , L I M A S S O L C Y P R U S

PRINCIPLES OF GOOD FEEDBACK (NICOL AND MACFARLANE-DICK, 2006)

1. Helps clarify what good performance is

2. Facilitates the development of self-assessment in learning

3. Delivers high quality information to learners about their learning

4. Encourages teacher and peer dialogue around learning

5. Encourages positive motivational beliefs and self-esteem

6. Provides oppurtunities to close the gap between current and desired performance.

7. Provides information to supervisors

9

E A R L I C O N F E R E N C E , 2 7 A U G 2 0 1 5 , L I M A S S O L C Y P R U S

ENTRUSTABLE PROFESSIONAL ACTIVITY

Task based instead of construct based approach

Crucial question: would I entrust this learner unsupervised with this task? (with my sick mother, animal or teach my daughter/son…)

An EPA is a task that an individual can be trusted to perform unsupervised, in a given professional context, once sufficient competence has been demonstrated. International Competency-Based Medical Education Collaborators, March 18, 2014

Ten Cate, Chen, Hoff, Peters, Bok & Van der Schaaf, 2015

10

E A R L I C O N F E R E N C E , 2 7 A U G 2 0 1 5 , L I M A S S O L C Y P R U S

COMBINE EPA APPROACH WITH LEARNING ANALYTICS

11

Application of probabilistic student models that enable feedback based on multi sorted assessments

Measurement, collection, analysis and reporting of data about trainees in their contexts, for the purpose of understanding, and optimising learning and the utilising of environments in which it occurs (Solar, 2013)

Personalized feedback

Visualizing learners’ development

E A R L I C O N F E R E N C E , 2 7 A U G 2 0 1 5 , L I M A S S O L C Y P R U S

ELECTRONIC PORTFOLIO

12

E A R L I C O N F E R E N C E , 2 7 A U G 2 0 1 5 , L I M A S S O L C Y P R U S

THE OVERALL WATCHME ARCHITECTURE

13

E A R L I C O N F E R E N C E , 2 7 A U G 2 0 1 5 , L I M A S S O L C Y P R U S

ASSESSMENT ARGUMENTS (MISLEVY, 2006)

What EPAs should be assessed and how

does the learner develop during the

curriculum?

What performance indicators should be

used to gain insight into a learner’s

competence?

What instruments should be used to assess

the EPAs?

StudentModel

EvidenceModel

TaskModel

E A R L I C O N F E R E N C E , 2 7 A U G 2 0 1 5 , L I M A S S O L C Y P R U S

ARCHITECTURE OF BAYESIAN STUDENT MODEL

EPA Level

CompetencyLevel

Assessment outcomes

Narratives

States

Feedback decision

assessment

assessment

assessment

JIT FEEDBACK

PORTFOLIO MEBN FRAGMENTS

EPA decision

Interest selector

15

E A R L I C O N F E R E N C E , 2 7 A U G 2 0 1 5 , L I M A S S O L C Y P R U S

BAYESIAN STUDENT MODEL

represents the actual internal cognitive state of each learner as well as their

actual learning context.

contains enough pedagogical knowledge in order to be able to translate the

internal state and context into meaningful messages and information for

visualization.

is a back-end component, meaning that no direct user interaction is made

with this module.

will make suggestions to help the learners, assessors or supervisors improve

their performance.

16

E A R L I C O N F E R E N C E , 2 7 A U G 2 0 1 5 , L I M A S S O L C Y P R U S

Domain SM vs Individual SM

Domain SM (DSM)

Will encapsulate all the pedagogical knowledge required to model one

domain

Possible models are: Anaesthesiology Training, Veterinary Education,

Teacher Education and General Undergraduate Medical Education

Variations of these proposed models:

Veterinary Education (NL vs HU)

Teacher Education (NL vs EE)

General Undergraduate Medical Education (NL vs DE)

17

E A R L I C O N F E R E N C E , 2 7 A U G 2 0 1 5 , L I M A S S O L C Y P R U S

Domain SM vs Individual SM

Individual SM (ISM)

Will encapsulate all the pedagogical knowledge required to

model one learner.

The Individual SM will be a personalized domain model

⚠ One ISM cannot handle more than one domain at once

⚠ For multiple roles of the same individual (e.g. a learner is

both a supervisor and a trainee) the ISM will cover only the

learner’s role

18

E A R L I C O N F E R E N C E , 2 7 A U G 2 0 1 5 , L I M A S S O L C Y P R U S

JUST IN TIME PERSONALIZED FEEDBACK

Type of feedbackContent of feedbackMoment of feedback

19

The Bayesian student model must indicate appropriateness of:

Some users are motivated by competition? Others

find competition demotivating.

Users want to know whether they are on

track.

Users want to be able to highlight and

save useful feedback.

Users would like an overview of weekly,

monthly and yearly goals.

E A R L I C O N F E R E N C E , 2 7 A U G 2 0 1 5 , L I M A S S O L C Y P R U S

TASK AND EVIDENCE MODELS

E A R L I C O N F E R E N C E , 2 7 A U G 2 0 1 5 , L I M A S S O L C Y P R U S

TYPES OF FEEDBACK

21

E A R L I C O N F E R E N C E , 2 7 A U G 2 0 1 5 , L I M A S S O L C Y P R U S

TYPES OF FEEDBACK

22

E A R L I C O N F E R E N C E , 2 7 A U G 2 0 1 5 , L I M A S S O L C Y P R U S

SURVEY REQUIREMENTS PERSONALIZED FEEDBACK

Two rounds, n = 8

Experts perceived the preliminary design

for personalized feedback to be indicative

for high quality and useful feedback

Most of the comments experts made

were in congruence of the principles of

good feedback

Result: least attention was paid to

support student’s self-esteem and

motivation

23

E A R L I C O N F E R E N C E , 2 7 A U G 2 0 1 5 , L I M A S S O L C Y P R U S

Example program codes: JIT Feedback

24

{

"EPA": "TT_UU_EPA5", "PerformanceIndicator": "TT_UU_PI1", "PerformanceIndicatorLevel": "TT_UU_LEVEL3", "Translations": { "en": “check more often when existing tests and digital testing systems are inadequate and design new tests (incl. correction sheets).", "nl": “vaker te controleren of bestaande toetsen en digitale toetssystemen ontoereikend zijn en ontwerp hiervoor een nieuwe toetsen (incl. correctiemodel)." }

E A R L I C O N F E R E N C E , 2 7 A U G 2 0 1 5 , L I M A S S O L C Y P R U S

Example Personalized feedback in EPASS portfolio

25

E A R L I C O N F E R E N C E , 2 7 A U G 2 0 1 5 , L I M A S S O L C Y P R U S

Example Personalized Feedback in EPASS portfolio

26

E A R L I C O N F E R E N C E , 2 7 A U G 2 0 1 5 , L I M A S S O L C Y P R U S

1. Utrecht University, NL

2. University Medical Centre Utrecht, NL

3. Szent Istvan University, Hungary

4. University of Tartu, Estonia

5. Universitätsmedizin Charité Berlin, Germany

6. University of California San Francisco, USA

7. Maastricht University, NL

8. Mateum, NL

9. University of Reading, UK

10.Jayway, Denmark

11.NetRom, Rumania/NL

THANK you for your attention!

27

E A R L I C O N F E R E N C E , 2 7 A U G 2 0 1 5 , L I M A S S O L C Y P R U S

ARCHITECTURE & API DESIGN

Student Model Server

Data Merger

Natural Language Processor

Student Model

API Dispatcher

Bayesian Network Manager

BN Model Storage

EPASS External API

JIT/VIZ External API

Privacy Manager

Numerical Data Processor

Error reporting

Data Storage

28

E A R L I C O N F E R E N C E , 2 7 A U G 2 0 1 5 , L I M A S S O L C Y P R U S

2. 2. FACILITATES THE DEVELOPMENT OF SELF-ASSESSMENT

(REFLECTION) IN LEARNING

Timeline overview:Detailed visualization:

29