IMPROVING PERSONALIZED FEEDBACK AT THE WORKPLACE WITH A LEARNING ANALYTICS ENHANCED E-PORTFOLIO MARIEKE VAN DER SCHAAF, UTRECHT UNIVERSITY GERALDINE CLAREBOUT, MAASTRICHT UNIVERSITY
www.project-watchme.eu
@Project_WatchMe
This project has received funding from the European Union’s Seventh Framework Programme for research, technological development and demonstration under grant agreement no 619349
1. Utrecht University, NL 2. University Medical Centre Utrecht, NL 3. Szent Istvan University, Hungary 4. University of Tartu, Estonia 5. Universitätsmedizin Charité Berlin, Germany 6. University of California San Francisco, USA 7. Maastricht University, NL 8. Mateum, NL 9. University of Reading, UK 10.Jayway, Denmark 11.NetRom, Romania/NL
1 I N T R O D U C T I O N W A T C H M E
Develop complex competences
Integrated in context
Demands long learning trajectories in workplace
Deliberate practice: feedback and reflection
2
IMPROVE USE OF DATA IN E-PORTFOLIO WITH LEARNING ANALYTICS
3
Application of probabilistic student models that enable feedback based on multi sorted assessments
Measurement, collection, analysis and reporting of data about trainees in their contexts, for the purpose of understanding, and optimising learning and the utilising of environments in which it occurs (Solar, 2013)
Personalized feedback
Visualizing learners’ development
I N T R O D U C T I O N W A T C H M E
THREE WAYS OF PERSONALIZED FEEDBACK
• Regular e-portfolio feedback, e.g. assessment forms
Probabilistic, derived from data in the portfolio: • Performance estimations, based on earlier scores
or narratives • Pedagogical estimations, e.g level of a student’s
frustration, consistency
I N T R O D U C T I O N W A T C H M E
EPASS
Intuitive user interface Monitoring of progress Mobile web application
for entering forms on smartphone or tablet
I N T R O D U C T I O N W A T C H M E
EXAMPLE DATA POINTS E-PORTFOLIO
Aggregation of data points Written or electronic test Skills test in simulation (OSCE) Workplace-based assessment
(practice observation) Case-based discussion Multisource feedback Product evaluation
6 I N T R O D U C T I O N W A T C H M E
PERSONALIZED FEEDBACK
9
Some users are motivated by competition? Others
find competition demotivating.
Users want to know whether they are on
track.
Users want to be able to highlight and
save useful feedback.
Users would like an overview of weekly,
monthly and yearly goals.
I N T R O D U C T I O N W A T C H M E
JIT Written feedback: supervisors’ feedback and rubric-based improvement advice.
Visualisations of students’ progress and development within competencies (e.g. graphs)
10 To create the interface(s) within the e-portfolio, in which information about development and progress will be shown to students.
VIZ .
architecture
Appears in: • Dashboard • Assessment forms • Pages with competencies
• Feedback categories: • Improvement feedback • Supervisor feedback
11
WRITTEN FEEDBACK (JIT)
I N T R O D U C T I O N W A T C H M E
• Appears in: • Dashboard • Competency pages
• Visualisations • Timeline • Current Performance • General Overview • Supervisor View
12
VISUALISATION
I N T R O D U C T I O N W A T C H M E
13
EXAMPLE - TIMELINE
How am I developing?
What competencies need my attention?
I N T R O D U C T I O N W A T C H M E
How am I doing right now?
What are my strengths and weaknesses?
14
EXAMPLE - CURRENT PERFORMANCE
I N T R O D U C T I O N W A T C H M E
15
EXAMPLE – SUPERVISOR VIEW
How are my students progressing?
Which of my students require special attention?
I N T R O D U C T I O N W A T C H M E