NMC Online ConferenceDecember 2005

Assessing Learning in a MORPG

Patricia Youngblood, PhD Director of Evaluation, SUMMIT(Stanford University Medical Media & Information Technology)

Phillip M. Harter, MDResidency Program DirectorDivision of Emergency Medicine Stanford University School of Medicine

NMC Online ConferenceDecember 2005

Comparison of the Virtual ED & HPSHuman Patient

Simulator

Virtual ED (Emergency Department)

NMC Online ConferenceDecember 2005

Virtual ED (Emergency Department)

NMC Online ConferenceDecember 2005

Debriefing with instructor

Student being assessed

Standardized team member

NMC Online ConferenceDecember 2005

Study Design: Pretest/Posttest

30 medical students & interns

Virtual ED Simulator (n=16)pretest case 4 training cases posttest case

Human Patient Simulator “HPS” (n=14)pretest case 4 training cases posttest case

Is there a significant difference in performance between the two student

groups?

NMC Online ConferenceDecember 2005

EMCRM Rating Scale of Team Performance (1-5)

• Knowledge of the environment• Anticipation of & planning for potential problems• Assumption of leadership role• Communication with other team members• Distribution of workload/delegation of responsibility• Attention allocation• Utilization of information• Utilization of resources• Recognition of limitations/call for help early enough• Professional behavior/interpersonal skills• Overall team leadership skills (1= not acceptable; 5=excellent)

NMC Online ConferenceDecember 2005

4. Communication with other Team Members

1 2 3 4 5

Average performance (3): Communicates clearly with all team members; speaks clearly, calls team members by name, can be heard easily

Below Average (1-2): Cannot be heard easily, doesn’t identify to whom he is speaking, talks ‘down’ to other members, does not close the communication loop

Above Average (4-5): Closes the communication loop, thinks/talks out loud so other members can follow his problem solving approach

Guidelines for Scoring Team Leadership in the Virtual ED

Comparison of Pre & Post Mean ScoresM

ea

n S

co

res

0.00

10.00

20.00

30.00

40.00

50.00

HPS Group

Pretest Sum Scores Posttest Sum Scores Pretest Sum Scores Posttest Sum Scores

Virtual ED Group

NMC Online ConferenceDecember 2005

Results Summary

• Users’ performance rating scores improved between pretest case and posttest case with both the MORPG and the Human Patient Simulator;

• Analysis of performance data shows no statistically significant differences between the two treatment groups;

• Reliability of the EMCRM rating scale was high (Cronbach’s alpha = .96);

• Users liked both types of simulation training.

NMC Online ConferenceDecember 2005

The Stanford Research Team