SCHOOL  OF  MEDICINE  

Fall  2014  

Educational  Technology  Innovations  Evaluation  

Report  2013-­‐14  Christy  Boscardin,  PhD  

A special thank you and acknowledgment to our innovative and dedicated medical educators who are true pioneers:

June Chan, ScD Susannah Cornes, MD Vanja Douglas, MD Kathy Hyland, PhD Descartes Li, MD Dan Lowenstein, MD  

 

 

Christy  Boscardin  Christian  Burke  

Chandler  Mayfield  Dylan  Alegria  Patricia  Nason  

Holly  Nishimura  Julia  Vandermeer  

 

2    

UCSF Educational Technology Evaluation Report 2013-14

BACKGROUND

The proliferation of Information and Communications Technology (ICT) in recent years is changing

the educational environment. A recent survey conducted by the Pew Internet & American Life Project

found that more than 56% of American adults have smart phones and 34% own a tablet computer.

According to the latest data from the Babson survey, about 7.1 million American students are taking

at least one online course now (Chronicle of Higher Ed). These emerging technologies are changing

the landscape of how we learn and acquire knowledge in and outside the traditional educational

structures (Downes, 2011; Fournier & Kop, 2010). However, research on learning technology

suggests that how the tools are used and supported are critical for successful implementation and

desired educational outcome (Salomon, Perkins, & Globerson, 1991).

PURPOSE OF THIS REPORT

At UCSF, we are considering several new instructional technology tools to promote pedagogical

innovations in medical education. In this evaluation, we report on the findings related to:

1) Level of learner adoption and engagement with these several new instructional technology tools

(i.e. reasons for learner adoption, barriers to adoption, features of the tool that worked/need

improvement, etc)

2) Impact on learning outcomes including changes in learning process and performance (i.e. self-

regulated learning, access to content)

3) Impact on instructional planning and delivery (i.e. inquiry based: iRocket reader; hypothesis

driven: NeuroExam; flipped classroom: BMB)

DESCRIPTION OF THE TOOLS:

IROCKET READER: The iROCKET Reader organizes learning by concept, helping learners to

pinpoint what areas they understand, and what areas they need to continue to study. Each concept is

like a leaf on the branch of a tree. Each leaf covers one or two important learning objectives using a

combination of images, text and rich-media, such as videos. For each leaf learners can check their

understanding by answering several short self-assessment questions. Students can mark leaves that

need further study and track their own progress relative to their peers.

3    

The iROCKET Reader also allows faculty to see how learners are progressing through the material

and to see learning trends. Noticing if a large group of learners are struggling with the same concepts

allows faculty to address this more quickly than with traditional learning methods and also allows

faculty to quickly improve their learning materials.

The iROCKET Reader was developed and hosted by Odigia, a learning technology company based in

North Carolina. The iROCKET Reader pilot focused on core content in Genetics and

Epidemiology/Evidence-Based Medicine traditionally delivered through the course syllabi in

Prologue, Organs, M&N, BMB and M3. The new learning content on iROCKET was developed in

partnership with Curriculum Ambassadors in the summer of 2013, course faculty, agraduate student

and post-doc. For more info: http://vimeo.com/75066548  

NEURO-EXAM APP: The NeuroExam Tutor presents an innovative approach to learning the

neurological physical exam. With clinical cases, physician-authored information and a library of real

patient videos, the NeuroExam Tutor helps medical students, residents and practicing physicians

perfect their understanding and execution of the neurological exam. The Neuroexam Tutor was

developed as a partnership between University of California, San Francisco and Bandwdth

Educational Publishing. It was made possible by the work of many educators, technologists,

designers, producers and developers, including:

• Susannah Cornes MD, Vanja Douglas MD, Dylan Alegria MS, S. Andrew Josephson MD,

Dan Lowenstein MD, and the Technology Enhanced Learning team in the UCSF School

of Medicine Office of Medical Education.

4    

For more info: http://vimeo.com/album/2234882    

Features:

• More than 60 high quality videos

• In-depth descriptions of how to execute more than 50 different physical exam maneuvers

• 6 interactive cases with real patient videos

• Descriptions of 8 exam categories with explanations of terminology and grading scales

• Quick reference flashcards for 6 common neurological complaints

• Pearls and pitfalls from the master clinicians at UCSF

The NeuroExam Tutor App was first pilot in the Neurology Core Clerkship in 2013-14.

BMB ONLINE AND FLIPPED CLASSROOM: The BMB Online Lecture project entailed creating 19

hours of new online learning content in the form of short chunked highly produced online videos.

These online lectures replaced content that was delivered via-in person lectures in the first year Brain,

Mind & Behavior course. The goal was to create more flexibility in the curriculum, more

opportunities for active and self-directed learning and to use technology to enhance the delivery of

core curriculum content. The videos were produced by TEL in partnership with UCSF ETS and the

faculty content experts. For more info: http://vimeo.com/album/2362353  

EVALUATION FRAMEWORK:

THEMATIC CONCEPT TARGETED FOR 2013-14: SELF-REGULATED LEARNING

Self-regulated learning promotes life-long learning through use of metacognitive, motivational,

and behavioral strategies to regulate the learner's own learning process. In self-regulated learning, the

focus is on the learners taking the initiative in the learning process. Self-regulated learning is

conceived of as a learning process in which learners employ self-regulatory skills, such as self-

assessing, self-directing, controlling and adjusting, in order to acquire knowledge (Zimmerman,1989).

The ability to self-regulate is a key aspect of lifelong learning (Blumberg, 2000; Denton, et al 2000;

Dolmans & Schmidt, 1996; Schutz & Davis, 2000). The complexity and the rapid advances in

medical knowledge require our learners to be adaptive and develop life-long learning skills through

self-regulated learning. Effective teaching and learning empowers the learner become more self-

regulated and develop skills for adaptive expertise and life-long learning.

5    

In this evaluation, we draw on Zimmerman’s model of self-regulated learning to focus on three

phases of cognitive and behavioral strategies to evaluate the efficacy of the various new educational

technologies in enabling learners to develop these critical skills (Zimmerman, 2002). The three phases

of the self-regulation model includes: 1) forethought, 2) self-monitoring/performance, and 3)

evaluation. The forethought stage involves goal setting, planning, and motivational processes. The

second phase includes prompting the learners to monitor, and focus on the learning through inquiry.

This is closely linked to metacognitive skills. The last phase of self-regulated learning involves self-

reflection and appraisal of their learning strategies and performance.

UNITED THEORY OF ACCEPTANCE AND USE OF TECHNOLOGY MODEL

In addition to the SRL, using the United Theory of Acceptance and Use of Technology Model (Venkatesh et al. 2003), we focus on three aspects of the evaluation construct: 1) Adoption Level, 2) Perceived Benefits to Learning, 3) Challenges to Adoption.

Depending on the particular technology, selected items will be chosen for the survey:

Constructs Definition Potential Questions Level of Adoption and Usage

To determine whether the technology is being adopted and the degree of diffusion in the current context

• Frequency of Use • Ease of Use • Level of encouragement by the instructor to adopt • Level of encouragement from peers for

adoption/usage • Level of alignment with current course structure • Usability compared to other format (i.e. traditional

syllabus, podcast, coursera, etc)

Benefits To determine the degree to which the students perceive that the use of this technology will benefit their learning

• Assisted in obtaining content knowledge • Assisted in organizing and synthesizing content

(related to self-regulated learning: SRL) • Assisted in self-assessment and monitoring (related

to SRL) • Assisted in identifying areas for improvement

(related to SRL) • More individualized learning

Challenges/Areas of Improvement

To determine the challenges to adoption (including technological barriers,

• Level of recommendation to others • Any technological features that discourages

participation • Any course/contextual barriers to adoption (time

commitment, not a requirement, etc)

6    

context, social norm, etc)

• Areas for improvement

EVALUATION METHOD:

For 2013-14, we will focus on 3 specific instructional technology: 1) iRocket Reader, 2) NeuroExam App, and 3) BMB: Online Video lectures (Flipped Classroom).

Tools Evaluation Questions Addressed

Method & Data Source

iRocket Reader: online syllabus User Adoption

• First Year Medical Students (MS1)

• Second Year Medical Students (MS2) Survey

Changes in learning process (self-regulated learning, etc)

• MS1 and MS2 Survey

Impact on instruction • MS1 and MS2 Survey • Instructor Interview

NeuroExam App: physical exam tutoring App

User Adoption

• MS3 End of Neurology Clerkship Survey

• CPX Interstation Survey from UCSF and UC Irvine

Changes in learning process (self-regulated learning, etc)

• MS3 End of Neurology Clerkship Survey

• CPX Interstation Survey from UCSF and UC Irvine

Impact on instruction • Performance Exam Scores (CPX) • Developer Survey

BMB: Flipped classroom User Adoption • MS1 Survey

• MS1 Video User Log

Changes in learning process (self-regulated learning, etc)

• MS1 Survey • MS1 Focus Group

Impact on instruction • Instructor Survey

RESULTS FOR IROCKET READER:

7    

Instructional Challenge/Problem iRocket is Trying to Address:

v Current syllabus in PDF form created challenge to keep content of the course reader up to

date (static document). Inefficient production process.

v Traditional course reader is not dynamic and interactive. It does not allow for learner

flexibility.

Goals of the Tool:

v Develop Interactive, inquiry-based platform that engages the learner

v Improved presentation of images

v Ability to author content in teams and edit in real time

v Opportunities for innovative teaching modalities

LEARNER ENGAGEMENT/ADOPTION:

v Forty-five MS1s and forty-one MS2s participated and completed the evaluation survey. Both

MS1s and MS2s responded that they found the iRocket Reader only slightly useful. On a 5-

point scale for the level of usefulness, the mean for MS1s was (x1=1.71, sd=0.82) and the

mean for MS2s (x2=1.71, sd=0.98).

v Both MS1s and MS2s reported difficulty with the access to the iRocket materials.

v Both MS1s (62%) and MS2s (73%) found the chunking of information into individual

concepts (e.g. by leaf) was only slightly or not at all useful.

LEARNING PROCESS (SELF-REGULATED LEARNING):

v Students reported that the “Check My Learning” feature of iRocket Reader was one of the

most useful features of the technology. This feature provides an opportunity for students to

self-assess and monitor their learning progression.

v Other features of the technology (i.e. Tree-Branch-Leaf outline, Progress Leaf Map, Links to

other websites) which were designed to elicit forethought (planning stages of the self-

regulation process) and evaluation (self-appraisal and reflection and planning next steps)

were not as successful in facilitating these strategies as indicated by student responses.

INSTRUCTIONAL IMPACT:

v The course directors found the dynamic features of the iRocket reader very useful. Ability to

author content in teams and edit in real time were both key features that made the adoption of

iRocket worthwhile.

8    

v Despite the low enthusiasm from the students, both course directors thought that the iRocket

Reader platform provided a forum to implement the inquiry approach to learning the content.

v Although the interactive of the iRocket reader provided innovative teaching modalities,

technological limitations of the tool made access to the materials more difficult for the

students.

RESULTS FOR NEUROEXAM APP:

Instructional Challenge/Problem:

v To help medical students, residents and physicians overcome “neurophobia,” the feeling many people get when given the seemingly impossible task to learn and master the comprehensive version that has been traditionally taught by head-to-toe approach.

Goals of the Tool:

v To facilitate hypothesis driven approaches to physical exam by offering six interactive

case-based videos with real patients and health care experts.

v To provide a novel approach to learning the neurological physical exam by

providing a challenging series of assessments aimed at diagnosing neurological

disorders in patients.

v Provide videos and an explanation of normal physical exam findings, but also

provides interactive cases to hone your neuro exam acumen.

LEARNER ENGAGEMENT/ADOPTION:

v BMB: Of the 33 students who completed the user engagement survey, 70% of the

students used the App during BMB.

v Clerkship: About 32% (n=38 out of 118) of MS3s used the app during their clerkship.

v Of the students who used the App (n=27), 93% of the students agreed (some strongly

agreed) that App was useful.

v A lack of access to iPad was one of the biggest barriers to adoption of the App.

LEARNING PROCESS (SELF-REGULATED LEARNING):

9    

v Overall, the self-regulation skills between NeuroExam App users (NET) and non-users were

similar. Students indicated generally high self-regulations skills regardless of the app usage.

INSTRUCTIONAL IMPACT:

v Clinical Performance Examination (CPX): There were no significant differences in student performance on history taking, physical exam skills, or neurological specific items between users of the NeuroExam app vs. non-app users on the standardized patient neurological case.

RESULTS FOR BMB FLIPPED CLASSROOM:

Instructional Challenge/Problem:

v As the medical school curriculum changes to allow more workplace learning, large lectures will

become difficult to schedule. To facilitate this change, we used online lectures to provide some of

the didactic instruction in an asynchronous manner. This further allowed faculty to redesign

formally passive didactic lecture sessions into more active learning experiences.

Goals of the Tool:

v To deliver didactic content online and allow in-class sessions to focus on interactive

learning sessions including patient presentations.

v To allow instructors to show learning environments that are not possible in a lecture

(videos of clinical settings, clinical procedures).

10    

v To provide specific self-assessment questions to accompany the online lecture material

that students could use to track their own understanding of the material.

v To better prepare students for active learning sessions.

LEARNER ENGAGEMENT/ADOPTION:

v Overall, students had generally positive attitude towards BMB online lectures. On a five

point scale (1= not very helpful to 5 = excellent way to learn the material), the average

response was 3.9 (sd = 0.82).

v Majority of the students either were indifferent (23%) or would like to see more (51%)

classes use online lectures.

v Most (93%) watched the videos only once. Out of those that watched more than once,

pharmacology videos were viewed more than once due to difficulty of the material.

v Students reported that since they were able to watch the videos in faster speed, the time

allocation was less than to traditional in-class lecture time.

v In contrast to other Kahn Academy videos (designed to be about 10 minutes), students

varied in their preference for the length of videos. The suggested length ranged from 15

minutes to full length of in-class lecture. Some also commented that time was not an issue

and time should be allocated based on the amount of content needed to be cover.

v Most students (98%) watch the videos using their personal computer compared to other

devices including tablet/ipad.

v Some of the problems cited for barrier to adoption/engagement include: 1) poor audio

quality (radiology), 2) lack of coherence to the syllabus (Opiods).

v Student use of self assessment questions was mixed. Some appreciated them, but others

prefer a quiz that they can take long after watching the videos specifically as a study tool.

v More than half (54%) of the students indicated that they would like to see about the same or

more of the class lectures online relative the amount allocated in BMB. Only 7% responded

that they would like to see much less in comparison to BMB.

LEARNING PROCESS (SELF-REGULATED LEARNING):

v In comparison to sitting in a lecture, 45% of the students thought that the online lectures enabled

them to learn the material about the same and 29% thought it was better than going to the live

lecture.

11    

v Students reported that dividing the lectures into shorter modules with clearly defined objectives

helped them understand the structure of the lecture and what they were expected to learn.

INSTRUCTIONAL IMPACT:

v Overall, there was no significant difference in student ratings on the quality of the couse

compared to previous year.

  2013  (1-­‐5:  Poor  to  Excellent)  

2014  (1-­‐5:  Poor  to  Excellent)  

Overall  quality  of  course   4.60   4.66  Overall  quality  of  lectures   4.47   4.24  Overall  quality  of  small  groups   4.09   4.35  

v Some instructors indicated that presenting online lectures gave them opportunity to use more

videos and visual elements during lecture.

v The format also allowed them to break down each lecture into smaller components.

v It also provided an opportunity to review the alignment between the syllabus and lecture.

v Some instructors just transferred their live lectures to online lecture with minimal adjustments

or modifications.

SUMMARY AND RECOMMENDATIONS

v Given the low adoption of the tools, evaluation of the overall impact of the tools on

learning was limited and not generalizable.

v Instructors and course directors found that the development and the features of the new

technologies afforded new pedagogical approaches and provided forum for innovation.

v Despite the lack of significant impact of the tools on the development of self-regulated

skills, further studies examining the impact of the tools on the student’s learning plan and

approach to learning should be considered in the future.

v A high percentage of the students found the NeuroExam App useful and we should

consider creating or finding more apps to blend with the curriculum.

12    

v Overall, BMB online lectures were positively received and despite being the pilot year,

the format did not significantly impact the perceived quality of the course.

v Without explicit communication around the utility and the purpose of the new

educational technology, user adoption and engagement with the tool will be low.

v For increase adoption, providing ease of access and multiple platforms to access the tool

will be a key element.

v Instructors and developers will need to be explicit about the specific features that will

allow for interactivity and facilitate engagement with the learners.