The Leaders in Professional, Continuing, and Online Education1 Lessons Learned From An Adaptive...

The Leaders in Professional, Continuing, and Online Education 1www.upcea.edu

Lessons Learned From An Adaptive Learning Pilot In A Highly Attended General Education Course

UPCEA Mid-Atlantic ConferencePhiladelphia, PAOctober 8, 2015

DANIEL M INTZ, PROGRAM CHAIR

INFORMATION SYSTEMS MANAGEMENT

COLLEGIATE ASSOCIATE PROFESSOR

UNIVERS ITY OF MARYL AND UN IVERS ITY COLLEGE


Agenda• Introductory Remarks

• What We Were Trying to Solve

• What We Learned - Overview

• Background Information

• What Happened

• What We Learned - Details

• Next Step


Introduction - Me•IT Subject Matter Expert (SME)• Program Manager• Former US Department of Transportation CIO• Former CXO, various X’s

•Taught graduate classes for seven years (on-line)• Syracuse University• UMUC

• Graduated in Information Systems Management at the University of Maryland College Park

• Masters in International Management at University of Maryland University College, 2005

• Started as the Program Chair in Information Systems Management at UMUC nine months ago


Introduction – Me My career arc has been a single point

Career Arc

Graduated 40 years ago from the

University of Maryland College Park in

Information Systems Management

Became the Program Chair 9 months ago at the

University of Maryland University College for

Information Systems Management

14 miles


IFSM 201 - Concepts and Applications of Information Technology• Partially based on State of Maryland requirement to teach about technology and to learn about databases

• General Education requirement

• Two area of focus:• Introduction to computer technology (hardware, software, communications, security, ethics)• Introduction to ‘office’ software using Microsoft Office (Word, Excel, Powerpoint, Access) or equivalent (such as OpenOffice)


IFSM 201 - Concepts and Applications of Information Technology• Anecdotal feedback•Course is a bit ‘stuffed’ – covers a great deal of material•Generally positive, though not universal, feedback for the Open Learning Initiative (OLI) modules•The combined ‘standard’ UMUC Learning Management System (LMS) and OLI components takes some getting used to by new faculty and all students


What We Were Trying to Solve• Concerned about performance and retention rates in Undergraduate Gateway courses, especially STEM (science, technology, engineering and mathematics) courses

• Enrollments in these courses represented approximately 21% of all courses

• Accounted for 6-7% more withdrawals and failures than average

• Learning outcomes assessments indicated proficiency levels were often lower than other courses


What We Learned - Overview• Will focus on two areas of interest:•Statistical results which are providing direction for future research•Organizational and process lessons which represent necessary, often overlooked, steps supporting our path forward


What We Learned - Overview• Analysis• With the Open Learning Initiative (OLI) from Carnegie Mellon we have both

low-stakes activities (ungraded) and high-stakes activities (graded)• We found a strong correlation between the two• Working on next areas for research focus

• Process• Integration of Learning Management System (LMS) external pilot integration• Nature of grant process on planning• Running multiple program pilots at the same time• Course refresh and pilot management• Organizing and faculty management issues to optimize statistics collection


Background Information• UMUC

• Information Systems Management at UMUC

• Adaptive Learning

• OLI


UMUC• One of the largest public universities in the US

• Serves over 84,000 mostly non-traditional students• Over 8600 class sessions in AY 2014, more than 243K online course

enrollments• Over 81% are employed, full- or part-time• More than half are working parents• Over 35% are the 1st in their family to attend college• 93 degrees, specializations & certificates; more than 1000 courses online

• Highly diverse student body• 46% minority enrollment

• Open-enrollment


Information Systems Management• 13 Courses•One General Education• Two required for all business majors

• Total classes (sections) in Academic Year 14-15: 560

• Over 85% on-line, remaining are hybrid

• Total enrollments in AY 14-15: 15,253

• Faculty (all but one adjunct): over 200

• About 800 graduates/year


Adaptive Learning• Differentiated: there are different

pathways that students can take within a learning environment, often pre-set categories

• Personalized: a different pathway for each individual student, often rules based

• Adaptive: data driven, dynamically takes data from students and modifies learning pathway


Open Learning Initiative (OLI)• OLI is an online learning and assessment tool featuring educational activities and quizzes

• Courses were developed by Carnegie Mellon, in some cases with input from UMUC and partner institutions; adapted for UMUC students•OLI is administered by Carnegie Mellon and is run on their servers


Open Learning Initiative (OLI)• Courses using OLI at UMUC:• IFSM – 1 course (IFSM 201) (part of original grant)•Biology – 2 courses (part of original grant)•Statistics – 1 course (part of original grant)•French – 2 courses•Psychology – 1 course•Spanish – 4 courses


Open Learning Initiative (OLI)• OLI features two different types of learning activities:•Low-stakes activities• Learn by doing (LBD)• Did I get this (DIGT)• Students were allowed multiple attempts at these activities

•High-stakes quizzes (unit & module)• Measured specific learning objectives• Factored into class grade• Only one attempt was permitted on Quizzes


Learn by Doing


Did I Get This


Learning Dashboard


UMUC <-> CMU UMUC has years of experience and expertise in developing objects that are

• content rich,

• interactive,

• designed by experts in online pedagogy…but also

• one-way: they convey information but cannot collect information


UMUC <-> CMU Combined UMUC’s rich, interactive learning objects with OLI’s learning dashboard

• increased student engagement in the material

• highlighted “speed bumps” in student’s grasp of the content

• created a product that is greater than the sum of its parts


How It Came About• Gateways to Success project•Grant funded by Carnegie Corporation -> Dr. Marie Cini, then Vice President & Dean of the Undergraduate school at UMUC• Partnership with Carnegie Mellon University re OLI• Initial partnership w/Prince George Community College to develop or transition courses• Computer (IFSM) developed by UMUC staff under this grant• Statistics was transitioned using OLI developed materials• Biology material was developed as part of this grant plus a different grant

•Data has been accumulated since 2012


Learned Thus Far - Analysis


Analysis - Dataset•Total initial dataset for IFSM 201:•10,015 students•368 sections•Fall, 2012 – Spring, 2014•We performed analysis for 2,693 students, selected semesters with high quality data


Analysis - Variables•Variables included the number of:•Students who got the activity correct on the first try•Hints students received•Errors students made•Attempts they needed to get an activity correct

•Initial analysis focused on the first variable, “If they got an activity correct on the first try”


Analysis – Research Question

How does student performance on low-stakes activities relate to performance on high-stakes

activities (quiz scores) in IFSM 201?


Analysis – Methods• Three main outcomes were analyzed:•Correlations among Learn by Doing (LBD), Did I Get This (DIGT), and Quizzes•The predictive relation from LBD and DIGT to Quiz scores using multiple regression•The total amount of variance in Quiz scores explained by the LBD and DIGT in the multiple regression model


Analysis – Correlations• LBD and DIGT were strongly correlated at r=.97

• LBD and Quiz were moderately correlated at r=.60

• DIGT and Quiz were moderately correlated at r=.61

• Each of these correlations was statistically significant


Analysis – Multiple Regression• Both LBD and DIGT significantly predicted Quiz scores

• DIGT predicted quiz scores slightly better than LBD

• Student performance on the low-stakes activities significantly predicted Quiz scores by explaining 37% of the variance in that outcome


Analysis - Multiple Regression Model

Learn by Doing (LBD)

Did I get this? (DIGT)

Quiz score (Quiz)

𝑏=.37

𝑏=.24


Analysis – Summary• The two low stakes activities were very highly correlated with one another

• Each of the low stakes activities was significantly correlated with Quiz scores

• For every one unit increase in LBD, the multiple regression model predicts .24 units increase in Quiz scores

• For every one unit increase in DIGT, the multiple regression model predicts .37 units increase in Quiz scores


Learned Thus Far - Process


LEODesire2Learn

(LEO)

WebTycho(Proprietary,

in-house)LMS

OLI

OLI

OLI

Process - Integrating the UMUC LMS & OLI

2012

20152013


Process – Staffing Challenges• Coordinating course design staff with CMU OLI staff

• Ongoing course refresh

• Faculty training•Adding OLI• Integrating OLI•Analytics control/treatment group segregation

• Robust interaction with analytics team


Next Steps• Analyzing current data – low stakes activities• Look at students who did not utilize low-stakes activities• Expand analysis to other variables• Versions of the classroom• Attempts• Hints accessed• Errors• Faculty interaction (see also generating new data)• Learning outcomes


Next Steps• Further investigations into faculty impact• Initial A/B test will start later this month• We will provide ‘extra credit’ for completing low-stakes activities (up to

4% added to score, based on % of low-stakes activities completed)• In 2016, we will also investigate impact of additional faculty training and

use of the Learning Dashboard


Questions?


Daniel [email protected]

mailto:[email protected]

Date post:	05-Jan-2016
Category:	Documents
Upload:	duane-cunningham
View:	212 times
Download:	0 times

The Leaders in Professional, Continuing, and Online Education1 Lessons Learned From An Adaptive...

Documents