An Interdisciplinary Approach to Institutional Level Evaluation of An Interdisciplinary Approach to Institutional Level Evaluation of Online Learning Environments
15th Annual Sloan-C International Conference on Online Learning | Orlando, FL | October 2009
• Introductions
• APUS Mission and Demographicsg p
• Assessment Website
• Goals of the Program Review Process
• Program Review Process, Tools, and Resources
• Use of Continuous Improvement in Business Programin Business Program
• Use of Data to Drive Decisions
• Conclusion and Dialogue
Frank McCluskey, Ph.D.Executive Vice President and Provost
Phil Ice, Ph.D.Director, Course Design, Research &
DevelopmentDevelopment
Chad Patrizi, ABDDean of Business Administration
Jennifer Stephens, Ph.D.Associate Vice-President, Dean of Assessment
Dave BecherDirector, Academic Information Analysis
To educate the nation's military and public service communities and public service communities by providing respected, relevant,affordable, and student-focusedonline programs which prepare them for service and leadership them for service and leadership in a diverse, global society.
• Founded in 1991 as American MilitaryUniversity to provide affordable and convenient distance learning to service gmembers
• In 2002, expanded reach to national security,public safety and other professionals seeking affordable education and online convenience affordable education and online convenience, establishing American Public University System with member institutions: American Military University and American Public University
• We are a 100% online for-profit school that serves 53,000+ students who study online from 100 countries.
External ReviewExternal Review
Data Collection
Data Collection
AnalysisAnalysisFollow‐UpFollow‐Up
Program Review Meeting
Program Review Meeting
Three Year Plan
Three Year Plan
Electronic Repository of Information
Library and Learning Resources
Curriculum Assessment
External Reviewer Feedback
• Course books
• Electronic resources
• Learning strategies
• Student learning outcomes
• Instructional strategies
• Evaluation procedures
• Expert reviewer report
• Industry Advisory Council report
• Evaluation procedures
• Academic rigor
Electronic Repository of Information
A l i f f lt
Faculty
• Student demographic
Students
C i l M i
Learning Outcomes Assessment
• Analysis of facultycredentials andexpertise to ensurebreadth and diversity
• Student demographicinformation
• Enrollment History
• Growth trends
• Curricular Mapping
• Assessment measures
• Fact book
Electronic Repository of Information
Program Benchmarking
Program Directory Summary
Program Review Findings
• Program benchmarkingwith similar programsand institutions
• Evaluation of findings
• Programrecommendations
• Three year proposed
• Dean’s observations
• Meeting minutes
• Three year proposedstrategic plan
D a v e – insert slidesDave insert slides
• Introduction
• Bachelor of Business AdministrationProgram (BBA) vs. Institutional EOC (end of course survey)
• Without Micro and Macro Economics• Without Micro and Macro Economics
• Fact Book
4.60
BBA Program vs Institutional EOC Data 2006
4.40
4.00
4.20
3 60
3.80
3.40
3.60
Clear course objectives Course assignments supported course objectives
Course materials relevant to objectives
Course assignments fairly evaluated work
Instructor's teaching style appropriate Knowledgeable instructorInstructor provided feedback to inquiries within 72 hours
Instructor provided grades and appropriate feedback within one week
Interactions between the instructor and the students were positive
Interaction among classmates created a sense of community
Electronic classroom easy to navigateAssignments easy to post Technical support questions answered within 2 days
BBA Program Average (n=1139) 4.42 4.42 4.50 4.31 4.27 4.45 4.35 4.29 4.40 4.01 4.53 4.49 3.87
Institutional Average (n=28344) 4.44 4.45 4.46 4.37 4.35 4.44 4.38 4.27 4.44 4.02 4.39 4.48 3.83
4.50
BBA Program vs Institutional EOC Data 2006 w/o Micro and Macro Economics
3.50
4.00
2.00
2.50
3.00
0.50
1.00
1.50
0.00
0.50
BBA Pro g ram Averag e (n=113 9 ) 0 .0 0 0 .00 0 .0 0 4 .3 9 4 .3 8 0 .0 0 0 .0 0 0 .0 0 0 .0 0 0 .0 0 0 .0 0 0 .0 0 0 .0 0
Ins t itut io nal Averag e (n=28 3 4 4 ) 4 .4 4 4 .4 5 4 .4 6 4 .3 7 4 .3 5 4 .4 4 4 .3 8 4 .2 7 4 .4 4 4 .0 2 4 .3 9 4 .4 8 3 .8 3
Clear co urse o b ject ives
Co urse ass ig nments sup p orted
Course materials relevant to o b ject ives
Co urse ass ig nments
fairly evaluated
Ins tructo r's teaching s tyle ap p ro p riate
Kno wled g eab le ins tructo r
Ins tructo r p ro vid ed
feed b ack to
Instructo r p ro vided g rad es and app ro p riate
Interact ions b etween the
ins tructo r and
Interact io n amo ng
classmates
Electro nic class ro o m easy
to navigate
Ass ig nments easy to p o s t
Technical sup p ort
q ues t ions
C i f Community of Inquiry Framework
a process model of learning in online and blended educational environments
grounded in a collaborative constructivist view of higher educationview of higher education
assumes effective online learning requires the development of a community of the development of a community of learners that supports meaningful inquiry and deep learningand deep learning
social presence cognitive presence
LEARNINGLEARNINGLEARNINGLEARNING
teaching presence
Social Presence
the ability of participants in a community of inquiry to project themselves socially and emotionally -- as ‘real’ people
the degree to which participants in g p pcomputer mediated communication feel socially and emotionally connected
Cognitive Presence
the extent to which learners are able to construct and confirm meaning through sustained reflection and discourse in a critical community of i i inquiry
Teaching Presence
the design, facilitation and direction of cognitive and social processes for the purpose of realizing personally meaningful and d ti ll th hil l i educationally worthwhile learning
outcomes
Community of Inquiry Surveyy
9 social presence items (3 affective expression, 3 open communication, 3 group cohesion) p , g p )
12 cognitive presence items (3 triggering, 3 exploration 3 integration 3 resolution)exploration, 3 integration, 3 resolution)
13 teaching presence items (4 design & facilitation 6 facilitation of discourse 3 direct facilitation, 6 facilitation of discourse, 3 direct instruction)
CoI Survey Validation
tested in graduate courses at four tested in graduate courses at four institutions in the US and Canada
principal component factor analysis
three factor model predicted by CoIp yframework confirmed
Arbaugh, Cleveland-Innes, Diaz, Garrison, Arbaugh, Cleveland Innes, Diaz, Garrison, Ice, Richardson, Shea & Swan - 2008
APUS Utilization
American Public University System
100% online100% online
monthly course starts
CoI is the end of course survey
eight semesters of data collectioneight semesters of data collection
Application
comparison of at the University, School, Program, Course and Instructor levels
assessment of instructional design efficacy
t f l diff assessment of learner differences –demographic factors
assessment of integration of new technologies
Frank McCluskey, [email protected]
Executive Vice President and Provost
Phil Ice, [email protected]
Director, Course Design, Research & Development
Chad Patrizi, [email protected]
Dean of Business Administration
Jennifer Stephens, [email protected]
Associate Vice-President, Dean of Assessment
Dave [email protected]
Director, Academic Information Analysis