8/14/2019 Evaluating Online Learning
1/80
8/14/2019 Evaluating Online Learning
2/80
8/14/2019 Evaluating Online Learning
3/80
Prepared by WestEdWith Edvance Research, Inc.
For
U.S. Department of Edu cation
Office of Innovation and Improvement
2008
Evaluating Online LearningChallenges and Strategies for SuccessI n n O v a t I O n S I n E d u C a t I O n
8/14/2019 Evaluating Online Learning
4/80
this repor ws proce er u.S. deprme o Ecio Corc no. Ed-01-CO-0012, tsk Orer
d010, wih WesE. Shro Kiey Hor sere s he corcig ocers represeie. the coe
o his repor oes o ecessrily refec he iews or policies o he u.S. deprme o Ecio, or
oes he meio o re mes, commercil procs or orgizios imply eorsemes by he u.S.
goerme. this pblicio lso cois uRLs or iormio cree miie by prie org-
izios. this iormio is proie or he reers coeiece. the u.S. deprme o Ecio
is o resposible or corollig or greeig he ccrcy, relece, imeliess, or compleeess o
his osie iormio. Frher, he iclsio o iormio or uRL oes o refec he imporce o
he orgizio, or is i iee o eorse y iews expresse, or procs or serices oere.
U.S. Department o Education
Mrgre Spelligs
Secretary
Ofce o Innovation and Improvement
dog Mesecr
Assistant Deputy Secretary
Technology in Education Programs
Cheryl Gree
Director
Jly 2008
this repor is i he pblic omi. ahorizio o reproce i i whole or i pr is gre. While
permissio o repri his pblicio is o ecessry, he ciio shol be: u.S. deprme o
Ecio, Oce o Ioio Improeme, Evaluating Online Learning: Challenges andStrategies or Success, Wshigo, d.C., 2008.
to orer copies o his repor (orer mber Ed004344P),
write o: Ed Pbs, Ecio Pblicios Ceer, u.S. deprme o Ecio, P.O. Box 1398,
Jessp, Md 20794-1398;
or ax yor reqes o: 301-470-1244;
or e-mailyor reqes o: [email protected];
or calli yor reqes oll-ree: 1-877-433-7827 (1-877-4-Ed-PuBS). those who se elecommi-
cios eice or he e (tdd) or eleypewrier (ttY), shol cll 1-877-576-7734. I 877 ser-
ice is o ye ilble i yor re, cll 1-800-872-5327 (1-800-uSa-LEaRn; ttY: 1-800-437-0833);or order online : hp://www.epbs.e.go.
this repor is lso ilble o he deprmes Web sie :
hp://www.e.go/mis/le/cemic/elolie/
O reqes, his pblicio is ilble i lere orms, sch s Brille, lrge pri, or comper
iskee. For more iormio, plese coc he deprmes alere Form Ceer 202-260-0852
or 202-260-0818.
8/14/2019 Evaluating Online Learning
5/80
iii
ContentsList o Illustrations and Tables iv
Foreword v
Acknowledgments vii
Introduction 1
Commo Chlleges o Elig Olie Lerig (1)
the Fere Elios (2)
Whs i this Gie (3)
Part I: Challenges o Evaluating Online Learning 7
Meeig he nees o Mliple Skeholers (7)
Bilig o he Exisig Bse o Kowlege (12)
Elig Mlicee Olie Resorces (20)
Fiig approprie Compriso Grops (26)
Solig d Collecio Problems (33)
Ierpreig he Impc o Progrm Mriy (40)
trslig Elio Fiigs Io acio (43)
Part II: Recommendations or Evaluating Online Learning 49
Geerl Recommeios (49)
Recommeios or Meeig he nees o Mliple Skeholers (50)Recommeios or uilizig Bilig o he Exisig Bse o
Kowlege (50)
Recommeios or Elig Mlicee Resorces (52)
Recommeios or Progrms Cosierig Compriso Sy (52)
Recommeios or Gherig vli Elio d (53)
Recommeios or tkig Progrm Mriy Io acco (54)
Recommeios or trslig Elio Fiigs Io acio (54)
Featured Online Learning Evaluations 55
Appendix A: Resources 59
disce Lerig (59)Elio Mehos tools (60)
Resorces From Higher Ecio (61)
Appendix B: Research Methodology 63
Glossary o Common Evaluation Terms 65
Notes 67
8/14/2019 Evaluating Online Learning
6/80
iv
Illustrations and TablesFIGURES
1. Excerpt From Appleton eSchools Online Program Perceiver Instrument 17
2. Excerpt From Digital Learning Commons Meeting 21st Century Learning Challenges
in Washington State 24
3. Example o Tabulated Results From an Arizona Virtual Academy Online Parent SurveyAbout the Quality o a Recent Student Workshop 38
4. Excerpts From AZVAs Next Steps Plan, In Response to Recommendations
From the K12 Quality Assurance Audit 47
TABLES
1. Selected Variables o Profled Online Learning Evaluations 4
2. Evaluation Design Characteristics 28
8/14/2019 Evaluating Online Learning
7/80
v
Foreword
Ecio i his cory hs eole rmiclly rom he ys o oe echer i oe-room
schoolhose. toy, se lerig is o loger coe o physicl spce. Compers he
Iere he broke hrogh school wlls, giig ses greer opporiies o persolize heir
ecio, ccess is resorces, receie exr help or more-chllegig ssigmes, egge
i lerig i ew iqe wys.
alhogh olie lerig is reliely ew eerprise i he K12 re, i is expig rpily,
wih icresig mbers o proiers oerig serices more ses choosig o pricipe. as
wih y ecio progrm, olie lerig iiiies ms be hel ccoble or resls. ths,
i is criicl or ses heir press well s miisrors, policymkers, erso
he iormig hem bo progrm se ocomes , i rele, bo how well
priclr progrm compres o riiol ecio moels. to his e, rigoros elios re
esseil. they c ieiy wheher progrms olie resorces re perormig s promise, ,
eqlly impor, hey c poi o res or improeme.
the elios highlighe i his gie represe bro specrm o olie opios, rom pro-
grms h proie olie corses o Web sies h ere ecio resorces. the elios
hemseles rge rom ierl ssessmes o exerl, scieic reserch sies. all emosre
how progrm leers elors he bee ble o impleme srog elio prcices espie
some chlleges ihere o exmiig lerig i olie eirome.
this gie complemes oher pblicio, Connecting Students to Advanced Courses Online, pb-
lishe ls yer by he u.S. deprme o Ecio. Boh re pr o he Ioios i Ecio
series, which ieies exmples o ioie prcices rom cross he cory h re helpig
ses chiee.
My hope is h his gie will ssis elors progrm leers who seek o se o gie
progrm improeme ime chieig posiie ocomes or or ios ses.
Mrgre Spelligs, Secrery
u.S. deprme o Ecio
8/14/2019 Evaluating Online Learning
8/80
8/14/2019 Evaluating Online Learning
9/80
Acknowledgments
this gie ws eelope er he spiceso he u.S. deprme o Ecios Oce oIoio Improeme. Shro Hor wsprojec irecor.
a exerl isory grop proie eebcko ree he sy scope, ee he seleciocrieri, clriy he ex. Members icleRober Blomeyer, presie, Blomeyer & Clem-ee Coslig Serices; tom Clrk, presie,ta Coslig; tri dis, presie, Ier-iol Sociey or techology i Ecio, ssis proessor irecor o eEcio,texs a&M uiersiy; Helee Jeigs, icepresie, Mcro Ieriol, Ic.; Liz Ppe,presie chie execie ocer o virlHigh School Globl Cosorim; Lis Es,irecor, Ceer or Elio Cpciy Bil-ig, Wexor, Ic.
S i he deprme o Ecio who proi-e ip reiewe rs icle Se Bek,
Cyhi Cbell, tom Corwi, Ke deie,di d, Lorezo Esers, Mereih Frce,See Frie, Cheryl Gree, virgii Geles,Robi Gilchris, dog Herber, neie Lser,Bri Leker, Megh Lerch, tim Mger, Li Wilso. this gie wol o be possi-ble wiho he sppor o tim Mger, irecor,Oce o Eciol techology.
the see olie progrms h pricipe ihe eelopme o his gie he cse s-ies o which i is bse were geeros wih boh
heir ime eio.
Alabama Connecting Classrooms, Educators, &
Students Statewide Distance Learning
albm deprme o Ecio50 n. Ripley S.P.O. Box 302101Mogomery, aL 36104hp://ccessl.se.l.s
vii
Algebra I OnlineLoisi virl SchoolLoisi deprme o EcioP.O. Box 94064Bo Roge, La 70804-9064hp://www.loisiirlschool.e/?lgebr
Appleton eSchool2121 E. Emmers drie
appleo, WI 54915hp://www.s.k12.wi.s/Eschool
Arizona Virtual Academy4595 S. Plo vere RoSie 517/519tcso, aZ 85714hp://www.z.org
Chicago Public Schools Virtual High School125 S. Clrk S.Chicgo, IL 60603hp://cler.cps.k12.il.s/ohsp/isce_
lerig.hml
Digital Learning CommonsCorbe Bilig4507 uiersiy Wy nESie 204Sele, Wa 98105hp://www.lerigcommos.org
ThinkportMryl Pblic teleisio11767 Owigs Mills Bl.
Owigs Mills, Md 21117hp://www.hikpor.org
i prership wih:
Johs Hopkis Ceer or techology iEcio6740 alexer Bell drieSie 302Colmbi, Md 21046hp://ce.jh.e/iex.hm
8/14/2019 Evaluating Online Learning
10/80
8/14/2019 Evaluating Online Learning
11/80
Introduction
This guide is designed as a resource or leaders and evaluators o K12 online learning programs. In
this guide, the term online learning is used to reer to a range o education programs and resources
in the K12 arena, including distance learning courses oered by universities, private providers, or
teachers at other schools; stand-alone virtual schools that provide students with a ull array o online
courses and services; and educational Web sites that oer teachers, parents, and students a range o
resources.1 The guide eatures seven evaluations that represent variety in both the type o program or
resource being evaluated, and in the type o evaluation. These evaluations were selected because they
oer useul lessons to others who are planning to evaluate an online learning program or resource.
O corse, elig olie lerig is o lo-
geher iere rom ssessig y oher ype o
ecio progrm, , o some egree, el-
ors my ce he sme ki o esig
lysis isses i boh isces. Sill, olie
progrm elors my ecoer some -
icipe chlleges i he irl re owig,
or exmple, o he isce bewee progrm
sies ses, pricips miliriywih he echology beig se, lck o
rele elio ools. this gie exmies
rge o chlleges h olie progrm el-
ors re likely o mee, some h re iqe
o olie seigs ohers h re more ge-
erl. I lso escribes how olie eiromes
c someimes oer ges o elors
by preseig opporiies or sremlie
collecio lysis, or exmple.
the gie speciclly ocses o chlleges
respose sregies. all o he elios
escribe here illsre srog ssessme prc-
ices robs igs, hey re moels
or emosrig how progrm leers
elors c hle he chlleges o el-
ig olie lerig.
Common Challenges o Evaluating
Online Learning
Olie lerig is reliely ew eelopme
i K12 ecio b is rpily expig
i boh mber o progrms pricips.
accorig o repor by he norh americ
Cocil or Olie Lerig (naCOL), as o
Sepember 2007, 42 ses he [h] sigi-c spplemel olie lerig progrms (i
which ses erolle i physicl schools ke
oe or wo corses olie), or sigic ll-
ime progrms (i which ses ke mos or
ll o heir corses olie), or boh.2 I iio,
he Iere hoses eer-expig mber
o Web sies wih bro rge o ecio
resorces or ses, pres, echers.
Gie his expsio erh o exisig
reserch o he opic, i is criicl o cocrigoros elios o olie lerig i K12
seigs o esre h i oes wh people hope
i will o: help improe se lerig.
Howeer, hose erkig sch elios
my well ecoer mber o echicl
mehoologicl isses h c mke his ype
1
8/14/2019 Evaluating Online Learning
12/80
2
Evaluating Online Learning: Challenges and Strategies or SuccessInnOvatIOnS In EduCatIOn
o reserch icl o exece. For exmple, he
sc reserch lierre o K12 olie lerig
elio proies ew exisig rmeworks o
help elors escribe lyze progrms,
or ools, sch s sreys or rbrics, hey c
se o collec or ssess progrm qliy.
aoher commo chllege whe ses re
syig olie is he icly o exmiig
wh is hppeig i mliple, geogrphiclly
is lerig sies. a mlicee ec-
io resorcessch s s Web sies oerig
wie rge o eres or irl schools h
oer corses rom mliple eorsre lso
hr o ele, s re progrms h ilize
echologies isrciol moels h re
ew o sers.
Frhermore, elios o olie lerig o-
e occr i he coex o poliiclly lo-
e ebe bo wheher sch progrms re
worh he iesme how mch ig
is eee o r high-qliy progrm; bo
wheher olie lerig relly proies ses wih high-qliy lerig opporiies;
bo how o compre olie riiol
pproches. uersbly, ers pol-
icymkerso o meio ses heir
presw h show wheher olie
lerig c be s eecie s riiol ec-
iol pproches which olie moels re
he bes. these skeholers my or my o
hik bo elio i echicl erms, b ll
o hem re ierese i how ses perormi hese ew progrms. a he sme ime, my
olie progrm leers he mliple gols i
mi, sch s icrese se eggeme or
icrese se ccess o high-qliy corses
echers. they rge h es scores loe
re ieqe mesre or cprig impor-
iereces bewee riiol olie
lerig seigs. a, like ecors i y
seigriiol or oliehey my eel
rl repiio bo iiig elors o
ke criicl look heir progrm, erig h
i will hmper he progress o heir progrm,
rher h sreghe i.
this gie will iscss how elors he
rie o compre riiol olie lerig
pproches, wh chlleges hey he eco-
ere, wh lessos hey he lere.
The Featured Evaluations
this gie ieiolly eres riey o o-
lie progrms resorces, iclig irl
schools, progrms h proie corses olie,
Web sies wih bro eciol resorces.
Some sere eire se, while ohers sere
priclr isric. this gie lso icles is-
ic kis o elios, rom ierlly le
ormie elios (see Glossry o Commo
Elio terms, p. 65) o scieic reserch
sies by exerl expers. I some cses, pro-
grm isiers iiie he elios; i ohers,
here were exerl resos or he elio.
the ere elios icle wie rge
o collecio lysis ciiiesrom
ormie elios h rely primrily o sr-
ey, ieriew, obserio , o scieic
experimes h compre ocomes bewee
olie riiol seigs. I ech isce,
elors chose he mehos crelly, bse
o he prpose o he elio he specic
se o reserch qesios hey sogh o swer.
the gol i choosig rge o elios or
his gie ws o oer exmples h col be
isrcie o progrm leers elors i
ierse circmsces, iclig hose i ry-
8/14/2019 Evaluating Online Learning
13/80
3
grm or resorce, wheher he progrm seres
isric- or se-leel iece, sge o
mriy. I selecig he ere elios,
he reserchers rew rom s wie rge o
chrcerisics s possible while keepig he
qliy crieri high. a ll escripio o he
mehoology se o sy he elio(s) o
he selece sies c be o i ppeix B:
Reserch Mehoology.
the l selecio icle elios o he
ollowig olie progrms resorces: al-
bm Coecig Clssrooms, Ecors, & S-
es Sewie disce Lerig, opere by
he albm deprme o Ecio; algebr
I Olie, opere by he Loisi deprme
o Ecio; appleo eSchool, opere by
Wiscosis appleo are School disric; ari-
zo virl acemy, pblic chrer school;
Chicgo Pblic Schools virl High School;
digil Lerig Commos i Wshigo se;
thikpor, Web sie opere by Mryl
Pblic teleisio Johs Hopkis Ceer or
techology i Ecio. aiiol iorm-
io bo ech progrm is elio(s) is
icle i ble 1.
Whats in This Guide
this gie ws eelope s resorce or el-
ors, wheher exerl, hir-pry reserchers,
or progrm miisrors oher s whore cosierig cocig heir ow ierl
elio. Some o he elios highlighe
here were crrie o by exerl elors,
while ohers were coce by progrm s
or he s o pre orgizio. I ll cses,
he reserch ws erke by experiece
proessiols, his pblicio is ime
ig sges o mriy, wih ryig egrees o
ierl cpciy mos o ilble -
ig. the ere elios re o wiho
fws, b hey ll illsre resoble sregies
or cklig commo chlleges o elig
olie lerig.
to selec he ere elios, reserchers
or his gie compile iiil lis o c-
ies by serchig or K12 olie lerig
elios o he Web i pblishe oc-
mes, he expe he lis hrogh reer-
rls rom six-member isory grop (see lis
o members i he ackowlegmes secio,
p. ii) oher kowlegeble expers i he
el. Fory orgizios were o he l lis
or cosierio.
a mrix o selecio crieri ws re
reise bse o eebck rom he isory
grop. the hree qliy crieri were:
the elio cosiere mliple ocome
mesres, iclig se chieeme.
the elio igs were wiely comm-
ice o key skeholers o he progrm orresorce beig sie.
Progrm leers ce o elio resls.
Reserchers wre sies p o hree pois o
ech o hese hree crieri, sig pblicly il-
ble iormio, reiew o elio repors,
gp-llig ieriews wih progrm leers.
all he icle sies score les six o he
possible ie pois cross hese hree crieri.
Sice gol o his pblicio ws o showcse
riey o ypes o elios, he poeil
sies were coe s o sch iiol chrcer-
isics s: ierl s. exerl elor, ype o
elio esig, ype o olie lerig pro-
8/14/2019 Evaluating Online Learning
14/80
44
Evaluating Online Learning: Challenges and Strategies or SuccessInnOvatIOnS In EduCatIOn
Table 1. Selected Variables o Profled Online Learning Evaluations
Name of Program
or Resource
Type of Program or Resource/
Year Initiated
Type of Evaluation
Featured in GuideaYear
Evaluation
Started
Evaluation
Objective
Alabama Connect-ing Classrooms,Educators, &Students StatewideDistance Learning
Online courses and interactivevideo-conerence classes orstudents across state / Piloted inspring 2006; statewide imple-mentation in all 2006
External; ormative &summative; includes com-parisons with traditionalinstructional settings
2006 Monitoring programimplementation; programimprovement; sharing bestpractices
Algebra I OnlineLouisiana
Online algebra courses or stu-dents across the state / 2002
External and internal;ormative and summative;includes comparisons with
traditional instructionalsettings
2003; com-parative studyin 200405
Determine i program is e-ective way to provide stu-dents with certifed algebra
teachers and to support thein-class teachers certifca-tion eorts
Appleton eSchool Online courses or students en-rolled in districts high schools(some students take all coursesonline) / 2002
Internal; ormative; evalua-tion process based on inter-nally developed rubric
Rubric pilotedin 2006
Program improvement;sharing best practices
ArizonaVirtualAcademy
Virtual charter school or stu-dents enrolled in public schoolsand homeschool students (no
more than 20%) / 2003
Formative and summative;external and internal
2003 State monitoring; qualityassurance; program im-provement
Chicago PublicSchools VirtualHigh School
Online courses or students en-rolled in districts high schools /2002
External; ormative 2002 Assess need or mentortraining and other studentsupports; identiy ways toimprove completion andpass rates
Digital LearningCommons
Web site with online coursesand a wide array o resources orteachers and students / 2003
External and internal; or-mative
2003 Understand usage o site;assess impact on studentachievement and collegereadiness
Thinkport Maryland PublicTelevision withJohn Hopkins
Web site with a wide array oresources or teachers and stu-dents / 2000
External and internal;ormative and summative,including randomized con-trolled trial
2001; ran-domizedcontrolled trialin 2005
Understand usage o site;assess impact o elec-tronic feld trip on studentperormance
a See Glossary o Common Evaluation Terms on page 65.b Run by the nonprot College Board, the Advanced Placement (AP) program oers college-level course work to high school students. Many institutions o higher education
oer college credits to students who take AP courses.c North Central Regional Educational Laboratory.
8/14/2019 Evaluating Online Learning
15/80
55
Cost of Evaluation Funding Source for
Evaluation
Data
Collected
Data
Collection Tools
Improvements Resulting
From Evaluation
$60,000 in 2007;$600,000 in 2008
Specifc allocationin program budget(originating romstate o Alabama)
Student enrollment, comple-tion, grades; APb coursepass rates; student andteacher satisaction; de-scription o implementationand challenges
Surveys, interviews,observations
Teacher proessional devel-opment; improvements totechnology and administrativeoperations
$110,000 or themost labor-intensivephase, including the
comparative analysisduring 200405
General programunds, grants romNCREL,c
BellSouth Founda-tion, and U.S. De-partmento Education
Student grades and statetest scores; pre- and post-tests; student use and satis-
action data; ocus groups;teacher characteristicsand teachers certifcationoutcomes
Pre- and posttestsdeveloped by eval-uator, surveys
Teacher proessional develop-ment; increased role or in-classteachers, curriculum improve-
ments, new technologies used;expansion to middle schools
No specifc allocation;Approx. $15,000 tomake the rubric andevaluation processavailable in Webormat
General programunds (originatingrom charter grantrom state o Wis-consin)
Internal descriptions and as-sessments o key programcomponents (using rubric);mentor, student, and teachersatisaction data; coursecompletion and grades
Internally developedrubric and surveys
Mentor proessional develop-ment; course content improve-ments; expanded interactivity incourses; improvements to pro-gram Web site and printed mate-rials, sharing o best practices
No specifc allocation General programunds (originatingrom state o Ari-
zona)
Student enrollment, grades,and state test scores; par-ent, teacher, and student
satisaction data; internal& external assessments onkey program components
Electronic surveys;externally devel-oped rubric
Wide range o operational andinstructional improvements
Approx. $25,000 Dist ricts Ofce o Technology Services
Student enrollment, coursecompletion, grades, andtest scores; student use andsatisaction data; mentorassessments o needs
Surveys, interviews,ocus groups
Designated class periods oronline learning; more onsitementors; training or mentors
Approx. $80,000 orthe college-readinessstudy
Bill & Melinda GatesFoundation
Student transcripts; studentgrades and completionrates; use and satisactiondata
Surveys Improvements to student ori-entation; curriculum improve-ments; development o schooluse plans to encourage bestpractices
Estimated $40,000or the randomizedcontrolled trial (parto a comprehensiveevaluation)
Star Schools Grant Student test scores oncustom-developed contentassessment; inormationabout delivery o curriculum;use and satisaction data
Test o content knowl-edge developed byevaluator, surveys,teacher implementa-tion logs
Changes to teaching materials;changes to online content andormat
8/14/2019 Evaluating Online Learning
16/80
66
Evaluating Online Learning: Challenges and Strategies or SuccessInnOvatIOnS In EduCatIOn
primrily reers who re milir wih bsic
elio prcices. For reers who re less
milir wih elio, glossry o commo
erms is ilble o pge 65.
Pr I o his gie ocses o some o he likely
chlleges ce by olie progrm elors,
i is orgize io he ollowig secios:
Meeig he nees o Mliple Skeholers
Bilig o he Exisig Bse o Kowlege
Elig Mlicee Olie Resorces
Fiig approprie Compriso Grops
Solig d Collecio Problems
Ierpreig he Impc o Progrm Mriy
trslig Elio Fiigs Io acio
Ech secion o Pr I presens prcicl inor-
mion bou one o he chllenges o elu-
ing online lerning nd proides exmples o
how he eured eluions he ddressed
i. Pr II synhesizes he lessons lerned rom
meeing hose chllenges nd oers recom-
mendions bsed s well on reserch nd
conersions wih expers in eluing online
lerning. these re gered o progrm leders
who re considering n eluion nd o ssis
hem nd heir eluors s hey work ogeh-
er o design nd complee he process. Brie
profles o ech o he seen online progrms
cn be ound he end o he guide, nd de-
ils bou ech eluion re summrized in
ble 1.
8/14/2019 Evaluating Online Learning
17/80
77
P A R T I
Challenges ofEvaluating OnlineLearning
Meeting the Needs o MultipleStakeholders
Eery goo elio begis wih clerly s-
e prpose specic se o qesios o be
swere. these qesios rie he elio
pproch help eermie he specic
collecio echiqes h elors will se.
Someimes, howeer, progrm elors
hemseles i he icl posiio o eeig o
lll seerl prposes oce, or eeig o -
swer wie riey o reserch qesios. Mos
skeholers he he sme bsic qesioIs
i workig? B o eeryoe ees working
i he sme wy. While policymkers my be
ierese i gis i srize es scores,
progrm leers my be eqlly ierese i
oher iicors o sccess, sch s wheher he
progrm is ressig he ees o riio-
lly errepresee sbgrops, or procig
ocomes h re oly iirecly rele o es
scores, like se eggeme.
nrlly, hese qesios will o be swerei he sme wy. For exmple, i skeholers
w cocree eiece bo he impc o
se chieeme, elors migh coc
romize corolle rilhe gol s-
r or ssessig progrm eecsor qsi-
experimel esig h compres es scores o
progrm pricips wih ses i mche
compriso grops. B i skeholers w o
kow, or exmple, how progrm hs bee
implemee cross my sies, or why i is
leig o priclr ocomes, he hey migh
op or escripie sy, icorporig sch
echiqes s sreys, ocs grops, or obser-
ios o progrm pricips o gher qli-
ie process .
Whe mliple skeholers he ierig i-
eress qesios, how c elors mee
hese rios expecios?
to sisy he ems o mliple skehol-ers, elors oe combie ormie
smmie compoes (see Glossry o Com-
mo Elio terms, p. 65). I he cse o
albm Coecig Clssrooms, Ecors, &
Ses Sewie disce Lerig (aCCESS),
escribe below, he elors he bee ery
procie i esigig series o elios
h, colleciely, yiel iormio h hs il-
iy boh or progrm improeme or -
ersig progrm perormce. I he cseo arizo virl acemy (aZva), he schools
leership em hs me he mos o he my
elio ciiies hey re reqire o com-
plee by sig igs rom hose ciiies
or heir ow improeme prposes pig-
gybckig o hem wih collecio eors
o heir ow. I ech isce, progrm leers
8/14/2019 Evaluating Online Learning
18/80
8
Evaluating Online Learning: Challenges and Strategies or SuccessInnOvatIOnS In EduCatIOn
wol likely sy h heir elios re rs
oremos iee o improe heir pro-
grms. Ye, whe clle po o show chiee-
me resls, hey c o h s well.
Combine Formative and Summative Evaluation
Approaches to Meet Multiple Demands
From he begiig o heir ioleme wih
aCCESS, elors rom he Ieriol Sociey
or techology i Ecio (IStE) he ke
combie smmie ormie pproch
o syig his se-r progrm h oersboh Web-bse iercie ieocoerec-
ig corses. the origil eelopme proposl
or aCCESS icle ccobiliy pl h
clle or ogoig moiorig o he progrm
o ieiy res or improeme o geer-
e sel iormio h col be shre wih
oher schools hrogho he se. I iio,
rom he begiig, progrm leers se
policymkers expresse ieres i gherig
bo he progrms impc o se lerig.to ccomplish hese mliple gols, IStE com-
plee wo sccessie elios or albm,
ech o which h boh ormie smmie
compoes. a hir elio is er wy.
the rs elio, rig he progrms pilo
implemeio, ocse o proiig eebck
h col be se o moiy he progrm, i
ee be, o geerig iormio o shre
wih albm schools. Elio ciiies his sge icle lierre reiew ob-
serio isis o six romly selece pilo
sies, where elors coce ieriews
sreys. they lso r ocs grops, or
which reserchers ieriewe respoes i
grop seig. Elors chose hese mehos
o geere qliie iormio bo how
he pilo progrm ws beig implemee
wh chges migh be eee o sreghe i.
the seco elio ook more o smm-
ie pproch, lookig o see wheher or o
aCCESS ws meeig is oerll objecies. Firs,
elors coce sreys ieriews
o ses echers, s well s ieriews
wih school miisrors persoel he
progrms hree Regiol Sppor Ceers. I
iio, hey ghere se erollme
chieeme , sewie corse erollme
compleio res, oher progrm o-come , sch s he mber o ew isce
corses eelope he mber o prici-
pig schools.
this seco elio lso se qsi-exper-
imel esig (see Glossry o Commo El-
io terms, p. 65) o proie iormio o
progrm eecs. Elors compre chiee-
me ocomes bewee aCCESS pricips
ses sewie, bewee ses iiercie ieocoerecig corses s-
es i riiol seigs, bewee s-
es who pricipe i olie corses
hose who ook corses oere i he ierc-
ie ieocoerecig orm.
as o erly 2008, he elors were coc-
ig hir elio, iegrig he rom
he rs wo sies ocsig o se
chieeme. Lookig he, aCCESS leers
pl o coie gherig lly i -
icipio o cocig logiil sies h
will ieiy aCCESSs ll impc o se
progress chieeme.
togeher, he crelly ple elio c-
iiies coce by aCCESSs elors he
geere seerl igs recommeios
8/14/2019 Evaluating Online Learning
19/80
99
Reasons and Contexts for Formative Versus Summative Evaluations
Formative and summative evaluations can each serve important unctions or programs. Formative evaluations,
sometimes called process evaluations, are conducted primarily to fnd out how a program is being implemented
and how it might be strengthened. Summative evaluations, also called outcome evaluations, are appropriate or
better-established programs, when program leaders have settled on their best policies and practices and want to
know, or example, what results the program is yielding.
Ideally, ormative evaluations are developed as partnerships that give all stakeholders a hand in planning and help-
ing conduct the evaluation. Explicitly raming a ormative evaluation as a collaboration among stakeholders can
help in more ways than one. Practitioners are more likely to cooperate with and welcome evaluators rather than eel
wary or threateneda common reaction. In addition, practitioners who are invited to be partners in an evaluation
are more likely to eel invested in its results and to implement the fndings and recommendations.
Even more than ormative evaluations, summative evaluations can be perceived by practitioners as threatening
and, in many cases, program sta are not eager to welcome evaluators into their midst. Even in these situations,
however, their reaction can be mitigated i evaluators work diligently to communicate the evaluations goals. Evalu-
ators should make clear their intention to provide the program with inormation that can be used to strengthen it, or
to give the program credible data to show unders or other stakeholders. In many cases, summative evaluations do
not uncover fndings that are unexpected; they merely provide hard data to back up the anecdotes and hunches o
program leaders and sta.
Program leaders who are contemplating an evaluation also will want to consider the costs o whatever type o study
they choose. Some ormative evaluations are relatively inormal. For example, a ormative evaluation might consist
primarily o short-term activities conducted by internal sta, like brie surveys o participants, to gather eedback about
dierent aspects o the program. This type o evaluation is inexpensive and can be ideal or leaders seeking ongoinginormation to strengthen their program. In other instances, ormative evaluation is more structured and ormal. For
instance, an external evaluator may be hired to observe or interview program participants, or to conduct feld surveys
and analyze the data. Having an external evaluator can bring increased objectivity, but it also adds cost.
In many cases, summative evaluations are more ormal and expensive operations, particularly i they are using
experimental or quasi-experimental designs that require increased coordination and management and sophisticated
data analysis techniques. Typically, external evaluators conduct summative evaluations, which generally extends
the timeline and ups the costs. Still, experimental and quasi-experimental designs may provide the most reliable
inormation about program eects.
Finally, program leaders should consider that an evaluation need not be exclusively ormative or summative. As
the ACCESS case illustrates (see pp. 710), sometimes it is best or programs to combine elements o both, either
concurrently or in dierent years.
h lrey he bee se o sreghe he
progrm. For exmple, heir igs sgges
h ses pricipig i he isce ler-
ig corses re compleig corses high res
, i he cse o he College Bor ace
Plceme (aP) corses,* re chieig scores
comprble o ses gh i riiol
* Run by the nonprot College Board, the Advanced Placement program oers col-
lege-level course work to high school students. Many institutions o higher education
oer college credits to students who take AP courses.
8/14/2019 Evaluating Online Learning
20/80
10
Evaluating Online Learning: Challenges and Strategies or SuccessInnOvatIOnS In EduCatIOn
10
seigs. these kis o col be criicl o
miiig ig poliicl sppor or he
progrm i he re. aCCESS lso is bilig
logiil bse o proie core o
or se i re elios , progrm le-
ers hope, o help eermie aCCESSs log-erm
impc o se progress chieeme.
Wih hese collecio ools processes
i plce, aCCESS hs rme isel o ress he
ees expecios o rios skeholers
isie osie he progrm.
Make the Most o Mandatory Program
Evaluations
While ll leers s o olie ecio
progrms re likely o w o ers heir
ifece o se lerig, some he o
choice i he mer. My olie progrms ms
elier smmie se ocome becse
er or reglory boy ems i. I he
cse o arizo virl acemy (aZva), K12
sewie pblic chrer school, he progrmms comply wih seerl mory elio
reqiremes: Firs, school leers re reqire
by he se o arizo o sbmi l e-
ecieess reiew, which is se o eermie
wheher or o he schools chrer will be re-
ewe. For his yerly repor, aZva s ms
proie o se erollme, reeio,
mobiliy, se es perormce. the repor
lso ms icle ppil pre siscio
, which aZva collecs olie he e oech corse, eile sel-elio o
operiol miisrie eciecy.
aZva lso ms swer o K12 Ic., he ec-
io compy h spplies he progrms cr-
riclm or ll gre leels. K12 Ic. hs is ow
ieres i elig how well is crriclm
procs re workig i esrig h i is
prere wih high-qliy school. aZvas i-
recor, Mry Gior, sys h rom he seco
yo ope yor school, here is expecio
[o he pr o K12 Ic.] h yo will collec
, lyze hem, se hem o mke eci-
sios. K12 Ic. hs esblishe bes prcices or
cemic chieeme. they ke gre prie i
beig -rie compy, she s. I co-
cs qliy ssrce is aZva pproxi-
mely eery wo yers, which cosis o sie
isi coce by K12 Ic. persoel ex-
esie qesioire, complee by aZva, h
ocmes rios specs o he progrm, sch
s isrcio, orgiziol srcre, pr-
e-school relios. K12 Ic. lso reqires aZva
o proce eile l School Improe-
me Pl (SIP), which coers progrm oper-
ios s well s se chieeme. the pl
ms icle lysis o se perormce
o srize se ess, iclig compri-
so o he perormce o aZva ses o he
perormce o ll ses cross he se.
Ech o hese meshose o he se
hose o aZvas crriclm proierhs im-
por prpose. B he mliple reqiremes
p o wh col be see s sbsil
bre or y smll orgizio. aZvas smll
cerl s chooses o look i ierely.
alhogh he reqiremes geere yer-ro
work or aZva employees, hey he me he
mos o hese ciiies by sig hem or heirow prposes, oo. Ech o he my me
elio ciiies seres ierl prpose:
S members pore oer es scores, corse
compleio , ser siscio o
eermie how hey c improe heir progrm.
the SIP is se s giig ocme o or-
gize iormio bo wh specs o he
8/14/2019 Evaluating Online Learning
21/80
1111
progrm ee xig o moior he schools
progress owr is se gols. alhogh he
process is ime-cosmig, eeryoe bees:
K12 Ic. is ssre h he school is oig wh
i shol, aZva hs srcre pproch o
improig is progrm, i c emosre
o he se ohers h se perormce
is meeig expecios.
aZva is ble o mke he mos o is mory
elios becse he schools clre sp-
pors i: S members icorpore collec-
io lysis io heir eeryy resposibil-
iies, rher h iewig hem s exr bres
o heir worklo. Frhermore, aZvas le-
ers iiie collecio lysis eors
o heir ow. they reqely coc olie
sreys o pres o gge he eecieess o
priclr serices. More recely, hey lso he
beg o srey echers bo heir proessio-
l eelopme ees heir siscio wih
he riigs proie o hem. Were ryig o
o sreys er eery sigle proessiol e-
elopme [sessio], o o wh ws mos
eecie, sys Gior. do hey w more o
his, less o his? Ws his oo mch ime? Ws
his eogh ime? th eebck hs bee ery
goo. aZvas K8 pricipl, Brige Schleier,
corms h echers resposes o he sreys
re ke ery seriosly. Wheeer srey
comes p we see ee, she sys, we
will eiely p h o he ge or he
ex moh o proessiol eelopme.
togeher, hese my eors proie aZva wih
comprehesie iormio h helps he school
ress exerl ccobiliy ems, while
lso serig ierl progrm improeme ob-
jecies. Js s impor, aZvas rios el-
io ciiies re iegre sppor ech
oher. For isce, he SIP is bse o he -
igs rom he elio ciiies me
by he se K12 Ic., he lers i
process icles pe o progress me
owr SIP gols. More broly, he ormie
elio ciiies help he school leers o
se specic cemic gols eelop pl
or rechig hem, which limely helps hem
improe he chieeme ocomes ssesse by
he se. Oe lesso aZva illsres is how o
mke he mos o elios h re iiie
exerlly by reig eery collecio c-
iiy s opporiy o ler somehig l-
ble h c sere he progrm.
Summary
as he boe progrm elios emosre,
someimes he bes pproch o meeig he
ees o mliple skeholers is beig proc-
ie. the seps re srighorwr b criicl:
Whe cosierig elio, progrm le-
ers shol rs ieiy he rios skehol-ers who will be ierese i he elio
wh hey will w o kow. they migh
cosier cocig ieriews or ocs grops
o collec his iormio. Leers he ee
o si hrogh his iormio prioriize
heir ssessme gols. they shol eelop
cler isio or wh hey w heir elio
o o work wih elors o choose
elio ype h will mee heir ees. I i
is me o sere seerl iere skeholergrops, elors progrm leers migh
ecie o coc mli-meho sy h
combies ormie smmie elio
ciiies. they migh lso cosier eelop-
ig mliyer elio pl h resses
sepre gols i iere yers. I he repor-
ig phse, progrm leers elors c
8/14/2019 Evaluating Online Learning
22/80
1212
Evaluating Online Learning: Challenges and Strategies or SuccessInnOvatIOnS In EduCatIOn
cosier commicig igs o iere
skeholer ieces i wys h re ilore
o heir ees ieress.
I isces where olie progrms prici-
pe i mory elios, progrm leers
shol seek o bil o hese eors se
hem or ierl prposes s well. they c
leerge he iormio lere i smm-
ie elio o improe he progrm, cqire
ig, or esblish he progrms creibiliy.
Elors c help progrm leers piggybck
o y mory ssessme ciiies by se-
lecig complemery elio mehos h
will proie o js he reqire b lso
iormio h progrm s c se or heir
ow improeme prposes.
Building on the Existing Base o
Knowledge
uer orml circmsces, elors re-qely begi heir work by reiewig ilble
reserch lierre. they lso my serch or
cocepl rmework mog similr sies or
look or exisig collecio ools, sch s
sreys rbrics, h c be borrowe or
pe. Ye, compre o my oher opics i
K12 ecio, he boy o reserch lierre
o K12 olie lerig is reliely ew
rrow. ailble escripie sies re oe
ery specic oer igs h re o es-ily geerlize o oher olie progrms or re-
sorces. Empiricl sies re ew. Oher kis
o ools or elors re limie, oo. Rece
eors he le o mliple ses o srs or
K12 olie lerig (see Srs or K12
Olie Lerig, p. 13). Howeer, here sill re
o wiely ccepe ecio progrm ocome
mesres, mkig i icl or elors o
gge sccess relie o oher olie or ri-
iol progrms.
Gie he exisig bse o kowlege o K12
olie lerig, how shol elors procee?
O corse, elors will rs w o cosl
he K12 olie lerig reserch h oes ex-
is. alhogh he el is compriely limie, i
is growig ech yer lrey hs geere
mber o sigic resorces (see ppeix
a, p. 59). amog oher orgizios, he norh
americ Cocil or Olie Lerig (naCOL)
hs eelope collece ozes o pblic-
ios, reserch sies, oher resorces se-
l or elors o K12 olie lerig pro-
grms. I some cses, elors lso my w
o look o higher ecio orgizios, which
he richer lierre o olie lerig el-
io, iclig seerl pblicios h ieiy
srs bes prcices (see ppeix a).
I some cses, hese resorces c be pe
or K12 seigs, b i oher cses, reserchers
he o, hey o o rsle well.
aoher pproch is o eelop elio ools
echiqes rom scrch. as escribe be-
low, his my be s simple s eig s-
rizig he ocome mesres se mog
mliple schools or eors, like he elors
o digil Lerig Commos i. Or i my be
mch more mbiios eor, s whe apple-
o eSchools leers eelope ew moelor elig irl schools wih olie sys-
em or compilig elio . Filly, some
elors respo o limie kowlege bse
by ig o i. For exmple, he elors o
Loisis algebr I Olie progrm he pb-
lishe heir elio igs or he bee
o oher elors progrm miisrors.
8/14/2019 Evaluating Online Learning
23/80
1313
I iere b eqlly helpl shio, he
leers o appleo eSchool he coribe o
he el, by eelopig Web sie h llows
miisrors o olie progrms o shre heir
bes prcices i pblic orm.
Clearly Defne Outcome Measures
Elors ms be ble o clerly ricle key
progrm gols, ee ocomes h lig wih
hem, he ieiy specic ocome me-
sres h c be se o rck he progrms
progress i meeig he gols. Presely, how-
eer, ocome mesres or elig olie
lerig progrms re o cosisely ee,which mkes i icl or skeholers o gge
progrms sccess, compre i o oher pro-
grms, or se improeme gols h re bse
o he experiece o oher progrms. Frher-
more, he lck o cosise ocome mesres
crees echicl heches or elors. a re-
ce ricle co-hore by Liz Ppe, presie
chie execie ocer o virl High School
Globl Cosorim, opro ework o o-
lie schools, escribes he problem his wy:
Although standards or online course andprogram eectiveness have been identied,
data-driven yardsticks or measuring
against those standards are not generally
agreed upon or in use. There is no general
agreement about what to measure and
how to measure. Even or measures
that most programs use, such as course
completion rates, there is variation in the
metrics because the online programs that
measure course completion rates do not
measure in the same manner.3
the elors o Wshigo ses digil
Lerig Commos (dLC) ecoere js sch
problem whe empig o clcle he
mber o olie corse-kers sere by he
progrm. dLC is cerlly hose Web porl
h oers wie rge o olie corses rom
Standards for K12 Online LearningAs online learning in K12 education has expanded, there has been an eort to begin to codiy current best prac-
tices into a set o standards that educators can look to in guiding their own perormance. Several organizations
have released sets o these emerging standards based on practitioner input to date.
In late 2006, the Educational Technology Cooperative o the Southern Regional Education Board (SREB) issued
Standards for Quality Online Courses. These standards are available along with other key resources rom SREB at
http://www.sreb.org/programs/EdTech/SVS/index.asp.
A year later, NACOL published National Standards of Quality for Online Courses, which endorsed SREBs standards
and added a ew others. The national standards cover six broad topic areas: course content, instructional design,
student assessment, technology, course evaluation and management, and 21st-century skills.
NACOL also developed National Quality Standards for Online Teachingin 2008 and is currently working on pro-gram standards. The standards or online courses and teaching can be ound at http://www.nacol.org along with
many other resources.
The National Education Association also has published standards or online courses and online teachers, both
available at http://www.nea.org.
8/14/2019 Evaluating Online Learning
24/80
14
Evaluating Online Learning: Challenges and Strategies or SuccessInnOvatIOnS In EduCatIOn
meros prie eors. Whe elors
rom Cohe Reserch Elio rie o -
lyze dLCs corse-kig compleio res,
hey o rge o reporig prcices mog
he eors: Some rcke se pricipio
hrogho he corse, while ohers repore
oly o he mber o ses who complee
corse receie l gre. also, i some
cses, eors i o iereie bewee s-
es who wihrew rom corse ses
who receie F, cofig wo iere s-
e ocomes h migh he isic implic-
ios or progrm improeme.
Ppe e l. oe iiol problems i he wys
h corse compleio is ee:
When does the measure begin? How is
completion dened? Do students have a
no penalty period o enrollment in the
online course during which they may
drop rom the course and will not be
considered when calculating the course
completion rate? Is completion denedas a grade o 60 or 65? How are students
who withdrew rom the course ater the
no penalty period counted, especially i
they withdrew with a passing grade?4
Followig he recommeios o heir el-
ors, dLC s me eors o commice e-
iios o corse compleio wihrwl h
were ierlly cosise me sre ech
eor ws reporig ccre bse o
hese coersios. the resl ws higher leel
o cosisecy ccrcy i reporig.
there is growig eio o he problem o -
ee ocome mesres i he el o el-
ig olie lerig. a 2004 repor by Chy
Cgh e l., speciclly recommee
h srs be eelope or reporig he
cemic progrmmic ocomes o is-
ce lerig progrms.5 a naCOL eor is
er wy o eelop bechmrks or mesr-
ig progrm eecieess oerll srs
or progrm qliy. Mewhile, he bes el-
ors c o is o esre ierl cosisecy i
he ocome mesres se cross ll o heir
ow sorces. alhogh he reserch bse
is limie, elors my be ble o similr
sies or ies o how o ee ocomes.
Moig orwr, olie progrm elors c
prociely wys o shre reserch mehos eiios rech cosess o he
bes wys o mesre progrm eecieess.
Work Collaboratively to Develop New
Evaluation Tools
I he erly yers o appleo eSchool, school
leers becme wre h hey eee oer-
ll elio sysem o eermie he schools
sreghs wekesses. aer ilig o
exisig comprehesie ool h wol heirees, Be vogel, appleos pricipl Go-
erce Bor chir, Coie Rke, apple-
os progrm leer, beg o eelop heir
ow elio process. their gol ws o e-
sig isrme h wol ieiy he core
compoes ecessry or ses o be sc-
cessl i olie progrm. I iio, hey
we o cree process h col be se o
promp iloge mog progrm leers, s,
goerce bor members, exerl col-leges, bo he compoes o sccessl
olie lerig experiece, i orer o proie
irecio or re growh ehceme.
throgh exesie ierl iscssios
coslio wih exerl colleges, icl-
ig s rom sch olie progrms s virl
8/14/2019 Evaluating Online Learning
25/80
15
High School Flori virl School, vo-
gel Rke ieie eigh key progrm
compoes eelope rbric or me-
srig hem clle he Olie Progrm Percei-
er Isrme (OPPI). (See Key Compoes o
appleo eSchools Olie Progrm Perceier
Isrme, p. 16.) vogel sys, [Or gol ws]
o gre o hose iersl core compoes
h re ecessry i K12 olie progrm o
llow ses o be sccessl. We were ble
o shre [or iiil hoghs] wih oher people
i he K12 relm, sy, Wh re we miss-
ig? a wh oher pieces shol we ? I
js ki o eelope rom here.
the core compoes ieie i he OPPI re
wh appleo leers see s he esseil bil-
ig blocks or spporig ses i olie
eirome. the eigh compoes ress he
olie progrm sers eire experiece, rom
rs lerig bo he corse or progrm o
compleig i.
Whe eelopig he OPPI, vogel Rke
reserche my exisig olie progrm el-
ios i higher ecio, b o hem
iscie or bilig comprehesie rbric
he K12 leel. vogel oes, or exmple, h
hig ce-o-ce meors or coches or s-
es kig olie corses is criicl he K12
leel, wheres i is o cosiere so impor
or oler ses who re syig olie i
higher ecio. to cpre his progrm ele-me, vogel Rke icle Progrm Sp-
por s key compoe i he OPPI rbric, o-
csig o he riig gie o meors (slly
pres i he appleo eSchool moel) s well
s riig or locl school cocs who sp-
por coorie locl se ccess o o-
lie corses (see g. 1, Excerp rom appleo
eSchools Olie Progrm Perceier Isrme,
p. 17). to ssess meor percepio o progrm
qliy, he elors sreye hem ollowig
he compleio o ech olie corse.
as pr o he OPPI process, vogel Rke
eelope hree-phse pproch o ierl
elio, which hey reer o s sel-is-
coery process. I he discoery Phse, pro-
grm persoel ll o repor h escribes
he schools prcices i ech o he eigh res
ieie i he OPPI. the progrm ecisio-
mkers se he rbric o eermie wh leel o
progrm perormce is beig ie or ech
eleme: ecie, eelopig, procie, or ex-
emplry. I iio, progrm leers e-mil sr-
eys o ses, meors (slly pres),
echers he e o ech corse, giig hem
opporiy o comme o he progrms
perormce i ech o he eigh OPPI res.
I he Ocome Phse, resls rom he disco-
ery Phse repor sreys re smmrize,
geerig mericl rig i ech progrmre. a he sme ime, iormio o se
ocomes is reiewe, iclig se gres,
gre poi erges, corse compleio
res. Progrm ecisio-mkers syhesize ll
o his i ocome shee se i o
se gols or re growh eelopme.
Filly, i he Shrig o Bes Prcices Phse,
progrm leers my selec priclr prcices
o shre wih oher progrms. appleo hs pr-
ere wih oher olie progrms o orm heWiscosi eSchool nework, cosorim o ir-
l schools h shre resorces. the neworks
Web sie icles Bes Prcices Porolio
schools sig he OPPI re iie o sbmi ex-
mples rom heir elios. 6 Prcices h re
eermie o be i he procie or exem-
plry rge re cosiere or plceme i he
8/14/2019 Evaluating Online Learning
26/80
16
Evaluating Online Learning: Challenges and Strategies or SuccessInnOvatIOnS In EduCatIOn
Key Components of Appleton eSchools Online Program
Perceiver Instrument (OPPI)
Practitioners and program administrators use the OPPI to evaluate program perormance in eight dierent areas:
1. Program Inormation: System provides and updates inormation necessary or prospective users to understand the
program being oered and determine whether it may be a good ft or students.
2. Program Orientation: System provides an introduction or orientation that prepares students to be successul in the
online course.
3. Program Technology: System provides and supports program users hardware and sotware needs in the online
environment.
4. Program Curriculum: System provides and supports an interactive curriculum or the online course.
5. Program Teaching: System provides and supports teaching personnel dedicated to online learning and their
online students.
6. Characteristics and Skills Displayed by Successul Online Students: System identifes and provides opportunities
or students to practice characteristics necessary or success in an online environment.
7. Program Support: System provides and supports a system o support or all online students and mentors (e.g.,
parents) and coaches.
8. Program Data Collection: System collects data and uses that data to inorm program decision-makers and share
inormation with other programs.
porolio, pricipig schools re cie or
heir coribios. the eire elio sysem
is Web-bse, llowig or sremlie col-
lecio, lysis, shrig.
appleos ecisio o eelop is ow elio
rbric process proies seerl ges.
Besies reslig i perecly ilore elio
process, appleo leers lso he he biliy o
ele heir progrm y ime wiho wiigor ig or relyig o hir-pry elor.
Sill, eelopig elio process is ypiclly
expesie my o be prcicl opio or
my progrms. appleos leers spe my
hors reserchig eelopig ocome me-
sres (i.e., he escripios o prcice or ech
progrm compoe er ech leel o progrm
perormce). they lso iese bo $15,000
o heir progrm gr s o py Web e-
eloper o esig he olie sysem or compil-
ig isplyig elio . For ohers -
empig o eelop his ype o ilore rbric
process, ccessig osie experise is criicl
o ll gps i kowlege or cpciy. appleo
leers collbore exesiely wih experiece
colleges rom oher irl schools, priclrly
s hey were eelopig heir rbric.
Share Evaluation Findings With Other
Programs and Evaluators
alhogh he OPPI rbric ws eelope speci-
clly or appleo, rom he begiig vogel
Rke iee o shre i wih oher progrms.
8/14/2019 Evaluating Online Learning
27/80
1717
Figure 1. Excerpt From Appleton eSchools Online Program Perceiver Instrument*
Compoe 7. Progrm Sppor. the prpose o he sppor ework is o proie iiol sppor
o he se h complemes he isrcor. this icles o oly sppor perso rom home,
b lso oher school resorces h my icle coselor, socil worker, ec.
ELEMENT
LEVEL OF PROGRAM PERFORMANCE
DEFICIENT DEVELOPING PROFICIENT EXEMPLARY
Overall StudentSupport Structure Plan
No student supportstructure plan in placeor the plan is poorly
written and/or incom-plete. Students are notreceiving necessaryresources.
Student support struc-ture plan is in placebut parts o the planare incomplete or maybe unclear. The planprovides or resourcesnecessary or studentsuccess but some gapsin the plan may exist.
Student support structureplan is in place and isclear in its purpose andobjectives. The planprovides or resourcesnecessary or studentsuccess.
Exemplary student sup-port structure plan is inplace and is a provenmodel or assuring stu-dents have necessarysupport and ongoingsupport.
Mentor responsibilitieshave been developedand shared
Mentor responsibilitieshave not been devel-oped or are incompleteand/or unclear or havenot been appropriatelyshared.
Mentor responsibilitieshave been developedand shared but partsmay be unclear and/or all mentors have notreceived the inormation.
Mentor responsibilitiesare well written, clear,and have been shared
with all mentors.
Mentor responsibilitiesare extremely well writ-ten, clear, and shared innumerous ormats withall mentors.
Mentors are providednecessary trainingand support
No mentor training pro-gram provided or thereare no written objectivesthat clearly outline thepurpose o the men-tor training program.No ongoing support isavailable.
Mentor training programis provided and thereare written objectivesthat outline the purposeo the mentor training.Some objectives maybe unclear. Ongoingsupport is available, butmay be inconsistent.
Mentor training programis provided and thereare objectives that out-line the purpose o thementor training. Fewobjectives are unclear.Ongoing support isavailable.
Mentor training programis provided and thereare extremely clear andconcise objectives thatoutline the purpose othe mentor training pro-gram. Ongoing supportis consistently provided.
Mentors providepositive support orstudent
No system in place tomonitor mentor supportand/or concerns thatmay arise.
System in place tomonitor mentor supportand/or concerns thatmay arise, but systemmay break down romtime to time.
System in place to mon-itor mentor support and/or concerns that mayarise. System is reliablemost o the time.
Proven system in placeto monitor mentor sup-port and/or concernsthat may arise. Thesystem provides op-portunities or two-waycommunication.
Mentors have the abil-ity to communicate
with teacher
No system in place toallow mentors to com-municate with teacher ina timely manner.
System in place toallow mentors to com-municate with teacherin a timely manner butsystem may break downrom time to time.
System in place to allowmentors to communi-cate with teacher in atimely manner. Systemis reliable most o thetime.
Proven system inplace to allow men-tors to communicate
with teacher in a timelymanner. The systemprovides check-ins withteacher on a regularbasis.
* The U.S. Department o Education does not mandate or prescribe particular curricula or lesson plans. The inormation in this gure was provided by the
identied site or program and is included here as an illustration o only one o many resources that educators may nd helpul and use at their option. The
Department cannot ensure its accuracy. Furthermore, the inclusion o inormation in this gure does not refect the relevance, timeliness, or completeness o
this inormation; nor is it intended to endorse any views, approaches, products, or services mentioned in the gure.
8/14/2019 Evaluating Online Learning
28/80
18
Evaluating Online Learning: Challenges and Strategies or SuccessInnOvatIOnS In EduCatIOn
echer who my o be cerie o elier l-
gebr isrcio. B oce i his clssroom,
ech se hs his or her ow comper
pricipes i olie clss eliere by
highly qlie (i.e., cerie) lgebr echer.
the i-clss echer gies ses ce-o-ce
ssisce, oersees lb ciiies, procors ess,
is geerlly resposible or miiig
mosphere h is cocie o lerig. the
olie echer eliers he lgebr isrcio,
swers ses qesios i olie is-
cssio bor, gres ssigmes i e-mil,
proies ses wih eebck o homework
ess, sbmis gres. the olie
i-clss echers commice reqely wih
ech oher o iscss ses progress col-
lbore o how o help ses ler he pr-
iclr coe beig coere. this iercio
bewee echers o oly bees ses; i
lso seres s orm o proessiol eelop-
me or he i-clss isrcors. I iio o
proiig ll ses wih high-qliy lgebr
isrcio, secory gol o he progrm
is o icrese he isrciol skills o he i-
clss echers sppor hem i erig heir
mhemics echig cerice.
alhogh is oers beliee h he alge-
br I Olie moel oere gre promise or
ressig Loisis shorge o mhem-
ics echers, whe he progrm ws lche
i 2002 hey h o eiece o bck p his
belie. the key qesio ws wheher sch progrm col proie ses wih lerig
opporiies h were s eecie s hose
i riiol seigs. I i were s eecie,
he progrm col proie imely cos-
eecie solio or he mhemics echer
shorge. Loisi eee hr eiece o
show wheher he progrm ws creible.
this rbric is crrely ccessible ree o chrge
hrogh he Web sie o he Wiscosi eSchool
nework, escribe boe.* vogel explis h
he OPPI is mbrell elio sysem re
reily pble o oher progrms: Ierlly,
his sysem oes sk people o he oer-
whelmig mo o kowlege. I llows peo-
ple o mke weks s eee or heir pric-
lr progrms, b hey o he o cree he
whole wheel oer gi [by esigig heir ow
elio sysem]. the OPPI sysem lso llows
or ggregio o resls cross mliple pro-
grms mechism h wol llow grops o
schools i se, or exmple, o lyze heir
combie . to ssis schools sig he OPPI
or he rs ime, appleo oers coslio
serices o ech oher sers how o ierpre
commice key igs. throgh heir eors
o shre heir elio ool cree he o-
lie orm, appleo leers he eelope
ecie ioie wy o bil he kowl-
ege bse o olie lerig progrms.
I Loisi, elors rom Ecio deel-
opme Ceer (EdC) he se more coe-
iol chels or shrig igs rom he
elio o he ses algebr I Olie pro-
grm. this progrm ws cree by he Loisi
deprme o Ecio o ress he ses
shorge o highly qlie lgebr echers, es-
pecilly i rb rrl seigs. I iio,
isrics esirig o proie cerie echers
ccess o pegogy riig meorig sohey c bil cpciy or srog mhemics
isrcio re eligible o pricipe. I alge-
br I Olie corses, ses physiclly e
clss i sr bricks--morr clssroom
heir home school, which is mge by
* Registration is required to access the OPPI and some consultation with its developers
may be needed to implement the process ully.
8/14/2019 Evaluating Online Learning
29/80
19
Followig mber o ierl elio cii-
ies rig he progrms rs wo yers, i 2004
he progrms leers egge exerl el-
io em cosisig o Rebecc Crey o EdC,
orgizio wih experiece i reserchig
olie lerig; Lr Odwyer o Boso Col-
leges Lych School o Ecio; Gle
Kleim o he Friy Isie or Eciol
Ioio norh Croli Se uiersiy,
College o Ecio. the elors were im-
presse by he progrm leers willigess o
ergo rigoros elio. We i he
o o lo o coicig, sys EdCs Crey.
they we i o be s rigoros s possible,
which ws gre , I hik, lile bi sl.
the progrm lso ws gie boos i he orm
o gr rom he norh Cerl Regiol E-
ciol Lborory (nCREL), eerlly e
ecio lborory. the gr e primry
reserch o he eecieess o olie lerig
proie he projec wih $75,000 beyo is
iiil $35,000 elio bge rom he se
legislre. the iiol ig llowe EdC
o ocs grops i-clss obserios, s
well s o gme is ow elio cpciy
by hirig exerl cosl wih exesie
experise i reserch mehoology lysis.
the EdC elors chose qsi-experime-
l esig (see Glossry o Commo Elio
terms, p. 65) o compre ses erolle i
he olie lgebr progrm wih hose syig
lgebr oly i riiol ce-o-ce clss-room orm. to exmie he impc o he alge-
br I Olie corse, hey se hierrchicl lier
moelig o lyze poses scores oher
collece rom he reme corol
grops. to eermie i ses i olie ler-
ig progrms egge i iere ypes o peer-
o-peer iercios i hey perceie heir
lerig experieces ierely h ses i
riiol clssrooms, he elors sreye
ses i boh eiromes coce
obserios i hl o he reme clssrooms.
I ol, he elors sie algebr I Olie
corses ce-o-ce corses i six isrics.
aer compleig heir ssessme, he elors
proce l repors or he Loisi depr-
me o Ecio nCREL ler wroe
wo ricles bo he progrm or proessiol
jorls. the rs ricle, pblishe i he Jour-
nal o Research on Technology in Education,7
escribe algebr I Olie s ible moel or
proiig eecie lgebr isrcio. I he
sy, olie ses showe comprble (
someimes sroger) es scores, sye o sk,
spe more ime iercig wih clssmes
bo mh coe h ses i riiol
clssroom seigs. the elors specle
h his ws resl o he progrms iqe
moel, which brigs he olie ses ogeh-
er wih heir peers reglrly schele ime.
the elors o ew res or cocer s
well. For exmple, higher percege o o-
lie ses repore h hey i o he
goo lerig experiece, ig h is boh
sppore corice by reserch sies
o olie lerig rom higher ecio. the
elio lso o h he algebr I Olie
ses el less coe i heir lgebr skills
h i riiol ses, ig he el-
ors eel is priclrly ripe or rher resercheors. (For iiol iscssio, see Ierpre-
ig he Impc o Progrm Mriy, p. 40.)
the algebr I Olie elors wroe pb-
lishe seco ricle i he Journal o Asyn-
chronous Technologiesh ocse o he pro-
grm s proessiol eelopme moel or
8/14/2019 Evaluating Online Learning
30/80
20
Evaluating Online Learning: Challenges and Strategies or SuccessInnOvatIOnS In EduCatIOn
Gie he lck o commoly se ocome me-
sres or olie lerig elios, iiil
progrms shol les srie or ierl cosis-
ecy, s dLC hs. I workig wih mliple e-
ors or school sies, olie progrm leers ee
o ricle cler se o bsiess rles or wh
re o be collece how, isribig
hese gielies o ll pries who re collecig
iormio. Lookig he, wiho hese com-
mo gielies, elors will be hr presse
o compre heir progrms ocomes wih oh-
ers. Some o he elors ere i his gie
he me coribios o he el o olie
lerig elio, like appleos leers, who
eelope elio moel h c be bor-
rowe or pe by oher progrms, he
elors o algebr I Olie, who pblishe
heir sy igs i proessiol jorls.
Evaluating Multiaceted Online
Resources
Like my riiol ecio progrms, o-
lie lerig resorces someimes oer pr-
icips wie rge o lerig experieces.
their mlicee oerigs re boo or s-
es or echers wih ierse ieress, b c
be ilemm or elors seekig iorm
igs bo eecieess. I he cse o
eciol Web sie like Wshigos dLC, or
exmple, iere ypes o sers will explore
iere resorces; some ses my ke olie corse while ohers my be reserchig
colleges or seekig meor. virl schools h
se mliple corse proiers prese similr
corm, ee he sme olie corse
my oer iereie lerig experieces i,
or exmple, ses iiie more or less co-
c wih he corse isrcor or receie ryig
cerie or iexperiece mh echers.8
I his piece, he elors escribe he pro-
grms pirig o olie i-clss echers s
ible olie moel or proiig [he i-clss]
echers wih eecie moel or heic
embee proessiol eelopme h is
rele o heir clssroom experieces.
O corse, o ll progrms will he he re-
sorces o coribe o he reserch lierre i
his mer. I Loisis cse, he elors
h exesie experise i olie elio
ook he ime iiiie reqire or pblish-
ig heir igs i cemic jorls. I so
oig, hey sere wo prposes: proiig he
progrm leers wih he eiece hey eee
o coely procee wih algebr I Olie
pblishig mch-eee reserch o ses
h migh be cosierig similr pproches.
Summary
the lierre o K12 olie lerig is grow-ig. Seerl pblicios resorces oc-
me emergig bes prcices policies i
olie lerig (see ppeix a). For progrm
leers elors who re eelopig
elio, he qliy srs rom SREB
naCOL proie bsic rmework or lookig
he qliy o olie corses echers. there
lso is growig boy o sies rom which
elors c rw lessos p mehos;
elors ee o reie he wheel. a hesme ime, hey ms exercise cio whe p-
plyig igs, ssmpios, or mehos rom
oher sies, s olie progrms resorces
ry remeosly i whom hey sere, wh
hey oer, how hey oer i. Wh works
bes or oe progrm elio my o be
pproprie or oher.
8/14/2019 Evaluating Online Learning
31/80
21
egrees o ce-o-ce sppor rom pre
or coch. (a similr lck o iormiy c be
o i riiol seigs wih iere i-
srcors sig ryig isrciol moels.)
Whe ce wih mlicee resorce, how
is elor o ers ocme he
olie lerig experiece, mch less eer-
mie wh le i s?
Seerl o he elios ere i his gie
ecoere his isse, lbei i isic wys.
dLC elors were chllege o ssess howses experiece beee rom he
Web sies bro rge o resorces. Elors
o Mryl Pblic teleisios thikpor Web
sie, wih is exesie echer se re-
sorces rom my proiers, similrly srggle
o ssess is impc o se chieeme. I
ery iere exmple, he elors or he
arizo virl acemy (aZva) ce he chl-
lege o elig hybri corse h icl-
e boh olie ce-o-ce compoes i which ses iiil experieces
rie cosierbly.
Combine Breadth and Depth to Evaluate
Resource-rich Web Sites
Wih is wie rge o serices resorces
or ses echers, dLC is sprwlig,
ierse projec o ele. throgh his ce-
rlly hose Web sie, ses c ccess oer
300 olie corses, iclig ll core sbjecs rios elecies, pls ace Plceme
(aP) corses Eglish s Seco Lgge.
dLC lso oers ses olie meors, college
creer plig resorces, exesie
igil librry. I iio, dLC oers oher re-
sorces ools or echers, iclig olie
crricl, ciiies, igosics. For schools
h sig p o se dLC, he progrm proies
riig or school persoel o ssis hem i
implemeig he Web sies resorces.
Iiilly, dLCs elio sregy ws o col-
lec bro iormio bo how he Web sie
is se. Ler, progrm leers shie heir
sregy o ocs o ewer rrower opics
h col sbsie he progrms eccy.
the elors ocse o se chiee-
me i he olie corses o school-leel
sppors or ecors o help hem mke he
mos o dLCs resorces. togeher, he series o
dLC elioshere he bee les e
isic eors o ecombie breh
eph, he bil o ech ohers igs rom
yer o yer, he proce impor or-
mie smmie igs (see Glossry o
Commo Elio terms, p. 65).
I he projecs rs yer, debr Friem, le
miisror he uiersiy o Wshigo (
dLC prer orgizio), coce el-io h sogh iormio o whom dLC
seres wh school coiios poli-
cies bes sppor is se. to swer hese qes-
ios, he elors selece mehos esige
o elici iormio irecly rom pricips,
iclig iscssios wih dLC miisrors,
bor members, school leers, echers, s
well s se echer sreys h ske
bo heir se o compers he Iere
bo he iliy o he dLC riig. theelor lso looke ew iicors o s-
e chieeme, sch s he gres h s-
es receie or dLC olie corses.
the elio yiele bro igs bo
operiol isses oe he ee or dLC
o prioriize mog is my prposes
8/14/2019 Evaluating Online Learning
32/80
22
Evaluating Online Learning: Challenges and Strategies or SuccessInnOvatIOnS In EduCatIOn
miisrors (primrily o help eelop srey
qesios). to gi isigh io how well s-
es were perormig i he clsses, he el-
ors lyze gres compleio res or
ses erolle i dLC corses. the el-
io ciiies coce i he seco yer
gi poie o he ee or more school-leel
sppor or sig dLC resorces. the el-
ors o h some schools were excie
commie o sig he Web sies resorces,
b were erilizig i becse hey lcke
scie srcres, sch s riig ime
or echers o ler bo is oerigs, ier-
l commicio mechisms o rck se
progress, eqe echicl sppor.
Whe dLCs leers beg o coemple
hir-yer elio, hey we more h
bsic ocome , sch s se gres
compleio res. We c co how my
corses, we kow he orie sbjecs, we
kow he gre erges ll o h, sys
Jy Mrgrh-Hge, dLC presie chie ex-
ecie ocer. Wh hey eee, she explis,
ws o ge he so wh, meig hey we
o ers wh ierece [dLC] mkes.
the elio em kew h i is eor were
o proce relible iormio bo dLCs i-
fece o se chieeme, i wol ee
o zero i o oe, or js ew, o he Web
sies my compoes. Some eresdLCs
s igil librry, or exmplesimply wereo goo cies or he ki o sy hey
ple o coc. as Krl nelso, dLCs i-
recor o echology operios, explis,
I is ery icl o ele he eecieess
o o swer so wh qesio bo
librry bse, or exmple. Is js hr o
poi o ki sig librry bse he
ieces. I lso coere impor -
ig bo se chieeme i he olie
corses: the grees percege o ses
(52 perce) receie Fs, b he ex grees
percege (37 perce) receie as. to expli
hese ocomes, he elor poie o he lck
o iormiy i ses moiio ees,
he ype o cemic sppor ilble o
hem. the elor lso oe he ryig ql-
iy o he eors who proie corses, ig
h some eors re fexible resposie
o ses ees; ohers re obly ifexible.
Some re highly proessiol operios, ohers
less so.9 this elio lso escribe sbs-
il riio i how well schools were ble o
sppor he se o dLC resorces. the igs
helpe progrm miisrors who, i respose,
seppe p heir eors o ri ecors bo
he Web porls resorces how o se i
wih ses. the elio lso ws jmp-
ig-o poi or re ssessmes h wol
ollow p o he hemes o se chieeme
i he olie corses sppors or ecors
o help hem ke ge o dLCs oerigs.
I he projecs seco yer, projec leers
lche oher elio. this eor co-
sise o se ocs grops o ieiy s-
es expecios o dLCs olie corses, s-
e pre-corse preprio, oerll experiece,
sggesios or improig he corses
proiig beer sppor. as sepre eor,
hey lso corce wih iepee el-or, Cohe Reserch Elio, o ler
more bo he behior moiios o
ses oher sers, sch s school librr-
is, echers, miisrors. this spec o
he elio cosise o olie sreys wih
ses, echers, school librris; i-
eriews wih selece echers, librris,
8/14/2019 Evaluating Online Learning
33/80
23
es score goig p. ulimely, sys nelso,
dLCs leers chose o look primrily he o-
lie corses, belieig h his ws he ere
hey col bes ele.
dLC leers hire osie elors, Fos &
associes, o help hem rill ow io spe-
cic spec o se chieemeeermi-
ig he role h dLC olie corses ply i: 1)
eblig ses o gre rom high school
2) helpig ses become eligible l-
ly prepre or college. I his elio, he
reserchers ieie smple o 115 gre
seiors rom 17 schools who h complee dLC
corses. the elors isie he schools o
beer ers olie corse-kig policies
grio reqiremes o ieiy dLC
corses o he rscrips o hese 115 ses.
a he school sies, elors ieriewe school
cooriors exmie se chieeme
, se rscrips, dLC ocmes.
the elio ge dLCs leers wh hey we: cocree eiece o he impc o
dLC olie corses. this sy showe h 76
perce o ses who ook olie corse
hrogh dLC i so becse he clss ws o
ilble heir school h oe-hir o he
ses i he sy wol o he gre
wiho he creis rom heir olie corse. this
oher elio igs, show h we re
meeig or missio, sys Mrgrh-Hge. We
re ccomplishig wh we were se o o c-complish. a is relly impor or s o be
ble o s elier hose kis o messges
wih h ki o behi s. dLC hs se
is elio igs i mliple wys, icl-
ig whe mrkeig he progrm o osiers,
o emosre is rge o oerigs is e-
ec o ses (see g. 2, Excerp rom digil
Lerig Commos Meeting 21st Century Learn-
ing Challenges in Washington State, p. 24).
I wol be impossible o coc compre-
hesie elio o eeryhig h dLC hs
o oer, b cerily he elio sregy
o combiig breh eph hs gie i
gre el o sel iormio. dLCs leers
he se he igs rom ll he elios
o improe heir oerigs o emosre
eecieess o ers oher skeholers.
I Mryl, thikpor elors ce similrchllege i ryig o sy s Web sie h
compiles eciol resorces or echers
ses. a rs, he elio em rom Mc-
ro Ieriol, reserch, mgeme,
iormio echology rm, coce sch c-
iiies s gherig siscio , reiewig
Web sie coe, ocmeig how he sie
ws se. B oer ime, projec leers were
ske by ers o proie more cocree ei-
ece bo thikpors impc o se per-ormce. the elio ( he projec isel)
h o eole o mee his em.
I respose, he em ecie o rero he
elio i 2005, selig o wo-pr el-
io h wol oer boh breh eph.
Firs, he elors sreye ll regisere s-
ers bo sie sge siscio, sec-
o, hey esige romize corolle
ril (see Glossry o Commo Elio terms,p. 65) o sy how oe o he sies mos pop-
lr eres elecroic el ripece
ses lerig. Seerl el rips h bee
eelope er his gr; he oe selece ws
Phwys o Freeom, bo slery he u-
ergro Rilro. this priclr proc ws
chose or mber o resos: mos mile
8/14/2019 Evaluating Online Learning
34/80
24
Evaluating Online Learning: Challenges and Strategies or SuccessInnOvatIOnS In EduCatIOn
Figure 2. Excerpt From Digital Learning Commons Meeting 21st Century
Learning Challenges in Washington State*
* The U.S. Department o Education does not mandate or prescribe particular curricula or lesson plans. The inormation in this gure was provided by theidentied site or program and is included here as an illustration o only one o many resources that educators may nd helpul and use at their option. TheDepartment cannot ensure its accuracy. Furthermore, the inclusion o inormation in this gure does not refect the relevance, timeliness, or completeness o thisinormation; nor is it intended to endorse any views, approaches, products, or services mentioned in the gure.
8/14/2019 Evaluating Online Learning
35/80
25
school socil sies crricl icle he op-
ic; he elors h obsere he el rip i
clssrooms i erlier ormie sy were
wre o ses high ieres i is opics
ciiies; Web sie sisics iice h i
ws heily iewe ilize resorce.
the elors ge pre- posess o coe
kowlege bo slery he uergro
Rilro o ses whose echers se he
elecroic el rip corol grops o s-
es whose echers se riiol isrcio.
they o h he elecroic el rip h ery sbsil posiie impc o se ler-
ig, priclrly mog ses whose echers
h preiosly se i: these ses o experi-
ece echers score 121 perce higher o he
coe kowlege es h he ses whose
echers se riiol isrcio.
Like he dLC elio, he thikpor el-
io proe sel boh or ormie sm-
mie prposes. thikpors leers lereh echers who were ew o he elecroic
el rip eee more riig experiece
o sccesslly icorpore i io heir clss-
rooms. they lso lere h oce echers
kew how o se he ool, heir ses lere
he is coe r beer h heir peers i
riiol clsses. the elors wo-pr pl
ge thikpors leers wh hey eee:
bro iormio bo sge siscio
creible eiece h reqely seere hs rel impc o ses.
Use Multiple Methods to Capture Wide-ranging
Student Experiences in Online Courses
I he exmples boe, elors srggle o
wrp heir rms ro sprwlig Web resorc-
es h lcke iormiy. Someimes similr
chllege is o he micro leel, s whe
ses he heerogeeos experieces i
he sme olie clss. the leers o arizos
aZva srggle wih his problem whe hey se
o o ele oe o heir olie corses.
I 2006, aZva school leers beg o ex-
perime wih hybri corsesreglr olie
clsses spplemee wih weekly ce-o-ce
lessos rom clssroom echer. the i-per-
so compoe ws origilly esige i
ery srcre wy: Ses receie clss-
room isrcio eery week specic ime
locio, hey h o commi o his
weekly isrcio or eire semeser. I -
iio, ses col pricipe i he hybri
clss oly i hey were workig eiher o gre
leel or o more h oe leel below gre
leel. these resricios llowe he ce-o-ce
echers o oer he sme lessos o ll ses
rig he weekly sessio. School leers spe-
ciclly esige his srcre o brig ior-
miy o ses experieces mke i esier
o ele he clss. as aZvas irecor, Mry
Gior, explis, We we he hybri expe-
riece o be he sme or ll he kis so we
col clly eermie wheher or o i is
icresig se chieeme.
Howeer, whe progrm leers sreye pr-
es he semeser brek, Gior sys, Pr-
es oere some ery specic eebck. they
i like he semeser-log, oce--week com-mime, hey rge h he srcre
preee ses rom workig heir ow
pce. Ise, pres we rop-i moel
h wol oer ses more fexibiliy
ilore ssisce. I respose, she sys, We
olly oerhle he corse or he seco se-
meser me i iere ki o moel.
8/14/2019 Evaluating Online Learning
36/80
26
Evaluating Online Learning: Challenges and Strategies or SuccessInnOvatIOnS In EduCatIOn
I he ew orm, echers o o elier pre-
pre lessos, b, ise, work wih ses
oe-o-oe or i smll grops o y opic wih
which se is srgglig.
While he fexibiliy o he ew moel mees
se ees, ses rlly will he more
rie experieces sig i school leers
will o be ble o isole is eec o se
chieeme. I oher wors, obsere gis
col be e o y mber o cors, sch s
how reqely he se rops i, wheher
se works oe-o-oe wih echer ori smll grop, wh coe is coere
rig he rop-i sessio. I his isce, he
ees o progrm pricips ecessrily o-
weighe hose o he elio.
Howeer, becse he school hs mber o
oher collecio eors i plce, Gior
her colleges will sill be ble o gher ior-
mio bo wheher he hybri moel is help-
ig ses. School miisrors rck howreqely ses e he rop-i clss
chr ses cemic progress hrogh he
crriclm boh beore er hey prici-
pe i he hybri progrm. aZva lso sepre-
ly exmies se es scores or ses who
e he hybri progrm o reglr bsis. I
iio, aZva reqely ses sreys o pr-
es, ses, echers o gher iorm-
io bo he eecieess o my specs o
heir progrm, iclig he hybri clss. these
kis o ciiies c proie impor isighs
whe corolle sies re impossible.
Summary
thogh mlicee resorces c mke i i-
cl or elors o gge eecieess, goo
eliosespecilly hose sig mliple,
complemery reserch mehosc ieiy
he circmsces er which he progrm or
resorce is mos likely o sccee or il c
geere sel recommeios or sregh-
eig wek pois. Elors who re syig
mlicee resorces shol cosier sr-
egy h combies boh breh eph.
I syig eciol Web sie h oers
rry o resorces, elors migh collec bro
iormio bo sie sge he selec oe
or wo priclr eres o exmie i more
eph. Progrm leers c cilie his process
by clerly riclig wh ech resorce is i-
ee o o, or wh ocomes hey wol hope
o see i he resorce ws beig se eeciely.
From his lis, progrm leers elors
c work ogeher o eermie wh o sy
how. I some isces, i migh be logicl
o esig mliyer elio h ocses o
isic progrm compoes i iere yers,
or collecs bro iormio i he rs yer,
rrows i ocs i sbseqe yers.
I elig priclr corse or resorce h
oers ses wie rge o experieces,
elors migh cosier sig mix o qi-
ie qliie mehos o proie well-
roe ssessme o i. Rich, escripie i-
ormio bo ses experieces wih he
corse or resorce c be sel whe ryig o
ierpre bo se ocomes.
Finding Appropriate Comparison
Groups
Whe elors he reserch qesios bo
olie progrms impc o se chiee-
me, heir bes sregy or swerig hose
8/14/2019 Evaluating Online Learning
37/80
27
qesios is oe experimel esig, like
romize corolle ril, or qsi-experi-
mel esig h reqires hem o mche
compriso grops. these wo esigs re he
bes, mos wiely ccepe mehos or eer-
miig progrm eecs (see ble 2, p. 28).
Elors who se hese mehos ms procee
crelly. they ms rs esre h compri-
sos re pproprie, kig io cosierio
he poplio sere he progrms gols
srcre. For exmple, olie progrms
eice o crei recoery wol w o com-
pre heir se ocomes wih hose o oher
crei recoery progrms becse hey re ser-
ig similir poplios. Elors ms lso be
scrplos i heir eors o goo compri-
so gropsoe chllegig sk. For pr-
iclr olie clss, here my be o correspo-
ig ce-o-ce clss. Or i my be icl o
oi sel-selecio bis i ses (or ech-
ers) he chose o pricipe i olie pro-
grm. Or he olie progrm migh sere wie
rge o ses, possibly rom mliple ses,
wih bro rge o ges, or wih iere le-
els o preprio, mkig i icl o compre
o riiol seig or oher olie progrm.
Elors empig o coc romize
corolle ril c ecoer ee greer
chllege i eisig wy o romly ssig
ses o receie reme (see Glossry o
Commo Elio terms, p. 65) olie.
How migh elors work ro hese com-
priso grop iclies?
Seerl o he elios ere i his gie
sogh o compre isce lerig progrms
wih ce-o-ce lerig seigs, hey
ook rios pproches o he ihere echi-
cl chlleges o oig so. despie some i-
clies log he wy, he elors o olie
projecs i Loisi, albm, Mryl ll
sccesslly coce comprie sies h
yiele impor igs or heir progrms.
Identiy Well-matched Control Groups or
Quasi-experimental Studies
I esigig he elio or Loisis alge-
br I Olie, progrm leers i he Loisi
deprme o Ecio we o ress he
boom lie hey kew policymkers cre bomos: wheher ses i he progrms olie
corses were perormig s well s ses
syig lgebr i riiol clssroom.
to impleme he qsi-experimel esig o
heir elio (see Glossry o Commo El-
io terms, p. 65), he progrms miisrors
eee o ieiy riiol lgebr clssrooms
o se s corols. the ie ws o ieiy s-
r lgebr corses serig ses similr o
hose pricipig i he algebr I Olie pro-
grm he gie pre- pos-corse ess o
boh grops o compre how mch ech grop
lere, o erge. O corse, i esig sch
s his, mber o cors besies he clss
orm (olie or ce-o-ce) col ec s-
es perormce: se-leel cors, sch s
iiil biliy home eirome; ech-
er-leel cors, sch s experiece skill i
echig lgebr; school-leel cors, sch
s o-o-clss cemic sppors or ses.
Clerly, i ws impor o he elio o
he closes mches possible.
uer igh imelie, progrm miisrors
worke wih ll heir pricipig isrics o
ieiy riiol clssrooms h were mche
emogrphiclly o he olie clsses , he,
8/14/2019 Evaluating Online Learning
38/80
28
Evaluating Online Learning: Challenges and Strategies or SuccessInnOvatIOnS In EduCatIOn
miisere prees o geerl mhemics
biliy o ses i boh ses o clsses. Exerl
elors rom he Ecio deelopme Ce-
er (EdC) joie he eor his poi. alhogh
impresse by he work h he progrm s h
ccomplishe gie ime bge cosris,
he EdC elors were cocere bo he
qliy o mches bewee he reme
corol clsses. Fiig goo mches is i-
cl sk er he bes o circmsces , ihis cse, i proe ee more icl or o-
elors, h is, progrm school miis-
rors. the qliy o he mches ws especilly
problemic i smll isrics or opblic schools
h h ewer corol clssrooms rom which o
choose. EdC elor Rebecc Crey sys, to
heir crei, [he progrms miisrors] i he
bes hey col i he mo o ime hey h.
Sill, she s, he mches were ecessrily
h gre cross he corol he experime-
l. H we bee iole rom he begiig,
we migh he bee lile bi more srige
bo how he corol schools wol mch o
he iereio schools , mybe, he me
he selecio process lile bi more rigoros.
ulimely, he elio em se ses
prees scores o gge wheher he reme
corol grop ses sre he corse
wih comprble skills kowlege em-
ploye ce sisicl echiqes o help
corol or some o he poor mches.
I heir repor, he elors lso proie
o iereces bewee he corol reme
Table 2. Evaluation Design Characteristics
Design Characteristics Advantages Disadvantages
Experimental
design
Incorporates random assignment o participants
to treatment and control groups. The purpose o
randomization is to ensure that all possible ex-
planations or changes (measured and unmea-
surable) in outcomes are taken into account,
randomly distributing partic