of 53
7/31/2019 Interactive Learning Online At Public Universities
1/53
May 22, 2012
Interactive Learning Online
at Public Universities:Evidence from
Randomized TrialsWilliam G. Bowen
Matthew M. Chingos
Kelly A. Lack
Thomas I. Nygren
7/31/2019 Interactive Learning Online At Public Universities
2/53
Interactive Learning Online at Public Universities: Evidence from Randomized Trials May 22, 2012 1
Ihaka S+R is a sraegic consuling and research service provided by IHAK,a no-or-pro organizaion dedicaed o helping he academic communiy usedigial echnologies o preserve he scholarly record and o advance research andeaching in susainable ways. Ihaka S+R ocuses on he ransormaion o schol-arship and eaching in an online environmen, wih he goal o ideniying hecriical issues acing our communiy and acing as a caalys or change. JSOR,a research and learning plaorm, and Porico, a digial preservaion service, arealso par o IHAK.
Copyrigh 2012 IHA K. Tis work is licensed under he Creaive CommonsAtribuion No Derivaive Works 3.0 Unied Saes License. o view a copy ohe license, please see htp://creaivecommons.org/licenses/by-nd/3.0/us
http://creativecommons.org/licenses/by-nd/3.0/ushttp://creativecommons.org/licenses/by-nd/3.0/us7/31/2019 Interactive Learning Online At Public Universities
3/53
Interactive Learning Online at Public Universities: Evidence from Randomized Trials May 22, 2012 2
Preface
Higher educaion is acing serious challenges in he Unied Saes. Tere isincreasing concern abou rising coss, he qualiy o educaion, and ha henaion is losing is compeiive edge. Online learningspecically highlyineracive, closed-loop, online learning sysems ha we call ILO or IneraciveLearning Onlineholds he promise o broadening access o higher educaiono more individuals, while also lowering coss or sudens. Bu is he qualiyhere?
In our rs repor in his area, Barriers o Adopion o Online Learning Sysemsin U. S. Higher Educaion, we highlighed a broad, widely held concern abouhe qualiy o learning oucomes achieved hrough online learning. Bu do weacually know how ineracive online learning sysems really compare o hein-classroom experience? Tis second repor was designed o help nd answers.
We used a sricly quaniaive mehodology o compare he wo learning approachesin a rigorous way. In six dieren public insiuions, we arranged or he sameinroducory saisics course o be augh. In each insance, a conrol group wasenrolled in a radiional classroom-based course; hen, a reamen group ook ahybrid course using a prooype machine-guided mode o insrucion developed aCarnegie Mellon Universiy in concer wih one ace-o-ace meeing each week.Sudens were assigned o hese wo groups by means o a careully designedrandomizaion mehodology. Te research we conduced was designed o answerhese quesions:
Can sophisicaed, ineracive online courses be used o mainain or improvebasic learning oucomes (masery o course conen, compleion raes, andime-o-degree) in inroducory courses in basic subjecs such as saisics?
Are hese courses as eecive, or possibly more eecive, or minoriy andlow-socioeconomic-saus sudens and or oher groups subjec o sereoype
hrea? Or, are hese groups less well suied o an online approach?
Are such courses equally eecive wih no-so-well-prepared sudens andwell-prepared sudens?
Te resuls o his sudy are remarkable; hey show comparable learning oucomes orhis basic course, wih a promise o cos savings and produciviy gains over ime.
More research is needed. Even hough he analysis was rigorous, i was a singlecourse. We need o learn more abou he adapabiliy o exising plaorms oroering oher courses in dieren environmens. Ihaka S+R is commited oconinuing his research and shar ing our ndings broadly.
We look orward o coninuing o engage wih al l hose who care abou highereducaion o help deliver on he poenial ha new echnologies provide.
DEANNA MARCUM
Managing Director, Ithaka S+R
mailto:Deanna.Marcum%40ithaka.org?subject=mailto:Deanna.Marcum%40ithaka.org?subject=7/31/2019 Interactive Learning Online At Public Universities
4/53
Interactive Learning Online at Public Universities: Evidence from Randomized Trials May 22, 2012 3
Interactive Learning Online at Public Universities:
Evidence from Randomized Trials
2 Preface
4 Introduction
9 Educational Outcomes in Public Universities
23 Costs and Potential Savings
26 Summary Observations
29 Acknowledgements
30 Appendices
7/31/2019 Interactive Learning Online At Public Universities
5/53
Interactive Learning Online at Public Universities: Evidence from Randomized Trials May 22, 2012 4
Interactive Learning Online at
Public Universities: Evidence fromRandomized TrialsWilliam G. Bowen, Matthew M. Chingos, Kelly A. Lack, and Thomas I. Nygren1
May 22, 2012
Introduction
Te opic o online learning in higher educaion is o obvious imporance. Teserious economic and social problems acing he U.S.high unemploymen,slow growh, and severe inequaliiesare relaed, many believe, o ailures o he
1 The authors are all associated with Ithaka S+R (the Strateg y and Research arm of ITHAKA), which sponsored
this study. Bowen is a senior advisor to Ithaka S+R, Chingos is a senior research consultant at Ithaka S+R
and a fellow at the Brookings Institut ion's Brown Center on Education Policy, Lack is a research analyst,
and Nygren is a project director and senior business analyst for Ithaka S+R. The authors wish to thank the
foundat ions that suppor ted t his work: t he Carnegie Corporation of New York, t he William and Flora Hewlett
Foundation, the Spencer Foundation, and a fourth foundation that has asked to remain anonymous. We also
thank our colleagues at ITHAK Aand Larry Bacow, Johanna Brownell, Jackie Ewenstein, and Kevin Guthrie inparticularfor their generous help all along the way. But most of all, we wish to thank our faithful friends on
the participating campuses for their hard work and patience with us; their names are appended to this report.
A number of these individuals (as well as others) have commented on a draft of the report, but the authors
are, of course, fully responsible for the views expressed here and for any errors that remain.
Ithaka S+R has sponsored three s tudies of online learning, of which this is the longest lasting. The t wo other
studies are now available on the Ithaka S+R website. See Barriers to Adoption of Online Learning Systems
in U.S. Higher Education by Lawrence S. Bacow, William G. Bowen, Kevin M. Guthrie, Kelly A. Lack, and
Matthew P. Long, and Current Status of Research on Online Learning in Postsecondary Education by William
G. Bowen and Kelly A. Lack (both available online at htt p://www.sr.ithaka.org/).
7/31/2019 Interactive Learning Online At Public Universities
6/53
Interactive Learning Online at Public Universities: Evidence from Randomized Trials May 22, 2012 5
U.S. educaion sysem, including higher educaion.2 Levels o educaionalatainmen in his counry have been sagnan or almos hree decades, whilemany oher counries have been making grea progress in educaing largernumbers o heir ciizens. Tere is growing concern ha he U.S. is losing iscompeiive edge in an increasingly knowledge-driven world. Also, subsanialachievemen gaps relaed o race and socioeconomic saus persis and have agrea deal o do wih worr ying inequiies. Moreover, here are good reasons o
believe ha hese wo problems are closely relaed.3
The Cost Squeeze in Higher Education
A he same ime, higher educaion, especially in he public secor, is increasinglyshor o resources. Saes coninue o cu back appropriaions in he ace o scalconsrains and pressures o spend more on oher hings, such as healh careand reiremen expenses.4 Caliornia is a dramaic case in poin. Lack o undinghas caused Caliornia colleges and universiies o reduce he size o heir ener-ing classes a he very ime when increasing numbers o sudens are seekingo enroll.5 Higher uiion revenues migh be an escape valve, bu here is greaconcern abou uiion levels and increasing resenmen among sudens andheir amilies ha is having poliical reverberaions. Presiden Obama, in his
2 The authors agree that there is an important connection between educational outcomes and the economic
performance of a country. But we would warn against exaggerating the power of the connection. In the case
of the U.S., for example, the recent recession and the slow rate of growth seen in the last few years surely
owe more to the 2008 financial excesses than they do to deficiencies in the countrys higher education
system. As Jacob Weisberg pointed out in Newsweek in 2010 with respect to the recent recession, there are
no strong candidates for a single factor that would have caused the crisis in the absence of any others
(Weisbergs piece can be f ound online at http://www.thedailybeast.com/newsweek/2010/01/08/what-
caused-the-great-recession.html).
3 SeeEquity and Excellence in American Higher Education by William G. Bowen, Martin A. Kurzweil, and
Eugene M. Tobin (2005) for an ext ended discussion of the historical record and of t he likely connections,
going forward, between achievement gaps and overall levels of educational attainment. See also David
Leonhardts October 8, 2011 column in the New York Times, The Depression: If Only Things Were That Good,
in which he argues that the U.S. is worse off today t han it was in the 1930s because innovation is lagging
which he attributes in no small part to deficiencies in education (http://www.nytimes.com/2011/10/09/sun-
day-review/the-depression-if-only-things-were-that-good.html?_r=1&pagewanted=all.) Of course, lagging
rates of educational at tainment have their origins in low high school graduation rates. See Henr y M. Levin
and Cecilia E. Rouse, The True Cost of High School Dropout, New York Times, January 25, 2012. (http://
www.nytimes.com/2012/01/26/opinion/the-true-cost-of-high-school-dropouts.html). But these problems
are then compounded by low completion rates among those who both graduate from high school and enter
college; see Crossing the Finish Line: Completing College at Americas Public Universities (2009) by William
G. Bowen, Matthew M. Chingos, and Michael S. McPherson.
4 A report released in spring 2012 by the State Higher Education Executive Officers, entitled State Higher
Education Finance FY 2011 (http://www.sheeo.org/finance/shef/SHEF_FY2011-EARLY_RELEASE.pdf), docu-
ments the dire economic circumstances of many public institutions.
5 In November 2008, California State University became the first public university to limit enrollment when,
despite a 20% increase in applications from prospective first-year students, it decided to reduce its student
body by 10,000 students, following a $200 million decrease in tax revenue that academic year coupled with
an additional $66 million cut (see Under Financial Stress, More Colleges Cap Enrollments (November
26, 2008) in TIME, http://www.time.com/time/nation/article/0,8599,1861861,00.html). The University of
California and California Community College systems have since followed suit in the face of limited funding
available from the state (see the August 5, 2009 article Budget cuts devasta te California higher educa-
tion in The Washington E xaminer, http://washingtonexaminer.com/science-and-technology/2009/08/
budget-cuts-devastate-california-higher-education).
Levels of educational attainment in
this country have been stagnant for
almost three decades, while many
other countries have been making
great progress in educating larger
numbers of their citizens.
http://www.thedailybeast.com/newsweek/2010/01/08/what-caused-the-great-recession.htmlhttp://www.thedailybeast.com/newsweek/2010/01/08/what-caused-the-great-recession.htmlhttp://www.nytimes.com/2011/10/09/sunday-review/the-depression-if-only-things-were-that-good.html?_r=1&pagewanted=allhttp://www.nytimes.com/2011/10/09/sunday-review/the-depression-if-only-things-were-that-good.html?_r=1&pagewanted=allhttp://www.nytimes.com/2012/01/26/opinion/the-true-cost-of-high-school-dropouts.htmlhttp://www.nytimes.com/2012/01/26/opinion/the-true-cost-of-high-school-dropouts.htmlhttp://www.sheeo.org/finance/shef/SHEF_FY2011-EARLY_RELEASE.pdfhttp://www.time.com/time/nation/article/0,8599,1861861,00.htmlhttp://washingtonexaminer.com/science-and-technology/2009/08/budget-cuts-devastate-california-higher-educationhttp://washingtonexaminer.com/science-and-technology/2009/08/budget-cuts-devastate-california-higher-educationhttp://washingtonexaminer.com/science-and-technology/2009/08/budget-cuts-devastate-california-higher-educationhttp://washingtonexaminer.com/science-and-technology/2009/08/budget-cuts-devastate-california-higher-educationhttp://www.time.com/time/nation/article/0,8599,1861861,00.htmlhttp://www.sheeo.org/finance/shef/SHEF_FY2011-EARLY_RELEASE.pdfhttp://www.nytimes.com/2012/01/26/opinion/the-true-cost-of-high-school-dropouts.htmlhttp://www.nytimes.com/2012/01/26/opinion/the-true-cost-of-high-school-dropouts.htmlhttp://www.nytimes.com/2011/10/09/sunday-review/the-depression-if-only-things-were-that-good.html?_r=1&pagewanted=allhttp://www.nytimes.com/2011/10/09/sunday-review/the-depression-if-only-things-were-that-good.html?_r=1&pagewanted=allhttp://www.thedailybeast.com/newsweek/2010/01/08/what-caused-the-great-recession.htmlhttp://www.thedailybeast.com/newsweek/2010/01/08/what-caused-the-great-recession.html7/31/2019 Interactive Learning Online At Public Universities
7/53
Interactive Learning Online at Public Universities: Evidence from Randomized Trials May 22, 2012 6
2012 Sae o he Union address and in subsequen speeches, has decried risinguiions, called upon colleges and universiies o conrol coss, and proposed o
wihhold access o some Federal programs or colleges and universiies ha didno address aordabiliy issues or mee compleion ess.6
oday, a variey o higher educaion insiuions mus conron he challengeo how o manage coss in he ace o igher unding. Whi le he proporion oeducaion spending drawn rom uiion revenues rose across all insiuions,
increases in uiion oen oupaced increases in educaion and relaed spending(i.e. spending on insrucion, suden services, and some suppor and maine-nance coss relaed o hese uncions), calling ino quesion he susainabiliyo he curren unding model.7 Moreover, he rs sur vey o provoss and chieacademic ocers byInside Higher Ed ound ha on he quesion o ins iuionaleeciveness in conrolling coss, over 15 percen o all provoss gave heirinsiuions marks o 1 or 2 on eeciveness [on a scale rom 1 o 7, wih 7 being
very eecive].8 I is equally noeworhy ha very ew chie academic ocers(and especially hose a boh public and privae docoral universiies) gave heirinsiuions high marks on his meric. Recogniion o he problem is widespread;soluions have been hard o come by.
A undamenal source o he problem is he cos disease, based on he handicranaure o educaion wih is atendan lack o opporuniies or gains in produc-iviy, which one o he auhors o his repor [Bowen] promulgaed in he 1960s,in collaboraion wih William J. Baumol. Bu he ime may (nally!) be a hand
when advances in inormaion echnology will permi, under he righ circum-sances, increases in produciviy ha can be ranslaed ino reducions in he
6 See Remarks by the President in State of the Union Address, January 24, 2012 (transcript available at (http://
www.whitehouse.gov/the-press-office/2012/01/24/remarks-president-state-union-address). Three days later,
Obama spoke about college af fordability at the University of Michigan (transcript available at http://www.
whitehouse.gov/the-press-office/2012/01/27/remarks-president-college-affordability-ann-arbor-michigan).
This speech does not, however, contain more details concerning how affordability is to be measured or
what penalties are to be imposed on those who fail to pass the requisite tests. As Molly Broad, president of
the American Council on Education, said after the speech: The devil is in the [unspecified] details (Mixed
Reviews of Obama Plan to Keep Down College Costs, January 28, 2012, New York Times, http://www.nytimes.
com/2012/01/28/education/obamas-plan-to-control-college-costs-gets-mixed-reviews.html).7 According to the College Boards 2011 Trends in College Pricing Report (http://trends.collegeboard.org/
downloads/College_Pricing_2011.pdf), tuition at public two-year universities increased, on average, by
8.7% relative to the 2010-2011 academic year, and tuition at public four-year institutions for the 2011-2012
academic year increased, on average, by 8.3% for instate students and by 5.7% for out of state student s.
In keeping with the trend over the previous four years, students attending private institutions experienced
smaller percentage increases (4.5% for private not-for-profit four-year institutions and 3.2% for private for-
profit institutions).
8 See Scott Jaschik, Mixed Grades: A Survey of Provosts, Inside Higher Education, January 25, 2012, http://
www.insidehighered.com/news/survey/mixed-grades-survey-provosts.
Higher education, especially in the
public sector, is increasingly short
of resources. States continue to
cut back appropriations in the
face of scal constraints and
pressures to spend more on
other things, such as healthcare and retirement expenses.
http://www.whitehouse.gov/the-press-office/2012/01/24/remarks-president-state-union-addresshttp://www.whitehouse.gov/the-press-office/2012/01/24/remarks-president-state-union-addresshttp://www.whitehouse.gov/the-press-office/2012/01/27/remarks-president-college-affordability-ann-arbor-michiganhttp://www.whitehouse.gov/the-press-office/2012/01/27/remarks-president-college-affordability-ann-arbor-michiganhttp://www.nytimes.com/2012/01/28/education/obamas-plan-to-control-college-costs-gets-mixed-reviews.htmlhttp://www.nytimes.com/2012/01/28/education/obamas-plan-to-control-college-costs-gets-mixed-reviews.htmlhttp://trends.collegeboard.org/downloads/College_Pricing_2011.pdfhttp://trends.collegeboard.org/downloads/College_Pricing_2011.pdfhttp://www.insidehighered.com/news/survey/mixed-grades-survey-provostshttp://www.insidehighered.com/news/survey/mixed-grades-survey-provostshttp://www.insidehighered.com/news/survey/mixed-grades-survey-provostshttp://www.insidehighered.com/news/survey/mixed-grades-survey-provostshttp://trends.collegeboard.org/downloads/College_Pricing_2011.pdfhttp://trends.collegeboard.org/downloads/College_Pricing_2011.pdfhttp://www.nytimes.com/2012/01/28/education/obamas-plan-to-control-college-costs-gets-mixed-reviews.htmlhttp://www.nytimes.com/2012/01/28/education/obamas-plan-to-control-college-costs-gets-mixed-reviews.htmlhttp://www.whitehouse.gov/the-press-office/2012/01/27/remarks-president-college-affordability-ann-arbor-michiganhttp://www.whitehouse.gov/the-press-office/2012/01/27/remarks-president-college-affordability-ann-arbor-michiganhttp://www.whitehouse.gov/the-press-office/2012/01/24/remarks-president-state-union-addresshttp://www.whitehouse.gov/the-press-office/2012/01/24/remarks-president-state-union-address7/31/2019 Interactive Learning Online At Public Universities
8/53
Interactive Learning Online at Public Universities: Evidence from Randomized Trials May 22, 2012 7
cos o insrucion.9 Greaerand smareruse o echnology in eaching iswidely seen as a promising way o conroll ing coss while also reducing achieve-men gaps and improving access. Te exploding growh in online learning isoen cied as ev idence ha, a las, echnology may oer pahways o progress.10Online learning is seen by a growing number o people as a way o breakingree o cenury-old rigidiies in educaional sysems ha we have inheried. Temuch-discussed book on disrupive echnologies and universiies by ClayonChrisensen and Henry Eyring is perhaps he bes example o he atenion beinggiven o online echnologies as a way o changing prooundly he way we educaesudens.11
Tere are, however, also concerns ha a leas some kinds o online learning arelow qualiy and ha onl ine learning in general de-personalizes educaion.In thisregard, it is crit ically important to recognize issues o nomenclature: online learningis hardly one thing. I comes in a dizzying variey o favors, ranging rom simply
videoaping lecures and posing hem or any-ime access, o uploading maeri-als such as s yllabi, homework assignmens, and ess o he Inerne, all he wayo highly sophisicaed ineracive learning sysems ha use cogniive uors andake advanage o muliple eedback loops. Te varieies o online learning can be
used o each many kinds o subjecs o dieren populaions in diverse insiuional
9 Bowens co-author in the promulgation of the cost disease, William J. Baumol, has continued to discuss
its relevance not only for education but also for sectors such as the performing arts and heath care. For
the initial statement of this proposition, see William J. Baumol and William G. Bowen,Performing Ar ts: The
Economic Dilemma, Twentieth Century Fund(1968). In essence, the argument is that in fields such as t he
performing arts and education, there is less oppor tunity than in other fields to improve productivity (by, for
example, substituting capital for labor), that unit labor costs will there fore rise inexorably as these sec tors
have to compete for labor with other sectors in which productivity gains are easier to come by, and that the
relative costs of labor-intensive activities such as chamber music and teaching will therefore continue to
rise. As Bowen argued in his Romanes lecture, for a number of years advances in information technology
have in fact increased productivity, but these increases have been enjoyed primarily in the form of more
output (especially in research) and have generally led to higher, not lower, total costs. (For the text of the
Romanes lecture, see William G. Bowen, At a Slight Angle to the Universe: The Universit y in a Digitized,
Commercialized Age, Princeton University Press, 2001; the tex t is also available on the Andrew W. Mellon
Foundation website: http://www.mellon.org/internet/news_publications/publications/romanes.pdf.)
10 A November 2011 report by the Sloan Consort ium and the Babson Survey Research Group shows that be tween
fall 2002 and fall 2010, en rollment s in online courses increased much more quickl y than total enrollment s
in higher education. During this time period, the number of online course enrollments grew from 1.6 million
to 6.1 million, amounting to a compound annual rate of 18.3% (compared with a rate of 2% for course
enrollments in general)although between fall 2009 and fall 2010 online enrollments grew more slowly,
at 10.1%. More than three of every 10 students in higher education now take at least one course online. Inaddition to the growth in what we call online or hybrid courseshowever nebulous that terminology may
bewe also feel the pervasiveness of the Internet in higher education by the increasing use of it in the
form of course management systems or vir tual reading materials /elect ronic textbooks incorporated into the
curriculum. Even courses that are c alled traditional almost always involve some use of digital resources.
11 See Clayton M. Christensen, and Henry J. Eyring, The Innovative University: Changing the DNA of Higher
Education from the Inside Out, San Francisco: Jossey-Bass, 2011. An October 2, 2011 New York Times op-ed
piece by Bill Keller, aptly titled The University of Wherever, is another illustration of the high visibility and
high stakes of the debate over online education (http://www.nytimes.com/2011/10/03/opinion/the-universi-
ty-of-wherever.html?pagewanted=all).
There are also concerns that at
least some kinds of online learning
are low quality and that online
learning in general de-personalizes
education. In this regard, it is
critically important to recognize
issues of nomenclature: onlinelearning is hardly one thing. It
comes in a dizzying variety of
avors.
http://www.mellon.org/internet/news_publications/publications/romanes.pdfhttp://www.nytimes.com/2011/10/03/opinion/the-university-of-wherever.html?pagewanted=allhttp://www.nytimes.com/2011/10/03/opinion/the-university-of-wherever.html?pagewanted=allhttp://www.nytimes.com/2011/10/03/opinion/the-university-of-wherever.html?pagewanted=allhttp://www.nytimes.com/2011/10/03/opinion/the-university-of-wherever.html?pagewanted=allhttp://www.mellon.org/internet/news_publications/publications/romanes.pdf7/31/2019 Interactive Learning Online At Public Universities
9/53
Interactive Learning Online at Public Universities: Evidence from Randomized Trials May 22, 2012 8
setings. A key poin, i an obvious one, is ha here is no one approach ha isrigh or every suden or every seting. In imporan respecs, he online learn-ing markeplace refecs he diversiy o A merican higher educaion isel.12
As resisan as some may si ll be even o hink abou seeking produciv iy gainsin order o reduce eaching coss, here is simply no denying he need o lookmore closely han ever beore a he relaion beween cerain oupus (approxi-maed, or example, by degrees conerred) and inpus (he mix o labor and
capial ha denes educaional producion uncions).13
I is essenial ha helimied resources available o higher educaion be used as eecively as possible.For hese reasons, he research repored here is concerned wih boh educaionaloucomes and coss, seen as wo blades o he scissors.
Organization of This Report
Te nex secion o his repor describes a wo-year eor we have made o esrigorously he learning oucomes achieved by a prooype ineracive learningonline course delivered in a hybrid mode (wih some ace-o-ace insr ucion) onpublic universiy campuses in he Norheas and Mid-Alanic. Beore preseningour ndings, we devoe space o explaining our randomizaion mehodology
boh because he ndings can only be undersood agains he backdrop o he
mehodology and because he research design may be o independen ineres osome readers.14 Tis secionwhich conains he resuls o he main par o ourresearchis ollowed by a brieer discussion o he poenial cos savings hacan conceivably be achieved by he adopion o hybrid-orma online learningsysems. We explain why we avor using a cos simulaion approach o esimaepoenial savings, bu we relegae o Appendix B he highly provisional resuls
we obained by employing one se o assumpions in a cos simulaion model. Weend he main body o he repor wih a shor conclusion ha considers barrierso he adopion o online learning sysems ha are ruly ineracive, seps hamigh be aken o overcome such barriers, and he need o ake a sysem-wideperspecive in addressing hese exremely imporan issues.
12 As Henry Bienen (president emeritus of Northwestern and chairman of the board of Rasmussen College, a
for-profi t university, as well as chairman of ITHAKA) point s out, for many institutions seeking to address the
needs of adult learners and others who are not candidates for places in traditional colleges and universi-
ties, there is no choice: online education, in some form, is the only way that many people can acquire more
skills and earn a college degree, the returns on which have skyrocketed in the past three decades. But online
education is also increasingly common in colleges and universities that educate traditional students. It is
seen as a revenue-generating force in many institutions, both four-year and two-year and both public and
private. See Barr iers to Adoption of Online Learning Systems in U.S. Higher Educat ion by Bacow et al.13 Some argueand we heartily agreethat the output of higher education has broader dimensions and
includes both research results and also the contribution that the entire system of higher education makes
to the effective functioning of a democratic society. But it will not do to allow emphasis on these larger (and
hard-to-measure) contributions to obscure the need to look carefully, and with a somewhat skeptical eye, at how
effectively institutions utilize resources to achieve straightforward aims such as improving graduation rates.
14 Readers interested in methodology may be especially interested in Appendix C to this report, which contains
a detailed discussion of lessons learned from our experience in carrying out this complicated research
project. We wish only that we had had access to this recitation of what to do and what not to do before we
started on this adventure! We learned many of these lessons the hard way.
7/31/2019 Interactive Learning Online At Public Universities
10/53
Interactive Learning Online at Public Universities: Evidence from Randomized Trials May 22, 2012 9
Educational Outcomes in Public Universities
Te rs and mos ambiious par o our research was direced a assessing heeducaional oucomes associaed wih wha we erm ineracive learning onlineor ILO. By ILO we reer o highly sophisicaed, ineracive online coursesin which machine-guided insrucion can subsiue or some (hough no usu-ally all) radiional, ace-o-ace insrucion. Course sysems o his y pe akeadvanage o daa colleced rom large numbers o sudens in order o oer eachsuden cusomized insrucion, as well as allow insrucors o rack sudensprogress in deail so ha hey can provide heir sudens wih more argeed andeecive guidance. As several leaders o higher educaion made clear o us inpreliminary conversaions, absen real ev idence abou learning oucomes here isno possibiliy o persuading mos radiional colleges and universiies, and espe-cially hose regarded as hough leaders, o push hard or he inroducion o ILOechnologies ha begin o subsiue machine-guided insrucion or radiionalorms o eaching in appropriae setings.
We se ou o provide a leas enaive answers o hese quesions:
Can sophisicaed, ineracive online courses be used o mainain or improvebasic learning oucomes (masery o course conen, compleion raes, andime-o-degree)?
Are hese courses as eecive, or possibly more eecive, or minoriy andlow-socioeconomic-saus sudens and or oher groups subjec o sereoypehrea?
Are hey equal ly eecive wih no-so-well-prepared sudens and well-pre-pared sudens?
Are hey equal ly eecive in a var iey o campus setingscommuniy col-leges versus our-year colleges, commuer colleges versus colleges wih more
sudens in residence?
Research Design
In hinking abou research design, we began by looking closely a exisingresearch. Tere have been lierally housands o sudies o online learning, buunorunaely he grea majoriy are decien in one way or a noheroen orreasons beyond he conrol o he principal invesigaors.15 Very ew look direclya he eaching o large inroducory courses in basic elds a major public univer-siies, where he grea majoriy o undergraduae sudens pursue eiher associaeor baccalaureae degrees, presumably because very ew ILO courses have been
15 A detailed summary of existing research has been compiled by our staff (especially Lack); but it is too lengthy
to include here. See Current Sta tus of Research on Online Learning in Postsecondary Education by Bowen
and Lack.
The most ambitious part of our
research was directed at assessing
the educational outcomesassociated with what we term
interactive learning online or
ILO. By ILO we refer to highly
sophisticated, interactive online
courses in which machine-guided
instruction can substitute for some
(though not usually all) traditional,
face-to-face instruction.
7/31/2019 Interactive Learning Online At Public Universities
11/53
Interactive Learning Online at Public Universities: Evidence from Randomized Trials May 22, 2012 10
oered in hese setings.16 Very ew o he sudies use randomized assignmenechniques o creae reamen and conrol groups ha can be used o reduceoherwise ubiquious selecion eecs ha make i hard o inerpre ndings.
o overcome hese limiaions, we decided o work wih seven insances o aprooype ILO saisics course a six public universiy campuses (including woseparae courses in wo deparmens on one campus) ha agreed o cooperae ina careully designed research projec uilizing random assignmen echniques.
wo o hese campuses are par o he Sae Universiy o New York (SUNY);wo are par o he Universiy o Maryland; and wo are par o he Ciy Univer-siy o New York (CUNY). Te individual campuses involved in his sudy were,rom SUNY, he Universiy a A lbany and SUNY Insiue o echnology; romhe Universiy o Maryland, he Universiy o Maryland, Balimore Couny andowson Universiy; and, rom CUNY, Baruch College and Ciy College. Teseven courses, wih heir all 2011 enrollmens, are shown in able 1.
We also atemped o include hree communiy colleges in New York andMaryland. We were ulimaely unable o include daa rom hese campuses inour sudy or several reasons. A one o he hree communiy colleges, muliplechanges in leadership compromised he implemenaion o he randomizedresearch proocol. A he second communiy college, a large number o sudyparicipans never ook he course, and among hose who did, almos a quar-er swiched ino a orma dieren rom he one o which hey were randomlyassigned. Addiionally, daa on nal exam and sandardized es scores wereunavailable or a subsanial proporion o his campus sudy paricipans. A hehird communiy college, much o he daa were provided oo lae o incorporaeino our primary analysis. We srongly cauion readers agains assuming ha hendings repored here or our-year colleges necessarily apply o communiy col-leges. Vigorous eors nowihsanding, we were unable o obain hard evidenceon his key quesion.
16 Our focus on students attending public institutions is not meant to denigrate the importance of either the
private non-profit sector or t he for-profit sector. Nor is it meant to denigrate professional programs aimed
at working adults. But it is t he public colleges and universities, which educate more than three- quarters of
undergraduates at degree-g ranting institutions (according to the College Boards 2011 report, cited above),
that face the most consequential challenges in raising attainment rates and closing achievement gaps while
simultaneously reducing costs and rest raining tuition increases.
7/31/2019 Interactive Learning Online At Public Universities
12/53
Interactive Learning Online at Public Universities: Evidence from Randomized Trials May 22, 2012 11
TABLE 1. PARTICIPATING COURSES/INSTITUTIONS, FALL 2011
Course Enrollment Study Participants
Institution A 850 97
Institution B 877 229
Institution C 235 92
Institution D 86 16
Institution E, Department 1 337 31
Institution E, Department 2 473 50
Institution F 188 90
Total 3,046 605
Notes: Study participants are students who consented to be in our study and were randomly assigned to a traditional or
hybrid format of the introductory statistics class.
We do no claim ha hese six campuses are a sais ical ly valid sample o evenpublic higher educaion, never mind all o higher educaion. Bu his se o si xdoes include: (a) major urban universiies wih large commuing populaionso sudens, as well as universiies wih more residenial sudens; and (b) largenumbers o minoriy sudens and sudens rom low-socioeconomic-saus
amilies (as shown in ables 2 and 3). Tus, he populaion o insiuions andsudens in he sudy is boh large enough and diverse enough o allow us oexplore mos o he quesions lised above in he conex o our-year publicaioninsiuions.
More specically, his research was designed o es as rigorously as possiblehe learning eeciveness o a paricular ineracive saisics course developeda Carnegie Mellon Universiy (CMU)viewed as a prooype o oher ILO
The population of institutions
and students in the study is both
large enough and diverse enoughto allow us to explore most of
the questions listed above in the
context of four-year publication
institutions.
7/31/2019 Interactive Learning Online At Public Universities
13/53
Interactive Learning Online at Public Universities: Evidence from Randomized Trials May 22, 2012 12
courses.17 While he CMU course can be delivered in a ully online environmen,in his sudy i was used in a hybrid mode in which mos o he insrucion wasdelivered hrough he ineracive online maerials, bu he online insrucion
was supplemened by a one-hour-per-week ace-o-ace session in which sudenscould ask quesions or be given a rgeed assisance.
Te exac research proocol varied by campus in accordance wih local policies,pracices, and preerences, and we describe hese proocols in deail in Appendix
able A1, and on Ihaka S+Rs websie where here is a narraive descripion;Appendix able A1 also presens summar y daa on enrollmens and secion sizesin each orma (oen he hybrid-orma secions were somewha smaller han heradiional-orma secions). Te general procedure ollowed was: 1) a or beorehe beginning o he semeser, sudens regisered or he inroducory sais-ics course were asked o paricipae in our sudy, and modes incenives wereoered;18 2) sudens who consened o paricipae lled ou a baseline survey;3) sudy paricipans were randomly assigned o ake he class in a radiional orhybrid orma; 4) sudy paricipans were asked o ake he CAOS es o saisi-cal lieracy19 a he beginning o he semeser; and 5) a he end o he semeser,
17 We prefer the ILO acronym to others, including the OLI acronym used by CMU to stand for Open LearningInitiative. The term ILOfor interactive learning onlineis not specific to CMUs suite of courses, and
ILO emphasizes the interactive fea tures of this kind of online learning. This is in contrast with more com-
mon types of online learning which largely mimic classroom teaching without taking advantage of the unique
online environment to provide added value, that is, anything beyond that which can be achieved in a physi-
cal classroom.
The CMU statistics course (which can be accessed at http://oli.web.cmu.edu/openlearning/) includes textual
explanations of concepts and an inventory o f worked examples and practice problems, some of which require
the students to manipulate data for themselves using a statistical sof tware package. Both the st atistics
course and other courses in the OLI suite were originally intended to be comprehensive enough to allow stu-
dents to learn the material independently without the guidance of an instructor; since it was developed, how-
ever, the statistics course has been used at a variet y of higher education institutions, sometimes in a hybrid
mode. (Taylor Walsh describes the history of the development of this course, which was financed largely by
the Hewlett Foundation over a number of years, in her 2010 book Unlocking the Gates: How and Why Leading
Universities Are Opening Up Access to Their Courses, Princeton University Press, 2010.) Among the main
strengths of the CMU sta tistics course is its ability to embed interac tive assessments into each instructional
activity, and its three key feedback loops: system to student, as the student answers questions; system to
teacher, to inform student-instructor interactions; and system to course developer, to identif y aspects of the
course that can be improved. In addition to offering assessments to measure how well students understand a
particular concept, the CMU course also asks students to complete self-assessments, to give the instruct or
and/or learning scientists a sense of how well students think they understand the concept. However, while
instructors can delete and re-order modules, CMU does not offer much opportunity for customization, nor is
the course adaptive in terms of redirecting students to ex tra practice sessions or additional reading if their
incorrect answers indicate that they do not understand a concept and need more help. Thus, although the
CMU statistics course is cert ainly impressive, we refer to it as a prototype because we believe it is an early
representative of what will likely be a wave of even more sophisticated systems in the not-too-distant future.18 See Appendix A for a description of the research protocol and incentives used on each campus.
19 The CAOS test, or Comprehensive Assessment of Outcomes in Statistics, is a 40-item multiple-choice
assessment designed to measure students statist ical literacy and reasoning skills. One characteristic of
the CAOS test is that (for a variet y of reasons) scores do not increase by a large amount over the course
of the semester. Among students in our study who took the CAOS test at both the beginning and end of the
semester, the average score increase was 5 percentage points. For more information about the CAOS tes t,
see https://app.gen.umn.edu/artist/caos.html, or delMas, Robert, Joan Garfield, Ann Ooms, and Beth Chance,
Assessing Students Conceptual Understanding After a First Course in Statistics, 6.2 (2007): 28-58,
accessed July 28, 2010, http://www.stat.auckland.ac.nz/~iase/serj/SERJ6(2)_delMas.pdf.
http://oli.web.cmu.edu/openlearning/https://app.gen.umn.edu/artist/caos.htmlhttp://www.stat.auckland.ac.nz/~iase/serj/SERJ6%282%29_delMas.pdfhttp://www.stat.auckland.ac.nz/~iase/serj/SERJ6%282%29_delMas.pdfhttps://app.gen.umn.edu/artist/caos.htmlhttp://oli.web.cmu.edu/openlearning/7/31/2019 Interactive Learning Online At Public Universities
14/53
Interactive Learning Online at Public Universities: Evidence from Randomized Trials May 22, 2012 13
sudy paricipans were asked o ake he CAOS es o saisical lieracy again,as well as complee anoher quesionnaire. Appendix able A2 provides henumbers o sudens on each campus who were randomized ino each ormaand who compleed each daa collecion insrumen.
Adminisraive daa on paricipaing and non-paricipaing sudens were gah-ered rom he paricipaing ins iuions daabases. Te baseline survey admin-isered o sudens included quesions on sudens background characerisics,
such as socioeconomic saus, as well as heir prior ex posure o saisics andhe reason or heir ineres in possibly aking he saisics course in a hybridorma. Te end-o-semeser survey asked quesions abou heir experiences inhe saisics course. Sudens in sudy-aliaed secions o he saisics courseook a nal exam ha included a se o iems ha were idenical across al l heparicipaing secions a ha campus (or, in he case o he campus ha had wodeparmens paricipaing in he sudy, all par icipaing secions in ha depar-men). Te scores o sudy paricipans on his common porion o he exam
were provided o he research eam, along wih background adminisraive daaand nal course grades o all sudens (boh paricipans and, or comparisonpurposes, nonparicipans) enrolled in he saisics course in he all 2011
semeser. All o hese daa are described in deail on he Ihaka S+R websie,which also includes copies o he survey insrumens.
Our inenion was o provide a rigorous side-by-side comparison o speciclearning oucomes or sudens in his hybrid version o he saisics course andcomparable sudens in a radiionally-augh version o he same course. Werecognize, however, ha while we were reasonably successul in randomizingsudens beween reamen and conrol groups (see documenaion in he nexsecion o his repor), we could no randomize insrucors in eiher group andhus could no conrol or dierences in eacher qualiy.20 Tis is one reason,among ohers, ha we do no regard he research design o his projec as
20 Instructor surveys reveal that, on average, the instructors in traditional format sections were much more
experienced than their counterparts teaching hybrid-format sections (median years of teaching experience
was 20 and 5, respectively). Moreover, almost all of the instructors in the hybrid-format sect ions were using
the CMU online course for either the first or second time, whereas many of the instructors in the traditional-
format sect ions had t aught in this mode for years. The experience-advantage, therefore, is clearly in favor
of the teachers of the traditional-format sections. The questionnaires also revealed that a number of the
instructors in hybrid-format sections began with negative perceptions of online learning. In part for these
reasons, a leader of one of the sets of institutions in this study believes that results for the hybrid-format
sections would be improved vis--vis results in the traditional-format sections if the exper iment were
repeated and instructors in the hybrid-format sections were bet ter motivated and bet ter trained. But this is,
of course, a conjecture.
Our intention was to provide a
rigorous side-by-side comparison
of specic learning outcomesfor students in this hybrid
version of the statistics course
and comparable students in a
traditionally-taught version of the
same course. However, while we
were reasonably successful in
randomizing students between
treatment and control groups, we
could not randomize instructorsin either group and thus could not
control for differences in teacher
quality.
7/31/2019 Interactive Learning Online At Public Universities
15/53
Interactive Learning Online at Public Universities: Evidence from Randomized Trials May 22, 2012 14
anyhing close o perec.21 Sill, his is he rs eor o which we are awareo carr y ou he kind o randomized sudy o oucomes in large inroducorycourses on public universiy campuses ha we hink has been needed.
One wise decision we made was o conduc spring-erm pilos on as many cam-puses as possible in advance o he al l-erm 2011 research phase o he sudy
when we reaed oucomes as suiable or measuremen. Te spring-erm pilosidenied a number o pracical aspecs in which he sudy could be improved,
and a memo on lessons learned rom he spring-erm pilos is included in hisrepor as Appendix C.22
I remains only o add ha, as Appendix C illusraes, his is very diculresearch o do, in large par because so many deailshow bes o presenhe course, o recrui suden and aculy paricipans, o randomize sudens
beween reamen and conrol groups, o collec good daa including back-ground inormaion abou he suden paricipans, and o saisy InsiuionalReview Board requiremens in a imely wayneed o be worked ou wih heday-o-day involvemen o campus sa no direcly responsible o us. We havegrea respec or oher invesigaors who have coped wih hese problems, oenin setings ouside higher educaion.
Findings
Te grea advanage oindeed, he main moivaion orconducing a ran-domized experimen is ha sudens in he reamen and conrol groups areexpeced o have he same average characerisics, boh observed and unob-served. Te resuls in able 2 indicae ha he randomizaion worked properly inha radiional and hybrid-orma sudens in ac have similar characerisics.Tere are a handul o small dierences ha are saisically signican bu, ingeneral, he dierences beween sudens augh in he radiional orma andsudens augh in he hybrid orma are no meaningul.23
21 Randomization procedures were limited by the fact that Institutional Review Board (IRB) requirements
precluded randomization of students enrolled in the course without their consent. Instead, we had first to
use incentives to encourage students to participate in the study, with the understanding that they would then
be randomized between trea tment and control groups. We were able, however, to compare the character-
istics of participants and non-participants, and the two groups turned out to be very similar; see Table 3.
The study is, o f course, limited in that it involves only a single course, but having a common hybrid course
across the six campuses (i.e. the CMU statistics course) controls for one source of variance in outcomes.
We deliberately chose the CMU statistics course because we think that the greatest near-term opportunity
to take advantage of interact ive online technologies is in introductory-level courses that serve large student
populations in fields in which there is more or less one right answer to most questions. Somewhat dif ferent
pedagogies would be needed, we suspect, in courses that are more value-laden and dependent on discussion
of various perspectives.22 We are indebted to James Kemple, now Executive Director of the Research Alliance for New York City Public
Schools, and formerly the Director of the K-12 Education Policy division at MDRC, for much useful advice. Dr.
Kemple has long experience with randomized t rials. Lessons learned from the pilots included how to present
the project, the ef fective use of modest incentives for par ticipants, and techniques that could improve ran-
domization. We hope that others will benefit from our experience (see Appendix C) in mounting this research
project.
23 A regression of format assignment on all of the variables listed in Table 2 (and institution dummies) fails to
reject the null hypothesis of zero coefficients for all variables (except the institution dummies) with =0.12.
A Hotelling test fails to rejec t the null of no difference in means with =0.27.
7/31/2019 Interactive Learning Online At Public Universities
16/53
Interactive Learning Online at Public Universities: Evidence from Randomized Trials May 22, 2012 15
In addiion o esing he success o our eors o randomize sudens, able 2also serves o describe he populaion o sudens who paricipaed in our sudy.Tey are a very diverse group. Hal o he sudens come rom amilies wihincomes less han $50,000 and hal are rs-generaion college sudens. Fewerhan hal are whie, and he group is abou evenly divided beween sudens wihcollege GPAs above and below 3.0. Mos sudens are o radiional college-goingage (younger han 24), are enrolled ull-ime, and are in heir sophomore or
junior year.
Tese sudens are a diverse group, bu do hey resemble he enire populaiono sudens enrolled in he inroducory sais ics courses included in our sudy?Sudy paricipans were randomly assigned o a secion orma, bu he sudyparicipans hemselves are a sel-seleced populaionbecause o InsiuionalReview Board requiremens only sudens who agreed o be in he sudy wererandomly assigned, and scheduling complicaions also l imied he populaiono paricipans. Overall, 605 o he 3,046 sudens enrolled in hese saisicscourses paricipaed in he sudy. An even larger sa mple size would have beendesirable, bu he logisical challenges o scheduling a leas wo secions (onehybrid secion and one radiional secion) a he same ime, so as o enable
sudens in he sudy o atend he saisics course regardless o heir (random-ized) orma assignmen, resriced our prospecive paricipan pool o helimied number o paired ime slos available. Also, as already noed, Insi-uional Review Boards required suden consen in order or researchers orandomly assign hem o he radiional or hybrid orma. No sur prisingly, somesudens who were able o make he paired ime slos eleced no o paricipaein he sudy. All o hese complicaions nowihsanding, our nal sample o 605sudens is by no means smalli is in ac quie large in he conex o his ypeo research.24
24 Of the 46 studies examined in the Means et al. (2009) meta-analysis, only 5 had sample sizes of over 400,
and of the 51 independent effect sizes the authors abstract ed, 32 came from studies with fewer than 100
study participants.
The students who participated
in our study are a very diverse
group. Half of the students come
from families with incomes less
than $50,000, and half are rst-
generation college students. Fewer
than half are white.
7/31/2019 Interactive Learning Online At Public Universities
17/53
Interactive Learning Online at Public Universities: Evidence from Randomized Trials May 22, 2012 16
TABLE 2. RANDOMIZATION OF STUDY PARTICIPANTS
Traditional Hybrid Adj. Diff. Sig?
Male 46% 39% -7% +
Asian 24% 23% -1%
Black 14% 14% 0%
Hispanic 20% 14% -5% +
White 41% 46% 4%
Other/Missing 1% 3% 2%
Average Age 21.9 22.0 0.0
Age
7/31/2019 Interactive Learning Online At Public Universities
18/53
Interactive Learning Online at Public Universities: Evidence from Randomized Trials May 22, 2012 17
TABLE 3. STUDENT CHARACTERISTICS BY STUDY PARTICIPATION
Participant Non-Part. Adj. Diff. Sig?
Male 42% 44% -1%
Asian 23% 17% 1%
Black 14% 13% 0%
Hispanic 17% 10% 3% *
White 44% 47% 6% *
Other/Missing 2% 13% -10% **
Average Age 21.9 21.6 -0.3
Age
7/31/2019 Interactive Learning Online At Public Universities
19/53
Interactive Learning Online at Public Universities: Evidence from Randomized Trials May 22, 2012 18
Te resuls in able 3 indicae ha he 605 sudy par icipans, while no ul lyrepresenaive o all saisics sudens in any ormal sense, have broadly simi-lar characerisics. Tere are sais ically signican dierences beween sudyparicipans and non-paricipans on several characerisics, bu mos o hedierences are small in magniude. For example, paricipans are more likelyo be enrolled ull-ime, bu only by a margin o 90 versus 86 percen. Courseoucomes are also broadly similar, wih par icipans earning similar grades and
being only sl ighly less likely o complee and pass he course as compared onon-paricipans.
Our analysis o he daa is sraighorward; we compare he oucomes o su-dens randomly assigned o he radiional orma o he oucomes o sudensrandomly assigned o he hybrid orma. In a small number o cases4 perceno he 605 sudens in he sudyparicipans atended a dieren orma sec-ion han he one o which hey were randomly assigned. In order o preservehe randomizaion procedure, we associaed sudens wih he secion y pe o
which hey were randomly assigned. Tis is someimes called an inen o reaanalysis. Under cerain assumpions, he eec o acually ak ing he course inhe hybrid orma (as opposed o jus being randomly assigned o do so) can be
calculaed by increasing our esimaes by 4 percen.25
Tis is someimes calledhe reamen on he reaed esimae, which in our sudy is very similar o heinen o rea esimae because mos sudens ook he course in he orma o
which hey were randomly assigned.
How did learning oucomes compare across he reamen and conrol groups?We rs examine he impac o assignmen o he hybrid orma, rela ive o heradiional orma, in erms o he rae a which sudens compleed and passedhe course, heir perormance on a sandardized es o saisics (he CAOSes), and heir score on a se o nal exam quesions ha were he same in hewo ormas.26 Our main resuls are summarized in Figure 1 (page 19), and heregression resuls are repored in Appendix able A3.27 We nd no saisicallysignican dierences in learning oucomes beween sudens in he radiional and hybrid-orma secions. Hybrid-orma sudens did perorm sl ighly beterhan radiional-orma sudens on hree oucomes, achieving pass raes ha
25 The key assumption is that being randomly assigned to hybrid or traditional did not have an effect on student
outcomes independent of its effect on the format in which students were enrolled. This assumption would
be violated if, for example, students hoped for a cer tain outcome of the random assignment and were disap-pointed when they did not get their preferred assignment, which in turn caused them to do worse in
the course.
26 All of our results control for course-specific dummy variables, since students were randomized within cours-
es; these variables also control for unobserved student characteristics that are constant within institutions.
However, we obtain similar results when we do not control for institution dummies, as would be expected
given that the probability of being assigned to the hybrid sec tion was constant across courses (50%).
27 Note that the pass rate in Figure 1 and Appendix Table A3 cannot be used to calculate the percentage of stu-
dents who failed the course because the non-passing group includes students who never enrolled or withdrew
from the course without receiving a grade.
7/31/2019 Interactive Learning Online At Public Universities
20/53
Interactive Learning Online at Public Universities: Evidence from Randomized Trials May 22, 2012 19
wereabou hree percenage poins higher, CAOS scores abou one percenagepoin higher, and nal exam scores wo percenage poins higherbu none ohese dierences passes radiional ess o saisical signicance.28
Figure 1. Effect of Hybrid Format on Student Learning Outcomes
0 10 20 30 40 50 60 70 80%
Traditional
Hybrid
PassRate
CAOSPost-Test
FinalExam
54.7%
56.7%
76.4%
79.7%
46.9%47.5%
Notes: None of the traditional-hybrid differences above were statistically signicant at the 10% level. See Appendix Table
A3 for more information about the results depicted here.
I is imporan o repor ha hese dierences (or raher, he lack o saisicallysignican dierences) are ai rly precisely esimaedsee boh he acual coe-ciens and he small sandard errors o he eec es imaes repored in Appen-dix able A3.29 In oher words, we can be quie conden ha he averageeecs were in ac close o zero. As we explain shorly, we also nd ha he same
basic resuls hold or subgroups, and ha disribuions o key oucomes are verysimilar or boh he reamen and conrol group sudens. One commenaor,Michael S. McPherson, presiden o he Spencer Foundaion, observed ha wha
we have here are quie precisely esimaed zeros.
Ta is, i here had in ac been pronounced dierences in oucomes beweenradiional-orma and hybrid-orma groups, i is highly likely ha we would
have ound hem.30
Our ndings are srik ingly dieren in his consequenialrespec rom a hypoheical nding o no signican dierence which resuledrom a coecien o some magniude accompanied by a very large sandard erroror by big dierences in he disr ibuions o oucomes. A hypoheical nding ohis k ind would mean, in eec, ha we don know much: ha he rue resulscould be almos anywhere.
28 The effect on CAOS test scores in standard deviation units (using the distribution of the control group) was
0.05. We also examined performance using item-level CAOS post-test data. Specifically, we grouped the 40
items into the 20 items on which delMas et al.s (2007) national sample of students exhibited significant
growth (over the course of a semester) and the remaining 20 items. We found similar hybrid-format effec tsfor the two g roups of items.
29 We cluster standard errors by section in order to capture section-specific shocks to student outcomes (such
as the quality of the instructor). Students who were randomly assigned but never enrolled in the course are
grouped as a section within each course for t he purpose of computing clustered st andard errors.
30 Some degree of caution is warranted in interpreting the results for the CAOS post-test because the average
students CAOS score only increased by five percentage points over the course of the semester (among
students who took both the pre-test and the post-test ). This may have resulted in part from some students
not taking the CAOS test seriously because, in most cases, it was not part of their grade. However, it is
reassuring that the results for the CAOS test are consistent with results for pass rates and final exam scores.
In other words, we can be quite
condent that the average effects
were in fact close to zero. We alsond that the same basic results
hold for subgroups, and that
distributions of key outcomes are
very similar for both the treatment
and control group students.
7/31/2019 Interactive Learning Online At Public Universities
21/53
Interactive Learning Online at Public Universities: Evidence from Randomized Trials May 22, 2012 20
Tese ndings conrol or suden characerisics, including race/ehniciy,gender, age, ull-ime versus par-ime enrollmen saus, class year in college,parenal educaion, language spoken a home, and amily income. Tese conrolsare no sricly necessary since sudens were randomly assigned o secion or-ma, bu we include hem in order o increase he precision o our resuls and oconrol or any remaining imbalance in observable characerisics. However, weobain nearly idenical resuls when we do no include hese conrol variables
jus as we would expec given he apparen success o our random assignmenprocedure.
Our resuls are also robus o a variey o alernaive mehodologies used oanalyze he experimenal daa. Tese resuls are repored in Appendix able A4,and one is worh highlighing. A limiaion o our main resuls or CAOS pos-es and nal exa m scores is ha we only observe hese oucomes or sudens
who compleed he course and ook hese exams. Tis is unlikely o be a signi-can limiaion given ha we do no nd any signican eecs o secion ormaon course compleion and pass raes. Bu as an addiional check, we assignedsudens or whom we did no observe a CAOS pos-es score heir score on heCAOS pre-esin oher words, we assumed ha heir score did no change over
he course o he semeser. Sudens who did no ake eiher he pre-es or hepos-es were assigned he average pre-es score a heir insiuion. Te resul-ing se o real and impued pos-es scores yielded very similar resuls o hoseobained using only he real daa.
Te lack o dierences in mean oucomes beween ormas could mask dier-ences in he disribuion o oucomes.31 Figure 2 (page 21) shows ha his is nohe case or CAOS pos-es scores. Te disribuions o scores or radi ionaland hybrid orma sudens are largely similar, alhough scores are sl ighly morespread-ou or hybrid-orma sudens. We obain a similar nding or nal examscores (no shown).32 (Tis kind o comparison o disribuions is no possible orpass raes, which only ake on a value o 0 or 1 or an individual suden.)
Resuls broken down by individual insiuion (Appendix able A5) do noreveal any noeworhy paterns. Tese resuls are much noisier because heyare based on smaller numbers o sudens, bu hey do no indicae ha hehybrid orma was paricularly eecive or ineecive a any individual insiu-ionwih he possible excepion o Insiuion F, where coeciens are posiiveacross all our oucomes, alhough only saisically signican in he case o oneoucome.
31 We are indebted to Stephen Stigler, a professor at the University of Chicago and a member of the ITHAKA
board, for emphasizing to us the importance of considering this question.
32 In general, results that use final exam scores should be interpreted cautiously given limitations in these
exams and their implementation. Some institut ions included only a handful of questions that were common
across the sections of t he course (and we only use data from the common questions). At one institution,
common questions were administered to some students after the end of the semester because the actual
final exam only included common questions in two ou t of six sect ions. At another institution, f inal exam data
were not available for the students of two instructors (covering three out of six traditional-format sections).
Excluding either or both of these institutions produces similar results, but it should be noted that the ef fect
on final exams in standard deviation units is substant ial in size (0.19; see Appendix Table A4) and imprecisely
estimated.
Results broken down by
individual institution do not
reveal any noteworthy patterns.
7/31/2019 Interactive Learning Online At Public Universities
22/53
Interactive Learning Online at Public Universities: Evidence from Randomized Trials May 22, 2012 21
Figure 2. Distributions of CAOS Post-Test Scores
Traditional
Hybrid
Traditional
4
Density
CAOS Post-Test Score
Kernel =epanechinov, bandwidth=0.03335
2
0
0 .2 .4 .6 .8
Hybrid
We also calculaed resuls separaely or subgroups o sudens dened in ermso var ious characeris ics, including race/ehniciy, gender, parenal educa-ion, primary language spoken, CAOS pre-es score, hours worked or pay, andcollege GPA. We did no nd any consisen evidence ha he hybrid-orma
eec varied by any o hese characerisics (Appendix able A6).33
Tere wereno groups o sudens ha beneed rom or were harmed by he hybrid ormaconsisenly across muliple learning oucomes.
Our main resuls provide compelling evidence ha, on average, sudens learnedjus as much in he hybrid orma as hey would have had hey insead aken hecourse in he radiional ormawih learning measured in radiional ways,in erms o course compleion, course grades, and perormance on a naional eso saisical lieracy. Tis seemingly bland resul is in ac very imporan, in l igho perhaps he mos common reason given by aculy and deans or resising heuse o ILO-ype insr ucion: We worry ha basic suden learning oucomes(pass raes and masery o conen) will be hur, and we won expose our su-
dens o his risk . Te research repored here suggess srongly ha such worriesare no well ounded.
We also examined how much sudens liked he hybrid orma o he course rela-ive o radiional orma sudens (Figure 3 [page 22] and Appendix able A7).
We ound ha sudens gave he hybrid orma a modesly lower overal l rainghan he one given by sudens aking he course in radiional orma (he raing
was abou 11 percen lower). By similar margins, hybrid sudens repored eel ing
33 The one exception is our finding that completion and pass rates were significantly higher in the hybrid course
for students with family incomes of at least $50,000 per year, but not for students with family incomes of
less than $50,000 per year. However, we hesitate to attach much significance to this result given that we do
not find such a patt ern for our other measure of socioeconomic status (parental education) or for measures
of academic preparation. This finding could be the result of random noise in the coef ficients (especially given
the large number of coefficients reported in Appendix Table A6).
7/31/2019 Interactive Learning Online At Public Universities
23/53
Interactive Learning Online at Public Universities: Evidence from Randomized Trials May 22, 2012 22
ha hey learned less and ha hey ound he course more dicul.34 Tese hreedierences, hough modes in size, were saisically signican a he 10 percenlevel. Bu here were no saisically signican dierences in sudens repors ohow much he course raised heir ineres in he subjec mater.
Figure 3. Effect of Hybrid Format on Student Evaluations of Course
0.0
0.5
1.0
1.5
2.0
2.5
DifficultyAmountLearned
RaisedInterest
OverallRating
Traditional
Hybrid
Raiting(0
4)
2.3
2.0+
2.0+
2.5+
1.7 1.7
2.22.3
Notes: + indicates traditional-hybrid difference is statistically signicant at the 10% level. See Appendix Table A7 for moreinformation about the results depicted here.
A leader o one o he universi ies ha ac ively paricipaed in our sudyopined ha a deec o he CMU prooype course is ha i has no addiciveor Disney-like appeal; i was, as his person pu i, designed by cogniivescieniss (no oense inended!). In conras, some sudens in he radiionalorma may have been reaed o an occasional colorul sory, personal recol-lecions o he insrucor, or oher reamens someimes used by aculy oimprove sudens opinions o heir course.35
We also asked sudens how many hours per week hey spen outside o class
working on he sais ics class. Hybrid-orma sudens repored spending 0.3hour more each week, on average, han radiional-orma sudens. Tis dier-
34 Students responses to the open-ended questions on the end-of-semester surveys indicate that many
students in the hybrid format would have liked more face-to-face time with the instructor than one hour
each week; others felt that the instructor could have better used t he face-to-face time to make the weekly
sessions more structured and/or helpful in explaining the material and going over concepts students did not
understand. A number of students in the hybrid course also indicated they would have benefited from more
practice problems or examples, and many were frustrated by the dif ficulty of checkpoint assessments in the
course and by problems they encountered using the statistical software packages to complete assignments.
35 The question of what is really going on herewith no differences in learning outcomes as measured con-
ventionally combined with a (to be sure, small) difference in qualitative assessmentsrelates t o a larger
literature on measured learning outcomes versus more subjective measures of student satisfaction with
online or hybrid courses relative to their satis faction with face-to-face courses. Studies (some more rigorous
than others) pertaining to the lat ter topic abound; a few examples include: Hannay, Maureen, and Tracy
Newvine. Perceptions of Distance Learning: A Comparison of Online and Traditional Learning. 2.1 (2006):
1-11. Accessed April 24, 2012. http://jolt.merlot.org/documents/MS05011.pdf; Horspool, Agi, and Carsten
Lange. Applying the Scholarship of Teaching and Learning: Student Perceptions, Behaviours and Success
Online and Face-to-Face. 37.1 (2011): 73-88. Accessed April 24, 2012. http://www.tandfonline.com/doi/abs
/10.1080/02602938.2010.496532; and Meyer, Katrina A. Student Perceptions of Face-to-Face and Online
Discussions: The Advantage Goes To . .4: 53-69. Accessed April 24, 2012. http://sloanconsortium.org/
jaln/v11n4/student-percept ions-face-face-and-online-discuss ions-advantage-goes.
Our results indicate that hybrid-
format students took about
one-quarter less time to achieve
essentially the same learning
outcomes as traditional-format
students.
http://jolt.merlot.org/documents/MS05011.pdfhttp://www.tandfonline.com/doi/abs/10.1080/02602938.2010.496532http://www.tandfonline.com/doi/abs/10.1080/02602938.2010.496532http://sloanconsortium.org/jaln/v11n4/student-perceptions-face-face-and-online-discussions-advantage-goeshttp://sloanconsortium.org/jaln/v11n4/student-perceptions-face-face-and-online-discussions-advantage-goeshttp://sloanconsortium.org/jaln/v11n4/student-perceptions-face-face-and-online-discussions-advantage-goeshttp://sloanconsortium.org/jaln/v11n4/student-perceptions-face-face-and-online-discussions-advantage-goeshttp://www.tandfonline.com/doi/abs/10.1080/02602938.2010.496532http://www.tandfonline.com/doi/abs/10.1080/02602938.2010.496532http://jolt.merlot.org/documents/MS05011.pdf7/31/2019 Interactive Learning Online At Public Universities
24/53
Interactive Learning Online at Public Universities: Evidence from Randomized Trials May 22, 2012 23
ence, which is no saisically signican, implies ha, in a course where a radi-ional secion mees or 3 hours each week and a hybrid secion mees or 1 hour,he average hybrid-orma suden would spend 1.7 less hours each week in totaltime devoed o he course, a dierence o abou 25 percen. Tis resul is con-sisen wih oher evidence ha ILO-ype ormas do succeed in achieving hesame learning oucomes as radiional-orma insrucion in less imewhichpoenially has imporan implicaions or scheduling and he rae o coursecompleion.36
In sum, our resuls indicae ha hybrid-orma sudens ook abou one-quarerless ime o achieve essenially he same learning oucomes as radiional-ormasudens. Te hree main limiaions o his analysis are: (1) we were no able orandomly assign insrucors o secion ormaswhich would have been di-cul, i no impossible, o do, especially in a small scale sudy, or o conrol ordierences in how radiional-orma secions were augh;37 (2) he limiaionso he CMU prooype o an ILO courseno cusomizaion and no addiciveeaures; and (3) our inabiliy o repor resuls or communiy colleges. Despiehese limiaions, hese resuls refec a ser ious, rigorous, assessmen o he rela-ive ecacy o echnology-enhanced learning (ILO-syle hybrid insrucion)
compared o he radiional mode o insr ucion. Tey are, we believe, he besevidence available o dae on an imporan se o quesions. Tere is, wihoudoub, much more research ha can and should be carried ou bu, a he mini-mum, his sudy suppors a no-harm-done conclusion regarding a leas onecurren prooype o an ILO sysem.
36 The authors of this paper, interested to see whether the hybrid course might enable students to learn the
material in the statistics course in a shorter period of time, conducted a separate, quasi-experimental study
in the summer of 2011, involving one of the campuses used in our larger fall 2011 study. This summer probe
occurred over two shortened (five or six-week) summer sessions; while we collected a large amount of
data and used numerous controls in our analysis, we did not randomly assign st udents to the hybrid or theface-t o-face format so that we could obtain larger sample si zes (f rom what was already a much smalle r pool
of students taking the course during the summer). The results of the summer study revealed no statistically
significant differences in the percentage of students who passed the course, in final exam grades, or in the
end-of-semester CAOS test results, between the students who took the course in a hybrid format and stu-
dents who took the course in a face-to-face format. A separate st udy by Marsha Lovett, associate director of
the Eberly Cente r for Teaching Excellence, and her colleagues at CMU produced similar findings. In that study,
which was conducted at CMU, the performance of students who were randomly assigned to the face-to-face
format of a s tatis tics course, which met four times a week for 50 minutes each time, was compared with
the performance of students who were randomly assigned to a hybrid format, which met twice a week for 50
minutes each time. The researchers found little difference in the amount of time students reported spending
on the course outside of class each week (about 2.8 hours for the hybrid-format student s, compared with
about 2.7 hours per week for face-to-face s tudents). In addition, students also were able t o learn in eight
weeks the same amount of material that students would ordinarily take 15 weeks in a face-to-face formatto learn. In this respect, the use of the CMU course could be said to increase learning ef ficiency. (For more
information about this study, see Lovett, Meyer, and Thilles 2008 art icle in the , entitled The Open Learning
Initiative: Measuring the effectiveness of the OLI s tatistics course in accelerating student learning, available
online at http://jime.open.ac.uk/2008/14.)
37 One commentator, Michael McPherson, noted that right now, quite apart from any use of online technologies,
very different instructional methods are used to teach introductory statist ics courses even within the same
universitynever mind across universities. But little effor t appears to be made to compare learning out-
comes and cost effec tiveness across different approachesa point Derek B ok, former president of Harvard
who continues to write on this subject, keeps emphasizing.
http://jime.open.ac.uk/2008/14http://jime.open.ac.uk/2008/147/31/2019 Interactive Learning Online At Public Universities
25/53
7/31/2019 Interactive Learning Online At Public Universities
26/53
Interactive Learning Online at Public Universities: Evidence from Randomized Trials May 22, 2012 25
amorized over ime. Tere are also ransiion coss enailed in moving rom heradiional, mosly ace-o-ace model (ha may, however, employ some elemenso simple online models, such as video-aped lecures or homework assign-mens online) o a hybrid model ha akes advanage o more sophisicaed ILOsysems employing machine-guided insrucion, cogniive uors, embeddedeedback loops, and some orms o auomaed grading. Insrucors need o berained o ake ull advanage o such sysems. Tere may also be conracual lim-is on secion size ha were designed wih he radiional model in mind bu hado no make sense or a hybrid model. I is possible ha hese consrains can bechanged in a nex round o conrac negoiaions, bu ha oo wi ll ake ime.
o overcome (or avoid!) hese problems, we hink here is much o be said orcarrying ou simulaed cos probes, and we made a very rough atemp o do
jus his on hree o he campuses included in he lea rning oucomes par ohe sudy. We concepualize he research quesion here no as how much willinsiuions save righ now by shiing o hybrid learning? bu raher as under
wha assumpions wi ll cos savings be realized, over ime, by shiing o a hybridorma, and how large are hose savings likely o be? Our basic approach waso sar by looking in as much deail as possible a he acual coss o eaching a
basic course in radiional orma (usually, bu no always, he saisics course)in a base year. Ten, we worked wih leaders on hese campuses o simulae heprospecive, seady-sae coss o a hybrid-online version o he same course,looking hree o ve years ino he uure. Tese ex ploraory simulaions were
based on explici assumpions, especially abou sang, ha were incorporaedino spreadsheeswhich in urn allowed us o see how sensiive our resuls
were o variaions in key assumpions. We ocused heavily on personnel coss,because o boh heir imporance and our abiliy o examine hem wih someprecision. Oher coss, including space coss, were also considered. We hopedha he simulaions would, a he minimum, give us a leas a rough sense o hepoenial impac on coss o inroducing hybrid learning and, more specically,
show us how much room here would be or insiuions o share cos savingswih acul y and sudens on a coninuing basis.
We ocus on insrucor compensaion because hese coss comprise a subsan-ial porion o he recurr ing cos o eaching and are he mos sraighorwardo measure. Space coss are also an imporan caegory o coss ha are likely o
be reduced by shiing o a hybrid lear ning model (he mos imporan caegoryin some siuaions), bu such coss are more dicul o measure accuraely ahe level o an individual course. A hybrid model also aords boh aculy andsudens signicanly greaer scheduling fexibiliy, a poenially very imporan
bene ha will no be capured by our simulaions. On he oher hand, here arealso oher y pes o coss ha we do no consider here, such as increases in inor-maion echnology (I) suppor coss associaed wih moving a signi can shareo learning aciviies online. Such added coss can be ar rom riv ial.
We did exploraory simulaions or wo ypes o radi ional eaching models: (1)a model in which sudens are augh in secions o roughly 40 sudens per sec-ion; and (2) a model in which all sudens atend a common lecure and are henassigned o small discussion secions led by eaching assisans. We compare hecurren coss o each o hese radi ional eaching models o simulaed coss oa hybrid model in which more insrucion is delivered online, sudens atend
Our simulations illustrate that
hybrid learning offers opportunities
for signicant savings incompensation costs, but that the
degree of cost reduction depends
on the exact model of hybrid
learning used.
7/31/2019 Interactive Learning Online At Public Universities
27/53
Interactive Learning Online at Public Universities: Evidence from Randomized Trials May 22, 2012 26
weekly ace-o-ace sessions wih par-ime insrucors, and he course is over-seen by a enure-rack proessor (wih adminisraive responsibiliies delegaedo a par-ime insrucor).
We have decided, however, no o presen he acual calculaions and resuls ohese simulaions in he body o his repor. Tey are oo speculaive and subjeco considerable variaion depending on how a par icular campus waned oorganize is eaching. Te danger o specious precision is grea, and i would be
wrong o atach much imporance o paricular numbers. Suce i o say ha hecrude models we employed sugges savings in compensaion coss ranging rom36 percen o 57 percen in he all-secion model, and 19 percen in he lecure-secion model. Appendix B presens hese resuls and many more calculaionsand gures showing how sensiive poenial savings are as we var y assumpionsabou secion sizes and compensaion.
Tese simulaions illusrae ha hybrid learning oers opporuniies or sig-nican savings in compensaion coss, bu ha he degree o cos reduciondepends (o course) on he exac model o hybrid learning usedespecially herae a which insr ucors are compensaed and secion size. A large share o cossavings is derived rom shiing away rom ime spen by expensive proessors
oward boh machine-guided insrucion ha saves on sang coss overall andoward ime spen by less expensive sa in Q&A setings. O course, enuredproessors canno be laid o in order o realize hese savings and, in any case,orce reducions are no required o save signican amouns o money. Insiu-ions ha ace pressures o expand enrollmen are in an especially good posiiono realize savings by shiing he mix o eaching ime. When more sudens areo be augh, hybrid models make his possible wihou increasing he demandsmade on enured aculy. Recruimen coss may hereby be reduced along wihcompensaion coss per suden, and debaes over mainaining commimenso exising aculy are avoided. Over ime, cerainly, sa size can be aleredhrough atriion. A lso, he ime o proessors can be reallocaed oward smaller,more advanced classeswhich many preer o each (such reallocaions mayno save he insiuion money, hough hey may improve he overall educaionalexperience o many sudens).
In hese simulaions, we have assumed ha he number o sudens in he coursewill remain consan. However, as already suggesed, many insiuions aceincreasing demand or places in heir classes. Te hybrid learning model is veryatracive in such circumsances or wo primary reasons: (a) less space is neededin general; and (b) hybrid courses provide boh sudens and eachers wihgreaer scheduling fexibiliy. Increased enrollmen can a lso lead o increasedcompensaion cos savings (per suden) because he xed coss o he proessor
in charge o he course, and an adminisraive coordinaor, would be spread overa larger number o sudens. For he same reason, he larges savings will be real-ized in courses wih he larges enrollmen, all else equal.
Our simulaion approach underesimaes subsanially he savings rom mov-ing oward a hybrid model in many setings because we do no accoun or spacecoss. We are relucan o pu a dollar gure on space coss because capial cossare dicul o apporion accuraely down o he course level. However, i is moresraighorward o ca lculae he percenage change in he need or classroom
7/31/2019 Interactive Learning Online At Public Universities
28/53
Interactive Learning Online at Public Universities: Evidence from Randomized Trials May 22, 2012 27
space ha would resul rom shiing oward a hybrid model. Te hybrid coursemees or one hour each week, whereas he radiional course ypically meesor hree or our hours each week. Consequenly, he hybrid course requires 67percen o 75 percen less classroom use han he radiional course, assumingha he course is augh in secions, ha secion size is held consan, and hahe hybrid course does no have addiional space requiremens o is own, such asaddiional compuer labs.
In he shor run, insiuions canno sell or demolish heir buildings. However,in he long run, using hybrid models or some large inroducory courses wouldallow insiuions o expand enrollmen wihou a commensurae increase inspace cossa major cos savings (cos avoidance) relaive o wha insiuions
would have had o spend had hey sayed wih a radiional model o insrucion.An imporan poin here is ha he hybrid model need no jus save moneyican also suppor an increase in access o higher educaion. I serves he accessgoal boh by making i more aordable or he insiuion o enroll more sudensand by accommodaing more sudens because o greaer scheduling fexibiliy,
which is especia lly imporan or sudens wih complicaed lives who have obalance amily responsibiliies and work wih course compleion, as well as or
sudens who may live a disance rom campus.41
o repea, we regard his highly speculaive cos simulaion eor as primarilyillusraive o an approach ha we believe has meri. I is no surprise ha under aplausible se o assumpions ILO sysems have he poenial o save sang cossand classroom space, and o increase scheduling fexibiliy. o go beyond haobvious saemen requires, a he minimum, deailed knowledge o local campussiuaions and realisic assessmens o wha is easible.
Summary Observations
In he case o a opic as acive as online learning, where new aricles appearevery day and millions o dollars are being invesed by a wide variey o eni-ies, we should perhaps expec ha here wil l be infaed claims o specacularsuccesses. Te ndings in his sudy warn agains oo much hype. o he beso our knowledge, here is no compelling evidence ha online learning sysemsavailable odayno even highly ineracive sysems, o which here are veryewcan in ac deliver improved educaional oucomes across he board, ascale, on campuses oher han he one where he sysem was born, and on a sus-ainable basis. Tis is no o deny, however, ha hese sysems have grea poen-ial. We believe ha hey do, and ha vigorous eors should be made o aggres-sively explore uses o boh he relaively simple sysems ha are prolieraing al l
around us, oen o good eec, and more sophisicaed sysems ha are si ll inheir inancy. Tere is every reason o expec hese sysems o improve over ime,perhaps dramaically, and hus i is no oolish o believe ha learning oucomes
will also improve.
41 The MOOCs (Massive Open Online Courses), like edX and Udacity, are another example of a technologically-
driven effort t o give many more students, of all kinds, access to low-cost instruction of high quality. It
remains to be seen, however, whether the certificates and badges that MOOCs propose to confer will be
accepted as credits toward degrees by mainline universitiesand how much this will matter to students
themselves, as well as to employers and others who want to assess learning outcomes.
7/31/2019 Interactive Learning Online At Public Universities
29/53
Interactive Learning Online at Public Universities: Evidence from Randomized Trials May 22, 2012 28
Te barriers o adopion o even he simpler sysems, le alone hose o moresophisicaed sysems ha are ruly ineracive, are deailed in he companionIhaka S+R repor by Lawrence S. Bacow e al. ha has been cied previously.42Tere is no need o summarize hose ndings here. I is sucien o re-empha-size he need o:
a. work closely wih aculy, who undersandably wil l wan o pu a leas some-hing o heir own samp on all such courses;
b. conron direc ly and imaginaively worries abou loss o jobs;
c. encourage serious research by rused hird paries on evidence o learningoucomes;43 and
d. recognize, and even embrace, he need o use such echnologies o increaseproduciviy and lower insrucional coss wihou sacricing learning ou-comes.
Te research repored here demonsraes he poenial o ruly ineracivelearning sysems ha use machine-guided proocols (wha we have been call-ing ILO) o provide some orms o insrucion, in properly chosen courses,
in appropriae setings. Our ndings demonsrae ha such an approach needno negaively impac learning oucomesand conceivably could, in heuure, improve hem as hese sysems become ever more sophisicaed anduser-riendly. I is also enirely possible ha by (poenially) saving signicanamouns o resources, such sysems can lead o more, no less, opporuniyor sudens o bene rom exposure o modes o insrucion such as direcedsudyi scarce aculy ime can be benecially redeployed. Bu none o his will
be easy.
In he spir i o hinking ou loud, here are some houghs as o wha is requiredi we are o ake ull advanage o he poenial o ILO oerings. Tese observa-ions, we should noe, are based less on he specic ndings in his sudy han
on conversaions wih a wide