Date post: | 03-Jun-2018 |
Category: |
Documents |
Upload: | srijit-sanyal |
View: | 223 times |
Download: | 0 times |
of 53
8/12/2019 Sr Ithaka Interactive Learning Online at Public Universities
1/53
May 22, 2012
Interactive Learning Online
at Public Universities:Evidence from
Randomized TrialsWilliam G. Bowen
Matthew M. Chingos
Kelly A. Lack
Thomas I. Nygren
8/12/2019 Sr Ithaka Interactive Learning Online at Public Universities
2/53
Interactive Learning Online at Public Universities: Evidence from Randomized Trials May 22, 2012 1
Ihaka S+R is a sraegic consuling and research service provided by IHA K,a no-or-profi organizaion dedicaed o helping he academic communiy usedigial echnologies o preserve he scholarly record and o advance research andeaching in susainable ways. Ihaka S+R ocuses on he ransormaion o schol-arship and eaching in an online environmen, wih he goal o ideniying hecriical issues acing our communiy and acing as a caalys or change. JSOR,a research and learning plaorm, and Porico, a digial preservaion service, arealso par o IHAK.
Copyrigh 2012 IHA K. Tis work is licensed under he Creaive CommonsAtribuion No Derivaive Works 3.0 Unied Saes License. o view a copy ohe license, please see htp://creaivecommons.org/licenses/by-nd/3.0/us
http://creativecommons.org/licenses/by-nd/3.0/ushttp://creativecommons.org/licenses/by-nd/3.0/us8/12/2019 Sr Ithaka Interactive Learning Online at Public Universities
3/53
Interactive Learning Online at Public Universities: Evidence from Randomized Trials May 22, 2012 2
Preface
Higher educaion is acing serious challenges in he Unied Saes. Tere isincreasing concern abou rising coss, he qualiy o educaion, and ha henaion is losing is compeiive edge. Online learningspecifically highlyineracive, closed-loop, online learning sysems ha we call ILO or IneraciveLearning Onlineholds he promise o broadening access o higher educaiono more individuals, while also lowering coss or sudens. Bu is he qualiyhere?
In our firs repor in his area, Barriers o Adopion o Online Learning Sysemsin U. S. Higher Educaion, we highlighed a broad, widely held concern abouhe qualiy o learning oucomes achieved hrough online learning. Bu do weacually know how ineracive online learning sysems really compare o hein-classroom experience? Tis second repor was designed o help find answers.
We used a sricly quaniaive mehodology o compare he wo learning approachesin a rigorous way. In six differen public insiuions, we arranged or he sameinroducory saisics course o be augh. In each insance, a conrol group wasenrolled in a radiional classroom-based course; hen, a reamen group ook ahybrid course using a prooype machine-guided mode o insrucion developed aCarnegie Mellon Universiy in concer wih one ace-o-ace meeing each week.Sudens were assigned o hese wo groups by means o a careully designedrandomizaion mehodology. Te research we conduced was designed o answerhese quesions:
Can sophisicaed, ineracive online courses be used o mainain or improvebasic learning oucomes (masery o course conen, compleion raes, andime-o-degree) in inroducory courses in basic subjecs such as saisics?
Are hese courses as effecive, or possibly more effecive, or minoriy andlow-socioeconomic-saus sudens and or oher groups subjec o sereoype
hrea? Or, are hese groups less well suied o an online approach?
Are such courses equally effecive wih no-so-well-prepared sudens andwell-prepared sudens?
Te resuls o his sudy are remarkable; hey show comparable learning oucomes orhis basic course, wih a promise o cos savings and produciviy gains over ime.
More research is needed. Even hough he analysis was rigorous, i was a singlecourse. We need o learn more abou he adapabiliy o exising plaorms oroffering oher courses in differen environmens. Ihaka S+R is commited oconinuing his research and shar ing our findings broadly.
We look orward o coninuing o engage wih al l hose who care abou highereducaion o help deliver on he poenial ha new echnologies provide.
DEANNA MARCUM
Managing Director, Ithaka S+R
mailto:Deanna.Marcum%40ithaka.org?subject=mailto:Deanna.Marcum%40ithaka.org?subject=8/12/2019 Sr Ithaka Interactive Learning Online at Public Universities
4/53
Interactive Learning Online at Public Universities: Evidence from Randomized Trials May 22, 2012 3
Interactive Learning Online at Public Universities:
Evidence from Randomized Trials
2 Preface
4 Introduction
9 Educational Outcomes in Public Universities
23 Costs and Potential Savings
26 Summary Observations
29 Acknowledgements
30 Appendices
8/12/2019 Sr Ithaka Interactive Learning Online at Public Universities
5/53
Interactive Learning Online at Public Universities: Evidence from Randomized Trials May 22, 2012 4
Interactive Learning Online at
Public Universities: Evidence fromRandomized TrialsWilliam G. Bowen, Matthew M. Chingos, Kelly A. Lack, and Thomas I. Nygren1
May 22, 2012
Introduction
Te opic o online learning in higher educaion is o obvious imporance. Teserious economic and social problems acing he U.S.high unemploymen,slow growh, and severe inequaliiesare relaed, many believe, o ailures o he
1 The authors are all associated with Ithaka S+R (the Strategy and Research arm of ITHAKA), which sponsored
this study. Bowen is a senior advisor t o Ithaka S+R, Chingos is a senior research consultant at Ithaka S+R
and a fellow at the Brookings Institution's Brown Center on Educat ion Policy, Lack is a research analyst,
and Nygren is a project director and senior business analyst for Ithaka S+R. The authors wish to thank the
foundat ions that supported this work: the Carneg ie Corporation of New York, the William and Flora Hewlett
Foundation, the Spencer Foundation, and a fourth foundation that has asked to remain anonymous. We also
thank our colleagues at ITHAKAand L arry Bacow, Johanna Brownell, Jackie Ewenstein, and Kevin Guthrie inparticularfor their generous help all along the way. But most o f all, we wish to thank our faithful friends on
the participating campuses for their hard work and patience with us; their names are appended to this repor t.
A number of these individuals (as well as others) have commented on a draft of the repor t, but the authors
are, of course, fully responsible for the views expressed here and for any errors t hat remain.
Ithaka S+R has sponsored three studies of online learning, of which this is the longest lasting. The two other
studies are now available on the Ithaka S+R website. See Barriers to Adoption of Online Learning Systems
in U.S. Higher Education by Lawrence S. Bacow, William G. Bowen, Kevin M. Guthrie, Kelly A. Lack, and
Matthew P. Long, and Current Status of Research on Online Learning in Postsecondary Educat ion by William
G. Bowen and Kelly A. Lack (bo th available online at http://www.sr.ithaka.org/).
8/12/2019 Sr Ithaka Interactive Learning Online at Public Universities
6/53
Interactive Learning Online at Public Universities: Evidence from Randomized Trials May 22, 2012 5
U.S. educaion sysem, including higher educaion.2Levels o educaionalatainmen in his counry have been sagnan or almos hree decades, whilemany oher counries have been making grea progress in educaing largernumbers o heir ciizens. Tere is growing concern ha he U.S. is losing iscompeiive edge in an increasingly knowledge-driven world. Also, subsanialachievemen gaps relaed o race and socioeconomic saus persis and have agrea deal o do w ih worry ing inequiies. Moreover, here are good reasons o
believe ha hese wo problems are closely relaed.3
The Cost Squeeze in Higher Education
A he same ime, higher educaion, especially in he public secor, is increasinglyshor o resources. Saes coninue o cu back appropriaions in he ace o fiscalconsrains and pressures o spend more on oher hings, such as healh careand reiremen expenses.4Caliornia is a dramaic case in poin. Lack o undinghas caused Caliornia colleges and universiies o reduce he size o heir ener-ing classes a he very ime when increasing numbers o sudens are seekingo enroll.5Higher uiion revenues migh be an escape valve, bu here is greaconcern abou uiion levels and increasing resenmen among sudens andheir amilies ha is having poliical reverberaions. Presiden Obama, in his
2 The authors agree that there is an important connection between educational outcomes and the economic
performance of a c ountry. But we would warn against exaggerating the power of the connection. In the case
of the U.S., for example, the recent recession and the slow rate of g rowth seen in the last few years surely
owe more to the 2008 financial excesses than they do to deficiencies in the countrys higher education
system. As Jacob Weisberg pointed out in Newsweek in 2010 with respect to the recent recession, there are
no strong candidates for a single factor t hat would have caused the crisis in the absence of any others
(Weisbergs piece can be f ound online at http://www.thedailybeast.com/newsweek/2010/01/08/what-
caused-the-great-recession.html).
3 SeeEquity and Excellence in American Higher Education by William G. Bowen, Martin A. Kurzweil, and
Eugene M. Tobin (2005) for an extended discussion of the historical record and of the likely connections,
going forward, between achievement gaps and overall levels of educational at tainment. See also David
Leonhardts October 8, 2011 column in the New York Times, The Depression: If Only Things Were That Good,
in which he argues that the U.S. is worse off today than it was in the 1930s because innovation is lagging
which he attributes in no small part to deficiencies in education (http://www.nytimes.com/2011/10/09/sun-
day-review/the-depression-if-only-things-were-that-good.html?_r=1&pagewanted=all.) Of course, lagging
rates of educational at tainment have their origins in low high school graduation rates. See Henry M. Levin
and Cecilia E. Rouse, The True Cost of High School D ropout, New York Times, January 25, 2012. (http://
www.nytimes.com/2012/01/26/opinion/the-true-cost-of-high-school-dropouts.html). But these problems
are then compounded by low completion rates among those who both graduate from high school and enter
college; see Crossing the Finish Line: Completing College at Americas Public Universities(2009) by William
G. Bowen, Matthew M. Chingos, and Michael S. McPherson.
4 A report released in spring 2012 by the State Higher Education Executive Officers, entitled State Higher
Education Finance FY 2011 (http://www.sheeo.org/finance/shef/SHEF_FY2011-EARLY_RELEASE.pdf), docu-
ments the dire economic circumstances of many public institutions.
5 In November 2008, California State University became the first public university to limit enrollment when,
despite a 20% increase in applications from prospective first-year students, it decided to reduce its s tudent
body by 10,000 students, following a $200 million decrease in tax revenue that academic year coupled with
an additional $66 million cut (see Under Financial Stress, More Colleges Cap Enrollments (November
26, 2008) in TIME, http://www.time.com/time/nation/article/0,8599,1861861,00.html). The University of
California and California Community College systems have since followed suit in the face of limited funding
available from the state (see the August 5, 2009 article Budget cuts devastate California higher educa-
tion in The Washington Examiner, http://washingtonexaminer.com/science-and-technology/2009/08/
budget-cuts-devastate-california-higher-education).
Levels of educational attainment in
this country have been stagnant for
almost three decades, while many
other countries have been making
great progress in educating larger
numbers of their citizens.
http://www.thedailybeast.com/newsweek/2010/01/08/what-caused-the-great-recession.htmlhttp://www.thedailybeast.com/newsweek/2010/01/08/what-caused-the-great-recession.htmlhttp://www.nytimes.com/2011/10/09/sunday-review/the-depression-if-only-things-were-that-good.html?_r=1&pagewanted=allhttp://www.nytimes.com/2011/10/09/sunday-review/the-depression-if-only-things-were-that-good.html?_r=1&pagewanted=allhttp://www.nytimes.com/2012/01/26/opinion/the-true-cost-of-high-school-dropouts.htmlhttp://www.nytimes.com/2012/01/26/opinion/the-true-cost-of-high-school-dropouts.htmlhttp://www.sheeo.org/finance/shef/SHEF_FY2011-EARLY_RELEASE.pdfhttp://www.time.com/time/nation/article/0,8599,1861861,00.htmlhttp://washingtonexaminer.com/science-and-technology/2009/08/budget-cuts-devastate-california-higher-educationhttp://washingtonexaminer.com/science-and-technology/2009/08/budget-cuts-devastate-california-higher-educationhttp://washingtonexaminer.com/science-and-technology/2009/08/budget-cuts-devastate-california-higher-educationhttp://washingtonexaminer.com/science-and-technology/2009/08/budget-cuts-devastate-california-higher-educationhttp://www.time.com/time/nation/article/0,8599,1861861,00.htmlhttp://www.sheeo.org/finance/shef/SHEF_FY2011-EARLY_RELEASE.pdfhttp://www.nytimes.com/2012/01/26/opinion/the-true-cost-of-high-school-dropouts.htmlhttp://www.nytimes.com/2012/01/26/opinion/the-true-cost-of-high-school-dropouts.htmlhttp://www.nytimes.com/2011/10/09/sunday-review/the-depression-if-only-things-were-that-good.html?_r=1&pagewanted=allhttp://www.nytimes.com/2011/10/09/sunday-review/the-depression-if-only-things-were-that-good.html?_r=1&pagewanted=allhttp://www.thedailybeast.com/newsweek/2010/01/08/what-caused-the-great-recession.htmlhttp://www.thedailybeast.com/newsweek/2010/01/08/what-caused-the-great-recession.html8/12/2019 Sr Ithaka Interactive Learning Online at Public Universities
7/53
Interactive Learning Online at Public Universities: Evidence from Randomized Trials May 22, 2012 6
2012 Sae o he Union address and in subsequen speeches, has decried risinguiions, called upon colleges and universiies o conrol coss, and proposed o
wihhold access o some Federal programs or colleges and universi ies ha didno address affordabiliy issues or mee compleion ess.6
oday, a variey o higher educaion insiuions mus conron he challengeo how o manage coss in he ace o igher unding. Whi le he proporion oeducaion spending drawn rom uiion revenues rose across all insiuions,
increases in uiion ofen oupaced increases in educaion and relaed spending(i.e. spending on insrucion, suden services, and some suppor and maine-nance coss relaed o hese uncions), calling ino quesion he susainabiliyo he curren unding model.7Moreover, he firs survey o provoss and chieacademic officers byInside Higher Edound ha on he quesion o insiuionaleffeciveness in conrolling coss, over 15 percen o all provoss gave heirinsiuions marks o 1 or 2 on effeciveness [on a scale rom 1 o 7, wih 7 being
very effecive].8I is equal ly noeworhy ha veryew chie academic officers(and especially hose a boh public and privae docoral universiies) gave heirinsiuions high marks on his meric. Recogniion o he problem is widespread;soluions have been hard o come by.
A undamenal source o he problem is he cos disease, based on he handicrafnaure o educaion wih is atendan lack o opporuniies or gains in pro-duciviy, which one o he auhors o his repor (Bowen) promulgaed in he1960s, in collaboraion wih William J. Baumol. Bu he ime may (finally!) bea hand when advances in inormaion echnology wil l permi, under he righcircumsances, increases in produciviy ha can be ranslaed ino reducions in
6 See Remarks by the President in State of the Union Address, January 24, 2012 (transcript available at (http://
www.whitehouse.gov/the-press-office/2012/01/24/remarks-president-state-union-address). Three days later,
Obama spoke about college affordability at the University of Michigan (transcript available at http://www.
whitehouse.gov/the-press-office/2012/01/27/remarks-president-college-affordability-ann-arbor-michigan).
This speech does not, however, contain more details concerning how affordability is to be measured or
what penalties are to be imposed on those who fail to pass the requisite tests. As Molly Broad, president of
the American Council on Education, said af ter the speech: The devil is in the [unspecified] details (Mixed
Reviews of Obama Plan to Keep Down College Costs, January 28, 2012, New York Times, http://www.nytimes.
com/2012/01/28/education/obamas-plan-to-control-college-costs-gets-mixed-reviews.html).7 According to the College Boards 2011 Trends in College Pricing Report (http://trends.collegeboard.org/
downloads/College_Pricing_2011.pdf), tuition at public two-year universities increased, on average, by
8.7% relative to the 2010-2011 academic year, and tuition at public four-year institutions for the 2011-2012
academic year increased, on average, by 8.3% for instate students and by 5.7% for out of state students.
In keeping with the trend over the previous four years, students at tending private institutions experienced
smaller percentage increases (4.5% for private not-for-profit four-year institutions and 3.2% fo r private for-
profit institutions).
8 See Scott Jaschik, Mixed Grades: A Survey of Provosts, Inside Higher Education, January 25, 2012, http://
www.insidehighered.com/news/survey/mixed-grades-survey-provosts.
Higher education, especially in the
public sector, is increasingly short
of resources. States continue to
cut back appropriations in the
face of scal constraints and
pressures to spend more on
other things, such as healthcare and retirement expenses.
http://www.whitehouse.gov/the-press-office/2012/01/24/remarks-president-state-union-addresshttp://www.whitehouse.gov/the-press-office/2012/01/24/remarks-president-state-union-addresshttp://www.whitehouse.gov/the-press-office/2012/01/27/remarks-president-college-affordability-ann-arbor-michiganhttp://www.whitehouse.gov/the-press-office/2012/01/27/remarks-president-college-affordability-ann-arbor-michiganhttp://www.nytimes.com/2012/01/28/education/obamas-plan-to-control-college-costs-gets-mixed-reviews.htmlhttp://www.nytimes.com/2012/01/28/education/obamas-plan-to-control-college-costs-gets-mixed-reviews.htmlhttp://trends.collegeboard.org/downloads/College_Pricing_2011.pdfhttp://trends.collegeboard.org/downloads/College_Pricing_2011.pdfhttp://www.insidehighered.com/news/survey/mixed-grades-survey-provostshttp://www.insidehighered.com/news/survey/mixed-grades-survey-provostshttp://www.insidehighered.com/news/survey/mixed-grades-survey-provostshttp://www.insidehighered.com/news/survey/mixed-grades-survey-provostshttp://trends.collegeboard.org/downloads/College_Pricing_2011.pdfhttp://trends.collegeboard.org/downloads/College_Pricing_2011.pdfhttp://www.nytimes.com/2012/01/28/education/obamas-plan-to-control-college-costs-gets-mixed-reviews.htmlhttp://www.nytimes.com/2012/01/28/education/obamas-plan-to-control-college-costs-gets-mixed-reviews.htmlhttp://www.whitehouse.gov/the-press-office/2012/01/27/remarks-president-college-affordability-ann-arbor-michiganhttp://www.whitehouse.gov/the-press-office/2012/01/27/remarks-president-college-affordability-ann-arbor-michiganhttp://www.whitehouse.gov/the-press-office/2012/01/24/remarks-president-state-union-addresshttp://www.whitehouse.gov/the-press-office/2012/01/24/remarks-president-state-union-address8/12/2019 Sr Ithaka Interactive Learning Online at Public Universities
8/53
Interactive Learning Online at Public Universities: Evidence from Randomized Trials May 22, 2012 7
he cos o insrucion.9Greaerand smareruse o echnology in eaching iswidely seen as a promising way o conroll ing coss while also reducing achieve-men gaps and improving access. Te exploding growh in online learning isofen cied as ev idence ha, a las, echnology may offer pahways o progress.10Online learning is seen by a growing number o people as a way o breakingree o cenury-old rigidiies in educaional sysems ha we have inheried. Temuch-discussed book on disrupive echnologies and universiies by ClayonChrisensen and Henry Eyring is perhaps he bes example o he atenion beinggiven o online echnologies as a way o changing prooundly he way we educaesudens.11
Tere are, however, also concerns ha a leas some kinds o online learning arelow qualiy and ha onl ine learning in general de-personalizes educaion.In thisregard, it is crit ically important to recognize issues of nomenclature: online learningis hardly one thing.I comes in a dizzying variey o flavors, ranging rom simply
videoaping lecures and posing hem or any-ime access, o uploading maeri-als such as s yllabi, homework assignmens, and ess o he Inerne, all he wayo highly sophisicaed ineracive learning sysems ha use cogniive uors andake advanage o muliple eedback loops. Te varieies o online learning can be
used o each many kinds o subjecs o differen populaions in diverse insiuional
9 Bowens co-author in the promulgation of the cost disease, William J. Baumol, has continued to discuss
its relevance not only for education but also for sect ors such as the performing arts and heath care. For
the initial statement of this proposition, see William J. Baumol and William G. Bowen,Performing Ar ts: The
Economic Dilemma, Twentieth Century Fund (1968). In essence, the argument is that in fields such as t he
performing arts and education, there is less opportunity t han in other fields to improve productivity (by, for
example, substituting capital for labor), that unit labor costs will therefore rise inexorably as these sectors
have to compete for labor with ot her sectors in which productivity gains are easier to come by, and that the
relative costs of labor-intensive activities such as chamber music and teaching will therefore continue to
rise. As Bowen argued in his Romanes lecture, for a number of years advances in information technology
have in fact increased productivity, but these increases have been enjoyed primarily in the form o f more
output (especially in research) and have generally led to higher, not lower, total costs. (For the text of the
Romanes lecture, see William G. Bowen, At a Slight Angle to the Universe: The University in a Digitized,
Commercialized Age, Princeton University Press, 2001; the text is also available on the Andrew W. Mellon
Foundation website: http://www.mellon.org/internet/news_publications/publications/romanes.pdf.)
10 A November 2011 report by the Sloan Consortium and the Babson Survey Research Group shows that between
fall 2002 and fall 2010, en rollments in online courses increased much more quickly t han tot al enro llments
in higher education. During this time period, the number of online course enrollments grew from 1.6 million
to 6.1 million, amounting to a compound annual rate of 18.3% (compared with a rate o f 2% for course
enrollments in general)although between fall 2009 and fall 2010 online enrollments grew more slowly,
at 10.1%. More than three of every 10 students in higher education now t ake at least one course online. Inaddition to the growth in what we call online or hybrid courseshowever nebulous that terminology may
bewe also fee l the pervasiveness of the Internet in higher education by the increasing use of it in the
form o f course manag ement systems or vi rtual reading materials/electronic tex tbooks incorporated in to the
curriculum. Even courses that are called traditional almost always involve some use of digit al resources.
11 See Clayton M. Christensen, and Henry J. Eyring, The Innovative University: Changing the DNA of Higher
Education from the Inside Out, San Francisco: Jossey-Bass, 2011. An October 2, 2011 New York Times op-ed
piece by Bill Keller, aptly titled The University of Wherever, is another illustration of the high visibility and
high stakes of the debate over online education (http://www.nytimes.com/2011/10/03/opinion/the-universi-
ty-of-wherever.html?pagewanted=all).
There are also concerns that at
least some kinds of online learning
are low quality and that online
learning in general de-personalizes
education. In this regard, it is
critically important to recognize
issues of nomenclature: onlinelearning is hardly one thing. It
comes in a dizzying variety of
avors.
http://www.mellon.org/internet/news_publications/publications/romanes.pdfhttp://www.nytimes.com/2011/10/03/opinion/the-university-of-wherever.html?pagewanted=allhttp://www.nytimes.com/2011/10/03/opinion/the-university-of-wherever.html?pagewanted=allhttp://www.nytimes.com/2011/10/03/opinion/the-university-of-wherever.html?pagewanted=allhttp://www.nytimes.com/2011/10/03/opinion/the-university-of-wherever.html?pagewanted=allhttp://www.mellon.org/internet/news_publications/publications/romanes.pdf8/12/2019 Sr Ithaka Interactive Learning Online at Public Universities
9/53
Interactive Learning Online at Public Universities: Evidence from Randomized Trials May 22, 2012 8
setings. A key poin, i an obvious one, is ha here is no one approach ha isrigh or every suden or every seting. In imporan respecs, he online learn-ing markeplace reflecs he diversiy o A merican higher educaion isel.12
As resisan as some may si ll be even o hink abou seeking produciv iy gainsin order o reduce eaching coss, here is simply no denying he need o lookmore closely han ever beore a he relaion beween cerain oupus (approxi-maed, or example, by degrees conerred) and inpus (he mix o labor and
capial ha defines educaional producion uncions).13
I is essenial ha helimied resources available o higher educaion be used as effecively as possible.For hese reasons, he research repored here is concerned wih boh educaionaloucomes and coss, seen as wo blades o he scissors.
Organization of This Report
Te nex secion o his repor describes a wo-year effor we have made o esrigorously he learning oucomes achieved by a prooype ineracive learningonline course delivered in a hybrid mode (wih some ace-o-ace insrucion) onpublic universiy campuses in he Norheas and Mid-Alanic. Beore preseningour findings, we devoe space o explaining our randomizaion mehodology
boh because he findings can only be undersood agains he backdrop o he
mehodology and because he research design may be o independen ineres osome readers.14Tis secionwhich conains he resuls o he main par o ourresearchis ollowed by a brieer discussion o he poenial cos savings hacan conceivably be achieved by he adopion o hybrid-orma online learningsysems. We explain why we avor using a cos simulaion approach o esimaepoenial savings, bu we relegae o Appendix B he highly provisional resuls
we obained by employing one se o assumpions in a cos simulaion model. Weend he main body o he repor wih a shor conclusion ha considers barrierso he adopion o online learning sysems ha are ruly ineracive, seps hamigh be aken o overcome such barriers, and he need o ake a sysem-wideperspecive in addressing hese exremely imporan issues.
12 As Henry Bienen (president emeritus of Northwestern and chairman of the board of Rasmussen College, a
for-profit unive rsit y, as well as chairman of ITHAKA) points out, f or many institutions seeking to address the
needs of adult learners and others who are not candidates for places in traditional colleges and universi-
ties, there is no choice: online education, in some form, is the only way that many people can acquire more
skills and earn a college degree, the returns on which have skyrocketed in the past three dec ades. But online
education is also increasingly common in colleges and universities that educate traditional students. It is
seen as a revenue-generating force in many institutions, both four-year and two-year and both public and
private. See Barriers to Adoption of Online Learning Systems in U.S. Higher Education by Bacow et al.13 Some argueand we heartily agreethat the output of higher education has broader dimensions and
includes both research results and also the contribution that the entire system of higher education makes
to the effective functioning of a democratic society. But it will not do to allow emphasis on these larger (and
hard-to-measure) contributions to obscure the need to look carefully, and with a somewhat skeptical eye, at how
effectively institutions utilize resources to achieve straightforward aims such as improving graduation rates.
14 Readers interested in methodology may be especially interested in Appendix C to this report, which contains
a detailed discussion of lessons learned from our experience in carrying out this complicated research
project. We wish only that we had had access to this recitat ion of what to do and what not to do be fore we
started on this adventure! We learned many of these lessons the hard way.
8/12/2019 Sr Ithaka Interactive Learning Online at Public Universities
10/53
Interactive Learning Online at Public Universities: Evidence from Randomized Trials May 22, 2012 9
Educational Outcomes in Public Universities
Te firs and mos ambiious par o our research was direced a assessing heeducaional oucomes associaed wih wha we erm ineracive learning onlineor ILO. By ILO we reer o highly sophisicaed, ineracive online coursesin which machine-guided insrucion can subsiue or some (hough no usu-ally al l) radiional, ace-o-ace insrucion. Course sysems o his ype akeadvanage o daa colleced rom large numbers o sudens in order o offer eachsuden cusomized insrucion, as well as allow insrucors o rack sudensprogress in deail so ha hey can provide heir sudens wih more argeed andeffecive guidance. As several leaders o higher educaion made clear o us inpreliminary conversaions, absen real ev idence abou learning oucomes here isno possibiliy o persuading mos radiional colleges and universiies, and espe-cially hose regarded as hough leaders, o push hard or he inroducion o ILOechnologies ha begin o subsiue machine-guided insrucion or radiionalorms o eaching in appropriae setings.
We se ou o provide a leas enaive answers o hese quesions:
Can sophisicaed, ineracive online courses be used o mainain or improvebasic learning oucomes (masery o course conen, compleion raes, andime-o-degree)?
Are hese courses as effecive, or possibly more effecive, or minoriy andlow-socioeconomic-saus sudens and or oher groups subjec o sereoypehrea?
Are hey equal ly effecive wih no-so-well-prepared sudens and well-pre-pared sudens?
Are hey equal ly effecive in a var iey o campus setingscommuniy col-leges versus our-year colleges, commuer colleges versus colleges wih more
sudens in residence?
Research Design
In hinking abou research design, we began by looking closely a exisingresearch. Tere have been lierally housands o sudies o online learning, buunorunaely he grea majoriy are deficien in one way or a noherofen orreasons beyond he conrol o he principal invesigaors. 15Very ew look direclya he eaching o large inroducory courses in basic fields a major public univer-siies, where he grea majoriy o undergraduae sudens pursue eiher associaeor baccalaureae degrees, presumably because very ew ILO courses have been
15 A detailed summary of existing research has been compiled by our staff (especially Lack); but it is too lengthy
to include here. See Current Status of Research on Online Learning in Postsecondary Education by Bowen
and Lack.
The most ambitious part of our
research was directed at assessing
the educational outcomesassociated with what we term
interactive learning online or
ILO. By ILO we refer to highly
sophisticated, interactive online
courses in which machine-guided
instruction can substitute for some
(though not usually all) traditional,
face-to-face instruction.
8/12/2019 Sr Ithaka Interactive Learning Online at Public Universities
11/53
Interactive Learning Online at Public Universities: Evidence from Randomized Trials May 22, 2012 10
offered in hese setings.16Very ew o he sudies use randomized assignmenechniques o creae reamen and conrol groups ha can be used o reduceoherwise ubiquious selecion effecs ha make i hard o inerpre findings.
o overcome hese limiaions, we decided o work wih seven insances o aprooype ILO saisics course a six public universiy campuses (including woseparae courses in wo deparmens on one campus) ha agreed o cooperae ina careu lly designed research projec uilizing random assignmen echniques.
wo o hese campuses are par o he Sae Universiy o New York (SUNY);wo are par o he Universiy o Maryland; and wo are par o he Ciy Univer-siy o New York (CUNY). Te individual campuses involved in his sudy were,rom SUNY, he Universiy a A lbany and SUNY Insiue o echnology; romhe Universiy o Maryland, he Universiy o Maryland, Balimore Couny andowson Universiy; and, rom CUNY, Baruch College and Ciy College. Teseven courses, wih heir all 2011 enrollmens, are shown in able 1.
We also atemped o include hree communiy colleges in New York andMaryland. We were ulimaely unable o include daa rom hese campuses inour sudy or several reasons. A one o he hree communiy colleges, muliplechanges in leadership compromised he implemenaion o he randomizedresearch proocol. A he second communiy college, a large number o sudyparicipans never ook he course, and among hose who did, almos a quar-er swiched ino a orma differen rom he one o which hey were randomlyassigned. Addiionally, daa on final exam and sandardized es scores wereunavailable or a subsanial proporion o his campus sudy paricipans. A hehird communiy college, much o he daa were provided oo lae o incorporaeino our primary analysis. We srongly cauion readers agains assuming ha hefindings repored here or our-year colleges necessarily apply o communiy col-leges. Vigorous effors nowihsanding, we were unable o obain hard evidenceon his key quesion.
16 Our focus on students attending public institutions is not meant to denigrate the importance of either the
private non-profit sector o r the for-profit sector. Nor is it meant to denigrate professional programs aimed
at working adults. But it is the public colleges and universities, which educate more than three-quarters of
undergraduates at degree-granting institutions (according to the College Boards 2011 report, cited above),
that face the most consequential challenges in raising attainment rates and closing achievement gaps while
simultaneously reducing costs and restraining tuition increases.
8/12/2019 Sr Ithaka Interactive Learning Online at Public Universities
12/53
Interactive Learning Online at Public Universities: Evidence from Randomized Trials May 22, 2012 11
TABLE 1. PARTICIPATING COURSES/INSTITUTIONS, FALL 2011
Course Enrollment Study Participants
Institution A 850 97
Institution B 877 229
Institution C 235 92
Institution D 86 16
Institution E, Department 1 337 31
Institution E, Department 2 473 50
Institution F 188 90
Total 3,046 605
Notes: Study participants are students who consented to be in our study and were randomly assigned to a traditional or
hybrid format of the introductory statistics class.
We do no claim ha hese six campuses are a sais ical ly valid sample o evenpublic higher educaion, never mind all o higher educaion. Bu his se o si xdoes include: (a) major urban universiies wih large commuing populaionso sudens, as well as universiies wih more residenial sudens; and (b) largenumbers o minoriy sudens and sudens rom low-socioeconomic-saus ami-
lies (as shown in ables 2 and 3). Tus, he populaion o insiuions and su-dens in he sudy is boh large enough and diverse enough o allow us o exploremos o he quesions lised above in he conex o our-year public insiuions.
More specifically, his research was designed o es as rigorously as possiblehe learning effeciveness o a paricular ineracive saisics course developeda Carnegie Mellon Universiy (CMU)viewed as a prooype o oher ILO
The population of institutions
and students in the study is both
large enough and diverse enoughto allow us to explore most of
the questions listed above in
the context of four-year public
institutions.
8/12/2019 Sr Ithaka Interactive Learning Online at Public Universities
13/53
Interactive Learning Online at Public Universities: Evidence from Randomized Trials May 22, 2012 12
coures.17While he CMU course can be delivered in a ully onl ine environmen,in his sudy i was used in a hybrid mode in which mos o he insrucion wasdelivered hrough he ineracive online maerials, bu he online insrucion
was supplemened by a one-hour-per-week ace-o-ace session in which sudenscould ask quesions or be given a rgeed assisance.
Te exac research proocol varied by campus in accordance wih local policies,pracices, and preerences, and we describe hese proocols in deail in Appendix
able A1, and on Ihaka S+Rs websie where here is a narraive descripion;Appendix able A1 also presens summar y daa on enrollmens and secion sizesin each orma (ofen he hybrid-orma secions were somewha smaller han heradiional-orma secions). Te general procedure ollowed was: 1) a or beorehe beginning o he semeser, sudens regisered or he inroducory sais-ics course were asked o paricipae in our sudy, and modes incenives wereoffered;182) sudens who consened o paricipae filled ou a baseline survey;3) sudy paricipans were randomly assigned o ake he class in a radiional orhybrid orma; 4) sudy paricipans were asked o ake he CAOS es o sais i-cal lieracy19a he beginning o he semeser; and 5) a he end o he semeser,
17 We prefer the ILO acronym to others, including the OLI acronym used by CMU to stand for Open LearningInitiative. The term ILOfor interactive learning onlineis not specific to CMUs suite of courses, and
ILO emphasizes the interactive features of this kind of online learning. This is in contrast with more com-
mon types of online learning which largely mimic classroom teaching without taking advantage of the unique
online environment to provide added value, that is, anything beyond t hat which can be achieved in a physi-
cal classroom.
The CMU statistics course (which can be accessed at http://oli.web.cmu.edu/openlearning/) includes textual
explanations of concepts and an inventory of worked examples and practice problems, some of which require
the students to manipulate data for themselves using a statistical software package. Both the statistics
course and other courses in the OLI suite were originally intended to be comprehensive enough to allow stu-
dents to learn the material independently without the guidance of an instructor; since it was developed, how-
ever, the statistics course has been used at a variet y of higher education institutions, sometimes in a hybrid
mode. (Taylor Walsh describes the history of the development o f this course, which was financed largely by
the Hewlett Foundation over a number of years, in her 2010 book Unlocking the Gates: How and Why Leading
Universities Are Opening Up Access to Their Courses,Princeton University Press, 2010.) Among the main
strengths of the CMU statistics course is its ability to embed interactive assessments into each instructional
activity, and its three key feedback loops: system to student, as the student answers questions; system to
teacher, to inform student-instructor interactions; and system to course developer, to identify aspec ts of the
course that can be improved. In addition to of fering assessments to measure how well students understand a
particular concept, the CMU course also asks students to complete self-assessments, to give the instruct or
and/or learning scientists a sense of how well students think they understand t he concept. However, while
instructors can delete and re-order modules, CMU does not offer much oppor tunity for customization, nor is
the course adaptive in terms of redirect ing students to extra prac tice sessions or additional reading if their
incorrect answers indicate that they do not understand a concept and need more help. Thus, although the
CMU statistics course is cer tainly impressive, we refer to it as a proto type because we believe it is an early
representative of what will likely be a wave of even more sophisticated systems in the not-too-distant future.18 See Appendix A for a description of the research protocol and incentives used on each campus.
19 The CAOS test, or Comprehensive Assessment of Outcomes in Statistics, is a 40-item multiple-choice
assessment designed to measure students sta tistical literacy and reasoning skills. One characteristic of
the CAOS test is that (for a variet y of reasons) scores do not increase by a large amount over the course
of the semester. Among students in our study who took t he CAOS test at both the beginning and end of the
semester, the average score increase was 5 percentage points. For more information about the CAOS test,
see https://app.gen.umn.edu/artist/caos.html, or delMas, Robert, Joan Gar field, Ann Ooms, and Beth Chance,
Assessing Students Conceptual Understanding After a First Course in Statistics, 6.2 (2007): 28-58,
accessed July 28, 2010, http://www.stat.auckland.ac.nz/~iase/serj/SERJ6(2)_delMas.pdf.
http://oli.web.cmu.edu/openlearning/https://app.gen.umn.edu/artist/caos.htmlhttp://www.stat.auckland.ac.nz/~iase/serj/SERJ6%282%29_delMas.pdfhttp://www.stat.auckland.ac.nz/~iase/serj/SERJ6%282%29_delMas.pdfhttps://app.gen.umn.edu/artist/caos.htmlhttp://oli.web.cmu.edu/openlearning/8/12/2019 Sr Ithaka Interactive Learning Online at Public Universities
14/53
Interactive Learning Online at Public Universities: Evidence from Randomized Trials May 22, 2012 13
sudy paricipans were asked o ake he CAOS es o saisical lieracy again,as well as complee anoher quesionnaire. Appendix able A2 provides henumbers o sudens on each campus who were randomized ino each ormaand who compleed each daa collecion insrumen.
Adminisraive daa on paricipaing and non-paricipaing sudens were gah-ered rom he paricipaing ins iuions daabases. Te baseline survey admin-isered o sudens included quesions on sudens background characerisics,
such as socioeconomic saus, as well as heir prior ex posure o saisics andhe reason or heir ineres in possibly aking he saisics course in a hybridorma. Te end-o-semeser survey asked quesions abou heir experiences inhe saisics course. Sudens in sudy-affiliaed secions o he saisics courseook a final exam ha included a se o iems ha were idenical across all heparicipaing secions a ha campus (or, in he case o he campus ha had wodeparmens paricipaing in he sudy, all paricipaing secions in ha depar-men). Te scores o sudy paricipans on his common porion o he exam
were provided o he research eam, along wih background adminisraive daaand final course grades o all sudens (boh paricipans and, or comparisonpurposes, nonparicipans) enrolled in he saisics course in he all 2011
semeser. All o hese daa are described in deail on he Ihaka S+R websie,which also includes copies o he survey insrumens.
Our inenion was o provide a rigorous side-by-side comparison o specificlearning oucomes or sudens in his hybrid version o he saisics course andcomparable sudens in a radiionally-augh version o he same course. Werecognize, however, ha while we were reasonably successul in randomizingsudens beween reamen and conrol groups (see documenaion in he nexsecion o his repor), we could no randomize insrucors in eiher group andhus could no conrol or differences in eacher qualiy.20Tis is one reason,among ohers, ha we do no regard he research design o his projec as
20 Instructor surveys reveal that, on average, the instructors in traditional format sections were much more
experienced than their counterparts teaching hybrid-format sections (median years of teaching experience
was 20 and 5, respectively). Moreover, almost all of the instructors in the hybrid-format sections were using
the CMU online course for either the first or second time, whereas many of the instructors in the traditional-
format sec tions had taug ht in this mode for years. The experience-advantage, therefore, is clear ly in favor
of the teachers of the tradit ional-format sections. The questionnaires also revealed that a number of the
instructors in hybrid-format sections began with negative perceptions of online learning. In part for these
reasons, a leader of one of the se ts of institutions in this study believes that results for the hybrid-format
sections would be improved vis--vis results in the traditional-format sections if the exper iment were
repeated and instructors in the hybrid-format sections were better motivated and better trained. But this is,
of course, a conjecture.
Our intention was to provide a
rigorous side-by-side comparison
of specic learning outcomesfor students in this hybrid
version of the statistics course
and comparable students in a
traditionally-taught version of the
same course. However, while we
were reasonably successful in
randomizing students between
treatment and control groups, we
could not randomize instructorsin either group and thus could not
control for differences in teacher
quality.
8/12/2019 Sr Ithaka Interactive Learning Online at Public Universities
15/53
Interactive Learning Online at Public Universities: Evidence from Randomized Trials May 22, 2012 14
anyhing close o perec.21Sill, his is he firs effor o which we are awareo carry ou he kind o randomized sudy o oucomes in large inroducorycourses on public universiy campuses ha we hink has been needed.
One wise decision we made was o conduc spring-erm pilos on as many cam-puses as possible in advance o he al l-erm 2011 research phase o he sudy
when we reaed oucomes as suiable or measuremen. Te spring-erm pilosidenified a number o pracical aspecs in which he sudy could be improved,
and a memo on lessons learned rom he spring-erm pilos is included in hisrepor as Appendix C.22
I remains only o add ha, as Appendix C illusraes, his is verydifficulresearch o do, in large par because so many deailshow bes o presenhe course, o recrui suden and aculy paricipans, o randomize sudens
beween reamen and conrol groups, o collec good daa including back-ground inormaion abou he suden paricipans, and o saisy InsiuionalReview Board requiremens in a imely wayneed o be worked ou wih heday-o-day involvemen o campus saff no direcly responsible o us. We havegrea respec or oher invesigaors who have coped wih hese problems, ofenin setings ouside higher educaion.
Findings
Te grea advanage oindeed, he main moivaion orconducing a ran-domized experimen is ha sudens in he reamen and conrol groups areexpeced o have he same average characerisics, boh observed and unob-served. Te resuls in able 2 indicae ha he randomizaion worked properly inha radiional and hybrid-orma sudens in ac have similar characerisics.Tere are a handul o small differences ha are saisically significan bu, ingeneral, he differences beween sudens augh in he radiional orma andsudens augh in he hybrid orma are no meaningul. 23
21 Randomization procedures were limited by the fact that Institutional Review Board (IRB) requirements
precluded randomization of students enrolled in the course without their consent. Instead, we had f irst to
use incentives to encourage students to participate in the study, with the understanding that they would then
be randomized between t reatment and control groups. We were able, however, to compare the character-
istics of participants and non-part icipants, and the two groups turned out to be very similar; see Table 3.
The study is, of course, limited in that it involves only a single course, but having a common hybrid course
across the six campuses (i.e. the CMU statistics course) controls fo r one source of variance in outcomes.
We deliberately chose the CMU statist ics course because we think that the greatest near-term opport unity
to take advantage of interactive online technologies is in introductory-level courses that serve large student
populations in fields in which there is more or less one right answer to most questions. Somewhat different
pedagogies would be needed, we suspect, in courses that are more value-laden and dependent on discussion
of various perspectives.22 We are indebted to James Kemple, now Executive Director of the Research Alliance for New York City Public
Schools, and formerly the Director of t he K-12 Education Policy division at MDRC, for much useful advice. D r.
Kemple has long experience with randomized trials. Lessons learned from the pilot s included how to present
the project, the ef fective use of modest incentives for participants, and techniques that could improve ran-
domization. We hope that others will benefit fr om our experience (see Appendix C) in mounting this research
project.
23 A regression of format assignment on all of the variables listed in Table 2 (and institution dummies) fails to
reject the null hypothesis of zero coef ficients for all variables (except the institution dummies) with p=0.12.
A Hotelling test fails to reject the null of no difference in means with p=0.27.
8/12/2019 Sr Ithaka Interactive Learning Online at Public Universities
16/53
Interactive Learning Online at Public Universities: Evidence from Randomized Trials May 22, 2012 15
In addiion o esing he success o our effors o randomize sudens, able 2also serves o describe he populaion o sudens who par icipaed in our sudy.Tey are a very diverse group. Hal o he sudens come rom amilies w ihincomes less han $50,000 and hal are firs-generaion college sudens. Fewerhan hal are whie, and he group is abou evenly divided beween sudens wihcollege GPAs above and below 3.0. Mos sudens are o radiional college-goingage (younger han 24), are enrolled ull-ime, and are in heir sophomore or
junior year.
Tese sudens are a diverse group, bu do hey resemble he enire populaiono sudens enrolled in he inroducory sais ics courses included in our sudy?Sudy paricipans were randomly assigned o a secion orma, bu he sudyparicipans hemselves are a sel-seleced populaionbecause o InsiuionalReview Board requiremens only sudens who agreed o be in he sudy wererandomly assigned, and scheduling complicaions also l imied he populaiono paricipans. Overall, 605 o he 3,046 sudens enrolled in hese saisicscourses paricipaed in he sudy. An even larger sa mple size would have beendesirable, bu he logisical challenges o scheduling a leas wo secions (onehybrid secion and one radiional secion) a he same ime, so as o enable
sudens in he sudy o atend he saisics course regardless o heir (random-ized) orma assignmen, resriced our prospecive paricipan pool o helimied number o paired ime slos available. Also, as already noed, Insi-uional Review Boards required suden consen in order or researchers orandomly assign hem o he radiional or hybrid orma. No surprisingly, somesudens who were able o make he paired ime slos eleced no o paricipaein he sudy. All o hese complicaions nowihsanding, our final sample o 605sudens is by no means smalli is in ac quie large in he conex o his ypeo research.24
24 Of the 46 studies examined in the Means et al. (2009) meta-analysis, only 5 had sample sizes of over 400,
and of the 51 independent effect sizes the authors abs tracted, 32 came from studies with fewer than 100
study participants.
The students who participated
in our study are a very diverse
group. Half of the students come
from families with incomes less
than $50,000, and half are rst-
generation college students. Fewer
than half are white.
8/12/2019 Sr Ithaka Interactive Learning Online at Public Universities
17/53
Interactive Learning Online at Public Universities: Evidence from Randomized Trials May 22, 2012 16
TABLE 2. RANDOMIZATION OF STUDY PARTICIPANTS
Traditional Hybrid Adj. Diff. Sig?
Male 46% 39% -7% +
Asian 24% 23% -1%
Black 14% 14% 0%
Hispanic 20% 14% -5% +
White 41% 46% 4%
Other/Missing 1% 3% 2%
Average Age 21.9 22.0 0.0
Age
8/12/2019 Sr Ithaka Interactive Learning Online at Public Universities
18/53
Interactive Learning Online at Public Universities: Evidence from Randomized Trials May 22, 2012 17
TABLE 3. STUDENT CHARACTERISTICS BY STUDY PARTICIPATION
Participant Non-Part. Adj. Diff. Sig?
Male 42% 44% -1%
Asian 23% 17% 1%
Black 14% 13% 0%
Hispanic 17% 10% 3% *
White 44% 47% 6% *
Other/Missing 2% 13% -10% **
Average Age 21.9 21.6 -0.3
Age
8/12/2019 Sr Ithaka Interactive Learning Online at Public Universities
19/53
Interactive Learning Online at Public Universities: Evidence from Randomized Trials May 22, 2012 18
Te resuls in able 3 indicae ha he 605 sudy paricipans, while no ul lyrepresenaive o all saisics sudens in any ormal sense, have broadly simi-lar characerisics. Tere are sais ically significan differences beween sudyparicipans and non-paricipans on several characerisics, bu mos o hedifferences are small in magniude. For example, paricipans are more likelyo be enrolled ull-ime, bu only by a margin o 90 versus 86 percen. Courseoucomes are also broadly similar, wih paricipans earning similar grades and
being only sl ighly less likely o complee and pass he course as compared onon-paricipans.
Our analysis o he daa is sraighorward; we compare he oucomes o su-dens randomly assigned o he radiional orma o he oucomes o sudensrandomly assigned o he hybrid orma. In a small number o cases4 perceno he 605 sudens in he sudyparicipans atended a differen orma sec-ion han he one o which hey were randomly assigned. In order o preservehe randomizaion procedure, we associaed sudens wih he secion y pe o
which hey were randomly assigned. Tis is someimes called an inen o reaanalysis. Under cerain assumpions, he effec o acually ak ing he course inhe hybrid orma (as opposed o jus being randomly assigned o do so) can be
calculaed by increasing our esimaes by 4 percen.25
Tis is someimes calledhe reamen on he reaed esimae, which in our sudy is very similar o heinen o rea esimae because mos sudens ook he course in he orma o
which hey were randomly assigned.
How did learning oucomes compare across he reamen and conrol groups?We firs examine he impac o assignmen o he hybrid orma, rela ive o heradiional orma, in erms o he rae a which sudens compleed and passedhe course, heir perormance on a sandardized es o saisics (he CAOSes), and heir score on a se o final exam quesions ha were he same in hewo ormas.26Our main resuls are summarized in Figure 1 (page 19), and heregression resuls are repored in Appendix able A3. 27We find no saisicallysignifican differences in learning oucomes beween sudens in he radiional and hybrid-orma secions. Hybrid-orma sudens did perorm slighly beterhan radiional-orma sudens on hree oucomes, achieving pass raes ha
25 The key assumption is that being randomly assigned to hybrid or traditional did not have an effect on student
outcomes independent of its e ffect on the format in which students were enrolled. This assumption would
be violated if, for example, students hoped for a certain outcome of the random assignment and were disap-pointed when they did not get their preferred assignment, which in turn caused them to do worse in
the course.
26 All of our results control for course-specific dummy variables, since students were randomized within cours-
es; these variables also control for unobserved student characteristics that are constant within institutions.
However, we obtain similar results when we do not control for institution dummies, as would be expec ted
given that the probability of being assigned to the hybrid section was constant across courses (50%).
27 Note that the pass rate in Figure 1 and Appendix Table A3 cannot be used to calculate the percentage of stu-
dents who failed the course because the non-passing group includes students who never enrolled or withdrew
from the course without receiving a grade.
8/12/2019 Sr Ithaka Interactive Learning Online at Public Universities
20/53
Interactive Learning Online at Public Universities: Evidence from Randomized Trials May 22, 2012 19
wereabou hree percenage poins higher, CAOS scores abou one percenagepoin higher, and final exam scores wo percenage poins higherbu none ohese differences passes radiional ess o saisical significance.28
Figure 1. Effect of Hybrid Format on Student Learning Outcomes
0 10 20 30 40 50 60 70 80%
Traditional
Hybrid
PassRate
CAOSPost-Test
FinalExam
54.7%
56.7%
76.4%
79.7%
46.9%47.5%
Notes: None of the traditional-hybrid differences above were statistically signicant at the 10% level. See Appendix Table
A3 for more information about the results depicted here.
I is imporan o repor ha hese differences (or raher, he lack o saisicallysignifican differences) are ai rly precisely esimaedsee boh he acual coe-ficiens and he small sandard errors o he effec es imaes repored in Appen-dix able A3.29In oher words, we can be quie confiden ha he averageeffecs were in ac close o zero. As we explain shorly, we also find ha he same
basic resuls hold or subgroups, and ha disribuions o key oucomes are verysimilar or boh he reamen and conrol group sudens. One commenaor,Michael S. McPherson, presiden o he Spencer Foundaion, observed ha wha
we have here are quie precisely esimaed zeros.
Ta is, i here had in ac been pronounced differences in oucomes beweenradiional-orma and hybrid-orma groups, i is h ighly likely ha we would
have ound hem.30
Our findings are srik ingly differen in his consequenialrespec rom a hypoheical finding o no significan difference which resuledrom a coefficien o some magniude accompanied by a very large sandard erroror by big differences in he disr ibuions o oucomes. A hypoheical finding ohis k ind would mean, in effec, ha we don know much: ha he rue resulscould be almos anywhere.
28 The effect on CAOS test scores in standard deviation units (using the distribution of the control group) was
0.05. We also examined performance using item-level CAOS post-test data. Specifically, we grouped the 40
items into the 20 items on which delMas et al.s (2007) national sample of s tudents exhibited significant
growth (over the course of a semes ter) and the remaining 20 items. We found similar hybrid-format effec tsfor the two g roups o f items.
29 We cluster standard errors by section in order to capture section-specific shocks to student outcomes (such
as the quality of the instructor). Students who were randomly assigned but never enrolled in the course are
grouped as a section within each course for the purpose of computing clustered st andard errors.
30 Some degree of caution is warranted in interpreting the results for the CAOS post-test because the average
students CAOS score only increased by five percentage points over the course of the semester (among
students who took both the pre-test and the post-test). This may have resulted in part f rom some students
not taking the CAOS test seriously because, in most cases, i t was not part of their grade. However, it is
reassuring that the results for the CAOS test are consistent with results fo r pass rates and final exam scores.
In other words, we can be quite
condent that the average effects
were in fact close to zero. We alsond that the same basic results
hold for subgroups, and that
distributions of key outcomes are
very similar for both the treatment
and control group students.
8/12/2019 Sr Ithaka Interactive Learning Online at Public Universities
21/53
Interactive Learning Online at Public Universities: Evidence from Randomized Trials May 22, 2012 20
Tese findings conrol or suden characeris ics, including race/ehniciy,gender, age, ull-ime versus par-ime enrollmen saus, class year in college,parenal educaion, language spoken a home, and amily income. Tese conrolsare no sricly necessary since sudens were randomly assigned o secion or-ma, bu we include hem in order o increase he precision o our resuls and oconrol or any remaining imbalance in observable characerisics. However, weobain nearly idenical resuls when we do no include hese conrol variables
jus as we would expec given he apparen success o our random assignmenprocedure.
Our resuls are also robus o a variey o alernaive mehodologies used oanalyze he experimenal daa. Tese resuls are repored in Appendix able A4,and one is worh highlighing. A limiaion o our main resuls or CAOS pos-es and final exa m scores is ha we only observe hese oucomes or sudens
who compleed he course and ook hese exams. Tis is unlikely o be a signifi-can limiaion given ha we do no find any significan effecs o secion ormaon course compleion and pass raes. Bu as an addiional check, we assignedsudens or whom we did no observe a CAOS pos-es score heir score on heCAOS pre-esin oher words, we assumed ha heir score did no change over
he course o he semeser. Sudens who did no ake eiher he pre-es or hepos-es were assigned he average pre-es score a heir insiuion. Te resul-ing se o real and impued pos-es scores yielded very similar resuls o hoseobained using only he real daa.
Te lack o differences in mean oucomes beween ormas could mask differ-ences in he disribuion o oucomes.31Figure 2 (page 21) shows ha his is nohe case or CAOS pos-es scores. Te disribuions o scores or radi ionaland hybrid orma sudens are largely similar, alhough scores are sl ighly morespread-ou or hybrid-orma sudens. We obain a similar finding or final examscores (no shown).32(Tis kind o comparison o disribuions is no possible orpass raes, which only ake on a value o 0 or 1 or an individual suden.)
Resuls broken down by individual insiuion (Appendix able A5) do noreveal any noeworhy paterns. Tese resuls are much noisier because heyare based on smaller numbers o sudens, bu hey do no indicae ha hehybrid orma was paricularly effecive or ineffecive a any individual insiu-ionwih he possible excepion o Insiuion F, where coefficiens are posiiveacross all our oucomes, alhough only saisically significan in he case o oneoucome.
31 We are indebted to Stephen Stigler, a professor at the University of Chicago and a member of the ITHAKA
board, for emphasizing to us the importance of considering this question.
32 In general, results that use final exam scores should be interpreted cautiously given limitations in these
exams and their implementation. Some institutions included only a handful of questions that were common
across the sections of the course (and we only use data from the common questions). At one institution,
common questions were administered to some students a fter the end of the semester because the actual
final exam only included common quest ions in t wo out o f six sections . At ano ther ins titut ion, fina l exam data
were not available for the students of two instruct ors (covering three out of six traditional-format sections).
Excluding either or both of these institutions produces similar results, but it should be noted that the e ffect
on final exams in standard deviation units is substantial in size (0.19; see Appendix Table A4) and imprecisely
estimated.
Results broken down by
individual institution do not
reveal any noteworthy patterns.
8/12/2019 Sr Ithaka Interactive Learning Online at Public Universities
22/53
Interactive Learning Online at Public Universities: Evidence from Randomized Trials May 22, 2012 21
Figure 2. Distributions of CAOS Post-Test Scores
Traditional
Hybrid
Traditional
4
Density
CAOS Post-Test Score
Kernel =epanechinov, bandwidth=0.03335
2
0
0 .2 .4 .6 .8
Hybrid
We also calculaed resuls separaely or subgroups o sudens defined in ermso var ious characeris ics, including race/ehniciy, gender, parenal educa-ion, primary language spoken, CAOS pre-es score, hours worked or pay, andcollege GPA. We did no find any consisen evidence ha he hybrid-orma
effec varied by any o hese characerisics (Appendix able A6).33
Tere wereno groups o sudens ha benefied rom or were harmed by he hybrid ormaconsisenly across muliple learning oucomes.
Our main resuls provide compelling evidence ha, on average, sudens learnedjus as much in he hybrid orma as hey would have had hey insead aken hecourse in he radiional ormawih learning measured in radiional ways,in erms o course compleion, course grades, and perormance on a naional eso saisical lieracy. Tis seemingly bland resul is in ac very imporan, in l igho perhaps he mos common reason given by aculy and deans or resising heuse o ILO-ype insr ucion: We worry ha basic suden learning oucomes(pass raes and masery o conen) will be hur, and we won expose our su-
dens o his risk . Te research repored here suggess srongly ha such worriesare no well ounded.
We also examined how much sudens liked he hybrid orma o he course rela-ive o radiional orma sudens (Figure 3 [page 22] and Appendix able A7).
We ound ha sudens gave he hybrid orma a modesly lower overal l rainghan he one given by sudens aking he course in radiional orma (he raing
was abou 11 percen lower). By similar margins, hybrid sudens repored eel ing
33 The one exception is our finding that completion and pass rates were significantly higher in the hybrid course
for st udents with family incomes o f at least $50,000 per year, but not for s tudent s with family inc omes o f
less than $50,000 per year. However, we hesitate to att ach much significance to this result given that we do
not find such a pattern for our other measure of socioeconomic status (parental education) or for measures
of academic preparation. This finding could be the result of random noise in the coef ficients (especially given
the large number of coefficients reported in Appendix Table A6).
8/12/2019 Sr Ithaka Interactive Learning Online at Public Universities
23/53
Interactive Learning Online at Public Universities: Evidence from Randomized Trials May 22, 2012 22
ha hey learned less and ha hey ound he course more difficul. 34Tese hreedifferences, hough modes in size, were saisically significan a he 10 percenlevel. Bu here were no saisically significan differences in sudens repors ohow much he course raised heir ineres in he subjec mater.
Figure 3. Effect of Hybrid Format on Student Evaluations of Course
0.0
0.5
1.0
1.5
2.0
2.5
DifficultyAmountLearned
RaisedInterest
OverallRating
Traditional
Hybrid
Raiting(0
4)
2.3
2.0+
2.0+
2.5+
1.7 1.7
2.2 2.3
Notes: + indicates traditional-hybrid difference is statistically signicant at the 10% level. See Appendix Table A7 for moreinformation about the results depicted here.
A leader o one o he universi ies ha ac ively paricipaed in our sudyopined ha a deec o he CMU prooype course is ha i has no addiciveor Disney-like appeal; i was, as his person pu i, designed by cogniivescieniss (no offense inended!). In conras, some sudens in he radiionalorma may have been reaed o an occasional colorul sory, personal recol-lecions o he insrucor, or oher reamens someimes used by aculy oimprove sudens opinions o heir course. 35
We also asked sudens how many hours per week hey spen outside of class
working on he sais ics class. Hybrid-orma sudens repored spending 0.3hour more each week, on average, han radiional-orma sudens. Tis differ-
34 Students responses to the open-ended questions on the end-of-semester surveys indicate that many
students in the hybrid format would have liked more face-to-face time with t he instructor than one hour
each week; others felt that the instructor could have bet ter used the face-to-face time to make the weekly
sessions more structured and/or helpful in explaining the material and going over concepts students did not
understand. A number of students in the hybrid course also indicated they would have benefited from more
practice problems or examples, and many were frustrated by the difficulty o f checkpoint assessments in the
course and by problems they encountered using the statistical software packages to complete assignments.
35 The question of what is really going on herewith no differences in learning outcomes as measured con-
ventionally combined with a (to be sure, small) difference in qualitative assessmentsrelates to a larger
literature on measured learning outcomes versus more subjective measures of student satisfact ion with
online or hybrid courses relative to their satisfaction with face-to-face courses. Studies (some more rigorous
than others) pert aining to the latter t opic abound; a few examples include: Hannay, Maureen, and Tracy
Newvine. Perceptions of Distance Learning: A Comparison o f Online and Traditional Learning. 2.1 (2006):
1-11. Accessed April 24, 2012. http://jolt.merlot.org/documents/MS05011.pdf; Horspool, Agi, and Carsten
Lange. Applying the Scholarship of Teaching and Learning: Student Perceptions, Behaviours and Success
Online and Face-to-Face. 37.1 (2011): 73-88. Ac cessed April 24, 2012. http://www.tandfonline.com/doi/abs
/10.1080/02602938.2010.496532; and Meyer, Katrina A. Student Perceptions of Face-to-Face and Online
Discussions: The Advantage Goes To . .4: 53-69. Accessed April 24, 2012. http://sloanconsortium.org/
jaln/v11n4/student-percept ions-face-face-and-online-discussions-advantage-goes.
Our results indicate that hybrid-
format students took about
one-quarter less time to achieve
essentially the same learning
outcomes as traditional-format
students.
http://jolt.merlot.org/documents/MS05011.pdfhttp://www.tandfonline.com/doi/abs/10.1080/02602938.2010.496532http://www.tandfonline.com/doi/abs/10.1080/02602938.2010.496532http://sloanconsortium.org/jaln/v11n4/student-perceptions-face-face-and-online-discussions-advantage-goeshttp://sloanconsortium.org/jaln/v11n4/student-perceptions-face-face-and-online-discussions-advantage-goeshttp://sloanconsortium.org/jaln/v11n4/student-perceptions-face-face-and-online-discussions-advantage-goeshttp://sloanconsortium.org/jaln/v11n4/student-perceptions-face-face-and-online-discussions-advantage-goeshttp://www.tandfonline.com/doi/abs/10.1080/02602938.2010.496532http://www.tandfonline.com/doi/abs/10.1080/02602938.2010.496532http://jolt.merlot.org/documents/MS05011.pdf8/12/2019 Sr Ithaka Interactive Learning Online at Public Universities
24/53
Interactive Learning Online at Public Universities: Evidence from Randomized Trials May 22, 2012 23
ence, which is no saisically significan, implies ha, in a course where a radi-ional secion mees or 3 hours each week and a hybrid secion mees or 1 hour,he average hybrid-orma suden would spend 1.7 less hours each week in totaltime devoed o he course, a difference o abou 25 percen. Tis resul is con-sisen wih oher evidence ha ILO-ype ormas do succeed in achieving hesame learning oucomes as radiional-orma insrucion in less imewhichpoenially has imporan implicaions or scheduling and he rae o coursecompleion.36
In sum, our resuls indicae ha hybrid-orma sudens ook abou one-quarerless ime o achieve essenially he same learning oucomes as radiional-ormasudens. Te hree main limiaions o his analysis are: (1) we were no able orandomly assign insrucors o secion ormaswhich would have been di-ficul, i no impossible, o do, especially in a small scale sudy, or o conrol ordifferences in how radiional-orma secions were augh;37(2) he limiaionso he CMU prooype o an ILO courseno cusomizaion and no addiciveeaures; and (3) our inabiliy o repor resuls or communiy colleges. Despiehese limiaions, hese resuls reflec a serious, rigorous, assessmen o he rela-ive efficacy o echnology-enhanced learning (ILO-syle hybrid insrucion)
compared o he radiional mode o insrucion. Tey are, we believe, he besevidence available o dae on an imporan se o quesions. Tere is, w ihoudoub, much more research ha can and should be carried ou bu, a he mini-mum, his sudy suppors a no-harm-done conclusion regarding a leas onecurren prooype o an ILO sysem.
36 The authors of this paper, interested to see whether the hybrid course might enable students to learn the
material in the statistics course in a shorte r period of time, conducted a separate, quasi-experimental study
in the summer of 2011, involving one of the campuses used in our larger fall 2011 study. This summer probe
occurred over two shortened (five or six-week) summer sessions; while we collected a large amount of
data and used numerous controls in our analysis, we did not randomly assign students to the hybrid or theface-t o-face format so that we could obtain lar ger sample sizes (from what was already a much smaller pool
of students taking the course during the summer). The results of the summer study revealed no sta tistically
significant differences in the percentage of students who passed t he course, in final exam grades, or in the
end-of-semester CAOS test results, bet ween the students who took the course in a hybrid format and stu-
dents who took the course in a face-to-face format. A separate study by Marsha Lovett, associate director of
the Eberly Cente r for Teaching Excellence, and her colleagues at CMU produced similar findings. In that study,
which was conducted at CMU, the performance of students who were randomly assigned to the face-to-face
format of a stat istics course, which met four times a week for 50 minut es each time, was compared w ith
the performance of s tudents who were randomly assigned to a hybrid format, which met twice a week for 50
minutes each time. The researchers found little dif ference in the amount of time students repor ted spending
on the course outside of class each week (about 2.8 hours for the hybrid-format student s, compared with
about 2.7 hours per week for face-to-face students). In addition, students also were able to learn in eight
weeks the same amount of material that students would ordinarily take 15 weeks in a face-to-face formatto learn. In this respect, the use of the CMU course could be said to increase learning efficiency. (For more
information about this study, see Lovet t, Meyer, and Thilles 2008 art icle in the , entitled The Open Learning
Initiative: Measuring the effectiveness of t he OLI statistics course in accelerat ing student learning, available
online at http://jime.open.ac.uk/2008/14.)
37 One commentator, Michael McPherson, noted that right now, quite apart from any use of online technologies,
very different instructional methods are used to teach introductory statistics courses even within the same
universitynever mind across universities. But little ef fort appears to be made to compare learning out-
comes and cost effectiveness across different approachesa point Derek Bok, former president of Harvard
who continues to write on this subject, keeps emphasizing.
http://jime.open.ac.uk/2008/14http://jime.open.ac.uk/2008/148/12/2019 Sr Ithaka Interactive Learning Online at Public Universities
25/53
Interactive Learning Online at Public Universities: Evidence from Randomized Trials May 22, 2012 24
Costs and Potential Savings
I is an open quesion wheher subsequen progress in improving hybrid ILOcourses o his k indin paricular, effors o achieve greaer cusomizaionopporuniies or aculy,38and o make i more un (more addicive) or bohsudens and aculy o use he sysemwill lead o more posiive conclusionsconcerning learning effeciveness. We jus don know. Digial learning is sil l inearly days, and i is enirely possible ha uure versions o hybrid courses wi llno jus mainain he same learning oucomes bu increase suden learning rela-ive o he saus quo. Bu we do know enough now o jusiy careul exploraiono poenial cos savings.
Produciviymeasured as oupus divided by inpushas been increasedby he use o echnology in oher secors o he economy, ofen resuling inincreased oupu. Our muli-campus sudy o learning oucomes in undergradu-ae educaion has shown ha a leading prooype hybrid learning sysem did nolead o a saisically significan increase in oupus (suden learning), bu couldpoenially increase produciviy noneheless by using ewer inpusherebyachieving cos savings. Unil now, echnology-induced produciviy gains in
higher educaion have been aken mainly in he orm o increased oupumoreand aser research, and so on; he ime may be a hand when cos savings should
be sough and emphasized.39
Te cos blade o he scissors is a leas as chal lenging o sudy as he learningblade. Tis may be one reason why so ew sudies have paid much atenion ocoss. (Carol wiggs work wih he Naional Cener or Academic ransorma-ion projec is a noable excepion.40) A firs blush, i would seem o be sraigh-orward o compare he side-by-side coss o he hybrid-online version o hesaisics course and he radi ional version. No so. Our early effors o do jusha were unsuccessul. Te big problem, we learned, is ha conemporaneouscomparisons can be near-useless in projecing seady-sae coss because hecoss o doing almos anyhing or he firs ime are very differen rom he cosso doing he same hing numerous imes. Ta admoniion is especially rue inhe case o online learning. Te cos implicaions o some educaional iner-
venions can be measured immediaely and wih relaively litle difficuly. Forexample, he higher cos associaed wih a decrease in he size o a class is simplyhe cos o he addiional insrucors and space required o accommodae a largernumber o classes wih ewer sudens in each one. Tis cos w ill be more or lesshe same in he firs year he inervenion is implemened as in he enh year.
In he case o hybrid learning, however, here are subsanial sar-up cossha have o be considered in he shor run bu are l ikely o decrease over ime,
hereby making shor-erm coss significanly greaer han long-erm coss. Forexample, he developmen o sophisicaed hybrid courses wil l be a cosly efforha would only be a sensible invesmen i he sar-up coss were eiher paid or
by ohers (oundaions and governmens) or shared by many insiuions and
38 The pervasive desire for more ability to customize is a key finding of Barriers by Bacow et al. (see pages
21-22 in particular).
39 This is another conclusion of the Barriers study by Bacow et al.; see pages 22 and 24-25 in particular.
40 See the National Center for Academic Transformation website at www.thencat.org.
We conceptualize the research
question here not as how much
will institutions save right now by
shifting to hybrid learning? but
rather as under what assumptions
will cost savings be realized, over
time, by shifting to a hybrid format,
and how large are those savings
likely to be?
http://www.thencat.org/http://www.thencat.org/8/12/2019 Sr Ithaka Interactive Learning Online at Public Universities
26/53
Interactive Learning Online at Public Universities: Evidence from Randomized Trials May 22, 2012 25
amorized over ime. Tere are also ransiion coss enailed in moving rom heradiional, mosly ace-o-ace model (ha may, however, employ some elemenso simple online models, such as video-aped lecures or homework assign-mens online) o a hybrid model ha akes advanage o more sophisicaed ILOsysems employing machine-guided insrucion, cogniive uors, embeddedeedback loops, and some orms o auomaed grading. Insrucors need o berained o ake ull advanage o such sysems. Tere may also be conracual lim-is on secion size ha were designed wih he radiional model in mind bu hado no make sense or a hybrid model. I is possible ha hese consrains can bechanged in a nex round o conrac negoiaions, bu ha oo wi ll ake ime.
o overcome (or avoid!) hese problems, we hink here is much o be said orcarrying ou simulaed cos probes, and we made a very rough atemp o do
jus his on hree o he campuses included in he lea rning oucomes par ohe sudy. We concepualize he research quesion here no as how much willinsiuions save righ now by shifing o hybrid learning? bu raher as under
wha assumpions wi ll cos savings be realized, over ime, by shifing o a hybridorma, and how large are hose savings likely o be? Our basic approach waso sar by looking in as much deail as possible a he acual coss o eaching a
basic course in radiional orma (usually, bu no always, he saisics course)in a base year. Ten, we worked wih leaders on hese campuses o simulae heprospecive, seady-sae coss o a hybrid-online version o he same course,looking hree o five years ino he uure. Tese ex ploraory simulaions were
based on explici assumpions, especially abou saffing, ha were incorporaedino spreadsheeswhich in urn allowed us o see how sensiive our resuls
were o variaions in key assumpions. We ocused heavily on personnel coss,because o boh heir imporance and our abiliy o examine hem wih someprecision. Oher coss, including space coss, were also considered. We hopedha he simulaions would, a he minimum, give us a leas a rough sense o hepoenial impac on coss o inroducing hybrid learning and, more specifically,
show us how much room here would be or insiuions o share cos savingswih acul y and sudens on a coninuing basis.
We ocus on insrucor compensaion because hese coss comprise a subsan-ial porion o he recurring cos o eaching and are he mos sraighorwardo measure. Space coss are also an imporan caegory o coss ha are likely o
be reduced by shifing o a hybrid learning model (he mos imporan caegoryin some siuaions), bu such coss are more difficul o measure accuraely ahe level o an individual course. A hybrid model also affords boh aculy andsudens significanly greaer scheduling flexibiliy, a poenially very imporan
benefi ha will no be capured by our simulaions. On he oher hand, here arealso oher y pes o coss ha we do no consider here, such as increases in inor-maion echnology (I) suppor coss associaed wih moving a signi fican shareo learning aciviies online. Such added coss can be ar rom riv ial.
We did exploraory simulaions or wo ypes o radi ional eaching models: (1)a model in which sudens are augh in secions o roughly 40 sudens per sec-ion; and (2) a model in which all sudens atend a common lecure and are henassigned o small discussion secions led by eaching assisans. We compare hecurren coss o each o hese radi ional eaching models o simulaed coss oa hybrid model in which more insrucion is delivered online, sudens atend
Our simulations illustrate that
hybrid learning offers opportunities
for signicant savings incompensation costs, but that the
degree of cost reduction depends
on the exact model of hybrid
learning used.
8/12/2019 Sr Ithaka Interactive Learning Online at Public Universities
27/53
Interactive Learning Online at Public Universities: Evidence from Randomized Trials May 22, 2012 26
weekly ace-o-ace sessions wih par-ime insrucors, and he course is over-seen by a enure-rack proessor (wih adminisraive responsibiliies delegaedo a par-ime insrucor).
We have decided, however, no o presen he acual calculaions and resuls ohese simulaions in he body o his repor. Tey are oo speculaive and subjeco considerable variaion depending on how a paricular campus waned oorganize is eaching. Te danger o specious precision is grea, and i would be
wrong o atach much imporance o paricular numbers. Suffice i o say ha hecrude models we employed sugges savings in compensaion coss ranging rom36 percen o 57 percen in he all-secion model, and 19 percen in he lecure-secion model. Appendix B presens hese resuls and many more calculaionsand figures showing how sensiive poenial savings are as we vary assumpionsabou secion sizes and compensaion.
Tese simulaions illusrae ha hybrid learning offers opporuniies or sig-nifican savings in compensaion coss, bu ha he degree o cos reduciondepends (o course) on he exac model o hybrid learning usedespecially herae a which insrucors are compensaed and secion size. A large share o cossavings is derived rom shifing away rom ime spen by expensive proessors
oward boh machine-guided insrucion ha saves on saffing coss overall andoward ime spen by less expensive saff in Q&A setings. O course, enuredproessors canno be laid off in order o realize hese savings and, in any case,orce reducions are no required o save significan amouns o money. Insiu-ions ha ace pressures o expand enrollmen are in an especially good posiiono realize savings by shifing he mix o eaching ime. When more sudens areo be augh, hybrid models make his possible wihou increasing he demandsmade on enured aculy. Recruimen coss may hereby be reduced along wihcompensaion coss per suden, and debaes over mainaining commimenso exising aculy are avoided. Over ime, cerainly, saff size can be aleredhrough atriion. A lso, he ime o proessors can be reallocaed oward smaller,more advanced classeswhich many preer o each (such reallocaions mayno save he insiuion money, hough hey may improve he overall educaionalexperience o many sudens).
In hese simulaions, we have assumed ha he number o sudens in he coursewill remain consan. However, as already suggesed, many insiuions aceincreasing demand or places in heir classes. Te hybrid learning model is veryatracive in such circumsances or wo primary reasons: (a) less space is neededin general; and (b) hybrid courses provide boh sudens and eachers wihgreaer scheduling flexibiliy. Increased enrollmen can a lso lead o increasedcompensaion cos savings (per suden) because he fixed coss o he proessor
in charge o he course, and an adminisraive coordinaor, would be spread overa larger number o sudens. For he same reason, he larges savings w ill be real-ized in courses wih he larges enrollmen, all else equal.
Our simulaion approach underesimaes subsanially he savings rom mov-ing oward a hybrid model in many setings because we do no accoun or spacecoss. We are relucan o pu a dollar figure on space coss because capial cossare difficul o apporion accuraely down o he course level. However, i is moresraighorward o calculae he percenage change in he need or classroom
8/12/2019 Sr Ithaka Interactive Learning Online at Public Universities
28/53
Interactive Learning Online at Public Universities: Evidence from Randomized Trials May 22, 2012 27
space ha would resul rom shi fing oward a hybrid model. Te hybrid coursemees or one hour each week, whereas he radiional course ypically meesor hree or our hours each week. Consequenly, he hybrid course requires 67percen o 75 percen less classroom use han he radiional course, assumingha he course is augh in secions, ha secion size is held consan, and hahe hybrid course does no have addiional space requiremens o is own, such asaddiional compuer labs.
In he shor run, insiuions canno sell or demolish heir buildings. However,in he long run, using hybrid models or some large inroducory courses wouldallow insiuions o expand enrollmen wihou a commensurae increase inspace cossa major cos savings (cos avoidance) relaive o wha insiuions
would have had o spend had hey sayed wih a radiional model o insrucion.An imporan poin here is ha he hybrid model need no jus save