From Data to Information in Online and Blended Learning Research

Chuck Dziuban Patsy Moskal

Research Initiative for Teaching EffectivenessUniversity of Central Florida

Some of Chuck and Patsy’s Chapters

• Data Analysis

• Scholarship of Teaching and Learning

• Longitudinal Evaluation

• Big Data

• The Future

Data to InformationResearch

and Data

Analysis

Interpretation

JudgmentInformation

Data Analysis

What do you do when someone asks:

“How large does my sample have to be?”

Hypothesis testing answers this question

What is the chance of my sample data

The null hypothesis is true?

WHEN

H0

Statistical (Classical) Hypothesis Tests are a Function of 3 Things:

1) Significance Level .05? .01? …or something else?

2) Sample Size

Tiny? Small? Medium? Large? Huge?3) Some Effect Size

A difference that means something to me∆1 –Doesn’t matter∆2 –Really important to me

How much is enough?

∆2I don’t care about this

I care about this

∆1

Statistical Significance Testing (SD = 15)Sample

Size

27502500225020001750150012501000750500

x1=100x2=101ES=.06

.01

.02

.03

.04

.05

.07

.10

.14

.20

.29

So the strategy is…

1) Pick ∆2 first à This is important to me

2) Then pick a significance level .05, .01, or something else

3) Pick a sample size that will catch ∆2 but not ∆1

Data Analysis Resources

• National Research Center for Distance Education and Technological Advancements (DETA) https://uwm.edu/deta/

• Ferguson, G. A., & Takane, Y. (1981). Statistical analysis in psychology and education (5th ed.). New York: McGraw-Hill.

• Nie, N. H. (1975). SPSS: Statistical package for the social sciences (2nd ed.). New York: McGraw-Hill.

• Practical Assessment, Research and Evaluation (PARE).

Scholarship of Teaching and Learning(SoTL)

So…what is SoTL?

• Involves…• Systematic reflection of the

teaching process• Research• Dissemination of the teaching

process and its impact on student learning

A research context for SoTL

But do they want us in their world?• Tim Brown – Communications• Amanda Groff – Anthropology

Are students interested in class tweets?

Not really

Prefer official channel, e-mails, CMS

Brown & Groff (2011)

Word clouds online• Beatriz Reyes-Foster – Anthropology• Can word clouds help with concept formation?

deNoyelles & Reyes-Foster (2014)

Susan B. Anthony

John Lewis

For more information on SoTL…

Journals that publish SoTL

Longitudinal Evaluation

Longitudinal Evaluation

• Advantages• Allows for examination of trends over time• Can potentially follow cohorts• Aids in continuous quality improvement• Helps identify in “near real” time

• Challenges• Requires time and continuity!

Repeated studies, carried out over a period of time

Students’ Desktop vs Laptop use from 2006-2009

71.0%

59.4%

49.7%44.0%

65.4%

72.8%

82.1%88.3%

40.0%

50.0%

60.0%

70.0%

80.0%

90.0%

100.0%

2006 2007 2008 2009N= (13,641) N= (12,861) N= (11,730) N= (10,180)

Desktop

Laptop

Note: Adapted from The ECAR study of undergraduate students and information technology, 2009,by S. D. Smith, G. Salaway, J. B. Caruso, & R. N. Katz, 2009, Boulder, CO: ECAR, p. 43

Student Mobile Technology Use & Importance: 2012-2014

ECAR Study of Students and Information Technology (Dahlstrom & Bichsel, 2014)

Student Perceptions of Instruction:Overall Excellent Ratings

50 49 52 53 54 5454 53 55 56 57 54

0

20

40

60

80

100

2009 2010 2011 2012 2013 2014

F2F (n=1,044,164) Blended (n=94,045)

Student Perception of Instruction:The Form Items

• Organization of course • Explanation of requirements, grading, & expectations• Communication• Respect• Stimulation of interest• Creation of learning environment • Feedback• Aid in student success • Overall effectiveness of instructor• + 2 open-ended questions

Achieve course objectives

Create learning environment

Then...The probability of an overall rating of Excellent = .99

If...

A decision rule for the probability of a faculty member receiving an overall rating of Excellent

Communicate ideas

n=58,156

Excellent Very Good Good Fair Poor

A comparison of excellent ratings by course modalityunadjusted & adjusted for instructors satisfying the rule (n=431,261)

Course Modality Overall % Excellent

If Rule 1 % Excellent

Blended 55 99

Fully Online 57 99

F2F 54 99

Lecture Capture 51 99

Blended LC 48 99

Longitudinal Resources

• 2015 Online Report Card - Tracking Online Education in the United States• http://onlinelearningconsortium.org/read/online-report-card-tracking-

online-education-united-states-2015/• ECAR Student and Technology Research Study

• https://library.educause.edu/resources/2016/6/2016-student-and-technology-research-study

• Pew Research Center on Internet, Science and Technology• http://www.pewinternet.org/

Big Data

The OldSTATISTICSSampling Estimation Hypothesis

Testing

The New

BIG DATAModeling Prediction Machine

Learning

Big Data and Statistical Thinking

Blended(n=53,476)Face to Face (n=726,342)Online (n=121,257)

Blended – Face to Face Blended – Online Online – Face to Face

Modality Mean P

Modality Bonferroni Effect Size

4.194.114.10

P = .000

= .000= .006= .013

.075

.093

.009

Possibilities of Big Data• Identify strong and weak relationships• Develop useful if-then decision rules• Conduct network analysis• Identify affiliated classification groups• Discover patterns• Detect underlying clusters• Develop association rules• Construct new variables• Work with several variables simultaneously

Big Data Resources• Levitt, Steven D., and Stephen J Dubner. Think Like a

Freak: The Authors of Freakonomics Offer to Retrain Your Brain. First edition. William Morrow, an imprint of HarperCollinsPublishers, 2014. *

• Lawson, J. (2015). Data science in higher education: Step-by-step machine learning for institutional researchers.

• Silver, N. (2012). The signal and the noise: Why so many predictions fail--but some don't. New York: Penguin Press.*

The Future

• Changing baselines• The speed of light• Complexity• Uncertain mediation• Being wrong• Beware of false positives• Research context• Collaboration

The Future Resources• Complexity Academy http://complexityacademy.io/

• Schulz, K. (2010). Being wrong: Adventures in the margin of error. New York: Ecco. *

• Mullainathan, S., & Shafir, E. (2014). Scarcity: Why having too little means so much. Picador.

• Floridi, L. (2014). The fourth revolution: How the infosphere is reshaping human reality. Oxford University Press. *

Questions?

Research Initiative for Teaching Effectiveness

For more information contact:

Dr. Chuck Dziuban(407) 823-5478

Charles.Dziuban@ucf.edu

Dr. Patsy Moskal(407) 823-0283

Patsy.Moskal@ucf.edu

http://rite.ucf.edu