40
From Data to Information in Online and Blended Learning Research Chuck Dziuban Patsy Moskal Research Initiative for Teaching Effectiveness University of Central Florida
From Data to Information in Online and Blended Learning Research
Chuck Dziuban Patsy Moskal
Research Initiative for Teaching EffectivenessUniversity of Central Florida
Some of Chuck and Patsy’s Chapters
• Data Analysis
• Scholarship of Teaching and Learning
• Longitudinal Evaluation
• Big Data
• The Future
Data to InformationResearch
and Data
Analysis
Interpretation
JudgmentInformation
Data Analysis
What do you do when someone asks:
“How large does my sample have to be?”
Hypothesis testing answers this question
What is the chance of my sample data
The null hypothesis is true?
WHEN
H0
Statistical (Classical) Hypothesis Tests are a Function of 3 Things:
1) Significance Level .05? .01? …or something else?
2) Sample Size
Tiny? Small? Medium? Large? Huge?3) Some Effect Size
A difference that means something to me∆1 –Doesn’t matter∆2 –Really important to me
How much is enough?
∆2I don’t care about this
I care about this
∆1
Statistical Significance Testing (SD = 15)Sample
Size
27502500225020001750150012501000750500
x1=100x2=101ES=.06
.01
.02
.03
.04
.05
.07
.10
.14
.20
.29
So the strategy is…
1) Pick ∆2 first à This is important to me
2) Then pick a significance level .05, .01, or something else
3) Pick a sample size that will catch ∆2 but not ∆1
Data Analysis Resources
• National Research Center for Distance Education and Technological Advancements (DETA) https://uwm.edu/deta/
• Ferguson, G. A., & Takane, Y. (1981). Statistical analysis in psychology and education (5th ed.). New York: McGraw-Hill.
• Nie, N. H. (1975). SPSS: Statistical package for the social sciences (2nd ed.). New York: McGraw-Hill.
• Practical Assessment, Research and Evaluation (PARE).
Scholarship of Teaching and Learning(SoTL)
So…what is SoTL?
• Involves…• Systematic reflection of the
teaching process• Research• Dissemination of the teaching
process and its impact on student learning
A research context for SoTL
But do they want us in their world?• Tim Brown – Communications• Amanda Groff – Anthropology
Are students interested in class tweets?
Not really
Prefer official channel, e-mails, CMS
Brown & Groff (2011)
Word clouds online• Beatriz Reyes-Foster – Anthropology• Can word clouds help with concept formation?
deNoyelles & Reyes-Foster (2014)
Susan B. Anthony
John Lewis
For more information on SoTL…
Journals that publish SoTL
Longitudinal Evaluation
Longitudinal Evaluation
• Advantages• Allows for examination of trends over time• Can potentially follow cohorts• Aids in continuous quality improvement• Helps identify in “near real” time
• Challenges• Requires time and continuity!
Repeated studies, carried out over a period of time
Students’ Desktop vs Laptop use from 2006-2009
71.0%
59.4%
49.7%44.0%
65.4%
72.8%
82.1%88.3%
40.0%
50.0%
60.0%
70.0%
80.0%
90.0%
100.0%
2006 2007 2008 2009N= (13,641) N= (12,861) N= (11,730) N= (10,180)
Desktop
Laptop
Note: Adapted from The ECAR study of undergraduate students and information technology, 2009,by S. D. Smith, G. Salaway, J. B. Caruso, & R. N. Katz, 2009, Boulder, CO: ECAR, p. 43
Student Mobile Technology Use & Importance: 2012-2014
ECAR Study of Students and Information Technology (Dahlstrom & Bichsel, 2014)
Student Perceptions of Instruction:Overall Excellent Ratings
50 49 52 53 54 5454 53 55 56 57 54
0
20
40
60
80
100
2009 2010 2011 2012 2013 2014
F2F (n=1,044,164) Blended (n=94,045)
Student Perception of Instruction:The Form Items
• Organization of course • Explanation of requirements, grading, & expectations• Communication• Respect• Stimulation of interest• Creation of learning environment • Feedback• Aid in student success • Overall effectiveness of instructor• + 2 open-ended questions
Achieve course objectives
Create learning environment
Then...The probability of an overall rating of Excellent = .99
If...
A decision rule for the probability of a faculty member receiving an overall rating of Excellent
Communicate ideas
n=58,156
Excellent Very Good Good Fair Poor
A comparison of excellent ratings by course modalityunadjusted & adjusted for instructors satisfying the rule (n=431,261)
Course Modality Overall % Excellent
If Rule 1 % Excellent
Blended 55 99
Fully Online 57 99
F2F 54 99
Lecture Capture 51 99
Blended LC 48 99
Longitudinal Resources
• 2015 Online Report Card - Tracking Online Education in the United States• http://onlinelearningconsortium.org/read/online-report-card-tracking-
online-education-united-states-2015/• ECAR Student and Technology Research Study
• https://library.educause.edu/resources/2016/6/2016-student-and-technology-research-study
• Pew Research Center on Internet, Science and Technology• http://www.pewinternet.org/
Big Data
The OldSTATISTICSSampling Estimation Hypothesis
Testing
The New
BIG DATAModeling Prediction Machine
Learning
Big Data and Statistical Thinking
Blended(n=53,476)Face to Face (n=726,342)Online (n=121,257)
Blended – Face to Face Blended – Online Online – Face to Face
Modality Mean P
Modality Bonferroni Effect Size
4.194.114.10
P = .000
= .000= .006= .013
.075
.093
.009
Possibilities of Big Data• Identify strong and weak relationships• Develop useful if-then decision rules• Conduct network analysis• Identify affiliated classification groups• Discover patterns• Detect underlying clusters• Develop association rules• Construct new variables• Work with several variables simultaneously
Big Data Resources• Levitt, Steven D., and Stephen J Dubner. Think Like a
Freak: The Authors of Freakonomics Offer to Retrain Your Brain. First edition. William Morrow, an imprint of HarperCollinsPublishers, 2014. *
• Lawson, J. (2015). Data science in higher education: Step-by-step machine learning for institutional researchers.
• Silver, N. (2012). The signal and the noise: Why so many predictions fail--but some don't. New York: Penguin Press.*
The Future
• Changing baselines• The speed of light• Complexity• Uncertain mediation• Being wrong• Beware of false positives• Research context• Collaboration
The Future Resources• Complexity Academy http://complexityacademy.io/
• Schulz, K. (2010). Being wrong: Adventures in the margin of error. New York: Ecco. *
• Mullainathan, S., & Shafir, E. (2014). Scarcity: Why having too little means so much. Picador.
• Floridi, L. (2014). The fourth revolution: How the infosphere is reshaping human reality. Oxford University Press. *
Questions?
Research Initiative for Teaching Effectiveness
For more information contact:
Dr. Chuck Dziuban(407) 823-5478
Dr. Patsy Moskal(407) 823-0283
http://rite.ucf.edu
