Post on 25-Jan-2015
description
transcript
Ari Bader-Natal
Ari Bader-Natalari@grockit.com
February 28, 2011
Evolvinga Learning Analytics Platform
Thomas Lotzethomas@grockit.com
Ari Bader-Natalari@grockit.com
@AriBaderNatal
Ari Bader-Natal
Ari Bader-Natal
Evolving a Learning Analytics System @AriBaderNatal
http://grockit.com
Questions to ask when evolving a platform:
Ari Bader-Natal
Evolving a Learning Analytics System @AriBaderNatal
http://grockit.com
The first question
Questions to ask when evolving a platform:
Ari Bader-Natal
Evolving a Learning Analytics System @AriBaderNatal
http://grockit.com
The first question
The interesting questions
Questions to ask when evolving a platform:
Ari Bader-Natal
Evolving a Learning Analytics System @AriBaderNatal
http://grockit.com
The first question
The interesting questions
The infrastructure question
Questions to ask when evolving a platform:
Ari Bader-Natal
Evolving a Learning Analytics System @AriBaderNatal
http://grockit.com
The first question
The interesting questions
The infrastructure question
The subset question
Questions to ask when evolving a platform:
Ari Bader-Natal
Evolving a Learning Analytics System @AriBaderNatal
http://grockit.com
The first question
The interesting questions
The infrastructure question
The subset question
The missing data question
Questions to ask when evolving a platform:
Ari Bader-Natal
Evolving a Learning Analytics System @AriBaderNatal
http://grockit.com
The first question
The interesting questions
The infrastructure question
The subset question
The missing data question
The pro-active question
Questions to ask when evolving a platform:
Ari Bader-Natal
Evolving a Learning Analytics System @AriBaderNatal
http://grockit.com
The first question
The interesting questions
The infrastructure question
The subset question
The missing data question
The pro-active question
The effort question
Questions to ask when evolving a platform:
Ari Bader-Natal
Evolving a Learning Analytics System @AriBaderNatal
http://grockit.com
The first question
The interesting questions
The infrastructure question
The subset question
The missing data question
The pro-active question
The effort question
The discovery question
Questions to ask when evolving a platform:
Ari Bader-Natal
Evolving a Learning Analytics System @AriBaderNatal
http://grockit.com
The first question
The interesting questions
The infrastructure question
The subset question
The missing data question
The pro-active question
The effort question
The discovery question
The production questions
Questions to ask when evolving a platform:
Ari Bader-Natal
Evolving a Learning Analytics System @AriBaderNatal
http://grockit.com
The first question
The interesting questions
The infrastructure question
The subset question
The missing data question
The pro-active question
The effort question
The discovery question
The production questions
The audience question
Questions to ask when evolving a platform:
Ari Bader-Natal
Evolving a Learning Analytics System @AriBaderNatal
http://grockit.com
The context questionsWhat is Grockit? Where? Why? How?
Ari Bader-Natal
Evolving a Learning Analytics System Ari Bader-Natal
http://grockit.com
Ari Bader-Natal
Evolving a Learning Analytics System Ari Bader-Natal
http://grockit.com
E-Learning
Computer-Supported Collaborative Learning
Intelligent Tutoring Systems
Ari Bader-Natal
Evolving a Learning Analytics System Ari Bader-Natal
http://grockit.com
Ari Bader-Natal
Evolving a Learning Analytics System Ari Bader-Natal
http://grockit.com
System control(adaptive/IRT assessment)
Learner control(custom-specified study)
System control(adaptive study)
Asynchronous Coordination
Synchronous Coordination
Ari Bader-Natal
Evolving a Learning Analytics System @AriBaderNatal
http://grockit.com
Social and motivational value of having a cohort of peers:
• opportunities to ask for assistance (just-in-time guidance)
• opportunities for discussion (both on-task and off-task)
• opportunities to earn recognition (for assisting others)
Ari Bader-Natal
Evolving a Learning Analytics System @AriBaderNatal
http://grockit.com
Social and motivational value of having a cohort of peers:
• opportunities to ask for assistance (just-in-time guidance)
• opportunities for discussion (both on-task and off-task)
• opportunities to earn recognition (for assisting others)
Ari Bader-Natal
Evolving a Learning Analytics System Ari Bader-Natal
http://grockit.com
Ari Bader-Natal
Evolving a Learning Analytics System Ari Bader-Natal
http://grockit.com
Study plans
Learning goals
Ari Bader-Natal
Evolving a Learning Analytics System Ari Bader-Natal
http://grockit.com
Ari Bader-Natal
Evolving a Learning Analytics System Ari Bader-Natal
http://grockit.com
Recognition of achievements
Recognition from peer learners
Indicator of helpfulness
Ari Bader-Natal
Evolving a Learning Analytics System Ari Bader-Natal
http://grockit.com
The first questionWhat are people actually doing?
Ari Bader-Natal
Evolving a Learning Analytics System Ari Bader-Natal
http://grockit.com
Just look around in the database.
Ari Bader-Natal
Evolving a Learning Analytics System Ari Bader-Natal
http://grockit.com
Just look around in the database.
•select count(*) from learners;
Ari Bader-Natal
Evolving a Learning Analytics System Ari Bader-Natal
http://grockit.com
Just look around in the database.
•select count(*) from learners;
•select * from discussions;
Ari Bader-Natal
Evolving a Learning Analytics System Ari Bader-Natal
http://grockit.com
Just look around in the database.
•select count(*) from learners;
•select * from discussions;
•select * from responses;
Ari Bader-Natal
Evolving a Learning Analytics System Ari Bader-Natal
http://grockit.com
Just look around in the database.
•select count(*) from learners;
•select * from discussions;
•select * from responses;
•select avg(time) from responses
Ari Bader-Natal
Evolving a Learning Analytics System Ari Bader-Natal
http://grockit.com
The interesting questionsExploratory Data Analysis...
Ari Bader-Natal
Evolving a Learning Analytics System @AriBaderNatal
http://grockit.com
Ari Bader-Natal
Evolving a Learning Analytics System @AriBaderNatal
http://grockit.com
What is effective? What is engaging?
Ari Bader-Natal
Evolving a Learning Analytics System @AriBaderNatal
http://grockit.com
What is effective? What is engaging?
• How does frequency and duration of discussions during group study sessions differ between high school students and post-college learners?
Ari Bader-Natal
Evolving a Learning Analytics System @AriBaderNatal
http://grockit.com
What is effective? What is engaging?
• How does frequency and duration of discussions during group study sessions differ between high school students and post-college learners?
• Do students who work in groups spend more time on task than those who choose to work alone?
Ari Bader-Natal
Evolving a Learning Analytics System @AriBaderNatal
http://grockit.com
What is effective? What is engaging?
• How does frequency and duration of discussions during group study sessions differ between high school students and post-college learners?
• Do students who work in groups spend more time on task than those who choose to work alone?
• Which of the various interventions within Grockit lead to the largest learning gains?
Ari Bader-Natal
Evolving a Learning Analytics System @AriBaderNatal
http://grockit.com
What is effective? What is engaging?
• How does frequency and duration of discussions during group study sessions differ between high school students and post-college learners?
• Do students who work in groups spend more time on task than those who choose to work alone?
• Which of the various interventions within Grockit lead to the largest learning gains?
• Do peer-awarded points motivate more discussion in small group settings?
Ari Bader-Natal
Evolving a Learning Analytics System @AriBaderNatal
http://grockit.com
What is effective? What is engaging?
• How does frequency and duration of discussions during group study sessions differ between high school students and post-college learners?
• Do students who work in groups spend more time on task than those who choose to work alone?
• Which of the various interventions within Grockit lead to the largest learning gains?
• Do peer-awarded points motivate more discussion in small group settings?
• How does an Item Response Theory model compare to a Knowledge Tracing model, in terms of predictive accuracy or responses?
Ari Bader-Natal
Evolving a Learning Analytics System @AriBaderNatal
http://grockit.com
What is effective? What is engaging?
• How does frequency and duration of discussions during group study sessions differ between high school students and post-college learners?
• Do students who work in groups spend more time on task than those who choose to work alone?
• Which of the various interventions within Grockit lead to the largest learning gains?
• Do peer-awarded points motivate more discussion in small group settings?
• How does an Item Response Theory model compare to a Knowledge Tracing model, in terms of predictive accuracy or responses?
• Are video explanations more effective than written explanations?
Ari Bader-Natal
Evolving a Learning Analytics System @AriBaderNatal
http://grockit.com
Ari Bader-Natal
Evolving a Learning Analytics System @AriBaderNatal
http://grockit.com
Question:
Ari Bader-Natal
Evolving a Learning Analytics System @AriBaderNatal
http://grockit.com
Question:
Are the questions currently available appropriate for the current population of learners?
Ari Bader-Natal
Evolving a Learning Analytics System @AriBaderNatal
http://grockit.com
Question:
Are the questions currently available appropriate for the current population of learners?
• Are there enough hard questions?
Ari Bader-Natal
Evolving a Learning Analytics System @AriBaderNatal
http://grockit.com
Question:
Are the questions currently available appropriate for the current population of learners?
• Are there enough hard questions?
• Are there enough easy questions?
Ari Bader-Natal
Evolving a Learning Analytics System @AriBaderNatal
http://grockit.com
Question:
Are the questions currently available appropriate for the current population of learners?
• Are there enough hard questions?
• Are there enough easy questions?
Actionable:
Ari Bader-Natal
Evolving a Learning Analytics System @AriBaderNatal
http://grockit.com
Question:
Are the questions currently available appropriate for the current population of learners?
• Are there enough hard questions?
• Are there enough easy questions?
Actionable:
If there isn’t a good fit, ask the content authors to start focusing on writing what is currently most needed.
Ari Bader-Natal
Evolving a Learning Analytics System @AriBaderNatal
http://grockit.com
The infrastructure questionHow can we streamline analysis?
Ari Bader-Natal
Evolving a Learning Analytics System @AriBaderNatal
http://grockit.com
Ari Bader-Natal
Evolving a Learning Analytics System @AriBaderNatal
http://grockit.com
collection
Ari Bader-Natal
Evolving a Learning Analytics System @AriBaderNatal
http://grockit.com
collection
selection
Ari Bader-Natal
Evolving a Learning Analytics System @AriBaderNatal
http://grockit.com
collection
selection
analysis
Ari Bader-Natal
Evolving a Learning Analytics System @AriBaderNatal
http://grockit.com
collection
selection
analysis
visualization
Ari Bader-Natal
Evolving a Learning Analytics System @AriBaderNatal
http://grockit.com
collection
selection
analysis
visualization
distribution
Ari Bader-Natal
Evolving a Learning Analytics System @AriBaderNatal
http://grockit.com
collection
selection
analysis
visualization
distribution
Ari Bader-Natal
Evolving a Learning Analytics System @AriBaderNatal
http://grockit.com
collection
selection
analysis
visualization
distribution
Ari Bader-Natal
Evolving a Learning Analytics System @AriBaderNatal
http://grockit.com
collection
selection
analysis
visualization
distribution
Ari Bader-Natal
Evolving a Learning Analytics System @AriBaderNatal
http://grockit.com
collection
selection
analysis
visualization
distribution
Ari Bader-Natal
Evolving a Learning Analytics System @AriBaderNatal
http://grockit.com
Ari Bader-Natal
Evolving a Learning Analytics System @AriBaderNatal
http://grockit.com
The subset questionHow can we ask the same question of different subsets of the available data?
Ari Bader-Natal
Evolving a Learning Analytics System @AriBaderNatal
http://grockit.com
Example views
Ari Bader-Natal
Evolving a Learning Analytics System @AriBaderNatal
http://grockit.com
Multiple views at once.
Ari Bader-Natal
Evolving a Learning Analytics System @AriBaderNatal
http://grockit.com
Multiple views at once.
Subject (e.g. Algebra)
Question
Game
Question Session
User (e.g. Ari)
Game Session
Response Chat Message
Ari Bader-Natal
Evolving a Learning Analytics System @AriBaderNatal
http://grockit.com
Multiple views at once.
Subject (e.g. Algebra)
Question
Game
Question Session
User (e.g. Ari)
Game Session
Response Chat Message
Ari Bader-Natal
Evolving a Learning Analytics System Ari Bader-Natal
http://grockit.com
Ari Bader-Natal
Evolving a Learning Analytics System @AriBaderNatal
http://grockit.com
Ari Bader-Natal
Evolving a Learning Analytics System @AriBaderNatal
http://grockit.com
The missing data questionWhat about actions that leave no trace?
Ari Bader-Natal
Evolving a Learning Analytics System @AriBaderNatal
http://grockit.com
Ari Bader-Natal
Evolving a Learning Analytics System @AriBaderNatal
http://grockit.com
The pro-active questionHow can we test out new hypotheses?
The effort questionHow can we reduce friction in
experimentation and hypothesis testing?
Ari Bader-Natal
Evolving a Learning Analytics System Ari Bader-Natal
http://grockit.com
http://xkcd.com/552/
Ari Bader-Natal
Evolving a Learning Analytics System Ari Bader-Natal
http://grockit.com
R. Kohavi, R. Henne, and D. Sommerfield. Practical guide to controlled experiments on the web. In Proceedings of the 13th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM, 2007.
http://bit.ly/exp-long
http://bit.ly/exp-short
Eric Ries: The one line split-test, or how to A/B test all the time.
http://bit.ly/one-line-split
Ari Bader-Natal
Evolving a Learning Analytics System Ari Bader-Natal
http://grockit.com
Ari Bader-Natal
Evolving a Learning Analytics System Ari Bader-Natal
http://grockit.com
InitialQuestionsShould new students be given easier questions when playing alone,
until they answer at least 5 questions correctly?This test is being assigned and evaluated for new users only.
EasyFirst NormalSubsequent activity (1657 persons) (1721 persons) (difference)
Participated in further study 73.2% (1213) 67.5% (1161) (2.6% to 8.9%) **Participated in group study 31.4% (520) 27.0% (464) (1.3% to 7.5%) *
Logged in again later 30.8% (510) 28.4% (489)Participated in group discussion 22.1% (366) 18.4% (316) (1% to 6.5%) *
Reviewed past questions 13.9% (230) 12.8% (221)
Ari Bader-Natal
Evolving a Learning Analytics System @AriBaderNatal
http://grockit.com
The discovery questionHow can we deliver findings effectively?
Ari Bader-Natal
Evolving a Learning Analytics System @AriBaderNatal
http://grockit.com
Ari Bader-Natal
Evolving a Learning Analytics System @AriBaderNatal
http://grockit.com
Ari Bader-Natal
Evolving a Learning Analytics System @AriBaderNatal
http://grockit.com
The production questionsScalability? Performance? Reliability?
Ari Bader-Natal
Evolving a Learning Analytics System @AriBaderNatal
http://grockit.com
Ari Bader-Natal
Evolving a Learning Analytics System @AriBaderNatal
http://grockit.com
The audience questionHow can this help teachers and learners?
Ari Bader-Natal
Evolving a Learning Analytics System Ari Bader-Natal
http://grockit.com
Ari Bader-Natal
Evolving a Learning Analytics System Ari Bader-Natal
http://grockit.com
Performance (per-skill)
Ari Bader-Natal
Evolving a Learning Analytics System @AriBaderNatal
http://grockit.com
My unresolved questionIs evolving the right approach?
Ari Bader-Natal
Evolving a Learning Analytics System @AriBaderNatal
http://grockit.com
YAGNI: “You Ain’t Gonna Need It”
Ron Jeffries: “Always implement things when you actually need them, never when you just foresee that you need them.”
Ari Bader-Natal
Evolving a Learning Analytics System @AriBaderNatal
http://grockit.com
In the age of #BigData, how do you choose tools for analysis before your dataset is undeniably Big? Start with Big tools or YAGNI?
YAGNI: “You Ain’t Gonna Need It”
Ron Jeffries: “Always implement things when you actually need them, never when you just foresee that you need them.”
Ari Bader-Natal
Evolving a Learning Analytics System @AriBaderNatal
http://grockit.com
Your unresolved questions???
Ari Bader-Natal
Thanks!
Ari Bader-Natalari@grockit.com
February 28, 2011
Evolvinga Learning Analytics Platform
@AriBaderNatal
Thomas Lotzethomas@grockit.com