Scaling Up Learning Analytics
Rebecca Ferguson, The Open University, UK
The Open University (OU)
• The Open University: largest in UK• Distance university• Making use of big data for 45 years• Informal learning: iTunes, YouTube…• MOOCs on FutureLearn,
OpenLearn and elsewhere• Learning analytics research and events• LACE project
2
open.ac.uk
http://www.laceproject.eu
Learning analytics
3
solaresearch.org
…the measurement, collection, analysis and
reporting of data about learners and their contexts,
for purposes of understanding and optimizing
learning and the environments in which it occurs.
Educators use analytics to…
• Monitor the learning process• Explore student data• Identify problems• Discover patterns• Find early indicators for success• Find early indicators for poor marks or drop-out• Assess usefulness of learning materials• Increase awareness, reflect and self reflect• Increase understanding of learning environments• Intervene, supervise, advise and assist• Improve teaching, resources and the environment
4
Dyckhoff, A. L., Lukarov, V., Muslim, A., Chatti, M. A., & Schroeder, U. (2013). Supporting Action Research with Learning Analytics. Paper presented at LAK13.
Learners use analytics to…
• Monitor their own activities and interactions• Monitor the learning process• Compare their activity with that of others• Increase awareness, reflect and self reflect• Improve discussion participation• Improve learning behaviour• Improve performance• Become better learners• Learn!
5
Dyckhoff, A. L., Lukarov, V., Muslim, A., Chatti, M. A., & Schroeder, U. (2013). Supporting Action Research with Learning Analytics. Paper presented at LAK13.
Analytics at scale: UK schools
6
• Aligned with clear aims
• Huge and sustained effort
• Agreed proxies for learning
• Clear and standardised visualisation
• Driving behaviour at every level BUT
• Stressed, unhappy learners• Analytics with little value for learners or educators• Omission of key areas, such as collaboration
Analytics at scale: Course SignalsDeveloped at Purdue University, USA
7
Arnold, K. E., & Pistilli, M. (2012). Course Signals at Purdue: Using Learning Analytics To Increase Student Success. Paper presented at LAK12, Vancouver, Canada.
Developing institutional strengthsThe OU is developing its capabilities in 10 key areas
8
The university needs world class capability in data science to continually mine the data and build rapid prototypes of simple tools, and a clear pipeline for the outputs to be mainstreamed into operations
We need to ensure we have the right architecture and processes for collecting the right data and making it accessible for analytics
– we need a ‘big data’ mind-set
Benefits will be realised through existing business processes
impacting on students directly and through enhancement of
the student learning experience – we will develop an ‘analytics
mind-set’ in these areas
The strategic roadmap will build these
capabilities prioritised using the indicators and
drivers of student success
Easily accessible OU dataLearning design and analytics at the OU
9
Relating design and outcomesLearning design and analytics at the OU
10
Innovaating in technology-enhanced learning (TEL)
The TEL Complex
11
The many elements of the ‘TEL Complex’ must all be taken into account
as an innovation is designed, developed
and embedded
Scanlon, E., Sharples, M., Fenton-O'Creevy, M., Fleck, J., Cooban, C., Ferguson, R., Cross, S. & Waterhouse, P. 2013. Beyond Prototypes. London: TEL Programme.
Rapid Outcomes Modelling Approach (ROMA)
The ROMA Framework
12
Ferguson, R., Macfadyen, L., Clow, D., Tynan, B., Alexander, S., & Dawson, S.. (2015). Setting learning analytics in context: overcoming the barriers to large-scale adoption. Journal of Learning Analytics, 1(3), 120-144.
Adapted from: Young, J., & Mendizabal, E. (2009). Helping researchers become policy entrepreneurs: How to develop engagement strategies for evidence‐based policy‐making. ODI Briefing Papers. London, UK: ODI.
Define (and redefine)
your policy objectives
What does success look like?
13
Academic analytics can guide future change
Student perspectives●Overall, I am satisfied with the quality of this module●Overall, I am satisfied with my study experience●I would recommend this module to other students●I was satisfied with the support provided by my tutor on this module●I enjoyed studying this module●This module met my expectations
Academic perspectives●The students were well prepared●The students met specified learning outcomes●The students defined and achieved their own learning goals
University perspectives●The module enhanced the university’s reputation●The module aligned well with others●The module generated income
What does success look like?
●Students demonstrate the skills necessary to network, collaborate, browse and reflect
●Students show progress towards defined learning outcomes●Students communicate well… when asked to collaborate●Students access and share links… when encouraged to browse●Students return to materials... when encouraged to reflect●Students engage with course content●Students seek out new challenges●Students persist when the work is challenging●Students persist in the face of failure●Students ask for help… when they are stuck
after several attempts●Students compare their learning strategies with those of experts●Students adapt their learning strategies to resemble those of experts
14
Learning analytics help to identify appropriate interventions
Policy objectivesOU Strategic Analytics Investment Programme
15
VisionTo use and apply information strategically to retain students and enable them to progress and achieve their study goals.
This vision requires• Discursive changes
to the communication of data and analytics
• Procedural changes in how learners are supported
• Behavioural changes associated with sustainable change in learner support.
Define (and redefine)
your policy objectives
Political contextMapping people and processes
16Tynan, B. & Buckingham Shum, S. (2013). Designing systemic learning analytics at the Open University.
http://www.slideshare.net/sbs/designing‐systemic‐learning‐analytics‐at‐the‐open‐university
Key stakeholdersOU Strategic Analytics Investment Programme
17
Define(and
redefine) your policy objectives
A community of stakeholders working in different areas:• Intervention and Evaluation• Data Usability• Ethics Framework• Predictive Modelling• Learning Experience Data• Professional Data• Student Tools
Key stakeholders are• University administrators• Students• Educators
Desired behaviour changesOU Strategic Analytics Investment Programme
18
Define(and
redefine) your policy objectives
VisionTo use and apply information strategically to retain students and enable them to progress and achieve their study goals.
Desired behaviour changes• Staff will use and apply
information strategically• Students will extend their
learning journeys• Students will complete their
learning journeys• Students will set learning goals• Students will work effectively
towards study goals
Engagement strategyOU Strategic Analytics Investment Programme
19
Define(and
redefine) your policy objectives
• Data in action is provided to stakeholders through a live portal, enabling them to understand learner behaviour and make adjustments and interventions that will have an immediate positive impact.
• Data on action is a more reflective process that takes place after an adjustment or intervention.
• Data for action takes advantage of predictive modelling and innovation in order to isolate particular variables and make changes based on a variety of analysis tools.
Internal capacity to effect changeOU Strategic Analytics Investment Programme
20
Define(and
redefine) your policy objectives
Includes• Recruitment• Capacity building• Developing an ethical
framework for the use of learning analytics.
MonitoringOU Strategic Analytics Investment Programme
21Tynan, B. & Buckingham Shum, S. (2013). Designing systemic learning analytics at the Open University.
http://www.slideshare.net/sbs/designing‐systemic‐learning‐analytics‐at‐the‐open‐university
Policy objectivesUniversity of Technology, Sydney, Australia
22
VisionA university where staff and students understand data and, regardless of its volume and diversity, can use and reuse it, store and curate it, apply and develop the analytical tools to interpret it.
Teaching and learningEnsure that all stakeholders have the capacity to understand and interpret contemporary data‐rich environments.ResearchEnable researchers to think and act differently when designing their research methodologies and practices.AdministrationIdentify opportunities to obtain, generate, visualize, and communicate data and analyses that can improve core business outcomes.UniversityMine existing institutional data to identify areas that can provide direct evidence or assistance to staff and students.
Political contextUniversity of Technology, Sydney, Australia
23Ferguson, R., Macfadyen, L., Clow, D., Tynan, B., Alexander, S., & Dawson, S.. (2015). Setting learning analytics in
context: overcoming the barriers to large-scale adoption. Journal of Learning Analytics, 1(3), 120-144.
• Project initiated and led by Deputy Vice‐Chancellor and Vice‐President (Teaching and Learning)
• Pilot projects were completed successfully to secure ongoing funding.
• Critical to the success of the initial pilot projects was the existence of an Advanced Analytics Institute with internationally regarded researchers in big data, data sciences and analytics sciences.
• This enabled the establishment of a Connected Intelligence Centre.
Key stakeholdersUniversity of Technology, Sydney, Australia
24
Define(and
redefine) your policy objectives
• 190 staff attended a one‐day ‘Data Intensive University Forum’, thus beginning a university‐wide conversation.
• Working party included Deputy Vice‐Chancellors, a senior member of library staff, and representatives of all faculties and administrative areas with relevant expertise
• Stakeholder buy‐in and ongoing participation in the project have been critical to its success.
Desired behaviour changesUniversity of Technology, Sydney, Australia
25
Define(and
redefine) your policy objectives
• Provide information that can be used to decrease student attrition
• Provide a more detailed understanding of factors affecting low pass rates in subjects with very high failure rates over time
• Provide students with more information about their own study and engagement patterns
• Enable a more fine‐grained understanding of the influences of a range of possible interventions on pass rates and completions
• Provide valuable input to learning futures projects
Engagement strategyUniversity of Technology, Sydney, Australia
26
Define(and
redefine) your policy objectives
• Give attention to institutional culture, ensuring engagement and buy‐in from key stakeholders through good communication and governance
• Invest in pilot projects of significant concern to the university and reporting of outcomes
• Invest in infrastructure: tools, applications, services
• Invest in expertise: recruitment of critical staff
• Provide leadership and engage institutional leaders
Internal capacity to effect changeUniversity of Technology, Sydney, Australia
27
Define(and
redefine) your policy objectives
• Students and staff must be sufficiently numerate and equipped to make use of the analyses that analytics projects produce.
• A subject has been developed and trialled with staff
• The course develops students’ ability to engage with complex, extended arguments underpinned by numerical data as a key to participation as informed citizens in issues of significance to our culture and society
• The course available as an elective and will become compulsory
MonitoringUniversity of Technology, Sydney, Australia
28
• UTS has been engaged in a variety of learning analytics projects to assess scale and impact
• For example, the Outreach Programme rings as many commencing undergraduate students as possible. Early results consistently show a significant decrease in attrition in the group of students contacted.
Define(and
redefine) your policy objectives
29
Slides online at www.slideshare.net/R3beccaF
Rebecca Ferguson @R3beccaFhttp://r3beccaf.wordpress.com/
Join the LACE project at the LACE SoLAR Flare on 9 October in Milton Keynes, UK, or online
lanyrd.com/2015/laceflare/