• Ways of Seeing LearningLearning Analytics for Learners

Mary Loftus Michael G MaddenNUI Galway NUI Galwaymary.loftus@nuigalway.ie michael.madden@nuigalway.ie@marloft

• The authors acknowledge the support of Ireland’s Higher Education Authority through the IT Investment Fund and ComputerDISC in NUI Galway.

https://youtu.be/0pDE4VX_9Kk?t=1m21s

And today…

• Data, Artificial Intelligence & Machine Learning are having a similar effect across society

• This revolution is not only showing us to ourselves in new ways – it is shaping how we live.

Overview of this Presentation• Why Ways of Seeing & What is Learning Analytics

• What kind of Education do we want to have for our children?

• How will AI, Machine Learning & Data Analytics change our world?

• Why try to Measure Learning? What are there downsides?

• Data is political – how do we use it fairly & avoid unintended consequences?

• White-box Algorithms and transparency

• Research Values & Questions

• Bayesian Networks, Open Learner Models & Data Gathering

• Current Research Status & Timeline

Learning Analytics – a Definition• “Learning analytics is the measurement, collection, analysis

and reporting of data about learners and their contexts, for purposes of understanding and optimising learning and the

environments in which it occurs”• Call for Papers of the 1st International Conference on Learning Analytics & Knowledge (LAK 2011)

Learning Sciences

Data Mining

Data Visualization

Psychology

Central to education’s purpose is “the coming into presence of unique individual beings”Education “spaces might open up for uniqueness to come into the world” – Biesta, G. J. J. (2015). Good Education in an Age of Measurement: Ethics, Politics, Democracy. Routledge.

Learning Analytics – The Story So Far• Predicting student outcomes - identifying ‘at-risk students’

• Personalisation of student learning

• Multi-modal analytics – analyses of audio, video, location data

• Discourse and writing analytics

• Measuring ‘student engagement’ & disengagement

• Levels of Learning Analytics:

• Teachers

• Course

• Institution

• National

If we can make learning more visible, can we...

• Reduce the need for formal testing and examinations?

• Do more problem-based learning & assessment?

• Provide more formative feedback for students?

• Model student’s conceptual understanding?

• Support metacognition?

Ethics & Other Tensions in Learning Analytics • In the next few slides, I want to examine the values that I am forming

as a researcher that will shape this research:

• Student Vulnerability & Agency

• Measurement as a powerful but double-edged tool

• Recognition of data and algorithms as powerful political tools

• The potential for Unintended Consequences when we build algorithmic systems in social systems

Student Vulnerability, Agency, and Learning Analytics

• Prinsloo & Slade examine how we:

• decrease student vulnerability,

• increase student agency,

• empower students as participants in learning analytics

• moving students from quantified data objects to qualified and qualifying selves

• “In light of increasing concerns about surveillance, higher education institutions (HEIs) cannot afford a

simple paternalistic approach to student data”• Prinsloo & Slade (2016)

Measuring Gets Results – But Care Needs to be Taken…

• When we measure, we can clearly see improvement and impact

• However, the Hawthorne effect is a kind of "tell-me-what-you-measure-I-will-tell-you-how-people-react-to-it" effect

• Drucker is often quoted as saying: “What gets measured gets managed” – but Demming said: “Eliminate management by objective. Eliminate management by numbers, numerical goals. Substitute leadership”

Data is Political

• We need to take care in our research to ensure fairness and not replicate societal biases and discrimination

•There is nothing about doing data analysis that is neutral. What and how data is collected, how the data is cleaned

and stored, what models are constructed, and what questions are asked — all of this is political.

dana boyd (2017)

Unintended Consequences…

• Criminal Justice Systems – discriminating on the basis of race?

• Employment Screening – address, age, gender?

• Advertising – different ads served depending on race? (Sweeney 2013)

• Even if humans are there as a ‘final check’, there is potential for ‘Moral Crumple Zones (Elish 2016)

Most data analysis makes prejudicial decisions as part of clustering without having any understanding of the people or properties that they are using. It’s merely math! But that math — and the decisions that are determined by it — have serious social ramifications.

dana boyd (2017)

White-Box Algorithms & Transparency

• Students (and citizens) need to be able to ‘see into’ algorithms that impact their opportunities and quality of life.

The problem with contemporary data analytics is that we’re often categorizing people without providing human readable

descriptors.

dana boyd (2017)

“Action can never manifest through a predictable, deterministic series of consequences, since the subject, by acting, is placed within a complicated web of relationships which cannot be predicted before hand. In the same sense, Action is irreversible.”

Hannah Arendt

“For apart from inquiry, apart from the praxis, individuals cannot be truly human.

Knowledge emerges only through invention and re-invention, through the restless, impatient, continuing, hopeful inquiry human beings pursue in the world, with the world, and with each other.”

Paulo Freire

Research Values

• Grounded in the Student perspective• Students as owners of their learning data• Links learning analytics to learning design• Machine Learning with an emphasis on white-box

modelling & visibility as well as prediction• Data literacy capacity building for students

Research Questions

1. Can a Learning Analytics system provide an interface for students to engage in metacognitive activities around their own learning, thereby improving individual learning?

2. Can we retool an existing learning analytics system using machine learning, modelling and classifiers to provide this metacognitive interface to students?

3. Can such a system help students visualize, track and reflect on their own learning and development goals and help them to improve performance?

4. Metacognition

1. Student Actions

2. Bayesian Models & Open

Learning Models

3. Student Reflection

Bayesian Networks

Millán et al (2010)

Example of Simple Bayesian Network

Madden et al (2008)

Open Learner Models

Bull et al (2016)

Data Gathering

Kitto et al (2016)

Current Research Status: Messy!

• Qualitative Research – talking to students

• Student Data being gathered from systems

• Identifying Data Models for this data and trying them for size

Research Timeline

• Literature Review

• Research Questions

Ethical Approval

• Data Modelling• Qualitative

Research

Data Gathering • Share improved

models with students

• Assess impact

Write up

2017 2018 2019

References• Biesta, G. J. J. (2015). Good Education in an Age of Measurement: Ethics, Politics, Democracy. Routledge.

• boyd, danah. (2017, April 12). Toward Accountability. Retrieved 18 April 2017, from https://points.datasociety.net/toward-accountability-6096e38878f0

• Bull, S., Ginon, B., Boscolo, C., & Johnson, M. (2016). Introduction of learning visualisations and metacognitive support in a persuadable open learner model. In Proceedings of the Sixth International Conference on Learning Analytics & Knowledge (pp. 30–39). ACM. Retrieved from http://dl.acm.org/citation.cfm?id=2883853

• Elish, M. C. (2016). Moral Crumple Zones: Cautionary Tales in Human-Robot Interaction (We Robot 2016) (SSRN Scholarly Paper No. ID 2757236). Rochester, NY: Social Science Research Network. Retrieved from https://papers.ssrn.com/abstract=2757236

• Millán, E., Loboda, T., & Pérez-de-la-Cruz, J. L. (2010). Bayesian networks for student model engineering. Computers & Education, 55(4), 1663–1683. https://doi.org/10.1016/j.compedu.2010.07.010

• Sweeney, L. (2013). Discrimination in Online Ad Delivery. Queue, 11(3), 10:10–10:29. https://doi.org/10.1145/2460276.2460278

• Madden, Michael G. (NUI, Galway), Lyons, William and Kavanagh, Ita (Limerick Institute of Technology).“A Data-Driven Exploration of Factors Affecting Student Performance in a Third-Level Institution”, Proceedings of AICS-2008: 19th Irish Conference on Artificial Intelligence and Cognitive Science, Cork, August 2008.

• Other refs in Abstract

Thank youMary Loftus

@marloft

mary.loftus@nuigalway.ie