PAZAPA: identifying and supporting at-risk students

Serge VerlindeJordi Heeren

Nathalie NouwenKU Leuven, Leuven Language Institute

2

French proficiency in Flanders is on the decline

French is definitely needed on the labourmarket

uncomfortable position of French language teachers at higher education institutions

Reseach question

3

How toidentify& support at-risk students (for French)

byproviding relevant feedback at the right moment?

Rather than providing students with the message ‘we missed you last week; please catch up the next days’ […], intervention can now focus on concrete learning behaviors: working on the tendency to procrastinate, searching for alternative learning materials that may be of more challenging nature. In otherwords: the learning feedback will change from product feedback to feedback provided at the process level, the most effective type of feedback.

Tempelaar, Rienties and Nguyen (2017). Towards Actionable Learning AnalyticsUsing Dispositions

Subject of the study

4

• 367 students1st year of fac. of Law (2015-2016)university of Leuven (KU Leuven)

• selected from dataset of 3909 students (2015-2017)

Teaching approach: blended learning

5

language for specific purposes course

basic grammar (and vocabulary) exercises

student data

initial testresult

learningmanagementsystem

end mark -continuous assessment

Student data

6

• previous studies• end mark for previous studies• number of hours for French and Mathematics• mother tongue

• self-evaluation (0-10)

Initial test

7

• C-test

o 100 itemso often used as placement test

Initial test

8

• Did the students take the test seriously? YES

Initial test

9

• Predictive value of test scores?o Correlation test score – exam score:

• R = .55 (p<0,001)o Regression analysis

• R² = 0.50%

variables β pEnd resultsecundary school

.292 <0.001

Test score .265 <0.001Secundary school type

.252 <0.001

…

YES

Initial test

10

• Division into groups (n = 367)

exam < 10 exam >= 10 % successtest <= 40 32 4 11.1%test <= 50 44 22 33.3%test <= 60 45 67 59.8%test <= 70 16 80 83.3%test > 70 5 52 91.2%

I: 102

II: 112

III: 153

LMS: exercises

11

• 20 grammatical topics, free order• common “fill in the blanks” exercises

• 1 exercise = 2 attemptso first attempt => correction + fact sheeto second attempt => focus on mistakes

correction + fact sheet

• goal: 7/10 for 17/20 topics > reward > final mark

Exercises

12

• feedback_1: correction + fact sheet (after both attempts)

Exercises

13

• feedback_2: score compared to score of their peers

Exercises

14

• feedback_3: dashboard

Exercises

15

• Do weaker students start their exercises later?

start ac. year

wake-up call

I: 42.4% III: 65%

YES

(II: 61.9%)

Exercises

16

• Do exercises help students get better marks?

>= 17/20 ex. >= 7

< 17/20 ex.>= 7

pass 198 27fail 102 40

YES, probably…

Chi2 = 15.251; df = 1; p = 0.0001

Exercises

17

• Are the exercises too difficult?

% first attempt(s)

% last attempt

averageattempts/topics

I 35.2% 75.3% 2.6

II 38.8% 76.4% 2.7

III 43.9% 76.2% 2.1

NO

Exercises

18

• Are some exercises more difficult than others? YES

Exam mark

19

• exam mark * self-evaluation

Research question

20

How toidentify& support at-risk students (French course)

byproviding relevant feedback at the right moment?

Feedback BEFORE analyses

21

• Mixed product and process feedback

for every ex.: correction + fact sheet => second attempt

for every ex.: score comparison (peers)

wake-up call 1 (mid-Nov.): mail with variable message

wake-up call 2 (mid-March): mail with variable message

dashboard

Feedback AFTER analyses

22

• Process feedback

for every ex.: correction + fact sheet => second attempt

for every ex.: score comparison (peers)

wake-up call 1 (mid-Nov.): email with variable message

wake-up call 2 (mid-March): email with variable message

dashboard

asap: personalized email

Feedback AFTER analyses

23

• email to students witho test score <= 50 (if test taken seriously)

o self-evaluation < 5

• content:o start asap with the exerciseso making exercises increases your chances of successo start with exercises on [items]o be aware that you will need at least 3 attempts => 7/10o later on: do not forget to focus on [items]

Feedback AFTER analyses

24

• email to students witho test score <= 50 (if test taken seriously)

o self-evaluation > 6• content:

o you may have overestimated yourselfo start asap with the exerciseso making exercises increases your chances of successo start with exercises on [items]o be aware that you will need at least 3 attempts => 7/10o later on: do not forget to focus on [items]

Conclusion

25

descriptive learning analytics

predictive learning analytics

better identification of at-risk studentsdispositional learning analytics

process feedback

Future work

26

• More motivated group division?• Findings supported by evidence from other groups?• Impact of early (process) feedback?

• Analysis of answers?

Impact of early (feedback) process?

27

2015-2016

2016-2017

I: 42.4% III: 65%(II: 61.9%)

I: 60,3% III: 62,6%(II: 61.8%)

Impact of early (feedback) process?

28

YESBUT… (data from the day before the wake-up call)

2015-16 numberof

2016-17

20 5.6% 0 1.4% 5216 60.2% < 5 74.2% 27059 16.4% 5-9 11.8% 4342 11.7% 10-16 8.2% 3022 6.1% >= 17 4.4% 16

need for constant monitoring…

29

Thank you for your attention!

serge.verlinde@kuleuven.be