Italy - Remedial Exams Student Achievement

7/29/2019 Italy - Remedial Exams Student Achievement

1/20

Remedial Exams and Student Achievement 1'

&

$

%

The Effects of Remedial Exams on Student

Achievement: Evidence from Upper Secondary

Schools in Italy

(Erich Battistin, Ilaria Covizzi and Antonio Schizzerotto)

University of Padova and IRVAPP

Brussels, 20th June 2011

JRC-CRELL Catch the Train: Skills, Education and Jobs


2/20

& % Remedial Exams and Student Achievement 2'

&

$

%

Introduction

Until the 2006/07 school year students in higher secondaryschools (colleges) in Italy who did not meet proficiency

expectations were given a debito formativo (educational

debt), that is a final mark signaling failure in one or more

subjects.

Such lack in achievement was to be recovered in the followingyears, with no clear deadline and with mild (or simply no)

enforcement from schools.

The absence of any regulation soon became one of the main

suspects for the low performance at 2003 and 2006 PISA tests.

Aiming to improve effectiveness in learning, the Minister ofEducation of the time decided to introduce remedial exams

in Italy (that - curiously enough - had already been abolished

from the school year 1992/93).


3/20


&

$

%

Introduction

Under the new progression rule (which is still in operation):

low performing students are compelled to recover their

educational lack before the beginning of the new school

year by sitting a remedial exam.

students who have not been able to recover their debt, or have

been assigned too many debts to be recovered at the end of the

school year, face grade retention.

all schools, although with a certain degree of autonomy, have to

implement remedial education programmes for

low-achieving students, and prepare them for the exam.

failure in one single subject will lead to the remedial exam.

Thus, students are compelled to reach adequate standards in

all subjects to avoid grade retention.


4/20


&

$

%

Introduction

The autonomous province of Trento did not comply with thenew rules: (i) very good performance at PISA tests of local

students, and (ii) already longstanding tradition in offering effective

remedial courses to low-achieving students.


5/20


&

$

%

Punchline

General idea: set up the comparison of areas with remedialexam vis-a-vis other areas (i.e. educational debt).

Works if: the two groups of areas had similar performance

prior to the policy roll out.


6/20


&

$

%

Punchline

Natural experiment resulting from the geographical

discontinuity in the implementation of the reform.

Use geographical variation to study the reduced form policy

effect of being at risk of grade retention on short-run

achievement gains, vis-a-vis the regime operated until the

school year 2007/08.

(in progress) Build bridges between structural channels and

policy effects by relating family and school inputs to output.(i.e. competences developed).


7/20


&

$

%

Main Findings

Between group heterogeneity. Clear pattern of zero or nearly

positive (average) effects of the remedial exam for students onacademic tracks and negative (average) effects for students in

vocational schools.

The intervention exacerbates preexisting inequalities in

achievement among students from different school tracks.


8/20


&

$

%

Main Findings

Evidence that schools may have reacted differently. Schools

react to the reform by admitting to the next grade students

who in the status quo would have been given an

educational debt. This is more so in academic schools.

In academic schools the introduction of a remedial exam comes

with less stringent rules to pass students to the next grade.

This implies zero effect on average.

In vocational schools the effects of less stringent rules are

largely offset by the negative effects on effort caused by theremedial exam. These in turn imply more heterogeneous effects

across students and negative average effects.


9/20


&

$

%

Evaluation Design

Exploits a matched pair comparison of students in schools

either side of the border: schools in province of Trento were

matched to similar schools in the surrounding provinces of

Bolzano, Brescia and Vicenza.


10/20


&

$

%

Evaluation Design

School selection controls for observable dimensions such as

school track, school size (as measured by trends in enrollment),

school resources, as well as unobservable dimensions (such as

reputation of the school) gathered from general knowledge of

the socioeconomic background in which they operate.

We focused on students attending the second and the third

year during the school year 2008/09, thus aged between

14 and 15.

For each school we randomly selected two classes in the

second year (i.e. for the cohort of students enrolled for the first

time in school year 2007/08) and two classes in the third year

(i.e. for the cohort of students enrolled for the first time in

school year 2006/07). About 2000 students and 44 schools.


11/20


&

$

%

Evaluation Design

Ensures variability in the duration of enrollment at school across

the different regimes introduced by the reform.


12/20


&

$

%

Data

Main sources:

1. Specially commissioned standardized test prepared by the

National Institute for the Evaluation of the Educational

System. Can in fact be considered as a small scale PISA

survey.a

2. Large set of demographics and (contemporaneous)family inputs collected through integrated data surveyed

from students and their parents.

3. Transcript files from schools on all marks since enrollment

as well as on the national exam at the end of lower secondary

school (8th grade; not yet standardized nationwide at thattime).

aDifferently from the PISA format, all students answered all questions in the

assessment, so that multiple measurements are available for each student.


13/20


&

$

%

Data

Additional sources:1. Specially granted information from the 2003 and 2006 PISA

surveys, where only schools in narrowly defined areas that

match closely the evaluation design are considered.


14/20


&

$

%

Testable Implications of the Evaluation Design

No pre-reform differences in achievement for schools in

the provinces considered using PISA 2006 data.

Provinces: narrowly defined areas as in the evaluation design. Regions: same

regions as in the design.


15/20


&

$

%


Full support with respect to the socio-demographic

indicators across cohorts and school types. Good degree of

homogeneity for students in schools either side of the

province border.

.2 .4 .6 .8 1Propensity Score

Older Cohort

.2 .4 .6 .8 1Propensity Score

Younger Cohort

Propensity scores for students in academic schools with different policy regimes

Academic Schools

.2 .4 .6 .8Propensity Score

Older Cohort

0 .2 .4 .6 .8Propensity Score

Younger Cohort

Propensity scores for students in academic schools with different policy regimes

Vocational Schools


16/20


&

$

%


No pre-reform differences in retention rates.

Academic Vocational

Matching OLS Matching OLS

Passed, with no debt 0.0417 0.0407 0.0236 0.0629

(0.0502) (0.0732) (.0706) (0.0487)

Admitted with debt -0.0352 -0.0226 -0.0377 -0.0778

(0.0491) (0.0682) (0.0692) (0.0591)

Standard errors clustered by class

Regressors include gender, class size and its square


17/20


&

$

%

Reduced Form Effects: Academic Schools

Mean Quantiles

Matching OLS Weighting 0.25 0.50 0.75

Older Cohort (enrolled in 3rd year at interview)

Reading 7.847 7.774 11.275 -9.311 12.224 8.670

(14.704) (9.382) (15.443) (12.937) (16.557) (18.263)

Math -9.866 -2.791 -3.995 2.821 -11.087 -6.190

(12.990) (10.186) (20.215) (14.356) (11.797) (7.474)Science 7.921 12.687 16.139 -5.999 4.894 24.127

(11.922) (8.921) (10.462) (12.954) (17.502) (12.329)

Younger Cohort (enrolled in 2nd year at interview)

Reading 2.208 9.034 10.495 0.455 13.173 -6.190

(15.567) (15.034) (15.949) (10.856) (12.361) (15.967)

Math 13.742 16.579 18.552 14.186 13.127 1.069

(12.538) (13.422) (23.405) (9.760) (12.692) (10.624)

Science 21.843 18.749 16.479 15.204 17.316 5.554

(12.344) (9.072) (16.799) (10.286) (12.556) (14.342)



18/20


&

$

%

Reduced Form Effects: Vocational Schools

Mean Quantiles

Matching OLS Weighting 0.25 0.50 0.75

Older Cohort

Reading -21.793 -25.092 -22.740 4.192 -8.140 -22.298

(15.398) (9.621) (15.125) (16.316) (9.502) (9.538)

Math -5.849 -5.339 -5.187 -8.443 5.342 -22.775

(16.514) (10.376) (22.581) (16.309) (6.407) (9.487)Science -4.908 -8.047 -7.224 0.580 -10.473 -6.173

(14.577) (9.882) (16.465) (10.781) (11.813) (15.983)

Younger Cohort

Reading -16.947 -19.833 -20.843 -6.777 -16.214 -21.124

(12.176) (8.982) (14.937) (14.084) (13.420) (16.320)

Math -1.633 -6.195 -0.050 -29.413 -8.792 13.573

(16.902) (15.512) (27.105) (20.393) (11.148) (10.678)

Science -29.962 -32.333 -30.518 -25.160 -19.966 -28.386

(13.480) (11.019) (17.383) (18.650) (16.450) (15.007)



19/20


&

$

%

A Twist of Heterogeneity

0

.5

1

200 400 600 800 200 400 600 800

Younger Cohort Older Cohort

remedial exam debt

cdf

PISA score

Weighted cumulated density functions

Science

Academic Schools

0

.5

1

200 400 600 800 200 400 600 800

Younger Cohort Older Cohort

remedial exam debt

cdf

PISA score

Weighted cumulated density functions

Science

Vocational Schools


20/20

Remedial Exams and Student Achievement 20' $

Reduced Form Effects: Retention Rates

Academic Vocational

Matching OLS DD Matching OLS DD

Passed 0.1227 0.1366 0.0801 0.0863 0.0782 0.0227(0.0523) (0.0443) (0.0413) (0.0441) (0.0449) (0.0420)


Differences in retention rates at the end of the second year (first policy-on year)

for the cohort of students enrolled in the school year 2006/07. The DD is

calculated using results for the same cohort in the previous year.

Date post:	03-Apr-2018
Category:	Documents
Upload:	barathy-chandrasegran
View:	220 times
Download:	0 times

Italy - Remedial Exams Student Achievement

Documents