Post on 11-Jan-2016
description
transcript
IATEFL/TEASIG, Innsbruck 2011
READING LITERACY OF AUSTRIAN SCHOOL LEAVERS: BETWEEN PISA AND
"MATURA”
Irene Thelen-Schaefer, BIFIE Wien
IATEFL/TEASIG, Innsbruck 2011
Presentation overview The study Theoretical background The research questions The instruments Operationalisation Results Findings Limitations Future research
IATEFL/TEASIG, Innsbruck 2011
Background to the study
An interest in researching the predictive potential in PISA reading tests
IATEFL/TEASIG, Innsbruck 2011
Cross-cultural assessment and translationPISA = Programme for international student assessment
In order to make a comparison of competence possible
Adaptation of tests across borders and culturesSet the specific psychological construct into the right context
Exclude cultural bias
IATEFL/TEASIG, Innsbruck 2011
The Study
Background test: Reading in German (L1)
• German translation of PISA reading literacy tasks 2009
Reading in English (L2)
• English version PISA reading literacy tasks 2009
Austrian standardised school leaving exam
• (“Matura”)
Research focus
• the comparability of items
• the predictability of L2 reading proficiency
IATEFL/TEASIG, Innsbruck 2011
Theoretical background 1: The skill reading
L1 reading and L2 readingAre we testing the same thing?Same basic cognitive comprehension process across L1
and L2 contexts
BUT
distinctive features in L2 reading:Language proficiency (“language threshold“)L2 processing skillsBackground knowledge
IATEFL/TEASIG, Innsbruck 2011
Theoretical background 2: The learnerLearning to read: L1/L2 differences
L1 LearnerYoung learnerOral skills developed before starting readingLearning to read and write at the same timeLarge knowledge of linguistic structures and vocabulary
L2 LearnerCognitively matureThey are L1 literate, but: they have not developed L2 oral comprehension yet different starting point in L2 readingL2 reading processes occur in a dual-language system
IATEFL/TEASIG, Innsbruck 2011
Theoretical background 3: The processes
Lower level processes
Automatisation is essentialAutomatisation of word recognitionSpeed of fixationWorking memoryRauding (Reading + auding; Carver, 1984)
Higher level processes Motivation / purpose of
reading Two levels of understanding
“a text model of comprehension““a situation model of reader
interpretation“; (Grabe, 2009)
Strategies Background knowledge
(Schema theory)
IATEFL/TEASIG, Innsbruck 2011
The Studyꞌs Research Questions
1. Can L1 reading results predict L2 reading results?
2. Can reading results from PISA English predict results of the “Matura” English L2?
IATEFL/TEASIG, Innsbruck 2011
The Instruments
Tests PISA German PISA English "Matura"
No of tasks 3 3 4
Time 45 min 45 min 50 min
Test methods MCQ; Text answer +
justif.
MCQ; Text answer +
justif.
MCQ; MM; T/F/NG; NF
No of items 11 18 25
Student feedback questionnaire
IATEFL/TEASIG, Innsbruck 2011
Operationalisation
Participating schools
Allgemeinbildende Höhere Schulen• (AHS; grammar schools)
Urban and rural areas • (Vienna, Linz, Admont, Hallein, Reutte, Villach…)
Public and private schools
Different types of AHS • BORG, BG, BRG, WIKU
• Different branches (foreign languages, science, arts, technology…)
IATEFL/TEASIG, Innsbruck 2011
Operationalisation
Test takers (TT)18-year-old students50 females, 51 males101 TT completed all three tests96 questionnaires returned
Teachers as test administratorstrained test administratorsprecise instructions given
IATEFL/TEASIG, Innsbruck 2011
(Provisional) Results
Tests compared
PISA German
PISA English
“Matura“
Mean 8.54 7.85 17.43
Mean % 77.36% 71.36% 69.72%
Std. Deviation
1.45 1.65 3.91
Mode 9 8 20
IATEFL/TEASIG, Innsbruck 2011
(Provisional) Results
All three tests show that
The test takers did well on the tests (FV, skew...)
The sample is relatively homogeneous (SD)
Normal distribution on the PISA English test and the
“Matura”
IATEFL/TEASIG, Innsbruck 2011
(Provisional) Results - Facility values
FV % 1 2 3 4 5 6 7 8 9 10 11
PISA G 79.2 63.4 90.1 90.1 35.6 97.0 93.1 83.2 58.4 74.3 90.1
PISA E 80.2 58.4 87.1 69.3 16.8 94.1 86.1 73.3 58.4 78.3 83.2
Findings
FV equal or slightly better in PISA German than in PISA English
Exceptions: items 4 and 5
The PISA tasks
IATEFL/TEASIG, Innsbruck 2011
(Provisional) Results – Item analysisItem 4
FV Corrected Item-total Correlation
Cronbach‘s Alpha if Item
Deleted
PISA German 90.1% .058 .265
PISA English 69.3% .279 .281
Possible reasons: technical language Frequent reasoning for test difficulty given in
questionnaires by TT: (lack of) vocabulary
IATEFL/TEASIG, Innsbruck 2011
(Provisional) Results - Item analysis
Item 5
Possible reasons:Test methodAspect targeted (“access and retrieve”) – answer to item 5 is after answer to item 6
FV Corrected Item-total Correlation
Cronbach‘s Alpha if Item
Deleted
PISA German 35.3% .050 .278
PISA English 16.7% .108 .358
IATEFL/TEASIG, Innsbruck 2011
(Provisional) Results – Item analysisItem 10
Corrected Item-total Correlation
Cronbach‘s Alpha if Item Deleted
PISA German .307 .126
PISA English .054 .379
Possible reasons: Partial credit in English only Possible answers for partial credit are included in the
“incorrect”-answer key in GermanItem 10 is the same in both languages re translation, but: it is the best item in PISA German and the worst in PISA English
IATEFL/TEASIG, Innsbruck 2011
(Provisional) Results - Reliability and discrimination
PISA tasks Low coefficients on both reliability and discriminationGerman: eight items discriminate positivelyEnglish: all items discriminate positively; but only item 4 is above .25 (at .279)
These plus the number of items might be the reasons for the low Cronbach’s alpha coefficients on the PISA tasks
PISA German PISA English “Matura”
Cronbach‘s Alpha .270 .371 .723
IATEFL/TEASIG, Innsbruck 2011
(Provisional) Findings - Item analysis
The “Matura”
Problematic items mainly in one test format: T/F/NG
Possible reasons:
Difficult for TT to distinguish between F and NG
This test format has now been replaced by T/F
justification.
IATEFL/TEASIG, Innsbruck 2011
(Provisional) Results - CorrelationsPISA German PISA English
PISA English .454**
“Matura“ .268** .383**
Findings
Empirical data show that research question 1 on a low possible overlap between reading in L1 and L2 can be stated.Surprisingly, TT did nearly as well on the PISA English tasks than on the PISA German tasks.
**. Correlation is significant at the 0.01 level (2-tailed).
IATEFL/TEASIG, Innsbruck 2011
Possible impact on the results
Motivation Unknown test methods in the PISA tasks Order of items Partial credit in PISA English (item 10) Tests target different purposes or aspects Instructions Examples
IATEFL/TEASIG, Innsbruck 2011
Further steps to be conducted
Further item analysis, incl. factor analysis PISA
English – “Matura”
Correlations of test results and questionnaire items,
e.g. Do avid readers have higher scores?
IATEFL/TEASIG, Innsbruck 2011
Limitations
Only 11 items in the PISA tasks
Items low reliability values
Possible memory effect PISA English – PISA German
Due to time pressure no counter-balanced design
possible
IATEFL/TEASIG, Innsbruck 2011
Ideas for further research
Longitudinal study as a diagnostic instrument:
Can reading results from the PISA reading literacy test taken by 16-year-olds in their L2 predict the ”Matura“ reading results?
Diagnostic purpose
Advice for students and parents
IATEFL/TEASIG, Innsbruck 2011
List of referencesAlderson, J.C. (1984). Reading in a foreign language: a reading problem or
a language problem? In J. C. Alderson and A. H. Urquhart (eds.). Reading in a foreign language. London: Longman.
Alderson, J.C. (2000). Assessing reading. Cambridge, UK: Cambridge University Press.
Bernhardt, E. (2005). Progress and procrastination in second language reading. Annual Review of Applied Linguistics 25, 133-150.
Carver, R. (1984). Rauding theory predictions of amount comprehended under different purposes and speed reading conditions. Reading Research Quarterly 19 (2), 205-218.
Grabe, W. (2009). Reading in a second language. Cambridge, UK: Cambridge University Press.
Koda, K. (2005). Insights into second language reading. Cambridge, UK: Cambridge University Press.
Stanovich, K. (1980). Toward an interactive-compensatory model of individual differences in the development of reading fluency. Reading Research Quarterly 16 (1), 32-71.
Walter, C. (2008). Phonology in second language reading: not an optional extra. TESOL QUARTERLY 42 (3), 455-474.
IATEFL/TEASIG, Innsbruck 2011
THANK YOU FOR YOUR ATTENTION!
CONTACT ADDRESS
i.thelen-schaefer@bifie.at