What do the kids think?
A quantitative analysis of feedback questionnaires in standardised reading tests
Eva Konrad & Annabell Marinell
Outline
• Research questions• Secondary school types in Austria• The research instrument• Results• Conclusions
A. Marinell & E. Konrad IATEFL TEASIG 2011
Research questions
• Is there a connection between – the test takers’ familiarity with the topics – their interest in the texts used – how difficult the test takers felt the texts to be– their familiarity with the test methodsand how well they felt it tested their reading ability?
• Is there a connection between the test takers’ total scores and how well they felt the standardised reading test tested their reading ability?
• Is there a difference in test takers’ responses from different school types?
A. Marinell & E. Konrad IATEFL TEASIG 2011
Secondary school types in Austria
A. Marinell & E. Konrad IATEFL TEASIG 2011
AHS BHS
1. HUM: comprise business studies, theory in the respective area as well as compulsory work placements
2. HAK: secondary schools for occupations in the business sector
3. HTL: secondary schools for engineering
4. BAKIP: nursery schools
5. BASOP: colleges for social education
6. HLFS: colleges for agriculture and forestry
1. Classical secondary academic school
2. Secondary academic school emphasizing mathematics and science
3. Secondary academic school emphasizing economics
4. Upper-secondary academic school specializing in instrumental music
5. Upper-secondary academic school specializing in fine arts
Research instrument: test booklet
• 4 independent reading tasks• Test formats:
• Multiple Choice (MCQ)• Multiple Matching (MM)• Sequencing (SEQ)• Note Form (NF)• True/False/Justification (TFJ)
(dropped: True/False/Not given)
• Answer sheet• Questionnaire
A. Marinell & E. Konrad IATEFL TEASIG 2011
Research instrument: questionnaire
A. Marinell & E. Konrad IATEFL TEASIG 2011
Questionnaire excerpt
Selected questions for research
1. How familiar were you with the topic of the texts?2. How interesting did you find the texts?3. How difficult did you find the texts?4. How familiar were you with the test methods?5. How well could you show your reading competency
in this test?
A. Marinell & E. Konrad IATEFL TEASIG 2011
Face validity“The extent to which a test meets the expectations of those involved in its use, e.g. administrators, teachers, candidates and test score users; the acceptability of a test to its stakeholders.”
(McNamara 2000)
For our research:How well did the test takers feel the standardised reading test was able to measure their reading competency?
A. Marinell & E. Konrad IATEFL TEASIG 2011
Test populationTrial 2008: AHS Trial 2011: AHS Trial 2011: BHS
(N=1665) (N=3041) (N=914) School type
Booklet 1 184 Booklet 1 303 Booklet 1 124 BAKIP
Booklet 2 192 Booklet 2 306 Booklet 2 128 HAK
Booklet 3 186 Booklet 3 273 Booklet 3 160 HAK
Booklet 4 184 Booklet 4 297 Booklet 4 115 HTL
Booklet 5 183 Booklet 5 299 Booklet 5 112 HTL
Booklet 6 181 Booklet 6 388 Booklet 6 92 HUM
Booklet 7 168 Booklet 7 385 Booklet 7 91 HUM
Booklet 8 182 Booklet 8 385 Booklet 8 92 HUM
Booklet 9 117 Booklet 9 405
Booklet 10 88
A. Marinell & E. Konrad IATEFL TEASIG 2011
Method
Programme
• MSB statistics programme• Developed by Gerhard Marinell and Gabriela Steckenberger
A. Marinell & E. Konrad IATEFL TEASIG 2011
Method
MSB statistics interface
Data matrix
A. Marinell & E. Konrad IATEFL TEASIG 2011
Significance level as quoted in Henning (1987): α=0.05
Question 1
Is there a connection between • the test takers’ familiarity with the topics • interest in the texts used• how difficult they felt the texts to be and how well they felt their reading ability was tested?
Is there a connection between these factors and the test takers’ total score?
A. Marinell & E. Konrad IATEFL TEASIG 2011
A. Marinell & E. Konrad IATEFL TEASIG 2011
face validity x familiarity with topic
face validity x interest in texts
face validity x difficulty of texts
0 20 40 60 80 100 120
25
40.6
90.6
38.9
61.1
100
22.5
57.5
27.5
2008 AHS2011 AHS2011 BHS
Values in %
A. Marinell & E. Konrad IATEFL TEASIG 2011
total score x familiarity with topic
total score x interest in texts
total score x difficulty of texts
0 20 40 60 80 100 120
15.6
25
43.8
22
50
97
17.5
22.5
30
2008 AHS2011 AHS2011 BHS
Values in %
Values in %
face validity x familiarity with topic
total score x familiarity with topic
face validity x interest in texts
total score x interest in texts
face validity x difficulty of texts
total score x difficulty of texts
0 20 40 60 80 100 120
25
15.6
40.6
25
90.6
43.8
38.9
22
61.1
50
100
97
22.5
17.5
57.5
22.5
27.5
30
2008 AHS2011 AHS2011 BHS
Question 2
Is there a connection between the test takers’ familiarity with the test methods and how well they felt their reading ability was being tested?
A. Marinell & E. Konrad IATEFL TEASIG 2011
A. Marinell & E. Konrad IATEFL TEASIG 2011
face validity x multiple choice
face validity x note form
face validity x multiple matching
0 20 40 60 80 100 120
50.0
23.3
42.9
100
100
100
89
60
86
2008 AHS2011 AHS2011 BHS
Values in %
Question 3
Is there a connection between the test takers‘ perception of how well their reading competence was tested and their total scores?
A. Marinell & E. Konrad IATEFL TEASIG 2011
Values in %
face validity x total score
0 20 40 60 80 100 120
75
100
80
2008 AHS 2011 AHS 2011 BHS
Conclusions
• We could observe certain tendencies:– Correlations in all of the factors were more frequent in 2011
than in 2008 – Correlations with face validity were more frequent than with
total score• Further analysis of feedback questionnaires is needed• We will be able to say more about BHS after more trials• We still have several questions:
– Are there differences between the different BHS school types?– What further information might we gain from the open
questions?
A. Marinell & E. Konrad IATEFL TEASIG 2011
Bibliography
• Alderson, C. J., Clapham, C., & Wall, D. (2005). Language test construction and evaluation. Cambridge: CUP.
• Bachman, L. F. (2004). Statistical analyses for language assessment. Cambridge: CUP.
• Henning, G. (1987). A guide to language testing: development, evaluation, research. Cambridge, Mass: Newbury House Publishers.
• McNamara, T. (2000). Language testing. Oxford: OUP.
Thank you for your attention and enjoy the rest of the
conference!
A. Marinell & E. Konrad IATEFL TEASIG 2011