Date post: | 12-Jan-2016 |
Category: |
Documents |
Upload: | marsha-floyd |
View: | 216 times |
Download: | 0 times |
Overview of ICCS field trial
National Research Coordinators Meeting Windsor, June 2008
NRCMeetingWindsor
June 2008
Content of presentation
• Participation in field trial• General outcomes of field trial• Feedback from national centres
– Summary in NRC(June08)1.pdf
• Problems identified in field trial• Consequences of field trial
NRCMeetingWindsor
June 2008
Field trial outcomes
• Field trial data analysis– revealed that generally the instruments worked
well• Cognitive test well targeted to ability & reliable• Questionnaire scales reliable and fit structure
– informed the selection of item material for main survey
– confirmed general feasibility of on-line surveys for teacher and schools
• Review of implementation procedures– Generally procedures worked well– Confirmed the feasibility of using regional
instruments– Some possibilities for improvement &
streamlining
NRCMeetingWindsor
June 2008
Field trial participation
• 32 countries undertook field trial
• One country to field trial in June 2008
• 31 data sets included in international field trial analysis report
• Approximately– 19,000 students – 9,000 teachers– 700 schools
NRCMeetingWindsor
June 2008
School response rates
• 21 countries had school response rates of at least 85 percent BEFORE replacement
• 4 countries had school responses rates of at least 85 percent AFTER replacement
• Only 7 countries had school response rates below 85 percent
• Period for field trial close to the end of the year contributed to problems with school participation
NRCMeetingWindsor
June 2008
Student response rates
• Overall, about 90 percent of students participated within schools
• Response rates (slightly) below 85 percent were found in only two countries
NRCMeetingWindsor
June 2008
Teacher response rates
• Across countries, there was a median teacher response rate within schools of 90 percent
• When combined with school response rates the overall response rate for the teacher survey was 89 percent (median across countries)
• In 10 out of 31 countries the overall response rate was below 75 percent
NRCMeetingWindsor
June 2008
Results of NRC Survey
• Overall satisfaction with manuals
• Main problems encountered:– Convincing schools to participate– Explaining sampling procedures– Implementing some test administration
procedures– Reaching a high student and teacher
participation within schools
NRCMeetingWindsor
June 2008
Issues raised regarding survey instruments
• Length of student test and questionnaires (7 countries)
• Questions in instruments seen as too difficult for students of this age (4 countries)
• General directions too long and repetitive (5 countries)
NRCMeetingWindsor
June 2008
Issues raised regarding the teacher sampling design
• Listing of all teacher teaching in target grade seen as a problem
• Two national centres saw the procedures as a major obstacle for school participation
• Data about teachers’ month/year of birth seen as inappropriate
NRCMeetingWindsor
June 2008
Issues raised regarding teacher participation
• Teachers not teaching CCE saw questionnaire as irrelevant to them
• Two national centres viewed questionnaire as too long
• Teacher participation more difficult to obtain than student participation
NRCMeetingWindsor
June 2008
Issues raised regarding regional modules
• Problems with layout of European instrument (part 2 being upside-down and back-to-front)
• Problems with administration and length of general directions
NRCMeetingWindsor
June 2008
Issues raised regarding manuals and software
• More than 80% found the manuals “very helpful” but suggestions to:– Simplify and shorten manuals – Refine software tools for sampling and
data entry
• Need to improve the scoring guides for coding of open-ended questions
NRCMeetingWindsor
June 2008
Issues identified by the ISC
• Need for improvement of adaptation notes
• Provision for countries to record their national adaptations to general directions
• Inclusion of checklist in manual for submission for verification
NRCMeetingWindsor
June 2008
Looking ahead
• Reduction in assessment time– Testing time now 45 minutes– Student questionnaire time now about 35
minutes– Regional instruments shorter– Some reduction in teacher and school
questionnaires
• Reduction in teacher sample size– Cluster size of 15 for main survey– On average, no more than 15 teachers in target
population at schools
NRCMeetingWindsor
June 2008
Expert advice
• Project Advisory Committee Meeting 7-8 June here in Windsor
• Discussion of outcomes and proposals
• Some changes to the proposed instruments following PAC advice– PAC suggestions will be included in
presentations
NRCMeetingWindsor
June 2008
Questions or comments?