Proposed Cognitive Test for the Main Study

National Research Coordinators Meeting Windsor, June 2008

NRCMeetingWindsor

June 2008

• Feedback from group discussions, comments, recommendations

Contents

≅ 50min

• Group discussion of specific recommendations ≅ 60min

≅ 70 min

• First impressions

• Summary of recommendations

• Possible outcomes • Item deletions, minor changes.

NRCMeetingWindsor

June 2008

First impressions• Items worked well, can comprise single

scale (NRC 2, p10)• CIVED trend set can be scaled with and

used for single (NRC 2 pp16,17)• Test time per item seemed appropriate

according to ‘not reached’ – still to be investigated further

• No pattern of gender DIF by item type or substance (NRC 2, p12)

• Some patterns of ‘regional’ item-by-country interaction (NRC 2, p15)

NRCMeetingWindsor

June 2008

First impressions (cont.)

• Open-ended response items showed higher ‘non-attempts’ and slightly more variability (item-by-country interaction and opportunity for inter-rater reliability issues) (NRC 2, p17)

• Targeting good (NRC 2, p9)

NRCMeetingWindsor

June 2008

Open-ended response item scoring guides

• Revision:– Structural changes informed by

suggestions from countries*– Addition of “real” student responses (thank

you to Julie Nelson at NFER)– Addition of relevant scoring codes on the

advice of countries*

* Feedback from NRCs through questions, voluntary reports and field-trial operations survey.

– Finalisation of scoring guides at the marker training seminar in Hamburg

NRCMeetingWindsor

June 2008

Summary of Item Recommendations

Recommendation MCQ

Include unchanged 65 + 6*

Include only with change 11

Delete 3 + 1*

Include and consider possible change 2

Possible deletion or inclusion with change 0

Possible delete 2 + 5*

Total 90

OE

3

1

1

2

1

0

8

Total

74

12

5

4

1

7

98 *T/F

NRCMeetingWindsor

June 2008

Possible Outcomes• Shorten test to 45 minutes• Increase number of clusters to 7• Delete two open-ended response

items and approximately 15 MCQ to maintain test-taking rate.

• Some updates on meeting documents resulting from further analyses and PAC meeting.– True/False items (NRC 5, pp 25-26)– PAC recommendations/suggestions

(tabled document)

NRCMeetingWindsor

June 2008

Latest News

• Some updates on meeting documents resulting from further analyses and PAC meeting.– True/False items (NRC 5, pp 25-26)– PAC recommendations/suggestions

(tabled document)

NRCMeetingWindsor

June 2008

Part 2Group discussions of items

NRCMeetingWindsor

June 2008

Context of Group Discussions

• Assessment framework developed in consultation with NRCs and other experts before the field-trial.

• All countries invited to submit items before field-trial

• All items made available for online feedback by NRCs before the field-trial

• All items revised on the basis of the online feedback for the field-trial and approved for inclusion at the NRC meeting in Rome.

NRCMeetingWindsor

June 2008

Context for Group Discussions

• Field-trial empirical data collected from 31 countries and approx. 19, 000 students

• Recommendations for changes to instrument are based on empirical (and some qualitative) data from the field trial.

NRCMeetingWindsor

June 2008

Context for Group Discussions

• The only changes that can be made to the cognitive instrument now are:

1. Deletion of items2. Minor changes to items only if we are

very confident that a change will fix a problem (and not introduce a new problem)

3. Amendments to the scoring guides

NRCMeetingWindsor

June 2008

Context for Group Discussions

• All items with “good” data are to be included in the MS instrument

• Groups to consider each of the ISC recommendations and make one of three recommendations for each:

1. Accept definitive ISC recommendation

2. Reject definitive ISC recommendation (delete item)

3. Offer alternative “solution” to ISC recommendation

NRCMeetingWindsor

June 2008

Session 2 – Until lunch

1. 30 minutes – finish group discussion

2. 60 minutes – BRIEF presentation from each group

NOTE: Please submit all your comments and feedback to us (me) and we will include them in our deliberations regarding the final instrument.

NRCMeetingWindsor

June 2008

Part 3 Feedback from group

discussions, comments, recommendations

NRCMeetingWindsor

June 2008

Booklet rotation

Clust.Booklet

# Pos 1 Pos 2 Pos 3

A 1 A B D

B 2 B C E

C 3 C D F

D 4 D E G

E 5 E F A

F 6 F G B

G 7 G A C

NRCMeetingWindsor

June 2008

Not reached for Latin American Countries

NRCMeetingWindsor

June 2008

Recommended delete

Possible delete

3 | | | | | | |48.2 |57.2 |23.2 | 2 | X|97.2 XX| XX| XX|67 XX|57.1 68.2 XXX|83.2 XXXX|78 XXXX|31 45 1 XXXX|76 XXXXXX|17 42 80 XXXXX|12 41 48.1 72 XXXXXX|23.1 59 73 77 98.2 XXXXXXX|61 75 91 XXXXXXXX|8 79 98.1 XXXXXXXX|11 37 60 85 89 90 94 XXXXXXXXX|14 29 47 96.2 XXXXXXXX|6 53 65 70 84 92 XXXXXX|22 27 63 81 0 XXXXXXX|44 64 68.1 83.1 88 XXXXXXX|2 4 7 9 13 32 50 71 93 97.1 XXXXXXXX|15 55 62 XXXXXXXXXX|5 20 25 30 58 XXXXXXXX|16 24 26 XXXXXXX|18 43 46 49 51 52 54 82 86 XXXX|1 21 28 XXXXX|3 19 34 35 56 66 96.1 XXXXX|40 74 -1 XXX| XXX|10 69 XX|33 36 95 XX| X| X| X|38 87 X|39 | | -2 |

NRCMeetingWindsor

June 2008

Recommended delete

Possible delete

3 | | | | | | |48.2 | |23.2 | 2 | X|97.2 XX| XX| XX| XX| 68.2 XXX|83.2 XXXX| XXXX| 45 1 XXXX|76 XXXXXX|17 42 80 XXXXX|12 41 48.1 72 XXXXXX|23.1 59 73 77 98.2 XXXXXXX|61 75 91 XXXXXXXX|8 79 98.1 XXXXXXXX|11 37 60 85 89 90 94 XXXXXXXXX|14 29 47 96.2 XXXXXXXX|6 53 65 70 84 92 XXXXXX|22 27 63 81 0 XXXXXXX|44 64 68.1 83.1 88 XXXXXXX|2 4 7 9 13 32 50 71 93 97.1 XXXXXXXX|15 55 62 XXXXXXXXXX|5 20 25 30 58 XXXXXXXX|16 24 26 XXXXXXX|18 43 46 49 51 52 54 82 86 XXXX|1 21 28 XXXXX|3 19 34 35 56 66 96.1 XXXXX| 74 -1 XXX| XXX|10 69 XX|33 36 95 XX| X| X| X|38 87 X|39 | | -2 |

NRCMeetingWindsor

June 2008

Recommended delete

Possible delete

3 | | | | | | | | | | 2 | X| XX| XX| XX| XX| XXX| XXXX| XXXX| 45 1 XXXX|76 XXXXXX|17 42 80 XXXXX|12 41 72 XXXXXX| 59 73 77 XXXXXXX|61 75 91 XXXXXXXX|8 79 XXXXXXXX|11 37 60 85 89 90 94 XXXXXXXXX|14 29 47 XXXXXXXX|6 53 65 70 84 92 XXXXXX|22 27 63 81 0 XXXXXXX|44 64 88 XXXXXXX|2 4 7 9 13 32 50 71 93 XXXXXXXX|15 55 62 XXXXXXXXXX|5 20 25 30 58 XXXXXXXX|16 24 26 XXXXXXX|18 43 46 49 51 52 54 82 86 XXXX|1 21 28 XXXXX|3 19 34 35 56 66 XXXXX| 74 -1 XXX| XXX|10 69 XX|33 36 95 XX| X| X| X|38 87 X|39 | | -2 |

NRCMeetingWindsor

June 2008

True/False Items

Composite model 1

Yuck!

NRCMeetingWindsor

June 2008

True/False Items

Composite model 2

Also yuck!

NRCMeetingWindsor

June 2008

True/False Items

36: laws can be made by an elected <parliament/legislature>.

Disc: 0.18

NRCMeetingWindsor

June 2008

True/False Items37: judges make legal decisions without being influenced by the government.

Disc: 0.16

NRCMeetingWindsor

June 2008

True/False Items38: the religion of the majority is the only legal religion.

Disc: 0.36

NRCMeetingWindsor

June 2008

True/False Items38: the religion of the majority is the only legal religion.

NRCMeetingWindsor

June 2008

True/False Items

39: people have the right to criticise the government.

Disc: 0.31

NRCMeetingWindsor

June 2008

True/False Items

39: people have the right to criticise the government.

NRCMeetingWindsor

June 2008

True/False Items40: only adults who pay income tax are allowed to vote.

Disc: 0.15

NRCMeetingWindsor

June 2008

True/False Items41: all adult voters have the right to vote on new laws in the national <parliament/ legislature>.

Disc: 0.18