Using test data to improve performance

Patrick Griffin

Developmental Assessment to Policy Development

Measure Intervene Resource PolicyGeneralise

Measurementof each

specific skill.At this stageeach studentis assessed

bothdiagnostically

andformatively

Themeasurementis generalised

by linkingempirically

to acompetencycontinuum

for each skill

Thegeneralisedinterpretation

from thecompetencycontinuumleads to aninstructionalinterventionplan at eachlevel and forsubgroups

Interventionstrategies arelinked to thenature andrange of

resourcesneeded toimplement

theinterventions.

Aggregatingresource

needs andintervention

planinformationinforms thesystem toformulate

policy

Raising achievement

• Existing tests• Developmental continuum• Collaborative teams (3 to 5)• Team leaders responsibility• Monitoring achievement

Generating an assessment framework

• Using existing tests and questionnaires• Calibrating and monitoring individuals• Reporting • Leaders meetings• Interpreting and using data• Team approach•

Data and intervention

1. Overview • Linking assessment to teaching

2. The role of teacher teams• Nature, composition• Roles and responsibilities• Functions of the teacher teams

3. Team leaders • Meetings for team leaders• Use of data and decision making• Planning for the teacher team

4. Teacher team meetings• Reviewing school and individual data• Interpreting software charts• Setting goals, strategies and evidence expectations• Monitoring and recording progress

5. Data use• Scope of the exercise• Planning and timelines• Data delivery

6. Reporting and interpretation

The accountability framework

Assessment and use of data in developmental

context

CEOM / UoM Project Team

Leaders Team

Interventionand comprehension

resourcesTeacher Team1

Teacher Team2

Teacher Teamn

Assessment Specialist input

Specialist input

The analysis and interpretation cycle

The role of Professional learning teams

• teacher teams of three to five;• Detailed monitoring of at least three students per team member;• two assessments over each calendar year• act on advice from leaders• Link PD input to levels on the developmental progression.• set student goals based on developmental progressions• collective responsibility for each pupil through genuine collaboration• use meetings for professional learning• own the process• expect difference in pupil progress

Team leaders

• organise the teacher team meetings;• use of data to inform teaching practice;• use of the reporting software o focus intervention;• maintain a logs and records of team meetings;• brief the UoM team on the outcome of team meeting

discussions;

Team leaders

• attend project meeting days• understand data and its implications• prepare data interpretations for team members• chair Teacher Team discussions• coordinate goal setting and strategies• monitor and reporting function for

– Teacher Teams– Pupil progress

• support for teachers working as a team• provide team leadership• attention to relationships among the team members

Team Leader Meetings

• UoM team reports to team leaders• team leaders discuss analysis and interpretation

– whole of data– school level– year level etc– individual students

• Practise using software and its interpretation

• Distribute and check assessment audits• information to help teachers

• organise the school team meetings • generate discussion on how the assessment data could

be used to inform teaching practice

• generate discussion on how to aid student learning based on the use of developmental descriptions of

student learning • Feedback reports on students results

Team Leader Meetings

Team Leader Meetings

• When and where?– At least 3 external meetings over one calendar year– Generally meeting where analyses can be explored

(laptops will be essential)• Purpose

– Review aggregate data– Understand the school and individual level data– Prepare reporting session for the team – Report on Teacher Team process and progress– Plan data collections and delivery

• Training for team leaders

Teacher meetings

• Team composition– Leadership and discussion

• Meetings – Location– Frequency– Duration– Agenda

• Planning

Teacher meetings

• Use a set agenda• Profiling individual students• Discuss strategies and resources• Rationale and evidence based• Follow up discussions

Teacher meetings

• How many meetings?• Review of school level data• Discussion groups re implications and

interpretations• Discussions of strategies

1. What additional commitment can or does the school need to make?

2. To what extent does this add to the existing learning team meetings?

3. To what extent does this add to the usual discussion of individual pupils that takes place in the school?

4. Does this process provide a structure to an existing sequence of activities?

Teacher meetings

• A Possible Agenda• Preparation

» Data analyses reports» Targetted students

• Data collection and administration• Reporting• Team meetings• Team strategy and documentation• Team leader meetings

Data use

• Review of instruments and skills audit• Collective review of student achievement• Reaching agreement on strategy and resources• Evidence types for intervention and monitoring

data use

Teacher working SOLO and formulating targets for each pupil, with or without

reference to a progress map

Teachers working as a team and using the

assessment instruments to discus and assess each pupil. The progress map becomes the framework

for discussion

Teacher working alone and then leading the discussion of each

individual pupil using the test and progress map as

a framework

Teachers working as a team on sets of students across instruments and

progress maps

Data collection- Instrument completion proceduresIdeal

Pragmatic but less than ideal

• Establishing the Baseline

• March

• Monitoring

• September

• Ongoing

• team meetings • professional development• team support structure

Data use

Data use

• Establishing the Baseline• March

• Monitoring• September

• Ongoing• team meetings • professional development• team support structure

• Collating the collective view

Reporting

• Among team leaders• At team meetings

5.0 X XX 4.0 XXX XXXXX XXXXX 3.0 XXXXXX XXXXXXX XXXXXXXXX 101 152 XXXXXXXX 114 XXXXXXXXXX 86 118 2.0 XXXXXXXXXXXXXXXXXX 81 XXXXXXXXXX 108 146 XXXXXXXXXXXXXXXXXXX 104 145 XXXXXXXXXXXXXXXXXX 93 94 XXXXXXXXXXXXX 63 155 XXXXXXXXXXXXXXXXXXXX 90 100 115 117 120 1.0 XXXXXXXXXXXXXXXX 69 98 103 134 XXXXXXXXXXXXXXXX 82 105 111 XXXXXXXXXXXXXXXXXXX 70 74 99 110 127 139 XXXXXXXXXXXX 61 73 102 113 119 154 XXXXXXXXXXX 64 77 107 109 130 136 141 0.0 XXXXXXXXX 68 87 124 132 137 138 XXXXXXXXX 72 83 85 135 143 XXXXXXX 89 91 131 149 XXXXXX 71 75 133 XXXXX 67 78 79 116 144 147 153 XXXX 62 65 80 88 97 -1.0 XXX 84 106 126 XX 66 96 X 76 112 140 X 92 95 125 150 -2.0 129 151 142 122 128 -3.0 121 123 148

logit Cognitive skill underpinning the correct response 2.04 Match exact words and paraphrase from origin 0.63 Locating information from text 2.97 Combining several ideas (format-all of the above) 1.8 Understanding implications

0.87 Understanding author's main purpose 0.35 Locating information from text 1.04 Locating information from text 1.73 Understanding meaning of vocabulary 0.96 Locating information from text 1.37 Locating information from text (format - negative questions, using background

knowledge) 2 Locating information from text (format - negative questions) 2.2 Locating information from text (understanding signal words-"prediction" in the stem)

1.06 Understanding meaning of words 0.11 Exact match of text with adjacent text 1.79 Understanding meaning of sentences 0.82 Locating information from text 0.8 Understanding relationship between events in text 0.7 Locating information from text

3.51 Combining several ideas and using outside knowledge 2.35 Locating specific information from text 1.4 Locating information from text

0.54 Understanding meaning of word 1.43 Locating information from text 3.81 Combining several ideas (format-all of the above) 1.51 Understanding meaning of sentences 0.71 Locating information from text 1.29 Locating information from text 2.71 Inferring meaning from context (format-negative question) 1.19 Locating information from text 0.01 Match exact words and paraphrase 3.07 Deducing meaning from context 3.1 Requiring interpretation beyond text level, unfamiliar topic

0 Matching word and visual stimulus 0.35 Matching word and visual stimulus 0.65 Locating information from text 2.39 Locating specific information from text (too many details in long options ) 2.23 Locating information from text & illustration 2.65 Locating specific information from text 4.18 Link a concept to a visual stimulus and bring outside knowledge to the solution 1.97 Locating information from text 2.44 Locating specific information from text 3.29 Understanding author's main purpose on the basis of the title 2.33 Locating specific information from text 0.55 Understanding relationship between events in text 1.86 Understanding implications 3.39 Understanding figurative meaning of word (format-negative question) 1.8 Locating information from text

2.11 Inferring meaning from context (option d attracts some above average pupils) 2.32 Locating specific information from text 0.39 Locating information from text 2.08 Locating information from context 4.03 Combining several ideas and using outside knowledge (format-all of the above) 2.77 Understanding main idea, choosing a title

1 Locating information from text 2.68 Locating specific information from text 3.82 Combining several ideas, requiring interpretation beyond text level 2.05 Understanding author's main purpose 2.71 Understanding figurative meaning

Skills Audits

Levels 1-3

Level 1: Pre Reading (a) Skills: locates words involving concrete concepts and everyday objects. Follows short

simple written instructions. Example test item skills locate familiar words in a short (one line) text match words to pictures follow short and familiar instructions

Level 2: Emergent Reading (a) Skills: Locates words in sentences involving prepositions and abstract concepts; uses cuing systems (by sounding out, using simple sentence structure, and familiar words) to interpret phrases by reading on. (b) Example Test Items

read familiar words and identify some new words use simple and familiar prepositions and verbs to interpret new words match words and very simple phrases

Level 3: Basic Reading (a) Skills: Interprets meaning (by matching words and phrases, completing a sentence, or matching adjacent words) in a short and simple text by reading on or reading back. (b) Example Test Items

use context and simple sentence structure to match words and short phrases use phrases within sentences as units of meaning locate adjacent words and information in a sentence

Levels 4-6

Level 4: Reading for Meaning (a) Skills: Reads on or reads back in order to link and interpret information located in various parts of the text. (b) Example Test Items

interpret sentence and paragraph level texts links and matches phrases across sentences read forwards and backwards in order to locate information in longer texts

Level 5: Interpretive Reading (a) Skills: Reads on and reads back in order to combine and interpret information from various parts of the text in association with external information (based on recalled factual knowledge) that “completes” and contextualizes meaning. (b) Example Test Items

locate, interpret, and read forward to join two pieces of adjacent information use multiple pieces of information to interpret general purpose of a document paraphrase and interpret a single non-adjacent piece of information

Level 6: Inferential Reading (a) Skills: Reads on and reads back through longer texts in order to combine information from various parts of the text so as to infer the writer’s purpose. (b) Example Test Items

interpret, and make inferences from, different types of texts by reading backwards and forwards to confirm links between widely separated information pieces

extract information from a non-traditional (left to right) document make judgments about an author's intentions or purpose beyond the text content

Levels 7-8

Level 7: Analytical Reading (a) Skills: Locates information in longer texts (narrative, document or expository) by reading on and reading back in order to combine information from various parts of the text so as to infer the writer’s personal beliefs (value systems, prejudices, and/or biases). (b) Example Test Items

combine several pieces of information from a range of locations in complex and lexically dense text

analyse detailed text or extended documents for an underlying message identify meaning from different styles of writing

Level 8: Critical Reading (a) Skills: Locates information in a longer texts by reading on and reading back in order to combine information from various parts of the text so as to infer and evaluate what the writer has assumed about both the topic and the characteristics of the reader – such as age, knowledge, and personal beliefs (value systems, prejudices, and/or biases). (b) Example Test Items

use text structure and organisation to identify an author's assumptions and purposes identify an author's motives, biases, beliefs in order to understand the main theme link text to establish multiple meanings including analogy and allegory

Mathematics levels

Mathematics Competency Levels

Level 1 Reads, writes and compares natural numbers, fractions and decimals. Uses single operations of +,-,x and : on simple whole numbers; works with simple measures such as time; recognises simple 3D shapes.

Level 2 Converts fractions with denominator of 10 to decimals. Calculates with whole numbers using one operation (x,-,+ or : ) in a one-step word problem; recognises 2D and 3D shapes.

Level 3 Identifies place value; determines the value of a simple number sentence; understands equivalent fractions; adds and subtracts simple fractions; carries out multiple operations in correct order; converts and estimates common and familiar measurement units in solving problems.

Level 4 Reads, writes and compares larger numbers; solves problems involving calendars and currency, area and volume; uses charts and tables for estimation; solves inequalities; transformations with 3D figures; knowledge of angles in regular figures; understands simple transformations with 2D and 3D shapes.

Level 5 Calculates with multiple and varied operations; recognises rules and patterns in number sequences; calculates the perimeter and area of irregular shapes; measurement of irregular objects; recognised transformed figures after reflection; solves problems with multiple operations involving measurement units, percentage and averages.

Level 6 Problem solving with periods of time, length, area and volume; embedded and dependent number patterns; develops formulae; recognises 3D figures after rotation and reflection and embedded figures and right angles in irregular shapes; use data from graphs and tables

Distribution overall?

0.00 5.00 10.00 15.00 20.00 25.00 30.00 35.00 40.00 45.00

locates words involving concrete concepts and everyday objects. Follows short simple written instructions.

Locates words in sentences involving prepositions and abstract concepts; uses cuing systems (by sounding out, using

simple sentence structure, and familiar words) to interpret phrases by reading on.

Interprets meaning (by matching words and phrases, completing a sentence, or matching adjacent words) in a short and

simple text by reading on or reading back.

Reads on or reads back in order to link and interpret information located in various parts of the text.

Reads on and reads back in order to combine and interpret information from various parts of the text in association with

external information (based on recalled factual knowledge) that “completes” and contextualizes meaning.

Reads on and reads back through longer texts in order to combine information from various parts of the text so as to infer

the writer’s purpose.

Locates information in longer texts (narrative, document or expository) by reading on and reading back in order to

combine information from various parts of the text so as to infer the writer’s personal beliefs (value systems, prejudices,

and/or biases).

Locates information in a longer texts by reading on and reading back in order to combine information from various parts

of the text so as to infer and evaluate what the writer has assumed about both the topic and the characteristics of the

reader – such

Fair measure across schools?

Developmental Assessment to Policy Development

Measure Intervene Resource PolicyGeneralise

Measurementof each

specific skill.At this stageeach studentis assessed

bothdiagnostically

andformatively

Themeasurementis generalised

by linkingempirically

to acompetencycontinuum

for each skill

Thegeneralisedinterpretation

from thecompetencycontinuumleads to aninstructionalinterventionplan at eachlevel and forsubgroups

Interventionstrategies arelinked to thenature andrange of

resourcesneeded toimplement

theinterventions.

Aggregatingresource

needs andintervention

planinformationinforms thesystem toformulate

policy

© Copyright The University of Melbourne 2008