Consider the Evidence

© New Zealand Ministry of Education – copying restricted to use by New Zealand education sector.Page 1

Consider the EvidenceEvidence-driven decision making

for secondary schools

A resource to assist schools to review their use of data and other evidence

6

Getting to Information


Evidence-driven decision making

This module is part of a resource about how we use data and other evidence to improve teaching, learning and student achievementToday we are looking at the second stage of this process – collecting and analysing evidence, and getting to information we can use


Evidence

Any facts, circumstances or perceptions that can be used as an input for an analysis or decision•how classes are compiled, how classes are allocated to teachers, test results, teachers’ observations, attendance data, portfolios of work, student opinions …Data are one form of evidence


Data

Known facts or measurements, probably expressed in some systematic or symbolic way (eg as numbers)•assessment results, gender, attendance, ethnicity …Data are one form of evidence


What evidence does a school have?

• Demographics• Student achievement• Perceptions• School processes• Other practice


The evidence-driven decision making cycle

ExploreCheck data and evidence to explore the issue Question

Clarify the issue and ask a question

Assemble Decide what data and evidence might be useful

Analyse data and evidence

TriggerData indicate a possible issue that could impact on student achievement

Evaluate the impact on the intervention

IntervenePlan an action aimed at improving student achievement

Interpret Insights that answer your question

SpeculateA teacher has a hunch about a problem or a possible action

ActCarry out the intervention

Reflecton what has been learned, how practice will change



.

Explore data Survey of students shows that this is only partially true

QuestionWhat are the characteristics of students who are poor at writing?

Assemble more data & other evidence: asTTle reading, homework, extracurric,Attendance, etc

Analyse NQF/NCEA results by standard

TriggerSignificant numbers not achieving well in writing

Analyse non NQF/NCEA data and evidence

InterveneCreate multiple opportunities for writing; include topics that can use sport as context; connect speaking and writing. PD for staff.

Interpret informationPoor writers likely to play sport, speak well, read less, do little HW

A teacher has a hunch - poor writers might spend little time on homework

Evaluate Has writing improved?

ReflectHow will we teach writing in the future?


Evidence-driven decision making Getting to information

Trigger Clues found in data, hunchesExplore Is there really an issue?Question What do you want to know?Assemble Get all useful evidence togetherAnalyse Process data and other evidenceInterpret What information do you have?Intervene Design and carry out actionEvaluate What was the impact?Reflect What will we change?



Trigger Clues found in data, hunchesExplore Is there really an issue?Question What do you want to know?

> Assemble Get all useful evidence togetherAnalyse Process data and other

evidenceInterpret What information do you have?Intervene Design and carry out actionEvaluate What was the impact?Reflect What will we change?


Assembling the evidence

• We want to know if our senior students are doing better in one area of NCEA Biology than another.

So … we need NCEA results for our cohort.• It could be that all Biology students do better in this

area than others. So … we also need data about national differences across

the two areas.


Are our data any good?

A school found that a set of asTTle scores indicated that almost all students were achieving at lower levels than earlier in the year. Then they discovered that the first test had been conducted in the morning, but the later test was in the afternoon and soon after the students had sat a two-hour exam.


Think critically about data

• Was the assessment that created this data assessing exactly what we are looking for?

• Was the assessment set at an appropriate level for this group of students?

• Was the assessment properly administered?• Are we comparing data for matched groups?


Cautionary tale 1

You want to look at changes in a cohort’s asTTle writing levels over 12 months. Was the assessment conducted at the same time both years? Was it administered under the same conditions?Has there been high turnover in the cohort? If so, will it be valid to compare results?


Cautionary tale 2

You have data that show two classes have comparable mathematics ability. But end-of-year assessments show one class achieved far better than the other. What could have caused this? Was the original data flawed? How did teaching methods differ? Was the timetable a factor? Did you survey student views? Are the classes comparable in terms of attendance, etc?



Trigger Clues found in data, hunches

Explore Is there really an issue?

Question What do you want to know?

Assemble Get all useful evidence together

> Analyse Process data and other evidenceInterpret What information do you have?

Intervene Design and carry out action

Evaluate What was the impact?

Reflect What will we change?


Analysing data and other evidence

• Schools need some staff members who are responsible for leading data analysis

• Schools have access to electronic tools to process data into graphs and tables

• All teachers do data analysis • Data is not an end in itself - it’s one of the many stages

along the way to evidence-driven decision making


Types of analysis

We can compare achievement data by subject or across subjects for • an individual student• groups of students• whole cohortsThe type of analysis we use depends on the question we want to answer


Inter-subject analysis

• Have my students not achieved a particular history standard because they have poor formal writing skills, rather than poor history knowledge?


Intra-subject analysis

• What are the areas of strength and weakness in my own teaching of this class?


Longitudinal analysis

• Are we producing better results over time in year 11 biology?


Basic analysis

2.1 2.2 2.3 2.4* 2.5* 2.6* ABS DET 1 Pamela N A N N N N 20 6 2 Lee A A A N N A 12 0 3 Manu E E E E N E 18 4 4 Keisha N A N N N N 7 8 5 Bron E M M N N A 3 0 6 Deane M M E M N A 2 1 7 Slane N A N N N N 22 8 8 Sam A A N A A A 12 8 9 Sione M M N N N N 2 2 10 Oran A A A A A A 7 0 11 Shirin E E E E A E 6 0 12 Hanna E E M M A M 0 1 13 Val E E E E N E 0 0 14 Liam N A M M N M 10 2 15 Morgan M M M M N M 15 0 16 Hone N A N N N N 17 4 17 Mahi A A N A A A 10 0


Basic analysis

• Divide the class into three groups on the basis of overall achievement

• Identify students who are doing so well at level 2 that they could be working at a higher level

• Find trends for males and females, those who are absent often, or have many detentions

• Compare this group’s external assessment success rate with the national cohort.


Reading levels – terms 1 and 4

.


Making sense of the results

Think about significance and confidenceHow significant are any apparent trends?How much confidence can we have in the information?


Making sense of the results

Is this information? Can we act on it?

This table shows that reading levels overall were higher in term 4 than in term 1. Scores improved for most students. 20% of students moved into Level 5. But the median score is still 4A.



Trigger Clues found in data, hunchesExplore Is there really an issue?Question What do you want to know?Assemble Get all useful evidence togetherAnalyse Process data and other

evidence> Interpret What information do we have?

Intervene Design and carry out actionEvaluate What was the impact?Reflect What will we change?


Making sense of information

Data becomes information when it is categorised, analysed, summarised and placed in context. Information therefore is data endowed with relevance and purpose. Information is developed into knowledge when it is used to make comparisons, assess consequences, establish connections and engage in dialogue. Knowledge … can be seen as information that comes laden with experience, judgment, intuition and values.

Empson (1999) cited in Mason (2003)


Information

Knowledge gained from analysing data and making meaning from evidenceInformation is knowledge (or understanding) that can inform your decisions. How certain you will be about this knowledge depends on a number of factors: where your data came from, how reliable it was, how rigorous your analysis was. So the information you get from analysing data could be a conclusion, a trend, a possibility.


Information

Summative information is useful for reporting against targets and as general feedback to teachers. Formative information is information we can act on – it informs decision-making that can improve learning.


Reading levels – terms 1 and 4


Questions to elicit information

• Did the more able students make significant progress, but not the lower quartile?

• How have the scores of individual students changed?• How many remain on the same level?• How much have our teaching approaches contributed to

this result? • How much of this shift in scores is due to students’

predictable progress? Is there any data that will enable us to compare our students with a national cohort?

• How does this shift compare with previous Year 9 cohorts?


Words, words, words …

Information can … establish, indicate, confirm, reinforce, back up, stress, highlight, state, imply, suggest, hint at, cast doubt on, refute …•Does this confirm that …?•What does this suggest?•What are the implications of …?•How confident are we about this conclusion?


Interrogate the information

• Is this the sort of result we envisaged? If not, why?• How does this information compare with the results of

other research or the experiences of other schools?• Are there other variables that could account for this

result?• Should we set this information alongside other data or

evidence to give us richer information? • What new questions arise from this information?



• Does this relate to student achievement - or does it actually tell us something about our teaching practices?

• Does this information suggest that the school’s strategic goals and targets are realistic and achievable? If not, how should they change, or should we change?

• Does the information suggest we need to modify programmes or design different programmes?

• Does the information suggest changes need to be made to school systems?



What effect is the new 6-day x 50-min period structure having on student engagement levels?



What effect is the new 6-day x 50-min period structure having on student engagement levels?Do student views align with staff views?Do positive effects outweigh negative effects? Is there justification for reviewing the policy?Does the information imply changes need to be made to teaching practices or techniques? Does the information offer any hint about what sort of changes might work?



Trigger Clues found in data, hunchesExplore Is there really an issue?Question What do you want to know?Assemble Get all useful evidence togetherAnalyse Process data and other evidenceInterpret What information do you have?Intervene Design and carry out actionEvaluate What was the impact?Reflect What will we change?

Date post:	07-Jan-2016
Category:	Documents
Upload:	jadzia
View:	26 times
Download:	0 times

Consider the Evidence

Documents