Post on 17-Jan-2016
transcript
+
Working with SMART2Workshop for leaders in the use of SMART2
Workshop presentation 20/09/2010
Andrew Fraser and Carmel Kriz
+Objectives
Gain greater understanding of using data from an inquiry frame of reference
Develop greater familiarity with the SMART2 web-based application
Undertake an initial analysis of school SMART2 data
Identify strengths and issues to explore further
Develop strategies for effective leadership in the analysis and use of information
+
Where was I last weekend?
+ Which gives you an accurate answer?
+
Data MattersLeading learning with an inquiry habit of mind
+
Setting the context
A climate of accountability
+Using data to learn
What comes to mind when you hear the word:
DATA
(60 seconds)
+Using data to learn
What comes to mind when you hear the word:
INFORMATION
(60 seconds)
+A commitment to action
(p. 16)
Schools need reliable, rich data on the performance of their students because they
have the primary accountability
for improving student outcomes.
Good quality data supports each school to
improve outcomes for all of the students.
+
What is accountability?
+What is Accountability?
Earl and LeMahieu, 1997
As cited in “Leading learning in a data-rich world” Earl and Katz (2006: 10)
Accountability is
the conversation
about what the information means
and
how it fits with everything else
that we know
and
about how to use it
to make positive changes
+
Opening up productive inquiry
Some principles
Internal motivation: Teachers improve through learning their craft, which happens best in PLCs which have access to valid data about achievement.
Data: the divide…
Judgment
Questions
Inquiry Habit of Mind
Professional Learning Communities Requires
imagination
What happens?
The underlying assumptions
External motivation: Teachers are motivated to improve by the idea that someone is watching over them, judging and visiting consequences on them if the targets aren’t met. Thanks to John DeCourcey
How do you use data in your school?
+
The data, used well, frame the right The data, used well, frame the right questions;questions;Used poorly, they rush to judgmentUsed poorly, they rush to judgment
+Asking the right questions require imagination
+Types of data
Outcome
• External tests (SC, HSC)• A – E Reporting Grades• Retention rates• Enrolment trends• Behavioural data
Demographic
• Gender• LBOTE• ESL• SES• AEDI• Languages• Mobility• Staff age/mobility
Process
• Observation of practice• Descriptive data• Policy• Quality of practice• Professional development• School organisation
Perceptual
• Parent survey• Focus groups• Satisfaction surveys• Student attitudinal data• Community perceptions
Data worth looking at:
“The value of the data emerges only when analysis provides insights that direct decisions for students.”
Stephen White,
Beyond the Numbers, 2005
+From Data to Professional Knowledge
Data are
Making useful information
Developing informed professional knowledge
+Data Information Knowledge
Information Actionable knowledge
Types of Decisions• To identify or clarify a problem • Set and monitor progress towards goals• Address individual and group needs• Monitor and evaluate practices• Validate proposals for change • Assess whether student needs are being met• Strategically allocate resources• Adapt a new practice to fit the situation
+Collaboration
“Data analysis is a team sport.”Doug Reeves
Develops team thinking
Promotes insights that numbers alone can’t produce
Provides a forum for legitimizing practice
A characteristic of “Schools that Learn”
Wonderful learning?
He scored 42/60 on the
midyear testLazy?
Superb teaching?
Dreadful cheat?
Evidence:
What does this indicate?
Construct:
Top mark in the class, student has been studying well, highly motivated…Lowest he’s ever achieved, no sign of any preparation, smart student, easy testWhole class is showing improvement, class average is better than ever gained before…Stole the answer sheet from the teacher’s desk the day before…
Almost any evidence can be an indicator of many different
constructs – finding the most
productive questions is the
art of data analysis.
Thanks to John DeCourcey
+
Using data well
Use it to ask the right questions
‘Triangulate’ the construct – what other way do I have of looking at it?
Decide on a direction – go top-down, or bottom-up
Slice the data different ways: mean, top, bottom, time, individual map
Test the ‘Strength’ of the evidence: is it a sound link to the construct?
Be suspicious of yourself: ask the next question
Thanks to John DeCourcey
+Three Principles of Data Analysis
Exploring and determining the antecedents for success
Collaborating with colleagues
Embracing Accountability - Learning from our data
Connecting Cause and Effect Data at the program/initiative level
+Some questions for consideration
“What does this ask us about how teaching and learning are going in our school?”
“What do we need to do about it?”
“What do we need to learn?”
+
The NAPLAN Scale
Achievement Scale
Band 1
Band 2
Band 3
Band 4
Band 5
Band 6
Band 7
Band 8
Band 9
Band 10
Year 3
Year 5
Year 7
Year 9
Each Year level Student Report shows 6 of the bands.
Year 9 Reports show bands 5 to 10
Year 7 Reports show bands 4 to 9
Year 5 Reports show bands 3 to 8
Year 3 Reports show bands 1 to 6
Students are at the national minimum standard
Achievement Bands
Students are below the national minimum standard
Year 3
Band 1
Band 2
Band 3
Band 4
Band 5
Band 6
Band 3
Band 4
Band 5
Band 6
Band 7
Band 8
Year 5
Year 7
Year 9
Band 4
Band 5
Band 6
Band 7
Band 8
Band 9
Band 5
Band 6
Band 7
Band 8
Band 9
Band 10
Students are proficientStudents working at proficiency would be in the top 2 achievement bands for the respective year level.
Students that are in Band 1 are deemed at operating at below minimum standard.
Students working at minimum standard would be in the second lowest band.
NAPLAN Scales
There are five separate national scales, one each for:
ReadingWriting
Spelling,Grammar/Punctuation
and Numeracy.
Note: There are no more benchmarks. Students that are in the lowest band are deemed to be operating below minimum standard.
Introducing …
1
2
3
4
5 6
7
8
Creating groupsSelect: Manage Groups
+
Using SMART2 to analyse NAPLAN data
+Using Means and Standard Deviations
Good starting point
Gives overall school performance
Helps frame questions for further investigation
Gives difference in mean between the school and the state
Can be found in Means &Standard Deviations
Question : How significant is the difference from the state????
+ Means and Standard Deviations
When student numbers are small, summaries based on mean (average) scores can be misleading. In small groups, means can be affected by a couple of high performing or low performing students.
Means sometimes obscure real differences within a group, even when the numbers are large. For example, a school with concentrations of both educationally advantaged and disadvantaged students may find that the school mean actually describes very few of its students.
For schools with year cohorts less than 5 students no school means and deviations will be displayed.
+ Means and Standard DeviationsSchool Reports
Select aspect – Reading or Numeracy
Compare the mean for the full cohort against the state mean
Compare full cohort with boys, girls, ATSI
Compare means in different strands of literacy (writing, reading etc) or numeracy (number, measurement )
+ Means and Standard DeviationsSchool Report
+Significance of difference in means
School mean – state mean, divided by the state SD.
This calculation is about substantive meaning of any difference. It is not a test of statistical difference.
+Significance of difference in means Rule of thumb method :
> 0.5 Well above state
0.5 > 0.2 Above state
0.2 > -0.2 Within state
-0.2 > -0.5 Below state
< -0.5 Well below state
+Further investigationsIs the Significant difference of the mean:Because of the results of a particular
group? Eg.boys, girlsBecause of the results of a particular class?
Eg. class group that has had 4 different teachers in one year
The same for both year levels (3 & 5, 7 & 9)?
Has this been the pattern in previous years?
+Means and Standard Deviations
Significant difference of the mean allows for comparison between year levels
Means and Standard Deviations allow for creation of means table for special groups
Means and Standard Deviations allow for comparison with school groups
+Trend Data
+Percentage in Bands
Choose Percentages in Bands from main menu.
View for full cohort, boys, girls, ATSI, or custom groups
Record bands where difference from the state is significant.
* less students in top band or top two bands
* more students in the bottom band (below National Minimum Standard)
* more students in the second bottom band (at the National Minimum Standard)
OR of course the reverse
+ Percentage in Bands
+Further Investigations
Check to see if there is there is a significant difference in performance between the strands e.g.
In Literacy - between writing and reading
In Numeracy - between Number/Patterns & Algebra and Data/ Measurement /Space and Geometry
More detailed investigation can be carried out through the Item Analysis.
Questions need to be asked re pedagogy in these areas – this could form part of the School Improvement Plan.
+Percentage in Bands
Record names of students in the bottom two bands
Create a group of these students. (Create /Edit and Delete Groups)
Go to Item Analysis to look at which items all this group have correct /incorrect
+Percentage in Bands
Check names of students in particular bands.
Are there any surprises?
Follow up responses of individual students whose band placement is of concern by going to Student Analysis
+Value added Percentages in Bands may be of use for
schools where they have tried to increase the movement of students into higher bands.
Care must be taken when using this information with schools who have less than 10 students. In schools with small numbers each individual student is worth a large percentage and this may impact greatly on the student numbers in a band.
+ Value added - Percentage in Bands
+Student Growth (Years 5, 7, 9)
Select Aspect – Reading or Numeracy
Select Student Growth
+ Student Growth
+ Student Growth Arrows indicate growth for individual students (3 to 5, 5
to 7, 7 to 9).
Length of arrow indicates amount of growth.
Orange arrows indicates growth that is greater than or equal to expected growth for starting point.
Blue arrows indicate growth that is less than expected growth for starting point.
Downward blue arrow indicates student’s score less than their last score
Click on arrows for students’ names and scores.
Read from left to right – lowest performing students in the cohort’s last NAPLAN on the left – highest performing on the right.
+Student Growth Check which students have made the most
growth.
Is it the lowest third, middle third or upper third based on prior performance?
Is there any relationship between amount of growth and any teaching emphasis or intervention for a particular group?
Is there a group of students who have not made expected growth and who are clustered together?
Make a group of these students and examine their responses in Item Analysis and/or Student Analysis looking for common errors
+ Student Growth
+Student Growth
Check the tables on lower half of screen:
Average scaled score growth
Expected growth
+Student Growth
The tram lines on the main graph indicate the growth made by 75 %, 50 % and 25% of students in the state.
The middle table on the right hand side indicates the % of students in the school in the bottom 25th percentile, and the top 25th percentile and between the 25th and 75th percentile.
Does the school’s growth fit the anticipated 25 50 25 pattern ?
+Item Analysis
Use this table together with Item Analysis Table in the Main Menu.
Print: Report - Analysis by Questions (for required aspect – reading or numeracy and required year level)
Report – Analysis by Questions provides information on % of students choosing correct / incorrect options.
When investigating questions that a large numbers of students answered incorrectly, it is often helpful to ascertain whether a number of them choose the same incorrect answer.
+
+Item Analysis
Go to Item Analysis
Select Year Level
Select Aspect
Sorting of items in different ways.
+Item Analysis
+Item Analysis Sort by School % Correct
Investigate items where school percentage correct is below state by 10% or more.
Using Report – Analysis by Questions to check for common response errors.
+Item Analysis Sort by Syllabus Outcomes
Check: Which Items have/have not been answered correctly by the majority of students in the Year.
Check for common incorrect responses.
+Item Analysis
Select Filter by Substrands (drop-down menu)
Investigate performance on items assessing particular skill
Using Report check for common response errors
Check out the actual item using SCAN.
+
+Item Analysis
can be viewed graphically - choose CHART
+Item Analysis
Select : Filter by Group
Investigate aspects of Item Analysis for various subgroups – boys, girls, custom groups.
+Item Analysis
Compare performance of Year groups in a particular skill area.
Are there any links items in this set of items?
What are the skills year groups are are finding difficult ?
+Leading, Inquiring, TransformingDeveloping an inquiry habit of mind
Being data literate
Creating a culture of Inquiry
+School leadership teams
Familiar with, and competent in, analysing and understanding NAPLAN data
Takes an active role in the analysing and use of the data with staff to bring about measurable improvements in student learning.
+ Towards 20112010
Term 3
Term 4
2011
Analyse NAPLAN data Identify issues to follow up Engage others in the analysis Examine other data
Formulate a whole-school response – School Improvement Plan
Develop specific strategies to implement Determine measureables, milestones,
indicators of improvement
Implement plan Explore changes – cause and effect Work with SMART 2 Teaching Strategies
Monitor and evaluate – evidence of change
Critique impact
Inquire
Plan
Implement
Reflect
+Monitoring change
How will you know change is or has occurred?
What evidence will you use to monitor the transformation of learners?
What evidence will you use to monitor the transformation of learning?
+Acknowledgements
Dr Philip Pettit and the Canberra – Goulburn CEO
Rosemary Vellar and Sydney CEO
Dr John DeCourcy
NSW Educational Measurement and School Accountability Directorate
Australian Government National Partnerships Agreement
+Further assistance
Further support and advice can obtained by contacting the Catholic Schools Office, Broken Bay:
Andrew Fraserandrew.fraser@dbb.catholic.edu.au
Carmel Kriz carmel.kriz@dbb.catholic.edu.au
Or your school’s consultant.
+Reflecting on questions
“What does this ask us about how teaching and learning are going in our school?”
“What do we need to do about it?”
“What do we need to learn?”
Cause and Effect data
Effect Data – outcomes or results in student learning and achievement
Cause Data – professional practices (adult actions) that create specific effects or results
+ Teacher inquiry and knowledge-building cycle to promote valued student outcomes
•What do they already know?•What sources of evidence
have we used?•What do they need to learn
and do?•How do we build on what
they know?
•How effective has what wehave learned and done beenIn promoting our students’Learning and well-being?
•How have we contributed toexisting student outcomes?
•What do we already know that wecan use to promote valued
outcomes?•What do we need to learn to do to
promote valued outcomes?•What sources of evidence/knowledge can we utilise?
What are our students’ learning needs?
What has been the impact of our
changed actions?
What are our own learning needs?
Design of tasks andexperiences
Teaching actionsH. Timperley, A Wilson, H Barrar & I Fung (2007)
Teacher Professional Learning and Development: Best Evidence Synthesis Iteration
Wellington, New Zealand: Ministry of Education
http://educationcounts.edcentre.govt.nz/goto/BES
+ Leader inquiry and knowledge-building cycle to promote valued student outcomes
What knowledgeand skills
do our teachers andstudents need?
What has been the impact of our changed actions
on teachers and students?
What knowledge and skillsdo we as leaders need?
What are our teachers’ learning needs?
What has been the impact of our
changed actions?
What are our own learning needs?
Develop leadershipknowledge
and refine
leadership skills.
Engage teachersIn new
learning experiences
Modified from H. Timperley (2010)
+Data Teams
Guidelines for effective Data Teams1. Have collaborative teams2. Provide adequate time for collaboration3. Engage in collective inquiry4. Focus on the cause and effect data5. Post graphs and charts so they are visible6. Subscribe to action orientation and
experimentation7. React to our data with sound instructional and
curricular decisions8. Implement an effective communication9. Are results driven10. Are devoted to continuous improvement
+
Data literacy – a thinking process
Standing back and thinking about what you need to know and why
Collecting or locating the necessary data
Finding ways to link data sources
Ensuring that data are worth considering
Being aware of their limitations
Thinking about what the results mean
Systematically considering an issue from a range of perspectives so that you feel that you have evidence to explain, support, and also challenge your point of view.
+Data Information Knowledge
ONLY WHEN
Shaped
Organised
Embedded in a context
That gives meaning and connectedness.
Adapted from: van Barneveld (2008) and Earl and Katz (2006)
Which effect data to focus on?“Will this piece of data help a classroom teacher change curriculum, assessment and instruction and thus improve student achievement?”
Douglas Reeves