John Anderson Todd Rogers Don Klinger University of Victoria University of Alberta Queen’s University

Charles Ungerleider Barry Anderson Victor GlickmanUniversity of British ColumbiaBC Ministry of Education Edudata Canada

Funding

Canadian Education Statistics CouncilCanadian Education Statistics Council

Social Sciences & Humanities Research CouncilSocial Sciences & Humanities Research Council

The project focus

Modeling the relationships of student, school, and home characteristics to

the achievement of learning outcomesin the domains of

reading, writing, mathematics and science

Utilizing hierarchical linear modeling

&

School Achievement Indicators Program

Education Quality & Accountability Office program

Alberta Provincial Language Arts & Mathematics Achievement Tests

BC Foundational Skills Assessment program

datasets

Outcomes

Data issuesGraduate researchFindings

Next Steps

Data issues Complexity of datasets

Problem solving – age 13

Problem solving – age 16

Math content – age 13

Math content – age 16

Student achievement tests

Student Questionnaires

Teacher Questionnaires

Principal Questionnaires

Data issues Organization of assessment program

School-based

First, it should be noted that for both Language Arts and Mathematics, most of the variation in achievement was among students :

• 77.1% in Language Arts

• 75.1% in Mathematics.

class level:

• 15.3% for Language Arts

• 15.7% for Mathematics.

school level

•10.1% for Language Arts

•11.3% for Mathematics.

The Alberta study

__________________________________________________ Test ρ

_________________________________________________

SAIP Math 2001

Problem solving – age 13 0.18Problem solving – age 16 0.15

Math content – age 13 0.19Math content – age 16 0.15

OSSLT

Reading 0.13Writing 0.10

__________________________________________________

PISA average is 0.34 and ranges from .04 to 0.63

Data issues Data integrity

Student Gender Distribution 

Gender: Inside questionnaire

Gender/Cover Male Female Total

Male 4,456 1,388 5,166

Female 1,563 4,689 5,589

Total 6,019 6,077 12,096

Data issues Missing Data

SAIP Math

Parental Educational Level (Items 24 a&b)

34% missing on mother

36% missing on father

Parental Vocational Status (Items 25a&b)

53% missing on mother

40% missing on father

Data issues

Large number of variables

Student beliefs about mathematics

Derived variables from Student QuestionnaireDerived variables from Student Questionnaire

•Math is more difficult than other school subjects

•I am not very interested in mathematics

•I learn lots of new things in mathematics

•Math is an important school subject

•Math is important for my future studies

•Many good jobs require math

Derived variables from Student QuestionnaireDerived variables from Student Questionnaire

• You & your parents work on math homework

• You & your parents work on other homework

• In math course we work in pairs or small groups

• In math we use math books & magazines

• In math we had guest speakers or experts

• In math we use computers

• In math we use the internet

• In math we use the computer lab

Instructional supports used by students

Derived variables from Student QuestionnaireDerived variables from Student Questionnaire

Instructional practices In math courses this year. . . • The teacher gives notes• The teacher shows us how to do problems • We participate in math projects • We are taught different ways to solve problems • The teacher assigns homework • We discuss quiz or tests • We work alone on assigned work • We work on exercises from textbook • We study the textbook • The teacher reads from the textbook • Teachers asks questions of students • Students ask teacher questions

Causes of math performance

• To do well in math you need hard work

• To do well in math you need encouragement - teachers

• To do well in math you need encouragement - parents

• To do well in math you need good teaching

Derived variables from Student QuestionnaireDerived variables from Student Questionnaire

Disciplinary climate

In math courses this year. . .

• There is noise or disorder in the classroom

• We lose 5-10 minutes because of disruptions

Derived variables from Student QuestionnaireDerived variables from Student Questionnaire

Graduate researchCSSE 2004

Potential and Pitfalls of Secondary Data Analyses of SAIP data. Todd Rogers & Teresa Dawber, U of AlbertaCSSE 2005: The COLO Project 2005 Graduate Symposium

Student and school indices in SAIP questionnairesCarmen Gress & Shelley Ross, UVic

Correlates of mathematics achievement: a meta-synthesis

Margot English, Shelley Ross, Carmen Gress, UVicIssues and results arising from the HLM analysis of the Ontario Secondary School Literacy Test.

Chloe Soiblelman, Jinyan Huang, Cheryl Poth, &

Don Klinger, Queen’s UniversityFactors that influence writing performance

Jiawen Zhou, University of AlbertaStability of SAIP Factor Analysis: Results from school questionnaire items

Ally Feng, University of Alberta

Findings

No grand general models

Student level (level 1) coefficients

_______________________________________________________________________ Correlate CONTENT PROBLEM

13 16 13 16 _______________________________________________________________________

Student math beliefs .36 .33 .38 .37

Instructional supports -.18 -.22 -.22 -.29

Instructional practices .03 .08 .04 .10

Causes of math -.08 0 -.06 0

Discipline climate 0 0 0 0

Gender 0 -.09 .10 0.7

School level (level 2) coefficients for average school math score

_______________________________________________________________________ Correlate CONTENT PROBLEM

13 * 16 * 13 * 16 * _______________________________________________________________________

Limits to learning -.14 -.21 -.14 -.18

Instructional supports -.12 -.19 -.10 -.19

Causes of math 0 -.22 0 -.22

Discipline climate 0 0 0 -.17Student math beliefs .13 0

.11 0

School climate 0 0 -.04 -.05Parent engagement 0 .05 0 .06Student status .07 .06 .05 0Student achievement .05 0 .05 0Instructional practices .09 0 0 0

Findings Perhaps no grand models, but

As Lindblom (1968, 1990) has pointed out time and again, the desire that models of complex social systems such as public

education have an instrumental use remains an elusive dream.

Models of complex social systems are likely to be, at best, enlightening – allowing incrementally expanding understandings of

complex and dynamic systems such as public schools

(Kennedy, 1999)

The low rho suggests that

• Canadian schools are relatively homogeneous

&

• Most variation in achievement results lie within classrooms and

between students

Findings The specificity of models to grade

and domain suggests that the correlates of learning outcomes have to be considered within the

context of specific learning situations

For example . . . . .

Findings

The SAIP Math models show that

Student attitudes about math are related to achievement

Student dependence is related to math achievement

The views of school principals in regard to instructional impediments are related to

average school math scores

Gender tends to have a much reduced relationship to achievement when other

variables are entered into the model

Climate, Discipline and Parental Involvement – non-operative

Next Steps

Linkage with other assessment programs

Work with other educational partners:

Teachers

Parents

Ministries

Communications

Data collection

Analysis and application