Date post: | 17-Dec-2015 |
Category: |
Documents |
Upload: | eric-shepherd |
View: | 216 times |
Download: | 0 times |
© C
CS
R
5Essentials: Background and methodology
Rachel LevensteinSenior Manager for Survey [email protected]
© C
CS
R
20 years of research in Chicago Public Schools (CPS)- Conversations with practitioners- Wide scan of the literature- Surveys
Organizing Schools for Improvement
Origin of the 5Essentials
© C
CS
R
Surveys of CPS
1999
2001
2003
2005
2007
2009
2011
2012
2013
2014
2015
0
0.2
0.4
0.6
0.8
1
Student and teacher response rates
Students Teachers
© C
CS
R
Initial measure development process conducted in CPS
• Prior research findings indicate a link between construct and student outcomes1. Search existing literature for survey questions
2. Focus groups Draft questions
3. Cognitive interviews Revise questions
4. Pilot survey questionsCheck reliability, revise questions
• Validation study
© C
CS
R
Development of survey questions: Focus groups
• Gather 5-10 practitioners or students together to talk about the concept.
• Moderated by a researcher, using a protocol.
• Identify themes and common nomenclature.
Get ideas for how to write survey questions
© C
CS
R
Development of survey questions: Cognitive Interviews• One-on-one interviews with practitioners,
students• “Tell me how you came to your answer”• “What kinds of programs were you thinking
about?”
Identify problems understanding or interpreting the questions.
© C
CS
R
Development of survey questions: Pilot testing
• Small- or large-scale pilot• Same conditions as the “regular” survey
– May be conducted as part of the “regular” survey
• Analyze responses and calculate reliability.
Identify if the measure holds together (or not).
© C
CS
R
Development of survey questions:Validation study
Program Coherence example: Newman et al., 2001
– Qualitative data collection: (School improvement plans, classroom observations, staff meeting observations, assignment samples)
• Reports highly correlated with survey measure
– Schools that improved on Program Coherence between 1994 and 1997 showed more improvement on ITBS than schools that did not improve—by 1/5 of a school year!
© C
CS
R
Practices change Understanding of constructs change General revisions for clarity Clients outside of Chicago inform some changes
May add, delete, or modify questions every few years.
Changes to content
© C
CS
R
To what extent do you agree or disagree with each of the following statements?
- You can see real continuity from one program to another in this school.
- Many special programs come and go in this school.- Once we start a new program, we follow up to
make sure that it’s working.- We have so many different programs in this school
that I can’t keep track of them all.- (Strongly disagree, disagree, agree, strongly agree)
Changes to content: Program Coherence
© C
CS
R
To what extent do you agree or disagree with each of the following statements?
- You can see real continuity from one program to another in this school.
- Once we start a new program, we follow up to make sure that it’s working.
- Many special programs come and go in this school.- Once we start a new program, we follow up to make sure
that it’s working.- We have so many different programs in this school that I
can’t keep track of them all.- (Strongly disagree, disagree, agree, strongly agree)
Changes to content: Program Coherence
© C
CS
R
How much influence do teachers have over school policy in each of the areas below?- Hiring new professional personnel.- Planning how discretionary school funds should be used.- Determining books and other instructional materials used
in classrooms.- Establishing the curriculum and instructional programs.- Setting standards for student behavior.(Not at all, A little, Some, To a great extent)
Changes to content: Teacher Influence
© C
CS
R
How much influence do teachers have over school policy in each of the areas below?- Hiring new professional personnel.- Planning how discretionary school funds should be used.- Determining books and other instructional materials used in
classrooms.- Establishing the curriculum and instructional programs.- Setting standards for student behavior.
(Not at all, A little, Some, To a great extent)
(No influence, A little influence, Some influence, A great deal of influence)
Changes to content: Teacher Influence
© C
CS
R
Putting a measure into the 5Essentials
• Two years of data before it becomes an Essential
• Fit within 5Essentials framework– Factor analysis