Date post: | 29-Mar-2015 |
Category: |
Documents |
Upload: | johana-lum |
View: | 214 times |
Download: | 0 times |
Using Mixed Methods to Develop a Questionnaire
on Accountability for Teachers and Principals
Nora Gannon, Tysza Gandha & Katherine Ryan 2009 CREATE Conference
October 9, 2009
Accountability and Evaluation
Using Questionnaires in Evaluation
Using Mixed Methods in Questionnaire Development
Our Mixed Methods Design and
Implementation
Summary
Overview
No Child Left Behind (NCLB)
Credible Evidence
Improving Evaluation Methods
Accountability and Evaluation
Key Questions◦ What are the intended and unintended
consequences of the accountability assessments utilized in this state?
◦ What are the perspectives of teachers and principals on how these consequences influence: Instructional practices? Local assessment practices? Use of test data? School policies and practices? The Teaching profession?
Instate Assessment Consequences Evaluation (IACE)
◦Illuminate large-scale patterns in schools reform as a result of accountability systems in place
◦Provide broad-based views of stakeholders’ perspectives (e.g., teachers and principals)
◦Time and cost efficient
Use of Questionnaires in Large-scale Educational Evaluations
Reference: Desimone & le Floch, 2004
Advantages to Using Questionnaires
Challenge for Researchers
• Administer to many people to learn perspectives of diverse stakeholders
How do we know if a respondent’s understand questionnaire items as we intended?
• Large scale survey study can examine a wide range of contexts (e.g., school settings)
How can we strengthen our inferences across contexts?
Reduce response bias
Reduce response burden
Improve item clarity
GOAL: Understand thought process and improve data
interpretability
Reference: Czaja & Blair, 2005
◦Purposeful
◦Systematic
◦Comprehensive
Mixed Methods for Instrument Development
Questionnaire Development: Combination
Sequential/Concurrent Mixed Methods Design
qual Data analysis
qual Data collection
QUAN Data analysis
QUAN Data collection QUAL
Data analysis
QUAL Data collection
QUAN & qual
Phase 2
QUAL
Phase 1
Integration of Entire Analysis
PHASE 1: QUALITATIVE
qual Data analysis
qual Data collection
QUAN Data analysis
QUAN Data collection QUAL
Data analysis
QUAL Data collection
QUAN & qual
Phase 2
QUAL
Phase 1
Integration of Entire Analysis
PHASE 1: QUALITATIVE
Methods• Item Development• Cognitive Labs• Expert Review
Literature Review◦ 5 dimensions: Instructional practices, Local assessment
practices, Use of test data, School policies and practices, Teaching profession
◦ Dimension Task Lists
Item development◦ Item Bank◦ Reverse engineering
Content Review
Item Development
PHASE 1: QUALITATIVE
Item DevelopmentExample of Partial Task List for Changes in Instructional Practices
Test Preparation Kinds of test preparation strategies and frequency of use o Test taking tips (marking answers correctly,
eliminating wrong answers etc) o Using tests of similar format to state test o Using items of similar format of items on
state test o Using old test items from state test o Using entire tests from past as practice o Commercial test prep o Focus on content that is likely to be tested
Teachers’ perspectives in test preparation (i.e. ethics) o Easy to understand and pull information from o Teachers receive training in reading reports
PHASE 1: QUALITATIVE
Literature Review◦ 5 dimensions: Instructional practices, Local assessment
practices, Use of test data, School policies and practices, Teaching profession
◦ Dimension Task Lists
Item development◦ Item Bank◦ Reverse engineering
Content Review
Item Development
PHASE 1: QUALITATIVE
What is the purpose of a cognitive lab?◦ Gain insight into participant understandings.
How do you sample for cognitive labs?◦ Choose participants that represent your
population.
Implementation: the think-aloud process◦ Be careful not to influence responses!
Cognitive Labs
PHASE 1: QUALITATIVE
Single vs. Dual Rating Scales
Cognitive Lab Findings
PHASE 1: QUALITATIVE
How have you changed your instructional strategies as a result of NCLB testing?
Overall, my use of this strategy has… In
crea
sed
a
lot
Incr
ease
d
som
ewh
at
Not
ch
ange
d
Dec
reas
ed
som
ewh
at
Dec
reas
ed a
lot
I do
n’t
use
this
st
rate
gy
a. Project-based assignments
5 4 3 2 1
Single vs. Dual Rating Scales
Cognitive Lab Findings
PHASE 1: QUALITATIVE
Since NCLB testing began in grades 3-8, how have you changed your instructional strategies? To what extent were the changes a result of NCLB testing? [Please rate the item on both scales]
Overall, my use of this strategy has… To what extent was it a result
of NCLB testing?
Incr
ease
d
a lo
t
Incr
ease
d
som
ewh
at
Not
ch
ange
d
Dec
reas
ed
som
ewh
at
Dec
reas
ed a
lot
I do
n’t
use
this
st
rate
gy
To
a gr
eat
exte
nt
To
a m
oder
ate
exte
nt
To
a sm
all
exte
nt
Not
at
all
a. Project-based assignments
5 4 3 2 1 4 3 2 1
Global Questions
Cognitive Lab Findings
To what extent, and how, have NCLB testing changed your instruction since it began in all grades 3-8?
Changed it a lot,
for the better
Changed it somewhat,
for the better
Did not change
it
Changed it somewhat,
for the worse
Changed it a lot, for the worse
5 4 3 2 1
PHASE 1: QUALITATIVE
Cognitive Lab Findings
1. To what extent has NCLB testing had positive effects on your instruction since it began in grades 3-8?
To a great
positive extent
To a moderate positive extent
To a small positive extent
Not at all
4 3 2 1
2. To what extent has
NCLB testing had negative effects on instruction in your school since it began in grades 3-8?
To a great
negative extent
To a moderate negative
extent
To a small negative
extent
Not at all
4 3 2 1
PHASE 1: QUALITATIVE
Language
What does “test preparation materials” mean?
Cognitive Lab Findings
PHASE 1: QUALITATIVE
Since NCLB testing began in grades 3-8, how has the use of the following instructional strategies changed in your school? To what extent were the changes a result of NCLB testing?
[Please rate each item on both scales]
My school’s use of this strategy has… To what extent
was it a result of NCLB testing?
Inc
reas
ed
a
lo
t
Inc
reas
ed
s
om
ew
ha
t
No
t c
ha
ng
ed
De
cre
ase
d
so
me
wh
at
De
cre
ase
d
a l
ot
We
do
n’t
u
se
th
is
str
ate
gy
A l
ot
So
me
Ve
ry l
ittl
e
No
t a
t a
ll
a. Using test preparation materials to help students prepare for the NCLB test
5 4 3 2 1 4 3 2 1
Language
◦ Strategies for Addressing Language Issues
Examples
Alternate words
Multiple questions
Cognitive Lab Findings
PHASE 1: QUALITATIVE
Why use Expert Reviewers?
Expert Reviewers
PHASE 1: QUALITATIVE
When to engage expert reviewers?
Early• Can solicit broad, big picture feedback
• Have time to make major changes with feedback
Later• Can solicit highly specific (item-level) feedback
• Useful for final review of instruments in almost-final draft
Reference: Ramirez, 2002
PHASE 2: QUANTITATIVE and qualitative (concurrent)
qual Data analysis
qual Data collection
QUAN Data analysis
QUAN Data collection QUAL
Data analysis
QUAL Data collection
QUAN & qual
Phase 2
QUAL
Phase 1
Integration of Entire Analysis
Methods• Pilot Questionnaire• Cognitive Lab
Focus Group
PHASE 2: QUANTITATIVE AND qualitative
Teacher and Principal Questionnaires◦ Approximately 200 items; 60% shared between
teacher and principal versions
Sample- 383 principals - 2415 teachers
(1940 online, 475 mail)
Proportionate Probability Sample (N=400 schools)
◦ Stratified by geographic region and school type (i.e. elementary schools and middle schools)
◦ Sampling Criteria (AYP Subgroups)
Pilot QuestionnaireAdministration
PHASE 2: QUANTITATIVE AND qualitative
Teachers: 37% (902)
◦ Online: 44%◦ Paper-and-pencil: 9%
Principals: 44% (175)
Pilot QuestionnaireResponse Rates
PHASE 2: QUANTITATIVE AND qualitative
Population proportions
Response rate
proportions
Elementary 60% 57%
Middle School
40% 43%
Pilot QuestionnaireFindings
Refine content of Questionnaire based on◦ Low response endorsement◦ Low response rates◦ Low variance on scaled items
Pilot Questionnaire Findings
PHASE 2: QUANTITATIVE AND qualitative
IMPLEMENTATION: Cognitive lab in small group setting (4-6 people)
PURPOSE: Group narrative
SAMPLE: Depends on what you want to know
Cognitive Focus Group
Strengths Limitations• Group setting allows for
conversations that elicit normal narrative
• Cost-efficient way to learn about potentially different interpretations
• Have to use smaller number of items than individual cognitive lab
• Group relations need to be considered (e.g., include teachers and principal?)
PHASE 2: QUANTITATIVE AND qualitative
To what extent is the change in local assessment due to the state accountability assessment?
Mr. Smith: I think assessment is just a component of good teaching.
Ms. Brown: I think you are naïve…it’s all about preparing for the state test.
Ms. Rogers: Well, it depends on how you look at it, I guess… if you are cynical everything is because of accountability assessment!
Cognitive Focus Group
PHASE 2: QUANTITATIVE AND qualitative
Goal Method• Refine scope of
questionnaire• Comprehensive
literature review • Expert reviewers• Pilot administration
• Improving the interpretability of the questionnaire
• Cognitive labs• Cognitive focus groups
• Strengthen capacity to warrant inferences within and across contexts
• Cognitive labs• Purposeful sampling for
cognitive labs• Deliberate sample frame
in pilot
Summary: the value in using mixed methods
Questions?Thank you!