Date post: | 28-Dec-2015 |
Category: |
Documents |
Upload: | leonard-sims |
View: | 213 times |
Download: | 0 times |
RTI Professional Learning Webinar Series: Using Research to Select and Design Effective Interventions – Session 4October 15, 2009
Our session will start momentarily. While you are waiting, please do the following:
Enter/edit your profile information by going to:•Tools - Preferences - My Profile…•Fill out the info on the “identity” tab and click “OK”•To view the profile of another use, hover your mouse over his or her name in the participants window Configure your microphone and speakers by going to:•Tools – audio – audio setup wizard
Confirm your connection speed by going to:•Tools – preferences – connection speed
RTI Professional Learning Webinar Series:
Using Research to Select and Design
Effective Interventions
.
2
Session #4 10/15/09 3-4:30 pmCritical Review of Research Studies
on Interventions
Session #4 10/15/09 3-4:30 pmCritical Review of Research Studies
on Interventions
Counting ALL Participants After you enter your profile information to
join the webinar as usual:
1. Go to the Chat Window
2. Type your district name and school name or organization name or “parent”
3. Type the names of every person in attendance with you
4. Send it to “This Room”
3
Who are we? Teachers Administrators Counselors or Social Workers County Office or DOE Parents PK-5, 6-8, 9-12, post-secondary Statewide Organizations Other?
4
6
REGIONAL EDUCATION LABORATORY- SOUTHEAST (REL-SE)
Executive Director Dr. Ludwig van Broekhuizen
Toll Free: 800-755-3277 www.serve.org
Operated by the SERVE Center at UNCG
7
Serving Alabama, Florida, Georgia, Mississippi, North Carolina, and South Carolina
Georgia liaison: Dr. Kim [email protected]
404-657-6174
REL-SE MISSION
To serve the educational needs of the
Southeast, using applied research,
development, dissemination, and
training and technical assistance, to
bring the latest and best research and
proven practices into school
improvement efforts.
9
REL-SE Services
1. Outreach and dissemination of research, evaluation, and policy information
2. Technical assistance to SEAs and LEAs
3. “Issues & Answers” publications
4. Randomized Controlled Trial experimental studies on interventions of relevance to our region
• AMSTI/Alabama and K-PAVE/Mississippi
10
TODAY’S OBJECTIVES 1. Reach back to Session 3
2. Understand how to assess the rigor and findings of studies on interventions
3. Practice using the Critical Reading Protocol for Studies About Interventions
4. Debrief the Protocol process and discuss the relevance of this type of process for RTI decision making
5. Preview session #4
14
Types of Research Methods handout posted on GaDOE website
Carnegie Learning 2001 & Witzel Multisensory Algebra studies
Critical Reading Protocol For Studies About Interventions handout
We discussed 5 types of research methods. What do you remember?
What thoughts/questions remain?
Descriptive Correlational Quasi-Experimental Experimental Meta-Analysis
2 min. ChatFacilitators lead their groups
Individuals utilize chat box
16
Quick Poll 1: RecallStudies using which method can make
claims that the intervention studied caused changes in student achievement?
A. Descriptive B. Correlational C. Quasi-Experimental D. Experimental E. Meta-Analysis
17
Quick Poll 2: RecallStudies using which method can discover
relationships between variables (e.g., a particular instructional strategy and academic achievement)?
A. Descriptive B. Correlational C. Quasi-Experimental D. Experimental E. Meta-Analysis
18
Quick Poll 3: ID This Research Design Citation: Bennett, J., Lubben, F., & Hogarth, S. (2006).
Bringing science to life: A synthesis of the research evidence on the effect of context-based and STS approaches to science teaching. Science Education, 10, 347-370.
Research Question: What is the average effect across studies examining the impact of context-based and STS approaches to science teaching?
Findings: The 17 studies suggested that context-based and STS approaches to science teaching improved attitudes towards science as compared to control groups.
19
What type of research design do you think the researchers used? Why?
A. Descriptive B. Correlational C. Quasi-Experimental D. Experimental E. Meta-Analysis
20
Quick Poll 4: ID This Research Design Citation: Rainey, L., Harman, P., & McColskey, W. (1996). A
report of the integrated science student impact study at the University of Alabama. Greensboro, NC: SERVE Center
Research Question: Do students in the Integrated Science Program (“ISP”) perform significantly better on performance assessments as compared to students not enrolled in the ISP?
Methods: Students in the ISP and not in the ISP were matched based on race, gender, and academic ability.
Findings: In a test of science process skills, students enrolled in the ISP performed significantly better than students not in the ISP.
21
What type of research design do you think the researchers used? Why?
A. Descriptive B. Correlational C. Quasi-Experimental D. Experimental E. Meta-Analysis
22
Objective #2
Put our knowledge of research methods together with other key information to assess the rigor and findings of studies on interventions
23
Critical Reading Protocol for Studies About Interventions
Looks at key elements of a research study’s design and results in order to determine its quality and rigor.
This protocol can be used in the process of compiling information about interventions and strategies, and determining “what works.”
This protocol is in the GaDOE RTI manual.
24
Critical Reading Protocol for Studies About Interventions
See Protocol handout
1. Authorship: Who is the author? Is it the vendor or a third party? Who funded the study? Is there an obvious bias?
2. Sample: What was the sample? How closely do the participants in the study mirror your population?
25
Critical Reading Protocol for Studies About Interventions
3. Research design: What kind of design did the researchers use? What kind of questions can this design answer (and not answer)? How were the groups formed?
*use your Types of Research Methods handout
26
Critical Reading Protocol for Studies About Interventions
4. Results: What kinds of outcomes were reported on? Are the results statistically significant? Practically significant?
5. If you were to implement this intervention what outcomes could you reasonably expect to achieve?
27
Objective #3
Practice using the Critical Reading Protocol for Studies About Interventions
Participants read sample program evaluation publications and utilize Protocol to determine the quality and rigor of the study
Then, whole group discussion
28
Sample Publication #1: Carnegie Learning 2001 Cognitive Tutor
1 min. introduction to this publication
10 min. read and answer Protocol questions (in small facilitated groups and/or in chat box)
10 min. whole group discussion: What did we learn about this research study and its application for our RTI decision making?
29
POLL: Protocol Question #1 Authorship: Who is the author? Is it the vendor or
a third party? Who funded the study? Is there an obvious bias?
A.The author is/has a relationship to the vendor, and as such may be biased
B.The author is a third party, and as such should be neutral
C.Cannot determine
D.We have a question about this…?
30
POLL: Protocol Question #2
Sample: What was the sample? How closely do the participants in the study mirror your population?
A.The participants closely mirror our population
B.The participants are not similar to our population
C.Cannot determine
D.We have a question about this…?
31
POLL: Protocol Question #3Research design: What kind of design did the
researchers use? What kind of questions can this design answer (and not answer)? How were the groups formed?
A. The design was quasi-experimental
B. The design was correlational
C. The design was experimental
D. Cannot determine or we have a question about this…?
32
POLL: Protocol Question #4
Results part 1: What kinds of outcomes were reported on?
A. Results were Texas End of Course Exam
B. Results were local assessments
C. Cannot determine
D. We have a question about this…?
33
POLL: Protocol Question #4Results part 2: Are the results statistically
significant? Practically significant?
A. Results were statistically significant
B. Results were statistically AND practically significant (in our view, the difference between scores of C.T. students and comparison group students would justify C.T.’s cost, for us)
C. Cannot determine
D. We have a question about this…?
34
POLL: Protocol Question #5
If you were to implement this intervention what outcomes could you reasonably expect to achieve? Why?
A. We would expect to achieve similar results
B. We would not expect to achieve similar results
C. Cannot determine
D. We have a question about this…?
35
Sample Publication #2: Witzel 2005 Multisensory Algebra
1 min. introduction to this publication
10 min. read and answer Protocol questions (in small facilitated groups and/or in chat box)
10 min. whole group discussion: What did we learn about this research study and its application for our RTI decision making?
36
POLL: Protocol Question #1 Authorship: Who is the author? Is it the vendor or
a third party? Who funded the study? Is there an obvious bias?
A.The author is/has a relationship to the vendor, and as such may be biased
B.The author is a third party, and as such should be neutral
C.Cannot determine
D.We have a question about this…?
37
POLL: Protocol Question #2
Sample: What was the sample? How closely do the participants in the study mirror your population?
A.The participants closely mirror our population
B.The participants are not similar to our population
C.Cannot determine
D.We have a question about this…?
38
POLL: Protocol Question #3Research design: What kind of design did the
researchers use? What kind of questions can this design answer (and not answer)? How were the groups formed?
A. The design was quasi-experimental
B. The design was experimental
C. The design was descriptive
D. Cannot determine or we have a question about this…?
39
POLL: Protocol Question #4
Results part 1: What kinds of outcomes were reported on?
A. Results were SC’s “PACT” test scores
B. Results were scores on a local assessment designed specifically for use in the study
C. Cannot determine
D. We have a question about this…?
40
POLL: Protocol Question #4Results part 2: Are the results statistically
significant? Practically significant?
A. Results were statistically significant
B. Results were statistically AND practically significant (in our view, the amount of difference between scores of M.A. students and comparison would warrant M.A.’s cost, for us)
C. Cannot determine
D. We have a question about this…?
41
POLL: Protocol Question #5
If you were to implement this intervention what outcomes could you reasonably expect to achieve?
A. We would expect to achieve similar results
B. We would not expect to achieve similar results
C. Cannot determine
D. We have a question about this…?
42
OBJECTIVE #4
Debrief the Critical Reading Protocol process, and
Discuss the relevance of this type of process as we consider purchasing/creating interventions
43
Debrief & Discuss
1. What do we think about the Protocol? 2. How does this Protocol process compare to how we
normally think about the research behind interventions we are considering?
3. What is the value (or not) of this type of process as we consider purchasing/creating interventions?
3 min. ChatFacilitators lead their groups
Individuals utilize chat box
44
Objective #5Preview session #4
October 28, 2009 3-4:30pm
State of Research on RTI Math: What do we know? What do we not know?
Dr. Sybilla Beckmann, UGA
45
Thank you for being here
today!
Dr. Kim Anderson – [email protected] – 404-657-6174
The SERVE Center/REL-SE– 800-755-3277– www.serve.org
John Wight, GaDOE [email protected] 404-656-0478
49