Date post: | 28-Dec-2015 |
Category: |
Documents |
Upload: | octavia-burke |
View: | 215 times |
Download: | 0 times |
Developing Evidence-Based Products Using the Systematic
Review Process Session 3/Unit 9:
Inclusion and Exclusion Criteria
Screening and Coding Studies
Julia LavenbergOctober 30, 2007
NCDDR training course for NIDRR grantees
2
Outline
Study selection vs. data extraction
Requirements for each Exploration of screening and coding via a worked example
Application of inclusion and exclusion criteria and codes to planned reviews
3
Study Selection vs. Data Extraction
The process by which one chooses studies for inclusion in a systematic review.
Accomplished by instituting specific and detailed inclusion and exclusion criteria.
The process by which one locates and transcribes information from a primary study.
Accomplished by applying specific and detailed criteria to the information in a primary study.
4
Eligibility criteria
Provide readers with an idea of the research domain of interest
Aid the systematic reviewer in applying consistent and objective standards throughout the selection process
Clearly circumscribe the review
5
Inclusion and Exclusion Criteria: Categories
Publication types Study design Population Intervention Outcomes Measures
6
Publications Identify the types of reports to be included
Determine any geographic or linguistic limitations
“Studies eligible for this review may be published or unpublished reports (e.g.,
dissertations theses, government reports, school district reports, etc.) of school-based interventions conducted
in any country and reported in any language.”
(Lavenberg, 2007)
7
Study design Specify the research designs to be included
Also identify the research designs to be excluded
Address the rationale for inclusion and exclusion Consider theoretical framework Consider available evidence
8
Population Indicate the desired target population Stipulate any required characteristics Additionally, distinguish characteristics that would make the target population ineligible
“Studies of interventions that target children and youth who are enrolled in kindergarten through grade 12 (or the international equivalents) at public, private, parochial, or alternative schools and are between the ages of 4 and 20 years will be included in this review. Persons identified as attending ‘preschool’ or ‘college’ will not be included, even if they are within the acceptable age range.” (Lavenberg, in process)
9
Intervention
Detail the required characteristics of the intervention
Provide definitions List examples of what would be included and what would be excluded
10
Outcomes Indicate outcome variables of interest
“At least one quantitative measure of aggressive behavior must be reported in each study….[citations to literature here]… Therefore, both physical and verbal aggressive behaviors will be considered acceptable outcomes.”
(Lavenberg, in process)
11
Measures Address acceptable measures
“Standardized measures of aggressive behavior (e.g., Child Behavior Checklist – Teacher Report Form….) and unstandardized measures with adequate face validity (e.g., local administrative records…) will both be considered acceptable forms of reporting aggressive behavior. Measures reported in the Buros Institute of Mental Measures Yearbook…will be considered standardized measures; all others will be considered unstandardized measures.”
(Lavenberg, in process)
12
Example: Application to a review with > 900 records retrieved (Lavenberg, 2007)
Two phase screening process developed Phase I – title and abstract
Goal: differentiate potential studies from clearly inappropriate studies
Evaluate each study for topical relevance, setting, participants, and general study design
Phase II – method section Goal: create the pool of eligible studies Evaluate each study for intervention, design details, and outcome
13
Selection process for including reports
Potentially relevant reports identified
(n>900)
Reports retrieved for further review
(n=155)
Reports progressing to next level(n=109)
Reports used to code studies for inclusion
(n=50)
Reports excluded at Phase II screening(n=59)
Reports excluded at Phase I screening(n=46)
Reports excluded by title and
abstract review(n=750)
14
Data Extraction (i.e., “coding”)
Initiated after identifying appropriate pool of studies for the systematic review
Accomplished by means of a coding form and codebook
Goal: ensure reliable and orderly extraction of information from each study report
15
Forms… May be either paper-based or computer-based
Are best completed by trained coders Should have items organized such that the order reflects the manner in which information is presented in the study report
Contain the same categories of information as inclusion and exclusion criteria
16
Coding categories
That is, the data extracted will address: Publication type Design Participants Interventions Outcomes Measures
17
. . . in coding, much more detail
is required.
The data that is extracted from each study report is the basis of all subsequent analyses.
BUT:
18
Overarching principle:
Conserve as much of the original information as possible.
Fewer coder judgments = fewer errors
19
Publication characteristics
Document publication year Distinguish between types Create mutually exclusive categories
Must assign numerical values to these categorical variables
20
Study design and methodological characteristics
Consider identifying the recruitment pool to provide additional context
Describe the unit of assignment (e.g., individual, family, school, clinic, workplace)
Address the mechanism of allocation (e.g., random assignment, matching on pretest measures of outcome variables, etc.)
List unit of analysis Note whether unit of assignment matches unit of analysis
21
Participant characteristics Age
May need to capture data in multiple ways (e.g., school grade, chronological age, mental age, age range)
Gender Race/ethnicity Socioeconomic status Specific characteristics relevant to review topic (e.g., official diagnoses, developmental categories, baseline level of behavior)
Co-morbidity
22
Intervention characteristics Describe important aspects of the intervention Characteristics or features Total length of intervention, number of sessions, length of each session
Method of delivery Personnel implementing intervention Fidelity of implementation
Identify comparison group conditions
23
Outcome and measure characteristics
Identify outcome variables of interest Allow for and document variations
Identify outcome informant or source of information Determine if informant was blind to assignment
Identify personnel administering measure
Specify outcome measures used Indicate time of outcome assessment
24
Effect size characteristics Extract information required to calculate an effect size Sample size (intervention and control groups)
Pretest and posttest Attrition
Means, standard deviations d index F value Others: Chi-square, p value, t value, etc.
25
Worked example #1
Approaches to Parent Involvement for Improving the
Academic Performance of Elementary School Age Children
Nye, Turner, & Schwartz (2006)
http://www.campbellcollaboration.org/doc-pdf/Nye_PI_Review.pdf
26
Definition of parent involvement
“…the parent has a direct interaction with the child in either the delivery or monitoring of the program or intervention.” (page 11)
27
Inclusion criteria
Design: limited to randomized controlled trials (RCTs)
Rationale: “…in order to provide the least biased estimate of the effect of parent involvement on student achievement and to control for variation in threats to internal validity.” (pg. 12)
28
Inclusion criteria, continued
Participants: described in detail in the coding categories section of the review (pg. 13)
Age Grade Gender SES Ethnicity
29
Inclusion criteria, continued
Interventions: “…studies reported the following characteristics of the intervention program: 1) Parent involvement with their child in academic support activities outside of school (e.g., reading or completing supplemental math problems with the child), and 2) Parent involvement [as defined earlier] for a minimum of 20 days” (pg. 11)
30
Inclusion criteria, continued
Outcomes: “Included studies reported the following outcomes on children’s academic performance in: reading, mathematics, spelling, writing, language arts, or science. (pgs. 11-12)
31
Inclusion criteria, continued
Measures: not specified as an inclusion criteria in Nye, et al. Again, specification of this characteristic will depend on the topic and decisions made by the review team
e.g., depression as measured by Beck inventory
32
Nye, Turner, & Schwartz Coding categories and coding sheet (unique elements)
Report characteristics Subject characteristics
Treatment and comparison groups separately addressed
Sample source – types of schools Classroom assignments – high achieving, underachieving, average, academically at-risk
Developmental categories – emotionally disabled, physically disabled, hearing impaired, speech/language impaired, etc.
http://www.ncddr.org/training/1NyeTurnerSchwartz_PI_CodingForm_2006.doc
33
Coding categories, continued Intervention characteristics
Coder description required, in addition to numerical data (e.g., total length of intervention program, length of time per day of PI activity, etc.)
Fidelity of implementation Outcome measures – norm referenced vs criterion referenced vs rating scale vs survey
Design characteristics Recruitment pool Specific method of randomization and identification of who implemented the process
Effect size characteristics
34
Review topics Weight loss intervention to reduce physical disability in older adults
Improving the ambulatory transition of pre- and post-adolescents with cerebral palsy
Improving cultural competency to increase the employment of persons with disabilities
35
Review topics, continued
Use of assistive technology and employment supports in the employment of working age people with disabilities
Health promotion programs that improve primary access for adults with disability
38
Worked example #2
Effects of School-Based Cognitive-Behavioral Anger
Interventions: A Meta-Analysis
(Lavenberg, 2007)
39
Primary Research Question:
Are school-based cognitive-behavioral anger
interventions effective in reducing child and adolescent aggressive behavior in the
school setting?
40
Secondary Research Questions… Involved looking at whether the following were related to the effect size: Methodological issues (e.g., random assignment)
Setting Geographic location (urban, suburban, rural)
Delivery personnel Teachers vs. counselors
Stand-alone vs. embedded anger interventions
Participant characteristics (e.g., gender, age, baseline level of aggressive behavior)
41
Define and operationalize
critical elements:
Aggression Anger Cognitive-behavioral
Theory Intervention
Anger management intervention School-based intervention
42
Screening Forms and Manual
Copy sent to Joann Starks for posting
http://www.ncddr.org/training/3Lavenberg_2007_Screening_Coding_FormsManuals.doc