Post on 15-Aug-2015
transcript
Curtin University is a trademark of Curtin University of TechnologyCRICOS Provider Code 00301J
Jonathan HallettEvidence + Effectiveness in Health PromotionVALIDITY + RELIABILITY
Curtin University is a trademark of Curtin University of TechnologyCRICOS Provider Code 00301J
Lecture Overview
Revisit questionnaires
Reliability + validity
Methods for ensuring reliability + validity
Assessment check-in
Evidence and Effectiveness in Health Promotion
Curtin University is a trademark of Curtin University of TechnologyCRICOS Provider Code 00301J
Friendly Schools, Friendly Families
The primary outcome of FSFF was the prevalence of being bullied at the end of the three year intervention trial.
The purpose of FSFF was to assess the effectiveness of a parent and whole school bullying intervention, that includes policy review and enhancement, staff training, learning activities and parent and community involvement, aimed at preventing, reducing and managing bullying in primary schools.
Based on the Health Promoting Schools concept
Evidence and Effectiveness in Health Promotion
Curtin University is a trademark of Curtin University of TechnologyCRICOS Provider Code 00301J
Friendly Schools, Friendly Families
A three year group randomised control intervention trial
20 Perth metropolitan primary schools
Schools randomly allocated to high, moderate and low intervention groups
Followed cohorts of Year 2, 4 and 6 students and their teachers and parents
Evidence and Effectiveness in Health Promotion
Curtin University is a trademark of Curtin University of TechnologyCRICOS Provider Code 00301J
FSFF Research Objectives
Some of the objectives included:
Reduce reported frequency of children’s unresolved conflict/bullying
Improve children’s perceptions of positive outcome expectancies should they become involved in a bullying situation
Evidence and Effectiveness in Health Promotion
Curtin University is a trademark of Curtin University of TechnologyCRICOS Provider Code 00301J
Activity – Find questions that measure:
Prevalence of being bullied
Prevalence of bullying others
Telling someone about bullying (not parents)
Perception of prevalence of bullying at school
Evidence and Effectiveness in Health Promotion
Curtin University is a trademark of Curtin University of TechnologyCRICOS Provider Code 00301J
Answers
Prevalence of being bullied 7, 8, 9, 10, 11
Prevalence of bullying others 23, 24
Telling someone about bullying (not parents)
12, 13, 14, 15, 34g
Perception of prevalence of bullying at school
21, 22
Evidence and Effectiveness in Health Promotion
Curtin University is a trademark of Curtin University of TechnologyCRICOS Provider Code 00301J
Class Activity
Developing a Participant Satisfaction Sheet
The following page provides an example of a participation satisfaction sheet for a session conducted on responsible drinking.
- What are some noticeable features about this questionnaire? (e.g. size, layout, types of questions).
- What changes do you think you would make to improve on this feedback sheet if you were conducting a workshop on drinking behaviour?
Evidence and Effectiveness in Health Promotion
Curtin University is a trademark of Curtin University of TechnologyCRICOS Provider Code 00301J
Evidence and Effectiveness in Health Promotion
Curtin University is a trademark of Curtin University of TechnologyCRICOS Provider Code 00301J
Group activity
Review your case study
Write three SMART objectives
Develop a questionnaire with at least three different formats of questions (Likert, dichotomous, multi-choice etc.)
Identify the most appropriate mode of survey
Consider ethical issues related to your data collection
Evidence and Effectiveness in Health Promotion
Curtin University is a trademark of Curtin University of TechnologyCRICOS Provider Code 00301J
Reliability and Validity
What do we mean when we say something is reliable?
What about if something is valid?
Evidence and Effectiveness in Health Promotion
Curtin University is a trademark of Curtin University of TechnologyCRICOS Provider Code 00301J
Reliability + Validity
Reliability - does the questionnaire measure yield the same results on repeated trials? (stability - need this for validity)
Validity - does the questionnaire measure what you are actually trying to measure? (relationship between measure and indicator)
Evidence and Effectiveness in Health Promotion
Curtin University is a trademark of Curtin University of TechnologyCRICOS Provider Code 00301J
Reliability
The extent to which a measurement instrument is dependable, stable and consistent when repeated under identical conditions.
A reliable instrument will ascertain the same result on two or more occasions.
Evidence and Effectiveness in Health Promotion
Curtin University is a trademark of Curtin University of TechnologyCRICOS Provider Code 00301J
Types of reliability
Test-retest reliability describes the extent to which a stable evaluation of the attribute or behaviour can be obtained on two different occasions when no change is expected
Intra-rater reliability describes the extent to which the same person can rate the same performance consistently
Inter-rater reliability is the extent to which different people rate the same performance consistently
Evidence and Effectiveness in Health Promotion
Curtin University is a trademark of Curtin University of TechnologyCRICOS Provider Code 00301J
Sources of reliability error
Intra-rater reliability is higher than inter-rater reliability
For self-rated measures (self-report), intra- and inter-rater reliability measures are not relevant, so test-retest reliability is important
Evidence and Effectiveness in Health Promotion
Curtin University is a trademark of Curtin University of TechnologyCRICOS Provider Code 00301J
Testing Reliability
Observer dependent instrument- Inter-rater reliability (when measure applied only once each by different
observers)
- Intra-rater reliability (when measure taken more than once by same observer)
Self-report instrument- Test-retest reliability (most efficient method – give same measure to same
group at two points in time [about 1 to 2 weeks apart]; correlate results
- Split-half reliability (divide instrument questions in half; sum the scores and correlate the two results
Evidence and Effectiveness in Health Promotion
Curtin University is a trademark of Curtin University of TechnologyCRICOS Provider Code 00301J
Test-retest Method
Instrument administered to same participants twice (two weeks apart) under similar conditions.
Problems:- Non-independent respondents- Remember the answers- Testing effect- Take less effort the second time around- Change in characteristic itself, e.g. ever smoked vs number of
cigarettes in last seven days
Evidence and Effectiveness in Health Promotion
Curtin University is a trademark of Curtin University of TechnologyCRICOS Provider Code 00301J
Split-half Method
The items on the scale are divided into two matched halves and the scores for the two halves are correlated
Items need to be compatible / comparable
Evidence and Effectiveness in Health Promotion
Curtin University is a trademark of Curtin University of TechnologyCRICOS Provider Code 00301J
Split-half Method
1. On how many days in the last week did you drink alcohol?
2. On average, how many drinks did you consume when you drunk alcohol?
Evidence and Effectiveness in Health Promotion
Curtin University is a trademark of Curtin University of TechnologyCRICOS Provider Code 00301J
Split-half Method
3. Starting yesterday, going backwards through the last week, how many alcoholic drinks (minutes of exercise) did you consume on each day?
Monday __
Saturday __
Sunday __
Friday __ Thursday __
Wednesday __
Tuesday __
Evidence and Effectiveness in Health Promotion
Curtin University is a trademark of Curtin University of TechnologyCRICOS Provider Code 00301J
Validity
The degree to which a scale measures what it is intended to measure
Evidence and Effectiveness in Health Promotion
Curtin University is a trademark of Curtin University of TechnologyCRICOS Provider Code 00301J
Measurement Validity
Face validity
Content validity
Construct validity
Criterion validity
Predictive validity
Evidence and Effectiveness in Health Promotion
Curtin University is a trademark of Curtin University of TechnologyCRICOS Provider Code 00301J
Types of validity
Content validity refers to the extent to which an instrument covers all the important aspects or domains of interest that it intends to measure, and does not include items that are irrelevant.
Face validity indicates that an instrument appears to evaluate what it is supposed to and seems plausible to the users of the tool (evaluators and respondents)
Evidence and Effectiveness in Health Promotion
Curtin University is a trademark of Curtin University of TechnologyCRICOS Provider Code 00301J
Types of validity
Construct validity refers to the ability of an instrument to measure an abstract concept. Establishing this involves:
1.Explicitly describing the theoretical concepts and how they relate to each other.
2.Developing an assessment too that can measure this.
3.Testing the relationship between the constructs and the observed behaviours that it represents.
Evidence and Effectiveness in Health Promotion
Curtin University is a trademark of Curtin University of TechnologyCRICOS Provider Code 00301J
Types of validity
Criterion validity concerns the correlation of one tool with another, ideally a ‘gold standard’ that has been widely used and accepted in the field of interest.
But is there a ‘gold standard’?
Evidence and Effectiveness in Health Promotion
Curtin University is a trademark of Curtin University of TechnologyCRICOS Provider Code 00301J
Types of validity
Predictive validity is established by evaluating performance using the instrument at a specified time-point (e.g. baseline) and following the participants over time until the outcome to be predicted occurs. These studies determine the accuracy of the measurement tool to predict future outcomes.
Evidence and Effectiveness in Health Promotion
Curtin University is a trademark of Curtin University of TechnologyCRICOS Provider Code 00301J
Instrumentation processes which reduce validity1. Response sets. For example: tendency of respondents to
agree (pleasers) or tendency towards socially acceptable answers.
2. Interviews. For example: interviewer contamination – differential emphasis on questions or respondent may know objectives of study and answer accordingly.
3. Content validity – always a factor because no instrument can cover all aspects of a problem.
Evidence and Effectiveness in Health Promotion
Curtin University is a trademark of Curtin University of TechnologyCRICOS Provider Code 00301J
Improving Reliability + Validity
Having colleagues and experts review and make suggestions to the survey.
Interview a few participants as they are completing a survey which involves asking them to verbalise how they are answering the question. This will provide some indication of whether the questions are being interpreted as designed.
Conduct a pilot test of the questionnaire on a smaller sample
Have the survey proof read by an individual(s) unrelated to the project.
Evidence and Effectiveness in Health Promotion
Curtin University is a trademark of Curtin University of TechnologyCRICOS Provider Code 00301J
29
Valid / Reliable Questions
Valid and reliable results need valid and reliable approaches.
Minimum requirements:- Target group / community participation in face & content validity
- Test/retest approaches for reliability
Evidence and Effectiveness in Health Promotion
Curtin University is a trademark of Curtin University of TechnologyCRICOS Provider Code 00301J
Triangulation
Method of cross-checking data from multiple sources:
Methodological triangulation: use of more than one method of data collection (e.g. self-report survey and observation)
Investigator triangulation: use of more than one evaluator/researcher
Data triangulation: multiple sources of data (e.g. collecting data over different times and from different people)
Theory triangulation: multiple theories in interpreting phenomenon
Evidence and Effectiveness in Health Promotion
Curtin University is a trademark of Curtin University of TechnologyCRICOS Provider Code 00301J
Next Week
Focus Groups
Evidence and Effectiveness in Health Promotion