And the Survey Says…. Putting YOUR Healthy Youth Survey Results to Work

transcript

And the Survey Says….Putting YOUR Healthy Youth Survey

Results to Work

Spring 2003

Purpose: Help people understand & use

their HYS reports

Welcome!

• Introduction of presenters

• How many agencies are here today?

• Review Agenda

• What we will have accomplished by the end of the day?

Welcoming picture drawn by a Healthy Youth

Ice breaker

Ice Breaker

• In small groups, each person shares – Name

– Where you work

– Share one interesting or obscure fact about yourself

Background of HYS 2002

History of WA survey efforts

Content of HYS 2002

Sources of HYS 2002 items

Data processing and quality control steps

Past Washington Surveys: Primary Content

‘02‘01‘99‘97‘96‘95‘94‘93‘89‘88 ‘00‘92‘90 ‘91 ‘98

Health-risk Focused

Risk & Protective Factor Focused

Combination

Joint Survey Planning Committee

• Office of the Superintendent of Public Instruction

• Division of Alcohol & Substance Abuse, DSHS

• Department of Health

• Office of Community Development

• Governor’s Family Policy Council

• RMC Research (survey contractor)

HYS 2002: Collaborative Effort of OSPI, DOH, DSHS, and CTED

• Simple random sample of schools recruited at state level– County samples drawn as appropriate

• Survey consistently administered in the Fall of even years

• Survey given to 6th, 8th, 10th, and 12th graders

• Survey booklets will have one-page tear-off answer sheet

• Survey for 8th, 10th, and 12th graders will use 2 form “interleaved” administration

• The Healthy Youth Survey 2002 was completed…

HYS 2002: Participation

By 137,335 students…

In 752 schools…

In 203 school districts…

In 39 counties.

Risk & Protective Factor DataConsistent with CTC/MTF SurveyDevelopment led by DASA/DSHS & OSPI

WSSAHB Questionnaire• Demographics• Alcohol, Tobacco, other drugs• School risk/protective factors• Community risk/protective factors• Peer-Individual-Family risk/ protective factors

Youth Health Behavior DataConsistent with National YRBS & YTSDevelopment led by DOH

YRBS Questionnaire• Demographics• Alcohol, Tobacco, Other drugs• Nutrition• Physical Activity• Safety Behaviors (Helmets, Seatbelts)• Mental Health/Depression/Suicide• Additional Tobacco Indicators

Getting the Best of Both

Form AWSSAHB-

Form BYRBS-likeSurvey

6th Grade Survey

One Form

SurveyCore

+ Mutually Agreed Items

Core Survey Items

• Demographics• Alcohol, Tobacco and Other Drug Use Items• Key Violence-Related Items• School-Specific Asset Items• Depression

Sources of HYS 2002 Items

• Monitoring the Future (MTF)

• Youth Risk Behavior Survey (YRBS)

• Communities that Care (CTC)

• Youth Tobacco Survey (YTS)

• PRIDE Survey

Data Processing and Quality Control

• Administration procedures• Data cleaning procedures

Statistical Issues

Validity and reliability

Confidence intervals

Comparing state and local results

Generalizability

Validity and Reliability

• Definitions• How we assure validity: Items from established

instruments, validity checks• How we assure reliability: Standardized

administration procedures

Confidence Intervals

• What are they?• How to interpret the numbers • How to interpret the graphs• Practice activity (Ex. 1–2)

What is a Confidence Interval?

• The reported value is probably a little different than the “true” value for all your students

• We are “95% confident” that the “true” value is within the +/- range, called the “confidence interval”

Interpreting CIs Numerically

Smoked cigarettes: 9.2% (± 1.1%)

9.2 – 1.1 = 8.1

9.2 + 1.1 = 10.3

Between 8.1% and 10.3% smoked cigarettes

Interpreting CIs Graphically

0 5 10 15 20

Smokedcigarettes

Percent of students

Practice Activity

• Exercises 1 and 2

Why Are Confidence Intervals Different Sizes?

• Number of students• Inherent variability• Level of confidence (All 95% for HYS)• Sampling design

If the data are valid, why do we need confidence intervals?

• Confidence intervals account for variability among students, NOT validity of the data

• Variability is inherent in any population worth studying

• Variability causes uncertainty in the results• “Doubt is not a pleasant condition, but certainty is

absurd” – Voltaire

Comparing State and Local Results

• Looking at differences numerically• Looking at differences graphically• Practice activity (Ex. 3–4)

Significant Differences: Numerically (Ex. 1)

• Smoked cigarettes in the state: 9.2% (± 1.1%)

Between 8.1% and 10.3%

• Smoked cigarettes at my school: 14.1% (± 2.22.2%)

Between 11.911.9% to 16.316.3%

• Conclusion: Difference IS statistically significant

Significant Differences: Numerically (Ex. 2)

• Smoked cigarettes in the state: 9.2% (± 1.1%)

Between 8.1% and 10.3%

• Smoked cigarettes at my school: 14.1% (± 9.69.6%)

Between 4.54.5% to 23.723.7%

• Conclusion: Difference is NOT statistically significant

Significant Differences: Graphically (Ex. 1)

0 5 10 15 20 25

Smokedcigarettes

Percent of students

Local State

Significant Differences: Graphically (Ex. 2)

0 5 10 15 20 25

Smokedcigarettes

Percent of students

Local State

Practice Activity

• Exercises 3 and 4

Generalizability

• What is generalizability?• To whom can we generalize results?

• State & County sample

• Non-sampled counties, districts, and ESDs

• Schools

• Why 70% participation is required

Quotations From Teachers Who Administered the Survey

• About 25 students missed part of the session because of club photos

• Absent students are taking a PSAT test• Many students had too much homework. Chose to

do that rather than take the survey.• Seven students came too late to take the survey• The perfectionists, poor direction followers and

slow readers could not finish

Challenges to Generalizability

• Brainstorm challenges to generalizability – School level

– ESD, County, or District level

• How do these challenges affect data interpretation?

What if I don’t want to generalize?

• Confidence intervals matter if you want to generalize to a larger population

• You can describe the students surveyed, in that moment, without confidence intervals:– “Elmo Elementary School students who took the survey

said…”

Final Note About Significance

• Even if a difference is statistically significant, it might not be practically significant

• Example– State alcohol use: 12.8 (±0.2) %

– Local alcohol use: 14.4 (±1.2) %

Statistically significant difference, but should it influence program planning?

Comparing data over time

Things to think about

• Did the questions change?• Were the challenges to generalizability similar?• Is there a reason to think that things would have

changed? (a prevention program)

Surveys over time

• WSSAHB 2000 reports did not report confidence intervals – but the variability was still there!

• Future HYS reports will have confidence intervals, so 2002 vs. 2004 comparisons will be easier

What should I know about comparing 2000 to 2002 data?

• If you think you have comparable data, then:– If the 2000 percentage is INSIDE the 2002 confidence

interval, then there is not a “statistically significant” difference

– If the 2000 percentage is OUTSIDE the 2002 confidence interval, then you can’t be sure if the difference is “statistically significant” but the further away it is the more likely that a difference is significant

– If you really, really need to know whether it’s significant, you can contact DOH for a specialized data run

What if I am in a small school, and we have huge confidence intervals?• Having a confidence interval protects you (and

your program) from appearing to be ineffective when just a few students can make big changes

• Consider the input of teachers & staff from small-school environments when interpreting data – the data should be used to complement what they already know about their kids

Local Report Overview

Guide to the Local Report

How to use your local report

Local Report Contents

• Introduction and Overview– Key to the Notes

– Highlights of the Local Results

• Graphical summary of selected results• Selected results disaggregated by gender• Responses to all items

Guide to the Local Report, cont.

Healthy Youth Survey 2002Survey Results

Sample Middle School, Grade 8

Number of students surveyed:

Number of valid responses:

The impact of adolescent health risk behaviors remains a primary concern of citizens throughout the country. Many health problems experienced by adolescents are caused by a very few preventable behaviors….

The number of students who submitted survey forms

Healthy Youth Survey 2002Survey Results

Sample Middle School, Grade 8

Number of students surveyed:

Number of valid responses:

The impact of adolescent health risk behaviors remains a primary concern of citizens throughout the country. Many health problems experienced by adolescents are caused by a very few preventable behaviors….

The number of forms that were considered usable.

Selected Results Disaggregated by Gender

25. During the past 30 days, on how manydays did you smoke cigarettes?

Local(n = 59)

State(n = 4,960)

Female Male Female MaleNone 70.4% 93.8% 89.9% 84.9%1 or more 29.6 6.2 10.1 15.1

Note. Results are suppressed if any cell represents fewer than 15 students;p = .021 from a chi-square test.

To protect anonymity when data are broken down into identifiable groups.

Guide to the Local Report, cont.Item Results 42. How old were you the first time you smoked a whole

cigarette? Local

(n = 20) State

(n =4,687)

a. Never have 60.0% (± 21.5%) 74.7% (± 1.2%) b. 10 or younger 10.0 (± 13.1) 6.4 (± 0.7) c. 11 15.0 (±15.6) 6.5 (± 0.7) d. 12 5.0 (± 9.6) 6.8 (± 0.7) e. 13 10.0 (± 13.1) 4.8 (± 0.6) f. 14 0.0 (± 0.0) 0.8 (± 0.3) g. 15 0.0 (± 0.0) 0.0 (± 0.0) h. 16 0.0 (± 0.0) 0.0 (± 0.0) i. 17 or older 0.0 (± 0.0) 0.1 (± 0.1)

[Item 43 appears only on the elementary version of the survey.] 45. Do you think young people risk harming themselves if they

smoke from 1–5 cigarettes per day?B (one to five cigarettes a day?) C

Local (n = 8)

State (n =4,877)

a. Definitely no 12.5% (± 22.9%) 4.9% (± 0.6%) b. Probably no 12.5 (± 22.9) 11.5 (± 0.9) c. Probably yes 0.0 (± 0.0) 38.7 (± 1.4) d. Definitely yes 75.0 (± 30.0) 44.9 (± 1.4)

Number of students who responded to this item

Where Do Find Information About . . . ?

• General Information (e.g. demographics)• Alcohol, Tobacco, and Other Drug Use• Other Health Concerns (e.g. health/safety)• School climate (e.g., bullying)• Quality of Life (e.g., student outlook)• Risk and Protective Factors:

– Scale Results

– Item Results

Frequently Asked Questions

• Review handout• Note about optional items

How to use your local report

• Example• Practice Activity

Local Report Example

168. If you wanted to get some marijuana, how easy would it be for you to get some?

Local (n = 84)

State (n = 3,738)

a. Very hard 52.4% (± 10.7%) 56.6% (± 2.7%) b. Sort of hard 19.1 (± 8.4) 16.9 (± 1.5) c. Sort of easy 15.5 (± 7.7) 13.9 (± 1.3) d. Very easy 13.1 (± 7.2) 12.6 (± 1.4)

State Agencies Using the HYS Data

Federal Initiatives

• Drug-Free Schools and Communities Act • Department of Education Safe and Drug-Free

Schools Program • Principles of Effectiveness • Healthy People 2010: National Health Promotion

and Disease Prevention Objectives • National Drug Control Strategy • No Child Left Behind

Washington State Initiatives

• Washington Education Reform Act • Omnibus Alcohol and Controlled Substances Act• Violence Reduction Programs Act• Department of Health Priority Health Goals• Governor's Council on Substance Abuse

Why Do You Keep Changing Your Terminology?

Or…When is an Impact an Outcome?

Putting Your Data to Work

Program Planning and Evaluation

What’s the Problem?

• What is “a problem”?– Getting worse

– Worse than average

– Just plain bad

• Combine HYS data with other information, including knowledge of community stakeholders– For example, HYS data may show that the highest rates

of tobacco use are in a particular area of the county, where police know there is a problem with retailers who provide tobacco to minors

What’s the Problem?

Exercise 1: What’s the Problem?

• HYS data helped to identify a set of problems• Need to prioritize among them• Identify available resources

– Who is already working on the issue, or who has a stake in the issue

– Gaps in services or programs

– Modify existing programs, instead of creating all new ones

– Figure out what the community is “ready” for

What’s the Priority?

• Target Efforts– Describe a target audience (grade level, gender,

universal or selective approaches)– Write “SMART” goals and objectives to get

FOCUSED and CLEAR with partners, and evaluation• Specific• Measurable• Achievable• Relevant• Time-bound

Setting a Good Objective

• You have 1,000 students• Current smoking prevalence is 25% (250 kids)• SMART Objective: By 2004, we will implement a

comprehensive tobacco prevention program and reduce current smoking among our school’s students by 10%

• This translates into about 25 fewer smokers• This is a change from 25% to 22.5%

Keeping it Real

• Will your program reach enough kids to affect the overall prevalence?

• Remember the difference between a “percentage point” goal and a “percent change” goal!

Population Group Changes - Gender

Current Smoking Trends, by Gender

girls boys girls boys girls boys girls boys

6th 8th 10th 12th

Source: WSSAHB 2000, HYS 2002, state sample data

Exercise 2: What’s the Priority?

What Should I Do About It?

• Find “Best Practice” programs– Search for proven activities– The “Community Guide to Preventive Services”– CSAP “Science-based Prevention Programs”– Agency recommendations

• for example, DOH recommends conducting a retailer compliance program + public awareness about sources of tobacco

How Do I Know If It Worked?

Evaluate • Identify and collect process indicators, outcomes,

and impacts relative to your program– Process measures may include compliance checks &

compliance rates

– Outcomes may include perceived availability of tobacco & ‘usual sources’ of tobacco

– Impacts may include lifetime and current use of tobacco

Evaluation made easy

• What do I do when I go to work every day?• ….and then what happens?• ….and then what happens?• ….so that the WHY of going to work every day

happens.

Exercise 3: How Do I Know If It Worked?

Just in case you need more money…

• Healthy Youth Survey data can help– Provide justification of need for funding– Provide means for evaluating funded activities

• Find out about grant scoring• You will need to justify that HYS data are

– Valid– Reliable

Exercise 4: Grant Writing

Next Steps

Communicating With Your Data

Communications

• Nuts and bolts• Communicating about your program• Communicating to the media• Using data to make change

The Nuts and Bolts

• Round percents to whole numbers– 17.0% to 17.4% becomes 17%

– 17.5% to 17.9% becomes 18%

• Use language to acknowledge that these percents are estimates

• “About 17%”

• “14 to 20 percent”

• “plus or minus three percent”

Different Ways to Say ItInstead of saying: • “17.2% +/- 3.2% youth said that they ‘mostly’ or

‘definitely’ felt unsafe at school”

You could say:• “About 17% of our youth feel unsafe in school”• “About one in five youth feel unsafe in school”

Remember you can present it positively:• “About 83% of our youth feel SAFE in school”• “About four out of five youth feel SAFE in school”

Additional Tips

• If you had a low response rate for your survey, it would be best to say:

– “Elmo Elementary School students who took the survey said…”

Instead of

– “Elmo Elementary School students said…”

Communicating About Your Program

What data are appropriate for each type of communication?

• Your co-workers• Your neighbor at a cocktail party• Your grandmother

Communicating to the Media• What’s your SOCO? (Single Overriding

Communication Objective)– For broadcast media: 10-12 word “sound bite”

– For print media: 1-3 line quotation

• Message mapSingle Overriding Communication Objective

Detail 1 Detail 3 Detail 3

Fact Fact Fact

For Example: Tobacco Program Results

• SOCO: Our program has been successful in reducing youth smoking– Detail 1: Fewer youth are currently smoking than prior

to the program• Fact 1: 53% fewer 6th graders

– Detail 2: Fewer youth have ever tried smoking• Fact: 29% fewer 10th graders have ever smoked a

whole cigarette– Detail 3: Washington is doing better than the nation

• Fact: there was a 22% decrease in the nation for current smoking, so Washington’s decline is greater

Potential Pitfalls• When communicating with data, it’s easy to make

these mistakes:– Include information that’s not totally relevant to your

– Include information that is relevant, but goes beyond what you NEED (too much detail)

– Speculate beyond what you have in the data

– Treat non-significant or unimportant differences as significant

– Forget to double-check your results and/or the math

Keep “on message”

• The answer to every question is your SOCO.

“That’s an interesting point, but what I’d really like people to know is,

[ my SOCO]”

Message Mapping Exercise

In small groups• Choose 1 topic of interest from HYS• Develop SOCO and 3 supporting details• Use HYS local report to find 3 supporting facts for

each detail • Message map template in handouts

Using Data to Make Change

• The message should tell your audience what you want them to do:– Do you want them to be outraged? And then what…

– Do you want them to call a policymaker?

– Do you want them to change their beliefs or behaviors?

– Do you want them to support your program?

Choose and Describe Your Audience

• Potential Audiences– Co-workers– Area residents– Elected officials– Civic organizations– Health care providers– Media– Regulatory agencies– Activists

• Demographics• Characteristics

– Concerns– Attitudes– Levels of interests– Levels of involvement– Histories– Levels of knowledge– Opinions– Reasons for interest– Types of involvement

How Will You Reach Your Audience?

Area residents

• Community meetings

• Newspaper articles and ads

• Radio and TV talk shows

• Fliers

• Direct Mailings

Elected officials

• Frequent phone calls• Fact sheets• Personal visits• Invitations to

community meetings• News releases

Cardinal Rules of Communication with Data1. Accept and involve the public as a partner

– They’re going to comment on your stuff anyway– Know ahead of time what public opinion is around your topic– Anticipate (and prepare for) the “tough questions & smart answers” –

including criticism of the survey itself

2. Plan carefully and evaluate your efforts• Was your SOCO ‘heard’?• Did you reach your target audience?• Did they know what you wanted them to do?

3. Be honest, frank, and open• Be prepared to talk about the limitations of your data

4. Work with other credible sources• Have someone critique your findings or interpretation• …and double-check your math

Cardinal Rules of Communicating with Data (cont.)

5. Meet the needs of the media • Meet their deadlines• Help them understand what the data mean• Help them understand how the survey works, if needed

6. Speak clearly and with compassion• “data are just people with the tears wiped away”• Practice saying it out loud (numbers can trip your tongue)

7. Use AV or graphics with caution• Sometimes a graph hurts more than helps you• Visual aids might distract from your message• Don’t try to exaggerate with graphics

Thank You!! Sun setting on training session

And the Survey Says…. Putting YOUR Healthy Youth Survey Results to Work

Documents