Developing an Expert-System Developing an Expert-System for Health Promotion: for Health Promotion: An Experimental E-Learning PlatformAn Experimental E-Learning Platform

Work, Stress, & Health 2008APA-NIOSHWashington, DC

Chairperson: Rebekah K. Hersch, PhDISA Associates, Alexandria, Virginia

Symposium ParticipantsSymposium Participants

• Presenters– Joel B. Bennett, PhD; Organizational Wellness &

Learning Systems (Fort Worth, TX)– Mark Attridge, PhD; Attridge Studios (Minneapolis,

MN)– Rudy M. Yandrick; IntraView Analysts (Mechanicsburg,

PA)– Ashleigh Schwab, MA; Organizational Wellness &

Learning Systems (Fort Worth, TX)• Discussant– Tom F. Hilton, PhD; National Institute on Drug Abuse

(Bethesda, MD)

Assessing Program Design Assessing Program Design and Impact: Experimental and Impact: Experimental ResultsResults

Joel B. Bennett. PhDPresident

Organizational Wellness & Learning Systems

OverviewOverview

• Background (Need for Program)• Program Demonstration• Program Evaluation• Method– Sample of Workplace Professionals– Human Resources, EAP, Wellness

• Results• Commercialization

5. No standards for quality in evidence-based health media that translate into professional market

Trends paving way for prevention?

Background (Program Need)Background (Program Need)

2. Health care costs ; reductionist• All about pharma/biomed• no mech. for bringing OHP to scale

3. Vendor fragmentation: HR praxis• EAP, Wellness, Work-Life, benefit packages• No empowerment to integrate

1. Gulf between science to practice• Many efficacy studies, few dissemination studies• Paradigmatic sloth, one-shot studies

4. E-Health E-Learning• From web-’sites’ to interactives• Synchronous e-coaching capacity

One approach to meet the needOne approach to meet the need

DemonstrationDemonstration

The art and science of preventionThe art and science of prevention

AssessAssess DesignDesign

DeliverDeliverEvaluateEvaluate

9

Westside Industrial

National Registry of Evidence-BasedNational Registry of Evidence-BasedPrograms (NREPP)Programs (NREPP)

Effectiveness StudyEffectiveness Study

Study MethodologyStudy Methodology

• Recruitment of workplace Recruitment of workplace professionalsprofessionals

• SHRM, EAPA, EASNA, NWI, Regional SHRM, EAPA, EASNA, NWI, Regional Health Promotion (N = 218)Health Promotion (N = 218)

• Random AssignmentRandom Assignment– User Group had 3 to 6 mo. (User Group had 3 to 6 mo. (SHORT PERIODSHORT PERIOD))

• Sample MeasuresSample Measures– KEY OUTCOMES: KEY OUTCOMES: [1] Interest/ Engagement in [1] Interest/ Engagement in

Workplace AOD Prevention; [2] Ability to Address AOD; Workplace AOD Prevention; [2] Ability to Address AOD; [3] Role Improvement[3] Role Improvement

– Attitude Measures based on TRA, TPBAttitude Measures based on TRA, TPB

Sample (218 Professionals)Sample (218 Professionals)

• 72% Female; 87% Caucasian• Human Resources, Health Promotion,

Employee Assistance Professionals• 42 states; 15 Industries (healthcare,

education highest representation)• Highly Educated (65% with greater than

college degree)• Many (84%) had professional license• 65% held manager, executive, CEO status

QuestionsQuestions

• Will end-users experience:– Greater ability to implement programs? – Enjoyment in technology?– Improved role as a wellness advocate (in

assess, design, deliver, evaluate)?

WELLNESS

SUBSTANCE

ABUSE

More importantly… Greater interest/engagement in the 4 AOD

evidence-based programs? Greater intention to use AOD prevention? Felt ability to address worker AOD issues? Enhanced orientation to evidence-based

programs in general?

Sample MeasuresSample MeasuresM

ediat

ors

Outcomes

Venkatesh, V., Morris, M.G., Davis, G.D. and Davis, F.D., (2000). User acceptance of information technology: toward a unified view. MIS Quarterly. v27 i3. 425-478.

Preliminary ResultsPreliminary Results

Means (SD)

One-Way FControl Experimental

Performance Expectancy 4.58 (1.28) 5.08 (1.03) 9.56; p = .002

Effort Expectancy 4.71 (1.26) 5.05 (1.05) 4.61; p = .03

Attitudes Toward Technology 4.65 (1.27) 5.18 (1.00) 11.01;p< .001

Role Improvement 2.35 (.92) 2.52 (1.06) 1.46

Orientation to Evidence-Based Programs

4.37 (1.18) 4.97 (1.16) 13.81;p< .001

Ability to Address Worker Substance Abuse

4.54 (1.18) 4.92 (1.12) 5.58; p< .02

Engagement/Interest in Specific AOD Programs

1.94 (.96) 2.57 (1.32) 15.75;p< .001

?

These were distracter programs, not included in IntelliPrev™ Library but linked through 3rd-level lesson search

Key Outcome: Engagement/Interest in AOD

ENGAGEMENT

Interest/EngagementInterest/Engagement

0

10

20

30

40

50

60

70

80

Team Team AwarenessAwareness

WellnessWellnessOutreachOutreach

CopingCopingWith WorkWith Work& Family& Family

StressStress

HealthyHealthyWorkplaceWorkplace

HealthtracHealthtrac LifeStepsLifeSteps

IntelliPrev Control

% Any Interest Studied, Considered,Actual Use

** ** ** **

ConclusionsConclusions

Compared to non-users, IntelliPrev participants:

1.Greater ability to assess, design, deliver, and evaluate

2.Greater ease in implementing3.Greater orientation to evidence based

programs4.Greater ability to address substance

abuse5.More engagement of specific evidence-

based programs in the program library

Mark Attridge, PhDAttridge Studiosmark@attridgestudios.commark@attridgestudios.com

Return on Investment Return on Investment Calculations for Calculations for Behavioral Health:Behavioral Health:Development and ApplicationDevelopment and Application

22

OverviewOverview

• This presentation reviews the development and research rationale for a Return-on-Investment (ROI) calculator tool for a web-based application.

• This tool is available to the public but is targeted to employers.

• The calculator is designed to estimate the population prevalence, the cost burden to employers, and intervention savings and ROI for three conditions: – cardiovascular problems– depression– alcohol abuse

23

Logic of ModelLogic of Model

24

1000

Simulation (Inputs 1,2,3)Simulation (Inputs 1,2,3)

25

Simulation (Input 4)Simulation (Input 4)

26

60 40

46.50 18.50

15 50

Simulation (Inputs 5,6,7)Simulation (Inputs 5,6,7)

27

Simulation (Inputs 8,9)Simulation (Inputs 8,9)

Sample OutputSample Output

29

Depression Depression 7575

Number of Employees Number of Employees With Risk for this ProblemWith Risk for this Problem

Cardiovascular Cardiovascular 9898

Alcohol Alcohol 7373

Health care: 543KProductivity: 1.74KAbsence: 891K

CostCostBurden to Burden to EmployerEmployer

Simulation (Output)Simulation (Output)

Sample OutputSample Output

31

Depression Depression 3030

Number of Employees Number of Employees Expected to Participate Expected to Participate in Health Prevention or in Health Prevention or Intervention ProgramsIntervention Programs

Cardiovascular Cardiovascular 4949

Alcohol Alcohol 2525

Health care 56KProductivity 172K

Absence 59K

Savings Savings In Health Care In Health Care

and Work and Work PerformancePerformance

(assume 25% reduction (assume 25% reduction in disease burden)in disease burden)

Simulation (Output)Simulation (Output)

Sample OutputSample Output

33

Depression Depression $11.38$11.38

ROI Per Condition*

Cardiovascular Cardiovascular $8.25$8.25

Alcohol Alcohol $8.37$8.37

Savings / Investment= 9.18 ROI

Total ROI

* Assume $300 investment per participant/per year

Simulation (Output)Simulation (Output)

34

Distinguishing Factors of This ROI ModelDistinguishing Factors of This ROI Model

We increase the accuracy of ROI estimation by:We increase the accuracy of ROI estimation by: 1) Using recent data to estimate the prevalence rates of 1) Using recent data to estimate the prevalence rates of three common diseases with behavioral health linksthree common diseases with behavioral health links 2) Using research studies on the impact of diseases2) Using research studies on the impact of diseases

on health care costs and work performanceon health care costs and work performance 3) Including employee work productivity, which has 3) Including employee work productivity, which has

been shown to be the largest area of cost losses been shown to be the largest area of cost losses 4) Including organizational health climate as a factor4) Including organizational health climate as a factor

ResourcesResources

• See website for the ROI Calculator toolwww.IntelliPrev.com

• See handout for case example of results

• See handout for key references in this area in the research literature

35

THE HEALTH & PRODUCTIVITY THE HEALTH & PRODUCTIVITY CLIMATE INDEX: CLIMATE INDEX: MANUFACTURING MANUFACTURING PLANT PLANT CASE STUDYCASE STUDY

Rudy M. Yandrick, B.A.IntraView Workforce Analystsintraviewanalysts@comcast.net

OverviewOverview

The Health & Productivity Climate Index (HPC)•Key Elements– Risks & Strengths; Dimensions;

Levels•Case Study•Sample Data•Norms

Seven Dimensions of the HPCI Number of Items

AlphaN = 674

K= 9

Health-Related DimensionsHealth-Related DimensionsHealth & Wellness (physical, mood, substance use) 17 0.87

Coping with Stress 6 0.73

Productivity-Related DimensionsProductivity-Related DimensionsPresence & Engagement 6 0.83

Team Communication 6 0.87

Policy & Accountability 6 0.80

Health & Productivity-Related DimensionsHealth & Productivity-Related Dimensions

Work-life Balance 6 0.81

Help & Support 6 0.78

Introduction to the Health & Productivity Introduction to the Health & Productivity Climate Index Climate Index (HPCI)(HPCI)

Strength and Risk Items-ExampleStrength and Risk Items-Example

DIMENSION: COPING WITH STRESSDIMENSION: COPING WITH STRESS

Strength• Coping skill in response to high work stress• Manager responsiveness to employee stress• Knowledge about coping with work stress

Risk• Continual heavy work pressure• Recent stressful events in personal lives• Stress-management knowledge

Scores by Dimension: Test site vs. IntelliPrev™Scores by Dimension: Test site vs. IntelliPrev™MEANS & STANDARD DEVIATION COMPARISONSMEANS & STANDARD DEVIATION COMPARISONS

HPCI Climate IndexHPCI Climate Index: : OWLS’ 5-LEVEL SCHEMAOWLS’ 5-LEVEL SCHEMA

Health & Productivity Climate Index: Health & Productivity Climate Index: Three Years in Review: Three Years in Review: MUNICIPAL EMPLOYERMUNICIPAL EMPLOYER

*Small snapshot of full graph

OWLS Consulting with IntelliPrevfollowing 2005

6

11

16

21

26

Health &Wellness

Work-LifeBalance

Presence & Engagement

TeamCommunication

Policy &Accountability

Coping withStress

Help & SocialSupport

ProblemProblem

RiskRisk

AdaptingAdapting

HealthyHealthy

ResilientResilient

Health & Productivity Climate Index:Health & Productivity Climate Index:Test Worksite ResultsTest Worksite Results

IntelliPrev™ Norms IntelliPrev™ Norms (N= 674, K=9, 37 workgroups)(N= 674, K=9, 37 workgroups)

6

11

16

21

26

Health &Wellness

Work-LifeBalance

Presence &Engagement

TeamCommunication

Policy & Accountability

Coping withStress

Help& Support

High Score Work Group

Low Score Work Group

Median

ProblemProblem

RiskRisk

AdaptingAdapting

HealthyHealthy

ResilientResilient

16.915.89

19.37

17.7818.75

16.46 16.9918.2

19.14

21.4420.13

21.12

18.52

20.44

6

11

16

21

26

Health &Wellness

Work-LifeBalance

Presence & Engagement

TeamCommunication

Policy &Accountability

Coping withStress

Help & SocialSupport

ProblemProblem

RiskRisk

AdaptingAdapting

HealthyHealthy

ResilientResilient

Health & Productivity Climate IndexHealth & Productivity Climate IndexTest Worksite Means vs. IntelliPrev™ NormsTest Worksite Means vs. IntelliPrev™ Norms

Test Site

IntelliPrev

Strength and Risk ItemsStrength and Risk Items

DIMENSION: COPING WITH STRESS

Strength Mean diff.•Coping skill in response to high work stress -0.18•Manager responsiveness to employee stress -0.62*•Knowledge about coping with work stress -0.20

Risk•Continual heavy work pressure -0.59*•Recent stressful events in personal lives -0.86*•Stress-management knowledge -0.80*

*Asterisked items indicate relatively greater potential risks

Manager Responsiveness to Employee StressManager Responsiveness to Employee Stress: : Frequency Distribution from Test SiteFrequency Distribution from Test Site

25%

33%

25%

14%

2%0%

5%

10%

15%

20%

25%

30%

35%

StronglyDisagree

Disagree In Between Agree Strongly Agree

“Managers understand the level of job stress and strainemployees face and make an effort to reduce rather thanadd to their stress”

Among our recommendations...Among our recommendations...

• Priority one: Improve supervisor competency development (e.g. emotional intelligence)

• Increased team awareness• Stronger presence of EAP at the worksite• Wellness programming to encourage healthy

lifestyles• Administer the survey again next year

Ashleigh Schwab, M.S.Project Manager

Organizational Wellness & Learning Systems, Inc.

Process Evaluation of a User Process Evaluation of a User System Interface:System Interface:Predicting E-Learning Program Predicting E-Learning Program EffectivenessEffectiveness

GOAL: ‘Expert-system’ to assist professionals Select or craft ‘right’ programs Enhance role as behavioral health advocates Skills to Assess, Design, Deliver, and Evaluate

“High-Tech” beyond ‘text-only’ website Rich multi-media flash & narration Actual tools for application Interactive tools facilitate skills development

E-Learning (“High-Tech”)E-Learning (“High-Tech”)

E-Coaching (“High-Touch”)E-Coaching (“High-Touch”)

• Utilize web and phone to coach professional in their use of tools and technology transfer

• Can meet needs just-in-time and on demand

• Locating utilities and tools on the web reduce dependency on trainer and facilitate autonomy and experimentation for the learner

• Utilities are ‘real’ tools to be used in ‘live’ scenarios, not virtual simulations

MethodMethod

Procedure• Informed Consent• Access to e-Learning

platform• Users had 3-6 months of use• Tracked participation and

Engagement (Process Data)• Process data was used to

create a single score• Score was broken down

into 3 categories:• Low• Medium• High

Measures• Post-test online survey• Over 200 items, 25

different variables.• Variables developed

based on Unified Theory of Acceptance and Use of Technology

• Participants received $50 for completion

• One individual from each condition won a $500 raffle.

Key Outcome VariablesKey Outcome Variables

¹Adapted from Venkatesh²Adapted from Bandura

Three Process VariablesThree Process Variables

Determined by a users overall engagement and participation in the program.

Identified through:

• Completion of learning sections

• Receive weekly affirmation emails

• Completed the Health & Productivity Climate Index (HPCI)

• Work group completed the HPCI

• Received telephonic coaching

• Attended a webinar

COACHING

WEBINAR

Overall Engagement Score

2

3

1

Low Users

Medium Users

High Users

Total

48 (46%) 36 (35%) 19 (18%) 103

Control Group108

Grouping Score for Overall Engagement

SIX Contributions to the grouping variable:

1. Amount of sections completed2. Signed up to receive weekly positive affirmation emails3. Completed the Health & Productivity Climate Index™4. Invited a work group to complete the HPCI5. Received telephonic coaching6. Attended webinar

Frequency of Engagement ActivitiesFrequency of Engagement Activities

Completion of the HPCI

Work Group completion of the

HPCI

Requested to receive weekly emails

Received telephonic coaching

Attended webinar

31% 33% 18% 11% 9%

11%

11%

27%

37%

55%

Number of sections completed

None 1 or 2 3 or 4 5 or 6 7 or 8

Frequency of Engagement Activities (cont)Frequency of Engagement Activities (cont)

Method: AnalysesMethod: Analyses

Correlations• 3 Key Process Variables:

Overall Engagement Score, Reception of Coaching, Attendance to Webinar

• Key Outcome Variables: Role Improvement, Performance Expectations, Effort Expectancy, Self-Efficacy, General Knowledge.

One-way ANOVAs• Tukey’s Post Hoc

ProcessProcess

OutcomOutcomee

HypothesisHypothesis

H1: Experimental users in the ‘High User’ category will indicate greater improvement on all 5 outcomes than those in the control group and lower category users.

H2: Experimental users that received at least one coaching session or one webinar during the research trial will indicate greater improvement in the 5 outcomes than those who did not utilize either offering, or those in the control group

Case ExamplesCase Examples

Dual Coaching SessionGoal: To facilitate collaboration

between a HR Manager and the EAP contact in order to come up with an action plan to address organizational health risks

Result: Based on the facilitated discussion of the HPCI, the EAP and the HR Manager were able to devise a preliminary strategy for starting a wellness program.

“IntelliPrev gave my customer and I a starting point to begin building a Wellness Program”

Solo Coaching SessionGoal: To facilitate understanding of

the individual’s own ‘prevention style’ using an interactive assessment tool found within IntelliPrev™

Result: The coaching session ended with the individual having a much better understanding of the personal strengths they bring as a prevention advocate, as well as potential challenges. “This program – along with the consultation I received - helped me make a strong financial argument for prevention. Reviewing the results of the HPC with my manager helped me to show how prevention works. Very effective.”

Correlation Table (n)

CRITERION VARIABLES

Engagement Score M=1.71

(SD=0.76)

Reception of Coaching

M=1.26 (SD=0.47)

Attended Webinar

M=1.55(SD=.50)

Role Improvement .26** .18* .16*Performance Expectations .28** .25** .25**

Effort Expectancy .26** .18** .19**Self-Efficacy .27* .05 .08

General Knowledge .25* .29** .28**

*P < .05, **P<.01

Process VariablesProcess Variables

Results (1)Results (1)

Results (2)Results (2)

ANOVA for Outcomes and Overall Engagement Score

Means (SD) One-Way F

Tukey’s (p)

IntelliPrev Users

Control (C)N= 108

Low (L)N = 48

Medium (M)N= 36

High (H)N= 19

Role Improvement 2.35 (0.9) 2.21 (1.02) 2.85 (1.02) 2.83 (1.04) 4.08* M > L (.021)

Performance Expectations 4.58 (1.3) 4.78 (.97) 5.36 (.96) 5.43 (1.06) 5.96* M > C (.003)

H > C (.017)

Effort Expectancy 4.71 (1.26) 4.77 (1.12) 5.23 (.94) 5.46 (0.88) 3.674* H > C (.046)

PreventionSelf-Efficacy 5.07 (.72) 4.87 (.62) 5.19 (.59) 5.33 (0.91) 2.469

General Knowledge 4.37 (1.17) 4.69 (1.12) 5.16 (1.01) 5.42 (1.23) 7.392* M > C (.002)

H > C (.002)

ANOVA for Outcomes and Coaching Variable

Means (SD) One-Way

F

Tukey’s (p)

IntelliPrev™ Users

Control (C)N= 108

No Coaching (NC)N= 77

Received Coaching (RC) N=28

Role Improvement

2.35 (.92) 2.32 (1.01) 3.05 (1.01) 6.27* RC> C (.003)RC>NC (.003)

Performance Expectations

4.58 (1.27) 4.95 (1.02) 5.44 (.95) 6.65*RC > C (.002)

Effort Expectancy

4.71 (1.26) 4.95 (1.08) 5.34 (.91) 3.50*RC > C (.029)

Prevention Self-Efficacy

5.077 (.72) 4.98 (.63) 5.28 (.82) 1.99

General Knowledge

4.37 (1.17) 4.81 (1.14) 5.40 (1.13) 9.65* NC > C (.032)RC > C (.000)

Results (3)Results (3)

Results (4)Results (4)

ANOVA for Outcomes and Webinar Variable

Means (SD) One-Way

F

Tukey’s (p)

IntelliPrev™ Users

Control (C)N= 108

No Webinar (NW)N= 47

Attended Webinar (AW)N= 58

Role Improvement

2.35 (0.92) 2.22 (1.12) 2.76 (0.95) 4.49* AW > C (.035)

Performance Expectations

4.58 (1.27) 4.82 (0.99) 5.29 (1.01) 6.99* AW > C (.001)

Effort Expectancy

4.71 (1.26) 4.82 (1.05) 5.24 (1.01) 3.98* AW > C (.015)

Self-Efficacy 5.08 (0.72) 4.80 (0.61) 5.27 (0.69) 6.01* AW>NW (.002)

General Knowledge

4.37 (1.17) 4.74 (1.18) 5.16 (1.11) 8.71* AW> C (.000)

DiscussionDiscussion

1. High overall engagement, via both high-tech and high-touch options, had greatest impact on users:• Expectations for their performance • Effort expectancy• General knowledge about health promotion

2. High touch engagement, through telephonic coaching, resulted in significantly higher scores in these areas:• Role improvement; Performance expectancy; Effort expectancy;

General knowledge about health promotion

3. High touch engagement, through attending a webinar, resulted in significantly higher scores in all 5 outcome areas

Overall, IntelliPrev™ Users reported higher scores than the control participants, especially if engaged, as measured by process variables.

Discussion (Cont.)Discussion (Cont.)

IMPLICATIONS•E-learning can effectively engage motivation, enhance

advocacy roles, and promote knowledge of prevention •‘Passive’ use adds value, but outcomes are stronger for

those receiving ‘high-touch’ support STRENGTHS & LIMITATIONS

•(+) Process data is collected weeks/months prior to criteria data so findings have predictive value

• (-) Lack of pre-test data limits claims about causality FUTURE RESEARCH

• Discriminate Analysis• Multiple Webinars?• Experience with internet/software programs