Development and Evaluation of the Global Appraisal of Individual Needs (GAIN) Validity Measures

Rodney Funk, Michael L. Dennis, Melissa Ives, Chestnut Health Systems, Bloomington, IL

and Richard Lennox, Psychometric Technologies, Hillsborough, NC

Workshop at the Joint Meeting on Adolescent Treatment Effectiveness (JMATE), Baltimore, MD, March 28, 2006.

The content of this presentations are based on treatment & research funded by the Center for Substance Abuse Treatment

(CSAT), Substance Abuse and Mental Health Services Administration (SAMHSA) under contract 270-2003-00006 using

data provided by the CYT grantees (Nos. TI11317, TI11320, TI11321, TI11323, and TI11324). The authors thank Sarah

Knecht, Michelle White and the GAIN QA team for helping to identify common inconsistencies, Sandra McGuinness for the code

for the inconsistencies and Barth Riley of UI-C for Rasch code. The opinions are those of the author and do not reflect official

positions of the consortium or government. Available on line at www.chestnut.org/LI/Posters or by contacting Joan Unsicker at 720 West Chestnut, Bloomington, IL 61701, phone: (309) 827-6026, fax: (309) 829-4661, e-Mail: junsicker@Chestnut.Org

Acknowledgement

CYT Cannabis Youth Treatment Randomized Field Trial

Sponsored by: Center for Substance Abuse Treatment (CSAT), Substance Abuse and Mental Health Services Administration (SAMHSA), U.S. Department of Health and Human Services

Coordinating Center:Chestnut Health Systems, Bloomington, IL, and Chicago, ILUniversity of Miami, Miami, FLUniversity of Conn. Health Center, Farmington, CT

Sites:Univ. of Conn. Health Center, Farmington, CTOperation PAR, St. Petersburg, FLChestnut Health Systems, Madison County, ILChildren’s Hosp. of Philadelphia, Phil. ,PA

Design • Target Population: Adolescents with marijuana disorders

who are appropriate for 1 to 3 months of outpatient treatment.

• Inclusion Criteria: 12 to 18 year olds with symptoms of cannabis abuse or dependence, past 90 day use, and meeting ASAM criteria for outpatient treatment

• Data Sources: self report on the GAIN (Dennis et al 2003), collateral reports, on-site and laboratory urine testing, therapist alliance and discharge reports, staff service logs, and cost analysis.

• Random Assignment: to one of three treatments within site in two research arms and quarterly follow-up interview for 12 months, and long term follow up 30 to 42 months later.

Source: Dennis et al 2002, 2004

Adolescent Cannabis Users in CYT were as or More Severe Than Those in TEDS*

100%

15 intake

85%

46%

26%

78%

26%

47%

26%

71%

0%

20%

40%

60%

80%

First usedunder age

Dependence Weekly ormore use at

PriorTreatment

% o

f A

dmis

sion

s .

CYT Outpatient(n=600) TEDS Outpatient (n=16,480)* Adolescents with marijuana problems admitted to outpatient treatment

Source: Tims et al, 2002

Demographic Characteristics

62%

15%

55%50%

30%

83%

17%

0%

20%

40%

60%

80%

100%

Female Male AfricanAmerican

Caucasian Under 15 15 to 16 Singleparentfamily

Source: Tims et al, 2002

Institutional Involvement

25%

87%

47%

62%

0%

20%

40%

60%

80%

100%

In school Employed Current CJInvolvement

Coming fromControlled

Environment

Source: Tims et al, 2002

Patterns of Substance Use

9%17%

71%73%

0%

20%

40%

60%

80%

100%

Weekly Tobacco Use

WeeklyCannabis Use

Weekly AlcoholUse

Significant Timein ControlledEnvironment

Source: Tims et al, 2002

Multiple Problems are the NORM

86%

37%

12%

25%

61%

60%

66%

83%

83%

0% 20% 40% 60% 80% 100%

Any Marijuana Use Disorder

Any Alcohol Use Disorder

Other Substance Use Disorders

Any Internal Disorder

Any External Disorder

Lifetime History of Victimization

Acts of Physical Violence

Any (other) Illegal Activity

Three to Twelve Problems

Self-Reported in Past Year

Source: Dennis et al, 2004

Co-occurring Problems are Higher for those Self-Reporting Past Year Dependence

71%

57%

25%

42%

30%37%

22%

5%

13%

22%

0%

20%

40%

60%

80%

100%

Health ProblemDistress*

Acute MentalDistress*

AcuteTraumaticDistress*

AttentionDeficit

HyperactivityDisorder*

ConductDisorder*

Past Year Dependence (n=278) Other (n=322)

Source: Tims et al., 2002 * p<.05

Validity Measures• Number of Inconsistencies –Count of 49 paired items

consistently answered by over 90% of the clients, but that are inconsistent

• Denial/Misrepresentation – Sum of staff rating over 8 sections on a scale of 0-no problem, 1-estimating, 2-misunderstanding, 3-denial, 4-misrepresentation

• Context Effect – staff report of problems that might effect the interview (e.g.., someone present, interruptions, in juvenile justice setting)

• Proportion of Missing Data on 86 Items used in the GAIN’s core 10 Change measures: Substance Frequency Scale, Current Withdrawal Scale, Substance Problem Scale, Health Problem Scale, Emotional Problem Scale, Recovery Environment Risk Index, Social Risk Index, Illegal Activity Scale, Training Activity Scale and the Employment Activity Scale

Validity Measures (Continued)This last two measures are based on theresidual (actual vs expected answers) of On the 123 items of the GAIN’s 4 mainpsychopathology and psychopathy scales. They are based the outfit and infit statistics under the Rasch (1960)measurement model and are reported inlogits• Atypicalness a measure of endorsing high

severity items without first endorsing the typical prior items (e.g.., suicide without depression)

• Randomness a measure of answers that are more random than expected on the GAIN’s 4 main psychopathology and psychopathy scales

N

yt

i ni2

Atypicalness

)1( nini

ninini

pp

pxy

Residual

N

inini

nini

N

i

pp

px

1

2

1

))1((

)(

Randomness

Where yni is the observed response of person n to item i and pni is the probability of a correct response for person n on item i.

Correlation of Validity Measures

Bold indicates p < .05.

Denial/Misrepr. 0.07 -- -- -- --

Context Effect 0.10 0.31 -- -- --

%Missing Data -0.06 0.03 0.05 -- --

Atypicalness 0.03 0.20 0.05 -0.02 --

Randomness 0.00 0.24 0.14 -0.04 0.57

Inconsistencies -- -- -- ----

Den

ial/

Mis

repr

.

Con

text

Eff

ect

%M

issi

ng D

ata

Aty

pica

lnes

s

Ran

dom

ness

Inco

nsis

tenc

ies

----

--

--

--

--

While there is someoverlap, for the mostpart these measures

capture different aspects of validity

Trichotomization of Validity Measures

• Like all GAIN scales we trichotomized the validity measures into low/mod/high range to help line staff interpret them.

• Because they were close to normally distributed, we divided Inconsistencies, Atypicalness and Randomness into: Low 0-50%, Mod 51-90% and High 91-100%

• Because they were sharply right skewed, Missing data, Denial/misrepresentation and Context effects were divided into Low 0%, Mod 1-90%, and High 91-100%

Overview of Validation Test Results

Validity Measures

Denial/Misrep Rating

Fal

se N

egat

ives

Rel

ativ

e T

o U

rin

e at

Int

ake

X

Missing Data

Context

Inconsistencies

Randomness (aka infit)

Bia

s 3

Mon

th o

utco

mes

X

X

X

Atypicalness (aka outfit)C

onst

ruct

Val

idit

y at

Int

ake

X

X

X

X

X

X

Tes

t-R

etes

t at

Int

ake

X

X

X

X

X

X

Inte

rnal

Con

sist

ency

at

Int

ake

X

X

X

X

X

X: Continuous or trichotomous version of validity measure is a statistically significant (p<.05) predictor of worse values on the criterion in this column

Internal Consistency Results

0.00

0.10

0.20

0.30

0.40

0.50

0.60

0.70

0.80

0.90

1.00

Ave

rage

Cro

nbac

h’s

Alp

ha*

Low 0.87 0.87 0.87 0.87 0.87 0.87 0.87

Mod 0.88 0.88 0.86 0.88 0.86 0.87 0.87

High 0.89 0.84 0.80 0.83 0.82 0.84

Denial/Misrep Rating

Missing Data Context Inconsistecies Randomness Atypicalness Average

*average alpha across Substance Problem Scale, Internal Mental Distress Scale, Behavior Complexity Scale & Crime/Violence Scale

Inconsistencies are the best predictor of low alpha

Test-Retest Results

0.00

0.10

0.20

0.30

0.40

0.50

0.60

0.70

0.80

0.90

1.00

Ave

rage

Tes

t-R

etes

t Rho

*

Low 0.68 0.66 0.67 0.66 0.68 0.70 0.68

Mod 0.64 0.64 0.65 0.69 0.64 0.62 0.65

High 0.61 0.59 0.65 0.56 0.59 0.60

Denial/Misrep Rating

Missing Data Context Inconsistecies Randomness Atypicalness Average

*average test-retest rho over 2-14 days for days of alcohol use, marijuana use, any substance use, and number of cannabis disorder symptoms

Randomness is the best predictor of low test-retest Rho

Correlations with Intake Variables

Bold indicates p < .05.

Spearman Rho

Day

s of

Any

AO

D u

se

Day

s of

Alc

ohol

Use

Day

s of

M

arj.U

se

Day

s of

AO

D P

robl

ems

Pas

t M

onth

Pro

blem

s

Day

s of

Arg

uing

/Vio

lenc

e

Day

s of

Ille

gal A

ctiv

ity

Day

s of

Ill

ega

l Act

ivity

for

$

Denial/Misrep -0.06 -0.07 -0.04 -0.10 -0.13 -0.02 -0.04 0.00

Missing Data 0.00 0.02 -0.01 -0.02 0.03 -0.03 0.05 0.07

Context 0.04 -0.01 0.06 -0.09 -0.02 -0.04 0.00 0.00

Inconsistencies -0.01 -0.01 -0.04 0.02 -0.02 -0.05 -0.14 -0.09

Randomness -0.17 -0.12 -0.14 -0.25 -0.23 -0.05 0.01 0.09

Atypicalness 0.02 0.02 0.03 -0.03 0.02 0.11 0.17 0.20

Randomness on symptoms generally predicts lower than expected values

Atypicalness predicts higher than

expected values

Staff Ratings of Denial/Misrepresentation Predict False Negatives Relative to Urine Screens

6%

15%

5%

0%

2%

4%

6%

8%

10%

12%

14%

16%

Denial/Misrepresentation

LowModHigh

Only significant predictor of False Negatives, OR=4.0, 95% CI (2.03, 7.96)

But not all Denial/

Misrepresentation is

about drug use

Predicting 3-Month Outcomes

• Predicted 3-month variables with intake only and in second step, added the validity measures

• Dependent Variables: Substance Frequency, Substance Problems, Emotional Problems, Recovery Environment, Social Risk and Illegal Activity

• Context and Inconsistencies had a small significant positive relationship with Recovery Environment Risk at 3 months

• Randomness had a small significant positive relationship with Substance Frequency and Illegal Activity

Limitations

• We had very little missing data, denial/ misrepresentation, and false negatives in the data from this multi-site clinical trial

• This data set was for outpatient and limited in severity and diversity

• We plan to replicate the analyses with several larger data sets that are more diverse in terms of clinical severity, geography, demographics, level of care and type of service providers

Conclusions

• The 6 GAIN validity measures are good markers for predicting problems with internal consistency, reliability, and validity

• Even where there were problems, self report was still generally reliable and valid

• The small correlations between measures and differences in what they predicted demonstrate that they are measuring different facets of the problem

• Having developed metrics for identifying problem cases, the next step is to develop interventions to reduce the likilihood of these problems.

References

• Dennis, M. L., Godley, S. H., Diamond, G., Tims, F. M., Babor, T., Donaldson, J., Liddle, H., Titus, J. C., Kaminer, Y., Webb, C., Hamilton, N., & Funk, R. (2004). The Cannabis Youth Treatment (CYT) Study: Main Findings from Two Randomized Trials. Journal of Substance Abuse Treatment, in press

• Dennis, M. L., Titus, J. C., Diamond, G., Donaldson, J., Godley, S. H., Tims, F., Webb, C., Kaminer, Y., Babor, T., Roebeck, M. C., Godley, M. D., Hamilton, N., Liddle, H., Scott, C., & CYT Steering Committee. (2002). The Cannabis Youth Treatment (CYT) experiment Rationale, study design, and analysis plans. Addiction, 97, 16-34.

• Dennis, M. L., Titus, J. C., White, M. K., Unsicker, J., & Hodgkins, D. (2003). Global Appraisal of Individual Needs: Administration Guide for the GAIN and Related Measures. Bloomington, IL: Chestnut Health Systems. Retrieved from http://www.chestnut.org/li/gain .

• Rasch, G. (1960). Probabilistic models for some intelligence and attainment tests. Copenhagen: Danmarks Paedogogiske Institut. (Republished Chicago: The University of Chicago Press: 1980).

• Tims, F. M., Dennis, M. L., Hamilton, N., Buchan, B. J., Diamond, G. S., Funk, R., & Brantley, L. B. (2002). Characteristics and problems of 600 adolescent cannabis abusers in outpatient treatment . Addiction, 97, 46-57.