Risk Evaluation:Maximizing Risk Accuracy

MATSA/MASOC

Presentation to SORB

1/31/2013

Overview of Presentation

• Brief history of risk assessment and the different kinds of assessment that have been developed;

• Indication of where MA SORB Classification fits in these strategies;

• Summary of the criteria for evaluating risk instruments;

• Quick overview of the recent empirical evaluations of risk instruments;

• Suggest strategies for improving the MA SORB Classification.

BRIEF HISTORY OF RISK ASSESSMENT

Brief History

• First generation – Unstructured clinical judgment, including structured clinical guidelines (SCG).

• Second generation – Actuarial risk scales comprising static, historical factors.

• Third generation – the assessment of “criminogenic needs” or dynamic risk factors.

Bonta, 1996

Fixed or historical factors that cannot be changed

(such as age at first offense)

Potentially changeable factors, both stable, but potentially

changeable risk traits, and acute, rapidly changing factors.

Brief History

• Characteristics of Unstructured Clinical Judgments –• No items specified for considering risk level;• Method for combining items is not specified.

(Hanson & Morton-Bourgon, 2009)

First Generation

Brief History

• Characteristics of SCGs–• They identify items to use in the decision and

typically provide numerical values for each item;

• Although they also usually provide a method for combining the items into a total score, they do not specify a priori how the clinician should integrate the items;

• No tables linking the summary scores to recidivism rates.

(Hanson & Morton-Bourgon, 2009)

First Generation

Brief History

• Requirements of Empirical Actuarials –• Provide specific items to make the decision with

quantitative anchors, which are derived from empirical investigation;

• Method for combining the items into an overall score is specified;

• Tables linking the summary scores to recidivism rates are provided.

(Hanson & Morton-Bourgon, 2009)

Second Generation

Brief History

• Requirements of Mechanical Actuarials –• They provide specific items for the decision

with numeric values for each item, which are derived from a review of literature and theory;

• Method for combining the items into an overall score is specified;

• Tables linking the summary scores to recidivism rates are not provided.

(Hanson & Morton-Bourgon, 2009)

Second Generation

Brief History

• Additional condition Adjusted Actuarials –• Use appropriate actuarials (empirical or

mechanical);• The clinician adjusts the score (and the

recommendation) using factors external to the actuarial.

(Hanson & Morton-Bourgon, 2009)

Second Generation

MA SORB CLASSIFICATION FACTORSWhere Does It Fit?

MA SORB Classification Factors

• Somewhere between an unstructured judgment and an SCG –• It specifies a set of factors to be considered; but• It does not provide any quantification of these

factors (i.e., numeric item scores).• In many items it does not provide clear

specification of where the cutoff for “presence” or “absence” of a factor would be.

• Thus, it provides limited guidance both on the presence of items and on the combining of items.

Where Does It Fit?

Item 3. PsychopathyCode this by reference to the PCLR. Code PCLR scores of 30 or above as “Y”, scores of 21-29 as “?”, and scores of 20 or lower as “N”.

Y = 2

? = 1

N = 0

Example of SVR-20MA SORB Classification Factors

Item 2. Repetitive and Compulsive Behavior

Example of SORB Factors

?charges, convictions, self-report?

?includes both impulsive and compulsive behavior?

MA SORB Classification Factors

EVALUATING RISK TOOLSEvaluating Reliability and Validity

ASSESSING RELIABILITYDo Evaluators Agree?

Reliability

• Accuracy• Consistency

• Across raters• Across time• Across different measures of the same construct

• Freedom from variable error.

Reliability is --

Reliability

• Interrater

Interrater Reliability

R 1 R 2Agreement

Reliability

• Interrater• Internal Consistency

Internal Consistency

• Allows one to calculate various forms of reliability –• Item reliability• Reliability of subscales (e.g., sexual deviance,

criminality, etc.)• Internal consistency of items in the instrument

• Thus, quantification allows us to restructure items and their anchors to improve reliability.

Advantages of QuantificationReliability Checks

• Most popular SCGs and actuarials assessed in the comparative literature have acceptable reliability.

• Unstructured judgments have poor reliability.

• The reliability of MA SORB Classification Factors have not been assessed.

SCGs and ActuarialsReliability Results

ASSESSING VALIDITYPredicting Recidivism

Validity Answers the Question

• Does a test measure what it is suppose to measure?

• What does a test measure?• What can one do with the test? • What does a test score predict?

Validity Answers the Question

• Does a test measure what it is suppose to measure?

• What does a test measure?• What can one do with the test? • What does a test score predict?

Predicting Sexual Recidivism

Instrument Type d (95% CI)Empirical Actuarial .67 (.63 - .72)Mechanical Actuarial .66 (.58 - .74)SCG .46 (.29 - .62)Unstructured Judgmt .42 (.32 - .51)

(Hanson & Morton-Bourgon, 2009)

• Overall, controlling for a large number of study variables, Empirical and Mechanical were significantly better predictors of recidivism;

• SCGs using clinical judgment and SCGs that calculate total scores do not differ.

• In all studies examined, clinicians’ adjustment of actuarial scores consistently lowered predictive accuracy.

Predicting Sexual Recidivism

(Hanson & Morton-Bourgon, 2009)

• Across multiple areas of prediction, mechanical actuarial prediction (statistical prediction rules [SPRs]) has been shown to be superior to clinical judgment.

• A recent meta-analysis summarizes the results of years of research (Grove et al., 2000).

Why Is Clinical Judgment Inferior?

• All studies published in English from 1920s to mid 1990s.

• 136 studies on the prediction of health-related phenomena or human behavior.

(Grove et al., 2000)

(Grove et al., 2000)

• A large body of research has documented the reasons for the cognitive errors that clinicians make.

• For instance, clinicians are great at making observations and rating items, but they are worse than a formula at adding the items together and combining them.

Why Is Clinical Judgment Inferior?

• Allows one to use various strategies for improving validity of a measure–• Assess item correlation with outcome;• Adjust item cutoffs to maximize prediction;• Assess the validity of subscales (e.g., sexual deviance,

criminality, etc.);• Optimize item weights for decision-making and

predicting.

• Thus, one can restructure items, their anchors, cutoffs, and combinations to improve validity.

Advantages of QuantificationValidity Checks

STRATEGIES FOR IMPROVING MA SORB CLASSIFICATION

Potential Strategies

• Create separate adult and juvenile actuarials;• Create separate male and female actuarials;• Divide instrument into static and dynamic item

subsets;• Use recent meta-analytic literature to purge items

that are not likely to have predictive validity;

Improving the Current MA SORB Criteria

Examples of Poor Predictors

• Released from civil commitment vs. not committed (Knight & Thornton, 2007)

• Maximum term of incarceration;• Current home situation (?vague and

unspecified?);• Physical condition;• Documentation from a licensed mental health

professional specifically indicating that offender poses no risk to reoffend;

Examples of Poor Predictors

• Recent behavior while incarcerated;• Recent Threats;• Supplemental material;• Victim impact statement.

Potential Strategies

• Create separate adult and juvenile actuarials;• Create separate male and female actuarials;• Divide instrument into static and dynamic item

subsets;• Use recent meta-analytic literature to purge items

that are not likely to have predictive validity;• Transform items into a quantifiable format with

clear cutoffs;• Do a preliminary check on the predictive validity of

revised items using existing data bases.

Improving the Current MA SORB Criteria

Potential Strategies

• Do a small study on a subset of offenders to establish reliability.

• When using the revised instrument, require item and total scores for future validation studies.

Improving the Current Criteria

Potential Strategies

• Follow the lead of some other states and use existing static and dynamic instruments on which substantial research has already been done.

• MATSA and MASOC would be happy to consult on any strategy that MA SORB wishes to implement to improve the reliability and validity of current classification method.

Alternatively