Predictive Modeling Spring 2005 CAMAR meeting Louise Francis, FCAS, MAAA Francis Analytics and...

transcript

Predictive ModelingSpring 2005 CAMAR meeting

Louise Francis, FCAS, MAAAFrancis Analytics and Actuarial Data Mining, Inc

www.data-mines.com

Objectives

Introduce Predictive modeling Why use it? Describe some methods in depth

Trees Neural networks Clustering

Apply to fraud data

Predictive Modeling Family

Predictive Modeling

Classical Linear Models GLMs Data Mining

Why Predictive Modeling?

Better use of insurance data

Advanced methods for dealing with messy data now available

Major Kinds of Modeling

Supervised learning Most common situation A dependent variable

Frequency Loss ratio Fraud/no fraud

Some methods Regression CART Some neural networks

Unsupervised learning No dependent variable Group like records

together A group of claims with

similar characteristics might be more likely to be fraudulent

Some methods Association rules K-means clustering Kohonen neural

networks

Two Big Specialties in Predicative Modeling

Regression Logistic Regressions Poisson Regression

0 125 250 375 500

90% 5% .5944 328.9876

Mean=100047.5

Distribution for Severity/B10

Values in Thousands

Mean=100047.5

0 125 250 375 500 -1 0 1 2 3 4 5 6 7

5% 90% 5% 0 3

Mean=1.001

Distribution for Claims/B3

Mean=1.001

-1 0 1 2 3 4 5 6 7

Data Mining

Trees Neural Networks Clustering

Modeling ProcessInternal

Data Cleaning

External Data

Other Preprocessing

Build Model Validate Model Test Model

Deploy Deploy ModelModel

Data Complexities Affecting Insurance Data Nonlinear functions Interactions Missing Data Correlations Non normal data

Kinds of Applications

Classification Prediction

The Fraud Study Data

• 1993 Automobile Insurers Bureau closed Personal Injury Protection claims

• Dependent Variables• Suspicion Score

• Number from 0 to 10• Expert assessment of liklihood of fraud or abuse

• 5 categories• Used to create a binary indicator

• Predictor Variables• Red flag indicators • Claim file variables

Introduction of Two Methods

Trees Sometimes known as CART (Classification and

Regression Trees) Neural Networks

Will introduce backpropagation neural network

Decision TreesDecision Trees

Recursively partitions the data Often sequentially bifurcates the data – but can

split into more groups Applies goodness of fit to select best partition at

each step Selects the partition which results in largest

improvement to goodness of fit statistic

Goodness of Fit StatisticsGoodness of Fit Statistics Chi Square CHAID (Fish, Gallagher, Monroe- Discussion

Paper Program, 1990)

Deviance CART

cases j

2 log( ) (categorical)

D= (y ) (or RSS for continuous variables)

i ik ikk

Observed-Expected2

Expectedi k

Goodness of Fit StatisticsGoodness of Fit Statistics

Gini Measure CART i is impurity measure

( , ) ( ) ( ) ( )L L R Rt s i t p i t p i t

Goodness of Fit StatisticsGoodness of Fit Statistics

Entropy C4.5

2 2( ) log ( ) log ( )EE

I E pN

2log ( )k kk

An Illustration from Fraud data: GINI Measure

Fraud/No FraudLegal Representation No Yes Total

No 626 80 706Yes 269 425 694

Total 895 505 1400Percent 64% 36%

First SplitFirst Split

All Claims p(fraud) = 0.36

Legal Rep = Yes

P(fraud) = 0 .612

Legal Rep = No

P(fraud) = 0.113

Example cont:

Root Node: 0.461199

Fraud/No FraudNo Yes 1-p(i)^2 Row %

No 0.887 0.113 0.201 50.4%Yes 0.388 0.612 0.475 49.6%

0.337 ,201*.504 .475*.496

.461 .337 0.124improvement

Example of Nonlinear FunctionSuspicion Score vs. 1st Provider Bill

1000 3000 5000 7000

Provider Bill

4.00ne

Neural Network Fit of SUSPICION vs Provider Bill

|mp1.bill<1279.5

mp1.bill<153

mp1.bill<842.5

mp1.bill<2389

0.3387

1.2850 2.2550

3.6430 4.4270

An Approach to Nonlinear Functions: Fit A Tree

21Provider Bill

0 5000 10000 15000

Fitted Curve From Tree

Neural NetworksNeural Networks

Developed by artificial intelligence experts – but now used by statisticians also

Based on how neurons function in brain

Neural NetworksNeural Networks

• Fit by minimizing squared deviation between fitted and actual values

• Can be viewed as a non-parametric, non-linear regression

• Often thought of as a “black box”• Due to complexity of fitted model it is difficult

to understand relationship between dependent and predictor variables

The Backpropagation Neural Network

Three Layer Neural Network

Input Layer Hidden Layer Output Layer(Input Data) (Process Data) (Predicted Value)

Neural Network

Fits a nonlinear function at each node of each layer

0 1 10, 1 0 1 1 ( ... )

1( ; ... ) ( ... )

1 n nn n n w w x w xh f X w w w f w w x w xe

The Logistic Function

-1.2 -0.7 -0.2 0.3 0.8

Logistic Function for Various Values of w1

w1=-10w1=-5w1=-1w1=1w1=5w1=10

Universal Function Approximator• The backpropagation neural network with one

hidden layer is a universal function approximator

• Theoretically, with a sufficient number of nodes in the hidden layer, any continuous nonlinear function can be approximated

Nonlinear Function Fit by Neural Network

1000 3000 5000 7000

Provider Bill

Neural Network Fit of SUSPICION vs Provider Bill

Interactions

Functional relationship between a predictor variable and a dependent variable depends on the value of another variable(s)

3000 8000 13000 18000

Provider Bill

inj.type: 01 inj.type: 02

inj.type: 03 inj.type: 04

inj.type: 05

Neural Network Predicted for Provider Bill and Injury Type

Interactions

Neural Networks The hidden nodes pay a key role in

modeling the interactions CART partitions the data

Partitions capture the interactions

|mp1.bill<1279.5

mp1.bill<153

injtype:abcefghi

injtype:abfgi

injtype:abcefgh

injtype:abcefgi

mp1.bill<2675.5

mp1.bill<2017.5

0.14 0.30

0.68 1.00 2.10

3.70 4.20

Simple Tree of Injury and Provider Bill

324000 10000 16000 4000 10000 16000

4000 10000 16000

mp1.bill

injtype: 1 injtype: 2 injtype: 4

Missing Data

Occurs frequently in insurance data There are some sophisticated methods for

addressing this (i.e., EM algorithm) CART finds surrogates for variables with missing

values Neural Networks have no explicit procedure for

missing values

More Complex Example

Dependent variable: Expert’s assessment of liklihood claim is legitimate A classification application

Predictor variables: Combination of claim file variables (age of claimant, legal representation) red flag variables (injury is strain/sprain only, claimant

has history of previous claim) Used an enhancement on CART known as boosting

Red Flag Predictor VariablesRed Flag Variables

Subject Indicator Variable Description Accident ACC01 No report by police officer at scene ACC04 Single vehicle accident ACC09 No plausible explanation for accident ACC10 Claimant in old, low valued vehicle ACC11 Rental vehicle involved in accident ACC14 Property Damage was inconsistent with accident ACC15 Very minor impact collision ACC16 Claimant vehicle stopped short ACC19 Insured felt set up, denied fault Claimant CLT02 Had a history of previous claims CLT04 Was an out of state accident CLT07 Was one of three or more claimants in vehicle Injury INJ01 Injury consisted of strain or sprain only INJ02 No objective evidence of injury INJ03 Police report showed no injury or pain INJ05 No emergency treatment was given INJ06 Non-emergency treatment was delayed INJ11 Unusual injury for auto accident Insured INS01 Had history of previous claims INS03 Readily accepted fault for accident INS06 Was difficult to contact/uncooperative INS07 Accident occurred soon after effective date Lost Wages LW01 Claimant worked for self or a family member LW03 Claimant recently started employment

Claim File Variables

Variable Description

AGE Age of claimant

RPTLAG Lag from date of accident to date reported

TREATLAGLag from date of accident to earliest treatment by service provider

AMBUL Ambulance charges

PARTDIS The claimant partially disabled

TOTDIS The claimant totally disabled

LEGALREP The claimant represented by an attorney

Claim Variables Available Early in Life of Claim

Neural Network Measure of Variable Importance

• Look at weights to hidden layer

• Compute sensitivities:• a measure of how much the predicted value’s

error increases when the variables are excluded from the model one at a time

Variable Importance

Rank Rank Variable Importance

1 LEGALREP 100.0 ||||||||||||||||||||||||||||||||||||||||||2 TRTLAG 69.7 |||||||||||||||||||||||||||||3 AGE 54.5 ||||||||||||||||||||||4 ACC04 44.4 ||||||||||||||||||5 INJ 01 42.1 |||||||||||||||||6 INJ 02 39.4 ||||||||||||||||7 ACC14 35.8 ||||||||||||||8 RPTLAG 32.4 |||||||||||||9 AMBUL 29.3 ||||||||||||

10 CLT02 23.9 |||||||||

Testing: Hold Out Part of Sample

• Fit model on 1/2 to 2/3 of data

• Test fit of model on remaining data

• Need a large sample

Testing: Cross-Validation

• Hold out 1/n (say 1/10) of data• Fit model to remaining data• Test on portion of sample held out• Do this n (say 10) times and average the

results• Used for moderate sample sizes• Jacknifing similar to cross-validation

Results of Classification on Test Data

Fitted TreeActual 0 1

0 77.3% 22.7%1 14.3% 85.7%

Fitted Neural NetworkActual 0 1

0 81.5% 18.5%1 26.7% 73.3%

Unsupervised LearningUnsupervised Learning

Common Method: Clustering No dependent variable – records are grouped

into classes with similar values on the variable

Start with a measure of similarity or dissimilarity

Maximize dissimilarity between members of different clusters

Dissimilarity (Distance) Dissimilarity (Distance) MeasureMeasure Euclidian Distance

Manhattan Distance

1( ) i, j = records k=variable

mij ik jkkd x x

Binary Variables

Row Variable1 0

1 a b a+b0 c d c+d

a+c b+dCo

Binary Variables

Sample Matching

Rogers and Tanimoto

a b c d

2( )( ) 2( )

a d b c

Results for 2 Clusters

Cluster Lawyer Back Claim Or Sprain Chiro or PT Prior Claim1 77% 73% 56% 26%2 3% 29% 14% 1%

AverageSuspicious Suspicion

Cluster Claim Score

1 56% 2.992 3% 0.21

Beginners Library

Berry, Michael J. A., and Linoff, Gordon, Data Mining Techniques, John Wiley and Sons, 1997

Kaufman, Leonard and Rousseeuw, Peter, Finding Groups in Data, John Wiley and Sons, 1990

Smith, Murry, Neural Networks for Statistical Modeling, International Thompson Computer Press, 1996

Data Mining

CAMAR Spring Meeting

Louise Francis, FCAS, MAAALouise_francis@msn.com

www.data-mines.com

Predictive Modeling Spring 2005 CAMAR meeting Louise Francis, FCAS, MAAA Francis Analytics and...

Documents