CANE 2007 Spring MeetingVisualizing Predictive Modeling ResultsChuck Boucek (312) 879-3859

AgendaData ValidationHypothesis BuildingModel BuildingModel TestingMonitoringVisualization as a Diagnostic Tool

Data ValidationGoalsValidate reasonableness of dataUnderstand key patterns in dataUnderstand changes in data and underlying business through time

Data ValidationHistogram is a simple tool to for reasonability testing of modeling database

Data ValidationMosaic Plot shows the distribution of predictors in two dimensions

Data ValidationMissing Data plot shows the relationship of missing data elements

Data ValidationTime series plots identify consistency of data over timeClaims Match to Exposure1993199419951996199719981999200020012002200320040.00.10.20.30.40.50.60.70.80.91.0Company 1Company 2Company 3

Hypothesis BuildingGoalsPerform initial analysis of potential predictor variablesLimit the list of predictor variables to be employed in subsequent phases of model buildingFurther reasonability testing of data

Frequency0.00.3750.7501.1251.50000.010.020.3Severity100001750025000325004000000.010.020.3Pure Premium5000750010000125001500000.0010.010.020.3Loss Ratio0.2500.3750.5000.6250.75000.010.020.3025005000750010000Exposure ($MM)00.010.020.3050100150200Premium ($MM)00.010.020.3Demographic Variable 10.0010.0010.0010.0010.001

Hypothesis BuildingQuantile-Quantile plots help identify needed transformations of data

Hypothesis BuildingCorrelation Web concisely summarizes a correlation matrix

Model BuildingModel building is an iterative processUnderstanding patterns and relationships throughout this process is critical

Model BuildingPartial Plots are a key tool to visualize predictor variables throughout the model building processWhat is a Partial Plot? Linear Predictor = k + b1X1 + b2X2 + b3X3 + b4X4 Predicted value = (ek) x (eb1X1) x (eb2X2) x (eb3X3) x (eb4X4)Partial Plot demonstrates an individual predictor variables contribution to final prediction

Model BuildingPartial Plot demonstrates an individual predictor variables contribution to final prediction

Model BuildingPartial Plot with modified scatter plot of variable

Model BuildingTime Consistency plot is a critical tool for numeric predictors

Model BuildingPartial Plot for a factor variable

Frequency0.00.3750.7501.1251.500Severity1000017500250003250040000NoYesPure Premium50007500100001250015000Loss Ratio0.2500.3750.5000.6250.750025005000750010000Exposure (Pred. Count)050100150200Premium ($MM)Credit Variable 1NoYesNoYesNoYesNoYesNoYes

Model TestingLikely the most critical visualizations in predictive modeling workManagements perception of a projects success will likely depend on these visualizationsHoldout testsCross validation tests

Model TestingLift Chart shows overall model performance0.40.50.60.70.80.91.00.40.50.60.70.80.71.0PredictedActualLoss Ratio Lift Chart - Holdout Sample

Model TestingROC Curve shows overall model performanceNull, 0Perfect, 1prem, 0.51pred.loss, 0.56Holdout Sample ROC Curve

Model TestingClassical Cross Validation exhibit

Monitoring Model ResultsThe work does not end when the lift chart looks goodMonitoring toolsDecile managementException analysisModel vs. Actual Results

Monitoring Model ResultsDecile ManagementRetentionLoss RatioRate ActionTier/Schedule Mod

Monitoring Model ResultsAverage score over time

Monitoring Model ResultsLoss ratio of model exceptions

Visualization as Diagnostic ToolFrequency and severity models have been developedModel is underperforming in predicting loss ratioLikely cause of underperformance is severity model

Visualization as Diagnostic Tool

Visualization as Diagnostic Tool

Visualization as Diagnostic Tool

Visualization as Diagnostic ToolTwo different visualizations of the same model tell a very different story!