© 2014 Experian Information Solutions, Inc. All rights reserved. Experian and the marks used herein are service marks or registered trademarks of Experian Information Solutions, Inc.
Other product and company names mentioned herein are the trademarks of their respective owners. No part of this copyrighted work may be reproduced, modified, or distributed in
any form or manner without the prior written permission of Experian. Experian Public.
Inside the box — business scoring validations and model governance
Chuck Noel Compass Bank
Torsten Gerwien Experian
John Krickus Experian
#vision2014
2 © 2014 Experian Information Solutions, Inc. All rights reserved. Experian Public.
Model build and validation process
Metrics for validating model performance
Case study – BBVA Compass
Model governance policy and best practices
Agenda
3 © 2014 Experian Information Solutions, Inc. All rights reserved. Experian Public.
Model build and
validation process
John Krickus
Experian Business Information Services
4 © 2014 Experian Information Solutions, Inc. All rights reserved. Experian Public.
Bad definition: Firms filing bankruptcy / firms where GT 75% of trade dollars
are 91+ and / or negative trade comments for multiple quarters and at the end
Financial stability model process
Performance window
Month 0
Observation point
SOURCE: Experian Business Information Services
What archived data from the observation point, when all records had “clean” data,
would have predicted the “bads” at the end of the performance window in month 12?
All with “clean” data; no bad performance
Month 12
3% “bad”
rate
“Bads” observed in sample
5 © 2014 Experian Information Solutions, Inc. All rights reserved. Experian Public.
1) Population sampling 75/25 split is applied: 75% is for model development and 25% for model validation
2) Population segmentation
3) Proof of concept (POC) analysis to test alternative performance definitions, alternative segmentation scheme
4) Data treatment, variable selection and regression for each individual model
► Data treatment: Variable capping/false zero correction/variable transformations
► Variable selection: STEPDISC procedure in SAS is employed to do the variable selection
► Statistical regression
5) Finalize model and prepare documentation
Modeling build process Development steps
6 © 2014 Experian Information Solutions, Inc. All rights reserved. Experian Public.
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
1 2 3 4 5 6 7 8 9 10
Cum
ula
tive p
erc
ent
captu
red
Score decile, worst to best
Cumulative good
Cumulative bad
KS = 33.8
KS is a generally
accepted measure of a
model’s ability to separate
two populations
KS (Kolmogorov-Smirnoff) example
7 © 2014 Experian Information Solutions, Inc. All rights reserved. Experian Public.
Metrics for validating
model performance
Torsten Gerwien
Experian Decision Analytics
8 © 2014 Experian Information Solutions, Inc. All rights reserved. Experian Public.
Stability analysis – score distribution
The overall score distribution didn't change significantly; the difference in the two
distributions mainly exhibited that sample in 2010 has higher scores at low end
and lower scores at the high end
Score distribution comparison in percentile – IP214 CML
New (9/10/11/12-2010) Old (9/2008) Diff
100
90
80
70
60
50
40
30
20
10
0
Score
6
5
4
3
2
1
0
-1
-2
-3
-4
Dis
tributio
n d
iffere
nce
(poin
ts)
5 90 95 100 70 55 60 65 50 35 40 45 30 15 20 25 10 75 80 85
Percentile
9 © 2014 Experian Information Solutions, Inc. All rights reserved. Experian Public.
Stability analysis – divergence measure Score divergence result
Conclusion: No significant change
Sample Obs # Mean Median Min Max Var
Old 383,973 50.56 50 1.00 100 820.34
New 652,683 49.23 49 1.00 100 718.71
Divergence = 0.00
Def: DIV = ((NEW_MEAN – ORG_MEAN) ** 2) / (0.5*(NEW_VAR + ORG_VAR))
10 © 2014 Experian Information Solutions, Inc. All rights reserved. Experian Public.
Performance of industry-specific vs. all industry
80% bad capture for industry model vs. 50% for all-industry model (bottom 20% bads)
Bad definition: Two 60-day late pays or one 90-day late card payment
Model built off of Small Business Credit ShareSM financial trade and general business database, bad rate: 7.81%
Small Business Credit ShareSM: Bad capture bottom 20%, private card
SBCS = Small Business Credit ShareSM
11 © 2014 Experian Information Solutions, Inc. All rights reserved. Experian Public.
Superior ROI from Custom Model capabilities Experian business / SBCS data and analytic expertise
Combination of full range of business information and SBCS data from financial contributors
Experian Decision Analytics expertise
All combined in a Custom Model value offering delivered in weeks
Improving risk management results by 53%
3% lift
53% lift 41% lift 46% lift
SBCS = Small Business Credit ShareSM
12 © 2014 Experian Information Solutions, Inc. All rights reserved. Experian Public.
KS, bad capture for worst 10%, 20% and 30% of population
23
30%
43%
52%
29 29%
50%
59%
22
30%
40%
50%
31 31%
44%
59%
0%
10%
20%
30%
40%
50%
60%
70%
KS 10% 20% 30%
IPV1 IPV2 FSR IPV2+FSR
Intelliscore PlusSM v2
(IPV2)+ FSR matrix
provides the highest
KS statistic
However, bad capture
at the worst 20% shows
stronger performance
for IPV2
IPV2+FSR matrix
optimized for bad account
and $ capture
Performance results KS and bad account capture
13 © 2014 Experian Information Solutions, Inc. All rights reserved. Experian Public.
BBVA Compass
Validation results
Chuck Noel
SVP Global Risk Management
BBVA Compass
14 © 2014 Experian Information Solutions, Inc. All rights reserved. Experian Public.
Validation objectives
Objectives
Determine the ability of Experian information to predict Compass Bank medium size exposure bads for ongoing account management
Validated the performance of multiple Experian commercial risk models
Developed custom decision tree model to identify common characteristics of bads
Validated the predictiveness of Experian commercial triggers
BBVA Compass is a Sunbelt-based bank operating 687
branches in Texas, Alabama, Arizona, California, Florida,
Colorado and New Mexico. BBVA Compass ranks among
the Top-25 largest U.S. commercial banks based on
deposit market share
15 © 2014 Experian Information Solutions, Inc. All rights reserved. Experian Public.
Performance results KS and bad capture
The financial models (FC / FCC / FRC) had higher KS
The generic models (Intelliscore PlusSM v2 and FSR) have higher score (hit) rate
Focus will be on Intelliscore PlusSM v2 for custom decision tree
Bad Capture
Score 10% 20% 30% KS
Intelliscore PlusSM v2 39.73% 50.00% 55.21% 30.9
FSR5 30.55% 37.67% 51.37% 30.7
FTC 34.01% 52.38% 59.18% 33.8
FCC 45.67% 57.67% 67.33% 39.1
FRC 30.83% 61.25% 67.50% 42.9
16 © 2014 Experian Information Solutions, Inc. All rights reserved. Experian Public.
Decision tree model is used to illustrate benefits of customization
A decision tree is a statistical model that determines the optimal attributes for separating the good accounts from bad accounts
Each ‘branch node’ consists of an attribute which can be split on the attribute’s value ranges to further separate goods from bads
Attributes assessed for Compass model:
► Intelliscore PlusSM v2 risk score
► Business age
► Trade count
► Trade balance
► Trade payment behavior
► Collections
► Tax lines
► Judgments
Custom decision tree (DT) model
17 © 2014 Experian Information Solutions, Inc. All rights reserved. Experian Public.
Decisions tree example Numbers are example only, KS result is real
Score 1-24
Total = 200
1 Bad = 5
0 Good = 195
Within no trade segment, new “branch”
Low Intelliscore 1-24, 10% of applicants
Bad rate for this segment of 2.5%
5X greater bad rate then portfolio
Total = 8,000
1 Bad = 40
0 Good = 7,960
Start with total portfolio of 8,000 accounts
Bad rate of 0.5% or 40 accounts
No Trade
Total = 2,000
1 Bad = 15
0 Good = 1,885
Segment identified, higher bad rate 0.75%
No trade indicates 50% higher risk
KS improved 40% by using decision tree branches vs. generic
Intelliscore PlusSM v2 score
18 © 2014 Experian Information Solutions, Inc. All rights reserved. Experian Public.
Model governance policy
and best practices
Torsten Gerwien
Experian Decision Analytics
19 © 2014 Experian Information Solutions, Inc. All rights reserved. Experian Public.
Monitors attributes for soundness
Tracks the impact of changes on existing attributes
Provides regulatory audit support
All of which includes:
► Framework for effective attribute development and use
► Documentation management for robust version tracking
Business data attributes:
► Business Aggregates, Small Business Credit ShareSM Aggregates
► Delivered in batch file monthly (Commercial Risk DatabaseSM), archives, online via NetConnect
Attribute risk governance
Attribute governance is necessary for compliance
20 © 2014 Experian Information Solutions, Inc. All rights reserved. Experian Public.
OCC Bulletin 2000-16
2000
OCC Bulletin 2011-12
2011
.
The Supervisory Guidance on Model Risk Management OCC Bulletin 2011–12 extends the scope beyond model validation to policies, practices, standards for:
► Model development
► Model use and implementation
► Model governance and controls
Banks of all sizes are impacted
Model Risk Governance
21 © 2014 Experian Information Solutions, Inc. All rights reserved. Experian Public.
For additional information, please contact:
Hear the latest from Vision 2014
in the Daily Roundup:
www.experian.com/vision/blog
@ExperianVision | #vision2014
Follow us on Twitter
22 © 2014 Experian Information Solutions, Inc. All rights reserved. Experian Public.
Visit the Experian Expert Bar to learn more about
the topics and products covered in this presentation.