Vision 2014: Inside The Box Business Scoring Validations and Model Governance

© 2014 Experian Information Solutions, Inc. All rights reserved. Experian and the marks used herein are service marks or registered trademarks of Experian Information Solutions, Inc.

Other product and company names mentioned herein are the trademarks of their respective owners. No part of this copyrighted work may be reproduced, modified, or distributed in

any form or manner without the prior written permission of Experian. Experian Public.

Inside the box — business scoring validations and model governance

Chuck Noel Compass Bank

Torsten Gerwien Experian

John Krickus Experian

#vision2014

2 © 2014 Experian Information Solutions, Inc. All rights reserved. Experian Public.

Model build and validation process

Metrics for validating model performance

Case study – BBVA Compass

Model governance policy and best practices

Agenda


Model build and

validation process

John Krickus

Experian Business Information Services


Bad definition: Firms filing bankruptcy / firms where GT 75% of trade dollars

are 91+ and / or negative trade comments for multiple quarters and at the end

Financial stability model process

Performance window

Month 0

Observation point

SOURCE: Experian Business Information Services

What archived data from the observation point, when all records had “clean” data,

would have predicted the “bads” at the end of the performance window in month 12?

All with “clean” data; no bad performance

Month 12

3% “bad”

rate

“Bads” observed in sample


1) Population sampling 75/25 split is applied: 75% is for model development and 25% for model validation

2) Population segmentation

3) Proof of concept (POC) analysis to test alternative performance definitions, alternative segmentation scheme

4) Data treatment, variable selection and regression for each individual model

► Data treatment: Variable capping/false zero correction/variable transformations

► Variable selection: STEPDISC procedure in SAS is employed to do the variable selection

► Statistical regression

5) Finalize model and prepare documentation

Modeling build process Development steps


0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

1 2 3 4 5 6 7 8 9 10

Cum

ula

tive p

erc

ent

captu

red

Score decile, worst to best

Cumulative good

Cumulative bad

KS = 33.8

KS is a generally

accepted measure of a

model’s ability to separate

two populations

KS (Kolmogorov-Smirnoff) example


Metrics for validating

model performance

Torsten Gerwien

Experian Decision Analytics


Stability analysis – score distribution

The overall score distribution didn't change significantly; the difference in the two

distributions mainly exhibited that sample in 2010 has higher scores at low end

and lower scores at the high end

Score distribution comparison in percentile – IP214 CML

New (9/10/11/12-2010) Old (9/2008) Diff

100

90

80

70

60

50

40

30

20

10

0

Score

6

5

4

3

2

1

0

-1

-2

-3

-4

Dis

tributio

n d

iffere

nce

(poin

ts)

5 90 95 100 70 55 60 65 50 35 40 45 30 15 20 25 10 75 80 85

Percentile


Stability analysis – divergence measure Score divergence result

Conclusion: No significant change

Sample Obs # Mean Median Min Max Var

Old 383,973 50.56 50 1.00 100 820.34

New 652,683 49.23 49 1.00 100 718.71

Divergence = 0.00

Def: DIV = ((NEW_MEAN – ORG_MEAN) ** 2) / (0.5*(NEW_VAR + ORG_VAR))


Performance of industry-specific vs. all industry

80% bad capture for industry model vs. 50% for all-industry model (bottom 20% bads)

Bad definition: Two 60-day late pays or one 90-day late card payment

Model built off of Small Business Credit ShareSM financial trade and general business database, bad rate: 7.81%

Small Business Credit ShareSM: Bad capture bottom 20%, private card

SBCS = Small Business Credit ShareSM


Superior ROI from Custom Model capabilities Experian business / SBCS data and analytic expertise

Combination of full range of business information and SBCS data from financial contributors

Experian Decision Analytics expertise

All combined in a Custom Model value offering delivered in weeks

Improving risk management results by 53%

3% lift

53% lift 41% lift 46% lift

SBCS = Small Business Credit ShareSM


KS, bad capture for worst 10%, 20% and 30% of population

23

30%

43%

52%

29 29%

50%

59%

22

30%

40%

50%

31 31%

44%

59%

0%

10%

20%

30%

40%

50%

60%

70%

KS 10% 20% 30%

IPV1 IPV2 FSR IPV2+FSR

Intelliscore PlusSM v2

(IPV2)+ FSR matrix

provides the highest

KS statistic

However, bad capture

at the worst 20% shows

stronger performance

for IPV2

IPV2+FSR matrix

optimized for bad account

and $ capture

Performance results KS and bad account capture


BBVA Compass

Validation results

Chuck Noel

SVP Global Risk Management

BBVA Compass


Validation objectives

Objectives

Determine the ability of Experian information to predict Compass Bank medium size exposure bads for ongoing account management

Validated the performance of multiple Experian commercial risk models

Developed custom decision tree model to identify common characteristics of bads

Validated the predictiveness of Experian commercial triggers

BBVA Compass is a Sunbelt-based bank operating 687

branches in Texas, Alabama, Arizona, California, Florida,

Colorado and New Mexico. BBVA Compass ranks among

the Top-25 largest U.S. commercial banks based on

deposit market share


Performance results KS and bad capture

The financial models (FC / FCC / FRC) had higher KS

The generic models (Intelliscore PlusSM v2 and FSR) have higher score (hit) rate

Focus will be on Intelliscore PlusSM v2 for custom decision tree

Bad Capture

Score 10% 20% 30% KS

Intelliscore PlusSM v2 39.73% 50.00% 55.21% 30.9

FSR5 30.55% 37.67% 51.37% 30.7

FTC 34.01% 52.38% 59.18% 33.8

FCC 45.67% 57.67% 67.33% 39.1

FRC 30.83% 61.25% 67.50% 42.9


Decision tree model is used to illustrate benefits of customization

A decision tree is a statistical model that determines the optimal attributes for separating the good accounts from bad accounts

Each ‘branch node’ consists of an attribute which can be split on the attribute’s value ranges to further separate goods from bads

Attributes assessed for Compass model:

► Intelliscore PlusSM v2 risk score

► Business age

► Trade count

► Trade balance

► Trade payment behavior

► Collections

► Tax lines

► Judgments

Custom decision tree (DT) model


Decisions tree example Numbers are example only, KS result is real

Score 1-24

Total = 200

1 Bad = 5

0 Good = 195

Within no trade segment, new “branch”

Low Intelliscore 1-24, 10% of applicants

Bad rate for this segment of 2.5%

5X greater bad rate then portfolio

Total = 8,000

1 Bad = 40

0 Good = 7,960

Start with total portfolio of 8,000 accounts

Bad rate of 0.5% or 40 accounts

No Trade

Total = 2,000

1 Bad = 15

0 Good = 1,885

Segment identified, higher bad rate 0.75%

No trade indicates 50% higher risk

KS improved 40% by using decision tree branches vs. generic

Intelliscore PlusSM v2 score


Model governance policy

and best practices

Torsten Gerwien

Experian Decision Analytics


Monitors attributes for soundness

Tracks the impact of changes on existing attributes

Provides regulatory audit support

All of which includes:

► Framework for effective attribute development and use

► Documentation management for robust version tracking

Business data attributes:

► Business Aggregates, Small Business Credit ShareSM Aggregates

► Delivered in batch file monthly (Commercial Risk DatabaseSM), archives, online via NetConnect

Attribute risk governance

Attribute governance is necessary for compliance


OCC Bulletin 2000-16

2000

OCC Bulletin 2011-12

2011

.

The Supervisory Guidance on Model Risk Management OCC Bulletin 2011–12 extends the scope beyond model validation to policies, practices, standards for:

► Model development

► Model use and implementation

► Model governance and controls

Banks of all sizes are impacted

Model Risk Governance


For additional information, please contact:

[email protected]

[email protected]

Hear the latest from Vision 2014

in the Daily Roundup:

www.experian.com/vision/blog

@ExperianVision | #vision2014

Follow us on Twitter


Visit the Experian Expert Bar to learn more about

the topics and products covered in this presentation.

Date post:	06-May-2015
Category:	Business
Upload:	experian
View:	122 times
Download:	0 times