8/6/2019 Scoring Model

1/22

1

Issues in Credit Scoring

Model Development and Validation

Dennis Glennon

Risk Analysis Division

Economics Department

The Office of the Comptroller of the Currency

The opinions expressed are those of the author and do not necessarily reflect those of the Office of the

Comptroller of the Currency.

8/6/2019 Scoring Model

2/22

2

Model Development and ValidationModel Development and Validation

Outline

1. Credit Risk vs. Model Risk2. Model Risk Analysis

3. Model Purpose

i. Classification

ii. Prediction

8/6/2019 Scoring Model

3/22

3

Model Review

Scope of a Review

I. Credit Risk The risk to earnings or capital of an obligor's failure to meet the

terms of any contract with the bank or otherwise fail to perform as

agreed.

II. Model Risk

Although model risk contributes to the overall portfolio or creditrisk, it represents a conceptually distinct exposure that emerges

from an overly broad interpretation or application of a model

beyond that for which it was developed.

8/6/2019 Scoring Model

4/22

4

Model Review

Scope of a Review

I. Credit Risk Analysis

i. evaluate strategiesii. assess current portfolio performance

II. Model Risk Analysisi. evaluate model validity, reliability, and

accuracy

8/6/2019 Scoring Model

5/22

5

Model Review

Model Risk Analysis

I. Are the models developed using valid statistical or

industry-accepted methods?

i. Appropriate sample design

a. truncated/censored samplesb. over-sample

8/6/2019 Scoring Model

6/22

6

Model Review

Model Risk Analysis (continued)

ii. Valid model design

a. satisfy minimum statistical requirements

b. in-sample performance (including holdout sample)

c. out-of-time performance

8/6/2019 Scoring Model

7/22

7

Model Review

Model Risk Analysis (continued)

II. Are the models used in ways that are consistent with the

original purpose for which the model was developed?

i. Model purpose

a. classificationb. prediction

8/6/2019 Scoring Model

8/22

8

Model Purpose

Model Purpose

The underlying objective of a classification-based

model is different from that of a prediction model.

As such, a model should be evaluated within thescope of its primary objective.

8/6/2019 Scoring Model

9/22

9

Model Purpose

Models as Classification Tools

Banks are developing or purchasing models that are designed

as classification tools. That is, the models are developed forthe purpose of partitioning populations or portfolios into

groups by their expected relativeperformance.

Modeling Objective: Maximize the divergence or separation betweenthe distributions of good and bad accounts.

8/6/2019 Scoring Model

10/22

10

K-S = 64.0

P erform ance D istribution

0

2000

4000

6000

150

170

190

210

230

250

270

290

score

0

5 0 0 0 0

1 0 0 0 0 01 5 0 0 0 0

2 0 0 0 0 0

2 5 0 0 0 0

b a d s g o o d s

Performance Distribution

0100020003000400050006000

150

170

190

210

230

250

270

290

score

0

50000

100000150000

200000

250000

bads goods

K-S = 26.5

Classification Design: Example

A Comparison of Model PerformanceA Comparison of Model Performance

8/6/2019 Scoring Model

11/22

11

Model Purpose

Classification Objective

Interpretation: If, for example, the good/bad odds ratio

associated with the score interval between 200-210 is 30:1,then the odds ratio for the intervals above (below) 200-210will be greater (less) than 30:1.

Result: A model that maintains its ability to rank-orderperformance is considered to be reliable.

8/6/2019 Scoring Model

12/22

12

Model Purpose

Classification-Based Models

Valid Purpose: models developed under this

criteria are valid as decision tools if the objective isto simply identify segments of the population that,as a group, perform poorly.

Appropriate for identifying and excluding specificsegments of the population -- a strategy that, inpractice, often improves average portfolio performancerelative to a random-selection method.

8/6/2019 Scoring Model

13/22

13

Classification Model

Log Odds Curve

0

1

2

3

4

5

6

7

644 653 665 675 684 693 706 715 725 739 753

Score Bands

ln(good/bad)

Development (K-S = 32.1)

Validation (K-S = 34.3)

ln(20/1) = 3

bad rate = .05

ln(20/1) = 3

bad rate = .05

ln(4/1) = 1.39

bad rate = .20

ln(4/1) = 1.39

bad rate = .20

8/6/2019 Scoring Model

14/22

14

Model Purpose

Alternative Purpose: Predicting Performance.

Banks want models for risk-based pricing/re-pricing

and profitability analysis -- models that are designed

specifically to address the issue of trading risk for

margin (i.e., return).

For that purpose, banks need models that are accurate

predictors of performance.

8/6/2019 Scoring Model

15/22

15

Model Selection: Which model is better?

0

1

y

Score (quintiles)

y

1

020 40 80600 10010 30 50 70 90

010

20 40 60 80 10030 7050 90

Score (quintiles)

9 7 5 3 111 6 5 2 1

1 74 11

951 5 4

3

[0.1][0.08] [0.45] [0.44] [0.67] [0.92]

[0.3] [0.5] [0.7] [0.9]

K-S = 48 K-S = 48

[#B / (#G + #B)]

[bad rate][bad rate]

obs. bad (B) - y=1obs. good (G) - y=0

Model 2Model 1

8/6/2019 Scoring Model

16/22

16

Model Purpose: Prediction

Models as Prediction Tools

Purpose: to predict the expected frequency at

which accounts with similar attributes perform(e.g., respond, attrite, default). For example,predict the probability of default.

Modeling Objective: Minimize the difference betweenthe predicted and actual percentage of defaults withineach score range (i.e., maximize the goodness-of-fit).

8/6/2019 Scoring Model

17/22

17

Model Purpose: Prediction

Prediction-Based Models

Interpretation: If within the interval 200-210 the risk

model predicts a probability of default of .04, then for every100 account that score within that range, four shoulddefault.

A model that satisfies this condition is considered to be

accurate.

This is a much stronger condition than that associated with aclassification objective (i.e., reliable).

8/6/2019 Scoring Model

18/22

18

Model Purpose: Prediction

Prediction-Based Models

Valid Purpose: models developed under this approachare valid as actuarial tools; as such, they are appropriate in

situations in which the actual, not just the relative,

measure of performance is required.

8/6/2019 Scoring Model

19/22

19

Model Purpose: Prediction

Limitations of a Prediction-Based Model

The model-development process is significantly more

complex especially when data across all aspects of thebehavior decision (i.e., individual, market, and industry)

are limited.

8/6/2019 Scoring Model

20/22

20

Model Purpose

Conclusion: Models are developed for different purposes --

e.g., classification or prediction. As such, the choices of:

- sample design,- modeling technique, and

- validation procedures

are driven by the intended purpose for which the model will

ultimately be used.

8/6/2019 Scoring Model

21/22

21

Model Purpose

Observation: The choice of modeling objective isimportant not only because it defines how we assess

its validity, but also because it defines a full set oftechnical estimation procedures that are used toselect the best model under the chosen objective.

8/6/2019 Scoring Model

22/22

22

Issues in Credit Scoring

Model Development and Validation

The End

Model Development and ValidationModel Development and Validation

The EndThe End