Fair Isaac Proprietary Information Page i
Validation Analysis Results
Small Business Administration
Small Business Scoring Service (SBSS)
Version 6.0 - Pooled
January 2015
Validation Analysis
Fair Isaac Proprietary Information Page ii
This is the confidential, unpublished property of Fair Isaac Corporation. Receipt or possession
of it does not convey rights to divulge, reproduce, use, or allow others to use it without the
specific written authorization of Fair Isaac and use must conform strictly to the license
agreement between user and Fair Isaac.
© Fair Isaac Corporation - All Rights Reserved.
Fair Isaac Proprietary Information Page iii
TABLE OF CONTENTS
EXECUTIVE SUMMARY ......................................................................................................................................... 1
VALIDATION DESIGN ............................................................................................................................................. 2
PORTFOLIOS............................................................................................................................................................... 2
OBSERVATION AND PERFORMANCE DATES ............................................................................................................... 2
Observation Dates: July 2012 to November 2014 .............................................................................................. 2
Performance Period: 18 Month Window ............................................................................................................ 2
Performance Date: December 2014 .................................................................................................................... 2
DATA EXCLUSIONS .................................................................................................................................................... 3
PERFORMANCE DEFINITIONS ..................................................................................................................................... 3
POPULATION OVERVIEW ..................................................................................................................................... 4
POPULATION STATISTICS ........................................................................................................................................... 4
SUMMARY COUNTS ................................................................................................................................................... 4
DETAILED COUNTS .................................................................................................................................................... 5
SCOREABLE COUNTS ................................................................................................................................................. 7
DESCRIPTIVE STATISTICS .......................................................................................................................................... 9
Portfolio Group 1 – Performance Category by Score Distributions .................................................................... 9
New versus Existing Business ............................................................................................................................ 11
Loan Size ............................................................................................................................................................ 12
Delivery Method ................................................................................................................................................. 13
SCORE DISTRIBUTIONS AND GRAPHS ............................................................................................................ 14
SCORE DISTRIBUTION .............................................................................................................................................. 14
Fair Isaac Proprietary Information Page 1
Executive Summary
This validation analysis has been completed for SBA to provide information on the strength of
the SBSS 6.0 score - Term Loan/Line of Credit models. Minimal data availability has limited
the extent of the analysis. Based on the score reports, FICO recommends that the SBA continue
to use the SBSS 6.0 score for their portfolio.
This document contains the final project deliverables, including:
a. Validation Design
a. Portfolios
b. Observation and Performance Dates
c. Exclusions
d. Performance Definitions
b. Population Overview
a. Population Statistics
b. Summary Counts
c. Detail Counts
c. Score Reports
a. Summary Score Statistics
b. Trade-Off Curves
c. Score Distributions
d. Cumulative Score Distributions
Fair Isaac Proprietary Information Page 2
Validation Design
The SBA portfolio has been validated based on SBSS 6.0 Term Loan/Line of Credit models.
The following validation design decisions have been made at the on-set of the project.
Portfolios
Observation and Performance Dates
Data Exclusions
Performance Definitions
Portfolios FICO evaluated the following portfolio groups and segments on the scores noted in the table.
Recommendations have been made at the portfolio level.
Portfolio Scores Evaluated
Portfolio Group 1 Overall Portfolio SBSS 6.0 Term Loan/Line of Credit
Observation and Performance Dates For validations of scores, FICO uses two snapshots of data: one from the observation date and
one from the performance date.
Observation Dates: July 2012 to November 2014
A sample of applications was taken from the time period July 2012 to November 2014. The data
as of the observation date captures the SBSS 6 score as of the application date, and other key
metrics about the small business. Typically in a validation of origination score, both approved
and declined application data is available. For this effort, only approved application data was
provided.
Performance Period: 18 Month Window
This is the forecast period over which the scores predict account risk. The performance period
begins the month after the observation date. To measure non-event/event behavior, the statuses
over the entire window has been reviewed.
Performance Date: December 2014
This date marks the last month of the performance period. Using account status information over
the performance period, an account is assigned to a performance category for modeling purposes.
Fair Isaac Proprietary Information Page 3
Data Exclusions
The following tables show the accounts that have been excluded from the analysis during the
data collection phase.
Category Description
Cancelled Loan approved but cancelled before disbursement
Committed Loan approved but not yet disbursed
Performance Definitions
All accounts active as of the observation dates were used in the development of the models.
Accounts were segmented into three performance categories based on their behavior during the
performance period.
Performance Definition – Early Default Definition
Event Account transfers to liquidation or purchase within the first 18 months on book
Non-Event Account is past due, pre-paid in full, or current
Indeterminate
Account becomes delinquent or in deferment within the first 18 months on book
Months on book < 18
Account has no reported statuses in the performance file
Performance Definition – Early Payment Problem Definition
Event Account becomes delinquent, in deferment, transfers to liquidation, or purchase
within the first 18 months on book
Non-Event Account is past due, paid in full, or current
Indeterminate Months on book < 18
Account has no reported statuses in the performance file
Fair Isaac Proprietary Information Page 4
Population Overview
This section provides an overview of the portfolios being analyzed, including population
statistics, summary counts, and detailed counts. Population statistics show the overall risk
associated with each portfolio through non-event-to-event odds and event rates. These figures
are ratios of the non-event and event populations. The summary counts indicate the distribution
of non-events, events and indeterminates over the portfolio. The detailed counts indicate the
distribution applicants which the larger assigned categories of non-event, event and
indeterminate.
Population Statistics
The following table shows the statistics for Portfolio Group 1.
Early Default
Early Payment
Problem
Population Odds 25.20 8.33
Event Rate 0.28% 0.84%
Summary Counts
The following table shows the frequency of accounts for Portfolio Group 1. These counts
represent the scoreable population only.
Non-Event Event Indeterminate
# % # % # %
Early Default 1,890 6.98% 75 0.28% 25,119 92.70%
Early Payment Problem 1,890 6.98% 227 0.84% 24,967 92.15%
Fair Isaac Proprietary Information Page 5
Detailed Counts
Early Default
Category Order Assign
Exclusions Total
(Row)
Total
# (Category)
Exclusions –
Booked
Accounts
10 20 Cancelled 6 11
11 22 Commit 5
Event 1 30 Purchase 26
75 2 31 Liquidation 49
Indeterminate –
Approved
3 60 Delinquent 127
25,119
4 61 Deferred 25
5 62 Month on Books < 1 7,400
6 63 Month on Books < 6 9,529
7 64 Month on Books < 12 6,246
8 65 Months on Books < 18 1,791
9 66 No MISStatus History Available 1
Non-Event
12 70 Past Due 42
1,890 13 80 Paid in Full 121
14 81 Current 1,727
Total Accounts 27,095 27,095
Fair Isaac Proprietary Information Page 6
Early Payment Problem
Category Order
Assign
# Exclusions
Total
(Row)
Total
(Category)
Exclusions –
Booked
Accounts
10 20 Cancelled 6 11
11 22 Commit 5
Event
1 30 Purchase 26
227 2 31 Liquidation 49
3 40 Delinquent 127
4 41 Deferred 25
Indeterminate
- Approved
5 62 Month on Books < 1 7,400
24,967
6 63 Month on Books < 6 9,529
7 64 Month on Books < 12 6,246
8 65 Months on Books < 18 1,791
9 66 No MISStatus History Available 1
Non-Event
12 70 Past Due 42
1,890 13 80 Paid in Full 121
14 81 Current 1,727
Total Accounts 27,095 27,095
Fair Isaac Proprietary Information Page 7
Scoreable Counts
Data Source Scorecard Description
Event -
Early
Default (#)
Non-
Event (#)
Indeterminate
(#)
Total
(#)
Term Loan &
LOC -
Consumer
126
Non-derogatory, oldest trade line is less than 20
years old, and total number of trade lines is less
than 20.
10 147 1,909 2,066
127
Non-derogatory, oldest trade line is less than 20
years old, and total number of trade lines is greater
than 20.
15 171 2,184 2,370
128 Non-derogatory and oldest trade line is greater
than or equal to 20 years old. 6 158 2,720 2,884
129 Derogatory, one trade line is 60 or more days
delinquent. 6 112 1,335 1,453
130 Severe derogatory, two or more trade lines are 60
or more days delinquent. 4 94 1,486 1,584
Term Loan &
LOC -
Consumer
and D&B
131
Non-derogatory, oldest trade line is less than 20
years old, and total number of trade lines is less
than 20.
5 126 1,719 1,850
132
Non-derogatory, oldest trade line is less than 20
years old, and total number of trade lines is greater
than 20.
7 226 2,332 2,565
133 Non-derogatory and oldest trade line is greater
than or equal to 20 years old. 6 322 4,313 4,641
134 Derogatory, one trade line is 60 or more days
delinquent. 7 128 1,814 1,949
135 Severe derogatory, two or more trade lines are 60
or more days delinquent. 4 118 2,056 2,178
Fair Isaac Proprietary Information Page 8
Data Source Scorecard Description
Event -
Early
Default (#)
Non-
Event
(#)
Indeterminate
(#)
Total
(#)
Other Data
Combinations
145 &146 Term Loan & LOC – No Consumer report, D&B - 18 217 235
149 Term Loan & LOC – Application, Consumer,
Financial 3 110 415 528
150 Term Loan & LOC – Application, Consumer,
Financial, and D&B 2 152 494 648
152 Term Loan & LOC – Application and Financial - 1 8 9
153 Term Loan & LOC – Application, Financial, and
D&B - 4 13 17
199 Unscoreable - 3 73 76
75 1,890 25,119 27,084
Fair Isaac Proprietary Information Page 9
Descriptive Statistics
Portfolio Group 1 – Performance Category by Score Distributions
Fair Isaac Proprietary Information Page 10
Fair Isaac Proprietary Information Page 11
New versus Existing Business
Values
Early Default Definition
Event
Non-
Event
Indeterminates
Delq or Defer
Indeterminates
MOB = 0
Indeterminates
Insufficient
History Odds
Existing Business 32 1,112 79 4,441 10,947 34.75
New Business 43 778 73 2,940 6,575 18.09
Missing -- -- -- 19 45 --
Total 75 1,890 152 7,400 17,567 25.20
Values
Early Payment Problem Definition
Event
Non-
Event
Indeterminates
MOB = 0
Indeterminates
Insufficient
History Odds
Existing Business 111 1,112 4,441 10,947 10.01
New Business 116 778 2,940 6,575 6.71
Missing -- -- 19 45 --
Total 227 1,890 7,400 17,567 8.33
Fair Isaac Proprietary Information Page 12
Loan Size
Values
Early Default Definition
Event
Non-
Event
Indeterminates
Delq or Defer
Indeterminates
MOB = 0
Indeterminates
Insufficient
History Odds
$0-$350,000 74 1,889 129 4,238 10,707 25.53
$350,001+ 1 1 23 3,162 6,860 1.00
Total 75 1,890 152 7,400 17,567 25.20
Values
Early Payment Problem Definition
Event
Non-
Event
Indeterminates
MOB = 0
Indeterminates
Insufficient
History Odds
$0-$350,000 203 1,889 4,238 10,707 9.31
$350,001+ 24 1 3,162 6,860 0.04
Total 227 1,890 7,400 17,567 8.33
Fair Isaac Proprietary Information Page 13
Delivery Method
Values
Early Default Definition
Event
Non-
Event
Indeterminates
Delq or Defer
Indeterminates
MOB = 0
Indeterminates
Insufficient
History Odds
CLP - - - 53 164 -
CMLAND - - 1 95 101 -
OTH 7A - - 10 1,246 2,198 -
PLP 3 - 25 3,103 6,818 -
RUREXP - - - 8 23 -
SBAEXP - - - 3 5 -
SMLNAD 72 1,890 116 2,892 8,258 26.25
Total 75 1,890 152 7,400 17,567 25.20
Values
Early Payment Problem Definition
Event
Non-
Event
Indeterminates
MOB = 0
Indeterminates
Insufficient
History Odds
CLP - - 53 164 -
CMLAND 1 - 95 101 -
OTH 7A 10 - 1,246 2,198 -
PLP 28 - 3,103 6,818 -
RUREXP - - 8 23 -
SBAEXP - - 3 5 -
SMLNAD 188 1,890 2,892 8,258 10.05
Total 227 1,890 7,400 17,567 8.33
Fair Isaac Proprietary Information Page 14
Score Distributions and Graphs
Score Distribution
The table below tallies the total accounts by score range, as well as the cumulative % of the column population. Odds are presented for
both the odds associated with the row’s score range as well as the cumulative ascending odds throughout the total range of scores.
The cumulative percentages are based on accounts at or below the score range.
Fair Isaac Proprietary Information Page 15
Portfolio Group 1, SBSS 6.0, Early Default:
Non-event Event Indeterminates: Total Odds
Score
Range # Cum % # Cum % # Cum % # Cum % Actual
Cum
Ascending
80-<90 - 0.00% 0 0.00% 2 0.01% 2 0.01% -- --
90-<100 - 0.00% 0 0.00% 16 0.06% 16 0.06% -- --
100-<110 - 0.00% 0 0.00% 33 0.13% 33 0.12% -- --
110-<120 - 0.00% 0 0.00% 103 0.41% 103 0.38% -- --
120-<130 - 0.00% 0 0.00% 250 1.00% 250 0.92% -- --
130-<140 - 0.00% 0 0.00% 509 2.03% 509 1.88% -- --
140-<150 108 5.71% 8 10.67% 1,766 7.03% 1,882 6.95% 13.50 13.50
150-<160 280 14.81% 18 24.00% 3,801 15.13% 4,099 15.13% 17.20 15.56
160-<170 527 27.88% 32 42.67% 6,814 27.13% 7,373 27.22% 17.64 16.47
170-<180 832 44.02% 44 58.67% 10,757 42.82% 11,633 42.95% 25.42 18.91
180-<190 1,172 62.01% 60 80.00% 14,804 58.94% 16,036 59.21% 21.25 19.53
190-<200 1,479 78.25% 68 90.67% 18,869 75.12% 20,416 75.38% 38.38 21.75
200-<210 1,690 89.42% 74 98.67% 22,401 89.18% 24,165 89.22% 35.17 22.84
210-<220 1,835 97.09% 75 100.00% 24,315 96.80% 26,225 96.83% 145.00 24.47
220-<230 1,876 99.26% 75 100.00% 24,907 99.16% 26,858 99.17% -- 25.01
230-<240 1,886 99.79% 75 100.00% 25,033 99.66% 26,994 99.67% -- 25.15
240-<250 1,887 99.84% 75 100.00% 25,042 99.69% 27,004 99.70% -- 25.16
250-<998 1,887 99.84% 75 100.00% 25,043 99.70% 27,005 99.71% -- 25.16
998 1,890 100.00% 75 100.00% 25,119 100.00% 27,084 100.00% -- 25.20
Total 1,890 100.00% 75 100.00% 25,119 100.00% 27,084 100.00% -- 25.20
Fair Isaac Proprietary Information Page 16
Non-event Event Indeterminates: Total Odds
Risk Level Score Range # Cum % # Cum % # Cum % # Cum % Actual Cum
Ascending
High Less than 160 280 14.81% 18 24.00% 3,801 15.13% 4,099 15.13% 15.56 15.56
Medium 160 -< 200 1,199 63.44% 50 66.67% 15,068 59.99% 16,317 60.25% 23.98 21.75
Low 200 or more 408 21.59% 7 9.33% 6,174 24.58% 6,589 24.33% 58.29 25.16
Unscoreable 998 3 0.16% -- 0.00% 76 0.30% 79 0.29% -- --
Total 1,890 100.00% 75 100.00% 25,119 100.00% 27,084 100.00% 25.20 25.20
Fair Isaac Proprietary Information Page 17
Portfolio Group 1, SBSS 6.0, Early Payment Problem:
Non-event Event Indeterminates: Total Odds
Score
Range # Cum % # Cum % # Cum % # Cum % Actual
Cum
Ascending
80-<90 0 0.00% 0 0.00% 1 0.01% 2 0.00% -- --
90-<100 0 0.00% 0 0.00% 15 0.06% 16 0.06% -- --
100-<110 0 0.00% 0 0.00% 32 0.13% 33 0.13% -- --
110-<120 0 0.00% 0 0.00% 102 0.41% 103 0.41% -- --
120-<130 0 0.00% 0 0.00% 249 1.00% 250 1.00% -- --
130-<140 0 0.00% 1 0.44% 507 2.03% 509 2.03% 0.00 0
140-<150 108 5.71% 19 8.37% 1,754 7.03% 1,882 7.02% 6.00 5.68
150-<160 280 14.81% 55 24.23% 3,763 15.13% 4,099 15.07% 4.78 5.09
160-<170 527 27.88% 87 38.33% 6,758 27.13% 7,373 27.07% 7.72 6.06
170-<180 832 44.02% 129 56.83% 10,671 42.82% 11,633 42.74% 7.26 6.45
180-<190 1,172 62.01% 164 72.25% 14,699 58.94% 16,036 58.87% 9.71 7.15
190-<200 1,479 78.25% 192 84.58% 18,744 75.12% 20,416 75.07% 10.96 7.70
200-<210 1,690 89.42% 215 94.71% 22,259 89.18% 24,165 89.15% 9.17 7.86
210-<220 1,835 97.09% 223 98.24% 24,166 96.80% 26,225 96.79% 18.13 8.23
220-<230 1,876 99.26% 227 100.00% 24,754 99.16% 26,858 99.15% 10.25 8.26
230-<240 1,886 99.79% 227 100.00% 24,880 99.66% 26,994 99.65% -- 8.31
240-<250 1,887 99.84% 227 100.00% 24,889 99.69% 27,004 99.69% -- 8.31
250-<998 1,887 99.84% 227 100.00% 24,890 99.70% 27,005 99.69% -- 8.31
998 1,890 100.00% 227 100.00% 24,966 100.00% 27,084 100.00% -- 8.33
Total 1,890 100.00% 227 100.00% 24,966 100.00% 27,084 100.00% -- 8.33
Fair Isaac Proprietary Information Page 18
Non-event Event Indeterminates: Total Odds
Risk Level Score Range # Cum % # Cum % # Cum % # Cum % Actual Cum
Ascending
High Less than 160 280 14.81% 55 24.23% 3,763 15.13% 4,099 15.07% 5.09 5.09
Medium 160 -< 200 1,199 63.44% 137 60.35% 14,981 59.99% 16,317 60.00% 8.75 7.70
Low 200 or more 408 21.59% 35 15.42% 6,146 24.58% 6,589 24.62% 11.66 8.31
Unscoreable 998 3 0.16% -- 0.00% 76 0.30% 79 0.31% -- --
Total 1,890 100.00% 227 100.00% 24,966 100.00% 27,084 100.00% 8.33 8.33