Post on 07-Nov-2020
transcript
Model Risk Management
Nav Vaidhyanathan
Director, Head of Model Risk Management
Wintrust Financial Corporation
Global Association of Risk Professionals
August 2014
2
The views expressed in the following material are the author’s and do not
necessarily represent the views of the Global Association of Risk
Professionals (GARP), its Membership or its Management.
These materials reflect the views and opinions of the speaker and are not
intended to reflect the views, policies or practices of Wintrust Financial or its
affiliated companies. They are for informational purposes only.
3 | © 2014 Global Association of Risk Professionals. All rights reserved.
Agenda
Model Risk Management is a broad, evolving, and an expanding topic. This presentation
will cover only select components at a high level.
Inherent risk (model tier) and residual risk of model
A framework for model monitoring
Model risk aggregation framework
Model validation for stress test (e.g., DFAST) models
4 | © 2014 Global Association of Risk Professionals. All rights reserved.
Model Risk Axioms*
The following can potentially be considered as self-evident truths for Model Risk. Given
everything else equal,
All models have limitations
Model Risk is related to model limitations
Model Risk increases with more models (this might fall in the grey area)
− E.g., two Tier 1 models potentially pose higher risk than 1 Tier 1 model (everything else being the
same, i.e., similar limitations, similar validation status, similar materiality, similar findings, etc.)
Model Risk for Tier 1 model > Model Risk for Tier 2 model
Model Risk for validated model ≤ Model Risk for unvalidated model (for the same model)
More findings ⇒ more Model Risk (same category and level of findings)
Certain types of findings will have higher Model Risk than others
Models with deteriorating performance pose higher Model Risk
Model Risk is exhibited by the variance between model output and observed actual (this is just one of the ways that Model Risk exhibits itself)
* may not be axioms in the purest sense of the term
5 | © 2014 Global Association of Risk Professionals. All rights reserved.
Inherent Risk and Residual Risk of Model
Residual Risk Rating could be
driven by:
• Model validation findings:
• Category
• Level
• # of Open Findings
• Model performance /
monitoring
Model owners can reduce residual
risk – by improving models,
improving controls, and/or
remediating findings
Res
idu
al
Ris
k R
atin
g
High
Moderate
Low
Low (Tier 3) Moderate High (Tier 1)
Inherent Risk Rating (Model Tier)
Inherent Risk Rating could be driven by:
• Materiality
• Type of model
• Used for regulatory / financial reporting or critical business
decisions
• Feeds into another Tier 1 model?
Model owners cannot (usually) reduce inherent risk
6 | © 2014 Global Association of Risk Professionals. All rights reserved.
Residual Risk Rating Framework
Residual Risk Rating (High,
Moderate, Low) is based on
the Residual Risk Score range
E.g., RRS > 200 ⇒ High, RRS ≤
100 ⇒ Low
If the model is deemed unfit for
use, the RRR is High
The Residual Risk Score
Range to Residual Risk
Ratings mapping needs to be
monitored periodically
The ranges can be determined
by typical observations from
model validation and model
monitoring exercises
If a model is unvalidated, the
residual risk rating is HIGH
7 | © 2014 Global Association of Risk Professionals. All rights reserved.
Model Monitoring
Model monitoring should evaluate whether changes (including anticipated) in
products, exposures, activities, clients, or market conditions necessitate
adjustment, redevelopment, or replacement of the model
Ongoing activities include – review of input data and assumptions, outcome
analysis, model stability checks, review of overlays and expert judgments,
benchmarking, etc.
Key considerations
− Model performance monitoring
› Pros and cons of model performance measures
› Model type
› Model use
› Type of measure
− Periodicity of monitoring
− Thresholds for model recalibration or rebuild
− From performance measure to model performance rating
8 | © 2014 Global Association of Risk Professionals. All rights reserved.
A Bottom Up Framework For Model Risk Aggregation
Impact of Failure (e.g., on Capital)
Tier 1 Tier 2 Tier 3
Validated … …
Yes No
High Residual
Risk
Moderate
Residual Risk Low Residual
Risk
As part of an RCSA
process, the impact of
the model failure is
estimated by model
owners
The end nodes in the
decision tree can be
used for aggregating
model risk
For DFAST projection models, for Severely Adverse Scenario, the impact on capital can be assessed
using backtesting during Great Recession period for all models (adjust the variance by severity of the
scenario compared to the Great Recession)
By looking at the distribution, a conservative starting point for capital can be chosen for each model
Tier
9 | © 2014 Global Association of Risk Professionals. All rights reserved.
Some Considerations in Validation of Stress Test Models
Are estimates conditioned on macroeconomic scenarios?
Do the models produce outcomes that show separation and order between the baseline,
adverse, and severely adverse scenarios?
Are the scenario and assumptions consistent across all models?
Are exposures modeled using appropriate levels of segmentation?
Qualitative elements of the model
Relevance of third party vendor models to company specific characteristics, especially when
using proxy portfolio
Sensitivity analysis around assumptions
Use of weak models vs. expert judgment
For models not validated, compensating controls and conservativeness
Documentation
Capital buffer for model risk
C r e a t i n g a c u l t u r e o f
r i s k a w a r e n e s s ®
Global Association of
Risk Professionals
111 Town Square Place
14th Floor
Jersey City, New Jersey 07310
U.S.A.
+ 1 201.719.7210
2nd Floor
Bengal Wing
9A Devonshire Square
London, EC2M 4YN
U.K.
+ 44 (0) 20 7397 9630
www.garp.org
About GARP | The Global Association of Risk Professionals (GARP) is a not-for-profit global membership organization dedicated to preparing professionals and organizations to make
better informed risk decisions. Membership represents over 150,000 risk management practitioners and researchers from banks, investment management firms, government agencies,
academic institutions, and corporations from more than 195 countries and territories. GARP administers the Financial Risk Manager (FRM®) and the Energy Risk Professional (ERP®)
Exams; certifications recognized by risk professionals worldwide. GARP also helps advance the role of risk management via comprehensive professional education and training for
professionals of all levels. www.garp.org.
10 | © 2014 Global Association of Risk Professionals. All rights reserved.
Model Risk Management for Non Banks
Michelle McCarthy
Managing Director, Risk Management
Nuveen Investments
Global Association of Risk Professionals
August 2014
2
The views expressed in the following material are the author’s and
do not necessarily represent the views of the Global Association of
Risk Professionals (GARP), its Membership or its Management.
Disclaimers
These materials reflect the views and opinions of the speaker and are
not intended to reflect the views, policies or practices of Nuveen
Investments or its affiliated companies. They are not intended to
provide investment advice and are for informational purposes only.
3 | © 2014 Global Association of Risk Professionals. All rights reserved.
Contents
Differences between models and their uses at banks vs. the buy side
What should be in scope for model risk management activity in the asset
management business?
Potential tiering of asset management models, and effect on model risk
management activities
4 | © 2014 Global Association of Risk Professionals. All rights reserved.
Most literature on model risk management has concerned banks
OCC 2000-16
OCC 2011-12, FRB SR 11-7
The case of AXA Rosenberg brought the conversation to the buy side, without
much guidance on how bank practices can and should differ from asset managers’
practices
5 | © 2014 Global Association of Risk Professionals. All rights reserved.
Different uses of models in asset managers vs. banks
Asset managers are not computing regulatory capital
Bank-owned asset managers may need to provide operational risk capital
calculations, but this is not in service of the asset management business
We rarely use valuation models as the sole source of valuation; we frequently
use external sources
Many asset management models are an input to a decision, but not the sole
driver of that decision
6 | © 2014 Global Association of Risk Professionals. All rights reserved.
Potential scope of model management activity for asset managers
6
Includes models that are used for investing or hedging
Drive investing activity to some extent
Value key assets
Does not include calculators; model must provide estimates and involve
uncertainty to require model risk management
Good change controls always an important operational control, but calibrations and
validations should be reserved for models that estimate uncertainty
Would not include compliance systems
Would not include price-to-yield calculations
Would not include simple screens for fundamental stock characteristics
Would not include pure index replication models without meaningful uncertainty
Does not include analyses that help develop an investment thesis—only the
tools used to execute the thesis, or to value assets once purchased
7 | © 2014 Global Association of Risk Professionals. All rights reserved.
Tiering for materiality
A potential tiering of asset management models:
Tier 1 Tier 2 Tier 3 Tier 4Models that:
• have high potential direct
impact on company results
• are broadly applied across the
entire company
• are understood and
maintained by a relatively
narrow range of experts at the
company
• Result in instant electronic
trading with little or no ability
to intervene, and/or affecting
a large portion of the portfolio
• are tightly coupled to trading but permit
human intervention, and/or affect a
small portion of the business
• result in trading with little other input
than direction from a model (quant
strategies)
• provide hedge ratios for using
derivatives, or to balance proportions
of fixed income instruments, in
portfolios
• value smaller, less significant balance
sheet items than those captured in Tier
1
• apply a screen to a universe of
stocks, using some measure of
uncertainty, such as a measurement
of the dispersion of some
fundamental characteristic (low price-
to-earnings ratios, for example)
• compare characteristics of securities
to assist the portfolio manager in
finding securities that he or she
deems desirable
• may support a certain
amount of decision
making, but with many
other inputs to these
decisions outside the
model.
• are outside the
company’s areas of key
business risk
• are understood by a
broad range of business
people
Such as:
• Statistical arbitrage model
• High frequency trading model
• A model that is rigorously
adhered to, is key to a broad
set of strategies and is
disclosed as a critical feature
of the investment process
• Significant valuation models
where third party sources are
unavailable
• A strategy that strongly adheres to
rebalancing if Barra or Yieldbook
measures exceed a threshold
• An economic indicator model that is
the basis of an investment strategy
• High frequency trading models
• Models used for delta hedging or asset
allocation rebalancing
• Minor valuation models where third
party sources are unavailable
• Using Barra or Yieldbook to see
where a portfolio is overweight, in
order to select trades that will balance
the portfolio better
• Economic indicator screens to
provide general investment signals
• Index replication models with tilts
• Risk oversight models
• Valuation models where third party
valuations are available
• Measures of portfolios
that are not used to
drive action
• Index replication
models that approach
100% replication
8 | © 2014 Global Association of Risk Professionals. All rights reserved.
Slotting typical asset management models into tiers
Tier I Tier II Tier III Tier IV
Investment Models
Fundamental
Business Models x x
Stock Selection Screen x x
Quantitative
Stock Selection x x
Portfolio Construction x x
Index Replication with Tilts x
Index Replication x
Statistical Arbitrage x x
High Frequency Trading x x
Leverage x
Hedging x
Economic Models
Growth x x
Inflation x x
Industry x x
Credit x x
Liquidity x x x
Risk Models
Capital Loss x
Liquidity x x
Ex Ante Tracking Error x x
Ex Post Tracking Error x
Duration, Beta, Factor Exposure Limits x x
Valuation Models
Third Party Pricing Services x x
Pricing Model - Third Party Developed x x x
Pricing Model - In-House Developed x x x
9 | © 2014 Global Association of Risk Professionals. All rights reserved.
What steps may make sense for model management of the
different tiers?
Inventory
For material models (Tiers 1 to 2)
Initial independent validation
─ Where independence may or may not require the different reporting lines set forth for
banks: effective challenge
Documentation (model theory, assumptions, limitations, code, user instructions)
Change validation
Periodic formal, documented calibration
Periodic re-validation given results of calibration
Model management for Tier 3 models could be lighter in terms of frequency,
intensity, formality of independence
The challenge of third party vendor models
C r e a t i n g a c u l t u r e o f
r i s k a w a r e n e s s ®
Global Association of
Risk Professionals
111 Town Square Place
14th Floor
Jersey City, New Jersey 07310
U.S.A.
+ 1 201.719.7210
2nd Floor
Bengal Wing
9A Devonshire Square
London, EC2M 4YN
U.K.
+ 44 (0) 20 7397 9630
www.garp.org
About GARP | The Global Association of Risk Professionals (GARP) is a not-for-profit global membership organization dedicated to preparing professionals and organizations to make
better informed risk decisions. Membership represents over 150,000 risk management practitioners and researchers from banks, investment management firms, government agencies,
academic institutions, and corporations from more than 195 countries and territories. GARP administers the Financial Risk Manager (FRM®) and the Energy Risk Professional (ERP®)
Exams; certifications recognized by risk professionals worldwide. GARP also helps advance the role of risk management via comprehensive professional education and training for
professionals of all levels. www.garp.org.
10 | © 2014 Global Association of Risk Professionals. All rights reserved.