Solvency II model validation workshop 5

12 & 13 June 2012

© Lloyd’s

Agenda

Introduction

Five ways to improve your validation

Dry Run findings for major risk types

Lloyd’s review under BAU

Break-out – table discussions

Next steps

© Lloyd’s

Introduction

© Lloyd’s

Model validation workstream

solvency II syndicate timeline - key dates for 2012

January february march april may june july august september october November

IMSCRInternal Model

SCR

MVModel

Validation

31 May

Model Validation

Guidance

Data Audit Reviews

5 October

Validation Report

30 April

2013 SCR

Guidance

19 July

Draft SCR via LCR

(incl projected TPs)

20 September

Final SCR via LCR

(incl projected TPs)

8 & 11 May

Workshop

15 June

Data

Audit

Report13 & 18 April

Workshop

12 & 13

JuneWorkshop

Validation walkthrough follow up

& CAT MODEL REVIEW WORK

Workshop/Briefing Agent Deliverable - Dry Run / ProjectLloyd's PublicationKey shaded boxes Thematic ReviewAgent Deliverable - BAU

18 January

Cat/External models

Briefing

10 February

Draft Data Audit

Report Guidance

30 March

Final Data Audit

Report Guidance

31 July (PROV'L)

Standard Formula

SCR Re-run @ 31/12/11

You are here.

Please read the Model

Validation Guidance.

© Lloyd’s

five ways to improve your validation

© Lloyd’s

Top 5 ways to improve your validation

1. Evaluate your risk coverage

2. Rank the risks

3. Link to the syndicate experience

4. Clarify the steps in the validation cycle

5. Structure the tests

© Lloyd’s

Risk coverage & non/partially modelled risks

Solvency II requires all material risks to be covered by the model.

Easier said than done.

Tsunamis, floods, latent claims, tail risk not captured in the history,

dependencies between prior years…how well are these captured by your

model?

Begin with a qualitative assessment of what’s in and what’s out.

Data may be lacking and methods may be approximate – but the risk still

needs to be captured.

© Lloyd’s

Rank the risks

What are the biggest threats to syndicate solvency?

The most common reasons for threats to the ICA or RBC since 2003:

Reserve deteriorations on multiple years of account in a casualty portfolio

Severe natural catastrophes with significant losses from non-modelled

perils

At a more granular level, risk ranking:

Demonstrates an understanding of the risks

Allocates validation resource more effectively

• Rank risks using business definitions rather than statistical/actuarial ones.

© Lloyd’s

Clarify the steps in the validation cycle

Recall the four steps to the validation cycle:

1. Apply the validation test

2. Analyse the results

3. Escalate test results (if required)

4. Implement changes to the internal model (if required)

Common findings from the Dry Run on steps 2 and 3:

No clear statement of the result

Lack of justification or assessment of limitations

Lack of clarity around escalation process

Test results will rarely be clear-cut – they need to be explained.

© Lloyd’s

Link to the syndicate experience

The claims history is never sufficient but always necessary.

The model is an abstract exercise if there is no link to the past.

Use expert judgement to explain the impact of data limitations.

GAAP vs. economic basis

Best estimate vs. conservative reserving

Business no longer written, etc.

Consider tail risk not reflected in the experience – don’t just exclude poor

results.

The history also serves as a reference point for management.

© Lloyd’s

Structure the tests

Many agents have relied primarily on Sensitivity testing – Type I (“ST1”).

Across the board up/down movement in parameter type (e.g. CoVs)

Outcome: “Pass” if change in SCR is within a pre-defined range

Limitations:

It’s deterministic – so there’s no sense of the uncertainty in the input

Sensitivity within a pre-defined range doesn’t mean it’s the right value

ST1 has its advantages but won’t be sufficient on its own.

Validation of more material risks will need to build on ST1 with other tests.

© Lloyd’s

Validation tests: suggested hierarchy In

cre

asin

g r

isk m

ate

rialit

y

Reverse stress testing

P&L attribution

Stress & scenario Testing

Sensitivity testing (Type 2)

Testing against experience

Sensitivity testing (Type I)

Qualitative validation

© Lloyd’s

Dry Run findings for major risk types

© Lloyd’s

Reserving risk

© Lloyd’s

The validation elephant

Multi-year reserve deteriorations have been one of the leading threats to

ICA/RBC at Lloyd’s.

Does your reserve risk methodology reflect the risk of multi-year

deteriorations?

First, a brief detour down bootstrap lane…

© Lloyd’s

The bootstrap: last step in the algorithm*

Simulation No: XXXX

Simulated

AY/DY 1 2 3 4 5 Reserves

2007

2008 85 85

2009 282 35 317

2010 301 487 70 858 Result:

2011 211 341 403 66 1021 thin tailed distributions

TOTAL 2281

...etc.

Bootstrap AY/DY distribution parameters

Distributions sampled from independently

* England, P.D. (2001). Addendum to 'Analytic and bootstrap estimates of prediction errors in claims reserving.'

© Lloyd’s

Are statistical methods enough?

Reasons why most stochastic reserving methods (not just the bootstrap)

assume independence between accident years:

Mathematical tractability

Realism

The few that do will still require you to take a view on the calendar year

“trend”.

Back to validation…How material is the assumption of independence?

Two validation tools for dependencies (CEIOPS DOC 48/09 (5.245)):

Expert judgement of causal relationships

Statistical analysis

© Lloyd’s

Elephant sightings

The support from expert judgement for dependencies between accident years

will be very strong for some CoB (mostly long-tailed casualty):

Similar or same risks

Underwriting cycle, inflation, legal environment, etc.

Quantitative evidence…discussed in previous workshops.

Calendar Residuals

-2.0

-1.5

-1.0

-0.5

0.0

0.5

1.0

1.5

2.0

2.5

2002 2003 2004 2005 2006 2007 2008 2009

Calendar Year

© Lloyd’s

Avoid getting stomped on

Your validation process should include:

Criteria for the existence of accident year dependencies

An assessment of their materiality

An approach for incorporating them in the internal model (where required)

The above need not (should not ?) rely on complex statistical methods.

Lloyd’s encourages agents to build on work done in the best estimate

reserving process:

Actual vs. best estimate comparisons

Knowledge of large claim deteriorations

Legal and other risks…

© Lloyd’s

dependencies

© Lloyd’s

So many correlations, so few dependencies

Why are the following often seen in the same model?

Hundreds of correlations

Diversification benefits exceeding 40%

How would your validation process respond?

“Dependencies are very hard to estimate and validate.” CEIOPS DOC

48/09 (5.233)

“Wounds from correlation matrices are still sore.” Nassim Nicholas

Taleb

© Lloyd’s

Seeing the forest through the trees

Broadly speaking…two ways of validating dependencies:

Both types of validation have their uses and limitations – neither is a magic

bullet

Validation

component Bottom-up Top-down

Risk distributions Pairs of marginals Aggregate

Inputs/Outputs Inputs Outputs

Sample metrics Correlations Joint exceedance

probabilities

Key validation tool Sensitivity testing Scenarios

Premium risk example:

Marine Hull & Cargo

Sensitivity test

correlation between

Hull & Cargo; observe

impact on model

output

Scenario based on

high loss ratios for key

accounts; compare to

model output for Hull &

Cargo combined

© Lloyd’s

Distressed…or just sensitive?

Most agents have emphasised “bottom-up” validation based on sensitivity

testing of correlations. Limitations of this approach:

Tests the sensitivities – not the level of dependency itself

Alternative dependency structures not evaluated

Ignores other drivers – e.g. inflation

In top-down validation, management takes a view of the material

“dependencies” that could impair profits or bankrupt the syndicate, e.g.

Multi-year reserve deterioration on US casualty

Several severe cats in the current YoA

(Very) poor underwriting results on several classes

Combinations of the above, etc.

These are then compared to model outputs.

© Lloyd’s

How to diversify away tail risk

Many agents did not undertake this high level validation of outputs.

Granular risk distributions

+ non-tail dependent dependency structures

= diversification in the tail

This will be a key area of interest for Lloyd’s (and the FSA) in BAU.

This works both ways. If you have “dependencies” in your model that have

limited/no justification, take them out. You’re just slowing down your model.

© Lloyd’s

Other risk types

© Lloyd’s

Summary of issues on other risk types

See the draft Validation Guidance for more detailed discussion.

See Appendix I for sample validation tests.

Risk type / area Key issues / remarks

Premium ex. Cat

• Perils covered (e.g. non-modelled cat)

• Validation of data adjustments

• The underwriting cycle

Catastrophe • Draft LMA guidance – awaiting comment from FSA

• Non/partially modelled perils remains a key issue

Market • Obtaining comfort on ESG

Credit • Materiality of different drivers

Operational • Comprehensive and realistic scenarios

One year risk • Consistency with ultimate

© Lloyd’s

BAU 2012

© Lloyd’s

Lloyd’s “Lines of defence”

Recap from May IMSCR workshop:

Line Description Tests Outcome

1 LCR data validation • Defective values

• Unintended input errors

LCR that agent can “sign-

off” on

2 LCR benchmarking

• Risk vs. reward and

other comparisons

• By SCR/main risks

Identification of areas of

focus for Line 4

3 SCR document review

• Methodology &

assumption descriptions

• Review of required

quantitative information

Completed SCR document

review template

4 Model walkthrough

• Focus on issues

identified in Lines 1-3

• Similar format to Dry

Run walkthroughs

Revisions to model if

required

5 Validation report review • Review against VR

guidance requirements

Completed/updated VR

review template

© Lloyd’s

Line 2: LCR benchmarking

A tool for generating questions – not a pass/fail test or a Lloyd’s “Standard

Formula”.

DRAFT example: risk vs. reward. Consistent with S2 concept of market

valuation.

(*) Illustrative applications of risk-reward. Not for LCR review.

Plus other approaches: reserving risk vs. best estimate reserves, etc.

Risk type Risk metric Reward

Market (*) Standard deviation of

returns

Expected return above

risk-free rate

Reinsurance (*) Probability capital falls

below £XXm Net premium

Premium Ultimate 99.5th VaR Profit loading in premium

Reserving Ultimate 99.5th VaR Risk margin

Capital requirement One year 99.5th VaR Expected one year profit

© Lloyd’s

© Lloyd’s

© Lloyd’s

Line 4: Model walkthroughs

The feedback from the model walkthroughs identified “material” issues for

most agents.

Agents are not expected to re-submit before 30 June (unless specifically

requested to do so by Lloyd’s).

Lloyd’s does expect agents to make progress on the issues identified – and

will follow up in the next round of walkthroughs.

Also expect questions on issues identified in Lines 1-3.

© Lloyd’s

Line 5: Validation report review

Update for October 2012 submission: A statement confirming that the SCR is

“not materially mis-stated” is not – repeat, NOT – required.

A statement that the model is appropriate WILL be required

Lloyd’s will re-visit when Level 3 text finalised

This update does not alter the validation requirements.

© Lloyd’s

table discussions

© Lloyd’s

Suggested topics

The proposed Lloyd’s LCR review structure:

What weight should be given to the different “lines of defense”?

Is the burden on syndicates reasonable?

Benchmarking:

What are the appropriate metrics?

How should they be used?

The most challenging area of validation for your syndicate:

The underwriting cycle

Dependencies between prior years

Non-modelled perils

Other

The draft Model Validation guidance: comments, suggestions, questions.

© Lloyd’s

next steps

© Lloyd’s

What happens next?

Slides will be available on lloyds.com after the 13th.

Draft Model Validation guidance – if you didn’t read it over the long weekend,

please do so and provide feedback before 15 June.

LCR and SCR documentation is due19 July (on initial SBF).

And…a mix of Olympics and SCR reviews.