Making Sense out of Metrics

ISDA / PRMIA

17th August 2004

Ope

ratio

nal R

isk

Man

agem

ent In 2002………

A global survey of 76 banks on the existence of formal KRI programs:

Acknowledgement Raft International Limited

Ope

ratio

nal R

isk

Man

agem

ent What is a KRI?

The number of fails has increased by 2%

Is this a KRI?

Ope

ratio

nal R

isk

Man

agem

ent Some definitions……

Metric – something observed or calculated that is used to show the presence or state of a condition or trend; an instrument or gauge that measures something and registers the measurement; something such as a light, sign, or pointer that gives information, for example about which direction to follow

KRI – Key Risk Indicator KPI – Key Performance Indicator KCI – Key Control Indicator KMI – Key Management Indicator

Overall, prefer the general term “indicator”

Ope

ratio

nal R

isk

Man

agem

ent What is a Risk Indicator?

Risk indicators are usually monitored over time

Late trade processing in Bank X in May 2004:

London 9%, New York 10%, Singapore 9% In which branch

do we have a

problem?

0%

5%

10%

15%

Feb Mar Apr May

Late Trade Processing

London

New York

Singapore

Ope

ratio

nal R

isk

Man

agem

ent What is a Risk Indicator?

So, Singapore is the problem! Is it?

0%

5%

10%

15%

Feb Mar Apr May

Late Trade Processing

London

New York

Singapore

0

500

1000

1500

2000

2500

3000

3500

Feb Mar Apr May

absolute numbers

Data without context may not expose the entire problem!

Ope

ratio

nal R

isk

Man

agem

ent The Choice from the Multitude

A typical operation can identify hundreds of indicators

Some are risk, others performance indicators

Indicators’ relevance and weight change over time

Some indicators are meaningless on their own

Graphics adapted from Reason, J.: “Managing the Risks of Organizational Accidents", Aldershot: Ashgate, 1997

Ope

ratio

nal R

isk

Man

agem

ent Other factors

If a ratings agency is to rate operational exposure, how will they compare different organisations?

How will regulators evaluate the effectiveness of different organisation’s operational risk capabilities?

How does a business unit provide senior management quality information?

Can the organisation use operational metrics to provide stakeholder information?

Ope

ratio

nal R

isk

Man

agem

ent But, KRIs have been disappointing…

No means of consistently relate the occurrence of loss events and the location of problems/situations

No means of classifying types of KRIs Plenty of data but no idea of its relevance No way to determine relevance No observable best practice No means of comparison, either internally

or externally

Ope

ratio

nal R

isk

Man

agem

ent The solution?

An organisation needs a common language or framework which identifies areas of exposure and then allows…..– Metrics to be identified to measure, monitor

and manage those exposures– Data on losses, near misses and control

failures to be recorded– Ongoing assessment of the exposure– Performance measurement around the

exposure, including use of capital

Ope

ratio

nal R

isk

Man

agem

ent KRI Study - Background

Recognising the need, RiskBusiness developed a strawman framework for identifying risk points within an institution

RMA and RiskBusiness co-sponsored Part I of the Study to ascertain the feasibility of using the framework as a KRI framework

Seven institutions tested the framework in a risk mapping exercise

Ope

ratio

nal R

isk

Man

agem

ent Initial Participants

7 banks participated in Part I:– Citigroup– Deutsche Bank– Dresdner Kleinwort Wasserstein– JP Morgan Chase– KeyCorp– Royal Bank of Canada– State Street

ANZ and Abbey then joined to form the Study Steering Group for Part II

Ope

ratio

nal R

isk

Man

agem

ent The Study

Part I – Proof of Concept Part II – 3 primary activities :

– Broaden participation, transfer experience, build industry risk profile

– Define KRI Library– Develop detailed specifications

Future :– Industry benchmarking– Extend participation further

Ope

ratio

nal R

isk

Man

agem

ent The KRI Framework

Ope

ratio

nal R

isk

Man

agem

ent Validation

Initial risk maps were evaluated to establish “most risky” risk categories and business functions

Compared these to a similar evaluation of the QIS 3 data and to the complete Fitch Risk First database

Broad correlation, taking into account the nature of the two data sources

Ope

ratio

nal R

isk

Man

agem

ent Participant risk map deviations

Based on risk category:

RMA/RiskBusiness Risk Categories

Stage 1 Rankings (additive

High ratings)

Participant A

Participant B

Participant C

Participant D

Participant E

Participant F

Participant G

Risk Business

Data Management 1 2 5 3 2 1 2 2 3 Improper Practices 2 1 1 1 1 4 1 7 2 Infrastructure & Systems 3 5 7 2 3 2 3 5 1 Transaction Management 4 7 6 6 6 6 5 1 4 Internal Theft & Fraud 5 3 2 5 5 3 4 9 5

Ope

ratio

nal R

isk

Man

agem

ent Participant risk map deviations

Based on business function:RMA/RiskBusiness Business

Functions

Stage 1 Rankings (additive

High ratings)

Participant A

Participant B

Participant C

Participant D

Participant E

Participant F

Participant G

Risk Business

Payment/Settlement/Collection (cash/securities)

1 2 3 1 1 1 3 3 2

Instruction or Order Management 2 1 1 3 2 1 1 - 1 Custody and Actions (including assets)

3 6 2 4 3 3 8 - 3

Transaction/Fees Capture and Record Update

4 21 23 2 5 5 5 2 11

Relationship Management 5 3 4 19 11 14 18 1 14 Confirm/Affirm/Matching and Documentation

6 5 9 19 4 4 2 - 4

Transaction Maintenance and Administration

7 21 7 13 13 10 5 5 15

Infrastructure, Networks & Maintenance

8 3 36 21 6 7 7 - 8

Pricing and Quotations 9 13 13 28 9 7 3 - 8 IT Security 10 10 5 14 5 13 9 - 12

Ope

ratio

nal R

isk

Man

agem

ent Participants in Part II

Abbey† ABN Amro ABSA Acleda Bank Alliance and Leicester ANZ† Bank Austria Creditanstaldt Bank Julius Bäer Bank of America Bank Rakvat Indonesia Bank Vontobel BNP Paribas Byblos Bank Capital One Citigroup† Commerzbank De Lage Landen (sub of Rabobank) Deutsche Bank† Dresdner Kleinwort Wasserstein† Erste Bank Euroclear Federation de Caisses Desjardins du Quebec GMAC Halifax Bank of Scotland HSBC

Huntington National Bank Investec JP Morgan Chase† KeyCorp† Kookmin Bank Macquarie Bank Mizuho International National Australia Group National Bank of Canada Nomura International Northern Trust People’s Bank Royal Bank of Canada† Royal Bank of Scotland RZB San Paolo IMI SE Banken Southwest Bank of Texas Standard Bank of South Africa State Street† Sumitomo Mitsui TD Financial Group Woori Bank Washington Mutual

† = Lead Participant

Ope

ratio

nal R

isk

Man

agem

ent Defining Indicators

Identify candidates for each risk point Evaluate candidates against qualifying

criteria (effectiveness, comparability, ease of use/collection)

Agree descriptions for each qualifier Prioritise nominated indicators Participant comment and review Generate detailed specifications, stored

in KRI Library at www.KRIeX.org

Ope

ratio

nal R

isk

Man

agem

ent Considering each risk point…..

Research ‘Historical Events’

Identify ‘Problems in Progress’

Look for ‘Early Warning Flags’

Near misses, loss = 0

0 < Losses < threshold

Potential loss events, include events that could lead this or another institution to loss

After the problem has been generated

Before the problem has been generated

Losses > threshold

Ope

ratio

nal R

isk

Man

agem

ent Issues for consideration

What is a KRI...KPI…KCI…or MIS?– An indicator can perform multiple roles

depending on who is using it What about scaling and aggregation?

– Do we scale then aggregate, or vice versa? How many indicators should a firm be

monitoring?– The quest for the “Magic 10” – the KRI

Library has 1,600 indicators

Ope

ratio

nal R

isk

Man

agem

ent Issues for consideration

Top-down versus Bottom-Up– Operations develop metrics for ongoing

use, Management want information Combinations and clusters of indicators

– Experience has demonstrated that in many cases, it is groups of indicators which will provide the best management information

– Staff Turnover and Transaction Volumes and Error Rates and ……

Ope

ratio

nal R

isk

Man

agem

ent Next Steps - Benchmarking

Selected indicators will be driven totally by broad participant agreement

Consists of participants delivering KRI data to centralised function :– Data will be anonymous– Data will be collated, analysed and benchmark

values calculated

Participants have access to benchmarks for comparative purposes

First submission expected in Q2, 2005

Ope

ratio

nal R

isk

Man

agem

ent Next Steps – KRI Library

Next intake of participants into KRI Library starts in October 2004 – currently have 68 additional firms wishing to join

Insurance KRI Study starts during Q4 2004

Ongoing maintenance and extension to the Library

Ope

ratio

nal R

isk

Man

agem

ent In Summary…..

A well-developed and structured indicator program can deliver quality management information and could possibly be used as an adjustor to capital…or at least as a measure of efficient and effective use

Common language and standardisation is imperative

The indicator program must deliver value at all levels

Ope

ratio

nal R

isk

Man

agem

ent Contact details

RiskBusiness International Limited– URL : www.riskbusiness.com– Study URL : www.kriex.org

Mike Finlay, Managing Director – Europe, Asia, Australia and Africa– Telephone : +44 7721 969 224 – E-mail : mike.finlay@riskbusiness.com