Research Matters
February 25, 2009
Nick HighamDirector of Research
School of Mathematics
1 / 6
Rehabilitating Correlations, AvoidingInversion, and Extracting Roots
Nick HighamSchool of Mathematics
The University of Manchester
[email protected]@nhigham, nickhigham.wordpress.com
http://www.maths.manchester.ac.uk/~higham/talks
The Actuarial ProfessionMarch 20, 2013, London
http://www.ma.man.ac.uk/~higham/http://www.ma.man.ac.ukhttp://www.man.ac.ukmailto:[email protected]://twitter.com/nhighamhttp://nickhigham.wordpress.comhttp://www.maths.manchester.ac.uk/~higham/talks
Precision
SingleDoubleQuadVariable
AccuracyEstimatesGuarantees
Efficiency
SpeedParallelismEnergy efficiency
Correlations Matrix Inversion Matrix Roots
Outline
1 Rehabilitating Correlations
2 Matrix Inversion
3 Matrix Roots
University of Manchester Nick Higham Actuarial Matrix Computations 3 / 37
Correlations Matrix Inversion Matrix Roots
Correlation Matrix
An n × n symmetric matrix A is a correlation matrix ifIt has ones on the diagonal.All its eigenvalues are nonnegative.
Is this a correlation matrix?1 1 01 1 10 1 1
.
Spectrum: −0.4142, 1.0000, 2.4142.
University of Manchester Nick Higham Actuarial Matrix Computations 4 / 37
Correlations Matrix Inversion Matrix Roots
Correlation Matrix
An n × n symmetric matrix A is a correlation matrix ifIt has ones on the diagonal.All its eigenvalues are nonnegative.
Is this a correlation matrix?1 1 01 1 10 1 1
.
Spectrum: −0.4142, 1.0000, 2.4142.
University of Manchester Nick Higham Actuarial Matrix Computations 4 / 37
Correlations Matrix Inversion Matrix Roots
Correlation Matrix
An n × n symmetric matrix A is a correlation matrix ifIt has ones on the diagonal.All its eigenvalues are nonnegative.
Is this a correlation matrix?1 1 01 1 10 1 1
. Spectrum: −0.4142, 1.0000, 2.4142.
University of Manchester Nick Higham Actuarial Matrix Computations 4 / 37
Question from London Fund Management Company (2000)“Given a real symmetric matrix A which is almost acorrelation matrix . . .
What is the best approximating (in Frobenius norm?)correlation matrix?Is it unique?Can we compute it?
Typically we are working with 1400× 1400 at the moment,but this will probably grow to 6500× 6500.”
Correlations Matrix Inversion Matrix Roots
How to Proceed
× Make ad hoc modifications to matrix: e.g., shiftnegative e’vals up to zero then diagonally scale.
√Plug the gaps in the missing data, then compute anexact correlation matrix.
√Compute the nearest correlation matrix in theweighted Frobenius norm (‖A‖2 =
∑i,j wiwja
2ij ).
Given approx correlation matrix A find correlationmatrix C to minimize ‖A− C‖.
Constraint set is a closed, convex set, so uniqueminimizer.
University of Manchester Nick Higham Actuarial Matrix Computations 6 / 37
Correlations Matrix Inversion Matrix Roots
Development
Derived theory and algorithm:N. J. Higham, Computing the Nearest CorrelationMatrix—A Problem from Finance, IMA J. Numer.Anal. 22, 329–343, 2002.
Extensions in:Craig Lucas, Computing Nearest Covariance andCorrelation Matrices, M.Sc. Thesis, University ofManchester, 2001.
University of Manchester Nick Higham Actuarial Matrix Computations 7 / 37
Correlations Matrix Inversion Matrix Roots
Alternating Projections Algorithmvon Neumann (1933): for subspaces.Dykstra (1983): corrections for closed convex sets.
S1
S2
Easy to implement.Guaranteed convergence, at a linear rate.Can add further constraints/projections.
University of Manchester Nick Higham Actuarial Matrix Computations 8 / 37
Correlations Matrix Inversion Matrix Roots
Unexpected Applications
Some recent papers:
Applying stochastic small-scale damage functionsto German winter storms (2012)
Estimating variance components and predictingbreeding values for eventing disciplines andgrades in sport horses (2012)
Characterisation of tool marks on cartridge casesby combining multiple images (2012)
Experiments in reconstructing twentieth-centurysea levels (2011)
University of Manchester Nick Higham Actuarial Matrix Computations 9 / 37
http://onlinelibrary.wiley.com/doi/10.1029/2012GL050961/abstracthttp://onlinelibrary.wiley.com/doi/10.1029/2012GL050961/abstracthttp://journals.cambridge.org/action/displayAbstract?fromPage=online&aid=8646224http://journals.cambridge.org/action/displayAbstract?fromPage=online&aid=8646224http://journals.cambridge.org/action/displayAbstract?fromPage=online&aid=8646224http://www.sciencedirect.com/science/article/pii/S0079661111000759http://www.sciencedirect.com/science/article/pii/S0079661111000759
Correlations Matrix Inversion Matrix Roots
Newton Method
Qi & Sun (2006): Newton method based on theory ofstrongly semismooth matrix functions.
Applies Newton to dual (unconstrained) ofmin 12‖A− X‖
2F problem.
Globally and quadratically convergent.
H & Borsdorf (2010) improve efficiency and reliability:
use minres for Newton equation,Jacobi preconditioner,reliability improved by line search tweaks,extra scaling step to ensure unit diagonal.
University of Manchester Nick Higham Actuarial Matrix Computations 11 / 37
Correlations Matrix Inversion Matrix Roots
Alternating Projections vs Newton
Matrix n tol Code Time (s) Iters1. Random 100 1e-10 g02aa 0.023 4
alt proj 0.052 152. Random 500 1e-10 g02aa 0.48 4
alt proj 3.01 263. Real-life 1399 1e-4 g02aa 6.8 5
alt proj 100.6 68
University of Manchester Nick Higham Actuarial Matrix Computations 14 / 37
Correlations Matrix Inversion Matrix Roots
Performance of NAG Codes
University of Manchester Nick Higham Actuarial Matrix Computations 15 / 37
Correlations Matrix Inversion Matrix Roots
Factor Model (1)
ξ = X︸︷︷︸n×k
η︸︷︷︸k×1
+ F︸︷︷︸n×n
ε︸︷︷︸n×1
, ηi , εi ∈ N(0,1),
where var(ξi) ≡ 1, F = diag(fii). Implies
k∑j=1
x2ij ≤ 1, i = 1 : n.
“Multifactor normal copula model”.Collateralized debt obligations (CDOs).Multivariate time series.
University of Manchester Nick Higham Actuarial Matrix Computations 16 / 37
Correlations Matrix Inversion Matrix Roots
Factor Model (2)
Yields correlation matrix of form
C(X ) = D + XX T = D +k∑
j=1
xjxTj ,
D = diag(I − XX T ), X = [x1, . . . , xk ].
C(X ) has k factor correlation matrix structure.
C(X ) =
1 yT1 y2 . . . y
T1 yn
yT1 y2 1 . . ....
... . . . yTn−1ynyT1 yn . . . y
Tn−1yn 1
, yi ∈ Rk .University of Manchester Nick Higham Actuarial Matrix Computations 17 / 37
Correlations Matrix Inversion Matrix Roots
Factor Structure
Nearest correlation matrix with factor structure.
Principal factors method (Andersen et al., 2003) has noconvergence theory and can converge to an incorrectanswer.
Algorithm based on spectral projected gradientmethod (Borsdorf, H & Raydan, 2010).
Respects the constraints, exploits their convexity,and converges to a feasible stationary point.NAG routine g02aef (Mark 23, 2012).
University of Manchester Nick Higham Actuarial Matrix Computations 18 / 37
Correlations Matrix Inversion Matrix Roots
Factor Structure
Nearest correlation matrix with factor structure.
Principal factors method (Andersen et al., 2003) has noconvergence theory and can converge to an incorrectanswer.Algorithm based on spectral projected gradientmethod (Borsdorf, H & Raydan, 2010).
Respects the constraints, exploits their convexity,and converges to a feasible stationary point.NAG routine g02aef (Mark 23, 2012).
University of Manchester Nick Higham Actuarial Matrix Computations 18 / 37
Correlations Matrix Inversion Matrix Roots
Variations
Specific rank or bound on correlations.
Block structure.
Bounds on individual correlations.
Other requirements?
University of Manchester Nick Higham Actuarial Matrix Computations 19 / 37
Correlations Matrix Inversion Matrix Roots
Outline
1 Rehabilitating Correlations
2 Matrix Inversion
3 Matrix Roots
University of Manchester Nick Higham Actuarial Matrix Computations 20 / 37
Correlations Matrix Inversion Matrix Roots
Avoiding Inversion
Fundamental Tenet of Numerical AnalysisDon’t invert matrices.
Ax = b : use Gaussian elimination (with pivoting), notx = A−1b.xT A−1y = xT (A−1y).
(A−1)ii = eTi A−1ei .
trace(A−1) ≈ m−1∑m
k=1 vTk A
−1vk ,vk ∼ uniform{−1,1}. (Bekas et al., 2007)
University of Manchester Nick Higham Actuarial Matrix Computations 21 / 37
Correlations Matrix Inversion Matrix Roots
Avoiding Inversion
Fundamental Tenet of Numerical AnalysisDon’t invert matrices.
Ax = b : use Gaussian elimination (with pivoting), notx = A−1b.xT A−1y = xT (A−1y).
(A−1)ii = eTi A−1ei .
trace(A−1) ≈ m−1∑m
k=1 vTk A
−1vk ,vk ∼ uniform{−1,1}. (Bekas et al., 2007)
University of Manchester Nick Higham Actuarial Matrix Computations 21 / 37
Correlations Matrix Inversion Matrix Roots
How to Invert (1)
Use a condition estimator to warn when A is nearlysingular.
>> A = hilb(16);>> X = inv(A);Warning: Matrix is close to singular orbadly scaled. Results may be inaccurate.RCOND = 9.721674e-19.
University of Manchester Nick Higham Actuarial Matrix Computations 22 / 37
Correlations Matrix Inversion Matrix Roots
How to Invert (2)
To compute variance–covariance matrix (X T X )−1 ofleast squares estimator, use QR factorization X = QR((X T X )−1 = R−1R−T ) and do not explicitly form X T X .
Quality of computed inverse Ŷ ≈ A−1 measured by
‖ŶA− I‖‖Ŷ‖‖A‖
or‖AŶ − I‖‖Ŷ‖‖A‖
but not both.
University of Manchester Nick Higham Actuarial Matrix Computations 23 / 37
Correlations Matrix Inversion Matrix Roots
Computing the Sample Variance
Sample variance of x1, . . . , xn:
s2n =1
n − 1
n∑i=1
(xi − x̄)2, where x̄ =1n
n∑i=1
xi . (1)
Can compute using one-pass formula:
s2n =1
n − 1
( n∑i=1
x2i −1n
( n∑i=1
xi
)2). (2)
For x = (10000,10001,10002) using 8-digit arithmetic,(1) gives: 1.0, (2) gives 0.0.
(2) can even give negative results in floating pointarithmetic!
University of Manchester Nick Higham Actuarial Matrix Computations 24 / 37
Correlations Matrix Inversion Matrix Roots
Computing the Sample Variance
Sample variance of x1, . . . , xn:
s2n =1
n − 1
n∑i=1
(xi − x̄)2, where x̄ =1n
n∑i=1
xi . (1)
Can compute using one-pass formula:
s2n =1
n − 1
( n∑i=1
x2i −1n
( n∑i=1
xi
)2). (2)
For x = (10000,10001,10002) using 8-digit arithmetic,(1) gives: 1.0, (2) gives 0.0.
(2) can even give negative results in floating pointarithmetic!
University of Manchester Nick Higham Actuarial Matrix Computations 24 / 37
Correlations Matrix Inversion Matrix Roots
Casio fx-992VB
University of Manchester Nick Higham Actuarial Matrix Computations 25 / 37
Correlations Matrix Inversion Matrix Roots
Spreadsheets in the CloudStandard deviation of x = [n, n + 1, n + 2]T .
n Exact Google Sheet107 1 1108 1 0
McCullough & Yalta (2013)
University of Manchester Nick Higham Actuarial Matrix Computations 26 / 37
Correlations Matrix Inversion Matrix Roots
Spreadsheets in the CloudStandard deviation of x = [n, n + 1, n + 2]T .
n Exact Google Sheet107 1 1108 1 0
McCullough & Yalta (2013)
University of Manchester Nick Higham Actuarial Matrix Computations 26 / 37
Correlations Matrix Inversion Matrix Roots
Spreadsheets in the CloudStandard deviation of x = [n, n + 1, n + 2]T .
n Exact Google Sheet107 1 1108 1 0
McCullough & Yalta (2013)
University of Manchester Nick Higham Actuarial Matrix Computations 26 / 37
Correlations Matrix Inversion Matrix Roots
Outline
1 Rehabilitating Correlations
2 Matrix Inversion
3 Matrix Roots
University of Manchester Nick Higham Actuarial Matrix Computations 27 / 37
Correlations Matrix Inversion Matrix Roots
Cayley and Sylvester
Term “matrix” coined in 1850by James Joseph Sylvester,FRS (1814–1897).
Matrix algebra developed byArthur Cayley, FRS (1821–1895).Memoir on the Theory of Ma-trices (1858).
University of Manchester Nick Higham Actuarial Matrix Computations 28 / 37
Correlations Matrix Inversion Matrix Roots
Cayley and Sylvester on Matrix Functions
Cayley considered matrix squareroots in his 1858 memoir.
Tony Crilly, Arthur Cayley: Mathemati-cian Laureate of the Victorian Age,2006.
Sylvester (1883) gave first defini-tion of f (A) for general f .
Karen Hunger Parshall, James JosephSylvester. Jewish Mathematician in aVictorian World, 2006.
.University of Manchester Nick Higham Actuarial Matrix Computations 29 / 37
Correlations Matrix Inversion Matrix Roots
Matrix Roots in Markov Models
Let vectors v2011, v2010 represent risks, credit ratings orstock prices in 2011 and 2010.Assume a Markov model v2011 = Pv2010, where P is atransition probability matrix.P1/2 enables predictions to be made at 6-monthlyintervals.
P1/2 is matrix X such that X 2 = P. What are P2/3, P0.9?
Ps = exp(s log P) .
Problem: log P, P1/k may have wrong sign patterns⇒“regularize”.
University of Manchester Nick Higham Actuarial Matrix Computations 30 / 37
Correlations Matrix Inversion Matrix Roots
Matrix Roots in Markov Models
Let vectors v2011, v2010 represent risks, credit ratings orstock prices in 2011 and 2010.Assume a Markov model v2011 = Pv2010, where P is atransition probability matrix.P1/2 enables predictions to be made at 6-monthlyintervals.
P1/2 is matrix X such that X 2 = P. What are P2/3, P0.9?
Ps = exp(s log P) .
Problem: log P, P1/k may have wrong sign patterns⇒“regularize”.
University of Manchester Nick Higham Actuarial Matrix Computations 30 / 37
Correlations Matrix Inversion Matrix Roots
Chronic Disease Example
Estimated 6-month transition matrix.Four AIDS-free states and 1 AIDS state.2077 observations (Charitos et al., 2008).
P =
0.8149 0.0738 0.0586 0.0407 0.01200.5622 0.1752 0.1314 0.1169 0.01430.3606 0.1860 0.1521 0.2198 0.08150.1676 0.0636 0.1444 0.4652 0.1592
0 0 0 0 1
.Want to estimate the 1-month transition matrix.
Λ(P) = {1,0.9644,0.4980,0.1493,−0.0043}.
H & Lin (2011).Lin (2011, for survey of regularization methods.
University of Manchester Nick Higham Actuarial Matrix Computations 31 / 37
Correlations Matrix Inversion Matrix Roots
MATLAB: Arbitrary Powers
>> A = [1 1e-8; 0 1]A =1.0000e+000 1.0000e-008
0 1.0000e+000
>> A^0.1ans =
1 00 1
>> expm(0.1*logm(A))ans =
1.0000e+000 1.0000e-0090 1.0000e+000
University of Manchester Nick Higham Actuarial Matrix Computations 32 / 37
Correlations Matrix Inversion Matrix Roots
MATLAB Arbitrary Power
New Schur algorithm (H & Lin, 2011, 2013) reliablycomputes Ap for any real p.New backward-error based inverse scaling andsquaring alg for matrix logarithm (Al-Mohy, H &Relton, 2012)—faster and more accurate.Alternative Newton-based algorithms available for A1/q
with q an integer, e.g., for
Xk+1 =1q[(q + 1)Xk − X q+1k A
], X0 = A,
Xk → A−1/q.
University of Manchester Nick Higham Actuarial Matrix Computations 33 / 37
Correlations Matrix Inversion Matrix Roots
Knowledge Transfer Partnership
University of Manchester and NAG (2010–2013)funded by EPSRC, NAG and TSB.
Developing suite of NAG Library codes for matrixfunctions.
Extensive set of new codes included in Mark 23 (2012),Mark 24 (2013).
Improvements to existing state of the art: faster andmore accurate.
My work also supported by ¤2M ERC Advanced Grant.
University of Manchester Nick Higham Actuarial Matrix Computations 34 / 37
Correlations Matrix Inversion Matrix Roots
Final Remarks
Fast moving developments in numerical linear algebraalgorithms.
Numerical reliability is essential.
Partnership with NAG enables rapid inclusion of ouralgorithms in the NAG Library.
Keen to hear about your matrix problems.
University of Manchester Nick Higham Actuarial Matrix Computations 37 / 37
References I
A. H. Al-Mohy and N. J. Higham.Improved inverse scaling and squaring algorithms forthe matrix logarithm.SIAM J. Sci. Comput., 34(4):C153–C169, 2012.
A. H. Al-Mohy, N. J. Higham, and S. D. Relton.Computing the Fréchet derivative of the matrix logarithmand estimating the condition number.MIMS EPrint 2012.72, Manchester Institute forMathematical Sciences, The University of Manchester,UK, July 2012.17 pp.Revised December 2012.
University of Manchester Nick Higham Actuarial Matrix Computations 1 / 8
References II
L. Anderson, J. Sidenius, and S. Basu.All your hedges in one basket.Risk, pages 67–72, Nov. 2003.|www.risk.net|.
C. Bekas, E. Kokiopoulou, and Y. Saad.An estimator for the diagonal of a matrix.Appl. Numer. Math., 57:1214–1229, 2007.
R. Borsdorf, N. J. Higham, and M. Raydan.Computing a nearest correlation matrix with factorstructure.SIAM J. Matrix Anal. Appl., 31(5):2603–2622, 2010.
University of Manchester Nick Higham Actuarial Matrix Computations 2 / 8
|
References III
T. Charitos, P. R. de Waal, and L. C. van der Gaag.Computing short-interval transition matrices of adiscrete-time Markov chain from partially observeddata.Statistics in Medicine, 27:905–921, 2008.
T. Crilly.Arthur Cayley: Mathematician Laureate of the VictorianAge.Johns Hopkins University Press, Baltimore, MD, USA,2006.ISBN 0-8018-8011-4.xxi+610 pp.
University of Manchester Nick Higham Actuarial Matrix Computations 3 / 8
References IV
P. Glasserman and S. Suchintabandid.Correlation expansions for CDO pricing.Journal of Banking & Finance, 31:1375–1398, 2007.
C.-H. Guo and N. J. Higham.A Schur–Newton method for the matrix pth root and itsinverse.SIAM J. Matrix Anal. Appl., 28(3):788–804, 2006.
University of Manchester Nick Higham Actuarial Matrix Computations 4 / 8
References V
N. J. Higham.Accuracy and Stability of Numerical Algorithms.Society for Industrial and Applied Mathematics,Philadelphia, PA, USA, second edition, 2002.ISBN 0-89871-521-0.xxx+680 pp.
N. J. Higham.Computing the nearest correlation matrix—A problemfrom finance.IMA J. Numer. Anal., 22(3):329–343, 2002.
University of Manchester Nick Higham Actuarial Matrix Computations 5 / 8
References VI
N. J. Higham and L. Lin.On pth roots of stochastic matrices.Linear Algebra Appl., 435(3):448–463, 2011.
N. J. Higham and L. Lin.An improved Schur–Padé algorithm for fractionalpowers of a matrix and their Fréchet derivatives.MIMS EPrint 2013.1, Manchester Institute forMathematical Sciences, The University of Manchester,UK, Jan. 2013.20 pp.
University of Manchester Nick Higham Actuarial Matrix Computations 6 / 8
References VII
L. Lin.Roots of Stochastic Matrices and Fractional MatrixPowers.PhD thesis, The University of Manchester, Manchester,UK, 2010.117 pp.MIMS EPrint 2011.9, Manchester Institute forMathematical Sciences.
B. D. McCullough and A. T. Yalta.Spreadsheets in the cloud—not ready yet.Journal of Statistical Software, 52:1–14, 2013.
University of Manchester Nick Higham Actuarial Matrix Computations 7 / 8
References VIII
K. H. Parshall.James Joseph Sylvester. Jewish Mathematician in aVictorian World.Johns Hopkins University Press, Baltimore, MD, USA,2006.ISBN 0-8018-8291-5.xiii+461 pp.
H.-D. Qi and D. Sun.A quadratically convergent Newton method forcomputing the nearest correlation matrix.SIAM J. Matrix Anal. Appl., 28(2):360–385, 2006.
University of Manchester Nick Higham Actuarial Matrix Computations 8 / 8
Rehabilitating CorrelationsMatrix InversionMatrix RootsAppendix
