Multi-way Analysis with Applications in the Chemical Sciences · Multi-way analysis with...

MRM

WU102-FM WU102/Smilde July 16, 2004 16:44 Char Count= 0

Multi-way Analysis withApplications in the ChemicalSciences

Age SmildeUniversity of Amsterdam, Amsterdam, andTNO Nutrition and Food Research, Zeist, The Netherlands

and

Rasmus BroKVL, Frederiksberg, Denmark

and

Paul GeladiSLU, Umeå, Sweden

iii

Innodata0470012102.jpg

MRM


ii

MRM


Multi-way Analysis with Applicationsin the Chemical Sciences

i

MRM


ii

MRM


Multi-way Analysis withApplications in the ChemicalSciences

Age SmildeUniversity of Amsterdam, Amsterdam, andTNO Nutrition and Food Research, Zeist, The Netherlands

and

Rasmus BroKVL, Frederiksberg, Denmark

and

Paul GeladiSLU, Umeå, Sweden

iii

MRM


Copyright C© 2004 John Wiley & Sons Ltd, The Atrium, Southern Gate, Chichester,West Sussex PO19 8SQ, England

Telephone (+44) 1243 779777

Email (for orders and customer service enquiries): [email protected] our Home Page on www.wileyeurope.com or www.wiley.com

All Rights Reserved. No part of this publication may be reproduced, stored in a retrieval system or transmitted inany form or by any means, electronic, mechanical, photocopying, recording, scanning or otherwise, except underthe terms of the Copyright, Designs and Patents Act 1988 or under the terms of a licence issued by the CopyrightLicensing Agency Ltd, 90 Tottenham Court Road, London W1T 4LP, UK, without the permission in writing of thePublisher. Requests to the Publisher should be addressed to the Permissions Department, John Wiley & Sons Ltd,The Atrium, Southern Gate, Chichester, West Sussex PO19 8SQ, England, or emailed to [email protected],or faxed to (+44) 1243 770620.

This publication is designed to provide accurate and authoritative information in regard to the subject mattercovered. It is sold on the understanding that the Publisher is not engaged in rendering professional services. Ifprofessional advice or other expert assistance is required, the services of a competent professional should besought.

Other Wiley Editorial Offices

John Wiley & Sons Inc., 111 River Street, Hoboken, NJ 07030, USA

Jossey-Bass, 989 Market Street, San Francisco, CA 94103-1741, USA

Wiley-VCH Verlag GmbH, Boschstr. 12, D-69469 Weinheim, Germany

John Wiley & Sons Australia Ltd, 33 Park Road, Milton, Queensland 4064, Australia

John Wiley & Sons (Asia) Pte Ltd, 2 Clementi Loop #02-01, Jin Xing Distripark, Singapore 129809

John Wiley & Sons Canada Ltd, 22 Worcester Road, Etobicoke, Ontario, Canada M9W 1L1

Wiley also publishes its books in a variety of electronic formats. Some content that appears in print may not beavailable in electronic books.

Library of Congress Cataloging-in-Publication Data

Smilde, Age K.Multi-way analysis with applications in the chemical sciences / Age Smilde and

Rasmus Bro and Paul Geladi.p. cm.

Includes bibliographical references and index.ISBN 0-471-98691-7 (acid-free paper)1. Chemistry – Statistical methods. 2. Multivariate analysis. I. Bro, Rasmus.

II. Geladi, Paul. III. Title.QD39.3.S7S65 2004540′.72 – dc22 2003027959

British Library Cataloguing in Publication Data

A catalogue record for this book is available from the British Library

ISBN 0-471-98691-7

Typeset in 10/12.5pt. Times and Helvetica by TechBooks Electronic Services, New Delhi, IndiaPrinted and bound in Great Britain by T J International, Padstow, CornwallThis book is printed on acid-free paper responsibly manufactured from sustainable forestryin which at least two trees are planted for each one used for paper production.

iv

http://www.wileyeurope.comhttp://www.wiley.com

MRM


CONTENTS

Foreword ix

Preface xi

Nomenclature and Conventions xiii

1 Introduction 1

1.1 What is multi-way analysis? 11.2 Conceptual aspects of multi-way data analysis 11.3 Hierarchy of multivariate data structures in chemistry 51.4 Principal component analysis and PARAFAC 111.5 Summary 12

2 Array definitions and properties 13

2.1 Introduction 132.2 Rows, columns and tubes; frontal, lateral and horizontal slices 132.3 Elementary operations 152.4 Linearity concepts 212.5 Rank of two-way arrays 222.6 Rank of three-way arrays 282.7 Algebra of multi-way analysis 322.8 Summary 34

Appendix 2.A 34

3 Two-way component and regression models 35

3.1 Models for two-way one-block data analysis: component models 353.2 Models for two-way two-block data analysis: regression models 463.3 Summary 53

Appendix 3.A: some PCA results 54Appendix 3.B: PLS algorithms 55

v

MRM


vi Contents

4 Three-way component and regression models 57

4.1 Historical introduction to multi-way models 574.2 Models for three-way one-block data: three-way component models 594.3 Models for three-way two-block data: three-way regression models 764.4 Summary 83

Appendix 4.A: alternative notation for the PARAFAC model 84Appendix 4.B: alternative notations for the Tucker3 model 86

5 Some properties of three-way component models 89

5.1 Relationships between three-way component models 895.2 Rotational freedom and uniqueness in three-way component models 985.3 Properties of Tucker3 models 1065.4 Degeneracy problem in PARAFAC models 1075.5 Summary 109

6 Algorithms 111

6.1 Introduction 1116.2 Optimization techniques 1116.3 PARAFAC algorithms 1136.4 Tucker3 algorithms 1196.5 Tucker2 and Tucker1 algorithms 1236.6 Multi-linear partial least squares regression 1246.7 Multi-way covariates regression models 1286.8 Core rotation in Tucker3 models 1306.9 Handling missing data 1316.10 Imposing non-negativity 1356.11 Summary 136

Appendix 6.A: closed-form solution for the PARAFAC model 136Appendix 6.B: proof that the weights in trilinear PLS1 can be obtained

from a singular value decomposition 144

7 Validation and diagnostics 145

7.1 What is validation? 1457.2 Test-set and cross-validation 1477.3 Selecting which model to use 1547.4 Selecting the number of components 1567.5 Residual and influence analysis 1667.6 Summary 173

8 Visualization 175

8.1 Introduction 1758.2 History of plotting in three-way analysis 1798.3 History of plotting in chemical three-way analysis 1808.4 Scree plots 180

MRM


Contents vii

8.5 Line plots 1848.6 Scatter plots 1908.7 Problems with scatter plots 1928.8 Image analysis 2018.9 Dendrograms 2028.10 Visualizing the Tucker core array 2048.11 Joint plots 2058.12 Residual plots 2168.13 Leverage plots 2168.14 Visualization of large data sets 2168.15 Summary 219

9 Preprocessing 221

9.1 Background 2219.2 Two-way centering 2289.3 Two-way scaling 2329.4 Simultaneous two-way centering and scaling 2389.5 Three-way preprocessing 2399.6 Summary 244

Appendix 9.A: other types of preprocessing 245Appendix 9.B: geometric view of centering 247Appendix 9.C: fitting bilinear model plus offsets across one mode

equals fitting a bilinear model to centered data 249Appendix 9.D: rank reduction and centering 250Appendix 9.E: centering data with missing values 251Appendix 9.F: incorrect centering introduces artificial variation 251Appendix 9.G: alternatives to centering 254

10 Applications 257

10.1 Introduction 25710.2 Curve resolution of fluorescence data 25910.3 Second-order calibration 27610.4 Multi-way regression 28510.5 Process chemometrics 28810.6 Exploratory analysis in chromatography 30210.7 Exploratory analysis in environmental sciences 31210.8 Exploratory analysis of designed data 32310.9 Analysis of variance of data with complex interactions 340

Appendix 10.A: an illustration of the generalized rank annihilationmethod 346

Appendix 10.B: other types of second-order calibration problems 347Appendix 10.C: the multiple standards calibration model of the

second-order calibration example 349

References 351

Index 371

MRM


viii

MRM


FOREWORD

The early days of chemometrics saw researchers from around the world developing andapplying mathematical and statistical methods to a wide range of problems in chemistry.Chemometricians with analytical chemistry backgrounds were interested in such tasks ascontrolling or optimizing complicated analytical instruments, resolving spectra of complexmixtures into the spectra of pure components and, in general, getting more chemical infor-mation from quantitative measurements. Early chemometricians with physical or organicchemistry backgrounds were interested in relating chemical properties and molecular struc-tural features of molecules to their chemical reactivity or biological activity. These scientistsfocused on the tools of classification and other areas of pattern recognition. The first fewmeetings of chemometricians tended to separate into problem areas until it was recognizedthat there was at least one common thread that tied us all together. That common thread canbe summarized in a single word: multivariate. It gradually became clear that the power inour individual studies came from viewing the world as multivariate instead of univariate.At first this seemed to defy the scientific method we were taught that allows one variableat a time to be varied. Control engineers and some statisticians knew for some time thatunivariate experimental designs were doomed to fail in a multivariate world. While study-ing variance is useful, understanding covariance can lead one to move rapidly from data toinformation to knowledge.

As chemometrics matured, researchers discovered that psychometricians had movedbeyond multivariate analysis to what they called multi-way data analysis. Psychometricstudies involving multiple subjects (people) given several tests over periods of time leadto data structures that fit into three-dimensional computer arrays represented by blocks ofdata or what is called three-way arrays. Of course each subjects data (tests × time) could beanalyzed separately by removing a matrix or slice from a block of data. Psychometriciansrecognized that this approach would lose the covariance among subjects so they begandeveloping data models and analysis methodology to analyze entire blocks of data or three-way arrays at one time.

Chemometricians soon discovered that chemistry is rich with experimental designs andanalytical tools capable of generating multi-way data. An environmental chemist, for ex-ample, may acquire rainwater samples from several selected locations (first way) and atmultiple times (second way) over the course of a study and have each sample analyzed forseveral analytes of interest (third way). In analytical chemistry, an instrument capable of

ix

MRM


x Foreword

generating a three-way array of data from a single sample is called a third order instrumentbecause a three-way array is a third order tensor. Such instruments (e.g. GC/GC/MS) havespecial powers not obtainable by second order (GC/MS) or first order (MS) instruments. Itis not simply that so-called higher order instruments generate more data that makes themmore powerful. But it is the structure or form of the data that is the key and to access thepower the multi-way analysis tools described in this book must be employed.

This book is the first of its kind to introduce multi-way analysis to chemists. It beginsin Chapter 1 by answering the question ‘What is multi-way analysis?’ and then carefullycovers the possible data structures and the models starting with simple two-way data andextending on to four-way and higher data. Chapter 2 goes into definitions, properties andother mathematical details that must be appreciated before applying the tools of multi-waydata analysis. The following chapters describe models and associated data analysis toolsand algorithms for two- and three-way data.

Further chapters go into the details one must learn to implement multi-way analysis toolssuccessfully. Several important questions are posed such as: How should data be prepro-cessed before analysis? How can models be validated and what diagnostics are available todetect problems? What visualization tools are available that attempt to show humans in ourextremely limited three-dimensional world the powerful advantages of multi-way data?

The last chapter provides a look at selected applications of multi-way analysis thathave appeared in the chemometrics literature. This section of the book is not meant to bea complete literature review of the subject. Rather, applications were selected to aid thereader in understanding what the tools can do and hopefully point the way for the reader todiscover new applications in his/her own field of investigation.

Most chemists are certainly unaware of the power of multi-way analysis. Analyticalchemists, with very few exceptions, are not developing new instruments that can takeadvantage of the second and third order advantages obtained from use of the tools describedso well in this book. Paradigm shifts in a field can only take place when the field is lookingfor and can accept new ways to operate. This book opens the door to anyone looking for aparadigm shift in his or her field of study. Multi-way analysis provides a new way to lookat experimental design and the scientific method itself. It can help us understand a worldthat is multivariate, often nonlinear and dynamic.

Bruce R. KowalskiSomewhere in Hay Gulch, Hesperus, Colorado

MRM


PREFACE

Goal of the project

In chemistry and especially in chemometrics, there is a growing need for new data analysistechnology. New data analysis technologies are usually first presented in papers in the litera-ture, but after a while the volume grows and there is a need for completeness and a systematicoverview. Review articles are written to give this needed overview. After a few years, a sim-ple review article is not enough for containing the existing knowledge and a monographis needed. The present monograph aims to give all this: an almost complete overview ofall the aspects of multi-way analysis that is needed for users in chemistry. Applying multi-way analysis requires an in-depth understanding of techniques and methodologies, but alsoa wide overview of applications and the relation of the techniques to problem definitionshould not be forgotten. Many problems in chemical research and industry are such thatthree-way data result from the measurements, but simply putting the data in a three-wayarray and running an algorithm is not enough. It is important to understand the backgroundof the problem in preprocessing, analyzing the data, presentation and interpretation of themodel parameters and interpretation of the results. The user is provided with an overview ofthe available methods as well as hints on the pitfalls and details of applying these. Throughnumerous examples and worked-out applications, an understanding is provided on how touse the methods appropriately and how to avoid cumbersome problems.

History of the project

The idea for this book has existed for many years. The project was finally realized in1995 when a contract was signed with a publisher. We knew that the task to be carriedout was enormous, but we underestimated it. Years went by without much production, justdiscussion. In the meantime the volume of literature on three-way analysis grew and the needfor this monograph became even more urgent. The result is that the structure of the bookhas undergone many improvements since the first synopsis and that the volume of exampleshas grown. In the process of writing, new ideas emerged and had to be tested and a deeperunderstanding of the literature was acquired. Some loose concepts had to be substantiatedby rigorous proofs. A special development was that of notation and nomenclature. Much

xi

MRM


xii Preface

was borrowed from psychology and sociology, but much was also modified and adapted tochemical formulation.

Some topics were left out, mainly because they were too new and not fully developed.Other topics were considered too narrowly specialized to really fit in. The applications inthis book do not give a complete overview of all the literature. They are selected examples ofimportant developments that are helpful in realizing how three-way analysis can be usefulalso in other applications.

How to read this book

The book was written with chemistry or related fields (biology, biochemistry, pharmaceuticaland clinical chemistry, process technology, quality control, etc.) in mind. The reader isexpected to have a basic knowledge of statistics and linear algebra. Although two-waymethods are introduced in Chapter 2, some background knowledge of latent variables andregression is useful. A course for newcomers would contain Chapters 1–4 and selected partsof Chapters 7–9, with Chapter 3 as a refresher for two-way methods. Such a course wouldprobably start with selected parts of Chapter 10 and return to the same chapter in the end.The advanced reader can start with Chapter 4 and read through to Chapter 10.

Acknowledgements

We would like to thank numerous people who helped us in different ways. Three groups arementioned separately: proofreaders of chapters, people who provided examples and thosewho were helpful as colleagues or management during the many years that we worked onthe book project. They are given in alphabetical order.

Proofreaders were: Karl Booksh, Kim Esbensen, Neal Gallagher, Margriet Hendriks,René Henrion, Henk Kiers, Lars Nørgaard, Mary-Beth Seasholtz, Nikos Sidiropoulos, Ric-cardo Leardi and Peter Schoenmakers.

Those who provided examples are: Peter Åberg, Claus Andersson, Dorrit Baunsgaard,Helén Bergner, Jennie Forsström, Paul Gemperline, René Henrion, Phil Hopke, JosefinaNyström, Riccardo Leardi, Magni Martens, Torbjörn Lestander, Carsten Ridder, LenaRingqvist and Barry Wise.

Helpful colleagues and management: Johanna Backman, Laila Brunes, Ulf Edlund, BertLarsson, Calle Nilsson, Britta Sethson, Lars Munck, Ellen Abeling, Renate Hippert and JanRamaker.

MRM


NOMENCLATURE ANDCONVENTIONS

x a scalarx a vector (always a column vector)X a matrix / two-way arrayX(I × JK),X(J × IK),X(K × IJ) matricized form of X with mode A, B or C kept intact[X1 X2] concatenation of the matrices X1 and X2X a three-way array or multi-way arraye, E, E vector, matrix or three-way array of residualsEx , Ey residuals in X and Y (regression)x̂, X̂, X̂ vector, matrix or three-way array of fitted (model) valuesi, j,k,l, . . . running indicesI,J,K ,L , . . . maximum of index i, j,k,l,..I,J preserved for size of XI,J,K preserved for size of XI,M preserved for size of YI,M,N preserved for size of Yp,q,r running indices for number of latent variables in the different

modesP,Q,R maximum values of p,q,rw, w, W weight, vector of weights, matrix of weightsr(A) rank of AkA k-rank of Avec A vectorization of Atr(A) trace of AA, B, C loading matricesD diagonal matrixD three-way superdiagonal arraydiag(a) matrix with the vector a on its diagonal and all off-diagonal

elements zeroI identity matrixI superdiagonal three-way array with ones on the diagonal

xiii

MRM


xiv Nomenclature and Conventions

G matricized core-arrayG three-way core-array�(X) range or column-space of Xℵ(X) null space of XSStot total sum of squares in a data setSSmod sum of squares explained by the modelSSres residual sum of squares2D plot two-dimensional plot3D plot three-dimensional plot

* Hadamard or Direct product� Khatri–Rao product⊗ Kronecker product� Tensor product‖.‖ Frobenius or Euclidian norm of vector or matrix

SPH

C01 WU102/Smilde May 20, 2004 14:53 Char Count= 0

1INTRODUCTION

1.1 What is Multi-way Analysis?

Multi-way analysis is the analysis of multi-way data. In this book a distinction will bemade between one-way, two-way, three-way and higher-way data. If a single measurementon a chemical system generates a single number and this measurement is repeated, then asequence of numbers is the result. For example, a titration is repeated four times, resultingin the volumes x1, x2, x3 and x4. This is a sequence of numbers and this sequence has tobe analyzed with one-way tools, such as calculating the mean and the standard deviation ofthese four numbers.

Going up one level to instruments that generate sequences of numbers, e.g. a UV–Visspectrum consisting of absorptions at J wavelengths. Upon taking I of those measurements,a matrix X of dimensions I × J results. This is a two-way array and such a matrix can beanalyzed appropriately with two-way analysis tools, such as principal component analysis.

If a single instrument generates a table of numbers for each sample, e.g., a fluorescenceemission/excitation landscape, this results in a matrix X (J × K ), where J excitation wave-lengths are used for measuring emission at K wavelengths. Taking I of these measurements,for example at I different occasions, generates a three-way array of size I × J × K . Sucharrays can be analyzed with three-way analysis methods, which is the issue of this book.It is even possible to generate four-way, five-way, in general multi-way data. Methods thatdeal with four- and higher-way data will also be discussed, although in less detail than themore commonly used three-way analysis tools.

1.2 Conceptual Aspects of Multi-way Data Analysis

The goal of this section is to introduce data, data bases and arrays. Data are measured; suchas a cardiologist measuring electrocardiogram data and a radiologist measuring computeraided tomography scans. This is data collection and is practiced a lot in academia andindustry. Putting the data together patient by patient will lead to construction of a data baseor data file. This data base may still be analog or it may be in digital form as computer

Multi-way Analysis With Applications in the Chemical Sciences. A. Smilde, R. Bro and P. GeladiC© 2004 John Wiley & Sons, Ltd ISBN: 0-471-98691-7

1

SPH


2 Multi-way Analysis With Applications in the Chemical Sciences

files. Some simple statistical and bookkeeping operations may be carried out on such a database. An array is only made when a statistician or data analyst constructs it from data in thebase, usually with a specific purpose. In analytical chemistry all these steps are often doneautomatically, especially with homogeneous variables in spectroscopy, chromatography,electrophoresis etc. The instrument collects the data in digital form, keeps the files in orderand simply constructs the desired array when needed, but this is still an exception.

Multi-way analysis is not only about algorithms for fitting multi-way models. There aremany philosophical and technical matters to be considered initially or during analysis. Thedata have to be collected. Problems such as detection limits, missing data and outliers mustbe handled. Preprocessing might be needed to bring the data in a suitable form. This getseven more complicated when qualitative and quantitative data are mixed. Some of the pointsto be considered are:

� the nature of a way/mode and how different combinations of these coexist;� mean-centering and linear scaling of ways;� nonlinear scalings;� the role of outliers;� missing data;� type of data: real, complex, integer, binary, ordinal, ratio, interval;� qualitative versus quantitative data;� rank and uniqueness of multi-way models.

Many of these aspects will be encountered in subsequent chapters and others are treated instandard statistical literature.

Problem definition

Apart from these data analytical issues, the problem definition is important. Defining theproblem is the core issue in all data analysis. It is not uncommon that data are analyzed bypeople not directly related to the problem at hand. If a clear understanding and consensusof the problem to be solved is not present, then the analysis may not even provide a solutionto the real problem. Another issue is what kind of data are available or should be available?Typical questions to be asked are: is there a choice in instrumental measurements to bemade and are some preferred over others; are some variables irrelevant in the context, forexample because they will not be available in the future; can measurement precision beimproved, etc. A third issue concerns the characteristics of the data to be used. Are theyqualitative, quantitative, is the error distribution known within reasonable certainty, etc.The interpretation stage after data analysis usually refers back to the problem definition andshould be done with the initial problem in mind.

Object and variable ‘ways’

In order to explain the basics of three-way analysis, it is easiest to start with two-way dataand then extend the concepts to three-way. Two-way data are represented in a two-way datamatrix typically with columns as variables and rows as objects (Figure 1.1). Three-way

SPH


Introduction 3

J

I

Variables

Objects

Figure 1.1. A two-way array has, e.g., the objects as rows and thevariables as columns. There are I objects and J variables.

I

J

K

Objects

Variables

Type 2Type 1

Figure 1.2. A typical three-way array has I objects, J type 1 variables and K type 2 variables.

data are represented as a three-way array (box) (Figure 1.2). The columns and rows ofthe two-way array are now ‘replaced’ by slices, like slicing a rectangular bread or cake indifferent ways. This is shown in Figure 1.3. Each horizontal slice (also sometimes calledhorizontal slab) holds the data for one object in a three-way array. Each vertical sliceholds data of one specific variable of one type (say absorbances at different wavelengths orretention times) and the back-to-front slices the variables of the other type, say judges ortime slots. Other types of arrangements can exist as well. Some three-way arrays have twotypes of object modes and only one type of variable mode, or even three objects modes andno variable mode. An example is in multivariate image analysis where the data are three-way arrays with x-coordinate, y-coordinate of the pixels as object ways and wavelength asthe variable way. See Figure 1.4. The whole discussion of what the nature of an object orvariable is may not be clear on all occasions. Basically an object is a physical or mentalconstruct for which information is sought. A variable provides means for obtaining thatinformation. For example, the physical health of persons might be of interest. Differentpersons are therefore objects in this respect. The blood pressure is a variable, and measuringthe blood pressure of the different objects provides information of the health status of theobjects.

For a three-way array of liquid-chromatography–ultraviolet spectroscopy data, the sam-ples are the objects whereas the absorbance at different retention times (of the chromatogra-phy mode) and the wavelengths (of the ultraviolet spectroscopy mode) are the two variablemodes. Hence, absorbance is measured as a function of two properties: retention time andwavelength. For a three-way array of food products × judges × sensory properties, thejudge mode can be an object mode or a variable mode. In a preliminary analysis the judges

SPH



I

JK

OBJECTS

Horizontal slices

I

J

K

I

K

Vertical slices

TYPE 1 VARIABLES

Frontal slices

TYPE 2 VARIABLES

J

Figure 1.3. Slicing of a three-way array can be done objectwise or along type 1 or type 2 variables.The slices are two-way arrays. This results in horizontal, vertical and frontal slices.

J123..

K

I

J

K

Figure 1.4. An image of I × J pixels for K different wavelengths gives a multivariate image whichis an I × J × K array. In this array there are two object ways.

can be seen as objects, where the purpose of the analysis is to assess if the judges agree.In a subsequent analysis with the judges as variables, the properties of the products areinvestigated, hence the products are the objects.

As can be understood from the above sensory problem, data alone do not determine whatis an object and what is a variable. This goes back to the problem definition. Only when astated purpose of the data is provided can the nature of the ways be deduced.

Some three-way models do not take the nature of a way into account, but the conceptsof objects and variables are important, e.g., for defining how an array is preprocessed andfor interpretation of the results. The same naturally holds for two-way analysis methods.

Types of objects and variables

A variable mode in a two-way or three-way array can consist of homogeneous variables,heterogeneous variables or a mixture of the two. Homogeneous variables have the samephysical units and are often measured on the same instrument. The variables of a two-way

SPH


Introduction 5

array may be absorbance measured in a spectrometer at different wavelengths. This meansthat all variables have the same scale (absorbance).

There is sometimes an intrinsic natural order in the variables. Spectral variables can forexample be meaningfully ordered according to wavelength/energy/wavenumber; elutionprofiles have an intrinsic order in time; geographical measurements have a two-dimensionalorder in space, etc. For homogeneous variables the order is often easy to detect visuallybecause of autocorrelation. For chemical reasons, spectra are often smooth, showing distinctpeaks and valleys. This property can be used in the analysis, for verifying and validatingthe results and it makes it easy to study the data or loadings of the way, either by themselvesor in scatter plots. Putting the wavelengths in random order will destroy much of theunderstanding of the data, even if the data analysis gives numerically identical results.

Also for objects there can be intercorrelations, e.g., related to distance or time, as inenvironmental sampling and sequential sampling in industrial processes. This property isimportant for the choice of model, validation, interpretation etc.

Heterogeneous variables are measured in different units, or measured with differentinstruments, often according to a large variety of principles. The variables in an investigationmay be:

� temperature in ◦C;� wind speed in m s−1;� wind direction in degrees/radians;� humidity in %;� pressure in pascals;� CO concentration in mg m−3;� particulate matter in mg m−3.

The different units used for measuring these variables require attention. For example, scalingis an important issue for such variables, because the possible widely different scales leadto variables of very different numerical values. Using unscaled data would lead to modelsfitting and reflecting mainly those specific variables that have high numerical values, ratherthan the whole data set as such.

Mixtures of homogeneous and heterogeneous variables also occur. Whole body impe-dance data for patients are used for measuring fat, water and bone content. The impedancesare measured at different frequencies. The frequencies are homogeneous variables, usuallytabulated as increasing frequencies, but to these data, the heterogeneous variables age, sex,drug dosage, length and weight are usually added. Note that sex is a categorical variable,age is usually tabulated as an integer and drug dosage is a real. The impedances may beexpressed as complex numbers. Again, scaling of such data needs to be considered.

1.3 Hierarchy of Multivariate Data Structuresin Chemistry

In order to set the stage for introducing different types of methods and models for analyzingmulti-way data it is convenient to have an idea of what type of multivariate data analysisproblems can be encountered in chemical practice. A categorization will be developed thatuses the types of arrangements of the data set related to the chemical problem.

SPH



BlocksWays 1 2 3

2

3

4

Figure 1.5. Relationship between data analysis methods for multi-block and multi-way arrangementsof the data. Blocks refer to blocks of data, and ways to the number of ways within a block.

A data set can have different features. Two general features of a data set will be dis-tinguished and used to categorize data sets. The first feature is whether the data set is twoway, three-way, or higher-way. In Section 1.1 this distinction has already been explained.Another feature is whether the data set can be arranged as a single block of data, two blocksor multiple blocks of data. Crossing both features leads to a categorization as visualized inFigure 1.5. The arrows in this figure indicate the relationship that is sought. Some problemsmay not fit exactly into this categorization, but it provides a simple overview of the bulk ofproblems encountered in data analysis. For two-block data, both blocks must have at leastone mode (or way) in common, e.g., the rows in the first block of data contain measurementson the same objects as the corresponding rows in the second block of data. For multi-blockdata, different relational schemes can be formulated; this is problem and context dependent.Note that a two-block problem does not necessarily need two blocks of the same order. Forexample, relationships between a three-way array and, e.g., a vector are positioned in themiddle of Figure 1.5. This is seen by setting the dimension in the second and third mode toone in one of the three-way arrays in the figure.

In the following a brief description is given of some methods available to investigate thetypical data analysis problems, as indicated in Figure 1.5.

Two-way one-block data

The simplest multivariate data set is a data set consisting of measurements (or calculatedproperties) of J variables on I objects. Such a data set can be arranged in an I × J matrix X.This matrix X contains variation which is supposed to be relevant for the (chemical) problemat hand. Several types of methods are available to investigate this variation depending onthe purpose of the research and the problem definition.

EXPLORATION

If the purpose of analyzing the data set is to find patterns, relations, differences and agree-ments between objects and/or variables, then decomposition methods can be used to sum-marize the data conveniently and explore the data set using plots and figures. Typically,

SPH


Introduction 7

subspace-based methods (projection methods) are used to handle collinearity problems. Awell-known technique to do this is principal component analysis (PCA). PCA will be de-scribed in more detail in Chapter 3. It suffices to say here that PCA is a dimension reducingtechnique, that projects objects and variables to low dimensional spaces [Pearson 1901].Plots of these projections can then be used for exploratory purposes.

While PCA is a linear projection method, there also exist nonlinear projection methods,e.g. multidimensional scaling [Mardia et al. 1979] and nonlinear PCA [Dong & McAvoy1996]. A good overview of nonlinear multivariate analysis tools is given by [Gifi 1990].

ELABORATION 1.1Exploratory and Confirmatory Analysis

There are two basic modes of analysis: exploratory and confirmatory analysis. Confirmatoryanalysis is usually related to formal statistical hypothesis testing. A model is hypothesized,data generated, and the significance of the model tested. In exploratory analysis the mainpurpose of the analysis is to learn from the data about interrelationships between variablesand objects. Exploratory analysis is typically a process where several models provide newinsight in and knowledge of what to do. Initially, data are generated according to a hypothesisbased on a very general idea. The results and information arising from the data give rise toa new set of ideas, which lead either to generation of new data or different analyses of thecurrent data. By repeated analyses generating new hypotheses, new data and new models,the knowledge of the problem increases and the analysis may become more focused. Thus,in exploratory analysis the main hypotheses comes from the data and the models, as opposedto confirmatory and traditional deductive thinking where the hypothesis is posed before dataare even acquired.

CLASSIFICATION PROBLEMS

Techniques that are suitable to detect grouping in the objects can be divided in supervisedand unsupervised classification methods. The purpose of classification is typically to assignsome sort of class membership to new samples. Mostly there are several classes, but, e.g.,in raw material identification, the purpose is to establish whether or not a new batch of rawmaterial is within the specified limits or not. Hence, there is only one well-defined class,being good raw material. In the case of supervised classification the data analyst knowsbeforehand which object in the data set belongs to which group. The purpose of the dataanalysis is then to verify whether this grouping can be found in the data set and to establishmeans of discriminating between the groups using a classification rule. Methods to servethis purpose are, e.g., discriminant analysis [Mardia et al. 1979].

In unsupervised classification, the grouping of the objects is not known beforehandand finding such groupings is the purpose of the analysis. Classical methods to performunsupervised classification include variants of cluster analysis [Mardia et al. 1979].

ESTIMATING UNDERLYING PHENOMENA

An important class of problems in chemistry is the estimation of underlying phenomena or la-tent variables. Suppose, e.g., that the I × J matrix X contains spectroscopic measurements

SPH



(absorbances at J wavelengths) made on I mixtures of R chemical constituents. Curveresolution techniques try to resolve the matrix X as a product CS′ based on Beer’s Law[Christian & O’Reilly 1986] where S (J × R) contains the pure spectra of the R chem-ical constituents and C (I × R) contains the relative concentrations of the R chemicalconstituents in the mixtures.

There is a wide variety of curve resolution techniques available [Lawton & Sylvestre1971, Liang et al. 1993, Malinowski 1991, Tauler 1995, Vandeginste et al. 1987, Windig &Guilment 1991]. In these techniques, constraints can be included to direct the solution tomeaningful spectra and concentration estimates.

Two-way two-block data

PREDICTION PROBLEMS

An enormous variety of problems in chemistry can be formulated in this type of data arrange-ment. Multivariate calibration, in which one data block contains spectral measurements andthe other block the known concentrations of the absorbing constituents, is one example[Martens & Næs 1989]. The purpose is then to build a calibration model, with which it ispossible to predict the concentrations of the constituents in a new sample using the spectralmeasurement of that sample and the calibration model (the relationship, as indicated withan arrow in Figure 1.5).

Another example is Quantitative Structure–Activity Relationships (QSAR) in which thefirst block contains measurements or calculated values of compounds and the second blockcontains measured biological activities. Here the purpose is to predict activities of newcompounds using (simple) measurements or calculated values of that new compound andthe model. Sometimes, the reverse is also sought: given a certain desired activity, whatshould a compound look like? This is known as the inverse problem and is considerablymore complicated than predicting activity.

The aim of regression methods in general is to predict one block of measurements (predic-tands) using another block of measurements (predictors). Examples of methods are MultipleLinear Regression [Draper & Smith 1998], Principal Component Regression [Martens &Næs 1989], Partial Least Squares Regression (PLS) [Wold et al. 1984], Ridge Regression[Hoerl & Kennard 1970], Projection Pursuit Regression [Friedman & Stuetzle 1981], Mul-tivariate Adaptive Regression Splines [Friedman 1991], Principal Covariates Regression[De Jong & Kiers 1992]. Clearly, there is an enormous number of regression methods.

EXPLORATION AND OPTIMIZATION PROBLEMS

For illustrating optimization problems, an example is given from the field of designedexperiments. If the yield of an organic synthesis reaction must be optimized, then thevariables that influence this yield, e.g. pH and temperature, can be varied according to anexperimental design [Box et al. 1978]. The experiments are carried out according to thisdesign and the measured yields are obtained in the second block of data. The first blockof data contains the settings of the controlled variables; pH and temperature. The effect onthe yield of varying the controlled variables can be estimated from the data, thus providingmeans for optimizing the yield [Box et al. 1978, Montgomery 1976].

SPH


Introduction 9

The methods to use for these types of problems are again the regression methods, asmentioned in the previous section. Methods also exist that are especially developed to dealwith designed data [Box et al. 1978].

In the above examples there is a natural way to order the complete data set in two blocks,where both blocks have one mode in common. In Chapter 3 the methods of multiple linearregression, principal component regression, and partial least squares regression will bediscussed on an introductory level.

Two-way multiblock data

EXPLORING, PREDICTING AND MODELING

Examples of this type of data are, e.g., from the field of process modeling and multivariatestatistical process control. Suppose that process measurements are taken from a chemicalreactor during a certain period of time. In the same time period, process measurements aretaken from the separation column following that reactor as a unit operation. The compositionof the product leaving the column is also measured in the same time period and with thesame measurement frequency as the process measurements. This results in three blocks ofdata, with one mode in common. Relationships between these blocks can be sought and,e.g., used to develop control charts [Kourti et al. 1995, MacGregor et al. 1994].

Methods to deal with this type of problems are Multiblock Partial Least Squares [Wangen& Kowalski 1988], Hierarchical PLS [Westerhuis et al. 1998, Wold et al. 1996] and LatentPath Modeling [Frank & Kowalski 1985, Jöreskog & Wold 1982]. The latter method is notvery popular in chemistry, but is common in the social sciences.

COMPARING CONFIGURATIONS

In sensory analysis different food products are subjected to judgements by a sensory panel.Suppose that several food products are judged by such a panel and scored on differentvariables (the meaning and number of variables may even differ for different assessors).In order to obtain objective descriptions of the samples and not subjective reflections ofthe individual judges, it is necessary to use tools that correct for individual differencesbetween judges and extract the consensus judgements. Techniques to tackle these types ofproblems are, e.g., Procrustes Analysis [Gower 1975, Ten Berge 1977] and HierarchicalPCA [Westerhuis et al. 1998, Wold et al. 1996].

Three-way one-block data

EXPLORATION

One of the typical purposes of using three-way analysis on a block of data is exploringthe interrelations in those data. An example is a three-way environmental data set consist-ing of measured concentrations of different chemical compounds on several locations ina geographical area at several points in time. Three-way analysis of such a data set canhelp in distinguishing patterns, e.g., temporal and spatial behavior of the different chemical

SPH



compounds. Another example is from the field of chromatography where retention valuesof different solutes are measured on different chromatographic columns at different mobilephase compositions. Again, three-way analysis can help in finding patterns [Smilde et al.1990], such as systematic differences between the chromatographic columns. Another ap-plication area is image analysis. Two-dimensional images of an object taken at differentwavelengths are stacked on top of each other and analyzed with three-way tools [Geladi &Grahn 1996].

CALIBRATION AND RESOLUTION

One of the earliest applications of three-way analysis in chemistry on a single block of datais in second-order calibration. Calibration is concerned with estimating the concentrationof a certain analyte in a mixture. In the case of second-order calibration, an instrument isused that generates a matrix of measurements for a single chemical analysis. If the standardX1 contains the pure response of the measured analyte and X2 is the measurement of themixture containing that analyte, then under certain conditions it is possible to quantify theanalyte in the mixture, even if this mixture contains unknown interferents. This is done bystacking X1 and X2 on top of each other and building a three-way model of that stackedarray. Not only the concentration estimate of the analyte is obtained but also estimates ofthe pure response profiles, e.g., pure spectra and chromatograms of the analyte [Sanchez &Kowalski 1988]. Second-order calibration will be explained in more detail in Chapter 10.

Three-way two-block data

Three-way two-block data can be encountered, e.g., in modeling and multivariate statisticalprocess control of batch processes. The first block contains the measured process variablesat certain points in time of different batch runs. The second block might contain the qualitymeasurements of the end products of the batches. Creating a relationship between theseblocks through regression analysis or similar, can shed light on the connection of the varia-tion in quality and the variation in process measurements. This can be used to build controlcharts [Boqué & Smilde 1999, Kourti et al. 1995]. Another application is in multivariatecalibration where, for example, fluorescence emission/excitation data of samples are usedto predict a property of those samples [Bro 1999].

Three-way multiblock data

An example of a three-way multiblock problem was published in the area of multivariatestatistical process control of batch processes [Kourti et al. 1995]. Suppose, e.g., that thefeed of a batch process is characterized by a composition vector of length L . If I differentbatch runs have been completed this results in a matrix X (I × L) of feed characteristics.The process measurements are collected in Z (I × J × K ) having I batches, J variablesand measured at K time points each. The M different end product quality measurementsare collected in Y (I × M). Investigating if there is a connection between X, Z and Y is acommon task in process analysis.

SPH


Introduction 11

Four-way data and higher

Applications and examples of multi-way data and multi-way analysis with four- and higher-way data are scarce. One application is in 3D-QSAR based on Comparative Molecular FieldAnalysis (COMFA), where a five-way problem is tackled [Nilsson et al. 1997]. In COMFA amolecule is placed in a hypothetical box with a grid of points. The energy of the molecule ina grid point is calculated when a hypothetical probe is placed at a certain position in the box.The five ways are: different probes, three different spatial directions in the molecule, andthe number of molecules. Five-way analysis has been used to study the variation in theresulting five-way array. Moreover, for each molecule an activity is measured. A predictiverelationship was sought between the five-way array and the vector of activities.

In image analysis, it has become possible to produce three-way images with rows,columns and depth planes as voxel coordinates. Measuring the images at several wave-lengths and following such a system over time will also give a five-way array. This arrayhas image coordinates H × I × J, K wavelength variables and M time samples. Otherexamples on higher-way data occur, e.g., in experimental design, spectroscopy and chro-matography [Durell et al. 1990, Bro 1998].

1.4 Principal Component Analysis and PARAFAC

A short introduction to principal component analysis for two-way arrays and PARAFACfor three-way arrays will be given here. These methods are used in the next chapter, andtherefore, a short introduction is necessary. Definitions, notations and other details on thesemethods are explained in later chapters.

In principal component analysis (PCA), a matrix is decomposed as a sum of vectorproducts, as shown in Figure 1.6. The ‘vertical’ vectors (following the object way) arecalled scores and the ‘horizontal’ vectors (following the variable way) are called loadings.A similar decomposition is given for three-way arrays. Here, the array is decomposed as asum of triple products of vectors as in Figure 1.7. This is the PARAFAC model. The vectors,of which there are three different types, are called loadings.

A pair of a loading vector and a matching score vector is called a component for PCAand similarly, a triplet of loading vectors is called a PARAFAC component. The distinctionbetween a loading and a score vector is relatively arbitrary. Usually, the term score vector is

= + + +...two-wayarray

Score 1

Loading 1

Score 2 Score 3

Loading 3Loading 2

Figure 1.6. In principal component analysis, a two-way array is decomposed into a sum of vectorproducts. The vectors in the object direction are scores and the ones in the variable direction areloadings.

SPH



= + + +...three-wayarray

A

B

C

A-loading 1

B-loading 1

C-loading 1

Figure 1.7. In PARAFAC, a three-way array is decomposed into a sum of triple products of vectors.The vectors are called loadings. There are three different types: called A,B and C for convenience.

only used if a mode is considered an object mode and loadings are used for variable modes.In some cases the distinction between objects and loadings, however, is not clear fromthe data alone and the more broad term loading vector is then generally used. Furthermore,since most decomposition methods to be discussed make no distinction in the decompositionof the array, it has become customary in three-way analysis to only use the term loadingvector. As seen in the Figures 1.6 and 1.7, a number of components can be found for an array,numbered 1, 2, 3, 4 etc. If the data can be well-modeled by a small number of componentsthe interpretation of the variations in the data can be analyzed through these componentsinstead of the raw data.

1.5 Summary

This chapter introduces multi-way analysis and describes important concepts of dataanalysis in general. In data analysis, sometimes a distinction is made between objectsand variables. Such distinctions are not always clear. Moreover, there is a large variety oftypes of variables, such as homogenous or heterogenous. It is important to keep these thingsin mind when doing multi-way analysis.

An overview is given of the different kinds of possible multi-way analyses and exist-ing methods to perform such analyses. A categorization is presented in which differentmulti-way analyses are divided into classes using two criteria: (i) the number of blocksinvolved, and (ii) the order of the arrays involved. Practical examples are given for eachclass, showing the relevance of the multi-way analysis problem.

Finally, two important decomposition methods for two-way analysis (PCA) and three-way analysis (PARAFAC) are introduced briefly, because these methods are needed in thefollowing chapter.

SPH


2ARRAY DEFINITIONSAND PROPERTIES

2.1 Introduction

There are different ways to represent two- and multi-way models. One way is to use rigorousmatrix algebra, another way is to use a pictorial description. When exactness is needed,matrix algebra is necessary. However, in order to understand the structure of the models,a pictorial description can be more informative. Hence, in this book both types of rep-resentations are used. In this section the pictorial representation will be introduced. Thenomenclature used for vectors, matrices and multiway arrays is given in the NomenclatureSection. Moreover, notational conventions for multi-way arrays are taken from Kiers [2000]with a few exceptions.

The rules for the pictorial description are: numbers (scalars) are represented by small cir-cles; vectors by line segments; matrices by rectangles; and three-way arrays by rectangularboxes. This is shown in Figure 2.1. These graphics also allow for a convenient representationof vector and matrix multiplication (see Figure 2.2). As is shown later in this chapter, moreadvanced concepts as dyads, triads and matricizing can also be visualized with these rules.

The matrix product is well known and can be found in any linear algebra textbook.It reduces to the vector product when a vector is considered as an I × 1 matrix, and atransposed vector is a 1 × I matrix. The product a′b is the inner product of two vectors.The product ab′ is called the outer product or a dyad. See Figure 2.2. These products haveno special symbol. Just putting two vectors or matrices together means that the product istaken. The same also goes for products of vectors with scalars and matrices with scalars.

2.2 Rows, Columns and Tubes; Frontal, Lateraland Horizontal slices

For two-way arrays it is useful to distinguish between special parts of the array, suchas rows and columns. This also holds for three-way arrays and one such a division

Multi-way Analysis With Applications in the Chemical Sciences. A. Smilde, R. Bro and P. GeladiC© 2004 John Wiley & Sons, Ltd ISBN: 0-471-98691-7

13

SPH



scalar vector matrix three-way array

a = a (I×1) =

1

I

A (I×J ) =

J

I

A (I×J×K ) =

I

JK

Figure 2.1. Pictorial representation of scalars, vectors, matrices and three-way arrays. The circle inthe matrix indicates the (1,1) element and in the three-way array indicates the (1,1,1) element; I, Jand K are the dimensions of the first, second and third mode, respectively.

Vector multiplication Matrix multiplication

a′b = inner product=scalar:

=

ab′ = outer product=matrix:

II

=

I

J

I

J

AB = matrix:

=

I

J

J

K

AB

I

K

AB

Figure 2.2. Pictorial representation of some vector and matrix manipulations. Note that for multipli-cation no special symbol is used.

I

JK

Horizontal slices

I

JK

Vertical slices

I

J

K

Frontal slices

Figure 2.3. Partitioning of a three-way array in slices (two-way arrays).

in parts was already introduced in Chapter 1: frontal, horizontal and vertical slices orslabs. For convenience, the figure is repeated here (Figure 2.3). There are three differentkinds of slices for the three-way array X (I × J × K ). The first ones are the horizon-tal slices: X1, . . . , Xi , . . . , XI ; all of size (J × K ). The second ones are the verticalslices: X1, . . . , X j , . . . , XJ ; all of size (I × K ). The last ones are the frontal slices:X1, . . . , Xk, . . . , XK ; all of size (I × J ). This shorthand notation is convenient but not al-ways unambiguous, e.g. X2 might mean three different things. Such ambiguity is removedin the text where necessary, e.g., using Xi=2 for the first mode. It is also possible to definecolumn, row and tube vectors in a three-way array. This is shown in Figure 2.4. In a three-way

Date post:	06-Oct-2020
Category:	Documents
Upload:	others
View:	3 times
Download:	0 times

Multi-way Analysis with Applications in the Chemical Sciences · Multi-way analysis with...

Documents