Digital Signal Processing with Kernel Methods

Digital Signal Processing with Kernel Methods

José Luis Rojo-ÁlvarezDepartment of Signal Theory and CommunicationsUniversity Rey Juan CarlosFuenlabrada (Madrid)andCenter for Computational SimulationUniversidad Politécnica de Madrid, Spain

Manel Martínez-RamónDepartment of Electrical and Computer EngineeringThe University of New MexicoAlbuquerque, New MexicoUSA

Jordi Muñoz-MaríDepartment of Electronics EngineeringUniversitat de ValènciaPaterna (València), Spain

Gustau Camps-VallsDepartment of Electronics EngineeringUniversitat de ValènciaPaterna (València), Spain

This edition first published © John Wiley & Sons Ltd

All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, ortransmitted, in any form or by any means, electronic, mechanical, photocopying, recording or otherwise,except as permitted by law. Advice on how to obtain permission to reuse material from this title is availableat http://www.wiley.com/go/permissions.

The right of José Luis Rojo-Álvarez, Manel Martínez-Ramón, Jordi Muñoz-Marí, Gustau Camps-Valls to beidentified as the authors of the editorial material in this work has been asserted in accordance with law.

Registered OfficesJohn Wiley & Sons, Inc., River Street, Hoboken, NJ , USAJohn Wiley & Sons Ltd, The Atrium, Southern Gate, Chichester, West Sussex, PO SQ, UK

Editorial OfficeThe Atrium, Southern Gate, Chichester, West Sussex, PO SQ, UK

For details of our global editorial offices, customer services, and more information about Wiley productsvisit us at www.wiley.com.

Wiley also publishes its books in a variety of electronic formats and by print-on-demand. Some content thatappears in standard print versions of this book may not be available in other formats.

Limit of Liability/Disclaimer of WarrantyMATLABⓇ and Simulink is a trademark of The MathWorks, Inc. and is used with permission. TheMathWorks does not warrant the accuracy of the text or exercises in this book. This work’s use or discussionof MATLABⓇ software or related products does not constitute endorsement or sponsorship by TheMathWorks of a particular pedagogical approach or particular use of the MATLABⓇ software.

While the publisher and authors have used their best efforts in preparing this work, they make norepresentations or warranties with respect to the accuracy or completeness of the contents of this work andspecifically disclaim all warranties, including without limitation any implied warranties of merchantabilityor fitness for a particular purpose. No warranty may be created or extended by sales representatives, writtensales materials or promotional statements for this work. The fact that an organization, website, or product isreferred to in this work as a citation and/or potential source of further information does not mean that thepublisher and authors endorse the information or services the organization, website, or product mayprovide or recommendations it may make. This work is sold with the understanding that the publisher is notengaged in rendering professional services. The advice and strategies contained herein may not be suitablefor your situation. You should consult with a specialist where appropriate. Further, readers should be awarethat websites listed in this work may have changed or disappeared between when this work was written andwhen it is read. Neither the publisher nor authors shall be liable for any loss of profit or any othercommercial damages, including but not limited to special, incidental, consequential, or other damages.

Library of Congress Cataloging-in-Publication data

Names: Rojo-Álvarez, José Luis, – author. | Martínez-Ramón, Manel, – author. |Muñoz-Marí, Jordi, author. | Camps-Valls, Gustau, – author.

Title: Digital signal processing with kernel methods / by Dr. José Luis Rojo-Álvarez,Dr. Manel Martínez-Ramón, Dr. Jordi Muñoz-Marí, Dr. Gustau Camps-Valls.

Description: First edition. | Hoboken, NJ : John Wiley & Sons, . | Includes bibliographicalreferences and index. |

Identifiers: LCCN (print) | LCCN (ebook) | ISBN (pdf) |ISBN (epub) | ISBN (cloth)

Subjects: LCSH: Signal processing–Digital techniques.Classification: LCC TK. (ebook) | LCC TK. .R (print) | DDC ./–dcLC record available at https://lccn.loc.gov/

Cover design by WileyCover image: ©AF-studio/Gettyimages

Set in /pt Warnock by SPi Global, Pondicherry, India

v

Contents

About the Authors xiiiPreface xviiAcknowledgements xxiList of Abbreviations xxiii

Part I Fundamentals and Basic Elements

1 From Signal Processing to Machine Learning . A New Science is Born: Signal Processing .. Signal Processing Before Being Coined .. : Birth of the Information Age .. s: Audio Engineering Catalyzes Signal Processing . From Analog to Digital Signal Processing .. s: Digital Signal Processing Begins .. s: Digital Signal Processing Becomes Popular .. s: Silicon Meets Digital Signal Processing . Digital Signal Processing Meets Machine Learning .. s: New Application Areas .. s: Neural Networks, Fuzzy Logic, and Genetic Optimization . Recent Machine Learning in Digital Signal Processing .. Traditional Signal Assumptions Are No Longer Valid .. Encoding Prior Knowledge .. Learning and Knowledge from Data .. From Machine Learning to Digital Signal Processing .. From Digital Signal Processing to Machine Learning

2 Introduction to Digital Signal Processing . Outline of the Signal Processing Field .. Fundamentals on Signals and Systems .. Digital Filtering .. Spectral Analysis .. Deconvolution .. Interpolation

vi Contents

.. System Identification .. Blind Source Separation . From Time–Frequency to Compressed Sensing .. Time–Frequency Distributions .. Wavelet Transforms .. Sparsity, Compressed Sensing, and Dictionary Learning . Multidimensional Signals and Systems .. Multidimensional Signals .. Multidimensional Systems . Spectral Analysis on Manifolds .. Theoretical Fundamentals .. Laplacian Matrices . Tutorials and Application Examples .. Real and Complex Signal Processing and Representations .. Convolution, Fourier Transform, and Spectrum .. Continuous-Time Signals and Systems .. Filtering Cardiac Signals .. Nonparametric Spectrum Estimation .. Parametric Spectrum Estimation .. Source Separation .. Time–Frequency Representations and Wavelets .. Examples for Spectral Analysis on Manifolds . Questions and Problems

3 Signal Processing Models . Introduction . Vector Spaces, Basis, and Signal Models .. Basic Operations for Vectors .. Vector Spaces .. Hilbert Spaces .. Signal Models .. Complex Signal Models .. Standard Noise Models in DSP .. The Role of the Cost Function .. The Role of the Regularizer . Digital Signal Processing Models .. Sinusoidal Signal Models .. System Identification Signal Models .. Sinc Interpolation Models .. Sparse Deconvolution .. Array Processing . Tutorials and Application Examples .. Examples of Noise Models .. Autoregressive Exogenous System Identification Models .. Nonlinear System Identification Using Volterra Models .. Sinusoidal Signal Models

Contents vii

.. Sinc-based Interpolation .. Sparse Deconvolution .. Array Processing . Questions and Problems .A MATLAB simpleInterp Toolbox Structure

4 Kernel Functions and Reproducing Kernel Hilbert Spaces . Introduction . Kernel Functions and Mappings .. Measuring Similarity with Kernels .. Positive-Definite Kernels .. Reproducing Kernel in Hilbert Space and Reproducing Property .. Mercer’s Theorem . Kernel Properties .. Tikhonov’s Regularization .. Representer Theorem and Regularization Properties .. Basic Operations with Kernels . Constructing Kernel Functions .. Standard Kernels .. Properties of Kernels .. Engineering Signal Processing Kernels . Complex Reproducing Kernel in Hilbert Spaces . Support Vector Machine Elements for Regression and Estimation .. Support Vector Regression Signal Model and Cost Function .. Minimizing Functional . Tutorials and Application Examples .. Kernel Calculations and Kernel Matrices .. Basic Operations with Kernels .. Constructing Kernels .. Complex Kernels .. Application Example for Support Vector Regression Elements . Concluding Remarks . Questions and Problems

Part II Function Approximation and Adaptive Filtering

5 A Support Vector Machine Signal Estimation Framework . Introduction . A Framework for Support Vector Machine Signal Estimation . Primal Signal Models for Support Vector Machine Signal Processing .. Nonparametric Spectrum and System Identification .. Orthogonal Frequency Division Multiplexing Digital Communications .. Convolutional Signal Models .. Array Processing . Tutorials and Application Examples

viii Contents

.. Nonparametric Spectral Analysis with Primal Signal Models .. System Identification with Primal Signal Model γ-filter .. Parametric Spectral Density Estimation with Primal Signal Models .. Temporal Reference Array Processing with Primal Signal Models .. Sinc Interpolation with Primal Signal Models .. Orthogonal Frequency Division Multiplexing with Primal Signal Models . Questions and Problems

6 Reproducing Kernel Hilbert Space Models for Signal Processing . Introduction . Reproducing Kernel Hilbert Space Signal Models .. Kernel Autoregressive Exogenous Identification .. Kernel Finite Impulse Response and the γ-filter .. Kernel Array Processing with Spatial Reference .. Kernel Semiparametric Regression . Tutorials and Application Examples .. Nonlinear System Identification with Support Vector

Machine–Autoregressive and Moving Average .. Nonlinear System Identification with the γ-filter .. Electric Network Modeling with Semiparametric Regression .. Promotional Data .. Spatial and Temporal Antenna Array Kernel Processing . Questions and Problems

7 Dual Signal Models for Signal Processing . Introduction . Dual Signal Model Elements . Dual Signal Model Instantiations .. Dual Signal Model for Nonuniform Signal Interpolation .. Dual Signal Model for Sparse Signal Deconvolution .. Spectrally Adapted Mercer Kernels . Tutorials and Application Examples .. Nonuniform Interpolation with the Dual Signal Model .. Sparse Deconvolution with the Dual Signal Model .. Doppler Ultrasound Processing for Fault Detection .. Spectrally Adapted Mercer Kernels .. Interpolation of Heart Rate Variability Signals .. Denoising in Cardiac Motion-Mode Doppler Ultrasound Images .. Indoor Location from Mobile Devices Measurements .. Electroanatomical Maps in Cardiac Navigation Systems . Questions and Problems

8 Advances in Kernel Regression and Function Approximation . Introduction . Kernel-Based Regression Methods .. Advances in Support Vector Regression .. Multi-output Support Vector Regression

Contents ix

.. Kernel Ridge Regression .. Kernel Signal-to-Noise Regression .. Semi-supervised Support Vector Regression .. Model Selection in Kernel Regression Methods . Bayesian Nonparametric Kernel Regression Models .. Gaussian Process Regression .. Relevance Vector Machines . Tutorials and Application Examples .. Comparing Support Vector Regression, Relevance Vector Machines, and

Gaussian Process Regression .. Profile-Dependent Support Vector Regression .. Multi-output Support Vector Regression .. Kernel Signal-to-Noise Ratio Regression .. Semi-supervised Support Vector Regression .. Bayesian Nonparametric Model .. Gaussian Process Regression .. Relevance Vector Machines . Concluding Remarks . Questions and Problems

9 Adaptive Kernel Learning for Signal Processing . Introduction . Linear Adaptive Filtering .. Least Mean Squares Algorithm .. Recursive Least-Squares Algorithm . Kernel Adaptive Filtering . Kernel Least Mean Squares .. Derivation of Kernel Least Mean Squares .. Implementation Challenges and Dual Formulation .. Example on Prediction of the Mackey–Glass Time Series .. Practical Kernel Least Mean Squares Algorithms . Kernel Recursive Least Squares .. Kernel Ridge Regression .. Derivation of Kernel Recursive Least Squares .. Prediction of the Mackey–Glass Time Series with Kernel Recursive Least

Squares .. Beyond the Stationary Model .. Example on Nonlinear Channel Identification and Reconvergence . Explicit Recursivity for Adaptive Kernel Models .. Recursivity in Hilbert Spaces .. Recursive Filters in Reproducing Kernel Hilbert Spaces . Online Sparsification with Kernels .. Sparsity by Construction .. Sparsity by Pruning . Probabilistic Approaches to Kernel Adaptive Filtering .. Gaussian Processes and Kernel Ridge Regression .. Online Recursive Solution for Gaussian Processes Regression

x Contents

.. Kernel Recursive Least Squares Tracker .. Probabilistic Kernel Least Mean Squares . Further Reading .. Selection of Kernel Parameters .. Multi-Kernel Adaptive Filtering .. Recursive Filtering in Kernel Hilbert Spaces . Tutorials and Application Examples .. Kernel Adaptive Filtering Toolbox .. Prediction of a Respiratory Motion Time Series .. Online Regression on the KINK Dataset .. The Mackey–Glass Time Series .. Explicit Recursivity on Reproducing Kernel in Hilbert Space

and Electroencephalogram Prediction .. Adaptive Antenna Array Processing . Questions and Problems

Part III Classification, Detection, and Feature Extraction

10 Support Vector Machine and Kernel Classification Algorithms . Introduction . Support Vector Machine and Kernel Classifiers .. Support Vector Machines .. Multiclass and Multilabel Support Vector Machines .. Least-Squares Support Vector Machine .. Kernel Fisher’s Discriminant Analysis . Advances in Kernel-Based Classification .. Large Margin Filtering .. Semi-supervised Learning .. Multiple Kernel Learning .. Structured-Output Learning .. Active Learning . Large-Scale Support Vector Machines .. Large-Scale Support Vector Machine Implementations .. Random Fourier Features .. Parallel Support Vector Machine .. Outlook . Tutorials and Application Examples .. Examples of Support Vector Machine Classification .. Example of Least-Squares Support Vector Machine .. Kernel-Filtering Support Vector Machine for Brain–Computer Interface

Signal Classification .. Example of Laplacian Support Vector Machine .. Example of Graph-Based Label Propagation .. Examples of Multiple Kernel Learning . Concluding Remarks . Questions and Problems

Contents xi

11 Clustering and Anomaly Detection with Kernels . Introduction . Kernel Clustering .. Kernelization of the Metric .. Clustering in Feature Spaces . Domain Description Via Support Vectors .. Support Vector Domain Description .. One-Class Support Vector Machine .. Relationship Between Support Vector Domain Description

and Density Estimation .. Semi-supervised One-Class Classification . Kernel Matched Subspace Detectors .. Kernel Orthogonal Subspace Projection .. Kernel Spectral Angle Mapper . Kernel Anomaly Change Detection .. Linear Anomaly Change Detection Algorithms .. Kernel Anomaly Change Detection Algorithms . Hypothesis Testing with Kernels .. Distribution Embeddings .. Maximum Mean Discrepancy .. One-Class Support Measure Machine . Tutorials and Application Examples .. Example on Kernelization of the Metric .. Example on Kernel k-Means .. Domain Description Examples .. Kernel Spectral Angle Mapper and Kernel Orthogonal

Subspace Projection Examples .. Example of Kernel Anomaly Change Detection Algorithms .. Example on Distribution Embeddings and Maximum Mean Discrepancy . Concluding Remarks . Questions and Problems

12 Kernel Feature Extraction in Signal Processing . Introduction . Multivariate Analysis in Reproducing Kernel Hilbert Spaces .. Problem Statement and Notation .. Linear Multivariate Analysis .. Kernel Multivariate Analysis .. Multivariate Analysis Experiments . Feature Extraction with Kernel Dependence Estimates .. Feature Extraction Using Hilbert–Schmidt Independence Criterion .. Blind Source Separation Using Kernels . Extensions for Large-Scale and Semi-supervised Problems .. Efficiency with the Incomplete Cholesky Decomposition .. Efficiency with Random Fourier Features .. Sparse Kernel Feature Extraction .. Semi-supervised Kernel Feature Extraction

xii Contents

. Domain Adaptation with Kernels .. Kernel Mean Matching .. Transfer Component Analysis .. Kernel Manifold Alignment .. Relations between Domain Adaptation Methods .. Experimental Comparison between Domain Adaptation Methods . Concluding Remarks . Questions and Problems

References Index

xiii

About the Authors

José Luis Rojo-Álvarez received the Telecommunication Engineering degree in from University of Vigo, Spain, and a PhD in Telecommunication Engineering in from the Polytechnic University of Madrid, Spain. Since , he has been a fullProfessor in the Department of Signal Theory and Communications, University ReyJuan Carlos, Madrid, Spain. He has published more than papers in indexed journalsand more than international conference communications. He has participated inmore than projects (with public and private fundings), and directed more than of them, including several actions in the National Plan for Research and FundamentalScience. He was a senior researcher at the Prometeo program in Ecuador (ArmyUniversity, to ) and research advisor at the Telecommunication Ministry. In he received the Rey Juan Carlos University Prize for Talented Researcher.

His main current research interests include statistical learning theory, digital signalprocessing, and complex system modeling, with applications to cardiac signals andimage processing. Specifically, he is committed to the development of new electrocar-diographic imaging systems, long-term cardiac monitoring intelligent systems, and bigdata for electronic recording and hospital information analysis at large scales.

He joined Persei vivarium, an eHealth company, as Chief Scientific Officer in. Currently, he is running a pioneer degree program on Biomedical Engineering,involving hospitals and companies in the electro-medicine and eHealth fields. In ,he also joined the Center for Computational Simulation (Universidad Politécnicade Madrid) for promoting eHealth technology transfer based on multivariate dataprocessing.

Manel Martínez-Ramón received an MsD in Telecommunications Engineering fromUniversitat Politècnica de Catalunya in , and a PhD in Communications Technolo-gies from Universidad Carlos III de Madrid (Spain) in . In he spent a -month postdoctoral period at the MIND Research Network (New Mexico, USA). Hewas an Associate Professor at Universidad Carlos III de Madrid until . There, heheld various positions from Associate Dean of the School of Engineering to AssociateVice-Chancellor for Infrastructures. He has taught more than different undergradu-ate and graduate classes in different universities.

Since August he has been a full professor with the Department of Electrical andComputer Engineering at the University of New Mexico, where he was permanentlyappointed Prince of Asturias Endowed Chair of the University of New Mexico, laterrenamed to King Felipe VI Endowed Chair, which is sponsored by the Household of

xiv About the Authors

the King of Spain. He is head of the machine learning track of this department andhe is the Associate Director of the Center of Emerging Energy Technologies of thisuniversity. He is currently a principal investigator of several projects funded by theNational Science Foundation and other agencies.

He has co-authored more than journal papers and about conference papers, andseveral books and book chapters. His research interests are in applications of machinelearning to cyberphysical systems, including first-responders systems, smart grids, andcognitive radio.

Jordi Muñoz-Marí was born in València, Spain, in , and received a BSc degreein Physics (), a BSc degree in Electronics Engineering (), and a PhD degreein Electronics Engineering () from the Universitat de València. He is currentlyan associate professor in the Electronics Engineering Department at the Universitatde València, where he teaches electronic circuits, digital signal processing, and datascience. He is a research member of the Image and Signal Processing (ISP) group. Hisresearch activity is tied to the study and development of machine-learning algorithmsfor signal and image processing.

Gustau Camps-Valls received BSc degrees in Physics () and in ElectronicsEngineering () and a PhD degree in Physics (), all from the Universitatde València. He is currently an Associate Professor (hab. Full Professor) in theDepartment of Electronics Engineering. He is a research coordinator in the Imageand Signal Processing (ISP) group. He is interested in the development of machine-learning algorithms for geoscience and remote-sensing data analysis. He is an authorof journal papers, more than conference papers, international bookchapters, and editor of the books Kernel Methods in Bioengineering, Signal and ImageProcessing (IGI, ), Kernel Methods for Remote Sensing Data Analysis" (JohnWiley & Sons, ), and Remote Sensing Image Processing (MC, ). He holdsa Hirsch’s index h = , entered the ISI list of Highly Cited Researchers in ,and Thomson Reuters ScienceWatch identified one of his papers on kernel-basedanalysis of hyperspectral images as a Fast Moving Front research. In , he obtainedthe prestigious European Research Council (ERC) consolidator grant on StatisticalLearning for Earth Observation Data Analysis. Since he has been a member of theData Fusion Technical Committee of the IEEE GRSS, and since of the MachineLearning for Signal Processing Technical Committee of the IEEE SPS. He is a memberof the MTG-IRS Science Team (MIST) of EUMETSAT. He is Associate Editor of theIEEE Transactions on Signal Processing, IEEE Signal Processing Letters, IEEE Geoscienceand Remote Sensing Letters, and invited guest editor for IEEE Journal of Selected Topicsin Signal Processing () and IEEE Geoscience and Remote Sensing Magazine ().

Valero Laparra Pérez-Muelas received a BSc degree in Telecommunications Engi-neering (), a BSc degree in Electronics Engineering (), a BSc degree inMathematics (), and a PhD degree in Computer Science and Mathematics ().Currently, he has a postdoctoral position in the Image Processing Laboratory (IPL) and

About the Authors xv

an Assistant Professor position in the Department of Electronics Engineering at theUniversitat de València.

Luca Martino obtained his PhD in Statistical Signal Processing from UniversidadCarlos III de Madrid, Spain, in . He has been an Assistant Professor in theDepartment of Signal Theory and Communications at Universidad Carlos III deMadrid since then. In August he joined the Department of Mathematics andStatistics at the University of Helsinki. In March , he joined the Universidadede Sao Paulo (USP). Currently, he is a postdoctoral researcher at the Universitat deValència. His research interests include Bayesian inference, Monte Carlo methods, andnonparametric regression techniques.

Sergio Muñoz-Romero earned his PhD in Machine Learning at Universidad CarlosIII de Madrid, where he also received the Telecommunication Engineering degree. Hehas led pioneering projects where machine-learning knowledge was successfully usedto solve real big-data problems. Currently, he is a researcher at Universidad Rey JuanCarlos. Since , he has worked at Persei vivarium as Head of Data Science and BigData. His present research interests are centered around machine-learning algorithmsand statistical learning theory, mainly in dimensionality reduction and feature selectionmethods, and their applications to bioengineering and big data.

Adrián Pérez-Suay obtained his BSc degree in Mathematics (), a Master’s degreein Advanced Computing and Intelligent Systems (), and a PhD degree in Com-putational Mathematics and Computer Science () about distance metric learning,all from the Universitat de València. He is currently a postdoctoral researcher at theImage Processing Laboratory (IPL) working on feature extraction and classificationproblems in remote-sensing data analysis, and has worked as assistant professor in theDepartment of Mathematics at the Universitat de València.

Margarita Sanromán-Junquera received the Technical Telecommunication Engineer-ing degree from Universidad Carlos III de Madrid, Spain, in , the Telecommunica-tion Engineering degree from Universidad Rey Juan Carlos, Spain, in , an MSc inBiomedical Engineering and Telemedicine from the Universidad Politécnica de Madrid,Spain, in , and a PhD in Multimedia and Communication from Universidad ReyJuan Carlos and Universidad Carlos III de Madrid, in . She is currently an AssistantProfessor in the Department of Signal Theory and Communications, Telematics, andComputing at Universidad Rey Juan Carlos. Her research interests include statisticallearning theory, digital processing of images and signals, and their applications tobioengineering.

Cristina Soguero-Ruiz received the Telecommunication Engineering degree and aBSc degree in Business Administration and Management in , and an MSc degree inBiomedical Engineering from the University Rey Juan Carlos, Madrid, Spain, in .She obtained her PhD degree in Machine Learning with Applications in Healthcare in in the Joint Doctoral Program in Multimedia and Communications in conjunctionwith University Rey Juan Carlos and University Carlos III. She was supported by FPUSpanish Research and Teaching Fellowship (granted in , third place in TEC area).

xvi About the Authors

She won the Orange Foundation Best PhD Thesis Award by the Spanish Official Collegeof Telecommunication Engineering.

Steven Van-Vaerenbergh received his MSc degree in Electrical Engineering fromGhent University, Belgium, in , and a PhD degree from the University of Cantabria,Santander, Spain, in . He was a visiting researcher with the Computational Neu-roengineering Laboratory, University of Florida, Gainesville, in . Currently, heis a postdoctoral associate with the Department of Telecommunication Engineering,University of Cantabria, Spain, where he is the principal researcher for a project onpattern recognition in time series. His current research interests include machinelearning, Bayesian statistics, and signal processing.

xvii

Preface

Why Did We Write This Book?

In we were finishing or had just finished our PhD theses in electronics andsignal processing departments in Spain. Each of us worked with complicated anddiverse datasets, ranging from the analysis of signals from patients in cooperation withhospitals, to satellite data imagery and antenna signals. All of us had grown in anacademic environment where neural networks were at the core of machine learning,and our theses also dealt with them. However, support vector machines (SVMs) had justarrived, and we enthusiastically adopted them. We were probably the Spanish pioneersusing them for signal processing. It took a bit to understand the fundamentals, butthen everything became crystal clear. It was a clean notation, a neat methodology,often involved straightforward implementations, and admitted many alternatives andmodifications. After understanding the SVM classification and regression algorithms(the two first ones that the kernel community delivered), we saw the enormous potentialfor writing other problems than maximum margin classifiers, and to accommodate theparticularities of signal and image features and models.

First, we started to write down some support vector algorithms for problems usingstandard signal models, the ones that we liked most, such as spectral analysis, decon-volution, system identification, or signal interpolation. Some concepts from both thesignal and the kernel worlds seemed to be naturally connected, including the conceptof signal autocorrelation, being closely related to the solid theory of reproducing kernelfunctions. Then, we started to send our brand-new algorithms to good machine-learning journals. Quite often, reviewers criticized that the approaches were trivial,and suggested resubmission to a signal-processing journal. And then, signal-processingreviewers apparently found no novelty whatsoever in redefining old concepts in kernelterms. It seemed that the clarity of the kernel methods methodology was playingagainst us, and everything was apparently obvious. Nevertheless, we were (and stillare) convinced that signal processing is much more than just filtering signals, and thatkernel methods are much more than maximum margin classifiers as well. Our visionwas that kernel methods should respect signal features and signal models as the onlyway to ensure model–data integration.

For years we worked in growing and designing kernel algorithms guided by therobustness requirements for the systems in our application fields. We studied other

xviii Preface

works around these fields, and some of them were really inspiring and useful in oursignal-processing problems. We even wrote some tutorials and reviews along theselines, aiming to put together the common elements of the kernel methods design undersignal-processing perspectives. However, we were not satisfied with the theoreticaltutorials, because our algorithms were designed according to our applications, andthe richness of the landscape given by the data was not reflected in these theoreticaltutorials, or even not fully conveyed by more application-oriented surveys. This is whywe decided to write a book that integrated the theoretical fundamentals, put togetherrepresentative application examples, and, if possible, to include code snippets and linksto relevant, useful toolboxes and packages. We felt that this could be a good way to helpthe reader work on theoretical fundamentals, while being inspired by real problems.This is, in some sense, the book we would have liked in the s for ourselves. Thisbook is not intended to be a set of tutorials, nor a set of application papers, and not just abunch of toolboxes. Rather, the book is intended to be a learning tour for those who likeand need to write their kernel algorithms, who need these algorithms for their signal-processing applications in real data, and who can be inspired by simple yet illustrativecode tutorials.

Needless to say, completing a book like this in the intersection of signal processing andkernel methods has been an enormous challenge. The literature of kernel methods insignal processing is vast, so we could not include all the excellent contributions workingin this cross-field during recent years. We tried our best in all chapters, though, byrevising the literature of what we feel are the main pillars and recent trends. The bookonly reflects our personal view and experience, though.

Structure and Contents

This book is divided into three parts: one for fundamentals, one focused on signalmodels for signal estimation and adaptive filtering, and another for classification,detection, and feature extraction. They are summarized next.

Part One: Fundamentals and Basic Elements

This consists of an introductory set of chapters and presents the necessary set of basicideas from both digital signal processing (DSP) and reproducing kernel Hilbert spaces(RKHSs). After an introductory road map (Chapter ), a basic overview of the field ofDSP is presented (Chapter ). Then, data models for signal processing are presented(Chapter ), on the one hand including well-known signal models (such as sinusoid-based spectral estimation, system identification, or deconvolution), and on the otherhand summarizing a set of additional fundamental concepts (such as adaptive filtering,noise, or complex signal models). Chapter consists of an introduction to kernelfunctions and Hilbert spaces, including the necessary concepts on RKHS and theirproperties for being used throughout the rest of the book. This chapter includes theelements of the SVM algorithm for regression, as an instantiation of a kernel algorithm,which in turn will be formally used as a founding optimization algorithm for thealgorithms in the next parts.

Preface xix

Part Two: Function Approximation and Adaptive Filtering

This presents a set of different SVM algorithms organized from the point of view ofthe signal model being used from Chapter and its role in the structured functionestimation. The key in this part is that SVM for estimation problems (not to be con-founded with the standard SVM for classification) raises a well-structured and foundedapproach to develop new general-purpose signal processing methods. Chapter startswith a simple and structured explanation of the three different sets of algorithms tobe addressed, which are primal signal models (linear kernel and signal model stated inthe primal problem, in the remainder of Chapter ), RKHS signal models (signal modelin the RKHS, conventional in kernel literature, to which Chapter is devoted), anddual signal models (signal model in the dual solution, closely related to function basisexpansion in signal processing, to which Chapter is devoted). These three chaptersrepresent the main axis along which the kernel trick is used to adapt the richness ofsignal processing model data, with emphasis on the idea that, far from using black-box nonlinear regression/classification model with an ad-hoc signal embedding model,one can actually adapt the SVM equations to the signal model from digital signalprocessing considerations on the structure of our data. Ending this part, Chapter provides an overview on the wide variety of kernel methods for signal estimationwhich can benefit from these proposed concepts for SVM regression, which includesuch widespread techniques as least-squares SVM, kernel signal-to-noise regression,or Bayesian approaches. A signal processing text for kernel methods in DSP must coverthe adaptive processing field, which after some initial basic proposals, seems to bereaching today a state of maturity in terms of theoretical fundamentals; all of them aresummarized in Chapter .

Part Three: Classification, Detection, and Feature Extraction

This presents a compendium of selected SVM algorithms for DSP which are notincluded in the preceding framework. Starting from the state of the art in SVMalgorithms for classification and detection problems in the context of signal processing,the rationale for this set of existing contributions is quite different from Part Two, giventhat likely the most fundamental concept of SVM classifiers, namely the maximummargin, holds in SVM classification approaches for signal processing. Chapter revisesthe conventional SVM classifier and its variants, introduces other kernel classifiersbeyond SVMs, and discusses particular advanced formulations to treat with semi-supervised, active, structured-output, and large-scale learning. Then, Chapter isdevoted specifically to clustering, anomaly detection, and one-class kernel classifiers,with emphasis in signal- and image-processing applications. Finally, Chapter is fullydevoted to the rich literature and theoretical developments on kernel feature extrac-tion, revisiting the classical taxonomy (unsupervised, supervised, semi-supervised, anddomain adaptation) from a DSP point of view.

Theory, Applications, and Examples

References on theoretical works and additional applications are stated and brieflycommented in each chapter. The book can be considered as self-contained, but still it

xx Preface

assumes some necessary previous and very basic knowledge on DSP. In the applicationexamples, references are given for the interested reader to be able to update or refreshthose concepts which are to be dealt with in each chapter.

Supporting material is also included in two forms. On the one hand, simple examplesfor fundamental concepts are delivered, so that the reader gains confidence and getsfamiliar with the basic concepts, some readers may find them trivial for some chapters,on the other hand, real and more advanced application examples are provided in severalchapters. Scripts, code, and pointers to toolboxes are mostly in MATLAB™. The sourcecode and examples can be downloaded from GitHub at the following link:

http://github.com/DSPKM/

In this dedicated repository, many links are maintained to other widely used softwaretoolboxes for machine learning and signal processing in kernel methods. This repositoryis periodically updated with the latest contributions, and can be helpful for the dataanalysis practitioner. The reader can use the code provided with this book for their ownresearch and analysis. We only ask that, in this case, the book is properly cited:

Digital Signal Processing with Kernel MethodsJosé Luis Rojo-Álvarez, Manel Martínez-Ramón, Jordi Muñoz-Marí, andGustau Camps-VallsJohn Wiley & Sons, .

When the code for the examples is simple and didactic, it is included in the text, sothat it can be examined, copied, and pasted. Those scripts and functions with increasedcomplexity are further delivered in the book repository. Owing to the multidisciplinarynature of the examples, they can be of different difficulty for each reader according tothe specific background, so that examples which seem extremely easy for some will beharder to work out for others. The reader is encouraged to spend some time with thecode with which they are more unfamiliar and to skip the examples which are alreadywell known.

xxi

Acknowledgements

We would like to acknowledge the help of all involved in the collation and review processof the book, without whose support the project could not have been satisfactorilycompleted. A further special note of thanks goes also to all the staff at John Wiley & Sons,Inc., whose contributions throughout the whole process, from inception of the initialidea to final publication, have been valuable. Special thanks also go to the publishingteam at John Wiley & Sons, Ltd., who continuously prodded via e-mail, keeping theproject on schedule. It has been really a pleasure to work with such a professional staff.

Our special thanks goes to the coauthors of several of the chapters, who made itpossible to cover material that has largely enriched those chapters, and who are listedin Table . We would like to express our deepest gratitude to them.

We also wish to thank all of the coauthors of the papers during these years. Withoutthem this work would not have been possible at all: J.C. Antoranz, J. Arenas-García,A. Artés-Rodríguez, T. Bandos, O. Barquero-Pérez, J. Bermejo, K. Borgwardt, L.Bruzzone, A.J. Caamaño-Fernández, S. Canú, C. Christodoulou, J. Cid-Sueiro, P. Conde-Pardo, E. Everss, J.R. Feijoo, M.J. Fernández-Getino, A.R. Figueiras-Vidal, C. Figuera,R. Flamary, A. García-Alberola, A. García-Armada, V. Gil-Jiménez, F.J. Gimeno-Blanes,L. Gómez-Chova, V. Gómez-Verdejo, A.E. Gonnouni, R. Goya-Esteban, A. Guerrero-Curieses, E. Izquierdo-Verdiguier, L.K. Hansen, R. Jenssen, M. Lázaro-Gredilla,N. Longbotham, J. Malo, J.D. Martín-Guerrro, M.P. Martínez-Ruiz, F. Melgani, I. Mora-Jiménez, N.M. Nasrabadi, A. Navia-Vázquez, A.A. Nielsen, F. Pérez-Cruz, K.B. Petersen,M. de Prado-Cumplido, A. Rakotomamonjy, I. Santamaría-Caballero, B. Schölkopf,E. Soria-Olivas, D. Tuia, J. Verrelst, J. Weston, R. Yotti, and D. Zhou.

This book was produced without any dedicated funding, but our research was partiallysupported by research projects that made it possible. We want to thank all agenciesand organizations for supporting our research in general, and this book indirectly.José Luis Rojo-Álvarez acknowledges support from the Spanish Ministry of Economyand Competitiveness, under projects PRINCIPIAS (TEC--C--R), FINALE(TEC--C--R), and KERMES (TEC--REDT), and from Comu-nidad de Madrid under project PRICAM (S/ICE-). Manel Martínez-Ramónacknowledges support from the Spanish Ministry of Economy and Competitivenessunder project TEC--R, from Comunidad de Madrid under project PRICAM(S/ICE-), and from the National Science Foundation under projects S&CC# and SBIR Phase II #. Jordi Muñoz-Marí acknowledges support fromthe Spanish Ministry of Economy and Competitiveness and the European RegionalDevelopment Fund, under project TIN--R, Gustau Camps-Valls and Luca

xxii Acknowledgements

Table 1 List of coauthors in specific chapters.

Margarita Sanromán-Junquera Universidad Rey Juan Carlos Chapters and Sergio Muñoz Romero Universidad Rey Juan Carlos Chapters , , and Cristina Soguero-Ruiz Universidad Rey Juan Carlos Chapters and Luca Martino Universitat de València Chapter Steven Van Vaerenbergh Universidad de Cantabria Chapter Adrián Pérez-Suay Universitat de València Chapter Valero Laparra Universitat de València Chapter

Martino acknowledge support by the European Research Council (ERC) under theERC-CoG- project . Steven Van Vaerenbergh is supported by the SpanishMinistry of Economy and Competitiveness, under PRISMA project (TEC--JIN).

José Luis Rojo-Álvarez, Manel Martínez-Ramón,Jordi Muñoz-Marí, and Gustau Camps-Valls

Leganés, Albuquerque, and València, December

xxiii

List of Abbreviations

ACD anomalous change detectionAL active learningALD approximate linear dependencyAP access pointAR autoregressiveARCH autoregressive conditional heteroscedasticityARMA autoregressive and moving averageARX autoregressive exogenousAUC area under the (ROC) curveAVIRIS Airborne Visible Infrared Imaging SpectrometerBCI brain–computer interfaceBER bit error rateBG Bernouilli–GaussianBRT bootstrap resampling techniquesBSS blind source separationBT breaking tiesCCA canonical correlation analysisCDMM color Doppler M modeCESNI continuous-time equivalent system for nonuniform interpolationCG conjugate gradientCI confidence intervalCNS cardiac navigation systemCOCO constrained covarianceCS compressive sensingCV cross-validationDCT discrete cosine transformDFT discrete Fourier transformDMGF double modulated Gaussian functionDOA direction of arrivalDSM dual signal modelDSP digital signal processingDWT discrete wavelet transformEAM electroanatomical mapEC elliptically contoured

xxiv List of Abbreviations

ECG electrocardiogramEEC error correction codeEEG electroencephalogramEM expectation–maximizationESD energy spectral densityFB-KRLS fixed budget kernel recursive least squaresFFT fast Fourier transformFIR finite impulse responseFT Fourier transformGM Gaussian mixtureGMM Gaussian mixture modelGP Gaussian processGPR Gaussian process regressionGRNN generalized regression neural networkHRCN high reliability communications networkHRV heart rate variabilityHSCA Hilbert–Schmidt component analysisHSIC Hilbert–Schmidt independence criterioni.i.d. independent and identically distributedICA independent component analysisICF incomplete Cholesky factorizationIIR infinite impulse responseIMSE integrated mean square errorIPM interior point methodIRWLS integrated reweighted least squaresKACD kernel anomaly change detectionKAF Kalman adaptive filteringKDE kernel density estimationKDR kernel dimensionality reductionKECA kernel entropy component analysisKEMA kernel manifold alignmentKF Kalman filterKFD kernel Fisher discriminantKGV kernel generalized varianceKICA kernel independent component analysisKKT Karush–Kuhn–TuckerKL Kullback–LeiblerKLMS kernel least mean squareskMI kernel mutual informationKMM kernel mean matchingKNLMS kernel normalized least mean squaresKOSP kernel orthogonal subspace projectionKPCA kernel principal component analysisKRLS kernel recursive least squaresKRLS-T kernel recursive least square trackerKRR kernel ridge regressionKSAM kernel spectral angle mapper

List of Abbreviations xxv

KSNR kernel signal-to-noise regression/ratioKTA kernel–target alignmentLapSVM Laplacian support vector machineLASSO least absolute shrinkage and selection operatorLDA linear discriminant analysisLFD linear Fisher discriminantLI linear interpolationLMF large margin filteringLMS least mean squaresLOO leave-one-outLS least squaresLS-SVM least-squares support vector machineLTI linear time invariantLUT look-up tableMA moving averageMAE mean absolute errorMAO most ambiguous and orthogonalMAP maximum a posterioriMCLU multiclass level uncertaintyMCMC Markov chain–Monte CarloMERIS medium resolution imaging spectrometerMIMO multiple input–multiple outputMKL multiple kernel learningML maximum likelihoodMLP multilayer perceptronMMD maximum mean discrepancyMMDE maximum mean discrepancy embeddingMMSE minimum mean square errorMNF minimum noise fractionMPDR minimum power distortionless responseMRI magnetic resonance imagingMS margin samplingMSE mean square errorMSSF modulated squared sinc functionMSVR multioutput support vector regressionMUSIC multiple signal classificationMVA multivariate analysisMVDR minimum variance distortionless responseNN neural networkNORMA naive online regularized risk minimization algorithmNW Nadayara–WatsonOA overall accuracyOAA one against allOAO one against oneOC-SVM one class support vector machineOFDM orthogonal frequency division multiplexingOKECA optimized kernel entropy component analysis

xxvi List of Abbreviations

OSP orthogonal subspace projectionPCA principal component analysisPCK probabilistic cluster kernelpdf probability density functionPLS partial least squaresPSD power spectral densityPSM primal signal modelPSVM parallel support vector machineQAM quadrature amplitude modulationQKLMS quantified kernel least mean squaresQP quadratic programmingQPSK quadrature-phase shift keyingRBF radial basis functionRHSIC randomized Hilbert–Schmidt independence criterionRKHS reproducing kernel in Hilbert spaceRKS random kitchen sinkRLS recursive least squaresRMSE root mean square errorROC receiver operating characteristicRSM reproducing kernel in Hilbert space signal modelRSS received signal strengthRV relevance vectorRVM relevance vector machineS/E signal to errorSAM spectral angle mapperSDP semi-definite programSE squared exponentialSMO sequential minimal optimizationSNR signal-to-noise ratioSOGP sparse online Gaussian processSOM self-organizing mapSR semiparametric regressionSRM structural risk minimizationSSL semisupervised learningSSMA semisupervised manifold alignmentSTFT short-time Fourier transformSVC support vector classificationSVD singular value decompositionSVDD support vector domain descriptionSVM support vector machineSVR support vector regressionSW-KRLS sliding window kernel recursive least squaresTCA transfer component analysisTFD time–frequency distributionTSVM transductive support vector machineWGP warped Gaussian processWGPR warped Gaussian process regression

1

Part I

Fundamentals and Basic Elements