Lecturer: Liqing ZhangDept. Computer Science & Engineering, Shanghai Jiao Tong UniversityStatistical Learning & Inference

Statistical Learning and Inference

*Statistical Learning and Inference*Books and ReferencesTrevor Hastie Robert Tibshirani Jerome Friedman , The Elements of statistical Learning: Data Mining, Inference, and Prediction, 2001, Springer-Verlag

V. Cherkassky & F. Mulier, Learning From Data, Wiley,1998Vladimir N. Vapnik, The Nature of Statistical Learning Theory, 2nd ed., Springer, 2000M. Vidyasagar, Learning and generalization: with applications to neural networks, 2nd ed., Springer, 2003G. Casella & R. Berger, Statistical Inference, Thomson, 2002T. Cover & J. Thomas, Elements of Information Theory, Wiley

Statistical Learning and Inference

*Statistical Learning and Inference*Overview of the CourseIntroductionOverview of Supervised LearningLinear Method for Regression and ClassificationBasis Expansions and RegularizationKernel MethodsModel Selections and InferenceSupport Vector MachineBayesian InferenceUnsupervised Learning

Statistical Learning and Inference

*Statistical Learning and Inference*Why Statistical Learning?---- R. Roger---- I. Hacking

Statistical Learning and Inference

Cloud Computing

Cloud Computing Service LayersDescriptionServices Complete business services such as PayPal, OpenID, OAuth, Google Maps, AlexaServicesApplicationFocused InfrastructureFocusedApplication Cloud based software that eliminates the need for local installation such as Google Apps, Microsoft OnlineStorage Data storage or cloud based NAS such as CTERA, iDisk, CloudNASDevelopment Software development platforms used to build custom cloud based applications (PAAS & SAAS) such as SalesForcePlatform Cloud based platforms, typically provided using virtualization, such as Amazon ECC, Sun GridHosting Physical data centers such as those run by IBM, HP, NaviSite, etc.

/

*Statistical Learning and Inference*ML: SARS Risk Prediction

Statistical Learning and Inference

*Statistical Learning and Inference*ML: Auto Vehicle Navigation

Statistical Learning and Inference

*Statistical Learning and Inference*Protein Folding

Statistical Learning and Inference

*Statistical Learning and Inference*The Scale of Biomedical Data

Statistical Learning and Inference

General Procedure in SL

EX. Pattern ClassificationObjective: To recognize horse in images

Procedure: Feature => Classifier => Cross+Valivation *Statistical Learning and Inference*

Statistical Learning and Inference

Classifier*Statistical Learning and Inference*Horse Non Horse

Statistical Learning and Inference

*Statistical Learning and Inference*Function Estimation ModelThe Function Estimation Model of learning examples:Generator (G) generates observations x (typically in Rn), independently drawn from some fixed distribution F(x)Supervisor (S) labels each input x with an output value y according to some fixed distribution F(y|x)Learning Machine (LM) learns from an i.i.d. l-sample of (x,y)-pairs output from G and S, by choosing a function that best approximates S from a parameterised function class f(x,), where is in the parameter set

Statistical Learning and Inference

*Statistical Learning and Inference*Function Estimation ModelKey concepts: F(x,y), an i.i.d. k-sample on F, functions f(x,) and the equivalent representation of each f using its index

Statistical Learning and Inference

*Statistical Learning and Inference*The loss functional (L, Q)the error of a given function on a given example

The risk functional (R)the expected loss of a given function on an example drawn from F(x,y) the (usual concept of) generalisation error of a given function The Problem of Risk Minimization

Statistical Learning and Inference

*Statistical Learning and Inference*The Problem of Risk MinimizationThree Main Learning ProblemsPattern Recognition:

Regression Estimation:

Density Estimation:

Statistical Learning and Inference

*Statistical Learning and Inference*General FormulationThe Goal of LearningGiven an i.i.d. k-sample z1,, zk drawn from a fixed distribution F(z)For a function class loss functionals Q (z ,), with in We wish to minimise the risk, finding a function *

Statistical Learning and Inference

*Statistical Learning and Inference*General FormulationThe Empirical Risk Minimization (ERM) Inductive PrincipleDefine the empirical risk (sample/training error):

Define the empirical risk minimiser:

ERM approximates Q (z ,*) with Q (z ,k) the Remp minimiserthat is ERM approximates * with kLeast-squares and Maximum-likelihood are realisations of ERM

Statistical Learning and Inference

*Statistical Learning and Inference*4 Issues of Learning TheoryTheory of consistency of learning processesWhat are (necessary and sufficient) conditions for consistency (convergence of Remp to R) of a learning process based on the ERM Principle?Non-asymptotic theory of the rate of convergence of learning processesHow fast is the rate of convergence of a learning process?Generalization ability of learning processesHow can one control the rate of convergence (the generalization ability) of a learning process?Constructing learning algorithms (i.e. the SVM)How can one construct algorithms that can control the generalization ability?

Statistical Learning and Inference

*Statistical Learning and Inference*Change in Scientific MethodologyTRADITIONAL

Formulate hypothesisDesign experimentCollect dataAnalyze resultsReview hypothesisRepeat/Publish

NEW

Design large experimentsCollect large dataPut data in large databaseFormulate hypothesisEvaluate hypothesis on databaseRun limited experiments Review hypothesisRepeat/Publish

Statistical Learning and Inference

*Statistical Learning and Inference*Learning & AdaptationAny method that incorporates information from training samples in the design of a classifier employs learning.Due to complexity of classification problems, we cannot guess the best classification decision ahead of time, we need to learn it.Creating classifiers then involves positing some general form of model, or form of the classifier, and using examples to learn the complete classifier.

Statistical Learning and Inference

*Statistical Learning and Inference*Supervised learningIn supervised learning, a teacher provides a category label for each pattern in a training set. These are then used to train a classifier which can thereafter solve similar classification problems by itself.Such as Face Recognition, Text Classification,

Statistical Learning and Inference

*Statistical Learning and Inference*Unsupervised learningIn unsupervised learning, or clustering, there is no explicit teacher or training data. The system forms natural clusters of input patterns and classifiers them based on clusters they belong to .

Data Clustering, Data Quantization, Dimensional Reduction,

Statistical Learning and Inference

*Statistical Learning and Inference*Reinforcement learningIn reinforcement learning, a teacher only says to classifier whether it is right when suggesting a category for a pattern. The teacher does not tell what the correct category is.

Agent, Robot,

Statistical Learning and Inference

*Statistical Learning and Inference*ClassificationThe task of the classifier component is to use the feature vector provided by the feature extractor to assign the object to a category.Classification is the main topic of this course.The abstraction provided by the feature vector representation of the input data enables the development of a largely domain-independent theory of classification.Essentially the classifier divides the feature space into regions corresponding to different categories.

Statistical Learning and Inference

*Statistical Learning and Inference*ClassificationThe degree of difficulty of the classification problem depends on the variability in the feature values for objects in the same category relative to the feature value variation between the categories.Variability is natural or is due to noise.Variability can be described through statistics leading to statistical pattern recognition.

Statistical Learning and Inference

*Statistical Learning and Inference*ClassificationQuestion: How to design a classifier that can cope with the variability in feature values? What is the best possible performance?Noise and Biological Variations Cause Class Spread

Classification error due to class overlap

Statistical Learning and Inference

ExamplesUser interfaces: modelling subjectivity and affect, intelligent agents, transduction (input from camera, microphone, or fish sensor) Recovering visual models: face recognition, model-based video, avatars Dynamical systems: speech recognition, visual tracking, gesture recognition, virtual instruments Probabilistic modeling: image compression, low bandwidth teleconferencing, texture synthesis *Statistical Learning and Inference*

Statistical Learning and Inference

*Statistical Learning and Inference*Course Webhttp://bcmi.sjtu.edu.cn/statLearnig/

Teaching Assistant: Liu YeEmail: lyyx000@yahoo.cn

Statistical Learning and Inference

AssignmentTo write a report on the topic you are working on, including: Problem definition Model and method Key issues to be solved Outcome *Statistical Learning and Inference*

Statistical Learning and Inference

*