Isaac Newton Institute - Cambridge

transcript

UNC, Stat & OR

Isaac Newton Institute - CambridgeIsaac Newton Institute - Cambridge

Object Oriented Data Analysis

J. S. Marron

Dept. of Statistics and Operations

Research, University of North Carolina

April 21, 2023

UNC, Stat & OR

Personal Opinions on Mathematical Personal Opinions on Mathematical StatisticsStatistics

What is Mathematical Statistics?

Validation of existing methods

Asymptotics (n ∞) & Taylor

expansion

Comparison of existing methods

(requires hard math, but

really “accounting”???)

UNC, Stat & OR

What could Mathematical Statistics be?

Basis for invention of new methods

Complicated data mathematical

Do we value creativity?

Since we don’t do this, others do…

(where are the ₤₤₤s???)

UNC, Stat & OR

Since we don’t do this, others do…

Pattern Recognition

Artificial Intelligence

Neural Nets

Data Mining

Machine Learning

UNC, Stat & OR

Possible Litmus Test:

Creative Statistics

Clinical Trials Viewpoint:

Worst Imaginable Idea

Mathematical Statistics Viewpoint:

UNC, Stat & OR

Object Oriented Data Analysis, IObject Oriented Data Analysis, I

What is the “atom” of a statistical analysis?

1st Course: Numbers

Multivariate Analysis Course : Vectors

Functional Data Analysis: Curves

More generally: Data Objects

UNC, Stat & OR

Object Oriented Data Analysis, IIObject Oriented Data Analysis, II

Examples:

Medical Image Analysis

Images as Data Objects?

Shape Representations as Objects

Micro-arrays

Just multivariate analysis?

UNC, Stat & OR

Object Oriented Data Analysis, IIIObject Oriented Data Analysis, III

Typical Goals:

Understanding population variation

Visualization

Principal Component Analysis +

Discrimination (a.k.a. Classification)

Time Series of Data Objects

UNC, Stat & OR

Object Oriented Data Analysis, IVObject Oriented Data Analysis, IV

Major Statistical Challenge, I:

High Dimension Low Sample Size (HDLSS)

Dimension d >> sample size n

“Multivariate Analysis” nearly useless Can’t “normalize the data”

Land of Opportunity for Statisticians Need for “creative statisticians”

UNC, Stat & OR

Object Oriented Data Analysis, VObject Oriented Data Analysis, V

Major Statistical Challenge, II:

Data may live in non-Euclidean space Lie Group / Symmet’c Spaces (manifold

Trees/Graphs as data objects

Interesting Issues: What is “the mean” (pop’n center)?

How do we quantify “pop’n variation”?

UNC, Stat & OR

Statistics in Image Analysis, IStatistics in Image Analysis, I

First Generation Problems:

Denoising

Segmentation

Registration

(all about single images)

UNC, Stat & OR

Statistics in Image Analysis, IIStatistics in Image Analysis, II

Second Generation Problems:

Populations of Images

Understanding Population Variation

Discrimination (a.k.a. Classification)

Complex Data Structures (& Spaces)

HDLSS Statistics

UNC, Stat & OR

HDLSS Statistics in Imaging

Why HDLSS (High Dim, Low Sample Size)?

Complex 3-d Objects Hard to Represent Often need d = 100’s of parameters

Complex 3-d Objects Costly to Segment Often have n = 10’s cases

UNC, Stat & OR

Medical Imaging – A Challenging Medical Imaging – A Challenging ExampleExample

Male Pelvis Bladder – Prostate – Rectum How do they move over time (days)? Critical to Radiation Treatment

(cancer) Work with 3-d CT Very Challenging to Segment

Find boundary of each object? Represent each Object?

UNC, Stat & OR

Male Pelvis – Raw DataMale Pelvis – Raw Data

One CT Slice

(in 3d

image)

Coccyx

(Tail Bone)

Rectum

Prostate

UNC, Stat & OR

Prostate:

manual segmentation

Slice by slice

Reassembled

UNC, Stat & OR

Prostate:

Slices:Reassembled in 3d

How to represent?

Thanks: Ja-Yeon Jeong

UNC, Stat & OR

Object RepresentationObject Representation

Landmarks (hard to find) Boundary Rep’ns (no

correspondence) Medial representations

Find “skeleton” Discretize as “atoms” called M-reps

UNC, Stat & OR

3-d m-reps3-d m-reps

Bladder – Prostate – Rectum (multiple objects, J. Y. Jeong)

• Medial Atoms provide “skeleton”

• Implied Boundary from “spokes” “surface”

UNC, Stat & OR

3-d m-reps3-d m-reps

M-rep model fitting

• Easy, when starting from binary (blue)

• But very expensive (30 – 40 minutes technician’s time)

• Want automatic approach

• Challenging, because of poor contrast, noise, …

• Need to borrow information across training sample

• Use Bayes approach: prior & likelihood posterior

• ~Conjugate Gaussians, but there are issues:

• Major HLDSS challenges

• Manifold aspect of data

UNC, Stat & OR

PCA for m-reps, IPCA for m-reps, I

Major issue: m-reps live in(locations, radius and angles)

E.g. “average” of: = ???

Natural Data Structure is:Lie Groups ~ Symmetric spaces

(smooth, curved manifolds)

)2()3(3 SOSO

359,358,3,2

UNC, Stat & OR

PCA for m-reps, IIPCA for m-reps, II

PCA on non-Euclidean spaces?(i.e. on Lie Groups / Symmetric Spaces)

T. Fletcher: Principal Geodesic Analysis

Idea: replace “linear summary of data”With “geodesic summary of data”…

UNC, Stat & OR

PGA for m-reps, Bladder-Prostate-PGA for m-reps, Bladder-Prostate-RectumRectum

Bladder – Prostate – Rectum, 1 person, 17 days

PG 1 PG 2 PG 3

(analysis by Ja Yeon Jeong)

UNC, Stat & OR

PG 1 PG 2 PG 3

UNC, Stat & OR

PG 1 PG 2 PG 3

UNC, Stat & OR

HDLSS Classification (i.e. HDLSS Classification (i.e. Discrimination)Discrimination)

Background: Two Class (Binary) version:

Using “training data” from Class +1, and from Class -1

Develop a “rule” for assigning new data to a Class

Canonical Example: Disease Diagnosis New Patients are “Healthy” or “Ill” Determined based on measurements

UNC, Stat & OR

HDLSS Classification (Cont.)HDLSS Classification (Cont.)

Ineffective Methods: Fisher Linear Discrimination Gaussian Likelihood Ratio

Less Useful Methods: Nearest Neighbors Neural Nets

(“black boxes”, no “directions” or intuition)

UNC, Stat & OR

Currently Fashionable Methods: Support Vector Machines Trees Based Approaches

New High Tech Method Distance Weighted Discrimination

(DWD) Specially designed for HDLSS data Avoids “data piling” problem of SVM Solves more suitable optimization problem

UNC, Stat & OR

Currently Fashionable Methods:

Trees Based ApproachesSupport Vector Machines:

UNC, Stat & OR

Distance Weighted DiscriminationDistance Weighted Discrimination

Maximal Data Piling

UNC, Stat & OR

Distance Weighted DiscriminationDistance Weighted Discrimination

Based on Optimization Problem:

More precisely work in appropriate penalty for violations

Optimization Method (Michael Todd): Second Order Cone Programming Still Convex gen’tion of quadratic

prog’ing Fast greedy solution Can use existing software

i ibw r1,

UNC, Stat & OR

DWD Bias Adjustment for MicroarraysDWD Bias Adjustment for Microarrays

Microarray data: Simult. Measur’ts of “gene

expression” Intrinsically HDLSS

Dimension d ~ 1,000s – 10,000s Sample Sizes n ~ 10s – 100s

My view: Each array is “point in cloud”

UNC, Stat & OR

DWD Batch and Source AdjustmentDWD Batch and Source Adjustment

For Perou’s Stanford Breast Cancer Data Analysis in Benito, et al (2004)

Bioinformaticshttps://genome.unc.edu/pubsup/dwd/

Adjust for Source Effects Different sources of mRNA

Adjust for Batch Effects Arrays fabricated at different times

UNC, Stat & OR

DWD Adj: Raw Breast Cancer dataDWD Adj: Raw Breast Cancer data

UNC, Stat & OR

DWD Adj: Source ColorsDWD Adj: Source Colors

UNC, Stat & OR

DWD Adj: Batch ColorsDWD Adj: Batch Colors

UNC, Stat & OR

DWD Adj: Biological Class ColorsDWD Adj: Biological Class Colors

UNC, Stat & OR

DWD Adj: Biological Class Colors & DWD Adj: Biological Class Colors & SymbolsSymbols

UNC, Stat & OR

DWD Adj: Biological Class SymbolsDWD Adj: Biological Class Symbols

UNC, Stat & OR

DWD Adj: Source ColorsDWD Adj: Source Colors

UNC, Stat & OR

DWD Adj: PC 1-2 & DWD directionDWD Adj: PC 1-2 & DWD direction

UNC, Stat & OR

DWD Adj: DWD Source AdjustmentDWD Adj: DWD Source Adjustment

UNC, Stat & OR

DWD Adj: Source Adj’d, PCA viewDWD Adj: Source Adj’d, PCA view

UNC, Stat & OR

DWD Adj: Source Adj’d, Class ColoredDWD Adj: Source Adj’d, Class Colored

UNC, Stat & OR

DWD Adj: Source Adj’d, Batch ColoredDWD Adj: Source Adj’d, Batch Colored

UNC, Stat & OR

DWD Adj: Source Adj’d, 5 PCsDWD Adj: Source Adj’d, 5 PCs

UNC, Stat & OR

DWD Adj: S. Adj’d, Batch 1,2 vs. 3 DWDDWD Adj: S. Adj’d, Batch 1,2 vs. 3 DWD

UNC, Stat & OR

DWD Adj: S. & B1,2 vs. 3 AdjustedDWD Adj: S. & B1,2 vs. 3 Adjusted

UNC, Stat & OR

DWD Adj: S. & B1,2 vs. 3 Adj’d, 5 PCsDWD Adj: S. & B1,2 vs. 3 Adj’d, 5 PCs

UNC, Stat & OR

DWD Adj: S. & B Adj’d, B1 vs. 2 DWDDWD Adj: S. & B Adj’d, B1 vs. 2 DWD

UNC, Stat & OR

DWD Adj: S. & B Adj’d, B1 vs. 2 Adj’dDWD Adj: S. & B Adj’d, B1 vs. 2 Adj’d

UNC, Stat & OR

DWD Adj: S. & B Adj’d, 5 PC viewDWD Adj: S. & B Adj’d, 5 PC view

UNC, Stat & OR

DWD Adj: S. & B Adj’d, 4 PC viewDWD Adj: S. & B Adj’d, 4 PC view

UNC, Stat & OR

DWD Adj: S. & B Adj’d, Class ColorsDWD Adj: S. & B Adj’d, Class Colors

UNC, Stat & OR

DWD Adj: S. & B Adj’d, Adj’d PCADWD Adj: S. & B Adj’d, Adj’d PCA

UNC, Stat & OR

DWD Bias Adjustment for Microarrays

Effective for Batch and Source Adj. Also works for cross-platform Adj.

E.g. cDNA & Affy Despite literature claiming contrary

“Gene by Gene” vs. “Multivariate” views

Funded as part of caBIG“Cancer BioInformatics Grid”

“Data Combination Effort” of NCI

UNC, Stat & OR

Interesting Benchmark Data SetInteresting Benchmark Data Set

NCI 60 Cell Lines Interesting benchmark, since same cells Data Web available:

http://discover.nci.nih.gov/datasetsNature2000.jsp

Both cDNA and Affymetrix Platforms

8 Major cancer subtypes

Use DWD now for visualization

UNC, Stat & OR

NCI 60: Views using DWD Dir’ns (focus on NCI 60: Views using DWD Dir’ns (focus on biology)biology)

UNC, Stat & OR

DWD in Face Recognition, I

Face Images as Data

(with M. Benito & D. Peña)

Registered using

landmarks

Male – Female Difference?

Discrimination Rule?

UNC, Stat & OR

DWD in Face Recognition, II

DWD Direction

Good separation

Images “make

sense”

Garbage at ends?

(extrapolation

effects?)

UNC, Stat & OR

Blood vessel tree dataBlood vessel tree data

Marron’s brain:

Segmented from

Reconstruct trees

Rotate to view

UNC, Stat & OR

Marron’s brain:

Segmented from

Reconstruct trees

Rotate to view

UNC, Stat & OR

Marron’s brain:

Segmented from

Reconstruct trees

Rotate to view

UNC, Stat & OR

Marron’s brain:

Segmented from

Reconstruct trees

Rotate to view

UNC, Stat & OR

Marron’s brain:

Segmented from

Reconstruct trees

Rotate to view

UNC, Stat & OR

Marron’s brain:

Segmented from

Reconstruct trees

Rotate to view

UNC, Stat & OR

Now look over many people (data

objects)

Structure of population (understand

variation?)

PCA in strongly non-Euclidean Space???

, ... ,,

UNC, Stat & OR

Possible focus of analysis:

• Connectivity structure only (topology)

• Location, size, orientation of segments

• Structure within each vessel segment

, ... ,,

UNC, Stat & OR

Present Focus:

Topology only

Already

challenging

Later address

others

Then add

attributes

To tree nodes

And extend

analysis

UNC, Stat & OR

Strongly Non-Euclidean Strongly Non-Euclidean SpacesSpaces

Statistics on Population of Tree-Structured Data Objects?

• Mean???• Analog of PCA???

Strongly non-Euclidean, since:• Space of trees not a linear space• Not even approximately linear

(no tangent plane)

UNC, Stat & OR

Strongly Non-Euclidean Strongly Non-Euclidean SpacesSpaces

PCA on Tree Space?

Key Idea (Jim Ramsay):

• Replace 1-d subspace

that best approximates data

• By 1-d representation

that best approximates data

Wang and Marron (2007) define notion of

Treeline (in structure space)

UNC, Stat & OR

PCA for blood vessel tree PCA for blood vessel tree datadata

Data Analytic Goals: Age, Gender

these?

UNC, Stat & OR

Preliminary Tree-Curve Preliminary Tree-Curve ResultsResults

First Correlation

OfStructure

To Age!

(BackTrees)

UNC, Stat & OR

HDLSS Asymptotics

Why study asymptotics?

UNC, Stat & OR

HDLSS Asymptotics

An interesting (naïve) quote:

“I don’t look at asymptotics, because

I don’t have an infinite sample size”

UNC, Stat & OR

HDLSS Asymptotics

An interesting (naïve) quote:

“I don’t look at asymptotics, because

I don’t have an infinite sample size”

Suggested perspective:

Asymptotics are a tool for finding simple

structure underlying complex entities

UNC, Stat & OR

HDLSS Asymptotics

Which asymptotics?

n ∞ (classical, very widely

d ∞ ???

Sensible?

Follow typical “sampling process”?

Say anything, as noise level

increases???

UNC, Stat & OR

HDLSS Asymptotics

Which asymptotics?

n ∞ & d ∞

n >> d: a few results around

(still have classical info in data)

n ~ d: random matrices (Iain J., et al)

(nothing classically estimable)

HDLSS asymptotics: n fixed, d ∞

UNC, Stat & OR

HDLSS Asymptotics

UNC, Stat & OR

HDLSS Asymptotics

Microarrays: # genes bounded

Proteomics, SNPs, …

A moot point, from perspective:

Asymptotics are a tool for finding

simple structure underlying complex

entities

UNC, Stat & OR

HDLSS Asymptotics

increases???

UNC, Stat & OR

HDLSS Asymptotics

increases???

Yes, there exists simple, perhaps

surprising, underlying structure

UNC, Stat & OR

HDLSS Asymptotics: Simple Paradoxes, I

For dim’al “Standard Normal” dist’n:

Euclidean Distance to Origin (as ):

- Data lie roughly on surface of sphere of radius

- Yet origin is point of “highest density”???

- Paradox resolved by:

“density w. r. t. Lebesgue Measure”

Z ,0~1

)1(pOdZ

UNC, Stat & OR

HDLSS Asymptotics: Simple Paradoxes, II

For dim’al “Standard Normal” dist’n: indep. of

Euclidean Dist. between and (as ):Distance tends to non-random constant:

Can extend to Where do they all go???

(we can only perceive 3 dim’ns)

dd INZ ,0~2

)1(221 pOdZZ

nZZ ,...,1

UNC, Stat & OR

HDLSS Asymptotics: Simple Paradoxes, III

For dim’al “Standard Normal” dist’n: indep. of

High dim’al Angles (as ):

- -“Everything is orthogonal”??? - Where do they all go???

(again our perceptual limitations) - Again 1st order structure is non-random

dd INZ ,0~2

)(90, 2/121

dOZZAngle p

UNC, Stat & OR

HDLSS Asy’s: Geometrical Representation, I

Assume , let

Study Subspace Generated by Data

a. Hyperplane through 0, of dimension

b. Points are “nearly equidistant to 0”, & dist

c. Within plane, can “rotate towards Unit Simplex”

d. All Gaussian data sets are“near Unit Simplex Vertices”!!!

“Randomness” appears only in rotation of simplex

d ddn INZZ ,0~,...,1

With P. Hall & A. Neeman

UNC, Stat & OR

HDLSS Asy’s: Geometrical Representation, II

Assume , let

Study Hyperplane Generated by Data

a. dimensional hyperplane

b. Points are pairwise equidistant, dist

c. Points lie at vertices of “regular hedron”

d. Again “randomness in data” is only in rotation

e. Surprisingly rigid structure in data?

d ddn INZZ ,0~,...,1

UNC, Stat & OR

HDLSS Asy’s: Geometrical Representation, III

Simulation View: shows “rigidity after rotation”

UNC, Stat & OR

HDLSS Asy’s: Geometrical Representation, III

Straightforward Generalizations:

non-Gaussian data: only need moments

non-independent: use “mixing conditions” (with P. Hall & A. Neeman)

Mild Eigenvalue condition on Theoretical Cov. (with J. Ahn, K. Muller & Y. Chi)

All based on simple “Laws of Large Numbers”

UNC, Stat & OR

HDLSS Asy’s: Geometrical Representation, IV

Explanation of Observed (Simulation) Behavior:

“everything similar for very high d”

2 popn’s are 2 simplices (i.e. regular n-

hedrons) All are same distance from the other class i.e. everything is a support vector i.e. all sensible directions show “data piling” so “sensible methods are all nearly the same” Including 1 - NN

UNC, Stat & OR

HDLSS Asy’s: Geometrical Representation, V

Further Consequences of Geometric Representation

1. Inefficiency of DWD for uneven sample size(motivates “weighted version”, work in progress)

2. DWD more “stable” than SVM(based on “deeper limiting distributions”)(reflects intuitive idea “feeling sampling

variation”)(something like “mean vs. median”)

3. 1-NN rule inefficiency is quantified.

UNC, Stat & OR

2nd Paper on HDLSS Asymptotics

Ahn, Marron, Muller & Chi (2007) Biometrika Assume 2nd Moments (and Gaussian)

Assume no eigenvalues too large in sense:

For assume i.e.

(min possible)

(much weaker than previous mixing conditions…)

)(1 do 1 d

UNC, Stat & OR

HDLSS Math. Stat. of PCA, I

Consistency & Strong Inconsistency:

Spike Covariance Model (Johnstone & Paul)

For Eigenvalues:

1st Eigenvector:

How good are empirical versions,

as estimates?

1,,1, ,,2,1 dddd d

1,,1 ˆ,ˆ,,ˆ uddd

UNC, Stat & OR

HDLSS Math. Stat. of PCA, II

Consistency (big enough spike):

Strong Inconsistency (spike not big enough):

0ˆ, 11 uuAngle

011 90ˆ, uuAngle

UNC, Stat & OR

HDLSS Math. Stat. of PCA, III

Consistency of eigenvalues?

Eigenvalues Inconsistent

But known distribution

Unless as well

,1,1̂

UNC, Stat & OR

HDLSS Work in Progress, II

Canonical Correlations: Myung Hee Lee

Results similar to those for those for

Singular values inconsistent

But directions converge under a much

milder spike assumption.

UNC, Stat & OR

HDLSS Work in Progress, III

Conditions for Geo. Rep’n & PCA Consist.:

John Kent example:

Can only say:

not deterministic

Conclude: need some flavor of mixing

dddddd ININX *100,02

212/1212/1

pwddOX p

UNC, Stat & OR

Conclude: need some flavor of mixing

Challenge: Classical mixing conditions

require notion of time ordering

Not always clear, e.g. microarrays

UNC, Stat & OR

Sungkyu Jung Condition:

Define:

Assume: Ǝ a permutation,

So that is ρ-mixing

ddX ,0~ tdddd UU

dtddd XUZ 2/1

100100

UNC, Stat & OR

HDLSS Deep Open Problem

In PCA Consistency:

Strong Inconsistency - spike

Consistency - spike

What happens at boundary

( )???

101101

UNC, Stat & OR

The Future of HDLSS Asymptotics?

1. Address your favorite statistical problem…

2. HDLSS versions of classical optimality

results?

3. Continguity Approach (~Random Matrices)

4. Rates of convergence?

5. Improved Discrimination Methods?

It is early days…

102102

UNC, Stat & OR

Some Carry Away Lessons

Atoms of the Analysis: Object Oriented

Viewpoint: Object Space Feature Space

DWD is attractive for HDLSS classification

“Randomness” in HDLSS data is only in rotations

(Modulo rotation, have constant simplex shape)

How to put HDLSS asymptotics to work?

Isaac Newton Institute - Cambridge

Documents