Robust estimation of a mean [6pt] in a multivariate ... · Robust estimation of a mean in a...

Robust estimation of a mean

in a multivariate Gaussian model: Part 1

Frejus, December 17, 2018

Arnak S. Dalalyan

ENSAE ParisTech / CREST

1. Various models of contamination

3

General notation

We first introduce the notation that are common to all the models ofcontamination considered in this talk.

Number of observations : n.

Dimension of the unknown parameter µ∗: p.

Observations (X1, . . . ,Xn) ∼ P n.

Number of outliers (possibly random): s ∈ {1, . . . , n}.

Set of outliers: S ⊂ {1, . . . , n}.

Proportion of outliers: ε = E[s/n] = E[|S|/n].

Setting (informal)

Among the n observations X1, . . . ,Xn, there is a small number s ofoutliers. If we remove the outliers, all the other Xi’s are iid drawn froma reference distribution Pµ∗ .

Dalalyan, A.S. Dec 17, 2018 3

4

Gaussian model with unknown mean

Assumption (model for inliers)

Throughout this presentation, we assume that the reference distributionPµ∗ is p-variate Gaussian Np(µ∗, Ip). The goal is to estimate theparameter µ∗ ∈ Rp.

0.7 0.8 0.9 1 1.1 1.2 1.3 1.4 1.5 1.60.5

1

1.5

inliersoutlierscontour plot

n = 30, s = 5, µ∗ = [1, 1]>


4

Gaussian model with unknown mean

Assumption (model for inliers)

Throughout this presentation, we assume that the reference distributionPµ∗ is p-variate Gaussian Np(µ∗, Ip). The goal is to estimate theparameter µ∗ ∈ Rp.

0.7 0.8 0.9 1 1.1 1.2 1.3 1.4 1.5 1.60.5

1

1.5

observationscontour plot

n = 30, s = 5, µ∗ = [1, 1]>


5

Huber’s contamination

Assumption (HC model for outliers)

There are unobserved iid random variables Z1, . . . , Zn ∼ B(ε) and adistribution Q, such that

L (Xi|Zi = 0) = Np(µ∗, Ip), L (Xi|Zi = 1) = Q,

the observations Xi corresponding to different i’s are independent.This is equivalent to

P n ={

(1− ε)Np(µ∗, Ip) + εQ}⊗n

.

In this model,

S = {i : Zi = 1}︸︷︷︸set of outliers

and s ∼ B(n, ε)︸︷︷︸nb of outliers

are both random.


5






P n ={

(1− ε)Np(µ∗, Ip) + εQ}⊗n

.

In this model,



are both random.

i Zi Xi ∼

1 0 Np(µ∗, Ip)

2 0 Np(µ∗, Ip)

3 1 Q4 0 Np(µ

∗, Ip)5 1 Q6 0 Np(µ

∗, Ip)7 0 Np(µ

∗, Ip)

.

.

....

.

.

.30 0 Np(µ

∗, Ip)

s = 5 ∼B(30, 0.2)


5






P n ={

(1− ε)Np(µ∗, Ip) + εQ}⊗n

.

In this model,



are both random.

i Zi Xi ∼

1 0 Np(µ∗, Ip)

2 1 Q3 0 Np(µ

∗, Ip)4 0 Np(µ

∗, Ip)5 0 Np(µ

∗, Ip)6 0 Np(µ

∗, Ip)7 1 Q

.

.

....

.

.

.30 0 Np(µ

∗, Ip)

s = 6 ∼B(30, 0.2)


5






P n ={

(1− ε)Np(µ∗, Ip) + εQ}⊗n

.

In this model,



are both random.

We write

Pn ∈MHCn (p, ε,µ∗).

for the model of Huber’scontamination.


6

Huber’s deterministic contamination

Assumption (HDC model for outliers)

There is a set S ⊂ {1, . . . , n} of cardinality s = [nε] and a distributionQ, such that

{Xi : i ∈ Sc} iid∼ Np(µ∗, Ip) ⊥⊥ {Xi : i ∈ S} iid∼ Q.

Similar to HC: the outliers are iid.

Different from HC: the set of outliers is determenistic.

Remark The number of outliers s should be smaller than n/2,otherwise Q would be the reference distribution and Np(µ∗, Ip) thecontamination.

We write Pn ∈MHDCn (p, ε,µ∗).


7

Parameter contamination

Assumption (PC model for outliers)

There is a set S ⊂ {1, . . . , n} of cardinality s = [nε] and a collection ofvectors {µi : i ∈ S}, such that

{Xi : i ∈ Sc} iid∼ Np(µ∗, Ip) ⊥⊥ {Xi : i ∈ S} ∼⊗i∈SNp(µi, Ip).

Similar to HC & HDC: the outliers are independent.

Different from HC & HDC: the outliers might have differentdistributions.

We write Pn ∈MPCn (p, ε,µ∗).


8

Adversarial contamination

Assumption (AC model for outliers)

For a sequence Y iiid∼ Np(µ∗, Ip), i = 1, . . . , n, and a random set

S ⊂ {1, . . . , n} of cardinality s = [nε] we have

Xi = Y i, ∀i ∈ Sc.

The set S is not independent of {Y i : i = 1, . . . , n}.

The observations {Xi : i ∈ S} may have arbitrary dependencestructure.

We write Pn ∈MACn (p, ε,µ∗).


9

Relation between the models

MHCn (p, ε,µ∗)

MHDCn (p, 2ε,µ∗)

MPCn (p, 2ε,µ∗)

MACn (p, 2ε,µ∗)


2. Problem formulation and overview of results

11

Historical approachBreakdown point

Assume the unknown parameter µ∗ is in Rp.

Let µ̂ be an estimator of µ∗. Thus,

µ̂ :

∞⋃n=1

Xn → Rp.

The breakdown point ε∗n of µ̂ is defined by

ε∗n =1

nmin

{s ∈ {1, . . . , n} : sup

y1,...,ys

‖µ̂(x1:(n−s),y1:s)‖ = +∞}.

Drawbacks:

does not take into account the impact of “mild” outliers,meaningless if the parameter space is bounded,does not depend on the norm under consideration,...


12

Minimax approachIn expectation

A more informative way of quantifying the robustness is theevaluation of the worst-case risk and its comparison to theminimax risk.

Worst-case risk of an estimator µ̂n:

R?n,p,ε(µ̂n) = sup

µ∗sup

Pn∈M?n(p,ε,µ∗)

EX∼Pn [‖µ̂n(X)− µ∗‖22].

Here,M?n(p, ε,µ∗) is one of the 4 models of contamination

considered in previous slides.For instance, RHC

n,p,ε(µ̂n) is the minimax risk for Huber’scontamination model.

Minimax risk:R?

n,p,ε = infµ̂n

R?n,p,ε(µ̂n).


12




RHCn,p,ε(µ̂n) = sup

µ∗sup

Pn∈MHCn (p,ε,µ∗)

EX∼Pn[‖µ̂n(X)− µ∗‖22].




Minimax risk:RHC

n,p,ε = infµ̂n

RHCn,p,ε(µ̂n).


12




RHDCn,p,ε(µ̂n) = sup

µ∗sup

Pn∈MHDCn (p,ε,µ∗)

EX∼Pn[‖µ̂n(X)− µ∗‖22].




Minimax risk:RHDC

n,p,ε = infµ̂n

RHDCn,p,ε(µ̂n).


12




RPCn,p,ε(µ̂n) = sup

µ∗sup

Pn∈MPCn (p,ε,µ∗)

EX∼Pn[‖µ̂n(X)− µ∗‖22].




Minimax risk:RPC

n,p,ε = infµ̂n

RPCn,p,ε(µ̂n).


12




RACn,p,ε(µ̂n) = sup

µ∗sup

Pn∈MACn (p,ε,µ∗)

EX∼Pn[‖µ̂n(X)− µ∗‖22].




Minimax risk:RAC

n,p,ε = infµ̂n

RACn,p,ε(µ̂n).


13

Minimax approachIn deviation

Most results in the literature provide bounds on the deviation, notfor the expectation.

Fix a confidence level δ ∈ (0, 1).

Worst-case deviation of an estimator µ̂n: r?n,p,ε(µ̂n) is solution to

minimize r

subject to PX∼Pn

(‖µ̂n(X)− µ∗‖22 > r

)≤ δ

∀µ∗ ∈ Rp, ∀P n ∈M?n(p, ε,µ∗).

Clearly, r?n,p,ε(µ̂n) depends on δ, but we will not be interested inthis dependence.

Minimax risk:r?n,p,ε = inf

µ̂n

r?n,p,ε(µ̂n).

Tchebychev’s inequality yields δr?n,p,ε(µ̂n) ≤ R?n,p,ε(µ̂n).


14

Common robust estimators of the mean

The most common robust estimators of the mean are perhaps thecoordinatewise median, the geometric median and the Huber’sestimator.

All these estimators can be defined as an M -estimator:

µ̂n ∈ arg minµ∈Rp

n∑i=1

Ψ(Xi − µ)

with

Ψ(x) =

‖x‖1, coordinatewise median,‖x‖2, geometric median,‖x‖22

2 ∧ λ(‖x‖2 − 0.5λ), Huber’s estimator.

In all the three cases, the function Ψ is convex and the estimatoris computable in polynomial time.


15

Overview of the resultsMinimax rates in deviation


15



15



16

Overview of the resultsTractable estimators

We will present three tractable estimators that improve on thecoordinatewise median.

1 The ellipsoid method (Diakonikolas et al., 2016).

2 The spectral method (Lai et al., 2016).

3 The iterative soft thresholding (Collier and Dalalyan, 2017).


3. The minimax rate

18

Minimax lower bound

Theorem 1 (Chen et al., 2015)

There is a constant c > 0 such that for every ε ∈ [0, 1] and everyδ ∈ (0, 1/2), it holds that

rHCn,p,ε ≥ c

(p

n+ ε2

).

Some remarks

By Tchebychev’s inequality, pn + ε2 is also a lower bound for the

minimax risk in expectation.

By inclusion, pn + ε2 is also a lower bound for the minimax risk in

models HDC and AC.

The same lower bound pn + ε2 holds true for the model PC.


19

Proof of the lower bound 1

1 From the classic parametric minimax theory: rHCn,p,ε &

pn .

2 Thus, we need only to show that rHCn,p,ε & ε

2.

3 Main steps of the proof:

Reduction to dimension 1: rHCn,p,ε ≥ rHC

n,1,ε.Construct a probability density function fε such that

f⊗nε ∈MHCn (1, ε, 0)

f⊗nε ∈MHCn (1, ε,∆ε)

with ∆ε � ε.

Parameter values µ∗ = 0 and µ∗ = ∆ε are indistinguishablefrom the observations X1, . . . ,Xn ∼ f⊗nε .

Therefore rHCn,p,ε & ‖∆ε − 0‖22 � ε2.


20


For a ∆ > 0, define f◦∆ = ϕ0 ∨ ϕ∆.


20


For a ∆ > 0, define f◦∆ = ϕ0 ∨ ϕ∆.

We have S∆ =∫f◦∆(x) dx = 1 + a∆ +O(∆2).


20


For a ∆ > 0, define f◦∆ = ϕ0 ∨ ϕ∆.

We have S∆ =∫f◦∆(x) dx = 1 + a∆ +O(∆2).

Then, f∆ = f◦∆/S∆ is a pdf.


20


For a ∆ > 0, define f◦∆ = ϕ0 ∨ ϕ∆.

We have S∆ =∫f◦∆(x) dx = 1 + a∆ +O(∆2).

Then, f∆ = f◦∆/S∆ is a pdf.

We choose ∆ε so that 1/S∆ε= 1− ε

and set fε = f∆ε .


21


fε = (1− ε)(ϕ0 ∨ ϕ∆ε)

ll ll

(1− ε)ϕ0

+ +

εq

fε =(1− ε)ϕ0 + εq

(1− ε)ϕ∆ε+ εq′

.


21


fε = (1− ε)(ϕ0 ∨ ϕ∆ε)

ll ll

(1− ε)ϕ0

+ +

εq

fε =(1− ε)ϕ0 + εq

(1− ε)ϕ∆ε+ εq′

.


21


fε = (1− ε)(ϕ0 ∨ ϕ∆ε)

ll ll

(1− ε)ϕ0

+ +

εq

fε =(1− ε)ϕ0 + εq

(1− ε)ϕ∆ε+ εq′

.


21


fε = (1− ε)(ϕ0 ∨ ϕ∆ε)

ll ll

(1− ε)ϕ0

+ +

εq

fε =(1− ε)ϕ0 + εq

(1− ε)ϕ∆ε+ εq′

.


22

Minimax upper bound

Theorem 2 (Chen et al., 2015)There are two constants C1, C2 > 0 such that

for every ε ≤ 1/5for every p ≤ C1nfor every δ ≥ e−C1n,

it holds that

rHCn,p,ε ≤ C2

(p

n+ ε2 +

log 1/δ

n

).

Some remarks:

The upper bound is attained by Tukey’s median.

The condition ε ≤ 1/5 can be replaced by ε ≤ 1/3− c′, with anarbitrarily small c′ > 0.

The estimator does not rely on the knowledge of ε.


23

Tukey’s median

The upper bound is attained byTukey’s median.

Tukey’s median is any maximaizer ofTukey’s depth:

µ̂TMn ∈ arg max

µ∈Rp

D(µ, {X1:n}).

Tukey’s (halfspace) depth is

D(µ,X1:n) = minu∈S1

n∑i=1

1(u>Xi ≤ u>µ).

µ̂TMn is computationally intractable for

large p.


23

Tukey’s median



µ̂TMn ∈ arg max

µ∈Rp

D(µ, {X1:n}).



n∑i=1

1(u>Xi ≤ u>µ).


large p.

depth of µ in X1:10?


23

Tukey’s median



µ̂TMn ∈ arg max

µ∈Rp

D(µ, {X1:n}).



n∑i=1

1(u>Xi ≤ u>µ).


large p.

depth of µ in X1:10?


23

Tukey’s median



µ̂TMn ∈ arg max

µ∈Rp

D(µ, {X1:n}).



n∑i=1

1(u>Xi ≤ u>µ).


large p.

depth of µ in theforest ?


23

Tukey’s median



µ̂TMn ∈ arg max

µ∈Rp

D(µ, {X1:n}).



n∑i=1

1(u>Xi ≤ u>µ).


large p.

D(µ,X1:n) = 1.


23

Tukey’s median



µ̂TMn ∈ arg max

µ∈Rp

D(µ, {X1:n}).



n∑i=1

1(u>Xi ≤ u>µ).


large p.

depth of µ in theforest ?


23

Tukey’s median



µ̂TMn ∈ arg max

µ∈Rp

D(µ, {X1:n}).



n∑i=1

1(u>Xi ≤ u>µ).


large p.

D(µ,X1:n) = 3.


24

Summary

We introduced four models of contamination by outliers:

Huber’s contaminationMHCn (p, ε,µ∗).

Huber’s deterministic contaminationMHDCn (p, ε,µ∗).

Parameter contaminationMPCn (p, ε,µ∗).

Adversarial contaminationMACn (p, ε,µ∗).

We have defined the worst case risks in expectation and indeviation, R?

n,p,ε(µ̂) and r?n,p,ε(µ̂).

We have defined the minimax risks R?n,p,ε = infµ̂R

?n,p,ε(µ̂) .

For every ε < 1/3−�, we have r?n,p,ε �pn + ε2.

This minimax rate is obtained by Tukey’s median, which is hard tocompute for large p.

QuestionWhat is the smallest rate of the worst-case risk that can be obtained byan estimator computable in poly(n, p, 1/ε) time?


24

Summary








We have defined the minimax risks r?n,p,ε = infµ̂ r?n,p,ε(µ̂) .





24

Summary













24

Summary













25

References I

M. Chen, C. Gao, and Z. Ren. Robust Covariance and Scatter MatrixEstimation under Huber’s Contamination Model. ArXiv e-prints, to appear inthe Annals of Statistics, 2015.

Olivier Collier and Arnak S. Dalalyan. Minimax estimation of amultidimensional linear functional in sparse gaussian models and robustestimation of the mean. submitted 1712.05495, arXiv, December 2017.URL https://arxiv.org/abs/1712.05495.

Ilias Diakonikolas, Gautam Kamath, Daniel M. Kane, Jerry Li, Ankur Moitra,and Alistair Stewart. Robust estimators in high dimensions without thecomputational intractability. In IEEE 57th Annual Symposium onFoundations of Computer Science, FOCS 2016, USA, pages 655–664,2016.

K. A. Lai, A. B. Rao, and S. Vempala. Agnostic estimation of mean andcovariance. In 2016 IEEE 57th Annual Symposium on Foundations ofComputer Science (FOCS), pages 665–674, Oct 2016.


https://arxiv.org/abs/1712.05495

Date post:	11-Jul-2020
Category:	Documents
Upload:	others
View:	4 times
Download:	0 times

Robust estimation of a mean [6pt] in a multivariate ... · Robust estimation of a mean in a...

Documents