Sum of SquaresA lens on average case complexity
Tensor PCA (for 4-tensors)
Tensor PCAgiven: a 4-tensor !: # $ → ℝ,
! = ( ⋅ *⊗$ +-!.,0,1,ℓ = *.*0*1*ℓ +-.,0,1,ℓ where
unknown signal * ∈ ℝ4, * = 1 and -.,0,1,ℓ ∼ 778 9:;<<7:#goal: recover the signal * ∈ ℝ4
( : controls the signal-to-noise ratio.
Tensor PCA (for 4-tensors)
TensorPCAgiven: a 4-tensor !: # $ → ℝ,
! = ( ⋅ *⊗$ +-where
unknown signal * ∈ℝ/, * = 1 and -2,3,4,ℓ ∼ 778 9:;<<7:#goal: recover * ∈ ℝ/
( ∶ <7>#:?#@7<A B:C7@D@EF
;C:C7@#:?D@E
F?A*7CG
IMPOSSIBLE
COMPUTATIONALLY
EASY
Sum
of S
quar
es D
egre
eDifficult to reason about
Tensor PCA (for 4-tensors)
! ∶ #$%&'(&)$#* +',$)-)./
0,',$)&'(-).
/(*1$,2
IMPO
SSIBLE
COM
PUTATIONALLY
EASY
Sum
of S
quar
es D
egre
eSum of squares:
• subsumes several well-known algorithmic techniques
• conjectured to be optimal in worst case settings.
• a sum of square lower bound almost always implies no known algorithm.
Signal strength vs Degree: Tensor PCA
IMP
OS
SIB
LE
! = #Θ %&'&
n%
De
gre
e o
f S
oS
pro
of
'Upper bound:[R-Rao-Schramm 16, Bhattiprolu-Guruswami-Lee 16]
Lower bound: [Hopkins-Kothari-Potechin-R-Schramm-S 17]
Tensor PCA in any dimensions,
Random CSPs
[R-Rao-Schramm 16, Kothari-Mori-O’Donnell-
Witmer]
Wish List
• Community Detection in stochastic block models.• Properties of random graphs: independent set, chromatic number,
vertex expansion …• Spiked Wigner models• Sparse PCA• Compressed Sensing• Mixtures models such as mixtures of Gaussians……………
Goal: Characterize optimal algorithms and thresholds for inference problems, and demonstrate the computational phase transitions predicted via statistical physics techniques.
SoS Degree Upper Boundfor TensorPCA
:[R-Rao-Schramm 16, Bhattiprolu-Guruswami-Lee 16]
TensorPCAgiven: a 4-tensor !: # $ → ℝ,
! = ( ⋅ *⊗$ +-!.,0,1,ℓ = *.*0*1*ℓ +-.,0,1,ℓ where
unknown signal * ∈ ℝ4, * = 1 and -.,0,1,ℓ ∼ 778 9:;<<7:#goal: recover * ∈ ℝ4
Natural approach to recover * approximately,
Maximize ! * = ∑.,0,1,ℓ !.,0,1,ℓ*.*0*1*ℓ
(Injective tensor norm)For a 4-tensor ! ∶ n $ → ℝ,
! .40 = maxC:‖C‖EF
G.,0,1,ℓ
!.,0,1,ℓ *.*0*1*ℓ = ⟨!, *⊗$⟩
Tensor Norm Certification
Certification Problem:given: a 4-tensor ! ∶ n $ → ℝ, !',),*,ℓ ∼ --. /0122-03goal: efficiently compute an upper bound 4(!) for ‖!‖'8) = max=:| = |@A⟨!, C
⊗$⟩
(Injective tensor norm)For a 4-tensor ! ∶ n $ → ℝ,
! '8) = max=:‖=‖@A F',),*,ℓ
!',),*,ℓ C'C)C*Cℓ = ⟨!, C⊗$⟩
Lies at the heart of random constraint satisfaction problems like random 3-SAT
By standard !-net argument," #$% ≤ ' (
with probability > 0.99for a random tensor ".
However, given a random tensor ", efficient algorithms are only known to certify upper bounds of O(n).
Typical TensorsGoal: upper bound for max
0 12⟨4, 6⊗8⟩
Spectral Upper Bound
Let ! be a symmetric "⊗$-dimensional tensor.
%&,& %&,(%(,&
%&,)
%),& %),)
⋯
⋯⋮
⋮⋱A =
Let / be the "(×"( matrix flattening of !.
Goal: upper bound for max4 5&
⟨!, 7⊗$⟩
Spectral Upper Bound
For any unit vector ! ∈ ℝ$,
&',' &',(
&(,'
&',$
&$,' &$,$
⋯
⋯⋮
⋮⋱A =
⟨/, !⊗1⟩ = ! ⊗ ! 34 ! ⊗ !
≤ max9∈ℝ:;
<34<<3<
= ‖4‖ = Θ ? w.h.p.
Recall that w.h.p., ‖/‖@AB ≤ C( ?).
∑/HIJℓ!H!I!J!ℓ
Goal: upper bound for maxL M'
⟨/, !⊗1⟩
Proof can be rewritten as degree 4 sum of squares
Random ?(×?(Gaussian matrix
Tensoring & Symmetrization
! "#$ = max) *+
,⊗., 0,⊗. 1+2
= max) *+
(,⊗.1)56⊗1 ,⊗.1+1
≤ 6⊗1+1
Symmetric under
permutations of dimensions {1, … , 2<}
= ‖6‖
Tensoring & Symmetrization
! "#$ = max) *+
,⊗., 0,⊗. 1+2
= max) *+
(,⊗.1)56 1 ,⊗.1+1
(avg of 7⊗1 over row & column symmetries)
≤ 6 1+1
M 1 = :;,< = ∘ 7⊗1 ∘ ?
Degree 4k SoS proof
Spectral Bound
! " #$ " ≤ &' ()
Theorem [R-Rao-Schramm, Bhattiprolu-Venkatesan]
!(+) -$-+-.-/, 1$1+1.1/= 1
4!+(6 -$-+1$1+ ⋅ 6 -.-/1.1/ + 6 -$-+1$1. ⋅ 6 -.-/1+1/ + … . . )
!(") = random (+"×(+" matrix whose entries have variance $"!
A Simple Spectral Algorithm
1. Compute a matrix ! " whose entries are polynomials of input:
! " ≔ $%,' (⊗"%,'
2. Compute the spectrum of !(") :
Output ! " ,- . ≥ 0 123
Runtime: 45 "eigenvalue of 4"×4" matrix
Theorem: [Hopkins-Kothari-Potechin-R-Schramm-S]For all problems satisfying a certain robust-inference property,
If SoS meta algorithm succeeds then a simpler spectral algorithmsucceeds.
SoS Degree Lower Bounds
Schema for a Lower boundPlanted Distribution
Given: ! = # ⋅ %⊗' +)unknown signal % ∈ ℝ,, % = 1 and )/,0,1,ℓ ∼ 445 6789947:
Fix degree 5. Suppose SoS algorithm recovers x, then
it solves a convex optimization problem <=``degree d SoS SDP relaxation”
whose solution yields %
Convex optimization problem >?“degree d SoS SDP relaxation”
is feasible & x is essentially only solution
Suppose degree d-SoS recovers x,
Schema for a Lower boundPlanted Distribution
Given: ! = # ⋅ %⊗' +)unknown signal % ∈ ℝ,, % = 1 and )/,0,1,ℓ ∼ 445 6789947:
Convex optimization problem ;<“degree d SoS SDP relaxation”
is feasible& x is essentially only solution
Suppose degree d-SoS recovers x,
Null Distribution (no signal)! = )
)/,0,1,ℓ ∼ 445 6789947:
Convex optimization problem ;<“degree d SoS SDP relaxation”
is infeasible
Contradict this!
Schema for a Lower boundPlanted Distribution
Given: ! = # ⋅ %⊗' +)unknown signal % ∈ ℝ,, % = 1 and )/,0,1,ℓ ∼ 445 6789947:
Null Distribution (no signal)! = )
)/,0,1,ℓ ∼ 445 6789947:
Lower bound strategy:Exhibit a feasible solution to the convex optimization
problem ;< <=> < ?@? ?AB C=DEFEGH@I for inputs from the null distribution.
Constructions by hand, technical & difficult to generalize • Random CSPs [Grigoriev, Schoenebeck, Tulsiani, Barak-Kothari, Kothari-Mori-O’Donnell-Witmer]
• Planted Clique [Meka-Potechin-Wigderson, Hopkins-Kothari-Potechin-R-Schramm, Montanari-Deshpande]
Pseudocalibration [Barak-Hopkins-Kothari-Kelner-Moitra-Potechin]
Planted Distribution! = # ⋅ %⊗' +)
unknown signal % ∈ ℝ,, % = 1 and )/,0,1,ℓ ∼ 445 6789947:
Null Distribution (no signal)! = )
)/,0,1,ℓ ∼ 445 6789947:
-- a general candidate construction of SoS SDP solutions for null distributions
Signal % : a solution to ;< solutions for null distribution
=: ?@9ABCA5 DA:9?B ! → 94F:7G %
ΠI ∘ = = projection of F onto low-degree polynomials in T
ΠI ∘ =(!) -- candidate SoS SDP solution
Pros: Systematic and general, recovers several existing constructions in a unified manner.Cons: Construction is still conjectural in general, proof of feasibility is extremely technical and difficult to generalize.
A bold conjecture:For broad families of statistical estimation problems,
low-degree (degree D) SoS semidefinite program distinguishes between the two distributions
if and only if,low-degree polynomials (degree ! "#$ %) distinguish between them.
Pseudocalibration
Planted Distribution& = ( ⋅ *⊗, +.
unknown signal * ∈ ℝ1, * = 1 and .4,5,6,ℓ ∼ 99: ;<=>>9<%
Null Distribution (no signal)& = .
.4,5,6,ℓ ∼ 99: ;<=>>9<%
-- a general candidate construction of SoS SDP solutions for null distributions
A bold conjecture:For broad families of statistical estimation problems,
low-degree (degree D) SoS semidefinite program distinguishes between the two distributions
if and only if,low-degree polynomials (degree ! "#$ %) distinguish between them.
Pseudocalibration-- a general candidate construction of SoS SDP solutions for null distributions
• Yields precise predictions on the performance of SoS SDPs for statistical estimation problems
• Predictions coincide with conjectured thresholds of computational phase transition in community detection and related models [Hopkins-Steurer]
Conclusions
• Sum of squares algorithms open an avenue to develop a general theory of efficient statistical estimation.
• Pseudocalibration potentially demonstrate computational phase transitions formally?
! has entries corresponding to elements of " #!$%&'
cadb
( = *+,- !⊗/ +,-
!$%&'
!⊗/01
cadb
cadbcadbcadbcadb
2 3!⊗/ has entries of the form
For example,!⊗4 5678, 9:;ℎ = !$%,&' ⋅ !>?,@A
Visualizing
a
jib
w
tsx
c
ℓkd
y
vuz
ab
w
tsx
cd
y
vuz
ji
ℓk
ab
w
tsx
cd
y
vuz
ji
ℓkAvg + +⋯+'(,* ≔
,⊗. → '0 1
1
2 0 2
average over permutation of
hyperedgematching from 1
→ 2
1
2
1 2
M . = 56,7 0 ∘ ,⊗. ∘ 9
' :;:<:=:>, ?;?<?=?>
=14!
<(D :;:<?;?< ⋅ D :=:>?=?> + D :;:<?;?= ⋅ D :=:>?<?> + … . . )
! " = "⊗%, !
= "⊗%', !⊗%' (/'
= "⊗*', +⊗' "⊗*',- where
+ isa6*×6* matrixflatteningof !,
i.e.,+ AB, Cℓ = !EF'ℓ ∀A, B, C, ℓ ∈ [6]
"⊗*' ∈ ℝ L M- is invariant under permutation of the NOPQR, i.e., action of the permutations on 1, … , 2C
= "⊗*', V ∘ +⊗' ∘ X "⊗*',- for every pair of permutations V, X ∈ YZ
= "⊗*', [\,] V ∘ +⊗' ∘ X "⊗*',-
≤ [_,` V ∘ +⊗' ∘ X
a/Z
! " = "⊗%, !
= ⟨"⊗(, ) "⊗(⟩ where) isa3(×3( matrixflatteningof !,i.e.,) >?, @ℓ = !BCDℓ ∀>, ?, @, ℓ ∈ [3]
≤ " ( )
≤ ) for " ≤ 1
! BKC ≤ )
A Spectral Bound
! " = "⊗%, !
= "⊗%', !⊗%' (/'
= "⊗*', +⊗' "⊗*',- where
!*,* isa6*×6* matrixflatteningof !,i.e.,!*,* AB, Cℓ = !EF'ℓ ∀A, B, C, ℓ ∈ [6]
≤ "⊗*' *+⊗'
,-
≤ +⊗' (/' for " ≤ 1= +
SoS Degree Lower BoundsApproach: Given tensor !Construct a linear functional "# ∶ ℝ &', … , &*
+, → ℝ such that,PropertyI:(largeonpolynomialT)"# ! & > B ⋅ "# 1Property II (non-negative on squares)"#[F
G] ≥ 0 ∀ F ∈ ℝ &',… , &*+,
Property III (respects constraints on x)"#[ 1 − & G ⋅ NG & ] ≥ 0 ∀ N ∈ ℝ &',… , &*
+,
B − ! & = ∑ℓ FℓG & + ∑ℓ NℓG & (1 − & G)"# ∘ "# ∘ "# ∘
< 0(Property I)
≥ 0(Property II)
≥ 0(Property III)
SoS Degree Lower BoundsApproach: Given tensor !Construct a linear functional "# ∶ ℝ &', … , &* +, → ℝ such that,PropertyI:(largeonpolynomialT)"# ! & > B ⋅ "# 1Property II (non-negative on squares)"#[FG] ≥ 0 ∀ F ∈ ℝ &',… , &* +,
Property III (respects constraints on x)"#[ 1 − & G ⋅ NG & ] ≥ 0 ∀ N ∈ ℝ &',… , &* +,
If O& is a solution to system O& G ≤ 1 and ! O& ≥ BThen the functional given by evaluation at O&,
"[F & ] = F(O&)
"# F = R OS∼UV[F( O&)]But there are no actual solutions to the system { & G ≤ 1, ! & ≥ B}, we are looking for a pseudodistribution Y
Pseudocalibration
• a candidate construction of the linear functional !" for sum-of-squares lower bounds.
• Discovered in the context of proving lower bounds for the planted clique problem.
• Still conjectural, but yields a valid linear functional for various problems like planted clique, tensor PCA
• Yields predictions on precise thresholds at which sum-of-squares SDPs fail, for various statistical estimation problems like community detection. The predicted thresholds coincide with conjectures arising from statistical physics considerations.
Pseudocalibration
Null distribution: ! random Gaussian tensor
given: tensor T
goal: construct a linear functional
"#:ℝ &', … , &* +, → ℝsuch that:
"# ! & ≥ / ⋅ "# '"# 1 − & 3 ⋅ 43 & ≥ 0 ∀ 4
"# 73(&) ≥ 0 ∀7
Planted distribution: ! = &⊗< +> random Gaussian tensor
Real
Sum of squares proofsSuppose ! " = ∑%&'ℓ !%&'ℓ "%"&"'"ℓ
) − ! " = ∑ℓ+ℓ, " + ∑ℓ-ℓ, " (1 − " ,)
123 45 1-26781 123 45 1-26781 ⋅ 1 − " ,
degree of the proof = max. deg { +ℓ, " , -ℓ,(")(1 − " ,)}
For each = ∈ ℕ,)@ ! = inf{ ) | ) − ! " ≥ 0 ℎ61 6 =8H788 = I4I +7445}
lim@→M
)@ ! = ! %N&
Finding degree =- sum of squares proof takes OP(@) computational time.
! − # $
= ! $ & − # $ + ! (1 − $ &)
= $⊗,, ! ⋅ /0 − ! $⊗, + ! (1 − $ &)
For every positive semidefinite matrix 1, and indeterminates 22,12 = 345 67 3849:;3 <6=>(2)
= 345 67 3849:;3 <6=> + 345 67 3849:;3 1 − $ , ≥ 0
SoS captures the spectral bound