DUALITY

WHY DUALITY?

minimize f(x)No constraints

Gradient descent Newton’s method

Quasi-newtonConjugate gradients

etc…

Constrained problems?minimize f(x)

subject to g(x) 0

h(x) = 0

????

minimize f(x)Non-differentiable

????

LAGRANGIAN

minimize f(x)

subject to Ax� b = 0

Simple case

minx

max�

f(x) + h�, Ax� bi

Lagrangian

“Saddle-point” form

SADDLE-POINT FORM

minimize f(x)

subject to Ax� b = 0

Simple case

minx

max�

f(x) + h�, Ax� bi

Lagrangian

“Saddle-point” form

max�

f(x) + h�, Ax� bi =(f(x), Ax� b = 0

1, otherwise

minx

max�

f(x) + h�, Ax� bi = minx

f(x)

INEQUALITY CONSTRAINTSminimize f(x)

subject to Ax� b = 0

Cx� d 0

Why does this work?non-negativeconstraint

minx

max�,⌫�0

f(x) + h�, Ax� bi+ h⌫, Cx� di

Cx� d 0 : ⌫ = 0

GENERAL FORM LAGRANGIAN

minimize f(x)

subject to g(x) 0

h(x) = 0

L(x,�, ⌫) = f(x) + h�, h(x)i+ h⌫, g(x)i

Lagrangian

f(x?) = minx

max�,⌫�0

f(x) + h�, h(x)i+ h⌫, g(x)iSaddle-point formulation

CALCULUS INTERPRETATION

linearconstraint

minimize f(x)

subject to h(x) = 0

�rf(x) = rh(x)� rf(x) +rh(x)� = 0

gradient of objective parallel to gradient of constraint

contours ofobjective

CALCULUS INTERPRETATIONminimize f(x)

subject to h(x) = 0

�rf(x) = rh(x)� rf(x) +rh(x)� = 0

minx

L(x,�) = f(x) + h�, h(x)i

@xL(x,�) = rf(x) +rh(x)� = 0

optimality condition

gradient of objective parallel to gradient of constraint

INEQUALITY CONDITIONSminimize f(x)

subject to g(x) 0

L(x, ⌫) = f(x) + h⌫, g(x)i

minx

max⌫�0

f(x) + h⌫, g(x)i

x? x?

Inactive: Active: ⌫ = 0 ⌫ > 0

INEQUALITY CONDITIONS

Inactive: Active: ⌫ = 0

⌫ > 0

@xL(x,�) = rf(x) +rg(x)⌫ = 0

OPTIMALITY CONDITIONS:KKT SYSTEMminimize f(x)

subject to g(x) 0

h(x) = 0L(x,�, ⌫) = f(x) + h�, h(x)i+ h⌫, g(x)i

Karush-Kuhn-TuckerPrimal/dual optimality

Primal feasibility

Dual feasibilityComp slackness

}rf(x?) + �rh(x?) + ⌫rg(x?) = 0

g(x?) 0

h(x?) = 0

⌫ � 0

⌫igi(x?) = 0

necessary conditions

DUAL FUNCTIONd(�, ⌫) = min

xL(x,�, ⌫)

= minx

f(x) + h�, h(x)i+ h⌫, g(x)i

�?Dual is concave

Why?Does this depend on convexity of f ?

DUAL FUNCTIONd(�, ⌫) = min

xL(x,�, ⌫)

= minx

f(x) + h�, h(x)i+ h⌫, g(x)i

�?Dual is lower bound to optimal objective

Why?

minx

f(x) + h�, h(x)i+ h⌫, g(x)i f(x?)

(because optimal x satisfies constraints)

GEOMETRIC INTERPRETATIONOF LOWER BOUND

minimize f(x)

subject to g(x) 0

h(x) = 0

minimize f(x) + X0(h(x)) + X�(g(x))

Implicit constraints

0

X�

1X�(z) =

(0, z 0

1, otherwise

GEOMETRIC INTERPRETATIONminimize f(x)

subject to g(x) 0

h(x) = 0

minimize f(x) + X0(h(x)) + X�(g(x))

Implicit constraints

0

X�

1 linear lower bound

X�(z) � hz, ⌫i, ⌫ � 0

GEOMETRIC INTERPRETATIONminimize f(x)

subject to g(x) 0

h(x) = 0

minimize f(x) + X0(h(x)) + X�(g(x))

Implicit constraints

minimize f(x) + h�, h(x)i+ h⌫, g(x)iLower bound

MAX OF DUALmax�,⌫�0

d(�, ⌫) = max�,⌫�0

minx

f(x) + h�, h(x)i+ h⌫, g(x)i

L(x,�, ⌫)

max�,⌫�0

d(�, ⌫) = max�,⌫�0

minx

L(x,�, ⌫) = minx

max�,⌫�0

L(x,�, ⌫)

Solution

???

Does maximizing dual = minimizing primal ???

WEAK/STRONG DUALITY

max�,⌫�0

d(�, ⌫) f(x?)

Weak duality

Always holds: even for non-convex problems

max�,⌫�0

d(�, ⌫) = f(x?)

Strong duality

Holds “most of the time” for convex problems

SLATER’S CONDITION

Slater’s condition holds if there is a strictly feasible point

Not strictly feasible

Strictly feasible

“Constraint qualification”

minimize f(x)

subject to g(x) 0

Ax = b

f(x) < 1g(x) < 0

Ax = b

SLATER’S CONDITION

Slater’s condition holds if there is a strictly feasible point

TheoremConvex+(linear equalities)+(Slater’s condition) = strong duality

max�,⌫�0

minx

L(x,�, ⌫) = minx

max�,⌫�0

L(x,�, ⌫)

max�,⌫�0

d(�, ⌫) = f(x?)

f(x) < 1g(x) < 0

Ax = b

minimize f(x)

subject to g(x) 0

Ax = b

NON-HOMOGENOUS SVM

Choose line with largest margin:2

kwk

Push data to “right” side of line:

xTw � b = 1<latexit sha1_base64="mJj/fapNTRfUJ3WMWUtg5TGNLnw=">AAAG/3icfZVNb9MwGMezwcoobxscuURskwZsUzOQ4IK0UYTgMDHQ3sSyVY7rNKaOndnO1uLlwHfgClduiCsfhRNfhcdZi9okXQ7Jk/x/fvz475cECaNKNxp/pqavXJ2pXZu9Xr9x89btO3Pzd/eUSCUmu1gwIQ8CpAijnOxqqhk5SCRBccDIftBtWn3/lEhFBd/R/YQcxajDaUgx0vDpY+9452w1cF+4XmtuobHWyC+3HHiDYMEZXNut+Zm/flvgNCZcY4aUOvQaiT4ySGqKGcnqfqpIgnAXdYhBsVL9OMjcpRjpSBU1+7FS6+UjzOr1pZGvh6kOnx8ZypNUE46hIWhhylwtXDtGt00lwZr1IUBYUqjHxRGSCGtwYiyVCYToahQo6MLn5AyLOEa8bXxGY7CfqMz4knYijaQUZwCVsBAITXravA3dLCuoOiJ8qLv2xSLjTISYTRFCdcbLzHoxh0S2BMQ7uaVj2a3CqhSlQYkC0TOuv+KvqDT4BH6APfYtL2Gcp/w0M8dm1St2rqmtHqVVWjexbbqPy0KcC+UWIoG6jqE6JIuDJIhZq+0CCALzodgSgmREb5b0iFJubYSgZZotGFGo+0UqFnrotB+nFV4LofXoXNihVzHdItTqVmBqjFK0E6Oyv9b5/1RVf6qAVOcB5NznKGDI7Z2XxBj1MtP3cVtod0gVIMKY4CQzizZqeYsFuRnRgf0YMXNQchZvjsibRTnUI+rrwhaAubKHFznhwq5ckmCRctiohpyk+SGV+e6yD5tn+P7Qtn9F4MyRZAuSvkuIRFrIR9BHDEZsbe1lEwHKaUw/w0CHkQ9Z25N51Bvyg+hyHmYHNnx+n4Qg2cnnA57+io0uAykfgnRyRhVJymFNDp6VmPETKaDX/D6B0LDGYKQXDzgj4JfgFX8A5WBvfc17srb+/unCxsvBz2HWue88cJYdz3nmbDhvnG1n18EOd74635zvtS+1H7WftV8X6PTUoM09Z+yq/f4HaV6QYA==</latexit>

xTw � b = 0<latexit sha1_base64="SJiXMqO6TewS9gzcdAU3zGIFkBw=">AAAG/3icfZVNb9MwGMezwcoobxscuURskwZsUzOQ4IK0UYTgMDHQ3sSyVY7rNKaOndnO1uLlwHfgClduiCsfhRNfhcdZi9okXQ7Jk/x/fvz475cECaNKNxp/pqavXJ2pXZu9Xr9x89btO3Pzd/eUSCUmu1gwIQ8CpAijnOxqqhk5SCRBccDIftBtWn3/lEhFBd/R/YQcxajDaUgx0vDpY+9452w1cF+4jdbcQmOtkV9uOfAGwYIzuLZb8zN//bbAaUy4xgwpdeg1En1kkNQUM5LV/VSRBOEu6hCDYqX6cZC5SzHSkSpq9mOl1stHmNXrSyNfD1MdPj8ylCepJhxDQ9DClLlauHaMbptKgjXrQ4CwpFCPiyMkEdbgxFgqEwjR1ShQ0IXPyRkWcYx42/iMxmA/UZnxJe1EGkkpzgAqYSEQmvS0eRu6WVZQdUT4UHfti0XGmQgxmyKE6oyXmfViDolsCYh3ckvHsluFVSlKgxIFomdcf8VfUWnwCfwAe+xbXsI4T/lpZo7NqlfsXFNbPUqrtG5i23Qfl4U4F8otRAJ1HUN1SBYHSRCzVtsFEATmQ7ElBMmI3izpEaXc2ghByzRbMKJQ94tULPTQaT9OK7wWQuvRubBDr2K6RajVrcDUGKVoJ0Zlf63z/6mq/lQBqc4DyLnPUcCQ2zsviTHqZabv47bQ7pAqQIQxwUlmFm3U8hYLcjOiA/sxYuag5CzeHJE3i3KoR9TXhS0Ac2UPL3LChV25JMEi5bBRDTlJ80Mq891lHzbP8P2hbf+KwJkjyRYkfZcQibSQj6CPGIzY2trLJgKU05h+hoEOIx+ytifzqDfkB9HlPMwObPj8PglBspPPBzz9FRtdBlI+BOnkjCqSlMOaHDwrMeMnUkCv+X0CoWGNwUgvHnBGwC/BK/4AysHe+pr3ZG39/dOFjZeDn8Osc9954Cw7nvPM2XDeONvOroMd7nx1vjnfa19qP2o/a78u0OmpQZt7zthV+/0PYsyQXw==</latexit>

h(li[xTi w � b])

<latexit sha1_base64="5HwNi5T9FjMiV2kY/JKzzlyeE+o=">AAAHC3icfZXNbtQwEMdToEtZvlrgxiWirdRCW20KEhxbFiE4VBTUL6lpI8fr3Zh17NR22t26eQTegStcuSGuPAQnXoVxuot2k2xzSMb5/zyeGX+FCaNKNxp/pq5dvzFduzlzq377zt1792fnHuwpkUpMdrFgQh6ESBFGOdnVVDNykEiC4pCR/bDbtPr+KZGKCr6j+wk5ilGH0zbFSMOvYPZRtMQC6h66vYAe77hn7mp4tBzMzjfWGvnjlg1vYMw7g2c7mJv+67cETmPCNWZIqUOvkegjg6SmmJGs7qeKJAh3UYcYFCvVj8PMXYyRjlRRsz8rtV6ebVavL478PUx1+9WRoTxJNeEYOoLWTpmrhWvzdVtUEqxZHwyEJYV4XBwhibCGqoy5MqEQXY1CBUP4nJxhEceIt4zPaAxTQVRmfEk7kUZSijOASlgbCE162rxvu1lWUHVE+FB3bcMi40yEmHXRhuiMl5n1og+JbAiId/KSjnm3CqtSlAYlCkXPuP6Kv6LS8DPUA8pjW3kI4zzlp5k5NqtecXBNbfQordK6ie3TfVYW4lwo9xAJxHUM0SFZTJIgZkttF0AYmk/FnmAkI3qzpEeUcltGMALTDCCjtu4XqVjoYaX9OK2otRBaj86FTb2K6RahoFuBqTFK0U6MyvW1lf9PVY2nCki1H0AufI5ChtzeRUmMUS8zfR+3hHaHVAEijAlOMrNgrcBbKMjNiA7KjxEzB6XK4s0RebMot/WI+rawBWCu7EFGTriwK5ckWKQcNqohJ2l+YGW+u+TD5hm2l23/NwTOHEm2wOmHhEikhXwKY8RQiK2tvWwiQDmN6TkkOrR88NqazKPekB9YV/MwO7Dh8/ckBMlOPh/w9VesdRVI+RCkkz2qSFIOa3LwrcSMn0gBo+bvCYSGNQaZXn7gjIArwSteAGVjb33Ne762/vHF/MbrweUw4zx2njhLjue8dDacd862s+tg59z56nxzvte+1H7UftZ+XaLXpgZ9HjpjT+33P6VHlDQ=</latexit>

minimizew,b

1

2kwk2 + C

Xh[yi(x

Ti w � b)]

<latexit sha1_base64="Tdy41nRS9BXV1itFE3xgrzeUj+M=">AAAHL3icfZVNb9Q4GMfD2yx0gS1w5GJRkAqUaqastByBQYg9VLCIlkp1GzkeZ2LGL8F2mBncfBW+A9+BK3td7QW47rfYx2EGzSQZckie5P/z48d/vyTJBbeu2/331OkzZ891fjl/Ye3Xi5cu/7Z+5eq+1YWhbI9qoc1BQiwTXLE9x51gB7lhRCaCvU5G/aC/fseM5Vq9ctOcHUkyVDzllDj4FK8/wJIrLvl7FvvxVlIinBGRInwyxifHO+gu6mNbSJQdTmO+OYn58avxveT2EULx+kZ3u1tdqBn0ZsFGNLtexFfOfcUDTQvJlKOCWHvY6+buyBPjOBWsXMOFZTmhIzJknkhrpxLKuSWJy2xdCx9btUnlSLm2dmvh62Hh0gdHnqu8cExRaAhaWgjkNAqeoAE3jDoxhYBQw6EeRDNiCHXg3FIqn2g9ciSx0AVWbEy1lEQNPBZcwnQxW3ps+DBzxBg9BqiBpUA4NnH+zxSVZU11GVNzHYWXgCwzYXoASaE63yv9Tj2HIaEEooaVpUvZgyLaFOtAyRI98Qhv4S1bJG/AD7AnvFUlLPNcvSv9sb/Xq3fueKieFG3aKA9tRnebgqyEZgudQ13HUB0x9UEyIoLVYQEkiX9ZbwlBvqD3G3rGuQo2QhD7fgwjSt20Tknt5k5jWbR4rbVzi3MRht7GjOpQPGrB7BJl+VCSpr/B+R9UW3+2hrTnAeQEK5IIgiYnDVGSSemnmA60Q3OqBjEhtGKlvxmiuHezJvczPrOfEuEPGs7SRwvyo7qcugX1aW0LwFyFw469VTqsXJZTXSjYqJ69LapDrcRoE8Pmmb/fDu2fMDhzDNuFpM9zZojT5g70IcGI3d39ciUwOxxLP48wZB2s5slkzs+in/MwO7Dhq/sqhJhhNR/wxFsh+hnI1RzkqzPazHAFa3L2bMU8zo2GXqv7CsLBGoORfn/AGQG/hF79B9AM9ne2e/e3d/76fePh49nP4Xx0PboRbUa96I/oYfQsehHtRTT6EH2KPkd/dz52/ul86Xz7jp4+NWtzLVq6Ov/9DyuuovA=</latexit>

DUAL: THE HARD WAY

Idea: make this differentiable

Standard form

minimizew,b

1

2kwk2 + C

Xh[yi(x

Ti w � b)]

<latexit sha1_base64="Tdy41nRS9BXV1itFE3xgrzeUj+M=">AAAHL3icfZVNb9Q4GMfD2yx0gS1w5GJRkAqUaqastByBQYg9VLCIlkp1GzkeZ2LGL8F2mBncfBW+A9+BK3td7QW47rfYx2EGzSQZckie5P/z48d/vyTJBbeu2/331OkzZ891fjl/Ye3Xi5cu/7Z+5eq+1YWhbI9qoc1BQiwTXLE9x51gB7lhRCaCvU5G/aC/fseM5Vq9ctOcHUkyVDzllDj4FK8/wJIrLvl7FvvxVlIinBGRInwyxifHO+gu6mNbSJQdTmO+OYn58avxveT2EULx+kZ3u1tdqBn0ZsFGNLtexFfOfcUDTQvJlKOCWHvY6+buyBPjOBWsXMOFZTmhIzJknkhrpxLKuSWJy2xdCx9btUnlSLm2dmvh62Hh0gdHnqu8cExRaAhaWgjkNAqeoAE3jDoxhYBQw6EeRDNiCHXg3FIqn2g9ciSx0AVWbEy1lEQNPBZcwnQxW3ps+DBzxBg9BqiBpUA4NnH+zxSVZU11GVNzHYWXgCwzYXoASaE63yv9Tj2HIaEEooaVpUvZgyLaFOtAyRI98Qhv4S1bJG/AD7AnvFUlLPNcvSv9sb/Xq3fueKieFG3aKA9tRnebgqyEZgudQ13HUB0x9UEyIoLVYQEkiX9ZbwlBvqD3G3rGuQo2QhD7fgwjSt20Tknt5k5jWbR4rbVzi3MRht7GjOpQPGrB7BJl+VCSpr/B+R9UW3+2hrTnAeQEK5IIgiYnDVGSSemnmA60Q3OqBjEhtGKlvxmiuHezJvczPrOfEuEPGs7SRwvyo7qcugX1aW0LwFyFw469VTqsXJZTXSjYqJ69LapDrcRoE8Pmmb/fDu2fMDhzDNuFpM9zZojT5g70IcGI3d39ciUwOxxLP48wZB2s5slkzs+in/MwO7Dhq/sqhJhhNR/wxFsh+hnI1RzkqzPazHAFa3L2bMU8zo2GXqv7CsLBGoORfn/AGQG/hF79B9AM9ne2e/e3d/76fePh49nP4Xx0PboRbUa96I/oYfQsehHtRTT6EH2KPkd/dz52/ul86Xz7jp4+NWtzLVq6Ov/9DyuuovA=</latexit>

minimizew,b

1

2kwk2 + C

Xmax{1� yi(x

Ti w � b), 0}

<latexit sha1_base64="rMtHtOMH4rJtpB0BZwHEg6rGEBg=">AAAHO3icfZVNj9Q2GMcDlCld+rLAkYvVBWkps6vJggRHYBBqD6tCxS4r4d3I8TgTM34JtsPM1JsvxHfod+iVXnroreqVO4/DTDWTZMgheZL/z48f//2StBDcusHgw4WLl7663Pv6yjdbV7/97vsftq9dP7a6NJQdUS20OUmJZYIrduS4E+ykMIzIVLBX6WQY9FfvmLFcq5duXrBTScaKZ5wSB5+S7adYcsUl/50lftpPK4RzIjKEz6f4/OwA3UVDbEuJsCQz7FG8h+YJ350l/OzldC+9g1AfDXCVbO8M9gf1hdpBvAh2osX1PLl2+W880rSUTDkqiLWv40HhTj0xjlPBqi1cWlYQOiFj5om0di6htNuSuNw2tfCxU5vV7lRbW7dXvr4uXfbw1HNVlI4pCg1By0qBnEbBHzTihlEn5hAQajjUg2hODKEOXFxL5VOtJ46kFrrAik2plpKokceCS5g6ZiuPDR/njhijpwC1sAwIx2bO/5KhqmqoLmdqqaPwEpB1JkwVIBlU5+PKHzRzGBJKIGpcW7qWPSiiS7EOlDzVM49wH/dtmb4BP8Ce8FaXsM5z9a7yZ34vbnbueKielF3apAhtJnfbgqyFdgtdQF1nUB0xzUEyIoLVYQGkqf+t2RKCYkUftvSccxVshCDxwwRGlLl5k5LaLZ3GsuzwWmvnVuciDL2LmTShZNKB2TXK8rEkbX+D8/9TXf3ZBtKdB5BzrEgqCJqdt0TY+JWfYzrSDi2pBsSE0IpV/laIkvhWQx7mfGE/JcKftJylj1fkx005cyvqs8YWgLkKBx97q3RYuaygulSwUT17W9YHXIXRLobNs3y/E9o/ZXDmGHYISX8tmCFOm5+gDwlGHB4eVxuBxUFZ+WWEIetoM09mS34RfZmH2YENX983IcSM6/mAJ+6H6EsgV0uQb85oc8MVrMnFsxPzuDAaeq3vGwgHawxG+vkBZwT8EuLmD6AdHB/sx/f2D17c33n0ZPFzuBLdjH6MdqM4ehA9in6OnkdHEY3eR39GH6K/en/0/un92/vvM3rxwqLNjWjt6n38BM2qprQ=</latexit>

minimizew,b

1

2kwk2 + C

Xmax{1� Y Xw + yb, 0}

<latexit sha1_base64="KbmZ4s5rsAngW5ULpRC1v5Kilgs=">AAAHVnicfZVdc9M4FIbNRxsou0thL7nRUJhhaejE3Z2BS7phmN2LDuwOLWVQm5EVORbRh5FkkqD69/Br9nY/fswyHLkJk9gOvrCP/T6Sjt4jyUkuuHW93n+XLl+5urHZuXZ968Z33/9wc/vW7WOrC0PZEdVCm5OEWCa4YkeOO8FOcsOITAR7nYz7QX/9gRnLtXrlZjk7lWSkeMopcfBpsH2AJVdc8o9s4CfdpEQ4IyJF+HyCz8/20S7qY1tIhCWZYo+wVgw9Qm9OJruzBKEu6uFysL3T2+tVF2oG8TzYiebXy8Gtjf/xUNNCMuWoINa+jXu5O/XEOE4FK7dwYVlO6JiMmCfS2pmEvO5L4jJb18LHVm1aWVNubd1f+vq2cOmTU89VXjimKDQELS0EchoFc9CQG0admEFAqOGQD6IZMYQ6sHClK59oPXYksTAEVmxCtZREDT0WXELdmC09NnyUOWKMngDUwFIgHJs6/3uKyrKmuoyphY7CS0BWmVAnQFLIzsel36/3YUhIgahRZelK70ERbYp1oGSJnkKpu7hri+Qd+AH2hLcqhVWeqw+lP/OP4vrgjofsSdGmjfPQZrzbFGQlNFvoHPI6g+yIqU+SERGsDgsgSfyf9ZYQ5Et6v6FnnKtgIwQD3x/AjFI3q1NSu4XTWBYtXmvt3HItwtTbmHEdGoxbMLtCWT6SpOlvcP4r1TaerSHt/QByjhVJBEHT84YIu770M0yH2qEFVYOYEHAolP5eiAbxvZrcz/jcfkqEP2k4Sw+W5IO6nLol9XnDqjDuRWlTH9c2CFQynInsvdJhXbOc6kLBNvbsfVGdfSVGDzBsrcX7T6H9MwYnkmGH0OeLnBnitHkIQ0iw6fDwuFwLzM/Q0i8iDL0O1/NkuuDn0bd5qB0cB9V9HULMqKoWPHE3RN8CuVqAfH2PNjNcwYqdP1sxj3OjYdTqvoZwsAJhphcPOEHghxHXfw/N4Hh/L/55b/+PX3ae/jr/dVyL7kR3owdRHD2Onka/RS+jo4hGn6K/or+jfzb/3fzc2eh0LtDLl+ZtfoxWrs72F5xJr5I=</latexit>

minimizew,b,v

1

2kwk2 + Ch1, vi

subject to v � 1� Y Xw + yb

v � 0<latexit sha1_base64="V/RchF1D5YrDENTB5wFsnG5Xrkg=">AAAHcXicfVVbjxM3FB6ugfTCAk+oLxYLiHbDKrNUKo/QoIo+rEqr7rIV3o08jidj4stge3LBO3+Of9F/0NfS50o9nk2qZGaCH2aO/X0+Pv6OfZzkglvX7/956fKVq9eud27c7H7x5Vdf39q5fefY6sJQdkS10OYkIZYJrtiR406wk9wwIhPB3iSTQcDfTJmxXKvf3SJnp5KMFU85JQ6GhjsYS6645B/Y0M96SW9aPkI4IyJF+HyGz88O0B4aICwIwlqxHkJThA10cBdb9whN8ZihCnqC/jiZ7S0SQFbD/eHObn+/XzXUNOKlsRst2+vh7Wv/4pGmhWTKUUGsfRv3c3fqiXGcClZ2cWFZTuiEjJkn0tqFTEr0UBKX2ToWBluxeaVa2e0+XBt9W7j02annKi8cUxQmApYWAjmNgm5oxA2jTizAINRwiAfRjBhCHai74conWk8cSSwsgRWbUS0lUSOPBZeQUmZLjw0fZ44Yo2dAatBSYDg2d/7nFJVlDXUZUyschU6gbHJC/oCSQnQ+Lv1B3YchIQSixpWkG94DItoQ6wDJEj33CPdwzxbJO9AD5Am9KoRNPlfT0p/5J3F9ccdD9KRowyZ5mDPZawKyApozdA5xnUF0xNQ3yYgIUocDkCT+t/pMMPI1fNDAM85VkBGMoR8MYUepW9RZUruV0lgWLVpr7dx6LsLW2ziTOmk4aaHZDZblY0ma+gbl/2e1rWdrlHY/QDnHiiRw9+fnDVCSeekXmI60QytWjcSEgMJQ+gfBGsYPavAg40v5KRH+pKEsfbEGv6jDqVtDf2pIFda9SG3q49oFgUyGcsneKx3ONcupLhRcY8/eF1VZLDF6jOFqrfrfhvkvGVQkww7B5y85M8Rp8x0sIUGmw8PjcithWV5Lv7IweB1t55P5ir+0Ps+H3EE5qL7bKMSMq2zBH/eC9TkiVysi3+7RZoYrOLHLfyvN49xoWLX6bmE4OIGw04sfVBB4MOL689A0jg/246f7B79+v/v8x+XTcSP6JrofPY7i6IfoefQqeh0dRTT6GP0V/R19uv5P514Hde5fUC9fWs65G220zt5/Bv624Q==</latexit>

h(z) = max{z � 1, 0}

<latexit sha1_base64="UUT015Jz/WZW9mOPLkNAJ0BXWUk=">AAAHKXicfZVNb9MwGMfDYGWMt/Fy4xIxJg3opgYNwQWJMYTgMDEQG5PmrXJcpzF17Mx2Rjsv34Ur8Gm4AVc+BxKPsxa1Sbockif5//z48d8vCVPOtGm1fp2bOX9htnFx7tL85StXr11fuHFzR8tMEbpNJJdqN8SacibotmGG091UUZyEnH4MextO/3hElWZSfDCDlO4nuCtYxAg28Km9cDtePr7vP/NRgvvIHq8EzRbK2wuLrdVWcfnVIBgGi97w2mrfmP2LOpJkCRWGcKz1XtBKzb7FyjDCaT6PMk1TTHq4Sy1OtB4kYe4vJdjEuqy5j7VavxhtPj+/NPZ1LzPR033LRJoZKgg0BC3KuG+k78brd5iixPABBJgoBvX4JMYKEwOuTKSyoZQ9g0MNXSBBPxOZJFh0LOIsgamgOrdIsW5ssFLyM0AVLALC0L6xbyI/z0uqiakY6b57ccgkE2PuUkRQnQ1y+6icQ2FXAhbdwtKJ7E7hdYo2oMSh7FsfNVFTZ+En8APscW9FCZM8E0e5PbArQblzw1z1OKvTeqlr03tYFZJCqLaQKdR1ANVhVR4kxdxZ7RZAGNr35ZYQpGP6RkWPGRPORgjadqMNI4rMoEwl0oycRklW47WUxozPhRt6HdMrQ+1eDaYnKM26Ca7665z/T9X1p0tIfR5ATpDAIcd+/6Qiwj7P7QCRjjT+iCpBlHMpaG7vuagd3CvJGzEb2k8wt7sVZ8n6mLxeliMzpr6qWOX6PZ3ayAalDQIz6Y45eiikW9c0JTITsI0tPcyK4yxH/jKCrTV6v+/av6RwIim6CTnfplRhI9UD6CIBmzY3d/KpABMsYcdQzihCkLUzncf9ET+MzuZh7uA4KO7TEKy6xWzBEzVddBbIxAhk0zPqWDEBK3b4rMUsSpWEXov7FMLACoSRnj7gBIEfRlD+PVSDnUerwdrq43dri89fDH8dc94d76637AXeE++599rb8rY94h17X7yv3rfG98aPxs/G71N05tywzS1v4mr8+QeO2qEZ</latexit>

DUAL: THE HARD WAY

Positive multipliers

Break it up!

minimizew,b,v

1

2kwk2 + Ch1, vi

subject to v � 1� Y Xw + yb

v � 0<latexit sha1_base64="V/RchF1D5YrDENTB5wFsnG5Xrkg=">AAAHcXicfVVbjxM3FB6ugfTCAk+oLxYLiHbDKrNUKo/QoIo+rEqr7rIV3o08jidj4stge3LBO3+Of9F/0NfS50o9nk2qZGaCH2aO/X0+Pv6OfZzkglvX7/956fKVq9eud27c7H7x5Vdf39q5fefY6sJQdkS10OYkIZYJrtiR406wk9wwIhPB3iSTQcDfTJmxXKvf3SJnp5KMFU85JQ6GhjsYS6645B/Y0M96SW9aPkI4IyJF+HyGz88O0B4aICwIwlqxHkJThA10cBdb9whN8ZihCnqC/jiZ7S0SQFbD/eHObn+/XzXUNOKlsRst2+vh7Wv/4pGmhWTKUUGsfRv3c3fqiXGcClZ2cWFZTuiEjJkn0tqFTEr0UBKX2ToWBluxeaVa2e0+XBt9W7j02annKi8cUxQmApYWAjmNgm5oxA2jTizAINRwiAfRjBhCHai74conWk8cSSwsgRWbUS0lUSOPBZeQUmZLjw0fZ44Yo2dAatBSYDg2d/7nFJVlDXUZUyschU6gbHJC/oCSQnQ+Lv1B3YchIQSixpWkG94DItoQ6wDJEj33CPdwzxbJO9AD5Am9KoRNPlfT0p/5J3F9ccdD9KRowyZ5mDPZawKyApozdA5xnUF0xNQ3yYgIUocDkCT+t/pMMPI1fNDAM85VkBGMoR8MYUepW9RZUruV0lgWLVpr7dx6LsLW2ziTOmk4aaHZDZblY0ma+gbl/2e1rWdrlHY/QDnHiiRw9+fnDVCSeekXmI60QytWjcSEgMJQ+gfBGsYPavAg40v5KRH+pKEsfbEGv6jDqVtDf2pIFda9SG3q49oFgUyGcsneKx3ONcupLhRcY8/eF1VZLDF6jOFqrfrfhvkvGVQkww7B5y85M8Rp8x0sIUGmw8PjcithWV5Lv7IweB1t55P5ir+0Ps+H3EE5qL7bKMSMq2zBH/eC9TkiVysi3+7RZoYrOLHLfyvN49xoWLX6bmE4OIGw04sfVBB4MOL689A0jg/246f7B79+v/v8x+XTcSP6JrofPY7i6IfoefQqeh0dRTT6GP0V/R19uv5P514Hde5fUC9fWs65G220zt5/Bv624Q==</latexit>

d(↵, �) = minw,b,v

1

2kwk2 + Ch1, vi+ h↵,1� Y Xw + yb� vi+ h�,�vi

<latexit sha1_base64="+ZspVEcekHl8Fv1lWkwWar5V0Gs=">AAAHlnicfZVdb9s2FIblrp1bb93SFWgL7IZY2iFdnMDKCrQ3Ldp5nxfBsmFJU5SJQdGUxZkiVZLyRxld7GfuF+xXDNihbA+2JFcX0hHfh4dHLz8UZYIb2+v93br20fUbH7dv3up88untzz7fufPFmVG5puyUKqH0eUQME1yyU8utYOeZZiSNBHsdjftefz1h2nAl/7DzjF2kZCR5zCmx0DTY+asz3MNEZAnp4hFJU/IYPUc45XLgpt2oO0HF1wgnRMQIX03x1eUR2u8jLAjCSrIuQhOENUEI7S8aF5lKER2gN+fT/Xl0MEEltEDQYpguWjUPdnZ7h73yQvUgXAa7wfI6Gdy58S8eKpqnTFoqiDFvw15mLxzRllPBig7ODcsIHZMRcyQ1Zp5GBXqUEpuYquYbG7VZaW3R6Txaa32b2/jZheMyyy2TFDqCFucCWYW8uWjINaNWzCEgVHOoB9GEaEItTMFGKhcpNbYkMjAElmxKFZgihw4LnsK8M1M4rPkosURrNQWohsVAWDaz7pcYFUVFtQmTKx35F49sMn5SAYmhOhcW7qiaQxNfApGj0tKN7F4RTYqxoCSRmjmEu7hr8uhP8APs8W9lCZs8l5PCXbqDsDq45b56kjdp48z3Ge/XhbQU6j1UBnVdQnVEVz+SEeGt9gsgitzv1Z4QZGt6v6YnnEtvIwQD1x/AF8V2XqVSZVdO4zRv8Fopa9fnwn96EzOuQoNxA2Y2KMNHKan7653/n2oaz1SQ5jyAXGFJItjZs6uamJJZ4eaYDpVFK6oCMSHgsCjcQx8NwocVuZ/wpf2UCHdec5a+WpNfVeXYrqk/1qzy4y6mNnZhZYPATPozlb2Tyq9rllGVS9jGjr3Ly7OzwGgPw9ZavT/2/b9ncCJpdgw5f82YJlbpb2CIFGw6Pj4rtgJc8pS/h3JWEYasw+08ma34ZfRhHuYOjoPyvg0helTOFjxx10cfArlcgXx7RpNoLmHFLp+NmMOZVjBqed9CWFiB8KWLB5wg8MMIq7+HenB2dBh+e3j025Pdl98tfx03gy+Dr4K9IAyeBi+Dn4OT4DSgwT+t2617rfvtB+0X7R/aPy3Qa61ln7vBxtU++Q9jzb6I</latexit>

d(↵, �) = minw,b,v

1

2kwk2 + Ch1, vi+ h↵,1i � h↵, Y Xwi+ h↵, yib� h↵, vi � h�, vi

<latexit sha1_base64="tzcqA64NUKubds0LRnDeP+OGOeQ=">AAAH1nicfVVdk9s0FHUKJW2gdAuPvGjYlm7ZdCdemIEXZnZJh4GHnbYMuxum2s3IihyLyJIryfmo1rx1eOUv8TP4Mcxw5SSdxHbqB/v6nnOvrs7VR5QJbmyv92/r1gcf3v6ofedu5+NP7n16f+/BZxdG5Zqyc6qE0oOIGCa4ZOeWW8EGmWYkjQS7jCZ9j19OmTZcyd/sImNXKRlLHnNKLLiGe/90RgeYiCwhXTwmaUqeoB8QTrkculk36k5R8RXCCRExwjczfHN9jA77CAuCsJKsi9AUYU0QQp3DpXeZqkTB67EOerqJoN8HszWwHbNYuiMfsQVMV/ylFy3rfOce7u33jnrlg+pGuDL2g9XzYvjg9n94pGieMmmpIMa8CnuZvXJEW04FKzo4NywjdELGzJHUmEUaFehRSmxiqph3NmLzsjVFp/Now/sqt/H3V47LLLdMUggELM4Fsgr55qAR14xasQCDUM2hHkQTogm10MKtVC5SamJJZGAILNmMKtBEjhwWPIV1w0zhsObjxBKt1QxINVoMDMvm1v0So6KooDZhco0j/+Mp2xy/KIASQ3UuLNxxNYcmvgQix6WkW9k9IpoQYwFJIjV3CHdx1+TRH6AHyOP/yhK2+VxOC3ftnobVwS331ZO8CZtkPmZyWAfSEqhHqAzquobqiK5OkhHhpfYLIIrcr9VIMLINvF/DE86llxGMoesPYUaxXVRZqbJrpXGaN2itlLWbvfBTb+JMqqThpIFmtliGj1NS19cr/47VNJ6pUJrzAOUGSxLBxp7f1MCUzAu3wHSkLFqzKiQmBJw1hXvorWH4sAL3E76SnxLhBjVl6ekGfFqFY7uB/lSTyo+7bG3swsoGgU76M5m9lsqva5ZRlUvYxo69zsuzt8DoAMPWWv8/8fHPGJxImp1BzucZ08Qq/TUMkYJMZ2cXxU4Clzzlb6CctYUh62g3n8zX/JX1fj70Do6D8r2LQvS47BZ8cddb7yNyuSby3RlNormEFbv6NtIczrSCUcv3DoaFFQgzXX7gBIELI6xeD3Xj4vgo/Obo+OW3+yc/rq6OO8EXwZfBQRAG3wUnwc/Bi+A8oK3HrbPWReuyPWj/2X7b/mtJvdVaxXwebD3tv/8H3uHRYg==</latexit>

DUAL: THE HARD WAY

This is too complicated. Let’s reduce it…Put all the v terms together

d(↵, �) = minw,b,v

1

2kwk2 + Ch1, vi+ h↵,1i � h↵, Y Xwi+ h↵, yib� h↵, vi � h�, vi

<latexit sha1_base64="tzcqA64NUKubds0LRnDeP+OGOeQ=">AAAH1nicfVVdk9s0FHUKJW2gdAuPvGjYlm7ZdCdemIEXZnZJh4GHnbYMuxum2s3IihyLyJIryfmo1rx1eOUv8TP4Mcxw5SSdxHbqB/v6nnOvrs7VR5QJbmyv92/r1gcf3v6ofedu5+NP7n16f+/BZxdG5Zqyc6qE0oOIGCa4ZOeWW8EGmWYkjQS7jCZ9j19OmTZcyd/sImNXKRlLHnNKLLiGe/90RgeYiCwhXTwmaUqeoB8QTrkculk36k5R8RXCCRExwjczfHN9jA77CAuCsJKsi9AUYU0QQp3DpXeZqkTB67EOerqJoN8HszWwHbNYuiMfsQVMV/ylFy3rfOce7u33jnrlg+pGuDL2g9XzYvjg9n94pGieMmmpIMa8CnuZvXJEW04FKzo4NywjdELGzJHUmEUaFehRSmxiqph3NmLzsjVFp/Now/sqt/H3V47LLLdMUggELM4Fsgr55qAR14xasQCDUM2hHkQTogm10MKtVC5SamJJZGAILNmMKtBEjhwWPIV1w0zhsObjxBKt1QxINVoMDMvm1v0So6KooDZhco0j/+Mp2xy/KIASQ3UuLNxxNYcmvgQix6WkW9k9IpoQYwFJIjV3CHdx1+TRH6AHyOP/yhK2+VxOC3ftnobVwS331ZO8CZtkPmZyWAfSEqhHqAzquobqiK5OkhHhpfYLIIrcr9VIMLINvF/DE86llxGMoesPYUaxXVRZqbJrpXGaN2itlLWbvfBTb+JMqqThpIFmtliGj1NS19cr/47VNJ6pUJrzAOUGSxLBxp7f1MCUzAu3wHSkLFqzKiQmBJw1hXvorWH4sAL3E76SnxLhBjVl6ekGfFqFY7uB/lSTyo+7bG3swsoGgU76M5m9lsqva5ZRlUvYxo69zsuzt8DoAMPWWv8/8fHPGJxImp1BzucZ08Qq/TUMkYJMZ2cXxU4Clzzlb6CctYUh62g3n8zX/JX1fj70Do6D8r2LQvS47BZ8cddb7yNyuSby3RlNormEFbv6NtIczrSCUcv3DoaFFQgzXX7gBIELI6xeD3Xj4vgo/Obo+OW3+yc/rq6OO8EXwZfBQRAG3wUnwc/Bi+A8oK3HrbPWReuyPWj/2X7b/mtJvdVaxXwebD3tv/8H3uHRYg==</latexit>

d(↵, �) = minw,b,v

1

2kwk2 + h↵,1i � h↵, Y Xwi+ h↵, yib+ hC1� ↵� �, vi

<latexit sha1_base64="0tUxxB+BYnZqQ/6FDHREs09r3gc=">AAAHunicfVVbb9s2FJa7de7cXdLtcS/E0g7p6gRWNmDDLkA7F8P6EKwrmjRFmRgUTVmcKVIlqdgeo5+zH9UfU6CHslTYklw9iIf8vnN4+B1eokxwY0ejN70bH31885P+rU8Htz/7/Isv9+58dWZUrik7pUoofR4RwwSX7NRyK9h5phlJI8FeRPOxx19cMW24ks/tKmMXKZlJHnNKLAxN9v4fTA8wEVlChnhG0pTcR78jnHI5cYthNLxCxXcIJ0TECF8v8PXlMUJo8ABhQVDlhrCSDEaxJmiADjcR9PJ8UQPbPqv1cFTHGpdBDisc2jKXIbpa8yZ7+6OjUfmhthFWxn5QfU8nd26+xVNF85RJSwUx5lU4yuyFI9pyKlgxwLlhGaFzMmOOpMas0qhA91JiE9PE/GAntizlLwaDexujr3Ib/3zhuMxyyyQFR8DiXCCrkC8AmnLNqBUrMAjVHPJBNCGaUAtl2grlIqXmlkQGpsCSLagCTeTUYcFT2BvMFA5rPkss0VotgNSixcCwbGndkxgVRQO1CZM1jnzHU7Y5vvBAiSE7FxbuuBlDE58CkbNS0q3oHhFdiLGAJJFaOoSHeGjy6F/QA+TxvTKFbT6XV4W7dIdhc3LLffYk78LmmfeZP2gDaQm0PVQGeV1CdkQ3F8mI8FL7DRBF7lnTE4xsAx+38IRz6WUEY+LGE1hRbFdNVqpsrTRO8w6tlbJ2sxZ+6V2ceZM0mXfQzBbL8FlK2vp65d+zuuYzDUp3HKBcY0kiOOfL6xaYkmXhVphOlUU1q0FiQsD1ULi73pqEdxvwOOGV/JQId95Slj7agB814dhuoH+2pPLzrksbu7BxQKCS/t5lr6Xy+5plVOUSjrFjr/Pyfi0wOsBwtOr+fe//mMGNpNkJxPw7Y5pYpb+HKVKQ6eTkrNhJ4JKn/D9Ip7YwRJ3u5pNlza+sD/OhdnAdlP9dFKJnZbWgxUNvfYjIZU3kuyOaRHMJO7ZqO2kOZ1rBrOV/B8PCDoSVrhu4QeDBCJvPQ9s4Oz4Kfzg6/ufH/Yd/VE/HreCb4NvgIAiDn4KHwV/B0+A0oL3bvbD3S+/X/m/9qM/78zX1Rq/y+TrY+vr2HQ7qyOw=</latexit>

C1� ↵� � = 0, with ↵, � � 0

derivative for v

remember this constraint!

0 ↵i C<latexit sha1_base64="hSFZPolLzkdH2M1bkt4MvmRDLWU=">AAAHK3icfZXLbtQwFIZTLkMptxYkNmwiChKXUk0KEixbBiFYVBRELxIukeNxJmYcO7WddgY3L8MWeBpWILY8BhLH6QyaSTLNIjnO//n4+LfjRBln2rTbP+fOnD13vnVh/uLCpctXrl5bXLq+o2WuCN0mkku1F2FNORN02zDD6V6mKE4jTnejfsfpu4dUaSbFezPM6H6Ke4LFjGADr8LFm37bR5z6CPMswSErGx0/XFxur7bLy68HwShY9kbXVrh0/i/qSpKnVBjCsdYfgnZm9i1WhhFOiwWUa5ph0sc9anGq9TCNCv9uik2iq5p72agNyvkWCwt3J95+yE38bN8ykeWGCgIdQYtz7hvpuxn7XaYoMXwIASaKQT0+SbDCxIAvU6lsJGXf4EjDEEjQIyLTFIuuRZylsBhUFxYp1ksMVkoeAVTDYiAMHRj7OvaLoqKahIqx7ruGQ6aZBHOXIobqbFDYtWoOhV0JWPRKS6eyO4U3KdqAkkRyYH20glZ0Hn0CP8Ae1ypLmOaZOCzsR/soqA5umKse501aP3N9+g/rQloK9R4yg7o+QnVYVSdJMXdWuw0QRfZdtScE2YTeqekJY8LZCEFoOyHMKDbDKpVKM3YapXmD11IaM7kWbupNTL8Khf0GTE9RmvVSXPfXOf+fahpPV5DmPIAcI4Ejjv3BcU1M8aCwQ0S60vhjqgJRzqWghb3jojC4U5E7CRvZTzC3ezVnycaEvFGVYzOhvqxZ5cY9WdrYBpUPBFbSHXT0QEi3r2lGZC7gM7b0IC8PtAL59xB8WuP2fdf/BYUTSdFNyPkmowobqR7AECnYtLm5U8wEmGAp+wzljCMEWbuzeTwY86PodB7WDo6D8j4LwapXrhY80YqLTgOZGINsdkadKCZgx46ejZhFmZIwanmfQRjYgTDTkwecIPDDCKq/h3qws7YaPF5de/tkef356Ncx793ybnv3vMB76q17r7wtb9sj3rH3xfvqfWt9b/1o/Wr9PkHPzI363PCmrtaff03Ioao=</latexit>

DUAL: THE HARD WAY

This is too complicated. Let’s reduce it…

d(↵, �) = minw,b,v

1

2kwk2 + Ch1, vi+ h↵,1i � h↵, Y Xwi+ h↵, yib� h↵, vi � h�, vi

<latexit sha1_base64="tzcqA64NUKubds0LRnDeP+OGOeQ=">AAAH1nicfVVdk9s0FHUKJW2gdAuPvGjYlm7ZdCdemIEXZnZJh4GHnbYMuxum2s3IihyLyJIryfmo1rx1eOUv8TP4Mcxw5SSdxHbqB/v6nnOvrs7VR5QJbmyv92/r1gcf3v6ofedu5+NP7n16f+/BZxdG5Zqyc6qE0oOIGCa4ZOeWW8EGmWYkjQS7jCZ9j19OmTZcyd/sImNXKRlLHnNKLLiGe/90RgeYiCwhXTwmaUqeoB8QTrkculk36k5R8RXCCRExwjczfHN9jA77CAuCsJKsi9AUYU0QQp3DpXeZqkTB67EOerqJoN8HszWwHbNYuiMfsQVMV/ylFy3rfOce7u33jnrlg+pGuDL2g9XzYvjg9n94pGieMmmpIMa8CnuZvXJEW04FKzo4NywjdELGzJHUmEUaFehRSmxiqph3NmLzsjVFp/Now/sqt/H3V47LLLdMUggELM4Fsgr55qAR14xasQCDUM2hHkQTogm10MKtVC5SamJJZGAILNmMKtBEjhwWPIV1w0zhsObjxBKt1QxINVoMDMvm1v0So6KooDZhco0j/+Mp2xy/KIASQ3UuLNxxNYcmvgQix6WkW9k9IpoQYwFJIjV3CHdx1+TRH6AHyOP/yhK2+VxOC3ftnobVwS331ZO8CZtkPmZyWAfSEqhHqAzquobqiK5OkhHhpfYLIIrcr9VIMLINvF/DE86llxGMoesPYUaxXVRZqbJrpXGaN2itlLWbvfBTb+JMqqThpIFmtliGj1NS19cr/47VNJ6pUJrzAOUGSxLBxp7f1MCUzAu3wHSkLFqzKiQmBJw1hXvorWH4sAL3E76SnxLhBjVl6ekGfFqFY7uB/lSTyo+7bG3swsoGgU76M5m9lsqva5ZRlUvYxo69zsuzt8DoAMPWWv8/8fHPGJxImp1BzucZ08Qq/TUMkYJMZ2cXxU4Clzzlb6CctYUh62g3n8zX/JX1fj70Do6D8r2LQvS47BZ8cddb7yNyuSby3RlNormEFbv6NtIczrSCUcv3DoaFFQgzXX7gBIELI6xeD3Xj4vgo/Obo+OW3+yc/rq6OO8EXwZfBQRAG3wUnwc/Bi+A8oK3HrbPWReuyPWj/2X7b/mtJvdVaxXwebD3tv/8H3uHRYg==</latexit>

d(↵, �) = minw,b,v

1

2kwk2 + h↵,1i � h↵, Y Xwi+ h↵, yib+ hC1� ↵� �, vi

<latexit sha1_base64="0tUxxB+BYnZqQ/6FDHREs09r3gc=">AAAHunicfVVbb9s2FJa7de7cXdLtcS/E0g7p6gRWNmDDLkA7F8P6EKwrmjRFmRgUTVmcKVIlqdgeo5+zH9UfU6CHslTYklw9iIf8vnN4+B1eokxwY0ejN70bH31885P+rU8Htz/7/Isv9+58dWZUrik7pUoofR4RwwSX7NRyK9h5phlJI8FeRPOxx19cMW24ks/tKmMXKZlJHnNKLAxN9v4fTA8wEVlChnhG0pTcR78jnHI5cYthNLxCxXcIJ0TECF8v8PXlMUJo8ABhQVDlhrCSDEaxJmiADjcR9PJ8UQPbPqv1cFTHGpdBDisc2jKXIbpa8yZ7+6OjUfmhthFWxn5QfU8nd26+xVNF85RJSwUx5lU4yuyFI9pyKlgxwLlhGaFzMmOOpMas0qhA91JiE9PE/GAntizlLwaDexujr3Ib/3zhuMxyyyQFR8DiXCCrkC8AmnLNqBUrMAjVHPJBNCGaUAtl2grlIqXmlkQGpsCSLagCTeTUYcFT2BvMFA5rPkss0VotgNSixcCwbGndkxgVRQO1CZM1jnzHU7Y5vvBAiSE7FxbuuBlDE58CkbNS0q3oHhFdiLGAJJFaOoSHeGjy6F/QA+TxvTKFbT6XV4W7dIdhc3LLffYk78LmmfeZP2gDaQm0PVQGeV1CdkQ3F8mI8FL7DRBF7lnTE4xsAx+38IRz6WUEY+LGE1hRbFdNVqpsrTRO8w6tlbJ2sxZ+6V2ceZM0mXfQzBbL8FlK2vp65d+zuuYzDUp3HKBcY0kiOOfL6xaYkmXhVphOlUU1q0FiQsD1ULi73pqEdxvwOOGV/JQId95Slj7agB814dhuoH+2pPLzrksbu7BxQKCS/t5lr6Xy+5plVOUSjrFjr/Pyfi0wOsBwtOr+fe//mMGNpNkJxPw7Y5pYpb+HKVKQ6eTkrNhJ4JKn/D9Ip7YwRJ3u5pNlza+sD/OhdnAdlP9dFKJnZbWgxUNvfYjIZU3kuyOaRHMJO7ZqO2kOZ1rBrOV/B8PCDoSVrhu4QeDBCJvPQ9s4Oz4Kfzg6/ufH/Yd/VE/HreCb4NvgIAiDn4KHwV/B0+A0oL3bvbD3S+/X/m/9qM/78zX1Rq/y+TrY+vr2HQ7qyOw=</latexit>

d(↵, �) = minw,b,v

1

2kwk2 + h↵,1i � h↵, Y Xwi+ h↵, yib

<latexit sha1_base64="4AgOjlY9CM50aZVRqLoTS1BqiD4=">AAAHlXicfVVbjxM3FJ5AaSBAu1BRHvpidQEtJawy20rtCxJ0EaIPKy5il63wbuRxPBk3vgy2Z5PgHal/s3+g/6JSj2cTlMxM8MPMsb/vHB9/x5ckF9y6weCfzqXLX135unv1Wu/6jZvffLt16/aR1YWh7JBqoc1xQiwTXLFDx51gx7lhRCaCvU8m+wF/f8aM5Vq9c/OcnUgyVjzllDgYGm793RvtYCLyjPTxmEhJHqInCEuuhn7aT/pnqHyAcEZEivD5FJ+f7iGEeo8QFgQt3BDWisEoNgT10ONVBP15PF0C6z7zi+EE9YZb24PdQdVQ04gXxna0aK+Ht678h0eaFpIpRwWx9kM8yN2JJ8ZxKljZw4VlOaETMmaeSGvnMinRfUlcZutYGGzFZpWyZa93f2X0Q+HS3048V3nhmKLgCFhaCOQ0CtqiETeMOjEHg1DDIR9EM2IIdVCBtVA+0XriSGJhCqzYlGqQXo08FlxC2ZktPTZ8nDlijJ4CqUFLgeHYzPk/UlSWNdRlTC1xFDqBss4JNQVKCtn5uPR79RiGhBSIGleSrkUPiGhDrAMkS/TMI9zHfVskf4EeIE/oVSms87k6K/2pfxzXJ3c8ZE+KNmySB5/JoyYgK6DpoXPI6xSyI6a+SEZEkDpsgCTxb+ueYOQr+H4DzzhXQUYwhn5/CCtK3bzOktotlcayaNFaa+dWaxGW3saZ1EnDSQvNrrEsH0vS1Dco/5nVNp+tUdrjAOUcK5LA0Z6dN0BJZqWfYzrSDi1ZNRITAq6P0t8L1jC+V4P3M76QnxLhjxvK0mcr8LM6nLoV9EVDqjDvRWlTH9cOCFQyXKnso9JhX7Oc6kLBMfbsY1FdnSVGOxiO1rL/MPg/Z3AjGXYAMV/lzBCnzU8whQSZDg6Oyo0ErrjknyCdpYUh6mgzn8yW/IX1ZT7UDq6D6ruJQsy4qhb8cT9YXyJytSTyzRFtZriCHbv4t9I8zo2GWavvBoaDHQgrvfjBDQIPRlx/HprG0d5u/PPu3ptftp/+vng6rkY/RD9GO1Ec/Ro9jV5Gr6PDiEb/dm507nS+797tPuk+7764oF7qLHy+i9Za99X/Z869OA==</latexit>

derivative for b

h↵, yi = 0<latexit sha1_base64="PBq3suWN2LcPuoCvfrfe6QvEuIo=">AAAHLXicfZVdj9Q2FIYDlOky/VrgBokbiwWJtstqsiDBDRKwFYKLVWnVXVbCy+jE40zMOHbWdmAGk/6a3tL+Gi6QUG/7Kyr1ODtTzSQZcpGc5H18fPz6I0khhXWDwcczZ899cb735caF/ldff/Ptd5sXLx1aXRrGD5iW2hwlYLkUih844SQ/KgyHPJH8eTLZC/rz19xYodVvblbw4xzGSqSCgcNPw80rhEogFGSRwTaZEUINkPsD0h9ubg12BvVF2kE8D7ai+fVsePH8v3SkWZlz5ZgEa1/Eg8IdezBOMMmrPi0tL4BNYMw95NbO8qQiN3JwmW1q4WOnNq1HXPX7N5a+vihdeu/YC1WUjiuGDVFLS0mcJmHMZCQMZ07OMABmBNZDWAYGmENnVlL5ROuJg8RiF1TxN0znOaiRp1LkOB3cVp4aMc4cGKPfINTCUiQcnzr/NCVV1VBdxtVCJ+ElIKtMBjKkSLE6H1d+t5nDQCgB1Li2dCV7UGSXYh0qWaKnntBtum3L5BX6gfaEt7qEVV6o15V/6W/Fzc6dCNVD2aVNitBm8mNbyGuh3UIXWNdLrA5Mc5AcZLA6LIAk8b82W2JQLOl7LT0TQgUbMRj6vSGOKHWzJpVrt3Ca5mWH11o7tzwXYehdzKQJDScdmF2hrBjn0PY3OP8/1dWfbSDdeRB5RxUkuLen71piDtPKzygbaUcWVAPiUmrFK389RMP4ekPey8TcfgbSH7WcZQ+X5IdNOXVL6uOWVaHf06lNfdzYIDiT4ajjJ0qHdc0LpkuF29jzk7I+0ipKblLcWov370P7nzieSIbvY86fC27AafMDdpGjTfv7h9VaQCiRi7dYziKimHW0nofpgp9Hn+dx7vA4qO/rEDDjerbwSbdD9DlQqAUo1me0mREKV+z82Yl5WhiNvdb3NYTDFYgjPX3gCYI/jLj5e2gHh7s78e2d3V/ubD14NP91bERXo2vRzSiO7kYPoifRs+ggYtHv0R/R++jP3l+9D71Pvb9P0bNn5m0uRytX75//AKupob0=</latexit>

remember this constraint!

DUAL: THE HARD WAY

This is too complicated. Let’s reduce it…

d(↵, �) = minw,b,v

1

2kwk2 + Ch1, vi+ h↵,1i � h↵, Y Xwi+ h↵, yib� h↵, vi � h�, vi

<latexit sha1_base64="tzcqA64NUKubds0LRnDeP+OGOeQ=">AAAH1nicfVVdk9s0FHUKJW2gdAuPvGjYlm7ZdCdemIEXZnZJh4GHnbYMuxum2s3IihyLyJIryfmo1rx1eOUv8TP4Mcxw5SSdxHbqB/v6nnOvrs7VR5QJbmyv92/r1gcf3v6ofedu5+NP7n16f+/BZxdG5Zqyc6qE0oOIGCa4ZOeWW8EGmWYkjQS7jCZ9j19OmTZcyd/sImNXKRlLHnNKLLiGe/90RgeYiCwhXTwmaUqeoB8QTrkculk36k5R8RXCCRExwjczfHN9jA77CAuCsJKsi9AUYU0QQp3DpXeZqkTB67EOerqJoN8HszWwHbNYuiMfsQVMV/ylFy3rfOce7u33jnrlg+pGuDL2g9XzYvjg9n94pGieMmmpIMa8CnuZvXJEW04FKzo4NywjdELGzJHUmEUaFehRSmxiqph3NmLzsjVFp/Now/sqt/H3V47LLLdMUggELM4Fsgr55qAR14xasQCDUM2hHkQTogm10MKtVC5SamJJZGAILNmMKtBEjhwWPIV1w0zhsObjxBKt1QxINVoMDMvm1v0So6KooDZhco0j/+Mp2xy/KIASQ3UuLNxxNYcmvgQix6WkW9k9IpoQYwFJIjV3CHdx1+TRH6AHyOP/yhK2+VxOC3ftnobVwS331ZO8CZtkPmZyWAfSEqhHqAzquobqiK5OkhHhpfYLIIrcr9VIMLINvF/DE86llxGMoesPYUaxXVRZqbJrpXGaN2itlLWbvfBTb+JMqqThpIFmtliGj1NS19cr/47VNJ6pUJrzAOUGSxLBxp7f1MCUzAu3wHSkLFqzKiQmBJw1hXvorWH4sAL3E76SnxLhBjVl6ekGfFqFY7uB/lSTyo+7bG3swsoGgU76M5m9lsqva5ZRlUvYxo69zsuzt8DoAMPWWv8/8fHPGJxImp1BzucZ08Qq/TUMkYJMZ2cXxU4Clzzlb6CctYUh62g3n8zX/JX1fj70Do6D8r2LQvS47BZ8cddb7yNyuSby3RlNormEFbv6NtIczrSCUcv3DoaFFQgzXX7gBIELI6xeD3Xj4vgo/Obo+OW3+yc/rq6OO8EXwZfBQRAG3wUnwc/Bi+A8oK3HrbPWReuyPWj/2X7b/mtJvdVaxXwebD3tv/8H3uHRYg==</latexit>

d(↵, �) = minw,b,v

1

2kwk2 + h↵,1i � h↵, Y Xwi+ h↵, yib+ hC1� ↵� �, vi

<latexit sha1_base64="0tUxxB+BYnZqQ/6FDHREs09r3gc=">AAAHunicfVVbb9s2FJa7de7cXdLtcS/E0g7p6gRWNmDDLkA7F8P6EKwrmjRFmRgUTVmcKVIlqdgeo5+zH9UfU6CHslTYklw9iIf8vnN4+B1eokxwY0ejN70bH31885P+rU8Htz/7/Isv9+58dWZUrik7pUoofR4RwwSX7NRyK9h5phlJI8FeRPOxx19cMW24ks/tKmMXKZlJHnNKLAxN9v4fTA8wEVlChnhG0pTcR78jnHI5cYthNLxCxXcIJ0TECF8v8PXlMUJo8ABhQVDlhrCSDEaxJmiADjcR9PJ8UQPbPqv1cFTHGpdBDisc2jKXIbpa8yZ7+6OjUfmhthFWxn5QfU8nd26+xVNF85RJSwUx5lU4yuyFI9pyKlgxwLlhGaFzMmOOpMas0qhA91JiE9PE/GAntizlLwaDexujr3Ib/3zhuMxyyyQFR8DiXCCrkC8AmnLNqBUrMAjVHPJBNCGaUAtl2grlIqXmlkQGpsCSLagCTeTUYcFT2BvMFA5rPkss0VotgNSixcCwbGndkxgVRQO1CZM1jnzHU7Y5vvBAiSE7FxbuuBlDE58CkbNS0q3oHhFdiLGAJJFaOoSHeGjy6F/QA+TxvTKFbT6XV4W7dIdhc3LLffYk78LmmfeZP2gDaQm0PVQGeV1CdkQ3F8mI8FL7DRBF7lnTE4xsAx+38IRz6WUEY+LGE1hRbFdNVqpsrTRO8w6tlbJ2sxZ+6V2ceZM0mXfQzBbL8FlK2vp65d+zuuYzDUp3HKBcY0kiOOfL6xaYkmXhVphOlUU1q0FiQsD1ULi73pqEdxvwOOGV/JQId95Slj7agB814dhuoH+2pPLzrksbu7BxQKCS/t5lr6Xy+5plVOUSjrFjr/Pyfi0wOsBwtOr+fe//mMGNpNkJxPw7Y5pYpb+HKVKQ6eTkrNhJ4JKn/D9Ip7YwRJ3u5pNlza+sD/OhdnAdlP9dFKJnZbWgxUNvfYjIZU3kuyOaRHMJO7ZqO2kOZ1rBrOV/B8PCDoSVrhu4QeDBCJvPQ9s4Oz4Kfzg6/ufH/Yd/VE/HreCb4NvgIAiDn4KHwV/B0+A0oL3bvbD3S+/X/m/9qM/78zX1Rq/y+TrY+vr2HQ7qyOw=</latexit>

d(↵, �) = minw,b,v

1

2kwk2 + h↵,1i � h↵, Y Xwi+ h↵, yib

<latexit sha1_base64="4AgOjlY9CM50aZVRqLoTS1BqiD4=">AAAHlXicfVVbjxM3FJ5AaSBAu1BRHvpidQEtJawy20rtCxJ0EaIPKy5il63wbuRxPBk3vgy2Z5PgHal/s3+g/6JSj2cTlMxM8MPMsb/vHB9/x5ckF9y6weCfzqXLX135unv1Wu/6jZvffLt16/aR1YWh7JBqoc1xQiwTXLFDx51gx7lhRCaCvU8m+wF/f8aM5Vq9c/OcnUgyVjzllDgYGm793RvtYCLyjPTxmEhJHqInCEuuhn7aT/pnqHyAcEZEivD5FJ+f7iGEeo8QFgQt3BDWisEoNgT10ONVBP15PF0C6z7zi+EE9YZb24PdQdVQ04gXxna0aK+Ht678h0eaFpIpRwWx9kM8yN2JJ8ZxKljZw4VlOaETMmaeSGvnMinRfUlcZutYGGzFZpWyZa93f2X0Q+HS3048V3nhmKLgCFhaCOQ0CtqiETeMOjEHg1DDIR9EM2IIdVCBtVA+0XriSGJhCqzYlGqQXo08FlxC2ZktPTZ8nDlijJ4CqUFLgeHYzPk/UlSWNdRlTC1xFDqBss4JNQVKCtn5uPR79RiGhBSIGleSrkUPiGhDrAMkS/TMI9zHfVskf4EeIE/oVSms87k6K/2pfxzXJ3c8ZE+KNmySB5/JoyYgK6DpoXPI6xSyI6a+SEZEkDpsgCTxb+ueYOQr+H4DzzhXQUYwhn5/CCtK3bzOktotlcayaNFaa+dWaxGW3saZ1EnDSQvNrrEsH0vS1Dco/5nVNp+tUdrjAOUcK5LA0Z6dN0BJZqWfYzrSDi1ZNRITAq6P0t8L1jC+V4P3M76QnxLhjxvK0mcr8LM6nLoV9EVDqjDvRWlTH9cOCFQyXKnso9JhX7Oc6kLBMfbsY1FdnSVGOxiO1rL/MPg/Z3AjGXYAMV/lzBCnzU8whQSZDg6Oyo0ErrjknyCdpYUh6mgzn8yW/IX1ZT7UDq6D6ruJQsy4qhb8cT9YXyJytSTyzRFtZriCHbv4t9I8zo2GWavvBoaDHQgrvfjBDQIPRlx/HprG0d5u/PPu3ptftp/+vng6rkY/RD9GO1Ec/Ro9jV5Gr6PDiEb/dm507nS+797tPuk+7764oF7qLHy+i9Za99X/Z869OA==</latexit>

derivative for wd(↵, �) = min

w,b,v

1

2kwk2 + h↵,1i � h↵, Y Xwi

<latexit sha1_base64="UtrlxXVeER7MkwhPbKuckRW7+qw=">AAAHfHicfVVbb9s2FFa7te68S9PtcS/Ekg7p4gRWOmx9KdAuxdA9BOuGJc1QJgZFUxZrXlSSiu0x+on9Afsde12LHSryYEty+SAd8vvO4eF3eElywa0bDv++cfOjj2/d7t35pP/pZ59/cXfr3penVheGshOqhTZnCbFMcMVOHHeCneWGEZkI9jKZHgX85SUzlmv1h1vk7FySieIpp8TB0Ghr0h/vYiLyjAzwhEhJHqDHCEuuRn42SAaXqPwW4YyIFOGrGb66OEQI9fcQFgTVbghrxWAUG4L6aH8VQX+ezWpgtLU9PBhWDbWNuDa2o7q9GN279Q6PNS0kU44KYu2reJi7c0+M41Swso8Ly3JCp2TCPJHWLmRSovuSuMw2sTDYic0rAct+//7K6KvCpY/OPVd54Zii4AhYWgjkNAoSojE3jDqxAINQwyEfRDNiCHUg9Foon2g9dSSxMAVWbEY1KKzGHgsuobrMlh4bPskcMUbPgNSipcBwbO78LykqywbqMqaWOAqdQFnnhNIBJYXsfFz6w2YMQ0IKRE0qSdeiB0R0IdYBkiV67hEe4IEtktegB8gTelUK63yuLkt/4ffj5uSOh+xJ0YVN8+Az3WsDsgLaHjqHvC4gO2Kai2REBKnDBkgS/3vTE4x8BT9q4RnnKsgIxsgfjWBFqVs0WVK7pdJYFh1aa+3cai3C0rs40yZpNO2g2TWW5RNJ2voG5f9ndc1nG5TuOEC5wookcLbnVy1QknnpF5iOtUNLVoPEhIBbovQ7wRrFOw34KOO1/JQIf9ZSlj5dgZ824dStoD+3pArzXpc29XHjgEAlw83J3igd9jXLqS4UHGPP3hTVDVlitIvhaC37D4L/MwY3kmHHEPPXnBnitPkOppAg0/HxabmRwBWX/C9IZ2lhiDrezCfzJb+2PsyH2sF1UH03UYiZVNWCPx4E60NErpZEvjmizQxXsGPrfyfN49xomLX6bmA42IGw0usf3CDwYMTN56FtnB4exA8PDn/7fvvJT/XTcSf6Ovom2o3i6MfoSfQ8ehGdRDR6G/0T/Ru9u/2+t9Pb6+1fU2/eqH2+itZa74f/AOmjuUQ=</latexit>

w �XTY ↵ = 0, or w = XTY ↵<latexit sha1_base64="iKsq7LOhn8QO0c57GsMiMUdB6SY=">AAAHRnicfZVBb9Q4FMcNC1MosLTLkYtFQQK2rSZlpeWCRClCy6GCXdFShNuR43EmZhw72A6dwc3n4NNwZfkKfAluiNNK+5zOoJkkQw7Ji/8/Pz//7ThxLoV13e6XM2d/OXe+s3Th4vKly1d+vbqy+tu+1YVhfI9pqc1BTC2XQvE9J5zkB7nhNIslfxkPd4L+8h03Vmj1wo1zfpjRgRKJYNRBU28lOsYb+ODoxStMqMxTih/g7jomjo8c9lgbXOJjaJsheitr3c1udeFmEE2CNTS5nvdWz/9H+poVGVeOSWrt66ibu0NPjRNM8nKZFJbnlA3pgHuaWTvO4hLfyqhLbV0Lja3aqLKiXF6+NdP6unDJ/UMvVF44rhh0BC0pJHYaBzNwXxjOnBxDQJkRUA9mKTWUObBsLpWPtR46GlsYgih+zHSWUdX3RIoM1onb0hMjBqmjxuhjgBpYAkSw1T9NcFnWVJdyNdVxeAnIPJNSGVIkUJ2PSr9Vz2FoKIGqQWXpXPagyDbFOlDSWI88Jutk3RbxG/AD7AlvVQnzvFDvSn/kN6L64E6E6mnRpg3z0Gf4e1PIKqHZQ+dQ1xFUR019kpzKYHXYAHHs/6n3hCCf0XcaeiqECjZC0PM7PZhR4sZ1KtNu6jTJihavtXZudi3C1NuYYR3qDVswO0dZMcho09/g/A+qbTxbQ9rzAHJCFI0lxaOThpjRUenHhPW1w1OqBnEpteKlvxmiXnSzJu+kYmI/o9IfNJxl2zPydl1O3Iz6pGFVGPd0aRMf1T4QWMlwBvK3Sod9zXOmCwWfsedvi+qsKwm+TeDTmr7fCf0fcziRDN+FnM9ybqjT5i4MkYFNu7v75UJAKJGJ91DONCKQtb+Yp6MpP4l+zsPawXFQ3Rch1Ayq1YInWQ/Rz0ChpqBYnNGmRijYsZNnK+ZJbjSMWt0XEA52IMz09AEnCPwwovrvoRnsb21G9za3/v5j7eGjya/jArqObqDbKEJ/oofoL/Qc7SGGPqCP6BP6t/O587XzrfP9FD17ZtLnGpq7ltD/1yCqFw==</latexit>

DUAL: THE HARD WAY

derivative for w

d(↵, �) = minw,b,v

1

2kwk2 + h↵,1i � h↵, Y Xwi

<latexit sha1_base64="UtrlxXVeER7MkwhPbKuckRW7+qw=">AAAHfHicfVVbb9s2FFa7te68S9PtcS/Ekg7p4gRWOmx9KdAuxdA9BOuGJc1QJgZFUxZrXlSSiu0x+on9Afsde12LHSryYEty+SAd8vvO4eF3eElywa0bDv++cfOjj2/d7t35pP/pZ59/cXfr3penVheGshOqhTZnCbFMcMVOHHeCneWGEZkI9jKZHgX85SUzlmv1h1vk7FySieIpp8TB0Ghr0h/vYiLyjAzwhEhJHqDHCEuuRn42SAaXqPwW4YyIFOGrGb66OEQI9fcQFgTVbghrxWAUG4L6aH8VQX+ezWpgtLU9PBhWDbWNuDa2o7q9GN279Q6PNS0kU44KYu2reJi7c0+M41Swso8Ly3JCp2TCPJHWLmRSovuSuMw2sTDYic0rAct+//7K6KvCpY/OPVd54Zii4AhYWgjkNAoSojE3jDqxAINQwyEfRDNiCHUg9Foon2g9dSSxMAVWbEY1KKzGHgsuobrMlh4bPskcMUbPgNSipcBwbO78LykqywbqMqaWOAqdQFnnhNIBJYXsfFz6w2YMQ0IKRE0qSdeiB0R0IdYBkiV67hEe4IEtktegB8gTelUK63yuLkt/4ffj5uSOh+xJ0YVN8+Az3WsDsgLaHjqHvC4gO2Kai2REBKnDBkgS/3vTE4x8BT9q4RnnKsgIxsgfjWBFqVs0WVK7pdJYFh1aa+3cai3C0rs40yZpNO2g2TWW5RNJ2voG5f9ndc1nG5TuOEC5wookcLbnVy1QknnpF5iOtUNLVoPEhIBbovQ7wRrFOw34KOO1/JQIf9ZSlj5dgZ824dStoD+3pArzXpc29XHjgEAlw83J3igd9jXLqS4UHGPP3hTVDVlitIvhaC37D4L/MwY3kmHHEPPXnBnitPkOppAg0/HxabmRwBWX/C9IZ2lhiDrezCfzJb+2PsyH2sF1UH03UYiZVNWCPx4E60NErpZEvjmizQxXsGPrfyfN49xomLX6bmA42IGw0usf3CDwYMTN56FtnB4exA8PDn/7fvvJT/XTcSf6Ovom2o3i6MfoSfQ8ehGdRDR6G/0T/Ru9u/2+t9Pb6+1fU2/eqH2+itZa74f/AOmjuUQ=</latexit>

w �XTY ↵ = 0, or w = XTY ↵<latexit sha1_base64="iKsq7LOhn8QO0c57GsMiMUdB6SY=">AAAHRnicfZVBb9Q4FMcNC1MosLTLkYtFQQK2rSZlpeWCRClCy6GCXdFShNuR43EmZhw72A6dwc3n4NNwZfkKfAluiNNK+5zOoJkkQw7Ji/8/Pz//7ThxLoV13e6XM2d/OXe+s3Th4vKly1d+vbqy+tu+1YVhfI9pqc1BTC2XQvE9J5zkB7nhNIslfxkPd4L+8h03Vmj1wo1zfpjRgRKJYNRBU28lOsYb+ODoxStMqMxTih/g7jomjo8c9lgbXOJjaJsheitr3c1udeFmEE2CNTS5nvdWz/9H+poVGVeOSWrt66ibu0NPjRNM8nKZFJbnlA3pgHuaWTvO4hLfyqhLbV0Lja3aqLKiXF6+NdP6unDJ/UMvVF44rhh0BC0pJHYaBzNwXxjOnBxDQJkRUA9mKTWUObBsLpWPtR46GlsYgih+zHSWUdX3RIoM1onb0hMjBqmjxuhjgBpYAkSw1T9NcFnWVJdyNdVxeAnIPJNSGVIkUJ2PSr9Vz2FoKIGqQWXpXPagyDbFOlDSWI88Jutk3RbxG/AD7AlvVQnzvFDvSn/kN6L64E6E6mnRpg3z0Gf4e1PIKqHZQ+dQ1xFUR019kpzKYHXYAHHs/6n3hCCf0XcaeiqECjZC0PM7PZhR4sZ1KtNu6jTJihavtXZudi3C1NuYYR3qDVswO0dZMcho09/g/A+qbTxbQ9rzAHJCFI0lxaOThpjRUenHhPW1w1OqBnEpteKlvxmiXnSzJu+kYmI/o9IfNJxl2zPydl1O3Iz6pGFVGPd0aRMf1T4QWMlwBvK3Sod9zXOmCwWfsedvi+qsKwm+TeDTmr7fCf0fcziRDN+FnM9ybqjT5i4MkYFNu7v75UJAKJGJ91DONCKQtb+Yp6MpP4l+zsPawXFQ3Rch1Ayq1YInWQ/Rz0ChpqBYnNGmRijYsZNnK+ZJbjSMWt0XEA52IMz09AEnCPwwovrvoRnsb21G9za3/v5j7eGjya/jArqObqDbKEJ/oofoL/Qc7SGGPqCP6BP6t/O587XzrfP9FD17ZtLnGpq7ltD/1yCqFw==</latexit>

d(↵) =1

2kXTY ↵k2 + h↵,1i � h↵, Y XXTY ↵i

<latexit sha1_base64="/CAC3Cq5e91D2laNkd99CdcX0XU=">AAAHenicfZVdb9s2FIbVdq0776Npd7kbommHZEkDKyuw3Qxol2HYLoJ1Q5O6KBPjiKYszhSpklRrj9Ev3C/Yz9jtPoAdyvZmS3J1IR3xfUgevvxKCimsGwx+v3b9xns3b/Vuv9//4MOPPr6zc/feudWlYfyMaanNMAHLpVD8zAkn+bAwHPJE8hfJ9CToL95wY4VWz9284Bc5TJRIBQOHRaMd3h/vUZBFBvvka/IZoRnIlNCr4eXzl4tyenV5TAjpHxAqgSzKDgnVimMpNUD65NG6Ql4O/6+8JEY7u4OjQf2QdhAvg91o+Twb3b35Dx1rVuZcOSbB2lfxoHAXHowTTPKqT0vLC2BTmHAPubXzPKnIwxxcZptaKOzUZrV9Vb//cK30VenSry68UEXpuGJYEbW0lMRpEgwkY2E4c3KOATAjMB/CMjDAHNq80ZRPtJ46SCx2QRV/y3Segxp7KkWOc8tt5akRk8yBMfotQi0sRcLxmfM/pKSqGqrLuFrpJPwEZJMJk4lIitn5uPLHzTYMhBRATWpLN1oPiuxSrEMlS/TME3pID22Z/IJ+oD3hr05hkxfqTeUv/aO42bkTIXsou7RpEepMD9pCXgvtGrrAvC4xOzDNQXKQweqwAJLE/9ysiUGxpp+09EwIFWzEYORPRjii1M2bVK7dymmalx1ea+3c+lyEoXcx0yY0mnZgdoOyYpJD29/g/H9UV3+2gXS3g8gVVZDgJp9dtcQcZpWfUzbWjqyoBsSlxOOi8g9CNIofNOSTTCztZyD9sOUse7omP23KqVtTv2tZFfpdTG3q48YGwZkM5yZ/rXRY17xgulS4jT1/XdbnY0XJHsWttfrfD/W/5XgiGX6Kbf5YcANOm8+xixxtOj09r7YCQolc/IrprCKKrY638zBb8cvo3TzOHR4H9XsbAmZSzxZ+6WGI3gUKtQLF9hZtZoTCFbv8dmKeFkZjr/V7C+FwBeJIFx88QfDCiJvXQzs4Pz6Kvzg6/unx7pNvllfH7ejT6H60F8XRl9GT6PvoWXQWsei36I/oz+ivW3/37vf2ewcL9Pq1ZZ1Poo2n9/hfloC4gw==</latexit>

d(↵) =1

2kXTY ↵k2 + h↵,1i � kXTY ↵k2

<latexit sha1_base64="eFs02FQiA1EBAAGFL3hN7T6/+es=">AAAHa3icfZVdb9s2FIbVT3feV9redbsglmZI1zSw0gLtzYB2GYbtIlg3NGmGMjGOaMpiTZEqSbX2GP2z/pHd7nbbTxiwQ9kebEmuLqQjvg/Jw5dfSSGFdYPBH5cuX7l67Xrvxkf9jz/59LPPt27eOrG6NIwfMy21OU3AcikUP3bCSX5aGA55IvnLZHIY9JdvubFCqxduVvCzHMZKpIKBw6Lh1kl/tEtBFhncI9+SrwnNQKaEXpyev/htXk4vzg8IIf37hEog87I9QrXiWEoNkD550OT7w63twf6gfkg7iBfBdrR4ng9vXvuXjjQrc64ck2Dtq3hQuDMPxgkmedWnpeUFsAmMuYfc2lmeVGQnB5fZphYKO7Vp7VfV7++slL4qXfrkzAtVlI4rhhVRS0tJnCbBMTIShjMnZxgAMwLzISwDA8yhr2tN+UTriYPEYhdU8XdM5zmokadS5DiZ3FaeGjHOHBij3yHUwlIkHJ86/1NKqqqhuoyrpU7CT0DWmTB7iKSYnY8rf9Bsw0BIAdS4tnSt9aDILsU6VLJETz2he3TPlslr9APtCX91Cuu8UG8rf+4fxM3OnQjZQ9mlTYpQZ3K/LeS10K6hC8zrHLMD0xwkBxmsDgsgSfyvzZoYFCv6YUvPhFDBRgyG/nCII0rdrEnl2i2dpnnZ4bXWzq3ORRh6FzNpQsNJB2bXKCvGObT9Dc7/T3X1ZxtIdzuIXFAFCe736UVLzGFa+RllI+3IkmpAXEo8Hyp/N0TD+G5DPszEwn4G0p+2nGXPVuRnTTl1K+oPLatCv/OpTX3c2CA4k+Gg5G+UDuuaF0yXCrex52/K+kCsKNmluLWW//dC/e85nkiGH2GbPxfcgNPmG+wiR5uOjk6qjYBQIhe/YzrLiGKro808TJf8Ivowj3OHx0H93oSAGdezhV+6F6IPgUItQbG5RZsZoXDFLr6dmKeF0dhr/d5AOFyBONL5B08QvDDi5vXQDk4O9uOH+we/PNp++t3i6rgRfRF9Fe1GcfQ4ehr9GD2PjiMWvY/+jP6K/r7+T+92707vyzl6+dKizu1o7ent/AcVw7PW</latexit>

d(↵) =� 1

2kXTY ↵k2 + h↵,1i

<latexit sha1_base64="YHOuSYKR+djuyT3yfe5SeRYDiWg=">AAAHWnicfZVdc9w0FIbdQLutCzQF7rjRkJZJaZpZB2bghpmWAAMXGQrTpGGqZOdYK6/FypIrye0uin8Rv4ZLPv4KMxx5d5ld21tf2Md6H0lHr77SUgrrhsM/r+289fb1G4Obt+Lb77z73p3du++fWV0Zxk+Zltqcp2C5FIqfOuEkPy8NhyKV/Hk6PQ7681fcWKHVMzcv+UUBEyUywcBh0Wj323i8T0GWOTwgX5FPyCOag8wIvTq/fPbLQqBXl0eEkPghoRLIouyAUK04llIDJI5Hu3vDw2HzkG6QLIO9aPk8Hd29/i8da1YVXDkmwdoXybB0Fx6ME0zyOqaV5SWwKUy4h8LaeZHW5H4BLrdtLRT2arPGnjqO76+Vvqhc9uWFF6qsHFcMK6KWVZI4TYJBZCwMZ07OMQBmBOZDWA4GmEMbN5ryqdZTB6nFLqjir5kuClBjT6UocO64rT01YpI7MEa/RqiDZUg4PnP+h4zUdUt1OVcrnYSfgGwyYa4QyTA7n9T+qN2GgZACqElj6UbrQZF9inWo5KmeeUIP6IGt0l/RD7Qn/DUpbPJCvar9pX+UtDt3ImQPVZ82LUOd6cOuUDRCt4YuMa9LzA5Me5AcZLA6LIA09T+3a2JQrunHHT0XQgUbMRj54xGOKHPzNlVot3KaFlWP11o7tz4XYeh9zLQNjaY9mN2grJgU0PU3OP8/1defbSH97SByRRWkuLtnVx2xgFnt55SNtSMrqgVxKfE0qP29EI2Sey35OBdL+xlIf95xlj1Zk5+05cytqd91rAr9LqY280lrg+BMhnORv1Q6rGteMl0p3Maev6ya86+mZJ/i1lr9Pwj1v+F4Ihl+gm3+WHIDTptPsYsCbTo5Oau3AkKJQvyG6awiiq2Ot/MwW/HL6M08zh0eB817GwJm0swWfulBiN4ECrUCxfYWbW6EwhW7/PZinpZGY6/NewvhcAXiSBefuLkwkvb10A3Ojg6Tzw6Pfvp87/HXy6vjZvRR9HG0HyXRF9Hj6PvoaXQasej36I/or+jvG/8Mdga3BrcX6M61ZZ0Poo1n8OF/kDetBw==</latexit>

DUAL: THE HARD WAYDual problem…

d(↵) =� 1

2kXTY ↵k2 + h↵,1i

<latexit sha1_base64="YHOuSYKR+djuyT3yfe5SeRYDiWg=">AAAHWnicfZVdc9w0FIbdQLutCzQF7rjRkJZJaZpZB2bghpmWAAMXGQrTpGGqZOdYK6/FypIrye0uin8Rv4ZLPv4KMxx5d5ld21tf2Md6H0lHr77SUgrrhsM/r+289fb1G4Obt+Lb77z73p3du++fWV0Zxk+Zltqcp2C5FIqfOuEkPy8NhyKV/Hk6PQ7681fcWKHVMzcv+UUBEyUywcBh0Wj323i8T0GWOTwgX5FPyCOag8wIvTq/fPbLQqBXl0eEkPghoRLIouyAUK04llIDJI5Hu3vDw2HzkG6QLIO9aPk8Hd29/i8da1YVXDkmwdoXybB0Fx6ME0zyOqaV5SWwKUy4h8LaeZHW5H4BLrdtLRT2arPGnjqO76+Vvqhc9uWFF6qsHFcMK6KWVZI4TYJBZCwMZ07OMQBmBOZDWA4GmEMbN5ryqdZTB6nFLqjir5kuClBjT6UocO64rT01YpI7MEa/RqiDZUg4PnP+h4zUdUt1OVcrnYSfgGwyYa4QyTA7n9T+qN2GgZACqElj6UbrQZF9inWo5KmeeUIP6IGt0l/RD7Qn/DUpbPJCvar9pX+UtDt3ImQPVZ82LUOd6cOuUDRCt4YuMa9LzA5Me5AcZLA6LIA09T+3a2JQrunHHT0XQgUbMRj54xGOKHPzNlVot3KaFlWP11o7tz4XYeh9zLQNjaY9mN2grJgU0PU3OP8/1defbSH97SByRRWkuLtnVx2xgFnt55SNtSMrqgVxKfE0qP29EI2Sey35OBdL+xlIf95xlj1Zk5+05cytqd91rAr9LqY280lrg+BMhnORv1Q6rGteMl0p3Maev6ya86+mZJ/i1lr9Pwj1v+F4Ihl+gm3+WHIDTptPsYsCbTo5Oau3AkKJQvyG6awiiq2Ot/MwW/HL6M08zh0eB817GwJm0swWfulBiN4ECrUCxfYWbW6EwhW7/PZinpZGY6/NewvhcAXiSBefuLkwkvb10A3Ojg6Tzw6Pfvp87/HXy6vjZvRR9HG0HyXRF9Hj6PvoaXQasej36I/or+jvG/8Mdga3BrcX6M61ZZ0Poo1n8OF/kDetBw==</latexit>

max<latexit sha1_base64="6Nc34djWBM+DLlOXzMOsY6xAtcc=">AAAHF3icfZXLbtQwFIbTQodSbi0s2US0lQqUalKQYNlShGBRURC9SHU7cjzOxIxjp7bTzuDmFdgCT8MOsWXJwyBxnM6gmSTTLJKT/J+Pj39fEqacadNs/pmavnJ1pnFt9vrcjZu3bt+ZX7i7p2WmCN0lkkt1EGJNORN01zDD6UGqKE5CTvfD7pbT90+p0kyKj6af0qMEdwSLGMHGfUIJ7rXmF5trzeLyq0EwCBa9wbXTWpj5i9qSZAkVhnCs9WHQTM2Rxcowwmk+hzJNU0y6uEMtTrTuJ2HuLyfYxLqsuY+1Wq8YWz43tzzy9TAz0Ysjy0SaGSoINAQtyrhvpO9G57eZosTwPgSYKAb1+CTGChMDHoylsqGUXYNDDV0gQc+ITBIs2hZxloDxVOcWKdaJDVZKngFUwSIgDO0Z+zby87ykmpiKoe67F4eMMzHmLkUE1dkgt+vlHAq7ErDoFJaOZXcKr1O0ASUOZc/6aBWt6iz8BH6APe6tKGGcZ+I0t8f2SVDu3DBXPc7qtG7q2nQfV4WkEKotZAp1HUN1WJUHSTF3VrsFEIb2Q7klBOmIvlXRY8aEsxGClt1qwYgi0y9TiTRDp1GS1XgtpTGjc+GGXsd0y1CrW4PpMUqzToKr/jrn/1N1/ekSUp8HkHMkcMix3zuviLC9c9tHpC2NP6RKEOVcCprbJRe1gqWSvBWzgf0Ec3tQcZZsjsibZTkyI+rrilWu34upjWxQ2iAwk+5QoydCunVNUyIzAdvY0pOsOLxy5K8g2FrD94eu/SsKJ5Ki25DzXUoVNlI9gi4SsGl7ey+fCDDBEvYZyhlGCLK2J/O4N+QH0eU8zB0cB8V9EoJVp5gteKJVF10GMjEE2eSMOlZMwIodPGsxi1IlodfiPoEwsAJhpBcPOEHghxGUfw/VYG99LXi6tv7+2eLGy8GvY9a77z3wVrzAe+5teG+8HW/XI17sffG+et8a3xs/Gj8bvy7Q6alBm3ve2NX4/Q/8F5uw</latexit>

subject to 0 ↵ C

h↵, yi = 0<latexit sha1_base64="rUBEKnAimXCMEOP86/g9vFmJGgc=">AAAHSHicfZXfjtQ2FMbDUgaYlrLQS26sLlSULqvJggQ3lYBBFb1YlVbdZSW8jE48zsSMYwfbgRlMHoSn6W3LG/Qtelf1Aonj7Ew1k2TIRXLi7+fjk89/khRSWDcY/H1m6+wX53rnL1zsf/nVpa8vb1+5emR1aRg/ZFpqc5yA5VIofuiEk/y4MBzyRPJnyXQY9GevubFCq9/dvOAnOUyUSAUDh02j7TvUOvIdIQMqOaEgiwxICIeU9rGZSli07pI5oQbIj2Qw2t4Z7A3qi7SDeBHsRIvr6ejKuY90rFmZc+WYBGufx4PCnXgwTjDJqz4tLS+ATWHCPeTWzvOkIjdycJltaqGxU5vVZlT9/o2V1uelS++feKGK0nHFsCNqaSmJ0yTYQcbCcObkHANgRmA9hGVggDk0bS2VT7SeOkgsDkEVf8N0noMaeypFjjPFbeWpEZPMgTH6DUItLEXC8ZnzP6ekqhqqy7ha6iS8BGSdyUCGFClW5+PK7zdzGAglgJrUlq5lD4rsUqxDJUv0zBO6S3dtmbxEP9Ce8FaXsM4L9bryL/ztuDm4E6F6KLu0aRH6TH9oC3kttHvoAut6gdWBaX4kBxmsDgsgSfxvzZ4YFCv6sKVnQqhgIwYjPxzhF6Vu3qRy7ZZO07zs8Fpr51bnInx6FzNtQqNpB2bXKCsmObT9Dc7/T3WNZxtIdx5E3lEFCe7s2buWmMOs8nPKxtqRJdWAuJRa8cpfD9Eovt6Qh5lY2M9A+uOWs+zhivywKaduRf2pZVUY93RqUx83NgjOZDgF+Sulw7rmBdOlwm3s+auyPu0qSm5S3FrL9+9D/8ccTyTDDzDnLwU34LS5hUPkaNPBwVG1ERBK5OItlrOMKGYdb+ZhtuQX0ed5nDs8Dur7JgTMpJ4tfNLdEH0OFGoJis0ZbWaEwhW7eHZinhZG46j1fQPhcAXil54+8ATBH0bc/D20g6P9vfjO3v6vd3cePFr8Oi5E16Jvo5tRHN2LHkRPoqfRYcSi99Ef0Z/RX70PvX96//b+O0W3ziz6fBOtXee3PgE4BKoC</latexit>

Solvable via “coordinate descent”LIBSVM

w = XTY ↵<latexit sha1_base64="EZE/b/C+np8MjqyQFo7Z6BqmcdQ=">AAAHJHicfZVNb9MwGMezwcooL9vgyCVimzRgTM1AggvSRhGCw8RAe0PLVjmu05g6dmY7W4uXT8IV+DTcEAcufBMkHmctapN0OSRP8v/58eO/XxIkjCrdaPyemr5ydaZ2bfZ6/cbNW7fn5hfu7CmRSkx2sWBCHgRIEUY52dVUM3KQSILigJH9oNu0+v4pkYoKvqP7CTmKUYfTkGKk4VNrfu7MfeEeHO98dH3Ekgi15hcba438csuBNwgWncG13VqY+eu3BU5jwjVmSKlDr5HoI4OkppiRrO6niiQId1GHGBQr1Y+DzF2OkY5UUbMfK7VePtCsXl8e+XqY6vD5kaE8STXhGBqCFqbM1cK1Q3XbVBKsWR8ChCWFelwcIYmwBkPGUplAiK5GgYIufE7OsIhjxNvGZzSGWSAqM76knUgjKcUZQCUsBEKTnjZvQzfLCqqOCB/qrn2xyDgTIWZThFCd8TKzXswhkS0B8U5u6Vh2q7AqRWlQokD0jOuv+qsqDT6BH2CPfctLGOcpP83MsXnsFTvX1FaP0iqtm9g23UdlIc6FcguRQF3HUB2SxUESxKzVdgEEgflQbAlBMqI3S3pEKbc2QtAyzRaMKNT9IhULPXTaj9MKr4XQenQu7NCrmG4RanUrMDVGKdqJUdlf6/x/qqo/VUCq8wBy7nMUMOT2zktijHqZ6fu4LbQ7pAoQYUxwkpklG7W8pYLcjOjAfoyYOSg5izdH5M2iHOoR9XXJKtvvxdSGxitsEJhJe8KREy7suiYJFimHbWzISZqfZJnvrviwtYbvD2z7VwROJEm2IOe7hEikhXwIXcRg09bWXjYRoJzG9DOUM4x8yNqezKPekB9El/Mwd3Ac5PdJCJKdfLbg6a/a6DKQ8iFIJ2dUkaQcVuzgWYkZP5ECes3vEwgNKxBGevGAEwR+GF7x91AO9tbXvCdr6++fLm68HPw6Zp17zn1nxfGcZ86G88bZdnYd7KTOF+er8632vfaj9rP26wKdnhq0ueuMXbU//wCBP5+G</latexit>

note:

DUAL: THE HARD WAYDual problem…

d(↵) =� 1

2kXTY ↵k2 + h↵,1i

<latexit sha1_base64="YHOuSYKR+djuyT3yfe5SeRYDiWg=">AAAHWnicfZVdc9w0FIbdQLutCzQF7rjRkJZJaZpZB2bghpmWAAMXGQrTpGGqZOdYK6/FypIrye0uin8Rv4ZLPv4KMxx5d5ld21tf2Md6H0lHr77SUgrrhsM/r+289fb1G4Obt+Lb77z73p3du++fWV0Zxk+Zltqcp2C5FIqfOuEkPy8NhyKV/Hk6PQ7681fcWKHVMzcv+UUBEyUywcBh0Wj323i8T0GWOTwgX5FPyCOag8wIvTq/fPbLQqBXl0eEkPghoRLIouyAUK04llIDJI5Hu3vDw2HzkG6QLIO9aPk8Hd29/i8da1YVXDkmwdoXybB0Fx6ME0zyOqaV5SWwKUy4h8LaeZHW5H4BLrdtLRT2arPGnjqO76+Vvqhc9uWFF6qsHFcMK6KWVZI4TYJBZCwMZ07OMQBmBOZDWA4GmEMbN5ryqdZTB6nFLqjir5kuClBjT6UocO64rT01YpI7MEa/RqiDZUg4PnP+h4zUdUt1OVcrnYSfgGwyYa4QyTA7n9T+qN2GgZACqElj6UbrQZF9inWo5KmeeUIP6IGt0l/RD7Qn/DUpbPJCvar9pX+UtDt3ImQPVZ82LUOd6cOuUDRCt4YuMa9LzA5Me5AcZLA6LIA09T+3a2JQrunHHT0XQgUbMRj54xGOKHPzNlVot3KaFlWP11o7tz4XYeh9zLQNjaY9mN2grJgU0PU3OP8/1defbSH97SByRRWkuLtnVx2xgFnt55SNtSMrqgVxKfE0qP29EI2Sey35OBdL+xlIf95xlj1Zk5+05cytqd91rAr9LqY280lrg+BMhnORv1Q6rGteMl0p3Maev6ya86+mZJ/i1lr9Pwj1v+F4Ihl+gm3+WHIDTptPsYsCbTo5Oau3AkKJQvyG6awiiq2Ot/MwW/HL6M08zh0eB817GwJm0swWfulBiN4ECrUCxfYWbW6EwhW7/PZinpZGY6/NewvhcAXiSBefuLkwkvb10A3Ojg6Tzw6Pfvp87/HXy6vjZvRR9HG0HyXRF9Hj6PvoaXQasej36I/or+jvG/8Mdga3BrcX6M61ZZ0Poo1n8OF/kDetBw==</latexit>

max<latexit sha1_base64="6Nc34djWBM+DLlOXzMOsY6xAtcc=">AAAHF3icfZXLbtQwFIbTQodSbi0s2US0lQqUalKQYNlShGBRURC9SHU7cjzOxIxjp7bTzuDmFdgCT8MOsWXJwyBxnM6gmSTTLJKT/J+Pj39fEqacadNs/pmavnJ1pnFt9vrcjZu3bt+ZX7i7p2WmCN0lkkt1EGJNORN01zDD6UGqKE5CTvfD7pbT90+p0kyKj6af0qMEdwSLGMHGfUIJ7rXmF5trzeLyq0EwCBa9wbXTWpj5i9qSZAkVhnCs9WHQTM2Rxcowwmk+hzJNU0y6uEMtTrTuJ2HuLyfYxLqsuY+1Wq8YWz43tzzy9TAz0Ysjy0SaGSoINAQtyrhvpO9G57eZosTwPgSYKAb1+CTGChMDHoylsqGUXYNDDV0gQc+ITBIs2hZxloDxVOcWKdaJDVZKngFUwSIgDO0Z+zby87ykmpiKoe67F4eMMzHmLkUE1dkgt+vlHAq7ErDoFJaOZXcKr1O0ASUOZc/6aBWt6iz8BH6APe6tKGGcZ+I0t8f2SVDu3DBXPc7qtG7q2nQfV4WkEKotZAp1HUN1WJUHSTF3VrsFEIb2Q7klBOmIvlXRY8aEsxGClt1qwYgi0y9TiTRDp1GS1XgtpTGjc+GGXsd0y1CrW4PpMUqzToKr/jrn/1N1/ekSUp8HkHMkcMix3zuviLC9c9tHpC2NP6RKEOVcCprbJRe1gqWSvBWzgf0Ec3tQcZZsjsibZTkyI+rrilWu34upjWxQ2iAwk+5QoydCunVNUyIzAdvY0pOsOLxy5K8g2FrD94eu/SsKJ5Ki25DzXUoVNlI9gi4SsGl7ey+fCDDBEvYZyhlGCLK2J/O4N+QH0eU8zB0cB8V9EoJVp5gteKJVF10GMjEE2eSMOlZMwIodPGsxi1IlodfiPoEwsAJhpBcPOEHghxGUfw/VYG99LXi6tv7+2eLGy8GvY9a77z3wVrzAe+5teG+8HW/XI17sffG+et8a3xs/Gj8bvy7Q6alBm3ve2NX4/Q/8F5uw</latexit>

subject to 0 ↵ C

h↵, yi = 0<latexit sha1_base64="rUBEKnAimXCMEOP86/g9vFmJGgc=">AAAHSHicfZXfjtQ2FMbDUgaYlrLQS26sLlSULqvJggQ3lYBBFb1YlVbdZSW8jE48zsSMYwfbgRlMHoSn6W3LG/Qtelf1Aonj7Ew1k2TIRXLi7+fjk89/khRSWDcY/H1m6+wX53rnL1zsf/nVpa8vb1+5emR1aRg/ZFpqc5yA5VIofuiEk/y4MBzyRPJnyXQY9GevubFCq9/dvOAnOUyUSAUDh02j7TvUOvIdIQMqOaEgiwxICIeU9rGZSli07pI5oQbIj2Qw2t4Z7A3qi7SDeBHsRIvr6ejKuY90rFmZc+WYBGufx4PCnXgwTjDJqz4tLS+ATWHCPeTWzvOkIjdycJltaqGxU5vVZlT9/o2V1uelS++feKGK0nHFsCNqaSmJ0yTYQcbCcObkHANgRmA9hGVggDk0bS2VT7SeOkgsDkEVf8N0noMaeypFjjPFbeWpEZPMgTH6DUItLEXC8ZnzP6ekqhqqy7ha6iS8BGSdyUCGFClW5+PK7zdzGAglgJrUlq5lD4rsUqxDJUv0zBO6S3dtmbxEP9Ce8FaXsM4L9bryL/ztuDm4E6F6KLu0aRH6TH9oC3kttHvoAut6gdWBaX4kBxmsDgsgSfxvzZ4YFCv6sKVnQqhgIwYjPxzhF6Vu3qRy7ZZO07zs8Fpr51bnInx6FzNtQqNpB2bXKCsmObT9Dc7/T3WNZxtIdx5E3lEFCe7s2buWmMOs8nPKxtqRJdWAuJRa8cpfD9Eovt6Qh5lY2M9A+uOWs+zhivywKaduRf2pZVUY93RqUx83NgjOZDgF+Sulw7rmBdOlwm3s+auyPu0qSm5S3FrL9+9D/8ccTyTDDzDnLwU34LS5hUPkaNPBwVG1ERBK5OItlrOMKGYdb+ZhtuQX0ed5nDs8Dur7JgTMpJ4tfNLdEH0OFGoJis0ZbWaEwhW7eHZinhZG46j1fQPhcAXil54+8ATBH0bc/D20g6P9vfjO3v6vd3cePFr8Oi5E16Jvo5tRHN2LHkRPoqfRYcSi99Ef0Z/RX70PvX96//b+O0W3ziz6fBOtXee3PgE4BKoC</latexit>

“kernel form”

d(↵) =� 1

2↵TY XXTY ↵+ h↵,1i

<latexit sha1_base64="3iQUkMDEXU42f6dryRFfcJGWnSc=">AAAHX3icfZXfjtw0FMbTQjslQNnCFeLGYlu0pdvVZEGCG6SWRQiQVhTU3Q6qd0cnHmdixrFT22lncPNOPA0SV8CDVOpxZgbNJJnmIjn297N9/PlP0lIK64bDv69cfevta9cHN96J333v/Zsf7N368NzqyjB+xrTUZpSC5VIofuaEk3xUGg5FKvmTdHYS9CfPubFCq8duUfKLAqZKZIKBw6rx3k/x5ICCLHO4S74hn5H7NAeZEbKsu3z822iEr2WJEBLfI1TCSj0kVCuOtdQAiePx3v7waNg8pBskq2A/Wj2PxreuvaITzaqCK8ckWPs0GZbuwoNxgklex7SyvAQ2gyn3UFi7KNKa3CnA5bathcpebd54VMfxnY3ap5XLvr7wQpWV44phQ9SyShKnSXCJTIThzMkFBsCMwHwIy8EAc+jlVlc+1XrmILU4BFX8BdNFAWriqRQFLiC3tadGTHMHxugXCHWwDAnH587/mJG6bqku52qtk1AIyDYTFgyRDLPzSe2P230YCCmAmjaWbvUeFNmnWIdKnuq5J/SQHtoq/R39QHtCqUlhmxfqee0v/f2kPbgTIXuo+rRZGdrM7nWFohG6LXSJeV1idmDak+Qgg9VhA6Sp/7XdEoNyQz/p6LkQKtiIwdifjHFGmVu0qUK7tdO0qHq81tq5zbUIU+9jZm1oPOvB7BZlxbSArr/B+f+pvvFsC+nvB5GXVEGKp3v+siMWMK/9grKJdmRNtSAuJd4Gtb8donFyuyWf5GJlPwPpRx1n2cMN+WFbztyG+n3HqjDucmkzn7QOCK5kuBz5M6XDvuYl05XCY+z5s6q5BGtKDigerXX5bmj/HccbyfBT7PPnkhtw2nyOQxRo0+npeb0TEEoU4g9MZx1R7HWym4f5ml9Fb+Zx7fA6aN67EDDTZrXwSw9D9CZQqDUodvdocyMU7tjVtxfztDQaR23eOwiHOxBnuvzEzQ8jaf8eusH58VHyxdHxL1/uP/h29eu4EX0SfRodREn0VfQg+iF6FJ1FLPoz+iv6J/r3+n+DweDmYG+JXr2yavNRtPUMPn4N08yu3g==</latexit>

kernel

KERNELS

kernel function

Dual SVM: only need to know similarity function

Kernel methods: replace inner product with some other similarity

(XXT )ij = hxi, xji =(big, when xi and xj are close

small, when xi and xj are far apart<latexit sha1_base64="zCXyyRQJiuetWOlrUOCO3uS02Ss=">AAAHyHicjVVbj9w0FM4UypThti1PiBeLbaUtTFeTpRK8IFoWoRZpRUHd7Uj1buR4nIl3fEltpzuDmxf+Er+GH4PEcTZTzSSZijwkJ+f7zsXfcZy0ENy6yeSfwY333r/5wfDWh6OPPv7k08/2bt85s7o0lJ1SLbSZpsQywRU7ddwJNi0MIzIV7EW6OA74i9fMWK7Vc7cq2Lkkc8UzTokDV7L398F0evH8fuL5ZYV+QFgQtEz4GG6X2JDgSdmcK0+hhq0QdmzpfMrnY3SVM4WqwG68iKhZ7bh86zAMUaEtqzAeXfusJEL87+CMGEQKYlw1wkzNmi6Svf3J4aS+UNeIG2M/aq5nye2b/+KZpqVkylFBrH0ZTwp37iExp4JB8tKygtAFmTNPpLUrmVboniQut20sOHuxZT2MajS6t+F9Wbrs+3PPVVE6pigEApaVAjmNwjjQjBtGnViBQajh0A+iOTGEOhjaViqfar1wJLVQAit2RbWUIJrHgkvYKaCLx4bPc0eM0VdA6tAyYNTaPs1QVbVQBxNZ48jV42kXyokIKTLozseVP2rnMCS0QNS8lnQre0BEH2IdIHmqlx7hMR7bMr0EPUCe8Fa3sM3n6nXlL/yDuF3c8dA9KfuwRRFiFt90AVkD3QhdQF8X0B0x7UUyIoLUYQOkqf+jHQlGsYEfd/CccxVkBCPxxwmsKHOrNktqt1Yay7JHa62d25xFWHofZ9EmJYsemt1iWT6XpKtvUP4tq6+ebVH68wDlDVYkDSfNmw4oybLyK0xn2qE1q0ViQmjFKn83WEl8twUf57yRnxLhpx1l6eMN+HEbztwG+ktHqlD3erSZj1sfCEwynMLsldJhX7OC6lLBZ+zZq7I+bSuMDjB8Wuv3+yH+ZwYnkmEnkPO3ghnitPkaSkiQ6eTkrNpJ4IpL/ie0s7YwZJ3t5pPlmt9Y7+bD7OA4qO+7KMTM62nBE4+D9S4iV2si353R5oYr2LHNs5fmcWE0VK3vOxgOdiCs9PoBJwj8MOL276FrnB0dxt8eHv3+cP/RT82v41b0ZfRVdBDF0XfRo+hJ9Cw6jejgi8GPgyeDp8Nfh8Xwari6pt4YNDGfR1vX8K//AM2L2KQ=</latexit>

k(xi, xj) =

(big, when xi and xj are “similar”

small, when xi and xj are “di↵erent”<latexit sha1_base64="9M6uTQsrt6GUc0ZeIuNPWPdg9HU=">AAAHv3iclVVdb9s2FJX7Mbdet6Xd416IpkXTzQustGj3MjRZhmF7CNYOSxqgTFWKpiTGFKmSVGOP1R/qP+qPGbBLRS5sSS4wPUhXPIfnXp5LUXEhuLGTycfBlavXrn8xvHFz9OWtr77+Zuv2nROjSk3ZMVVC6dOYGCa4ZMeWW8FOC81IHgv2Mp4devzlO6YNV/JvuyjYWU5SyRNOiYWhaOvDbGce8fE8On+IfkY4ZimXjoKgqRC2bG5dzNMxusiYRBUCajOKiJzWA+efBjRDb94YnnNB9IMHFcajS8TkRIj/ITHlScI0kxZERpjJaVNPtLU92Z3UF+oGYRNsB831PLp9/V88VbTMQYsKYsyrcFLYM0e05VQwEC8NKwidkZQ5khuzyOMK3c+JzUwb84O92LzuQTUa3V8ZfVXa5Kczx2VRWiYpTAQsKQWyCvkuoCnXjFqxgIBQzaEeRDOiCbXQqzUpFys1syQ2kAJLdkFVnoNxDguewwYBXxzWPM0s0VpdAKlDS4BR+/tHgqqqhVroyhJHtm5RO1FGhJdIoDoXVm6vraGJL4HItLZ0Td0jog8xFpAsVnOH8BiPTRmfgx9gj3+rS1jnc/mucq/dj2E7ueW+elL2YbPCz5n90AXyGujOUAXU9RqqI7q9SEaEt9pvgDh2f7VnQlCs4IcdPONcehshiNxhBCtK7KLNypVdOo3zssdrpaxd7YVfeh9n1iZFsx6aWWMZnuak6693/hOrL59pUfp1gPIeSxILgubvO2BO5pVbYDpVFi1ZLRITQklWuXs+isJ7Lfgw4439lAh32nGWHqzAB204sSvobx2rfN7L1iYubH0g0El/+LK3Uvl9zQqqSgmfsWNvy/qQrTDawfBpLd8f+vm/MjiRNDsCzT8LpolV+ntIkYNNR0cn1UYCl3DA/gPlLCMMqtPNfDJf8pvo83zoHRwH9X0Thei07hY88dhHnyNyuSTyzYom01zCjm2evTSHC60ga33fwLCwA2Gllw84QeCHEbZ/D93gZG83fLS79+Lx9v4vza/jRvBdcDfYCcLgabAf/B48D44DOtgaPBk8G+wPD4bpUA6LS+qVQTPn22DtGi7+A4pY1VE=</latexit>

EXAMPLE: TWO MOONS

ï1.5 ï1 ï0.5 0 0.5 1 1.5 2 2.5ï0.8

ï0.6

ï0.4

ï0.2

0

0.2

0.4

0.6

0.8

1

1.2�(x)

<latexit sha1_base64="Lr1DRCZb1PXVXGhk7CPIlNjXN+c=">AAAG/HicfZVNU9QwGMcLyor4Bnr00hGYAUVmy+DoEVzH0QMjOvIyQ2Enzabb2DQpSQq7hvodvOrVm+PV7+LJr+KTsuvstl16aJ/2/8uTJ/+8NEgZVbrZ/DM1fe36TOPG7M25W7fv3L03v3B/X4lMYrKHBRPyMECKMMrJnqaakcNUEpQEjBwEccvqB2dEKir4R91PyXGCupyGFCMNnw78NKIrvdX2/GJzvVlcbjXwBsGiM7h22wszf/2OwFlCuMYMKXXkNVN9bJDUFDOSz/mZIinCMeoSgxKl+kmQu8sJ0pEqa/ZjrdYrhpfPzS2PfD3KdPji2FCeZppwDA1BCzPmauHaAbodKgnWrA8BwpJCPS6OkERYgw1jqUwgRKxRoKALn5NzLJIE8Y7xGU3Ae6Jy40vajTSSUpwDVMFCIDTpafM2dPO8pOqI8KHu2heLjDMRYjZFCNUZLzcb5RwS2RIQ7xaWjmW3CqtTlAYlCkTPuP6av6ay4BP4AfbYt6KEcZ7ys9ycmKdeuXNNbfUoq9Pi1LaJn1SFpBCqLUQKdZ1AdUiWB0kQs1bbBRAE5kO5JQTpiN6q6BGl3NoIQdu02jCiUPfLVCL00Gk/yWq8FkLr0bmwQ69j4jLUjmswNUYp2k1Q1V/r/H+qrj9VQurzAHLhcxQw5PYuKmKCernp+7gjtDukShBhTHCSmyUbtb2lktyK6MB+jJg5rDiLt0fk7bIc6hH1dWkLwFzZk4uccmFXLkmxyDhsVENOs+KEyn13xYfNM3xfte1fEThzJNmBpO9SIpEW8jH0kYAROzv7+USAcprQzzDQYeRD1s5kHvWG/CC6mofZgQ1f3CchSHaL+YCnv2ajq0DKhyCdnFFFknJYk4NnLWb8VArotbhPIDSsMRjp5QPOCPgleOUfQDXY31j3Ntefvd9c3Ho5+DnMOg+dR86K4znPnS3njbPr7DnYiZ2vzjfne+NL40fjZ+PXJTo9NWjzwBm7Gr//AdBWj84=</latexit>

I want a mapping that linearizes the problem. I can’t write down that mapping, but I can write down a similarity measure for the mapped

points!

EXAMPLE: TWO MOONSkernel function

ï1.5 ï1 ï0.5 0 0.5 1 1.5 2 2.5ï0.8

ï0.6

ï0.4

ï0.2

0

0.2

0.4

0.6

0.8

1

1.2

farclose

k(xi, xj) =

(big, when xi and xj are “similar”

small, when xi and xj are “di↵erent”<latexit sha1_base64="9M6uTQsrt6GUc0ZeIuNPWPdg9HU=">AAAHv3iclVVdb9s2FJX7Mbdet6Xd416IpkXTzQustGj3MjRZhmF7CNYOSxqgTFWKpiTGFKmSVGOP1R/qP+qPGbBLRS5sSS4wPUhXPIfnXp5LUXEhuLGTycfBlavXrn8xvHFz9OWtr77+Zuv2nROjSk3ZMVVC6dOYGCa4ZMeWW8FOC81IHgv2Mp4devzlO6YNV/JvuyjYWU5SyRNOiYWhaOvDbGce8fE8On+IfkY4ZimXjoKgqRC2bG5dzNMxusiYRBUCajOKiJzWA+efBjRDb94YnnNB9IMHFcajS8TkRIj/ITHlScI0kxZERpjJaVNPtLU92Z3UF+oGYRNsB831PLp9/V88VbTMQYsKYsyrcFLYM0e05VQwEC8NKwidkZQ5khuzyOMK3c+JzUwb84O92LzuQTUa3V8ZfVXa5Kczx2VRWiYpTAQsKQWyCvkuoCnXjFqxgIBQzaEeRDOiCbXQqzUpFys1syQ2kAJLdkFVnoNxDguewwYBXxzWPM0s0VpdAKlDS4BR+/tHgqqqhVroyhJHtm5RO1FGhJdIoDoXVm6vraGJL4HItLZ0Td0jog8xFpAsVnOH8BiPTRmfgx9gj3+rS1jnc/mucq/dj2E7ueW+elL2YbPCz5n90AXyGujOUAXU9RqqI7q9SEaEt9pvgDh2f7VnQlCs4IcdPONcehshiNxhBCtK7KLNypVdOo3zssdrpaxd7YVfeh9n1iZFsx6aWWMZnuak6693/hOrL59pUfp1gPIeSxILgubvO2BO5pVbYDpVFi1ZLRITQklWuXs+isJ7Lfgw4439lAh32nGWHqzAB204sSvobx2rfN7L1iYubH0g0El/+LK3Uvl9zQqqSgmfsWNvy/qQrTDawfBpLd8f+vm/MjiRNDsCzT8LpolV+ntIkYNNR0cn1UYCl3DA/gPlLCMMqtPNfDJf8pvo83zoHRwH9X0Thei07hY88dhHnyNyuSTyzYom01zCjm2evTSHC60ga33fwLCwA2Gllw84QeCHEbZ/D93gZG83fLS79+Lx9v4vza/jRvBdcDfYCcLgabAf/B48D44DOtgaPBk8G+wPD4bpUA6LS+qVQTPn22DtGi7+A4pY1VE=</latexit>

EXAMPLE: TWO MOONSkernel function

ï1.5 ï1 ï0.5 0 0.5 1 1.5 2 2.5ï0.8

ï0.6

ï0.4

ï0.2

0

0.2

0.4

0.6

0.8

1

1.2

farclose

k(x, y) = exp

✓�kx� yk2

2�2

◆

k(xi, xj) =

(big, when xi and xj are “similar”

small, when xi and xj are “di↵erent”<latexit sha1_base64="9M6uTQsrt6GUc0ZeIuNPWPdg9HU=">AAAHv3iclVVdb9s2FJX7Mbdet6Xd416IpkXTzQustGj3MjRZhmF7CNYOSxqgTFWKpiTGFKmSVGOP1R/qP+qPGbBLRS5sSS4wPUhXPIfnXp5LUXEhuLGTycfBlavXrn8xvHFz9OWtr77+Zuv2nROjSk3ZMVVC6dOYGCa4ZMeWW8FOC81IHgv2Mp4devzlO6YNV/JvuyjYWU5SyRNOiYWhaOvDbGce8fE8On+IfkY4ZimXjoKgqRC2bG5dzNMxusiYRBUCajOKiJzWA+efBjRDb94YnnNB9IMHFcajS8TkRIj/ITHlScI0kxZERpjJaVNPtLU92Z3UF+oGYRNsB831PLp9/V88VbTMQYsKYsyrcFLYM0e05VQwEC8NKwidkZQ5khuzyOMK3c+JzUwb84O92LzuQTUa3V8ZfVXa5Kczx2VRWiYpTAQsKQWyCvkuoCnXjFqxgIBQzaEeRDOiCbXQqzUpFys1syQ2kAJLdkFVnoNxDguewwYBXxzWPM0s0VpdAKlDS4BR+/tHgqqqhVroyhJHtm5RO1FGhJdIoDoXVm6vraGJL4HItLZ0Td0jog8xFpAsVnOH8BiPTRmfgx9gj3+rS1jnc/mucq/dj2E7ueW+elL2YbPCz5n90AXyGujOUAXU9RqqI7q9SEaEt9pvgDh2f7VnQlCs4IcdPONcehshiNxhBCtK7KLNypVdOo3zssdrpaxd7YVfeh9n1iZFsx6aWWMZnuak6693/hOrL59pUfp1gPIeSxILgubvO2BO5pVbYDpVFi1ZLRITQklWuXs+isJ7Lfgw4439lAh32nGWHqzAB204sSvobx2rfN7L1iYubH0g0El/+LK3Uvl9zQqqSgmfsWNvy/qQrTDawfBpLd8f+vm/MjiRNDsCzT8LpolV+ntIkYNNR0cn1UYCl3DA/gPlLCMMqtPNfDJf8pvo83zoHRwH9X0Thei07hY88dhHnyNyuSTyzYom01zCjm2evTSHC60ga33fwLCwA2Gllw84QeCHEbZ/D93gZG83fLS79+Lx9v4vza/jRvBdcDfYCcLgabAf/B48D44DOtgaPBk8G+wPD4bpUA6LS+qVQTPn22DtGi7+A4pY1VE=</latexit>

EXAMPLE: TWO MOONS

subject to 0 ↵ C

h↵, li = 0

kernel

subject to 0 ↵ C

h↵, li = 0

Return to this later!!

d(↵) =� 1

2↵TY XXTY ↵+ h↵,1i

<latexit sha1_base64="3iQUkMDEXU42f6dryRFfcJGWnSc=">AAAHX3icfZXfjtw0FMbTQjslQNnCFeLGYlu0pdvVZEGCG6SWRQiQVhTU3Q6qd0cnHmdixrFT22lncPNOPA0SV8CDVOpxZgbNJJnmIjn297N9/PlP0lIK64bDv69cfevta9cHN96J333v/Zsf7N368NzqyjB+xrTUZpSC5VIofuaEk3xUGg5FKvmTdHYS9CfPubFCq8duUfKLAqZKZIKBw6rx3k/x5ICCLHO4S74hn5H7NAeZEbKsu3z822iEr2WJEBLfI1TCSj0kVCuOtdQAiePx3v7waNg8pBskq2A/Wj2PxreuvaITzaqCK8ckWPs0GZbuwoNxgklex7SyvAQ2gyn3UFi7KNKa3CnA5bathcpebd54VMfxnY3ap5XLvr7wQpWV44phQ9SyShKnSXCJTIThzMkFBsCMwHwIy8EAc+jlVlc+1XrmILU4BFX8BdNFAWriqRQFLiC3tadGTHMHxugXCHWwDAnH587/mJG6bqku52qtk1AIyDYTFgyRDLPzSe2P230YCCmAmjaWbvUeFNmnWIdKnuq5J/SQHtoq/R39QHtCqUlhmxfqee0v/f2kPbgTIXuo+rRZGdrM7nWFohG6LXSJeV1idmDak+Qgg9VhA6Sp/7XdEoNyQz/p6LkQKtiIwdifjHFGmVu0qUK7tdO0qHq81tq5zbUIU+9jZm1oPOvB7BZlxbSArr/B+f+pvvFsC+nvB5GXVEGKp3v+siMWMK/9grKJdmRNtSAuJd4Gtb8donFyuyWf5GJlPwPpRx1n2cMN+WFbztyG+n3HqjDucmkzn7QOCK5kuBz5M6XDvuYl05XCY+z5s6q5BGtKDigerXX5bmj/HccbyfBT7PPnkhtw2nyOQxRo0+npeb0TEEoU4g9MZx1R7HWym4f5ml9Fb+Zx7fA6aN67EDDTZrXwSw9D9CZQqDUodvdocyMU7tjVtxfztDQaR23eOwiHOxBnuvzEzQ8jaf8eusH58VHyxdHxL1/uP/h29eu4EX0SfRodREn0VfQg+iF6FJ1FLPoz+iv6J/r3+n+DweDmYG+JXr2yavNRtPUMPn4N08yu3g==</latexit>

maximize<latexit sha1_base64="aUghQffHXKPEjxOJoXQe1HdlNus=">AAAHHHicfZXLbtQwFIZToEMptxaWbCLaSgVKNSlIsGwpQrCoKIheUNOOHI8zMeNLajvtDG6egi3wNOwQWyQeBonjdAbNJJlmkZzk/3xs/7ZPopRRbZrNP1OXLl+ZblyduTZ7/cbNW7fn5u/sapkpTHawZFLtR0gTRgXZMdQwsp8qgnjEyF7U3XT63glRmkrxwfRTcshRR9CYYmTg08eQox7l9DNpzS00V5vF5VeDYBAseINruzU//TdsS5xxIgxmSOuDoJmaQ4uUoZiRfDbMNEkR7qIOsYhr3edR7i9xZBJd1tzHWq1XTDCfnV0a+XqQmfj5oaUizQwRGBqCFmfMN9J3U/TbVBFsWB8ChBWF8fg4QQphA0aMpbKRlF2DIg1dhIKcYsk5Em0bMsrBfaJzGyraSQxSSp4CVMFiIAzpGfsm9vO8pJqEiKHuuxeHjDMJYi5FDKOzQW7XyjkUckNAolNYOpbdKaxO0QaUJJI964cr4YrOok/gB9jj3oohjPNUnOT2yD4Oyp0b6kaPsjqtm7o23UdVgRdCtYVMYVxHMDqkypMkiDmr3QaIIvu+3BKCdETfrOgJpcLZCEHLbrZgRrHplykuzdDpkGc1XktpzOhauKnXMd0y1OrWYHqM0rTDUdVf5/x/qq4/XULq8wByFgoUMeT3zioinPHc9kPclsYfUiWIMCYFye2ii1rBYkneTOjAfoyY3a84izdG5I2yHJsR9VXFKtfv+dLGNigdEFhJV9nIsZBuX5MUy0zAMbbkOCsqWB76yyEcreH7A9f+JYGKpMgW5HybEoWMVA+hCw42bW3t5hMBKopamNthFELW9mR+UDuBH0QX87B2UA6K+yQEqU6xWvAMV1x0EUjFEKSTM+pEUQE7dvCsxWyYKgm9FvcJhIEdCDM9f0AFgR9GUP49VIPdtdXgyerau6cL6y8Gv44Z755331v2Au+Zt+699ra9HQ973PviffW+Nb43fjR+Nn6do5emBm3uemNX4/c/BIqeAA==</latexit>

Kij = k(xi, xj)<latexit sha1_base64="fshFn6ZzHIQPocUCl/70wyvl/mg=">AAAHKHicfZVdT9RAFIaLyor4BXLpTSOQgCLZoonemIAYozFENPKRUGims9PtsNOZMjOFXYf+Fm/VX+Od4db/YeKZsmt22y69aE/7PnPmzDsfDVNGlW42LyauXb8x2bg5dWv69p279+7PzD7YVSKTmOxgwYTcD5EijHKyo6lmZD+VBCUhI3thZ9Pqe6dEKir4F91LyWGC2pxGFCMNn4KZuQ+Boce5+8rtLHUDutINjpeDmfnmarO43Grg9YN5p39tB7OTf/2WwFlCuMYMKXXgNVN9aJDUFDOST/uZIinCHdQmBiVK9ZIwdxcTpGNV1uzHWq1bDDafnl4c+nqQ6ejloaE8zTThGBqCFmXM1cK1w3VbVBKsWQ8ChCWFelwcI4mwBlNGUplQiI5GoYIufE7OsEgSxFvGZzSBmSAqN76k7VgjKcUZQBUsAkKTrjbvIzfPS6qOCR/orn2xyCgTI2ZTRFCd8XKzVs4hkS0B8XZh6Uh2q7A6RWlQ4lB0jeuv+CsqC4/BD7DHvhUljPKUn+bmyDz1yp1raqtHWZ3WSW2bzpOqkBRCtYVIoa4jqA7J8iAJYtZquwDC0Hwut4QgHdI3K3pMKbc2QhCYzQBGFOlemUqEHjjtJ1mN10JoPTwXduh1TKcMBZ0aTI1QirYTVPXXOv+fqutPlZD6PICc+xyFDLnd84qYoG5uej5uCe0OqBJEGBOc5GbBRoG3UJI3Y9q3HyNm9ivO4o0heaMsR3pIfVuxyvZ7ObWR8UobBGbSnnLkhAu7rkmKRcZhGxtykhWnWe67Sz5srcH7sm3/hsCJJMkW5PyYEom0kI+hiwRs2trazccClNOEfoVyBpEPWVvjedQd8P3oah7mDo6D4j4OQbJdzBY8/RUbXQVSPgDp+IwqlpTDiu0/azHjp1JAr8V9DKFhBcJILx9wgsAPwyv/HqrB7tqq92x17dPz+fXX/V/HlPPQeeQsOZ7zwll33jnbzo6DnZ7zzfnu/Gj8bPxq/G5cXKLXJvpt5pyRq/HnH1x3oTA=</latexit>

minimize↵

d(↵) =1

2↵TY KY ↵� h↵,1i

<latexit sha1_base64="ux1IprgwHj25FirK0W/QjzT528E=">AAAHbnicfVXdjtQ2FA5QGJjSsoDETYWwuiAt7bKaLEhwgwQsqlpVq9KKXXa13h05Hmdixj/BdmCmJs/W5+gD9LbtC1TqcSaDZpIMvkiO/X0+Pv6OfZzkgls3GPx57vyFLy5e6l2+0v/y6ldfX9u4fuPQ6sJQdkC10OYoIZYJrtiB406wo9wwIhPB3iSTvYC/ec+M5Vq9drOcnUoyVjzllDgYGm4cY8kVl/x3NvSYiDwjJUJotDW376M+9J4ihDMiUjQfPHuNjn8+rjsAP0BYkLq7jbBWDGFD+sONzcHOoGqobcS1sRnV7dXw+sX/8EjTQjLlqCDWnsSD3J16YhyngpV9XFiWEzohY+aJtHYmkxLdk8RltomFwU5sWklW9vv3lkZPCpc+OfVc5YVjisJEwNJCIKdREA2NuGHUiRkYhBoO8SCaEUOoA2lXXPlE64kjiYUlsGIfqJaSqJHHgkvIJ7Olx4aPM0eM0R+A1KKlwHBs6vxPKSrLBuoyphY4Cp1AWeWETAElheh8XPrdpg9DQghEjStJV7wHRHQh1gGSJXrqEd7G27ZI3oIeIE/oVSGs8rl6X/oz/yBuLu54iJ4UXdgkD3Mm37cBWQHtGTqHuM4gOmKam2REBKnDAUgS/1tzJhj5Er7XwjPOVZARjKHfG8KOUjdrsqR2C6WxLDq01tq55VyErXdxJk3ScNJBsyssy8eStPUNyn9ida1nG5RuP0D5iBVJ4GZPP7ZASaaln2E60g4tWA0SEwIqQenvBmsY323Aexmv5adE+KOWsvT5Evy8CaduCf2hJVVYd57a1MeNCwKZDLWSvVM6nGuWU10ouMaevSuqmlhitIXhai3698P8lwwqkmH74POXnBnitPkOlpAg0/7+YbmWUNfW0i8sDF5H6/lkuuDX1uf5kDsoB9V3HYWYcZUt+OPtYH2OyNWCyNd7tJnhCk5s/e+keZwbDatW3zUMBycQdjr/QQWBByNuPg9t43B3J364s/vro81nL+qn43L0TfRttBXF0ePoWfRj9Co6iGj0R/RX9Hf0z6V/e7d6t3t35tTz5+o5N6OV1tv6H4w/te8=</latexit>

CLASSIFYING TEST DATAd(↵) =� 1

2↵TY XXTY ↵+ h↵,1i

<latexit sha1_base64="3iQUkMDEXU42f6dryRFfcJGWnSc=">AAAHX3icfZXfjtw0FMbTQjslQNnCFeLGYlu0pdvVZEGCG6SWRQiQVhTU3Q6qd0cnHmdixrFT22lncPNOPA0SV8CDVOpxZgbNJJnmIjn297N9/PlP0lIK64bDv69cfevta9cHN96J333v/Zsf7N368NzqyjB+xrTUZpSC5VIofuaEk3xUGg5FKvmTdHYS9CfPubFCq8duUfKLAqZKZIKBw6rx3k/x5ICCLHO4S74hn5H7NAeZEbKsu3z822iEr2WJEBLfI1TCSj0kVCuOtdQAiePx3v7waNg8pBskq2A/Wj2PxreuvaITzaqCK8ckWPs0GZbuwoNxgklex7SyvAQ2gyn3UFi7KNKa3CnA5bathcpebd54VMfxnY3ap5XLvr7wQpWV44phQ9SyShKnSXCJTIThzMkFBsCMwHwIy8EAc+jlVlc+1XrmILU4BFX8BdNFAWriqRQFLiC3tadGTHMHxugXCHWwDAnH587/mJG6bqku52qtk1AIyDYTFgyRDLPzSe2P230YCCmAmjaWbvUeFNmnWIdKnuq5J/SQHtoq/R39QHtCqUlhmxfqee0v/f2kPbgTIXuo+rRZGdrM7nWFohG6LXSJeV1idmDak+Qgg9VhA6Sp/7XdEoNyQz/p6LkQKtiIwdifjHFGmVu0qUK7tdO0qHq81tq5zbUIU+9jZm1oPOvB7BZlxbSArr/B+f+pvvFsC+nvB5GXVEGKp3v+siMWMK/9grKJdmRNtSAuJd4Gtb8donFyuyWf5GJlPwPpRx1n2cMN+WFbztyG+n3HqjDucmkzn7QOCK5kuBz5M6XDvuYl05XCY+z5s6q5BGtKDigerXX5bmj/HccbyfBT7PPnkhtw2nyOQxRo0+npeb0TEEoU4g9MZx1R7HWym4f5ml9Fb+Zx7fA6aN67EDDTZrXwSw9D9CZQqDUodvdocyMU7tjVtxfztDQaR23eOwiHOxBnuvzEzQ8jaf8eusH58VHyxdHxL1/uP/h29eu4EX0SfRodREn0VfQg+iF6FJ1FLPoz+iv6J/r3+n+DweDmYG+JXr2yavNRtPUMPn4N08yu3g==</latexit>

maximize<latexit sha1_base64="aUghQffHXKPEjxOJoXQe1HdlNus=">AAAHHHicfZXLbtQwFIZToEMptxaWbCLaSgVKNSlIsGwpQrCoKIheUNOOHI8zMeNLajvtDG6egi3wNOwQWyQeBonjdAbNJJlmkZzk/3xs/7ZPopRRbZrNP1OXLl+ZblyduTZ7/cbNW7fn5u/sapkpTHawZFLtR0gTRgXZMdQwsp8qgnjEyF7U3XT63glRmkrxwfRTcshRR9CYYmTg08eQox7l9DNpzS00V5vF5VeDYBAseINruzU//TdsS5xxIgxmSOuDoJmaQ4uUoZiRfDbMNEkR7qIOsYhr3edR7i9xZBJd1tzHWq1XTDCfnV0a+XqQmfj5oaUizQwRGBqCFmfMN9J3U/TbVBFsWB8ChBWF8fg4QQphA0aMpbKRlF2DIg1dhIKcYsk5Em0bMsrBfaJzGyraSQxSSp4CVMFiIAzpGfsm9vO8pJqEiKHuuxeHjDMJYi5FDKOzQW7XyjkUckNAolNYOpbdKaxO0QaUJJI964cr4YrOok/gB9jj3oohjPNUnOT2yD4Oyp0b6kaPsjqtm7o23UdVgRdCtYVMYVxHMDqkypMkiDmr3QaIIvu+3BKCdETfrOgJpcLZCEHLbrZgRrHplykuzdDpkGc1XktpzOhauKnXMd0y1OrWYHqM0rTDUdVf5/x/qq4/XULq8wByFgoUMeT3zioinPHc9kPclsYfUiWIMCYFye2ii1rBYkneTOjAfoyY3a84izdG5I2yHJsR9VXFKtfv+dLGNigdEFhJV9nIsZBuX5MUy0zAMbbkOCsqWB76yyEcreH7A9f+JYGKpMgW5HybEoWMVA+hCw42bW3t5hMBKopamNthFELW9mR+UDuBH0QX87B2UA6K+yQEqU6xWvAMV1x0EUjFEKSTM+pEUQE7dvCsxWyYKgm9FvcJhIEdCDM9f0AFgR9GUP49VIPdtdXgyerau6cL6y8Gv44Z755331v2Au+Zt+699ra9HQ973PviffW+Nb43fjR+Nn6do5emBm3uemNX4/c/BIqeAA==</latexit>

minimize↵

d(↵) =1

2↵TY KY ↵� h↵,1i

<latexit sha1_base64="ux1IprgwHj25FirK0W/QjzT528E=">AAAHbnicfVXdjtQ2FA5QGJjSsoDETYWwuiAt7bKaLEhwgwQsqlpVq9KKXXa13h05Hmdixj/BdmCmJs/W5+gD9LbtC1TqcSaDZpIMvkiO/X0+Pv6OfZzkgls3GPx57vyFLy5e6l2+0v/y6ldfX9u4fuPQ6sJQdkC10OYoIZYJrtiB406wo9wwIhPB3iSTvYC/ec+M5Vq9drOcnUoyVjzllDgYGm4cY8kVl/x3NvSYiDwjJUJotDW376M+9J4ihDMiUjQfPHuNjn8+rjsAP0BYkLq7jbBWDGFD+sONzcHOoGqobcS1sRnV7dXw+sX/8EjTQjLlqCDWnsSD3J16YhyngpV9XFiWEzohY+aJtHYmkxLdk8RltomFwU5sWklW9vv3lkZPCpc+OfVc5YVjisJEwNJCIKdREA2NuGHUiRkYhBoO8SCaEUOoA2lXXPlE64kjiYUlsGIfqJaSqJHHgkvIJ7Olx4aPM0eM0R+A1KKlwHBs6vxPKSrLBuoyphY4Cp1AWeWETAElheh8XPrdpg9DQghEjStJV7wHRHQh1gGSJXrqEd7G27ZI3oIeIE/oVSGs8rl6X/oz/yBuLu54iJ4UXdgkD3Mm37cBWQHtGTqHuM4gOmKam2REBKnDAUgS/1tzJhj5Er7XwjPOVZARjKHfG8KOUjdrsqR2C6WxLDq01tq55VyErXdxJk3ScNJBsyssy8eStPUNyn9ida1nG5RuP0D5iBVJ4GZPP7ZASaaln2E60g4tWA0SEwIqQenvBmsY323Aexmv5adE+KOWsvT5Evy8CaduCf2hJVVYd57a1MeNCwKZDLWSvVM6nGuWU10ouMaevSuqmlhitIXhai3698P8lwwqkmH74POXnBnitPkOlpAg0/7+YbmWUNfW0i8sDF5H6/lkuuDX1uf5kDsoB9V3HYWYcZUt+OPtYH2OyNWCyNd7tJnhCk5s/e+keZwbDatW3zUMBycQdjr/QQWBByNuPg9t43B3J364s/vro81nL+qn43L0TfRttBXF0ePoWfRj9Co6iGj0R/RX9Hf0z6V/e7d6t3t35tTz5+o5N6OV1tv6H4w/te8=</latexit>

Recall: w = XTY ↵<latexit sha1_base64="EZE/b/C+np8MjqyQFo7Z6BqmcdQ=">AAAHJHicfZVNb9MwGMezwcooL9vgyCVimzRgTM1AggvSRhGCw8RAe0PLVjmu05g6dmY7W4uXT8IV+DTcEAcufBMkHmctapN0OSRP8v/58eO/XxIkjCrdaPyemr5ydaZ2bfZ6/cbNW7fn5hfu7CmRSkx2sWBCHgRIEUY52dVUM3KQSILigJH9oNu0+v4pkYoKvqP7CTmKUYfTkGKk4VNrfu7MfeEeHO98dH3Ekgi15hcba438csuBNwgWncG13VqY+eu3BU5jwjVmSKlDr5HoI4OkppiRrO6niiQId1GHGBQr1Y+DzF2OkY5UUbMfK7VePtCsXl8e+XqY6vD5kaE8STXhGBqCFqbM1cK1Q3XbVBKsWR8ChCWFelwcIYmwBkPGUplAiK5GgYIufE7OsIhjxNvGZzSGWSAqM76knUgjKcUZQCUsBEKTnjZvQzfLCqqOCB/qrn2xyDgTIWZThFCd8TKzXswhkS0B8U5u6Vh2q7AqRWlQokD0jOuv+qsqDT6BH2CPfctLGOcpP83MsXnsFTvX1FaP0iqtm9g23UdlIc6FcguRQF3HUB2SxUESxKzVdgEEgflQbAlBMqI3S3pEKbc2QtAyzRaMKNT9IhULPXTaj9MKr4XQenQu7NCrmG4RanUrMDVGKdqJUdlf6/x/qqo/VUCq8wBy7nMUMOT2zktijHqZ6fu4LbQ7pAoQYUxwkpklG7W8pYLcjOjAfoyYOSg5izdH5M2iHOoR9XXJKtvvxdSGxitsEJhJe8KREy7suiYJFimHbWzISZqfZJnvrviwtYbvD2z7VwROJEm2IOe7hEikhXwIXcRg09bWXjYRoJzG9DOUM4x8yNqezKPekB9El/Mwd3Ac5PdJCJKdfLbg6a/a6DKQ8iFIJ2dUkaQcVuzgWYkZP5ECes3vEwgNKxBGevGAEwR+GF7x91AO9tbXvCdr6++fLm68HPw6Zp17zn1nxfGcZ86G88bZdnYd7KTOF+er8632vfaj9rP26wKdnhq0ueuMXbU//wCBP5+G</latexit>

wTx = hXTY ↵, xi =X

yi↵ihxi, xi =X

yi↵iK(xi, x)

<latexit sha1_base64="H3vPalkC9jBVtCTxSgsFU9KjGzg=">AAAHg3icfZVdb9s2FIaVtpm77CvdLndDNC2QrllgBRm2mwLtUhQbhmDdkKQZykSgaMrizA+VpBp5rP5lb/ZXBgzYoSwPtiRXF9IR34eHhy8pKi0Et248/nvr1u072x+N7n6888mnn33+xe69Ly+sLg1l51QLbS5TYpngip077gS7LAwjMhXsVTo7Cfqrt8xYrtWZmxfsSpKp4hmnxEFTsqturs8qhJ4gLIiaCoYur8/+wEQUOTlAFTaLRpBtKdE84WihhaDtUCV8jRxAf9lvqUfJ7t74cNxcqB/EbbAXtdfL5N72v3iiaSmZclQQa1/H48JdeWIcp4LVO7i0rCB0RqbME2ntXKY1eiiJy21XC42DWtX4WO/sPFxpfV267Icrz1VROqYodAQtKwVyGgUn0YQbRp2YQ0Co4VAPojkxhDrwey2VT7WeOZJaGAIrdkO1lERNPBZcwiIzW3ts+DR3xBh9A1APy4BwrHL+5wzVdUd1OVNLHYWXgKwzOREhRQbV+bj2R90choQSmiXsZg+KGFKsAyVPdeURPsAHtkz/BD/AnvDWlLDOc/W29tf+27g7uOOhelIOabMi9Jk97guyEfo9dAF1XUN1xHQnyYgIVocNkKb+925PCIoV/aSn55yrYCMEiT9JYEaZm3cpqd3SaSzLAa+1dm51LcLUh5hZF0pmA5hdoyyfStL3Nzj/PzU0nu0gw3kAeYcVSQVB1bueKElV+zmmE+3QkupATAitWO0fhCiJH3Tkk5y39lMi/GXPWfpsRX7WlTO3or7oWRXGXSxt5uPOBwIrGQ5Q9kbpsK9ZQXWp4DP27E3ZHJQ1RvsYPq3l+6PQ/zmDE8mwU8j5a8EMcdp8A0NIsOn09KLeCHDFJf8LyllGGLJONvOkWvJt9GEe1g6Og+a+CSFm2qwWPPFBiD4EcrUE+eaMNjdcwY5tn4OYx4XRMGpz30A42IEw08UDThD4YcTd30M/uDg6jI8Pv/vteO/pj+2v4270dXQ/2o/i6PvoafRT9DI6j2j0Pvpn6/bWndH26PHoaHS8QG9ttX2+itau0ZP/AGOcvJM=</latexit>

For a new data point, x

POLYNOMIAL KERNELK(x, y) = (xT y + 1)2

<latexit sha1_base64="BOtYHJGs5lkPKgUiZNwvTmzhYWY=">AAAHCnicfZXdbtMwFMezwcooXxu75CZim7TBmJppCG6QNooQCE0MtC9pWSvHdRtTx85sZ2vx8ga8A7dwyx3ilpfgilfhOGtRm6TLRXKc/8/H5xx/BTGjStdqf6amr12fqdyYvVm9dfvO3Xtz8/cPlEgkJvtYMCGPAqQIo5zsa6oZOYolQVHAyGHQrVv98IxIRQXf0/2YnESow2mbYqThV3Nu4d1Kb62/6r5wV3qNvf5jb7Wx0ZxbrK3XssctGt7AWHQGz25zfuav3xI4iQjXmCGljr1arE8MkppiRtKqnygSI9xFHWJQpFQ/ClJ3OUI6VHnN/izVelmyabW6PPL3ONHt5yeG8jjRhGPoCFo7Ya4Wrk3XbVFJsGZ9MBCWFOJxcYgkwhqKMubKBEJ0NQoUDOFzco5FFCHeMj6jEcwEUanxJe2EGkkpzgEqYG0gNOlp87btpmlO1SHhQ921DYuMMyFi1kUbojNeajbyPiSyISDeyUo65t0qrExRGpQwED3j+mv+mkqCT1APKI9tZSGM85SfpaZhnnj5wTW10aOkTOvGtk/3cVGIMqHYQ8QQVwOiQzKfJEHMltougCAwH/M9wYhH9HpBDynltoxgNE29CRm1dT9PRUIPK+1HSUmthdB6dC5s6mVMNw81uyWYGqMU7USoWF9b+f9U2Xgqh5T7AeTC5yhgyO1dFMQI9VLT93FLaHdI5SDCmOAkNUvWanpLObke0kH5MWLmqFBZvD0ib+flth5RX+e2AMyVPcfIKRd25ZIYi4TDRjXkNMnOq9R3V3zYPMP2qu3/isCZI8kOOH0fE4m0kI9gjAgKsbNzkE4EKKcR/QyJDi0fvLYm86g35AfW1TzMDmz47D0JQbKTzQd8/TVrXQVSPgTpZI8qlJTDmhx8SzHjx1LAqNl7AqFhjUGmlx84I+BK8PIXQNE42Fj3Nteffthc3Ho5uBxmnQfOQ2fF8Zxnzpbzxtl19h3s9J2vzjfne+VL5UflZ+XXJTo9Neiz4Iw9ld//AK1Xkyc=</latexit>

K(x, y) =⇣1 +

Xxiyi

⌘2= 1 +

X

i

xiyi +X

i

x2i y

2i + 2

X

i 6=j

xixjyiyj

<latexit sha1_base64="xSWR7iA+ewjCm57Iu4z0Wav9TP4=">AAAHYHicfVXLbtw2FFWe40wfsdNduiHqBLBrxxgNEqSbAEkdFC0Ko2lROwYiW6A41IgePmSSSqQw+sasuyrQf+i27aU8U8xImnBBXvKce3l5+Epyzowdjf64dv3GzVu3Bxt3hp99/sWXdze37p0YVWhCj4niSp8m2FDOJD22zHJ6mmuKRcLp62R26PHXb6k2TMnfbZXTM4GnkqWMYAtD8Sb7eafcr3bRs2HEaWp3wr3IFAKVMatiFmk2zewuOh+jZ+gKidkcW+qdjytfoT00bgYdiyS9RBe1B8v4ovIOF/Hm9uhg1BTUNcK5sR3My6t469af0USRQlBpCcfGvAlHuT1zWFtGOK2HUWFojskMT6nDwphKJDV6KLDNTBvzg71Y2UhYD4cPl0bfFDb97swxmReWSgKOgKUFR1YhLyKaME2J5RUYmGgG+SCSYY2JBalXQrlEqZnFiYEpQJV3RAmB5cRFnAnYX2pqd6Uy1lq9A1KHlgLD0tK6n1JU1y3UZlQucOQ7nrLKyTD3IVLIzoW1G7djaOxTwHLaSLoS3SO8DzEWkCxRpUPRfrRviuQC9AB5fK9JYZXP5NvanbtHYXtyy3z2uOjDZrn3me11AdEAXQ+VQ17nkB3W7UVSzL3U/gAkifut7QlGvoQfdvCMMellBCN2hzGsKLVVmyWUXSgdiaJHa6WsXd4Lv/Q+zqxNimc9NLPCMmwqcFdfr/z/rL75TIvSHwcoHyKJE45R+aEDClzWrorIRFm0YLVIlHMlae0eeCsOH7Tgw4zN5SeYu9OOsuTFEvyiDad2Cf2hdQVgr/zrSC+l8ieX5kQVEi6qo5dF8wrWEdqJ4PIs+rve/yWFN0fTIwj6S041tkp/C3MIEOLo6KReS2CSCfYeFrqwIog6Wc/H5YI/tz7Nh92BC9/U6yhYT5v9gDba99aniEwuiGx9RJNpJuFMzttemotyrWDWpl7DsHDGYKVXDbwR8CWE7Q+ga5yMD8LHB09+fbz9/Pv557ARfB18E+wEYfA0eB78GLwKjgMSfAz+Dv4J/r3912BjcHewdUW9fm3u81WwUgb3/wOfvbF1</latexit>

= h�(x),�(y)i

<latexit sha1_base64="YjBuogKnVGn8qU+X5CHsHXv1Cug=">AAAHEnicfZXPb9MwFMczYGWUXxscd4nYJm0wpnYaggvSRhGCw8RA7Ic0b5Xjuo2pY2e2s7V4OfA/8D9whSs3xJV/gBP/Cs9Zi9okXQ7Jc74fP7/3/CuIOdOmVvszdeXqtenK9Zkb1Zu3bt+5Ozt3b0/LRBG6SySX6iDAmnIm6K5hhtODWFEcBZzuB92G0/dPqdJMig+mH9OjCHcEazOCDfxqzs77z6uIYx/FIVvuraxeGP0VHyncnF2ordWyxy8a9YGx4A2enebc9F/UkiSJqDCEY60P67XYHFmsDCOcplWUaBpj0sUdanGkdT8KUn8pwibUec39LNV6WdJptbo08vcwMe1nR5aJODFUEOgIWjvhvpG+S9tvMUWJ4X0wMFEM4vFJiBUmBooz5soGUnYNDjQMgQQ9IzKKsGhZxFkEM0J1apFindBgpeQZQAWsDYShPWPftP00zakmpGKo+67hkHEmxNy5aEN0tp7a9bwPhV0IWHSyko55dwovU7QBJQxkz/poFa3qJPgI9YDyuFYWwjjPxGlqj+3jen5ww1z0OCnTurHr031UFKJMKPaQMcR1DNFhlU+SYu5K7RZAENj3+Z5gxCN6o6CHjAlXRjCattGEjNqmn6ciaYaVRlFSUmspjRmdC5d6GdPNQ81uCabHKM06ES7W11X+P1U2ns4h5X4AOUcCB7C5e+cFMcK91PYRaUnjD6kcRDmXgqZ20VnN+mJOboRsUH6CuT0oVJZsjchbebltRtRXuS0Ac+XOM3oipFu5NCYyEbBRLT1JsnMrRf4ygs0zbK+4/i8pnDmKboPTtzFV2Ej1EMaIoBDb23vpRIAJFrFPkOjQQuC1NZnHvSE/sC7nYXZgw2fvSQhWnWw+4ItWnXUZyMQQZJM96lAxAWty8C3FLIqVhFGz9wTCwBqDTC8+cEbAlVDPXwBFY299rb6x9uTdxsLmi8HlMOPNew+8Za/uPfU2vdfejrfrEe+z99X75n2vfKn8qPys/LpAr0wN+tz3xp7K73/5ZZZ5</latexit>

ConstantFirst order Second order

�(x) = (1, x0, · · · , xn, x20, · · · , x2

n, 2x0x1, 2x0x2, · · · , 2xn�1xn)

<latexit sha1_base64="Il8G2Jg07eQyKyqV5Mst0T2rVDA=">AAAHS3icfZXLbtQwFIbDpVMotwJLNhGlUgtDNRkVwQapUIRgUVFQb1LTRo7HMzHj2KntQAaTF+MdeABWsIUtO8SC43SCZpJMs0iO/X8+tn9fEiaMKt3pfDt3/sLFudb8pcsLV65eu35j8eatPSVSickuFkzIgxApwignu5pqRg4SSVAcMrIfDjetvv+BSEUF39GjhBzFaMBpn2KkoSpY3PGTiK5kq09XvHYWdNo+7gmtIOS2eNwtK1yogVIXKrPAa7unQbftljpUGP7Qy4FbDRaXOmud4nHrgTcOlpzxsx3cnPvu9wROY8I1ZkipQ6+T6CODpKaYkXzBTxVJEB6iATEoVmoUh7m7HCMdqapmKxu1rHArX1hYnqg9THX/yZGhPEk14RgagtZPmauFa/1ye1QSrNkIAoQlhfG4OEISYQ2uTqUyoRBDjUIFXficfMQijhHvGZ/RGJaSqNz4kg4ijaQUHwGqYX0gNMm0ed1387yi6ojwUndtwSLTTISYTdGH0RkvN91qDonsEBAfFJZOZbcKa1KUBiUKRWZcv+23VRq+Bz/AHlsqhjDNU/4hN8cGdkIlkaZ29Cht0oaJbTN8UBfiQqi3EAmM6xhGh2R1kgQxa7XdAGFo3lVbQpBM6Js1PaKUWxshCMxmADPq61GVioUunfbjtMFrIbSeXAs79SZmWIWCYQOmpihFBzGq+2ud/0819acqSHMeQD77HIUMudnnmhijLDej4tS7JVWBCGOCk9zcs1Hg3avImxEd248RMwc1Z/GzCflZVe7rCfVl5QjAWtmLkJxwYXcuSbBIORxUQ07S4sLLfXfFh8NTlldt+xcE7hxJtiDpm4RIpIW8D33EYMTW1l4+E6CcxvQTTLSMfMjam82jrOTH0dk8rA4c+OI9C0FyUKwHfP22jc4CKS9BOjujiiTlsCfH30bM+IkU0GvxnkFo2GMw09MP3BHwS/CqP4B6sNdd89bXHr1dX9p4Pv45XHLuOHedFcdzHjsbzitn29l1sPPF+eH8dH61vrZ+t/60/p6i58+N29x2pp75uX81ZKqx</latexit>

K(x, y) = (xT y + 1)p

<latexit sha1_base64="MKyoGV7nSLBvFfFi+Zh9rTTGiLY=">AAAHCnicfZXdbtMwFMezwcooXxu75CZim7TBmJppCG6QNooQCE0MtC9pWSvHdRtTx85sZ2vx8ga8A7dwyx3ilpfgilfhOGtRm6TLRXKc/8/H5xx/BTGjStdqf6amr12fqdyYvVm9dfvO3Xtz8/cPlEgkJvtYMCGPAqQIo5zsa6oZOYolQVHAyGHQrVv98IxIRQXf0/2YnESow2mbYqThV3Nu4d1Kb62/6r5wV3qNvf5jb7URN+cWa+u17HGLhjcwFp3Bs9ucn/nrtwROIsI1ZkipY68W6xODpKaYkbTqJ4rECHdRhxgUKdWPgtRdjpAOVV6zP0u1XpZsWq0uj/w9TnT7+YmhPE404Rg6gtZOmKuFa9N1W1QSrFkfDIQlhXhcHCKJsIaijLkygRBdjQIFQ/icnGMRRYi3jM9oBDNBVGp8STuhRlKKc4AKWBsITXravG27aZpTdUj4UHdtwyLjTIiYddGG6IyXmo28D4lsCIh3spKOebcKK1OUBiUMRM+4/pq/ppLgE9QDymNbWQjjPOVnqWmYJ15+cE1t9Cgp07qx7dN9XBSiTCj2EDHE1YDokMwnSRCzpbYLIAjMx3xPMOIRvV7QQ0q5LSMYTVNvQkZt3c9TkdDDSvtRUlJrIbQenQubehnTzUPNbgmmxihFOxEq1tdW/j9VNp7KIeV+ALnwOQoYcnsXBTFCvdT0fdwS2h1SOYgwJjhJzZK1mt5STq6HdFB+jJg5KlQWb4/I23m5rUfU17ktAHNlzzFyyoVduSTGIuGwUQ05TbLzKvXdFR82z7C9avu/InDmSLIDTt/HRCIt5CMYI4JC7OwcpBMBymlEP0OiQ8sHr63JPOoN+YF1NQ+zAxs+e09CkOxk8wFff81aV4GUD0E62aMKJeWwJgffUsz4sRQwavaeQGhYY5Dp5QfOCLgSvPwFUDQONta9zfWnHzYXt14OLodZ54Hz0FlxPOeZs+W8cXadfQc7feer8835XvlS+VH5Wfl1iU5PDfosOGNP5fc/RNGTZQ==</latexit>

In general

TEXT KERNELS

Based on…

Edit distance between words/paragraphs

Bag of words models

Genome mutations

CONJUGATE FUNCTIONf⇤(y) = max

xyTx� f(x)

Is it convex?

CONJUGATE OF NORM

f⇤(y) = maxx

yTx� kxk

f(x) = kxk

f⇤(y) =

(0, kyk⇤ 1

1, otherwise

dual normkyk⇤ , max

xyTx/kxk yTx kyk⇤kxk

Holder inequality

kyk⇤ 1 �! f⇤ = 0

kyk⇤ > 1 �! f⇤ = 1

why?

why?

EXAMPLES

f(x) = |x|, f⇤(y) = X1(y) =

(0, kxk1 1

1, kxk1 > 1

f(x) = kxk2, f⇤(y) = X2(y) =

(0, kxk2 1

1, kxk2 > 1

f(x) = kxk1, f⇤(y) = X1(y) =

(0, |x| 1

1, |x| > 1

HOW TO USE CONJUGATEminimize g(x) + f(Ax+ b)

minimize g(x) + f(y)

subject to y = Ax+ b

d(�) = minx,y

g(x) + f(y) + h�, Ax+ b� yi

= minx,y

g(x) + h�, Axi+ f(y)� h�, yi+ h�, bi

= h�, bi �maxx,y

�g(x)� hAT�, xi � f(y) + h�, yi

d(�) = h�, bi � g⇤(�AT�)� f⇤(�)

write this as…

EXAMPLE: LASSOminimize µ|x|+ 1

2kAx� bk2

how big?minimize g(x) + f(Ax+ b)

d(�) = h�, bi � g⇤(�AT�)� f⇤(�)

maximize �h�, bi � X1

✓�1

µAT�

◆� 1

2k�k2

change variables to eliminate negative sign

(µJ)⇤(y) = µJ⇤(y/µ)note:

maximize h�, bi � X1

✓1

µAT�

◆� 1

2k�k2

EXAMPLE: LASSOminimize µ|x|+ 1

2kAx� bk2

completesquare

maximize h�, bi � X1

✓1

µAT�

◆� 1

2k�k2

maximize � 1

2k�� bk22 +

1

2kbk2

subject to kAT�k1 µ

Zero is a solution when the optimal objective is

This coincides with dual variable

µ � kAT bk

µ|0|+ 1

2kA0� bk2 =

1

2kbk2

EXAMPLE: LASSOminimize µ|x|+ 1

2kAx� bk2

dual problem

� = b

maximize � k�� bk22 +1

2kbk2

subject to kAT�k1 µ

EXAMPLE: LASSOminimize µ|x|+ 1

2kAx� bk2

µ � kAT bkSolution is zero when

µguess =1

10kAT bk

EXAMPLE: SVM

f(z) = h(z) = CX

max{1� zi, 0}

x =

✓wb

◆

minimize g(x) + f(Ax)

g(x) = g(w, b) =1

2kwk2

minimizew,b

1

2kwk2 + h(Y Xw � yb)

<latexit sha1_base64="tJUsU411U6BtYc5U30QBGaThszs=">AAAHlXicfVXLctMwFDXPlPIqsGDBxkOBKbR04sIMbJgByjCw6PAY+mCqNiMrciyihyvJJEH1H/A1bOFH+Buu3AQS28EL+0rn3Hulc6XrOOPM2Hb796nTZ86eO99auLB48dLlK1eXrl3fMSrXhG4TxZXei7GhnEm6bZnldC/TFIuY0924v+nx3a9UG6bkJzvK6IHAPckSRrCFqc7SPSSYZIJ9ox03WIuLEKWYJ+h4gI4PN1bTlc97g4ej+H5nabm93i6fsG5EY2M5GD/vO9fO76KuIrmg0hKOjdmP2pk9cFhbRjgtFlFuaIZJH/eow8KYkYDkdwW2qalifrIRG5b7LxYX707N7uc2eXrgmMxySyUBR8CSnIdWhV6BsMs0JZaPwMBEM1hPSFKsMbGg00woFyvVtzg2kAJJOiBKCCy7DhFjiv3owKGY9ph0BApgiuUIUQBPBsWsg7Ip1QNmaOGQpUPrwn8z1eCcCag8xHBIs15qsdZqUFQ5ySTS2ySsZuMQW/7N5AdhLYsvM1AS2LeLCrdRjaGxz49lryzWTHSP8CbEWEDSWA1diNbQmsnjL6A0CO9H5RJm+Ux+LdyhexhVk1vmV4/zJqyfeZ/+ah0QJVD3UBms6xBWh3V1kxRzr7M/WnHsPlY9wcim8M0anjImvYxgdNxmB3aU2FGVJZSdKI1E3qC1UtZO18JvvYnTr5I6/QaamWEZ1hO4rq9X/i+rKZ+pUJrjAOUYSRxzHA6Pa6DAw8KNEOkqG05YFRLlXEm4FXe81YnuVODNlI3lJ5i7vZqy5MUU/KIKJ3YKfV2Tyuc9KW3iosoFgUr6lkqPpPLnmmZE5RIahKNHedk6CxSuILhak/F97/+KQq/TdAtivsuoxlbpB5BCgExbWzvFXMK4BRduYiGI2p3Px8MJf2z9nw+1g3ZQvudRsO6V1YIvWvPW/4hMTohsfkSTaibhxI6/jTSHMq0ga/mew7BwAmGnJx/oIPAriqo/nrqxs7EePVrf+PB4+fnL8U9pIbgV3A5Wgih4EjwP3gTvg+2ABN+DH8HP4FfrZutZ61Xr9Qn19Kmxz41g5mm9+wNNvMrI</latexit>

A = [Y X|� y]<latexit sha1_base64="YjRii8F6U9JQsd4bV5Bs8Zkmxao=">AAAHIHicfZVNb9MwGMczYGWUtw2OXCK2SQO2qRlIcEHaKEJwmBiIvaClqxzXaUwdO7OdrcXL5+AKfBpuiCN8FyQeZy1qk3Q5JE/y//nx479fEiSMKt1o/J65dPnKbO3q3LX69Rs3b92eX7izp0QqMdnFggl5ECBFGOVkV1PNyEEiCYoDRvaDXtPq+ydEKir4Bz1ISCtGXU5DipGGT60t97l7+PHAPXPXBq32/GJjvZFfbjnwhsGiM7x22guzf/2OwGlMuMYMKXXoNRLdMkhqihnJ6n6qSIJwD3WJQbFSgzjI3OUY6UgVNfuxUuvng8zq9eWxr4epDp+1DOVJqgnH0BC0MGWuFq4dptuhkmDNBhAgLCnU4+IISYQ1mDGRygRC9DQKFHThc3KKRRwj3jE+ozHMAFGZ8SXtRhpJKU4BKmEhEJr0tXkTullWUHVE+Eh37YtFJpkIMZsihOqMl5mNYg6JbAmId3NLJ7JbhVUpSoMSBaJvXH/VX1Vp8An8AHvsW17CJE/5SWaOzJpX7FxTWz1Kq7ReYtv0HpWFOBfKLUQCdR1BdUgWB0kQs1bbBRAE5n2xJQTJmN4s6RGl3NoIQds02zCiUA+KVCz0yGk/Tiu8FkLr8bmwQ69iekWo3avA1ASlaDdGZX+t8/+pqv5UAanOA8iZz1HAkNs/K4kx6mdm4OOO0O6IKkCEMcFJZpZs1PaWCnIzokP7MWLmoOQs3hqTt4pyqMfUVyWrbL/nUxsar7BBYCbt6UaOubDrmiRYpBy2sSHHaX6KZb674sPWGr0/sO1fEjiRJNmGnG8TIpEW8iF0EYNN29t72VSAchrTz1DOKPIha2c6j/ojfhhdzMPcwXGQ36chSHbz2YKnv2qji0DKRyCdnlFFknJYscNnJWb8RAroNb9PITSsQBjp+QNOEPhheMXfQznY21j3Hq9vvHuyuPli+OuYc+45950Vx3OeOpvOa2fH2XWwc+x8cb4632rfaz9qP2u/ztFLM8M2d52Jq/bnH0GPnfE=</latexit>

HOW TO USE CONJUGATEf(z) = h(z) = C

Xmax{1� zi, 0}

f⇤(�) = maxz

h�, zi � h(z)

= maxz

h�, zi � Cmax{0, 1� z}

= maxz

min{h�, zi, h�, zi � C(1� z)}

= maxz

min{h�, zi, h�+ C, zi � C}

=

(1T�, �i 2 [�C, 0]

1, otherwise

max occurs where two linear functions

intersect

HOW TO USE CONJUGATEg(x) = g(w, b) =

1

2kwk2

g⇤(y) = maxx

hy, xi � 1

2kwk2

= maxw,b

hyw, wi+ hyb, bi �1

2kwk2

=

(12kywk

2, yb = 0

1, otherwise

SVM DUAL

d(�) = �g⇤(�AT�)� f⇤(�)

x =

✓wb

◆

f⇤(�) =

(P�i,� 2 [�C, 0]

1, otherwise

↵ ��Same as before with

minimize g(x) + f(Ax)

g⇤(y) =

(12kywk

2, yb = 0

1, otherwise

A = [Y X|� y]<latexit sha1_base64="YjRii8F6U9JQsd4bV5Bs8Zkmxao=">AAAHIHicfZVNb9MwGMczYGWUtw2OXCK2SQO2qRlIcEHaKEJwmBiIvaClqxzXaUwdO7OdrcXL5+AKfBpuiCN8FyQeZy1qk3Q5JE/y//nx479fEiSMKt1o/J65dPnKbO3q3LX69Rs3b92eX7izp0QqMdnFggl5ECBFGOVkV1PNyEEiCYoDRvaDXtPq+ydEKir4Bz1ISCtGXU5DipGGT60t97l7+PHAPXPXBq32/GJjvZFfbjnwhsGiM7x22guzf/2OwGlMuMYMKXXoNRLdMkhqihnJ6n6qSIJwD3WJQbFSgzjI3OUY6UgVNfuxUuvng8zq9eWxr4epDp+1DOVJqgnH0BC0MGWuFq4dptuhkmDNBhAgLCnU4+IISYQ1mDGRygRC9DQKFHThc3KKRRwj3jE+ozHMAFGZ8SXtRhpJKU4BKmEhEJr0tXkTullWUHVE+Eh37YtFJpkIMZsihOqMl5mNYg6JbAmId3NLJ7JbhVUpSoMSBaJvXH/VX1Vp8An8AHvsW17CJE/5SWaOzJpX7FxTWz1Kq7ReYtv0HpWFOBfKLUQCdR1BdUgWB0kQs1bbBRAE5n2xJQTJmN4s6RGl3NoIQds02zCiUA+KVCz0yGk/Tiu8FkLr8bmwQ69iekWo3avA1ASlaDdGZX+t8/+pqv5UAanOA8iZz1HAkNs/K4kx6mdm4OOO0O6IKkCEMcFJZpZs1PaWCnIzokP7MWLmoOQs3hqTt4pyqMfUVyWrbL/nUxsar7BBYCbt6UaOubDrmiRYpBy2sSHHaX6KZb674sPWGr0/sO1fEjiRJNmGnG8TIpEW8iF0EYNN29t72VSAchrTz1DOKPIha2c6j/ojfhhdzMPcwXGQ36chSHbz2YKnv2qji0DKRyCdnlFFknJYscNnJWb8RAroNb9PITSsQBjp+QNOEPhheMXfQznY21j3Hq9vvHuyuPli+OuYc+45950Vx3OeOpvOa2fH2XWwc+x8cb4632rfaz9qP2u/ztFLM8M2d52Jq/bnH0GPnfE=</latexit>

maximize = �1

2kXTY �k2 � 1T�

subject to lT� = 0

� C � 0<latexit sha1_base64="vPTjs4n0jlmSRmryXqdxwzqm9o8=">AAAHfXicfZVbb9MwFMfDtVBuAx55sRhDXLapGUjwggQUIXiYuGgbQ3itHNdpTB072A60ePmKvPM9eOVynLXQJil5SE78//nY/ts5iTLBje10vh87fuLkqdOtM2fb585fuHhp5fKVPaNyTdkuVULp/YgYJrhku5ZbwfYzzUgaCfYuGnW9/u4z04YruWMnGTtIyVDymFNioam/kuCUjHnKvzJ08xFCGzghIkb4cL+38x5hAYkGBB/2tjYQVpL1dmZtCOM2NhbdRKK3M2t7hDrQDG0bXSzYPxTiTn9ltbPZKS9UD8JpsBpMr9f9y6d+4oGiecqkpYIY8yHsZPbAEW05Faxo49ywjNARGTJHUmMmaVSgtZTYxFQ139iojUsHi3Z7ba71Q27jhweOyyy3TFLoCFqcC2QV8h6iAdeMWjGBgFDNYT6IJkQTasHphVQuUmpkSWRgCCzZF6rSlMiBw4KnsL3MFA5rPkws0Vp9AaiGxUBYNrbuZYyKoqLahMmZjvyLRxYZv5+AxDA7FxZuq5pDEz8FIoelpQvZvSKaFGNBSSI1dgiv43WTRx/BD7DHv5VTWOS5/Fy4ntsIq4Nb7mdP8iZtlPk+o7t1IS2Feg+Vwbx6MDuiq4tkRHir/QGIIve22hOCbE7v1vSEc+lthKDvun1YUWwnVSpVduY0TvMGr5Wydn4v/NKbmFEV6o8aMLNAGT5MSd1f7/xfqmk8U0Ga8wByiCWJBEHjw5oIJaRwE0wHyqIZVYGYEFA/CnfDR/3wRkXuJnxqPyXC7decpU/m5CdVObZz6vOaVX7co62NXVj5QGAnfelkn6Ty55plVOUSPmPHPuVliSwwuoXh05q93/b9nzGoSJptQ85XGdPEKn0HhkjBpu3tvWIpwGVZags3izBkHSznp6UZ+Gn0fx72DspBeV+GED0sdwueeN1H/wO5nIF8eUaTaC7hxE6fjZjDmVYwanlfQlg4gbDSowdUEPhhhNXfQz3Y29oM721uvbm/+vjp9NdxJrgWXA9uBWHwIHgcvAheB7sBDb4FP4Kfwa/Tv1trrfXW5hF6/Ni0z9Vg4Wo9+AN7sbs+</latexit>

EXAMPLE: TV

d(�) = �g⇤(�AT�)� f⇤(�)

minimize g(x) + f(Ax)

minimize1

2kx� fk2 + µ|rx|

form dual problem

f(z) = µ|z| �! f⇤(y) = X1(y/µ)

(µJ)⇤(y) = µJ⇤(y/µ)

maximize �1

2krT�� fk2 � X1(�/µ)

g(z) =1

2kz � fk2 �! g⇤(y) =

1

2ky + fk2 � 1

2kfk2

EXAMPLE: TV

maximize�

� 1

2krT�� fk2

subject to k�k1 µsmooth

simple

maximize �1

2krT�� fk2 � X1(�/µ)

NON-CONVEX PROBLEM: SPECTRAL CLUSTERING

Two moons Swiss roll

Non-linearly separable classes

SPECTRAL CLUSTERING

Two moons

Non-linearly separable classes

(Dis)similarity matrix

xi 2 {�1, 1}

Wij = � � e�kdi�djk2/�2

Similar = negative

Dissimilar = positive

LABELING PROBLEM

Big W different labelsSmall W same label

Is this convex? Why? How hard is this problem?

d(�) = minimize xTWx+ h�, x2 � 1iDual function

Is this convex? Why? How hard is this problem?

minimize xTWx =X

i,j

xixjWij

subject to x2i = 1

<latexit sha1_base64="3pb4ayAhwoSMmI2jS6JQW54uJ44=">AAAHM3icfZXbjtQ2GMfDaaBDgV162RuLhYrDsppsK7UXXQm6FaIXq9KKPUh4N3I8nol3fAi2s2Rq8jJ9h75Db+EOcVEJ9bbv0M9hBs0kGXKRfM7/58/236c0F9y6weDdufMXLl7qXb7yRf/ql9eu31hbv3lgdWEo26daaHOUEssEV2zfcSfYUW4Ykalgh+lkN+iHZ8xYrtVzN83ZsSRjxUecEge/krUfseSKS/4HQ9+g8uT5YYl2ELaFTDzfPK1QmfAyOUWHUDytMO5j6wKY8JPtnThZ2xhsDeoHtYN4FmxEs+dZsn7pHzzUtJBMOSqItS/iQe6OPTGOU8GqPi4sywmdkDHzRFo7lWmF7kjiMtvUws9Oraxdqfr9Owt/XxRu9MOx5yovHFMUKoI2KgRyGgVf0JAbRp2YQkCo4dAfRDNiCHXg3lIqn2o9cSS10ARW7BXVUhI19FhwCVPGbOWx4ePMEWP0K4Ba2AgIx0rnfxmhqmqoLmNqrqNQCMgykxERUoygdz6u/HYzhyGhC0SNa0uXsgdFdCnWgZKluvQIb+JNW6Sn4AfYE0p1F5Z5rs4qf+Ifxs3GHQ+9J0WXNslDncmDtiBroV1D59CvE+gdMc1BMiKC1WEBpKn/vVkTgnxB323pGecq2AhB4ncTGNHITZuU1G7uNJZFh9daO7c4F2HoXcykCSWTDswuUZaPJWn7G5z/RHW1ZxtIdx5AXmNFUkFQ+bolSlJWforpUDs0pxoQE0IrVvnbIUri2w15N+Mz+ykR/qjlLH28ID9uyiO3oD5pbAGYq3DgsZdKh5XLcqoLBRvVs5dFfbBVGN3FsHnm5Xuh/s8MzhzD9iDprzkzxGlzH9qQYMTe3kG1EpgdkJWfRxiyDlfzpJzzs+jzPMwObPj6vQohZlzPB3zxZog+B3I1B/nqjDYzXMGanH07MY9zo6HV+r2CcLDGYKQfP3BGwJUQNy+AdnCwvRV/u7X923cbj36aXQ5Xoq+jW9HdKI6+jx5FT6Nn0X5Eoz+jv6M30dveX733vQ+9fz+i58/N6nwVLT29//4HU+mkMw==</latexit>

LABELING PROBLEM

d(�) = minimize xTWx+ h�, x2 � 1iDual function

X�ix

2i = xT diag(�)x

d(�) = minimize xT (W + diag(�))x� h�, 1i

=

(�h�, 1i, W + diag(�) ⌫ 0

�1, otherwise

LABELING PROBLEMDual function

d(�) = minimize xT (W + diag(�))x� h�, 1i

=

(�h�, 1i, W + diag(�) ⌫ 0

�1, otherwise

Pick the dual vector to be smallest constant we can get away with

smallesteigenvector

�i = ��min(W ), 8id(x) = h�min, 1i

x = argmin xT (W � �minI)x+ h�min, 1i = emin

LABELING PROBLEM

Final step: integer roundingx?approx = round(emin)

Approximate solutionx = argmin xT (W � �minI)x+ h�min, 1i = emin

OVERALL STRATEGY

Dual functiond(�) = minimize xTWx+ h�, x2 � 1i

minimize xTWx

subject to x2i = 1

Approximate solution

convex relaxation

Final step: integer roundingx?approx = round(emin)

Whyapproximate?

x = argmin xT (W � �minI)x+ h�min, 1i = emin

note: we could also definite SIMilarity matrix, and use largest eigenvalue

NEWTON’S METHODminimize f(x)

smooth problem

quadratic Approximation

minimize f(x)

subject to Ax = b

What about a constrained problem?

minimize1

2(x� x

k)TH(x� xk) + hx� x

k, gi

CONSTRAINED NEWTON

minimize f(x)

subject to Ax = b

Constrained problem

KKT ConditionsH(x� x

k) + g +AT� = 0

Ax = b

minimize1

2(x� x

k)TH(x� xk) + hx� x

k, gi

subject to Ax = b

L(x,�) =1

2(x� x

k)TH(x� xk) + hx� x

k, gi+ h�, Ax� bi

CONSTRAINED NEWTONKKT Conditions

H(x� xk) + g +A

T� = 0

Ax = b

Newton direction d = x� xk

✓H A

T

A 0

◆✓d

�

◆=

✓�g

b�Axk

◆

Solution is approximate minimizerSolution satisfies constraints

NEWTON ALGORITHM

✓H A

T

A 0

◆✓d

�

◆=

✓�g

b�Axk

◆

•Compute Hessian and gradient

•Solve Newton system

f(xk + ⌧d) f(xk) +⌧

10gT d

•Armijo search

•Update iteratexk+1 = xk + ⌧d

When stepsize=1, constraints are satisfied exactly

NEWTON COMPLEXITY

Suppose the Hessian of f is Lipschitz continuous. Then after a finitely many constrained Newton steps the unit stepsize is an Armijo step, and

Theorem

(⌧ = 1)kxk+1 � x?k Ckxk � x?k2

Furthermore, the constraint are exactly satisfied.

minimize f(x)

subject to Ax = b