+ All Categories
Home > Documents > Horseshoe Priors for Bayesian Neural Networks...Horseshoe Priors for Bayesian Neural Networks Soumya...

Horseshoe Priors for Bayesian Neural Networks...Horseshoe Priors for Bayesian Neural Networks Soumya...

Date post: 07-Jul-2020
Category:
Upload: others
View: 7 times
Download: 0 times
Share this document with a friend
32
Horseshoe Priors for Bayesian Neural Networks Soumya Ghosh Jiayu Yao Finale Doshi-Velez IBM Research MIT-IBM Watson AI lab Harvard Harvard 1
Transcript
Page 1: Horseshoe Priors for Bayesian Neural Networks...Horseshoe Priors for Bayesian Neural Networks Soumya Ghosh Jiayu Yao Finale Doshi-Velez IBM Research MIT-IBM Watson AI lab Harvard Harvard

Horseshoe Priors for Bayesian Neural Networks

Soumya Ghosh Jiayu Yao Finale Doshi-Velez

IBM Research MIT-IBM Watson AI lab

Harvard Harvard

�1

Page 2: Horseshoe Priors for Bayesian Neural Networks...Horseshoe Priors for Bayesian Neural Networks Soumya Ghosh Jiayu Yao Finale Doshi-Velez IBM Research MIT-IBM Watson AI lab Harvard Harvard

Bayesian Neural Networks

Inputs x Layers, parameterized by weights W Outputs y

!2

Being Bayesian:

p(W | �) � p(W | y, x, �)<latexit sha1_base64="Hj78BVbvqvoEleCeYI6+Lo5PhBM=">AAACMnicbVDLSgMxFM3UVx1foy7dBItQoZQZEXRZdKO7CvYBnaHcyaRtaOZBklHL0G9y45cILnShiFs/wvSBaNsDgcM555J7j59wJpVtvxq5peWV1bX8urmxubW9Y+3u1WWcCkJrJOaxaPogKWcRrSmmOG0mgkLoc9rw+5cjv3FHhWRxdKsGCfVC6EaswwgoLbWt66TohqB6BHjWGGI3ZAF2uZ4P4Bi7gnV7CoSI780FuUEJP5R+022rYJftMfA8caakgKaotq1nN4hJGtJIEQ5Sthw7UV4GQjHC6dB0U0kTIH3o0pamEYRUetn45CE+0kqAO7HQL1J4rP6dyCCUchD6OjlaW856I3GR10pV59zLWJSkikZk8lEn5VjFeNQfDpigRPGBJkAE07ti0gMBROmWTV2CM3vyPKmflB277NycFioX0zry6AAdoiJy0BmqoCtURTVE0CN6Qe/ow3gy3oxP42sSzRnTmX30D8b3D14PqY8=</latexit><latexit sha1_base64="Hj78BVbvqvoEleCeYI6+Lo5PhBM=">AAACMnicbVDLSgMxFM3UVx1foy7dBItQoZQZEXRZdKO7CvYBnaHcyaRtaOZBklHL0G9y45cILnShiFs/wvSBaNsDgcM555J7j59wJpVtvxq5peWV1bX8urmxubW9Y+3u1WWcCkJrJOaxaPogKWcRrSmmOG0mgkLoc9rw+5cjv3FHhWRxdKsGCfVC6EaswwgoLbWt66TohqB6BHjWGGI3ZAF2uZ4P4Bi7gnV7CoSI780FuUEJP5R+022rYJftMfA8caakgKaotq1nN4hJGtJIEQ5Sthw7UV4GQjHC6dB0U0kTIH3o0pamEYRUetn45CE+0kqAO7HQL1J4rP6dyCCUchD6OjlaW856I3GR10pV59zLWJSkikZk8lEn5VjFeNQfDpigRPGBJkAE07ti0gMBROmWTV2CM3vyPKmflB277NycFioX0zry6AAdoiJy0BmqoCtURTVE0CN6Qe/ow3gy3oxP42sSzRnTmX30D8b3D14PqY8=</latexit><latexit sha1_base64="Hj78BVbvqvoEleCeYI6+Lo5PhBM=">AAACMnicbVDLSgMxFM3UVx1foy7dBItQoZQZEXRZdKO7CvYBnaHcyaRtaOZBklHL0G9y45cILnShiFs/wvSBaNsDgcM555J7j59wJpVtvxq5peWV1bX8urmxubW9Y+3u1WWcCkJrJOaxaPogKWcRrSmmOG0mgkLoc9rw+5cjv3FHhWRxdKsGCfVC6EaswwgoLbWt66TohqB6BHjWGGI3ZAF2uZ4P4Bi7gnV7CoSI780FuUEJP5R+022rYJftMfA8caakgKaotq1nN4hJGtJIEQ5Sthw7UV4GQjHC6dB0U0kTIH3o0pamEYRUetn45CE+0kqAO7HQL1J4rP6dyCCUchD6OjlaW856I3GR10pV59zLWJSkikZk8lEn5VjFeNQfDpigRPGBJkAE07ti0gMBROmWTV2CM3vyPKmflB277NycFioX0zry6AAdoiJy0BmqoCtURTVE0CN6Qe/ow3gy3oxP42sSzRnTmX30D8b3D14PqY8=</latexit><latexit sha1_base64="Hj78BVbvqvoEleCeYI6+Lo5PhBM=">AAACMnicbVDLSgMxFM3UVx1foy7dBItQoZQZEXRZdKO7CvYBnaHcyaRtaOZBklHL0G9y45cILnShiFs/wvSBaNsDgcM555J7j59wJpVtvxq5peWV1bX8urmxubW9Y+3u1WWcCkJrJOaxaPogKWcRrSmmOG0mgkLoc9rw+5cjv3FHhWRxdKsGCfVC6EaswwgoLbWt66TohqB6BHjWGGI3ZAF2uZ4P4Bi7gnV7CoSI780FuUEJP5R+022rYJftMfA8caakgKaotq1nN4hJGtJIEQ5Sthw7UV4GQjHC6dB0U0kTIH3o0pamEYRUetn45CE+0kqAO7HQL1J4rP6dyCCUchD6OjlaW856I3GR10pV59zLWJSkikZk8lEn5VjFeNQfDpigRPGBJkAE07ti0gMBROmWTV2CM3vyPKmflB277NycFioX0zry6AAdoiJy0BmqoCtURTVE0CN6Qe/ow3gy3oxP42sSzRnTmX30D8b3D14PqY8=</latexit>

p(y� | x�, y, x, �)<latexit sha1_base64="ic+y4dKT/LqWr1V58Xz1J6QtqrM=">AAACB3icbVDLSsNAFJ3UV62vqEtBBotQSymJCLosunFZwT6gCWEymbRDZ5IwM5GG0p0bf8WNC0Xc+gvu/BunbRbaeuDC4Zx7Z+49fsKoVJb1bRRWVtfWN4qbpa3tnd09c/+gLeNUYNLCMYtF10eSMBqRlqKKkW4iCOI+Ix1/eDP1Ow9ESBpH9ypLiMtRP6IhxUhpyTOPk0rmVaHDaQBHXrUGsxoc1aDD9BMBOvPMslW3ZoDLxM5JGeRoeuaXE8Q45SRSmCEpe7aVKHeMhKKYkUnJSSVJEB6iPulpGiFOpDue3TGBp1oJYBgLXZGCM/X3xBhxKTPu606O1EAuelPxP6+XqvDKHdMoSRWJ8PyjMGVQxXAaCgyoIFixTBOEBdW7QjxAAmGloyvpEOzFk5dJ+7xuW3X77qLcuM7jKIIjcAIqwAaXoAFuQRO0AAaP4Bm8gjfjyXgx3o2PeWvByGcOwR8Ynz9HJpcD</latexit><latexit sha1_base64="ic+y4dKT/LqWr1V58Xz1J6QtqrM=">AAACB3icbVDLSsNAFJ3UV62vqEtBBotQSymJCLosunFZwT6gCWEymbRDZ5IwM5GG0p0bf8WNC0Xc+gvu/BunbRbaeuDC4Zx7Z+49fsKoVJb1bRRWVtfWN4qbpa3tnd09c/+gLeNUYNLCMYtF10eSMBqRlqKKkW4iCOI+Ix1/eDP1Ow9ESBpH9ypLiMtRP6IhxUhpyTOPk0rmVaHDaQBHXrUGsxoc1aDD9BMBOvPMslW3ZoDLxM5JGeRoeuaXE8Q45SRSmCEpe7aVKHeMhKKYkUnJSSVJEB6iPulpGiFOpDue3TGBp1oJYBgLXZGCM/X3xBhxKTPu606O1EAuelPxP6+XqvDKHdMoSRWJ8PyjMGVQxXAaCgyoIFixTBOEBdW7QjxAAmGloyvpEOzFk5dJ+7xuW3X77qLcuM7jKIIjcAIqwAaXoAFuQRO0AAaP4Bm8gjfjyXgx3o2PeWvByGcOwR8Ynz9HJpcD</latexit><latexit sha1_base64="ic+y4dKT/LqWr1V58Xz1J6QtqrM=">AAACB3icbVDLSsNAFJ3UV62vqEtBBotQSymJCLosunFZwT6gCWEymbRDZ5IwM5GG0p0bf8WNC0Xc+gvu/BunbRbaeuDC4Zx7Z+49fsKoVJb1bRRWVtfWN4qbpa3tnd09c/+gLeNUYNLCMYtF10eSMBqRlqKKkW4iCOI+Ix1/eDP1Ow9ESBpH9ypLiMtRP6IhxUhpyTOPk0rmVaHDaQBHXrUGsxoc1aDD9BMBOvPMslW3ZoDLxM5JGeRoeuaXE8Q45SRSmCEpe7aVKHeMhKKYkUnJSSVJEB6iPulpGiFOpDue3TGBp1oJYBgLXZGCM/X3xBhxKTPu606O1EAuelPxP6+XqvDKHdMoSRWJ8PyjMGVQxXAaCgyoIFixTBOEBdW7QjxAAmGloyvpEOzFk5dJ+7xuW3X77qLcuM7jKIIjcAIqwAaXoAFuQRO0AAaP4Bm8gjfjyXgx3o2PeWvByGcOwR8Ynz9HJpcD</latexit><latexit sha1_base64="ic+y4dKT/LqWr1V58Xz1J6QtqrM=">AAACB3icbVDLSsNAFJ3UV62vqEtBBotQSymJCLosunFZwT6gCWEymbRDZ5IwM5GG0p0bf8WNC0Xc+gvu/BunbRbaeuDC4Zx7Z+49fsKoVJb1bRRWVtfWN4qbpa3tnd09c/+gLeNUYNLCMYtF10eSMBqRlqKKkW4iCOI+Ix1/eDP1Ow9ESBpH9ypLiMtRP6IhxUhpyTOPk0rmVaHDaQBHXrUGsxoc1aDD9BMBOvPMslW3ZoDLxM5JGeRoeuaXE8Q45SRSmCEpe7aVKHeMhKKYkUnJSSVJEB6iPulpGiFOpDue3TGBp1oJYBgLXZGCM/X3xBhxKTPu606O1EAuelPxP6+XqvDKHdMoSRWJ8PyjMGVQxXAaCgyoIFixTBOEBdW7QjxAAmGloyvpEOzFk5dJ+7xuW3X77qLcuM7jKIIjcAIqwAaXoAFuQRO0AAaP4Bm8gjfjyXgx3o2PeWvByGcOwR8Ynz9HJpcD</latexit>

� <latexit sha1_base64="CPeEMjUCx3WaD3RGo/lTPbtpdYc=">AAAB8XicbVA9SwNBEJ2LXzF+RS1tFoNgFe5E0DJoYxnBJGISwt5mL1myt3vszinhyL+wsVDE1n9j579xk1yhiQ8GHu/NMDMvTKSw6PvfXmFldW19o7hZ2tre2d0r7x80rU4N4w2mpTb3IbVcCsUbKFDy+8RwGoeSt8LR9dRvPXJjhVZ3OE54N6YDJSLBKDrpoSN5hNQY/dQrV/yqPwNZJkFOKpCj3it/dfqapTFXyCS1th34CXYzalAwySelTmp5QtmIDnjbUUVjbrvZ7OIJOXFKn0TauFJIZurviYzG1o7j0HXGFId20ZuK/3ntFKPLbiZUkiJXbL4oSiVBTabvk74wnKEcO0KZEe5WwobUUIYupJILIVh8eZk0z6qBXw1uzyu1qzyOIhzBMZxCABdQgxuoQwMYKHiGV3jzrPfivXsf89aCl88cwh94nz/rCpEQ</latexit><latexit sha1_base64="CPeEMjUCx3WaD3RGo/lTPbtpdYc=">AAAB8XicbVA9SwNBEJ2LXzF+RS1tFoNgFe5E0DJoYxnBJGISwt5mL1myt3vszinhyL+wsVDE1n9j579xk1yhiQ8GHu/NMDMvTKSw6PvfXmFldW19o7hZ2tre2d0r7x80rU4N4w2mpTb3IbVcCsUbKFDy+8RwGoeSt8LR9dRvPXJjhVZ3OE54N6YDJSLBKDrpoSN5hNQY/dQrV/yqPwNZJkFOKpCj3it/dfqapTFXyCS1th34CXYzalAwySelTmp5QtmIDnjbUUVjbrvZ7OIJOXFKn0TauFJIZurviYzG1o7j0HXGFId20ZuK/3ntFKPLbiZUkiJXbL4oSiVBTabvk74wnKEcO0KZEe5WwobUUIYupJILIVh8eZk0z6qBXw1uzyu1qzyOIhzBMZxCABdQgxuoQwMYKHiGV3jzrPfivXsf89aCl88cwh94nz/rCpEQ</latexit><latexit sha1_base64="CPeEMjUCx3WaD3RGo/lTPbtpdYc=">AAAB8XicbVA9SwNBEJ2LXzF+RS1tFoNgFe5E0DJoYxnBJGISwt5mL1myt3vszinhyL+wsVDE1n9j579xk1yhiQ8GHu/NMDMvTKSw6PvfXmFldW19o7hZ2tre2d0r7x80rU4N4w2mpTb3IbVcCsUbKFDy+8RwGoeSt8LR9dRvPXJjhVZ3OE54N6YDJSLBKDrpoSN5hNQY/dQrV/yqPwNZJkFOKpCj3it/dfqapTFXyCS1th34CXYzalAwySelTmp5QtmIDnjbUUVjbrvZ7OIJOXFKn0TauFJIZurviYzG1o7j0HXGFId20ZuK/3ntFKPLbiZUkiJXbL4oSiVBTabvk74wnKEcO0KZEe5WwobUUIYupJILIVh8eZk0z6qBXw1uzyu1qzyOIhzBMZxCABdQgxuoQwMYKHiGV3jzrPfivXsf89aCl88cwh94nz/rCpEQ</latexit><latexit sha1_base64="CPeEMjUCx3WaD3RGo/lTPbtpdYc=">AAAB8XicbVA9SwNBEJ2LXzF+RS1tFoNgFe5E0DJoYxnBJGISwt5mL1myt3vszinhyL+wsVDE1n9j579xk1yhiQ8GHu/NMDMvTKSw6PvfXmFldW19o7hZ2tre2d0r7x80rU4N4w2mpTb3IbVcCsUbKFDy+8RwGoeSt8LR9dRvPXJjhVZ3OE54N6YDJSLBKDrpoSN5hNQY/dQrV/yqPwNZJkFOKpCj3it/dfqapTFXyCS1th34CXYzalAwySelTmp5QtmIDnjbUUVjbrvZ7OIJOXFKn0TauFJIZurviYzG1o7j0HXGFId20ZuK/3ntFKPLbiZUkiJXbL4oSiVBTabvk74wnKEcO0KZEe5WwobUUIYupJILIVh8eZk0z6qBXw1uzyu1qzyOIhzBMZxCABdQgxuoQwMYKHiGV3jzrPfivXsf89aCl88cwh94nz/rCpEQ</latexit>

y = hW(x) + noise<latexit sha1_base64="/XlkhuL+a7CalScycScWtlR3ZTM=">AAACBXicbZDLSsNAFIYn9VbrLepSF4NFqAglEUE3QtGNywr2Am0Ik+mkHTqZCTMTMYRu3Pgqblwo4tZ3cOfbOGmz0OoPAx//OYc55w9iRpV2nC+rtLC4tLxSXq2srW9sbtnbO20lEolJCwsmZDdAijDKSUtTzUg3lgRFASOdYHyV1zt3RCoq+K1OY+JFaMhpSDHSxvLt/RRewJHfj5AeYcSyzqR2fwSPIRdUEd+uOnVnKvgX3AKqoFDTtz/7A4GTiHCNGVKq5zqx9jIkNcWMTCr9RJEY4TEakp5BjiKivGx6xQQeGmcAQyHN4xpO3Z8TGYqUSqPAdObbqvlabv5X6yU6PPcyyuNEE45nH4UJg1rAPBI4oJJgzVIDCEtqdoV4hCTC2gRXMSG48yf/hfZJ3TV8c1ptXBZxlMEeOAA14IIz0ADXoAlaAIMH8ARewKv1aD1bb9b7rLVkFTO74Jesj2+V/JdX</latexit><latexit sha1_base64="/XlkhuL+a7CalScycScWtlR3ZTM=">AAACBXicbZDLSsNAFIYn9VbrLepSF4NFqAglEUE3QtGNywr2Am0Ik+mkHTqZCTMTMYRu3Pgqblwo4tZ3cOfbOGmz0OoPAx//OYc55w9iRpV2nC+rtLC4tLxSXq2srW9sbtnbO20lEolJCwsmZDdAijDKSUtTzUg3lgRFASOdYHyV1zt3RCoq+K1OY+JFaMhpSDHSxvLt/RRewJHfj5AeYcSyzqR2fwSPIRdUEd+uOnVnKvgX3AKqoFDTtz/7A4GTiHCNGVKq5zqx9jIkNcWMTCr9RJEY4TEakp5BjiKivGx6xQQeGmcAQyHN4xpO3Z8TGYqUSqPAdObbqvlabv5X6yU6PPcyyuNEE45nH4UJg1rAPBI4oJJgzVIDCEtqdoV4hCTC2gRXMSG48yf/hfZJ3TV8c1ptXBZxlMEeOAA14IIz0ADXoAlaAIMH8ARewKv1aD1bb9b7rLVkFTO74Jesj2+V/JdX</latexit><latexit sha1_base64="/XlkhuL+a7CalScycScWtlR3ZTM=">AAACBXicbZDLSsNAFIYn9VbrLepSF4NFqAglEUE3QtGNywr2Am0Ik+mkHTqZCTMTMYRu3Pgqblwo4tZ3cOfbOGmz0OoPAx//OYc55w9iRpV2nC+rtLC4tLxSXq2srW9sbtnbO20lEolJCwsmZDdAijDKSUtTzUg3lgRFASOdYHyV1zt3RCoq+K1OY+JFaMhpSDHSxvLt/RRewJHfj5AeYcSyzqR2fwSPIRdUEd+uOnVnKvgX3AKqoFDTtz/7A4GTiHCNGVKq5zqx9jIkNcWMTCr9RJEY4TEakp5BjiKivGx6xQQeGmcAQyHN4xpO3Z8TGYqUSqPAdObbqvlabv5X6yU6PPcyyuNEE45nH4UJg1rAPBI4oJJgzVIDCEtqdoV4hCTC2gRXMSG48yf/hfZJ3TV8c1ptXBZxlMEeOAA14IIz0ADXoAlaAIMH8ARewKv1aD1bb9b7rLVkFTO74Jesj2+V/JdX</latexit><latexit sha1_base64="/XlkhuL+a7CalScycScWtlR3ZTM=">AAACBXicbZDLSsNAFIYn9VbrLepSF4NFqAglEUE3QtGNywr2Am0Ik+mkHTqZCTMTMYRu3Pgqblwo4tZ3cOfbOGmz0OoPAx//OYc55w9iRpV2nC+rtLC4tLxSXq2srW9sbtnbO20lEolJCwsmZDdAijDKSUtTzUg3lgRFASOdYHyV1zt3RCoq+K1OY+JFaMhpSDHSxvLt/RRewJHfj5AeYcSyzqR2fwSPIRdUEd+uOnVnKvgX3AKqoFDTtz/7A4GTiHCNGVKq5zqx9jIkNcWMTCr9RJEY4TEakp5BjiKivGx6xQQeGmcAQyHN4xpO3Z8TGYqUSqPAdObbqvlabv5X6yU6PPcyyuNEE45nH4UJg1rAPBI4oJJgzVIDCEtqdoV4hCTC2gRXMSG48yf/hfZJ3TV8c1ptXBZxlMEeOAA14IIz0ADXoAlaAIMH8ARewKv1aD1bb9b7rLVkFTO74Jesj2+V/JdX</latexit>

Page 3: Horseshoe Priors for Bayesian Neural Networks...Horseshoe Priors for Bayesian Neural Networks Soumya Ghosh Jiayu Yao Finale Doshi-Velez IBM Research MIT-IBM Watson AI lab Harvard Harvard

Why Bother?

!3

• Need to guard against unintended consequences.

• Need to know when the model doesn’t know.

Page 4: Horseshoe Priors for Bayesian Neural Networks...Horseshoe Priors for Bayesian Neural Networks Soumya Ghosh Jiayu Yao Finale Doshi-Velez IBM Research MIT-IBM Watson AI lab Harvard Harvard

Predictive Uncertainty

!4

++

Page 5: Horseshoe Priors for Bayesian Neural Networks...Horseshoe Priors for Bayesian Neural Networks Soumya Ghosh Jiayu Yao Finale Doshi-Velez IBM Research MIT-IBM Watson AI lab Harvard Harvard

Larger Data and Modern Architectures

!5

Train: 60,000 Handwritten digits Test: 10,000 heldout digits

Convolution Neural Network ( LeNet variant )

Test Error ~ 1%

Page 6: Horseshoe Priors for Bayesian Neural Networks...Horseshoe Priors for Bayesian Neural Networks Soumya Ghosh Jiayu Yao Finale Doshi-Velez IBM Research MIT-IBM Watson AI lab Harvard Harvard

Are the predictions robust?

!6

7

1

0.99

7

1 0.85

0.1

4

1

7

0.955

9

0

0.9

Page 7: Horseshoe Priors for Bayesian Neural Networks...Horseshoe Priors for Bayesian Neural Networks Soumya Ghosh Jiayu Yao Finale Doshi-Velez IBM Research MIT-IBM Watson AI lab Harvard Harvard

Bayesian Neural Networks

!7

+ +

Distribution on weights:

Zp(y⇤ | W , x⇤)p(W | ytrain, xtrain)dW

source: Ghosh et al., AAAI 2016 Balan et al., NIPS 2016

Page 8: Horseshoe Priors for Bayesian Neural Networks...Horseshoe Priors for Bayesian Neural Networks Soumya Ghosh Jiayu Yao Finale Doshi-Velez IBM Research MIT-IBM Watson AI lab Harvard Harvard

Bayesian Neural Networks

!8

5

9

3

0.30

0.28

0.26

experiment inspired by: Gal et al., 2015

Page 9: Horseshoe Priors for Bayesian Neural Networks...Horseshoe Priors for Bayesian Neural Networks Soumya Ghosh Jiayu Yao Finale Doshi-Velez IBM Research MIT-IBM Watson AI lab Harvard Harvard

random variable

random variable

f(x) = hW(x)<latexit sha1_base64="Xq9dXC4/0b1K2GmMlddrniNulLI=">AAACJXicbVDLSgMxFM34rPVVdekmWIS6KTMi6EKh6MZlBfuAtpRMeqcNzTxI7kjLMD/jxl9x48Iigit/xXTahbYeCBzOOTe5OW4khUbb/rJWVtfWNzZzW/ntnd29/cLBYV2HseJQ46EMVdNlGqQIoIYCJTQjBcx3JTTc4d3UbzyB0iIMHnEcQcdn/UB4gjM0Urdw3UYYYXZPoqCXJl5pdJbSG0oH3WTRa/sMB5zJpJGmqcl1C0W7bGegy8SZkyKZo9otTNq9kMc+BMgl07rl2BF2EqZQcAlpvh1riBgfsj60DA2YD7qTZBuk9NQoPeqFypwAaab+nkiYr/XYd01yuqde9Kbif14rRu+qk4ggihECPnvIiyXFkE4roz2hgKMcG8K4EmZXygdMMY6m2LwpwVn88jKpn5cdwx8uipXbeR05ckxOSIk45JJUyD2pkhrh5Jm8kncysV6sN+vD+pxFV6z5zBH5A+v7B948prI=</latexit><latexit sha1_base64="Xq9dXC4/0b1K2GmMlddrniNulLI=">AAACJXicbVDLSgMxFM34rPVVdekmWIS6KTMi6EKh6MZlBfuAtpRMeqcNzTxI7kjLMD/jxl9x48Iigit/xXTahbYeCBzOOTe5OW4khUbb/rJWVtfWNzZzW/ntnd29/cLBYV2HseJQ46EMVdNlGqQIoIYCJTQjBcx3JTTc4d3UbzyB0iIMHnEcQcdn/UB4gjM0Urdw3UYYYXZPoqCXJl5pdJbSG0oH3WTRa/sMB5zJpJGmqcl1C0W7bGegy8SZkyKZo9otTNq9kMc+BMgl07rl2BF2EqZQcAlpvh1riBgfsj60DA2YD7qTZBuk9NQoPeqFypwAaab+nkiYr/XYd01yuqde9Kbif14rRu+qk4ggihECPnvIiyXFkE4roz2hgKMcG8K4EmZXygdMMY6m2LwpwVn88jKpn5cdwx8uipXbeR05ckxOSIk45JJUyD2pkhrh5Jm8kncysV6sN+vD+pxFV6z5zBH5A+v7B948prI=</latexit><latexit sha1_base64="Xq9dXC4/0b1K2GmMlddrniNulLI=">AAACJXicbVDLSgMxFM34rPVVdekmWIS6KTMi6EKh6MZlBfuAtpRMeqcNzTxI7kjLMD/jxl9x48Iigit/xXTahbYeCBzOOTe5OW4khUbb/rJWVtfWNzZzW/ntnd29/cLBYV2HseJQ46EMVdNlGqQIoIYCJTQjBcx3JTTc4d3UbzyB0iIMHnEcQcdn/UB4gjM0Urdw3UYYYXZPoqCXJl5pdJbSG0oH3WTRa/sMB5zJpJGmqcl1C0W7bGegy8SZkyKZo9otTNq9kMc+BMgl07rl2BF2EqZQcAlpvh1riBgfsj60DA2YD7qTZBuk9NQoPeqFypwAaab+nkiYr/XYd01yuqde9Kbif14rRu+qk4ggihECPnvIiyXFkE4roz2hgKMcG8K4EmZXygdMMY6m2LwpwVn88jKpn5cdwx8uipXbeR05ckxOSIk45JJUyD2pkhrh5Jm8kncysV6sN+vD+pxFV6z5zBH5A+v7B948prI=</latexit><latexit sha1_base64="Xq9dXC4/0b1K2GmMlddrniNulLI=">AAACJXicbVDLSgMxFM34rPVVdekmWIS6KTMi6EKh6MZlBfuAtpRMeqcNzTxI7kjLMD/jxl9x48Iigit/xXTahbYeCBzOOTe5OW4khUbb/rJWVtfWNzZzW/ntnd29/cLBYV2HseJQ46EMVdNlGqQIoIYCJTQjBcx3JTTc4d3UbzyB0iIMHnEcQcdn/UB4gjM0Urdw3UYYYXZPoqCXJl5pdJbSG0oH3WTRa/sMB5zJpJGmqcl1C0W7bGegy8SZkyKZo9otTNq9kMc+BMgl07rl2BF2EqZQcAlpvh1riBgfsj60DA2YD7qTZBuk9NQoPeqFypwAaab+nkiYr/XYd01yuqde9Kbif14rRu+qk4ggihECPnvIiyXFkE4roz2hgKMcG8K4EmZXygdMMY6m2LwpwVn88jKpn5cdwx8uipXbeR05ckxOSIk45JJUyD2pkhrh5Jm8kncysV6sN+vD+pxFV6z5zBH5A+v7B948prI=</latexit>

Distribution over Functions

!9

Page 10: Horseshoe Priors for Bayesian Neural Networks...Horseshoe Priors for Bayesian Neural Networks Soumya Ghosh Jiayu Yao Finale Doshi-Velez IBM Research MIT-IBM Watson AI lab Harvard Harvard

BNNs - applications

Model stochastic functions Depweg et al., ICLR 2017

Model uncertainty in deterministic functions Killian et al., NIPS 2017

Predictive uncertainties for active learning, sequential decision making Hernández-Lobato et al., ICML 2015, Gal et al., ICML 2017, Joshi et al., CVPR 2017, Zhang et al., AISTATS 2018, Depweg et al., ICML 2018, Riquelme et al., ICLR 2018

!10

Page 11: Horseshoe Priors for Bayesian Neural Networks...Horseshoe Priors for Bayesian Neural Networks Soumya Ghosh Jiayu Yao Finale Doshi-Velez IBM Research MIT-IBM Watson AI lab Harvard Harvard

Alternate distribution over functions

!11

Gaussian Processes Bayesian Neural Networks

f(x) � GP(m(x), K(x, x�))<latexit sha1_base64="xFl22ReGOlQcU3Yv8/UZt4O7oq8=">AAACDHicbVDLSgMxFM34rPVVdekmWMQWSpkRQZdFFwpuKtgHdIaSSTNtaDIzJHekZegHuPFX3LhQxK0f4M6/MX0stPVA4HDOudzc48eCa7Dtb2tpeWV1bT2zkd3c2t7Zze3t13WUKMpqNBKRavpEM8FDVgMOgjVjxYj0BWv4/aux33hgSvMovIdhzDxJuiEPOCVgpHYuHxQGRexqLrELbADpdXVUkEYr4dvCoIQHJ8WiSdllewK8SJwZyaMZqu3cl9uJaCJZCFQQrVuOHYOXEgWcCjbKuolmMaF90mUtQ0MimfbSyTEjfGyUDg4iZV4IeKL+nkiJ1HoofZOUBHp63huL/3mtBIILL+VhnAAL6XRRkAgMER43gztcMQpiaAihipu/YtojilAw/WVNCc78yYukflp2DL87y1cuZ3Vk0CE6QgXkoHNUQTeoimqIokf0jF7Rm/VkvVjv1sc0umTNZg7QH1ifP2RlmJ8=</latexit><latexit sha1_base64="xFl22ReGOlQcU3Yv8/UZt4O7oq8=">AAACDHicbVDLSgMxFM34rPVVdekmWMQWSpkRQZdFFwpuKtgHdIaSSTNtaDIzJHekZegHuPFX3LhQxK0f4M6/MX0stPVA4HDOudzc48eCa7Dtb2tpeWV1bT2zkd3c2t7Zze3t13WUKMpqNBKRavpEM8FDVgMOgjVjxYj0BWv4/aux33hgSvMovIdhzDxJuiEPOCVgpHYuHxQGRexqLrELbADpdXVUkEYr4dvCoIQHJ8WiSdllewK8SJwZyaMZqu3cl9uJaCJZCFQQrVuOHYOXEgWcCjbKuolmMaF90mUtQ0MimfbSyTEjfGyUDg4iZV4IeKL+nkiJ1HoofZOUBHp63huL/3mtBIILL+VhnAAL6XRRkAgMER43gztcMQpiaAihipu/YtojilAw/WVNCc78yYukflp2DL87y1cuZ3Vk0CE6QgXkoHNUQTeoimqIokf0jF7Rm/VkvVjv1sc0umTNZg7QH1ifP2RlmJ8=</latexit><latexit sha1_base64="xFl22ReGOlQcU3Yv8/UZt4O7oq8=">AAACDHicbVDLSgMxFM34rPVVdekmWMQWSpkRQZdFFwpuKtgHdIaSSTNtaDIzJHekZegHuPFX3LhQxK0f4M6/MX0stPVA4HDOudzc48eCa7Dtb2tpeWV1bT2zkd3c2t7Zze3t13WUKMpqNBKRavpEM8FDVgMOgjVjxYj0BWv4/aux33hgSvMovIdhzDxJuiEPOCVgpHYuHxQGRexqLrELbADpdXVUkEYr4dvCoIQHJ8WiSdllewK8SJwZyaMZqu3cl9uJaCJZCFQQrVuOHYOXEgWcCjbKuolmMaF90mUtQ0MimfbSyTEjfGyUDg4iZV4IeKL+nkiJ1HoofZOUBHp63huL/3mtBIILL+VhnAAL6XRRkAgMER43gztcMQpiaAihipu/YtojilAw/WVNCc78yYukflp2DL87y1cuZ3Vk0CE6QgXkoHNUQTeoimqIokf0jF7Rm/VkvVjv1sc0umTNZg7QH1ifP2RlmJ8=</latexit><latexit sha1_base64="xFl22ReGOlQcU3Yv8/UZt4O7oq8=">AAACDHicbVDLSgMxFM34rPVVdekmWMQWSpkRQZdFFwpuKtgHdIaSSTNtaDIzJHekZegHuPFX3LhQxK0f4M6/MX0stPVA4HDOudzc48eCa7Dtb2tpeWV1bT2zkd3c2t7Zze3t13WUKMpqNBKRavpEM8FDVgMOgjVjxYj0BWv4/aux33hgSvMovIdhzDxJuiEPOCVgpHYuHxQGRexqLrELbADpdXVUkEYr4dvCoIQHJ8WiSdllewK8SJwZyaMZqu3cl9uJaCJZCFQQrVuOHYOXEgWcCjbKuolmMaF90mUtQ0MimfbSyTEjfGyUDg4iZV4IeKL+nkiJ1HoofZOUBHp63huL/3mtBIILL+VhnAAL6XRRkAgMER43gztcMQpiaAihipu/YtojilAw/WVNCc78yYukflp2DL87y1cuZ3Vk0CE6QgXkoHNUQTeoimqIokf0jF7Rm/VkvVjv1sc0umTNZg7QH1ifP2RlmJ8=</latexit>

f(x) � hW(x)<latexit sha1_base64="1qSafpd3upOjzSIVuy/6/h3+F9A=">AAACBXicbZC7TsMwFIYdrqXcAowwWFRIZakShARjBQtjkehFaqLIcZ3Wqu1EtoOooiwsvAoLAwix8g5svA1OmwFafsnSp/+cI5/zhwmjSjvOt7W0vLK6tl7ZqG5ube/s2nv7HRWnEpM2jlkseyFShFFB2ppqRnqJJIiHjHTD8XVR794TqWgs7vQkIT5HQ0EjipE2VmAfRfWHU+gpyuEoyDyO9AgjlnXz3PiBXXMazlRwEdwSaqBUK7C/vEGMU06Exgwp1XedRPsZkppiRvKqlyqSIDxGQ9I3KBAnys+mV+TwxDgDGMXSPKHh1P09kSGu1ISHprNYU83XCvO/Wj/V0aWfUZGkmgg8+yhKGdQxLCKBAyoJ1mxiAGFJza4Qj5BEWJvgqiYEd/7kReicNVzDt+e15lUZRwUcgmNQBy64AE1wA1qgDTB4BM/gFbxZT9aL9W59zFqXrHLmAPyR9fkDbzaX5A==</latexit><latexit sha1_base64="1qSafpd3upOjzSIVuy/6/h3+F9A=">AAACBXicbZC7TsMwFIYdrqXcAowwWFRIZakShARjBQtjkehFaqLIcZ3Wqu1EtoOooiwsvAoLAwix8g5svA1OmwFafsnSp/+cI5/zhwmjSjvOt7W0vLK6tl7ZqG5ube/s2nv7HRWnEpM2jlkseyFShFFB2ppqRnqJJIiHjHTD8XVR794TqWgs7vQkIT5HQ0EjipE2VmAfRfWHU+gpyuEoyDyO9AgjlnXz3PiBXXMazlRwEdwSaqBUK7C/vEGMU06Exgwp1XedRPsZkppiRvKqlyqSIDxGQ9I3KBAnys+mV+TwxDgDGMXSPKHh1P09kSGu1ISHprNYU83XCvO/Wj/V0aWfUZGkmgg8+yhKGdQxLCKBAyoJ1mxiAGFJza4Qj5BEWJvgqiYEd/7kReicNVzDt+e15lUZRwUcgmNQBy64AE1wA1qgDTB4BM/gFbxZT9aL9W59zFqXrHLmAPyR9fkDbzaX5A==</latexit><latexit sha1_base64="1qSafpd3upOjzSIVuy/6/h3+F9A=">AAACBXicbZC7TsMwFIYdrqXcAowwWFRIZakShARjBQtjkehFaqLIcZ3Wqu1EtoOooiwsvAoLAwix8g5svA1OmwFafsnSp/+cI5/zhwmjSjvOt7W0vLK6tl7ZqG5ube/s2nv7HRWnEpM2jlkseyFShFFB2ppqRnqJJIiHjHTD8XVR794TqWgs7vQkIT5HQ0EjipE2VmAfRfWHU+gpyuEoyDyO9AgjlnXz3PiBXXMazlRwEdwSaqBUK7C/vEGMU06Exgwp1XedRPsZkppiRvKqlyqSIDxGQ9I3KBAnys+mV+TwxDgDGMXSPKHh1P09kSGu1ISHprNYU83XCvO/Wj/V0aWfUZGkmgg8+yhKGdQxLCKBAyoJ1mxiAGFJza4Qj5BEWJvgqiYEd/7kReicNVzDt+e15lUZRwUcgmNQBy64AE1wA1qgDTB4BM/gFbxZT9aL9W59zFqXrHLmAPyR9fkDbzaX5A==</latexit><latexit sha1_base64="1qSafpd3upOjzSIVuy/6/h3+F9A=">AAACBXicbZC7TsMwFIYdrqXcAowwWFRIZakShARjBQtjkehFaqLIcZ3Wqu1EtoOooiwsvAoLAwix8g5svA1OmwFafsnSp/+cI5/zhwmjSjvOt7W0vLK6tl7ZqG5ube/s2nv7HRWnEpM2jlkseyFShFFB2ppqRnqJJIiHjHTD8XVR794TqWgs7vQkIT5HQ0EjipE2VmAfRfWHU+gpyuEoyDyO9AgjlnXz3PiBXXMazlRwEdwSaqBUK7C/vEGMU06Exgwp1XedRPsZkppiRvKqlyqSIDxGQ9I3KBAnys+mV+TwxDgDGMXSPKHh1P09kSGu1ISHprNYU83XCvO/Wj/V0aWfUZGkmgg8+yhKGdQxLCKBAyoJ1mxiAGFJza4Qj5BEWJvgqiYEd/7kReicNVzDt+e15lUZRwUcgmNQBy64AE1wA1qgDTB4BM/gFbxZT9aL9W59zFqXrHLmAPyR9fkDbzaX5A==</latexit>

y(x) | f(x) � N (f(x), �2)<latexit sha1_base64="uph3aPp7mCNMeMierN/5SbxfLAY=">AAACGHicbZBNS8NAEIY39avWr6hHL4tFaEFqUgQ9Fr14kgr2A9pYNttNu3Q3CbsbMYT+DC/+FS8eFPHam//GTZqDtg4sPPvODDPzuiGjUlnWt1FYWV1b3yhulra2d3b3zP2DtgwigUkLBywQXRdJwqhPWooqRrqhIIi7jHTcyXWa7zwSIWng36s4JA5HI596FCOlpYF5FleeqrDP6RB6GUnK9RepMUYsuZ1WUvU0lUccPdSrA7Ns1aws4DLYOZRBHs2BOesPAxxx4ivMkJQ92wqVkyChKGZkWupHkoQIT9CI9DT6iBPpJNlhU3iiFb1YIPTzFczU3x0J4lLG3NWV6cpyMZeK/+V6kfIunYT6YaSIj+eDvIhBFcDUJTikgmDFYg0IC6p3hXiMBMJKe1nSJtiLJy9Du16zNd+dlxtXuR1FcASOQQXY4AI0wA1oghbA4Bm8gnfwYbwYb8an8TUvLRh5zyH4E8bsBx/8nfI=</latexit><latexit sha1_base64="uph3aPp7mCNMeMierN/5SbxfLAY=">AAACGHicbZBNS8NAEIY39avWr6hHL4tFaEFqUgQ9Fr14kgr2A9pYNttNu3Q3CbsbMYT+DC/+FS8eFPHam//GTZqDtg4sPPvODDPzuiGjUlnWt1FYWV1b3yhulra2d3b3zP2DtgwigUkLBywQXRdJwqhPWooqRrqhIIi7jHTcyXWa7zwSIWng36s4JA5HI596FCOlpYF5FleeqrDP6RB6GUnK9RepMUYsuZ1WUvU0lUccPdSrA7Ns1aws4DLYOZRBHs2BOesPAxxx4ivMkJQ92wqVkyChKGZkWupHkoQIT9CI9DT6iBPpJNlhU3iiFb1YIPTzFczU3x0J4lLG3NWV6cpyMZeK/+V6kfIunYT6YaSIj+eDvIhBFcDUJTikgmDFYg0IC6p3hXiMBMJKe1nSJtiLJy9Du16zNd+dlxtXuR1FcASOQQXY4AI0wA1oghbA4Bm8gnfwYbwYb8an8TUvLRh5zyH4E8bsBx/8nfI=</latexit><latexit sha1_base64="uph3aPp7mCNMeMierN/5SbxfLAY=">AAACGHicbZBNS8NAEIY39avWr6hHL4tFaEFqUgQ9Fr14kgr2A9pYNttNu3Q3CbsbMYT+DC/+FS8eFPHam//GTZqDtg4sPPvODDPzuiGjUlnWt1FYWV1b3yhulra2d3b3zP2DtgwigUkLBywQXRdJwqhPWooqRrqhIIi7jHTcyXWa7zwSIWng36s4JA5HI596FCOlpYF5FleeqrDP6RB6GUnK9RepMUYsuZ1WUvU0lUccPdSrA7Ns1aws4DLYOZRBHs2BOesPAxxx4ivMkJQ92wqVkyChKGZkWupHkoQIT9CI9DT6iBPpJNlhU3iiFb1YIPTzFczU3x0J4lLG3NWV6cpyMZeK/+V6kfIunYT6YaSIj+eDvIhBFcDUJTikgmDFYg0IC6p3hXiMBMJKe1nSJtiLJy9Du16zNd+dlxtXuR1FcASOQQXY4AI0wA1oghbA4Bm8gnfwYbwYb8an8TUvLRh5zyH4E8bsBx/8nfI=</latexit><latexit sha1_base64="uph3aPp7mCNMeMierN/5SbxfLAY=">AAACGHicbZBNS8NAEIY39avWr6hHL4tFaEFqUgQ9Fr14kgr2A9pYNttNu3Q3CbsbMYT+DC/+FS8eFPHam//GTZqDtg4sPPvODDPzuiGjUlnWt1FYWV1b3yhulra2d3b3zP2DtgwigUkLBywQXRdJwqhPWooqRrqhIIi7jHTcyXWa7zwSIWng36s4JA5HI596FCOlpYF5FleeqrDP6RB6GUnK9RepMUYsuZ1WUvU0lUccPdSrA7Ns1aws4DLYOZRBHs2BOesPAxxx4ivMkJQ92wqVkyChKGZkWupHkoQIT9CI9DT6iBPpJNlhU3iiFb1YIPTzFczU3x0J4lLG3NWV6cpyMZeK/+V6kfIunYT6YaSIj+eDvIhBFcDUJTikgmDFYg0IC6p3hXiMBMJKe1nSJtiLJy9Du16zNd+dlxtXuR1FcASOQQXY4AI0wA1oghbA4Bm8gnfwYbwYb8an8TUvLRh5zyH4E8bsBx/8nfI=</latexit>

Gaussian likelihoodsRest of this talk, Noisy data model:

Page 12: Horseshoe Priors for Bayesian Neural Networks...Horseshoe Priors for Bayesian Neural Networks Soumya Ghosh Jiayu Yao Finale Doshi-Velez IBM Research MIT-IBM Watson AI lab Harvard Harvard

!12

Gaussian Processes Bayesian Neural Networks

f(x) � GP(m(x), K(x, x�))<latexit sha1_base64="xFl22ReGOlQcU3Yv8/UZt4O7oq8=">AAACDHicbVDLSgMxFM34rPVVdekmWMQWSpkRQZdFFwpuKtgHdIaSSTNtaDIzJHekZegHuPFX3LhQxK0f4M6/MX0stPVA4HDOudzc48eCa7Dtb2tpeWV1bT2zkd3c2t7Zze3t13WUKMpqNBKRavpEM8FDVgMOgjVjxYj0BWv4/aux33hgSvMovIdhzDxJuiEPOCVgpHYuHxQGRexqLrELbADpdXVUkEYr4dvCoIQHJ8WiSdllewK8SJwZyaMZqu3cl9uJaCJZCFQQrVuOHYOXEgWcCjbKuolmMaF90mUtQ0MimfbSyTEjfGyUDg4iZV4IeKL+nkiJ1HoofZOUBHp63huL/3mtBIILL+VhnAAL6XRRkAgMER43gztcMQpiaAihipu/YtojilAw/WVNCc78yYukflp2DL87y1cuZ3Vk0CE6QgXkoHNUQTeoimqIokf0jF7Rm/VkvVjv1sc0umTNZg7QH1ifP2RlmJ8=</latexit><latexit sha1_base64="xFl22ReGOlQcU3Yv8/UZt4O7oq8=">AAACDHicbVDLSgMxFM34rPVVdekmWMQWSpkRQZdFFwpuKtgHdIaSSTNtaDIzJHekZegHuPFX3LhQxK0f4M6/MX0stPVA4HDOudzc48eCa7Dtb2tpeWV1bT2zkd3c2t7Zze3t13WUKMpqNBKRavpEM8FDVgMOgjVjxYj0BWv4/aux33hgSvMovIdhzDxJuiEPOCVgpHYuHxQGRexqLrELbADpdXVUkEYr4dvCoIQHJ8WiSdllewK8SJwZyaMZqu3cl9uJaCJZCFQQrVuOHYOXEgWcCjbKuolmMaF90mUtQ0MimfbSyTEjfGyUDg4iZV4IeKL+nkiJ1HoofZOUBHp63huL/3mtBIILL+VhnAAL6XRRkAgMER43gztcMQpiaAihipu/YtojilAw/WVNCc78yYukflp2DL87y1cuZ3Vk0CE6QgXkoHNUQTeoimqIokf0jF7Rm/VkvVjv1sc0umTNZg7QH1ifP2RlmJ8=</latexit><latexit sha1_base64="xFl22ReGOlQcU3Yv8/UZt4O7oq8=">AAACDHicbVDLSgMxFM34rPVVdekmWMQWSpkRQZdFFwpuKtgHdIaSSTNtaDIzJHekZegHuPFX3LhQxK0f4M6/MX0stPVA4HDOudzc48eCa7Dtb2tpeWV1bT2zkd3c2t7Zze3t13WUKMpqNBKRavpEM8FDVgMOgjVjxYj0BWv4/aux33hgSvMovIdhzDxJuiEPOCVgpHYuHxQGRexqLrELbADpdXVUkEYr4dvCoIQHJ8WiSdllewK8SJwZyaMZqu3cl9uJaCJZCFQQrVuOHYOXEgWcCjbKuolmMaF90mUtQ0MimfbSyTEjfGyUDg4iZV4IeKL+nkiJ1HoofZOUBHp63huL/3mtBIILL+VhnAAL6XRRkAgMER43gztcMQpiaAihipu/YtojilAw/WVNCc78yYukflp2DL87y1cuZ3Vk0CE6QgXkoHNUQTeoimqIokf0jF7Rm/VkvVjv1sc0umTNZg7QH1ifP2RlmJ8=</latexit><latexit sha1_base64="xFl22ReGOlQcU3Yv8/UZt4O7oq8=">AAACDHicbVDLSgMxFM34rPVVdekmWMQWSpkRQZdFFwpuKtgHdIaSSTNtaDIzJHekZegHuPFX3LhQxK0f4M6/MX0stPVA4HDOudzc48eCa7Dtb2tpeWV1bT2zkd3c2t7Zze3t13WUKMpqNBKRavpEM8FDVgMOgjVjxYj0BWv4/aux33hgSvMovIdhzDxJuiEPOCVgpHYuHxQGRexqLrELbADpdXVUkEYr4dvCoIQHJ8WiSdllewK8SJwZyaMZqu3cl9uJaCJZCFQQrVuOHYOXEgWcCjbKuolmMaF90mUtQ0MimfbSyTEjfGyUDg4iZV4IeKL+nkiJ1HoofZOUBHp63huL/3mtBIILL+VhnAAL6XRRkAgMER43gztcMQpiaAihipu/YtojilAw/WVNCc78yYukflp2DL87y1cuZ3Vk0CE6QgXkoHNUQTeoimqIokf0jF7Rm/VkvVjv1sc0umTNZg7QH1ifP2RlmJ8=</latexit>

f(x) � hW(x)<latexit sha1_base64="1qSafpd3upOjzSIVuy/6/h3+F9A=">AAACBXicbZC7TsMwFIYdrqXcAowwWFRIZakShARjBQtjkehFaqLIcZ3Wqu1EtoOooiwsvAoLAwix8g5svA1OmwFafsnSp/+cI5/zhwmjSjvOt7W0vLK6tl7ZqG5ube/s2nv7HRWnEpM2jlkseyFShFFB2ppqRnqJJIiHjHTD8XVR794TqWgs7vQkIT5HQ0EjipE2VmAfRfWHU+gpyuEoyDyO9AgjlnXz3PiBXXMazlRwEdwSaqBUK7C/vEGMU06Exgwp1XedRPsZkppiRvKqlyqSIDxGQ9I3KBAnys+mV+TwxDgDGMXSPKHh1P09kSGu1ISHprNYU83XCvO/Wj/V0aWfUZGkmgg8+yhKGdQxLCKBAyoJ1mxiAGFJza4Qj5BEWJvgqiYEd/7kReicNVzDt+e15lUZRwUcgmNQBy64AE1wA1qgDTB4BM/gFbxZT9aL9W59zFqXrHLmAPyR9fkDbzaX5A==</latexit><latexit sha1_base64="1qSafpd3upOjzSIVuy/6/h3+F9A=">AAACBXicbZC7TsMwFIYdrqXcAowwWFRIZakShARjBQtjkehFaqLIcZ3Wqu1EtoOooiwsvAoLAwix8g5svA1OmwFafsnSp/+cI5/zhwmjSjvOt7W0vLK6tl7ZqG5ube/s2nv7HRWnEpM2jlkseyFShFFB2ppqRnqJJIiHjHTD8XVR794TqWgs7vQkIT5HQ0EjipE2VmAfRfWHU+gpyuEoyDyO9AgjlnXz3PiBXXMazlRwEdwSaqBUK7C/vEGMU06Exgwp1XedRPsZkppiRvKqlyqSIDxGQ9I3KBAnys+mV+TwxDgDGMXSPKHh1P09kSGu1ISHprNYU83XCvO/Wj/V0aWfUZGkmgg8+yhKGdQxLCKBAyoJ1mxiAGFJza4Qj5BEWJvgqiYEd/7kReicNVzDt+e15lUZRwUcgmNQBy64AE1wA1qgDTB4BM/gFbxZT9aL9W59zFqXrHLmAPyR9fkDbzaX5A==</latexit><latexit sha1_base64="1qSafpd3upOjzSIVuy/6/h3+F9A=">AAACBXicbZC7TsMwFIYdrqXcAowwWFRIZakShARjBQtjkehFaqLIcZ3Wqu1EtoOooiwsvAoLAwix8g5svA1OmwFafsnSp/+cI5/zhwmjSjvOt7W0vLK6tl7ZqG5ube/s2nv7HRWnEpM2jlkseyFShFFB2ppqRnqJJIiHjHTD8XVR794TqWgs7vQkIT5HQ0EjipE2VmAfRfWHU+gpyuEoyDyO9AgjlnXz3PiBXXMazlRwEdwSaqBUK7C/vEGMU06Exgwp1XedRPsZkppiRvKqlyqSIDxGQ9I3KBAnys+mV+TwxDgDGMXSPKHh1P09kSGu1ISHprNYU83XCvO/Wj/V0aWfUZGkmgg8+yhKGdQxLCKBAyoJ1mxiAGFJza4Qj5BEWJvgqiYEd/7kReicNVzDt+e15lUZRwUcgmNQBy64AE1wA1qgDTB4BM/gFbxZT9aL9W59zFqXrHLmAPyR9fkDbzaX5A==</latexit><latexit sha1_base64="1qSafpd3upOjzSIVuy/6/h3+F9A=">AAACBXicbZC7TsMwFIYdrqXcAowwWFRIZakShARjBQtjkehFaqLIcZ3Wqu1EtoOooiwsvAoLAwix8g5svA1OmwFafsnSp/+cI5/zhwmjSjvOt7W0vLK6tl7ZqG5ube/s2nv7HRWnEpM2jlkseyFShFFB2ppqRnqJJIiHjHTD8XVR794TqWgs7vQkIT5HQ0EjipE2VmAfRfWHU+gpyuEoyDyO9AgjlnXz3PiBXXMazlRwEdwSaqBUK7C/vEGMU06Exgwp1XedRPsZkppiRvKqlyqSIDxGQ9I3KBAnys+mV+TwxDgDGMXSPKHh1P09kSGu1ISHprNYU83XCvO/Wj/V0aWfUZGkmgg8+yhKGdQxLCKBAyoJ1mxiAGFJza4Qj5BEWJvgqiYEd/7kReicNVzDt+e15lUZRwUcgmNQBy64AE1wA1qgDTB4BM/gFbxZT9aL9W59zFqXrHLmAPyR9fkDbzaX5A==</latexit>

• Exact Inference* * Only for Gaussian Likelihoods • Approximate Inference

• Scales Poorly with n • Scales Well

• Well calibrated uncertainties

• Predictive uncertainties can be poor

• Constraining the space of functions can be difficult

• Some are easy; depends on C

f � GP (., .); f � C<latexit sha1_base64="TU5zZde4qbCx4WoxFSYbids+sH8=">AAACAnicbZDLSsNAFIZP6q3WW9SVuBksQgUpiQgKbopd6LKCvUAbymQ6aYdOJmFmIpRQ3Pgqblwo4tancOfbOG2z0NYfBj7+cw5nzu/HnCntON9Wbml5ZXUtv17Y2Nza3rF39xoqSiShdRLxSLZ8rChngtY105y2Yklx6HPa9IfVSb35QKVikbjXo5h6Ie4LFjCCtbG69kGAOoqF6KZWKp+i8gm6QsZhAlW7dtEpO1OhRXAzKEKmWtf+6vQikoRUaMKxUm3XibWXYqkZ4XRc6CSKxpgMcZ+2DQocUuWl0xPG6Ng4PRRE0jyh0dT9PZHiUKlR6JvOEOuBmq9NzP9q7UQHl17KRJxoKshsUZBwpCM0yQP1mKRE85EBTCQzf0VkgCUm2qRWMCG48ycvQuOs7Bq+Oy9WrrM48nAIR1ACFy6gArdQgzoQeIRneIU368l6sd6tj1lrzspm9uGPrM8fNyCUFw==</latexit><latexit sha1_base64="TU5zZde4qbCx4WoxFSYbids+sH8=">AAACAnicbZDLSsNAFIZP6q3WW9SVuBksQgUpiQgKbopd6LKCvUAbymQ6aYdOJmFmIpRQ3Pgqblwo4tancOfbOG2z0NYfBj7+cw5nzu/HnCntON9Wbml5ZXUtv17Y2Nza3rF39xoqSiShdRLxSLZ8rChngtY105y2Yklx6HPa9IfVSb35QKVikbjXo5h6Ie4LFjCCtbG69kGAOoqF6KZWKp+i8gm6QsZhAlW7dtEpO1OhRXAzKEKmWtf+6vQikoRUaMKxUm3XibWXYqkZ4XRc6CSKxpgMcZ+2DQocUuWl0xPG6Ng4PRRE0jyh0dT9PZHiUKlR6JvOEOuBmq9NzP9q7UQHl17KRJxoKshsUZBwpCM0yQP1mKRE85EBTCQzf0VkgCUm2qRWMCG48ycvQuOs7Bq+Oy9WrrM48nAIR1ACFy6gArdQgzoQeIRneIU368l6sd6tj1lrzspm9uGPrM8fNyCUFw==</latexit><latexit sha1_base64="TU5zZde4qbCx4WoxFSYbids+sH8=">AAACAnicbZDLSsNAFIZP6q3WW9SVuBksQgUpiQgKbopd6LKCvUAbymQ6aYdOJmFmIpRQ3Pgqblwo4tancOfbOG2z0NYfBj7+cw5nzu/HnCntON9Wbml5ZXUtv17Y2Nza3rF39xoqSiShdRLxSLZ8rChngtY105y2Yklx6HPa9IfVSb35QKVikbjXo5h6Ie4LFjCCtbG69kGAOoqF6KZWKp+i8gm6QsZhAlW7dtEpO1OhRXAzKEKmWtf+6vQikoRUaMKxUm3XibWXYqkZ4XRc6CSKxpgMcZ+2DQocUuWl0xPG6Ng4PRRE0jyh0dT9PZHiUKlR6JvOEOuBmq9NzP9q7UQHl17KRJxoKshsUZBwpCM0yQP1mKRE85EBTCQzf0VkgCUm2qRWMCG48ycvQuOs7Bq+Oy9WrrM48nAIR1ACFy6gArdQgzoQeIRneIU368l6sd6tj1lrzspm9uGPrM8fNyCUFw==</latexit><latexit sha1_base64="TU5zZde4qbCx4WoxFSYbids+sH8=">AAACAnicbZDLSsNAFIZP6q3WW9SVuBksQgUpiQgKbopd6LKCvUAbymQ6aYdOJmFmIpRQ3Pgqblwo4tancOfbOG2z0NYfBj7+cw5nzu/HnCntON9Wbml5ZXUtv17Y2Nza3rF39xoqSiShdRLxSLZ8rChngtY105y2Yklx6HPa9IfVSb35QKVikbjXo5h6Ie4LFjCCtbG69kGAOoqF6KZWKp+i8gm6QsZhAlW7dtEpO1OhRXAzKEKmWtf+6vQikoRUaMKxUm3XibWXYqkZ4XRc6CSKxpgMcZ+2DQocUuWl0xPG6Ng4PRRE0jyh0dT9PZHiUKlR6JvOEOuBmq9NzP9q7UQHl17KRJxoKshsUZBwpCM0yQP1mKRE85EBTCQzf0VkgCUm2qRWMCG48ycvQuOs7Bq+Oy9WrrM48nAIR1ACFy6gArdQgzoQeIRneIU368l6sd6tj1lrzspm9uGPrM8fNyCUFw==</latexit>

Page 13: Horseshoe Priors for Bayesian Neural Networks...Horseshoe Priors for Bayesian Neural Networks Soumya Ghosh Jiayu Yao Finale Doshi-Velez IBM Research MIT-IBM Watson AI lab Harvard Harvard

!13

Gaussian Processes Bayesian Neural Networks

f(x) � GP(m(x), K(x, x�))<latexit sha1_base64="xFl22ReGOlQcU3Yv8/UZt4O7oq8=">AAACDHicbVDLSgMxFM34rPVVdekmWMQWSpkRQZdFFwpuKtgHdIaSSTNtaDIzJHekZegHuPFX3LhQxK0f4M6/MX0stPVA4HDOudzc48eCa7Dtb2tpeWV1bT2zkd3c2t7Zze3t13WUKMpqNBKRavpEM8FDVgMOgjVjxYj0BWv4/aux33hgSvMovIdhzDxJuiEPOCVgpHYuHxQGRexqLrELbADpdXVUkEYr4dvCoIQHJ8WiSdllewK8SJwZyaMZqu3cl9uJaCJZCFQQrVuOHYOXEgWcCjbKuolmMaF90mUtQ0MimfbSyTEjfGyUDg4iZV4IeKL+nkiJ1HoofZOUBHp63huL/3mtBIILL+VhnAAL6XRRkAgMER43gztcMQpiaAihipu/YtojilAw/WVNCc78yYukflp2DL87y1cuZ3Vk0CE6QgXkoHNUQTeoimqIokf0jF7Rm/VkvVjv1sc0umTNZg7QH1ifP2RlmJ8=</latexit><latexit sha1_base64="xFl22ReGOlQcU3Yv8/UZt4O7oq8=">AAACDHicbVDLSgMxFM34rPVVdekmWMQWSpkRQZdFFwpuKtgHdIaSSTNtaDIzJHekZegHuPFX3LhQxK0f4M6/MX0stPVA4HDOudzc48eCa7Dtb2tpeWV1bT2zkd3c2t7Zze3t13WUKMpqNBKRavpEM8FDVgMOgjVjxYj0BWv4/aux33hgSvMovIdhzDxJuiEPOCVgpHYuHxQGRexqLrELbADpdXVUkEYr4dvCoIQHJ8WiSdllewK8SJwZyaMZqu3cl9uJaCJZCFQQrVuOHYOXEgWcCjbKuolmMaF90mUtQ0MimfbSyTEjfGyUDg4iZV4IeKL+nkiJ1HoofZOUBHp63huL/3mtBIILL+VhnAAL6XRRkAgMER43gztcMQpiaAihipu/YtojilAw/WVNCc78yYukflp2DL87y1cuZ3Vk0CE6QgXkoHNUQTeoimqIokf0jF7Rm/VkvVjv1sc0umTNZg7QH1ifP2RlmJ8=</latexit><latexit sha1_base64="xFl22ReGOlQcU3Yv8/UZt4O7oq8=">AAACDHicbVDLSgMxFM34rPVVdekmWMQWSpkRQZdFFwpuKtgHdIaSSTNtaDIzJHekZegHuPFX3LhQxK0f4M6/MX0stPVA4HDOudzc48eCa7Dtb2tpeWV1bT2zkd3c2t7Zze3t13WUKMpqNBKRavpEM8FDVgMOgjVjxYj0BWv4/aux33hgSvMovIdhzDxJuiEPOCVgpHYuHxQGRexqLrELbADpdXVUkEYr4dvCoIQHJ8WiSdllewK8SJwZyaMZqu3cl9uJaCJZCFQQrVuOHYOXEgWcCjbKuolmMaF90mUtQ0MimfbSyTEjfGyUDg4iZV4IeKL+nkiJ1HoofZOUBHp63huL/3mtBIILL+VhnAAL6XRRkAgMER43gztcMQpiaAihipu/YtojilAw/WVNCc78yYukflp2DL87y1cuZ3Vk0CE6QgXkoHNUQTeoimqIokf0jF7Rm/VkvVjv1sc0umTNZg7QH1ifP2RlmJ8=</latexit><latexit sha1_base64="xFl22ReGOlQcU3Yv8/UZt4O7oq8=">AAACDHicbVDLSgMxFM34rPVVdekmWMQWSpkRQZdFFwpuKtgHdIaSSTNtaDIzJHekZegHuPFX3LhQxK0f4M6/MX0stPVA4HDOudzc48eCa7Dtb2tpeWV1bT2zkd3c2t7Zze3t13WUKMpqNBKRavpEM8FDVgMOgjVjxYj0BWv4/aux33hgSvMovIdhzDxJuiEPOCVgpHYuHxQGRexqLrELbADpdXVUkEYr4dvCoIQHJ8WiSdllewK8SJwZyaMZqu3cl9uJaCJZCFQQrVuOHYOXEgWcCjbKuolmMaF90mUtQ0MimfbSyTEjfGyUDg4iZV4IeKL+nkiJ1HoofZOUBHp63huL/3mtBIILL+VhnAAL6XRRkAgMER43gztcMQpiaAihipu/YtojilAw/WVNCc78yYukflp2DL87y1cuZ3Vk0CE6QgXkoHNUQTeoimqIokf0jF7Rm/VkvVjv1sc0umTNZg7QH1ifP2RlmJ8=</latexit>

f(x) � hW(x)<latexit sha1_base64="1qSafpd3upOjzSIVuy/6/h3+F9A=">AAACBXicbZC7TsMwFIYdrqXcAowwWFRIZakShARjBQtjkehFaqLIcZ3Wqu1EtoOooiwsvAoLAwix8g5svA1OmwFafsnSp/+cI5/zhwmjSjvOt7W0vLK6tl7ZqG5ube/s2nv7HRWnEpM2jlkseyFShFFB2ppqRnqJJIiHjHTD8XVR794TqWgs7vQkIT5HQ0EjipE2VmAfRfWHU+gpyuEoyDyO9AgjlnXz3PiBXXMazlRwEdwSaqBUK7C/vEGMU06Exgwp1XedRPsZkppiRvKqlyqSIDxGQ9I3KBAnys+mV+TwxDgDGMXSPKHh1P09kSGu1ISHprNYU83XCvO/Wj/V0aWfUZGkmgg8+yhKGdQxLCKBAyoJ1mxiAGFJza4Qj5BEWJvgqiYEd/7kReicNVzDt+e15lUZRwUcgmNQBy64AE1wA1qgDTB4BM/gFbxZT9aL9W59zFqXrHLmAPyR9fkDbzaX5A==</latexit><latexit sha1_base64="1qSafpd3upOjzSIVuy/6/h3+F9A=">AAACBXicbZC7TsMwFIYdrqXcAowwWFRIZakShARjBQtjkehFaqLIcZ3Wqu1EtoOooiwsvAoLAwix8g5svA1OmwFafsnSp/+cI5/zhwmjSjvOt7W0vLK6tl7ZqG5ube/s2nv7HRWnEpM2jlkseyFShFFB2ppqRnqJJIiHjHTD8XVR794TqWgs7vQkIT5HQ0EjipE2VmAfRfWHU+gpyuEoyDyO9AgjlnXz3PiBXXMazlRwEdwSaqBUK7C/vEGMU06Exgwp1XedRPsZkppiRvKqlyqSIDxGQ9I3KBAnys+mV+TwxDgDGMXSPKHh1P09kSGu1ISHprNYU83XCvO/Wj/V0aWfUZGkmgg8+yhKGdQxLCKBAyoJ1mxiAGFJza4Qj5BEWJvgqiYEd/7kReicNVzDt+e15lUZRwUcgmNQBy64AE1wA1qgDTB4BM/gFbxZT9aL9W59zFqXrHLmAPyR9fkDbzaX5A==</latexit><latexit sha1_base64="1qSafpd3upOjzSIVuy/6/h3+F9A=">AAACBXicbZC7TsMwFIYdrqXcAowwWFRIZakShARjBQtjkehFaqLIcZ3Wqu1EtoOooiwsvAoLAwix8g5svA1OmwFafsnSp/+cI5/zhwmjSjvOt7W0vLK6tl7ZqG5ube/s2nv7HRWnEpM2jlkseyFShFFB2ppqRnqJJIiHjHTD8XVR794TqWgs7vQkIT5HQ0EjipE2VmAfRfWHU+gpyuEoyDyO9AgjlnXz3PiBXXMazlRwEdwSaqBUK7C/vEGMU06Exgwp1XedRPsZkppiRvKqlyqSIDxGQ9I3KBAnys+mV+TwxDgDGMXSPKHh1P09kSGu1ISHprNYU83XCvO/Wj/V0aWfUZGkmgg8+yhKGdQxLCKBAyoJ1mxiAGFJza4Qj5BEWJvgqiYEd/7kReicNVzDt+e15lUZRwUcgmNQBy64AE1wA1qgDTB4BM/gFbxZT9aL9W59zFqXrHLmAPyR9fkDbzaX5A==</latexit><latexit sha1_base64="1qSafpd3upOjzSIVuy/6/h3+F9A=">AAACBXicbZC7TsMwFIYdrqXcAowwWFRIZakShARjBQtjkehFaqLIcZ3Wqu1EtoOooiwsvAoLAwix8g5svA1OmwFafsnSp/+cI5/zhwmjSjvOt7W0vLK6tl7ZqG5ube/s2nv7HRWnEpM2jlkseyFShFFB2ppqRnqJJIiHjHTD8XVR794TqWgs7vQkIT5HQ0EjipE2VmAfRfWHU+gpyuEoyDyO9AgjlnXz3PiBXXMazlRwEdwSaqBUK7C/vEGMU06Exgwp1XedRPsZkppiRvKqlyqSIDxGQ9I3KBAnys+mV+TwxDgDGMXSPKHh1P09kSGu1ISHprNYU83XCvO/Wj/V0aWfUZGkmgg8+yhKGdQxLCKBAyoJ1mxiAGFJza4Qj5BEWJvgqiYEd/7kReicNVzDt+e15lUZRwUcgmNQBy64AE1wA1qgDTB4BM/gFbxZT9aL9W59zFqXrHLmAPyR9fkDbzaX5A==</latexit>

m(x)<latexit sha1_base64="lY+c2Pwi7MEiEZn55TuyyAH5v5o=">AAAB63icbZDNSgMxFIXv1L9a/6ou3QSLUDdlRgRdFt24rOC0hXYomTTThiaZIcmIZegruHGhiFtfyJ1vY6adhbYeCHycey+594QJZ9q47rdTWlvf2Nwqb1d2dvf2D6qHR20dp4pQn8Q8Vt0Qa8qZpL5hhtNuoigWIaedcHKb1zuPVGkWywczTWgg8EiyiBFsckvUn84H1ZrbcOdCq+AVUINCrUH1qz+MSSqoNIRjrXuem5ggw8owwums0k81TTCZ4BHtWZRYUB1k811n6Mw6QxTFyj5p0Nz9PZFhofVUhLZTYDPWy7Xc/K/WS010HWRMJqmhkiw+ilKOTIzyw9GQKUoMn1rARDG7KyJjrDAxNp6KDcFbPnkV2hcNz/L9Za15U8RRhhM4hTp4cAVNuIMW+EBgDM/wCm+OcF6cd+dj0Vpyiplj+CPn8wd3Eo3Y</latexit><latexit sha1_base64="lY+c2Pwi7MEiEZn55TuyyAH5v5o=">AAAB63icbZDNSgMxFIXv1L9a/6ou3QSLUDdlRgRdFt24rOC0hXYomTTThiaZIcmIZegruHGhiFtfyJ1vY6adhbYeCHycey+594QJZ9q47rdTWlvf2Nwqb1d2dvf2D6qHR20dp4pQn8Q8Vt0Qa8qZpL5hhtNuoigWIaedcHKb1zuPVGkWywczTWgg8EiyiBFsckvUn84H1ZrbcOdCq+AVUINCrUH1qz+MSSqoNIRjrXuem5ggw8owwums0k81TTCZ4BHtWZRYUB1k811n6Mw6QxTFyj5p0Nz9PZFhofVUhLZTYDPWy7Xc/K/WS010HWRMJqmhkiw+ilKOTIzyw9GQKUoMn1rARDG7KyJjrDAxNp6KDcFbPnkV2hcNz/L9Za15U8RRhhM4hTp4cAVNuIMW+EBgDM/wCm+OcF6cd+dj0Vpyiplj+CPn8wd3Eo3Y</latexit><latexit sha1_base64="lY+c2Pwi7MEiEZn55TuyyAH5v5o=">AAAB63icbZDNSgMxFIXv1L9a/6ou3QSLUDdlRgRdFt24rOC0hXYomTTThiaZIcmIZegruHGhiFtfyJ1vY6adhbYeCHycey+594QJZ9q47rdTWlvf2Nwqb1d2dvf2D6qHR20dp4pQn8Q8Vt0Qa8qZpL5hhtNuoigWIaedcHKb1zuPVGkWywczTWgg8EiyiBFsckvUn84H1ZrbcOdCq+AVUINCrUH1qz+MSSqoNIRjrXuem5ggw8owwums0k81TTCZ4BHtWZRYUB1k811n6Mw6QxTFyj5p0Nz9PZFhofVUhLZTYDPWy7Xc/K/WS010HWRMJqmhkiw+ilKOTIzyw9GQKUoMn1rARDG7KyJjrDAxNp6KDcFbPnkV2hcNz/L9Za15U8RRhhM4hTp4cAVNuIMW+EBgDM/wCm+OcF6cd+dj0Vpyiplj+CPn8wd3Eo3Y</latexit><latexit sha1_base64="lY+c2Pwi7MEiEZn55TuyyAH5v5o=">AAAB63icbZDNSgMxFIXv1L9a/6ou3QSLUDdlRgRdFt24rOC0hXYomTTThiaZIcmIZegruHGhiFtfyJ1vY6adhbYeCHycey+594QJZ9q47rdTWlvf2Nwqb1d2dvf2D6qHR20dp4pQn8Q8Vt0Qa8qZpL5hhtNuoigWIaedcHKb1zuPVGkWywczTWgg8EiyiBFsckvUn84H1ZrbcOdCq+AVUINCrUH1qz+MSSqoNIRjrXuem5ggw8owwums0k81TTCZ4BHtWZRYUB1k811n6Mw6QxTFyj5p0Nz9PZFhofVUhLZTYDPWy7Xc/K/WS010HWRMJqmhkiw+ilKOTIzyw9GQKUoMn1rARDG7KyJjrDAxNp6KDcFbPnkV2hcNz/L9Za15U8RRhhM4hTp4cAVNuIMW+EBgDM/wCm+OcF6cd+dj0Vpyiplj+CPn8wd3Eo3Y</latexit>

K(x, x�)<latexit sha1_base64="miRB8bWTE7oNGLg/k4DhFbUWb5o=">AAAB73icbZDLSgMxFIbPeK31VnXpJljEClJmRNBl0Y3gpoK9QDuUTJppQzPJmGSkZehLuHGhiFtfx51vY9rOQlt/CHz85xxyzh/EnGnjut/O0vLK6tp6biO/ubW9s1vY269rmShCa0RyqZoB1pQzQWuGGU6bsaI4CjhtBIObSb3xRJVmUjyYUUz9CPcECxnBxlrNu9LwDA1PTjuFolt2p0KL4GVQhEzVTuGr3ZUkiagwhGOtW54bGz/FyjDC6TjfTjSNMRngHm1ZFDii2k+n+47RsXW6KJTKPmHQ1P09keJI61EU2M4Im76er03M/2qtxIRXfspEnBgqyOyjMOHISDQ5HnWZosTwkQVMFLO7ItLHChNjI8rbELz5kxehfl72LN9fFCvXWRw5OIQjKIEHl1CBW6hCDQhweIZXeHMenRfn3fmYtS452cwB/JHz+QNA9Y7J</latexit><latexit sha1_base64="miRB8bWTE7oNGLg/k4DhFbUWb5o=">AAAB73icbZDLSgMxFIbPeK31VnXpJljEClJmRNBl0Y3gpoK9QDuUTJppQzPJmGSkZehLuHGhiFtfx51vY9rOQlt/CHz85xxyzh/EnGnjut/O0vLK6tp6biO/ubW9s1vY269rmShCa0RyqZoB1pQzQWuGGU6bsaI4CjhtBIObSb3xRJVmUjyYUUz9CPcECxnBxlrNu9LwDA1PTjuFolt2p0KL4GVQhEzVTuGr3ZUkiagwhGOtW54bGz/FyjDC6TjfTjSNMRngHm1ZFDii2k+n+47RsXW6KJTKPmHQ1P09keJI61EU2M4Im76er03M/2qtxIRXfspEnBgqyOyjMOHISDQ5HnWZosTwkQVMFLO7ItLHChNjI8rbELz5kxehfl72LN9fFCvXWRw5OIQjKIEHl1CBW6hCDQhweIZXeHMenRfn3fmYtS452cwB/JHz+QNA9Y7J</latexit><latexit sha1_base64="miRB8bWTE7oNGLg/k4DhFbUWb5o=">AAAB73icbZDLSgMxFIbPeK31VnXpJljEClJmRNBl0Y3gpoK9QDuUTJppQzPJmGSkZehLuHGhiFtfx51vY9rOQlt/CHz85xxyzh/EnGnjut/O0vLK6tp6biO/ubW9s1vY269rmShCa0RyqZoB1pQzQWuGGU6bsaI4CjhtBIObSb3xRJVmUjyYUUz9CPcECxnBxlrNu9LwDA1PTjuFolt2p0KL4GVQhEzVTuGr3ZUkiagwhGOtW54bGz/FyjDC6TjfTjSNMRngHm1ZFDii2k+n+47RsXW6KJTKPmHQ1P09keJI61EU2M4Im76er03M/2qtxIRXfspEnBgqyOyjMOHISDQ5HnWZosTwkQVMFLO7ItLHChNjI8rbELz5kxehfl72LN9fFCvXWRw5OIQjKIEHl1CBW6hCDQhweIZXeHMenRfn3fmYtS452cwB/JHz+QNA9Y7J</latexit><latexit sha1_base64="miRB8bWTE7oNGLg/k4DhFbUWb5o=">AAAB73icbZDLSgMxFIbPeK31VnXpJljEClJmRNBl0Y3gpoK9QDuUTJppQzPJmGSkZehLuHGhiFtfx51vY9rOQlt/CHz85xxyzh/EnGnjut/O0vLK6tp6biO/ubW9s1vY269rmShCa0RyqZoB1pQzQWuGGU6bsaI4CjhtBIObSb3xRJVmUjyYUUz9CPcECxnBxlrNu9LwDA1PTjuFolt2p0KL4GVQhEzVTuGr3ZUkiagwhGOtW54bGz/FyjDC6TjfTjSNMRngHm1ZFDii2k+n+47RsXW6KJTKPmHQ1P09keJI61EU2M4Im76er03M/2qtxIRXfspEnBgqyOyjMOHISDQ5HnWZosTwkQVMFLO7ItLHChNjI8rbELz5kxehfl72LN9fFCvXWRw5OIQjKIEHl1CBW6hCDQhweIZXeHMenRfn3fmYtS452cwB/JHz+QNA9Y7J</latexit>

Completely specified by:

Intuitive, well understood parameterization

Need to specify:

h<latexit sha1_base64="qgYxBTCFiJYJlr1gvWqOWol7o+Y=">AAAB6HicbZBNS8NAEIYn9avWr6pHL4tF8FQSEeqx6MVjC/YD2lA220m7drMJuxuhhP4CLx4U8epP8ua/cdvmoK0vLDy8M8POvEEiuDau++0UNja3tneKu6W9/YPDo/LxSVvHqWLYYrGIVTegGgWX2DLcCOwmCmkUCOwEk7t5vfOESvNYPphpgn5ER5KHnFFjreZ4UK64VXchsg5eDhXI1RiUv/rDmKURSsME1brnuYnxM6oMZwJnpX6qMaFsQkfYsyhphNrPFovOyIV1hiSMlX3SkIX7eyKjkdbTKLCdETVjvVqbm//VeqkJb/yMyyQ1KNnyozAVxMRkfjUZcoXMiKkFyhS3uxI2pooyY7Mp2RC81ZPXoX1V9Sw3ryv12zyOIpzBOVyCBzWowz00oAUMEJ7hFd6cR+fFeXc+lq0FJ585hT9yPn8AzfuM7A==</latexit><latexit sha1_base64="qgYxBTCFiJYJlr1gvWqOWol7o+Y=">AAAB6HicbZBNS8NAEIYn9avWr6pHL4tF8FQSEeqx6MVjC/YD2lA220m7drMJuxuhhP4CLx4U8epP8ua/cdvmoK0vLDy8M8POvEEiuDau++0UNja3tneKu6W9/YPDo/LxSVvHqWLYYrGIVTegGgWX2DLcCOwmCmkUCOwEk7t5vfOESvNYPphpgn5ER5KHnFFjreZ4UK64VXchsg5eDhXI1RiUv/rDmKURSsME1brnuYnxM6oMZwJnpX6qMaFsQkfYsyhphNrPFovOyIV1hiSMlX3SkIX7eyKjkdbTKLCdETVjvVqbm//VeqkJb/yMyyQ1KNnyozAVxMRkfjUZcoXMiKkFyhS3uxI2pooyY7Mp2RC81ZPXoX1V9Sw3ryv12zyOIpzBOVyCBzWowz00oAUMEJ7hFd6cR+fFeXc+lq0FJ585hT9yPn8AzfuM7A==</latexit><latexit sha1_base64="qgYxBTCFiJYJlr1gvWqOWol7o+Y=">AAAB6HicbZBNS8NAEIYn9avWr6pHL4tF8FQSEeqx6MVjC/YD2lA220m7drMJuxuhhP4CLx4U8epP8ua/cdvmoK0vLDy8M8POvEEiuDau++0UNja3tneKu6W9/YPDo/LxSVvHqWLYYrGIVTegGgWX2DLcCOwmCmkUCOwEk7t5vfOESvNYPphpgn5ER5KHnFFjreZ4UK64VXchsg5eDhXI1RiUv/rDmKURSsME1brnuYnxM6oMZwJnpX6qMaFsQkfYsyhphNrPFovOyIV1hiSMlX3SkIX7eyKjkdbTKLCdETVjvVqbm//VeqkJb/yMyyQ1KNnyozAVxMRkfjUZcoXMiKkFyhS3uxI2pooyY7Mp2RC81ZPXoX1V9Sw3ryv12zyOIpzBOVyCBzWowz00oAUMEJ7hFd6cR+fFeXc+lq0FJ585hT9yPn8AzfuM7A==</latexit><latexit sha1_base64="qgYxBTCFiJYJlr1gvWqOWol7o+Y=">AAAB6HicbZBNS8NAEIYn9avWr6pHL4tF8FQSEeqx6MVjC/YD2lA220m7drMJuxuhhP4CLx4U8epP8ua/cdvmoK0vLDy8M8POvEEiuDau++0UNja3tneKu6W9/YPDo/LxSVvHqWLYYrGIVTegGgWX2DLcCOwmCmkUCOwEk7t5vfOESvNYPphpgn5ER5KHnFFjreZ4UK64VXchsg5eDhXI1RiUv/rDmKURSsME1brnuYnxM6oMZwJnpX6qMaFsQkfYsyhphNrPFovOyIV1hiSMlX3SkIX7eyKjkdbTKLCdETVjvVqbm//VeqkJb/yMyyQ1KNnyozAVxMRkfjUZcoXMiKkFyhS3uxI2pooyY7Mp2RC81ZPXoX1V9Sw3ryv12zyOIpzBOVyCBzWowz00oAUMEJ7hFd6cR+fFeXc+lq0FJ585hT9yPn8AzfuM7A==</latexit>

architecture non-linearity

p(W | �)<latexit sha1_base64="l3/kEntRR0JR3ojd4k4pxcsMyGg=">AAACBHicbVDLSsNAFL2pr1pfUZfdDBahbkoigi6LblxWsA9oQplMpu3QySTMTIQSunDjr7hxoYhbP8Kdf+OkzUJbDwwczrmHufcECWdKO863VVpb39jcKm9Xdnb39g/sw6OOilNJaJvEPJa9ACvKmaBtzTSnvURSHAWcdoPJTe53H6hULBb3eppQP8IjwYaMYG2kgV1N6l6E9ZhgnnVnyItYiDxu8iE+G9g1p+HMgVaJW5AaFGgN7C8vjEkaUaEJx0r1XSfRfoalZoTTWcVLFU0wmeAR7RsqcESVn82PmKFTo4RoGEvzhEZz9Xciw5FS0ygwk/nCatnLxf+8fqqHV37GRJJqKsjio2HKkY5R3ggKmaRE86khmEhmdkVkjCUm2vRWMSW4yyevks55wzX87qLWvC7qKEMVTqAOLlxCE26hBW0g8AjP8Apv1pP1Yr1bH4vRklVkjuEPrM8f3AuXkg==</latexit><latexit sha1_base64="l3/kEntRR0JR3ojd4k4pxcsMyGg=">AAACBHicbVDLSsNAFL2pr1pfUZfdDBahbkoigi6LblxWsA9oQplMpu3QySTMTIQSunDjr7hxoYhbP8Kdf+OkzUJbDwwczrmHufcECWdKO863VVpb39jcKm9Xdnb39g/sw6OOilNJaJvEPJa9ACvKmaBtzTSnvURSHAWcdoPJTe53H6hULBb3eppQP8IjwYaMYG2kgV1N6l6E9ZhgnnVnyItYiDxu8iE+G9g1p+HMgVaJW5AaFGgN7C8vjEkaUaEJx0r1XSfRfoalZoTTWcVLFU0wmeAR7RsqcESVn82PmKFTo4RoGEvzhEZz9Xciw5FS0ygwk/nCatnLxf+8fqqHV37GRJJqKsjio2HKkY5R3ggKmaRE86khmEhmdkVkjCUm2vRWMSW4yyevks55wzX87qLWvC7qKEMVTqAOLlxCE26hBW0g8AjP8Apv1pP1Yr1bH4vRklVkjuEPrM8f3AuXkg==</latexit><latexit sha1_base64="l3/kEntRR0JR3ojd4k4pxcsMyGg=">AAACBHicbVDLSsNAFL2pr1pfUZfdDBahbkoigi6LblxWsA9oQplMpu3QySTMTIQSunDjr7hxoYhbP8Kdf+OkzUJbDwwczrmHufcECWdKO863VVpb39jcKm9Xdnb39g/sw6OOilNJaJvEPJa9ACvKmaBtzTSnvURSHAWcdoPJTe53H6hULBb3eppQP8IjwYaMYG2kgV1N6l6E9ZhgnnVnyItYiDxu8iE+G9g1p+HMgVaJW5AaFGgN7C8vjEkaUaEJx0r1XSfRfoalZoTTWcVLFU0wmeAR7RsqcESVn82PmKFTo4RoGEvzhEZz9Xciw5FS0ygwk/nCatnLxf+8fqqHV37GRJJqKsjio2HKkY5R3ggKmaRE86khmEhmdkVkjCUm2vRWMSW4yyevks55wzX87qLWvC7qKEMVTqAOLlxCE26hBW0g8AjP8Apv1pP1Yr1bH4vRklVkjuEPrM8f3AuXkg==</latexit><latexit sha1_base64="l3/kEntRR0JR3ojd4k4pxcsMyGg=">AAACBHicbVDLSsNAFL2pr1pfUZfdDBahbkoigi6LblxWsA9oQplMpu3QySTMTIQSunDjr7hxoYhbP8Kdf+OkzUJbDwwczrmHufcECWdKO863VVpb39jcKm9Xdnb39g/sw6OOilNJaJvEPJa9ACvKmaBtzTSnvURSHAWcdoPJTe53H6hULBb3eppQP8IjwYaMYG2kgV1N6l6E9ZhgnnVnyItYiDxu8iE+G9g1p+HMgVaJW5AaFGgN7C8vjEkaUaEJx0r1XSfRfoalZoTTWcVLFU0wmeAR7RsqcESVn82PmKFTo4RoGEvzhEZz9Xciw5FS0ygwk/nCatnLxf+8fqqHV37GRJJqKsjio2HKkY5R3ggKmaRE86khmEhmdkVkjCUm2vRWMSW4yyevks55wzX87qLWvC7qKEMVTqAOLlxCE26hBW0g8AjP8Apv1pP1Yr1bH4vRklVkjuEPrM8f3AuXkg==</latexit>

prior on weights

Implied distribution on functions is poorly understood

Page 14: Horseshoe Priors for Bayesian Neural Networks...Horseshoe Priors for Bayesian Neural Networks Soumya Ghosh Jiayu Yao Finale Doshi-Velez IBM Research MIT-IBM Watson AI lab Harvard Harvard

Predictive Uncertainties?

Single layer network, with prior:

(Same results across many initialization strategies)

What is happening?

W � N (0, I)<latexit sha1_base64="ACN79W3NN9SiNlCY7zZvbl+vEIs=">AAACHXicbZDLSsNAFIYnXmu9RV26GSxCBSmJFHRZdKMbqWAv0IQymU7aoZNJmJkIJeRF3Pgqblwo4sKN+DZO0ija+sPAx3/OYc75vYhRqSzr01hYXFpeWS2tldc3Nre2zZ3dtgxjgUkLhywUXQ9JwignLUUVI91IEBR4jHS88UVW79wRIWnIb9UkIm6Ahpz6FCOlrb5ZdwKkRhixpJNCR9IA/hjXaTVnz0+s9Bh+81V61DcrVs3KBefBLqACCjX75rszCHEcEK4wQ1L2bCtSboKEopiRtOzEkkQIj9GQ9DRyFBDpJvl1KTzUzgD6odCPK5i7vycSFEg5CTzdma0oZ2uZ+V+tFyv/zE0oj2JFOJ5+5McMqhBmUcEBFQQrNtGAsKB6V4hHSCCsdKBlHYI9e/I8tE9qtlWzb+qVxnkRRwnsgwNQBTY4BQ1wCZqgBTC4B4/gGbwYD8aT8Wq8TVsXjGJmD/yR8fEFpt2iOw==</latexit><latexit sha1_base64="ACN79W3NN9SiNlCY7zZvbl+vEIs=">AAACHXicbZDLSsNAFIYnXmu9RV26GSxCBSmJFHRZdKMbqWAv0IQymU7aoZNJmJkIJeRF3Pgqblwo4sKN+DZO0ija+sPAx3/OYc75vYhRqSzr01hYXFpeWS2tldc3Nre2zZ3dtgxjgUkLhywUXQ9JwignLUUVI91IEBR4jHS88UVW79wRIWnIb9UkIm6Ahpz6FCOlrb5ZdwKkRhixpJNCR9IA/hjXaTVnz0+s9Bh+81V61DcrVs3KBefBLqACCjX75rszCHEcEK4wQ1L2bCtSboKEopiRtOzEkkQIj9GQ9DRyFBDpJvl1KTzUzgD6odCPK5i7vycSFEg5CTzdma0oZ2uZ+V+tFyv/zE0oj2JFOJ5+5McMqhBmUcEBFQQrNtGAsKB6V4hHSCCsdKBlHYI9e/I8tE9qtlWzb+qVxnkRRwnsgwNQBTY4BQ1wCZqgBTC4B4/gGbwYD8aT8Wq8TVsXjGJmD/yR8fEFpt2iOw==</latexit><latexit sha1_base64="ACN79W3NN9SiNlCY7zZvbl+vEIs=">AAACHXicbZDLSsNAFIYnXmu9RV26GSxCBSmJFHRZdKMbqWAv0IQymU7aoZNJmJkIJeRF3Pgqblwo4sKN+DZO0ija+sPAx3/OYc75vYhRqSzr01hYXFpeWS2tldc3Nre2zZ3dtgxjgUkLhywUXQ9JwignLUUVI91IEBR4jHS88UVW79wRIWnIb9UkIm6Ahpz6FCOlrb5ZdwKkRhixpJNCR9IA/hjXaTVnz0+s9Bh+81V61DcrVs3KBefBLqACCjX75rszCHEcEK4wQ1L2bCtSboKEopiRtOzEkkQIj9GQ9DRyFBDpJvl1KTzUzgD6odCPK5i7vycSFEg5CTzdma0oZ2uZ+V+tFyv/zE0oj2JFOJ5+5McMqhBmUcEBFQQrNtGAsKB6V4hHSCCsdKBlHYI9e/I8tE9qtlWzb+qVxnkRRwnsgwNQBTY4BQ1wCZqgBTC4B4/gGbwYD8aT8Wq8TVsXjGJmD/yR8fEFpt2iOw==</latexit><latexit sha1_base64="ACN79W3NN9SiNlCY7zZvbl+vEIs=">AAACHXicbZDLSsNAFIYnXmu9RV26GSxCBSmJFHRZdKMbqWAv0IQymU7aoZNJmJkIJeRF3Pgqblwo4sKN+DZO0ija+sPAx3/OYc75vYhRqSzr01hYXFpeWS2tldc3Nre2zZ3dtgxjgUkLhywUXQ9JwignLUUVI91IEBR4jHS88UVW79wRIWnIb9UkIm6Ahpz6FCOlrb5ZdwKkRhixpJNCR9IA/hjXaTVnz0+s9Bh+81V61DcrVs3KBefBLqACCjX75rszCHEcEK4wQ1L2bCtSboKEopiRtOzEkkQIj9GQ9DRyFBDpJvl1KTzUzgD6odCPK5i7vycSFEg5CTzdma0oZ2uZ+V+tFyv/zE0oj2JFOJ5+5McMqhBmUcEBFQQrNtGAsKB6V4hHSCCsdKBlHYI9e/I8tE9qtlWzb+qVxnkRRwnsgwNQBTY4BQ1wCZqgBTC4B4/gGbwYD8aT8Wq8TVsXjGJmD/yR8fEFpt2iOw==</latexit>

!14

Page 15: Horseshoe Priors for Bayesian Neural Networks...Horseshoe Priors for Bayesian Neural Networks Soumya Ghosh Jiayu Yao Finale Doshi-Velez IBM Research MIT-IBM Watson AI lab Harvard Harvard

Prior uncertainty

wj � N (0, �2w); b � N (0, �2

b )<latexit sha1_base64="KAfgq2qtdghwBAtJmW5k1oTkPxw=">AAACL3icbVBbS8MwGE29znmr+uhLcAgbyGiHoODLUBCfZIK7wFpLmmVbXNKWJHWMsn/ki39lLyKK+Oq/MN2G6LYDgcM55yPfd/yIUaks681YWl5ZXVvPbGQ3t7Z3ds29/ZoMY4FJFYcsFA0fScJoQKqKKkYakSCI+4zU/d5V6tefiJA0DO7VICIuR52AtilGSkueed33HqEjKYcOR6qLEUtuh3nrJNU6HD2UvH7hAkJ/UeY34hc8M2cVrTHgPLGnJAemqHjmyGmFOOYkUJghKZu2FSk3QUJRzMgw68SSRAj3UIc0NQ0QJ9JNxvcO4bFWWrAdCv0CBcfq34kEcSkH3NfJdGE566XiIq8Zq/a5m9AgihUJ8OSjdsygCmFaHmxRQbBiA00QFlTvCnEXCYSVrjirS7BnT54ntVLR1vzuNFe+nNaRAYfgCOSBDc5AGdyACqgCDJ7BCLyDD+PFeDU+ja9JdMmYzhyAfzC+fwA3OadR</latexit><latexit sha1_base64="KAfgq2qtdghwBAtJmW5k1oTkPxw=">AAACL3icbVBbS8MwGE29znmr+uhLcAgbyGiHoODLUBCfZIK7wFpLmmVbXNKWJHWMsn/ki39lLyKK+Oq/MN2G6LYDgcM55yPfd/yIUaks681YWl5ZXVvPbGQ3t7Z3ds29/ZoMY4FJFYcsFA0fScJoQKqKKkYakSCI+4zU/d5V6tefiJA0DO7VICIuR52AtilGSkueed33HqEjKYcOR6qLEUtuh3nrJNU6HD2UvH7hAkJ/UeY34hc8M2cVrTHgPLGnJAemqHjmyGmFOOYkUJghKZu2FSk3QUJRzMgw68SSRAj3UIc0NQ0QJ9JNxvcO4bFWWrAdCv0CBcfq34kEcSkH3NfJdGE566XiIq8Zq/a5m9AgihUJ8OSjdsygCmFaHmxRQbBiA00QFlTvCnEXCYSVrjirS7BnT54ntVLR1vzuNFe+nNaRAYfgCOSBDc5AGdyACqgCDJ7BCLyDD+PFeDU+ja9JdMmYzhyAfzC+fwA3OadR</latexit><latexit sha1_base64="KAfgq2qtdghwBAtJmW5k1oTkPxw=">AAACL3icbVBbS8MwGE29znmr+uhLcAgbyGiHoODLUBCfZIK7wFpLmmVbXNKWJHWMsn/ki39lLyKK+Oq/MN2G6LYDgcM55yPfd/yIUaks681YWl5ZXVvPbGQ3t7Z3ds29/ZoMY4FJFYcsFA0fScJoQKqKKkYakSCI+4zU/d5V6tefiJA0DO7VICIuR52AtilGSkueed33HqEjKYcOR6qLEUtuh3nrJNU6HD2UvH7hAkJ/UeY34hc8M2cVrTHgPLGnJAemqHjmyGmFOOYkUJghKZu2FSk3QUJRzMgw68SSRAj3UIc0NQ0QJ9JNxvcO4bFWWrAdCv0CBcfq34kEcSkH3NfJdGE566XiIq8Zq/a5m9AgihUJ8OSjdsygCmFaHmxRQbBiA00QFlTvCnEXCYSVrjirS7BnT54ntVLR1vzuNFe+nNaRAYfgCOSBDc5AGdyACqgCDJ7BCLyDD+PFeDU+ja9JdMmYzhyAfzC+fwA3OadR</latexit><latexit sha1_base64="KAfgq2qtdghwBAtJmW5k1oTkPxw=">AAACL3icbVBbS8MwGE29znmr+uhLcAgbyGiHoODLUBCfZIK7wFpLmmVbXNKWJHWMsn/ki39lLyKK+Oq/MN2G6LYDgcM55yPfd/yIUaks681YWl5ZXVvPbGQ3t7Z3ds29/ZoMY4FJFYcsFA0fScJoQKqKKkYakSCI+4zU/d5V6tefiJA0DO7VICIuR52AtilGSkueed33HqEjKYcOR6qLEUtuh3nrJNU6HD2UvH7hAkJ/UeY34hc8M2cVrTHgPLGnJAemqHjmyGmFOOYkUJghKZu2FSk3QUJRzMgw68SSRAj3UIc0NQ0QJ9JNxvcO4bFWWrAdCv0CBcfq34kEcSkH3NfJdGE566XiIq8Zq/a5m9AgihUJ8OSjdsygCmFaHmxRQbBiA00QFlTvCnEXCYSVrjirS7BnT54ntVLR1vzuNFe+nNaRAYfgCOSBDc5AGdyACqgCDJ7BCLyDD+PFeDU+ja9JdMmYzhyAfzC+fwA3OadR</latexit>

++

f(x) = b +J�

j=1

wj�(x; uj)<latexit sha1_base64="F19YSTSGZfbk19qglgh9199R8UE=">AAACEXicbZDLSsNAFIYnXmu9RV26GSxCilASERSkUHQjrirYCzQxTKaTdtrJhZmJtoS+ghtfxY0LRdy6c+fbOG2z0NYfBj7+cw5nzu/FjAppmt/awuLS8spqbi2/vrG5ta3v7NZFlHBMajhiEW96SBBGQ1KTVDLSjDlBgcdIw+tfjuuNe8IFjcJbOYyJE6BOSH2KkVSWqxu+MSjCMvTgEbRFErhpr2yN7q7hg9uDdtylxuAcJm6v6OoFs2ROBOfByqAAMlVd/ctuRzgJSCgxQ0K0LDOWToq4pJiRUd5OBIkR7qMOaSkMUUCEk04uGsFD5bShH3H1Qgkn7u+JFAVCDANPdQZIdsVsbWz+V2sl0j9zUhrGiSQhni7yEwZlBMfxwDblBEs2VIAwp+qvEHcRR1iqEPMqBGv25HmoH5csxTcnhcpFFkcO7IMDYAALnIIKuAJVUAMYPIJn8AretCftRXvXPqatC1o2swf+SPv8ASzvmrs=</latexit><latexit sha1_base64="F19YSTSGZfbk19qglgh9199R8UE=">AAACEXicbZDLSsNAFIYnXmu9RV26GSxCilASERSkUHQjrirYCzQxTKaTdtrJhZmJtoS+ghtfxY0LRdy6c+fbOG2z0NYfBj7+cw5nzu/FjAppmt/awuLS8spqbi2/vrG5ta3v7NZFlHBMajhiEW96SBBGQ1KTVDLSjDlBgcdIw+tfjuuNe8IFjcJbOYyJE6BOSH2KkVSWqxu+MSjCMvTgEbRFErhpr2yN7q7hg9uDdtylxuAcJm6v6OoFs2ROBOfByqAAMlVd/ctuRzgJSCgxQ0K0LDOWToq4pJiRUd5OBIkR7qMOaSkMUUCEk04uGsFD5bShH3H1Qgkn7u+JFAVCDANPdQZIdsVsbWz+V2sl0j9zUhrGiSQhni7yEwZlBMfxwDblBEs2VIAwp+qvEHcRR1iqEPMqBGv25HmoH5csxTcnhcpFFkcO7IMDYAALnIIKuAJVUAMYPIJn8AretCftRXvXPqatC1o2swf+SPv8ASzvmrs=</latexit><latexit sha1_base64="F19YSTSGZfbk19qglgh9199R8UE=">AAACEXicbZDLSsNAFIYnXmu9RV26GSxCilASERSkUHQjrirYCzQxTKaTdtrJhZmJtoS+ghtfxY0LRdy6c+fbOG2z0NYfBj7+cw5nzu/FjAppmt/awuLS8spqbi2/vrG5ta3v7NZFlHBMajhiEW96SBBGQ1KTVDLSjDlBgcdIw+tfjuuNe8IFjcJbOYyJE6BOSH2KkVSWqxu+MSjCMvTgEbRFErhpr2yN7q7hg9uDdtylxuAcJm6v6OoFs2ROBOfByqAAMlVd/ctuRzgJSCgxQ0K0LDOWToq4pJiRUd5OBIkR7qMOaSkMUUCEk04uGsFD5bShH3H1Qgkn7u+JFAVCDANPdQZIdsVsbWz+V2sl0j9zUhrGiSQhni7yEwZlBMfxwDblBEs2VIAwp+qvEHcRR1iqEPMqBGv25HmoH5csxTcnhcpFFkcO7IMDYAALnIIKuAJVUAMYPIJn8AretCftRXvXPqatC1o2swf+SPv8ASzvmrs=</latexit><latexit sha1_base64="F19YSTSGZfbk19qglgh9199R8UE=">AAACEXicbZDLSsNAFIYnXmu9RV26GSxCilASERSkUHQjrirYCzQxTKaTdtrJhZmJtoS+ghtfxY0LRdy6c+fbOG2z0NYfBj7+cw5nzu/FjAppmt/awuLS8spqbi2/vrG5ta3v7NZFlHBMajhiEW96SBBGQ1KTVDLSjDlBgcdIw+tfjuuNe8IFjcJbOYyJE6BOSH2KkVSWqxu+MSjCMvTgEbRFErhpr2yN7q7hg9uDdtylxuAcJm6v6OoFs2ROBOfByqAAMlVd/ctuRzgJSCgxQ0K0LDOWToq4pJiRUd5OBIkR7qMOaSkMUUCEk04uGsFD5bShH3H1Qgkn7u+JFAVCDANPdQZIdsVsbWz+V2sl0j9zUhrGiSQhni7yEwZlBMfxwDblBEs2VIAwp+qvEHcRR1iqEPMqBGv25HmoH5csxTcnhcpFFkcO7IMDYAALnIIKuAJVUAMYPIJn8AretCftRXvXPqatC1o2swf+SPv8ASzvmrs=</latexit>

wj<latexit sha1_base64="HwPGbp1zTlZgN+Y1A19X6nusNsU=">AAAB6nicbZBNS8NAEIYn9avWr6pHL4tF8FQSEfRY9OKxov2ANpTNdtKu3WzC7kYpoT/BiwdFvPqLvPlv3LY5aOsLCw/vzLAzb5AIro3rfjuFldW19Y3iZmlre2d3r7x/0NRxqhg2WCxi1Q6oRsElNgw3AtuJQhoFAlvB6Hpabz2i0jyW92acoB/RgeQhZ9RY6+6p99ArV9yqOxNZBi+HCuSq98pf3X7M0gilYYJq3fHcxPgZVYYzgZNSN9WYUDaiA+xYlDRC7WezVSfkxDp9EsbKPmnIzP09kdFI63EU2M6ImqFerE3N/2qd1ISXfsZlkhqUbP5RmApiYjK9m/S5QmbE2AJlittdCRtSRZmx6ZRsCN7iycvQPKt6lm/PK7WrPI4iHMExnIIHF1CDG6hDAxgM4Ble4c0Rzovz7nzMWwtOPnMIf+R8/gBgkI3Y</latexit><latexit sha1_base64="HwPGbp1zTlZgN+Y1A19X6nusNsU=">AAAB6nicbZBNS8NAEIYn9avWr6pHL4tF8FQSEfRY9OKxov2ANpTNdtKu3WzC7kYpoT/BiwdFvPqLvPlv3LY5aOsLCw/vzLAzb5AIro3rfjuFldW19Y3iZmlre2d3r7x/0NRxqhg2WCxi1Q6oRsElNgw3AtuJQhoFAlvB6Hpabz2i0jyW92acoB/RgeQhZ9RY6+6p99ArV9yqOxNZBi+HCuSq98pf3X7M0gilYYJq3fHcxPgZVYYzgZNSN9WYUDaiA+xYlDRC7WezVSfkxDp9EsbKPmnIzP09kdFI63EU2M6ImqFerE3N/2qd1ISXfsZlkhqUbP5RmApiYjK9m/S5QmbE2AJlittdCRtSRZmx6ZRsCN7iycvQPKt6lm/PK7WrPI4iHMExnIIHF1CDG6hDAxgM4Ble4c0Rzovz7nzMWwtOPnMIf+R8/gBgkI3Y</latexit><latexit sha1_base64="HwPGbp1zTlZgN+Y1A19X6nusNsU=">AAAB6nicbZBNS8NAEIYn9avWr6pHL4tF8FQSEfRY9OKxov2ANpTNdtKu3WzC7kYpoT/BiwdFvPqLvPlv3LY5aOsLCw/vzLAzb5AIro3rfjuFldW19Y3iZmlre2d3r7x/0NRxqhg2WCxi1Q6oRsElNgw3AtuJQhoFAlvB6Hpabz2i0jyW92acoB/RgeQhZ9RY6+6p99ArV9yqOxNZBi+HCuSq98pf3X7M0gilYYJq3fHcxPgZVYYzgZNSN9WYUDaiA+xYlDRC7WezVSfkxDp9EsbKPmnIzP09kdFI63EU2M6ImqFerE3N/2qd1ISXfsZlkhqUbP5RmApiYjK9m/S5QmbE2AJlittdCRtSRZmx6ZRsCN7iycvQPKt6lm/PK7WrPI4iHMExnIIHF1CDG6hDAxgM4Ble4c0Rzovz7nzMWwtOPnMIf+R8/gBgkI3Y</latexit><latexit sha1_base64="HwPGbp1zTlZgN+Y1A19X6nusNsU=">AAAB6nicbZBNS8NAEIYn9avWr6pHL4tF8FQSEfRY9OKxov2ANpTNdtKu3WzC7kYpoT/BiwdFvPqLvPlv3LY5aOsLCw/vzLAzb5AIro3rfjuFldW19Y3iZmlre2d3r7x/0NRxqhg2WCxi1Q6oRsElNgw3AtuJQhoFAlvB6Hpabz2i0jyW92acoB/RgeQhZ9RY6+6p99ArV9yqOxNZBi+HCuSq98pf3X7M0gilYYJq3fHcxPgZVYYzgZNSN9WYUDaiA+xYlDRC7WezVSfkxDp9EsbKPmnIzP09kdFI63EU2M6ImqFerE3N/2qd1ISXfsZlkhqUbP5RmApiYjK9m/S5QmbE2AJlittdCRtSRZmx6ZRsCN7iycvQPKt6lm/PK7WrPI4iHMExnIIHF1CDG6hDAxgM4Ble4c0Rzovz7nzMWwtOPnMIf+R8/gBgkI3Y</latexit>

uj<latexit sha1_base64="TCD8Huirq8KGLFYVIYoP6bPo/BI=">AAAB6nicbZBNS8NAEIYn9avWr6pHL4tF8FQSEeqx6MVjRfsBbSib7aRdu9mE3Y1QQn+CFw+KePUXefPfuG1z0NYXFh7emWFn3iARXBvX/XYKa+sbm1vF7dLO7t7+QfnwqKXjVDFssljEqhNQjYJLbBpuBHYShTQKBLaD8c2s3n5CpXksH8wkQT+iQ8lDzqix1n3af+yXK27VnYusgpdDBXI1+uWv3iBmaYTSMEG17npuYvyMKsOZwGmpl2pMKBvTIXYtShqh9rP5qlNyZp0BCWNlnzRk7v6eyGik9SQKbGdEzUgv12bmf7VuasIrP+MySQ1KtvgoTAUxMZndTQZcITNiYoEyxe2uhI2ooszYdEo2BG/55FVoXVQ9y3eXlfp1HkcRTuAUzsGDGtThFhrQBAZDeIZXeHOE8+K8Ox+L1oKTzxzDHzmfP12EjdY=</latexit><latexit sha1_base64="TCD8Huirq8KGLFYVIYoP6bPo/BI=">AAAB6nicbZBNS8NAEIYn9avWr6pHL4tF8FQSEeqx6MVjRfsBbSib7aRdu9mE3Y1QQn+CFw+KePUXefPfuG1z0NYXFh7emWFn3iARXBvX/XYKa+sbm1vF7dLO7t7+QfnwqKXjVDFssljEqhNQjYJLbBpuBHYShTQKBLaD8c2s3n5CpXksH8wkQT+iQ8lDzqix1n3af+yXK27VnYusgpdDBXI1+uWv3iBmaYTSMEG17npuYvyMKsOZwGmpl2pMKBvTIXYtShqh9rP5qlNyZp0BCWNlnzRk7v6eyGik9SQKbGdEzUgv12bmf7VuasIrP+MySQ1KtvgoTAUxMZndTQZcITNiYoEyxe2uhI2ooszYdEo2BG/55FVoXVQ9y3eXlfp1HkcRTuAUzsGDGtThFhrQBAZDeIZXeHOE8+K8Ox+L1oKTzxzDHzmfP12EjdY=</latexit><latexit sha1_base64="TCD8Huirq8KGLFYVIYoP6bPo/BI=">AAAB6nicbZBNS8NAEIYn9avWr6pHL4tF8FQSEeqx6MVjRfsBbSib7aRdu9mE3Y1QQn+CFw+KePUXefPfuG1z0NYXFh7emWFn3iARXBvX/XYKa+sbm1vF7dLO7t7+QfnwqKXjVDFssljEqhNQjYJLbBpuBHYShTQKBLaD8c2s3n5CpXksH8wkQT+iQ8lDzqix1n3af+yXK27VnYusgpdDBXI1+uWv3iBmaYTSMEG17npuYvyMKsOZwGmpl2pMKBvTIXYtShqh9rP5qlNyZp0BCWNlnzRk7v6eyGik9SQKbGdEzUgv12bmf7VuasIrP+MySQ1KtvgoTAUxMZndTQZcITNiYoEyxe2uhI2ooszYdEo2BG/55FVoXVQ9y3eXlfp1HkcRTuAUzsGDGtThFhrQBAZDeIZXeHOE8+K8Ox+L1oKTzxzDHzmfP12EjdY=</latexit><latexit sha1_base64="ck8pdC+ekZH4nUmSP+ZG7r8lEyk=">AAAB2XicbZDNSgMxFIXv1L86Vq1rN8EiuCozbnQpuHFZwbZCO5RM5k4bmskMyR2hDH0BF25EfC93vo3pz0JbDwQ+zknIvSculLQUBN9ebWd3b/+gfugfNfzjk9Nmo2fz0gjsilzl5jnmFpXU2CVJCp8LgzyLFfbj6f0i77+gsTLXTzQrMMr4WMtUCk7O6oyaraAdLMW2IVxDC9YaNb+GSS7KDDUJxa0dhEFBUcUNSaFw7g9LiwUXUz7GgUPNM7RRtRxzzi6dk7A0N+5oYkv394uKZ9bOstjdzDhN7Ga2MP/LBiWlt1EldVESarH6KC0Vo5wtdmaJNChIzRxwYaSblYkJN1yQa8Z3HYSbG29D77odOn4MoA7ncAFXEMIN3MEDdKALAhJ4hXdv4r15H6uuat66tDP4I+/zBzjGijg=</latexit><latexit sha1_base64="n9CoYaC86bbPTqcwZaqpK7PNAxc=">AAAB33icbZBLSwMxFIXv+Ky1anXrJlgEV2XGjS4FNy4r2ge0Q8mkd9rYTGZI7ghl6E9w40IR/5U7/43pY6GtBwIf5yTk3hNlSlry/W9vY3Nre2e3tFferxwcHlWPKy2b5kZgU6QqNZ2IW1RSY5MkKexkBnkSKWxH49tZ3n5GY2WqH2mSYZjwoZaxFJyc9ZD3n/rVml/352LrECyhBks1+tWv3iAVeYKahOLWdgM/o7DghqRQOC33cosZF2M+xK5DzRO0YTEfdcrOnTNgcWrc0cTm7u8XBU+snSSRu5lwGtnVbGb+l3Vziq/DQuosJ9Ri8VGcK0Ypm+3NBtKgIDVxwIWRblYmRtxwQa6dsishWF15HVqX9cDxvQ8lOIUzuIAAruAG7qABTRAwhBd4g3dPea/ex6KuDW/Z2wn8kff5Az/cjIE=</latexit><latexit sha1_base64="n9CoYaC86bbPTqcwZaqpK7PNAxc=">AAAB33icbZBLSwMxFIXv+Ky1anXrJlgEV2XGjS4FNy4r2ge0Q8mkd9rYTGZI7ghl6E9w40IR/5U7/43pY6GtBwIf5yTk3hNlSlry/W9vY3Nre2e3tFferxwcHlWPKy2b5kZgU6QqNZ2IW1RSY5MkKexkBnkSKWxH49tZ3n5GY2WqH2mSYZjwoZaxFJyc9ZD3n/rVml/352LrECyhBks1+tWv3iAVeYKahOLWdgM/o7DghqRQOC33cosZF2M+xK5DzRO0YTEfdcrOnTNgcWrc0cTm7u8XBU+snSSRu5lwGtnVbGb+l3Vziq/DQuosJ9Ri8VGcK0Ypm+3NBtKgIDVxwIWRblYmRtxwQa6dsishWF15HVqX9cDxvQ8lOIUzuIAAruAG7qABTRAwhBd4g3dPea/ex6KuDW/Z2wn8kff5Az/cjIE=</latexit><latexit sha1_base64="46aLFBoWDxH9bhturSsrpEjo9qI=">AAAB6nicbZA9T8MwEIYv5auUrwAji0WFxFQlLDBWsDAWQT+kNqoc99KaOk5kO0hV1J/AwgBCrPwiNv4NbpsBWl7J0qP37uS7N0wF18bzvp3S2vrG5lZ5u7Kzu7d/4B4etXSSKYZNlohEdUKqUXCJTcONwE6qkMahwHY4vpnV20+oNE/kg5mkGMR0KHnEGTXWus/6j3236tW8ucgq+AVUoVCj7371BgnLYpSGCap11/dSE+RUGc4ETiu9TGNK2ZgOsWtR0hh1kM9XnZIz6wxIlCj7pCFz9/dETmOtJ3FoO2NqRnq5NjP/q3UzE10FOZdpZlCyxUdRJohJyOxuMuAKmRETC5QpbnclbEQVZcamU7Eh+Msnr0LrouZbvvOq9esijjKcwCmcgw+XUIdbaEATGAzhGV7hzRHOi/PufCxaS04xcwx/5Hz+AFxEjdI=</latexit><latexit sha1_base64="TCD8Huirq8KGLFYVIYoP6bPo/BI=">AAAB6nicbZBNS8NAEIYn9avWr6pHL4tF8FQSEeqx6MVjRfsBbSib7aRdu9mE3Y1QQn+CFw+KePUXefPfuG1z0NYXFh7emWFn3iARXBvX/XYKa+sbm1vF7dLO7t7+QfnwqKXjVDFssljEqhNQjYJLbBpuBHYShTQKBLaD8c2s3n5CpXksH8wkQT+iQ8lDzqix1n3af+yXK27VnYusgpdDBXI1+uWv3iBmaYTSMEG17npuYvyMKsOZwGmpl2pMKBvTIXYtShqh9rP5qlNyZp0BCWNlnzRk7v6eyGik9SQKbGdEzUgv12bmf7VuasIrP+MySQ1KtvgoTAUxMZndTQZcITNiYoEyxe2uhI2ooszYdEo2BG/55FVoXVQ9y3eXlfp1HkcRTuAUzsGDGtThFhrQBAZDeIZXeHOE8+K8Ox+L1oKTzxzDHzmfP12EjdY=</latexit><latexit sha1_base64="TCD8Huirq8KGLFYVIYoP6bPo/BI=">AAAB6nicbZBNS8NAEIYn9avWr6pHL4tF8FQSEeqx6MVjRfsBbSib7aRdu9mE3Y1QQn+CFw+KePUXefPfuG1z0NYXFh7emWFn3iARXBvX/XYKa+sbm1vF7dLO7t7+QfnwqKXjVDFssljEqhNQjYJLbBpuBHYShTQKBLaD8c2s3n5CpXksH8wkQT+iQ8lDzqix1n3af+yXK27VnYusgpdDBXI1+uWv3iBmaYTSMEG17npuYvyMKsOZwGmpl2pMKBvTIXYtShqh9rP5qlNyZp0BCWNlnzRk7v6eyGik9SQKbGdEzUgv12bmf7VuasIrP+MySQ1KtvgoTAUxMZndTQZcITNiYoEyxe2uhI2ooszYdEo2BG/55FVoXVQ9y3eXlfp1HkcRTuAUzsGDGtThFhrQBAZDeIZXeHOE8+K8Ox+L1oKTzxzDHzmfP12EjdY=</latexit><latexit sha1_base64="TCD8Huirq8KGLFYVIYoP6bPo/BI=">AAAB6nicbZBNS8NAEIYn9avWr6pHL4tF8FQSEeqx6MVjRfsBbSib7aRdu9mE3Y1QQn+CFw+KePUXefPfuG1z0NYXFh7emWFn3iARXBvX/XYKa+sbm1vF7dLO7t7+QfnwqKXjVDFssljEqhNQjYJLbBpuBHYShTQKBLaD8c2s3n5CpXksH8wkQT+iQ8lDzqix1n3af+yXK27VnYusgpdDBXI1+uWv3iBmaYTSMEG17npuYvyMKsOZwGmpl2pMKBvTIXYtShqh9rP5qlNyZp0BCWNlnzRk7v6eyGik9SQKbGdEzUgv12bmf7VuasIrP+MySQ1KtvgoTAUxMZndTQZcITNiYoEyxe2uhI2ooszYdEo2BG/55FVoXVQ9y3eXlfp1HkcRTuAUzsGDGtThFhrQBAZDeIZXeHOE8+K8Ox+L1oKTzxzDHzmfP12EjdY=</latexit><latexit sha1_base64="TCD8Huirq8KGLFYVIYoP6bPo/BI=">AAAB6nicbZBNS8NAEIYn9avWr6pHL4tF8FQSEeqx6MVjRfsBbSib7aRdu9mE3Y1QQn+CFw+KePUXefPfuG1z0NYXFh7emWFn3iARXBvX/XYKa+sbm1vF7dLO7t7+QfnwqKXjVDFssljEqhNQjYJLbBpuBHYShTQKBLaD8c2s3n5CpXksH8wkQT+iQ8lDzqix1n3af+yXK27VnYusgpdDBXI1+uWv3iBmaYTSMEG17npuYvyMKsOZwGmpl2pMKBvTIXYtShqh9rP5qlNyZp0BCWNlnzRk7v6eyGik9SQKbGdEzUgv12bmf7VuasIrP+MySQ1KtvgoTAUxMZndTQZcITNiYoEyxe2uhI2ooszYdEo2BG/55FVoXVQ9y3eXlfp1HkcRTuAUzsGDGtThFhrQBAZDeIZXeHOE8+K8Ox+L1oKTzxzDHzmfP12EjdY=</latexit><latexit sha1_base64="TCD8Huirq8KGLFYVIYoP6bPo/BI=">AAAB6nicbZBNS8NAEIYn9avWr6pHL4tF8FQSEeqx6MVjRfsBbSib7aRdu9mE3Y1QQn+CFw+KePUXefPfuG1z0NYXFh7emWFn3iARXBvX/XYKa+sbm1vF7dLO7t7+QfnwqKXjVDFssljEqhNQjYJLbBpuBHYShTQKBLaD8c2s3n5CpXksH8wkQT+iQ8lDzqix1n3af+yXK27VnYusgpdDBXI1+uWv3iBmaYTSMEG17npuYvyMKsOZwGmpl2pMKBvTIXYtShqh9rP5qlNyZp0BCWNlnzRk7v6eyGik9SQKbGdEzUgv12bmf7VuasIrP+MySQ1KtvgoTAUxMZndTQZcITNiYoEyxe2uhI2ooszYdEo2BG/55FVoXVQ9y3eXlfp1HkcRTuAUzsGDGtThFhrQBAZDeIZXeHOE8+K8Ox+L1oKTzxzDHzmfP12EjdY=</latexit><latexit sha1_base64="TCD8Huirq8KGLFYVIYoP6bPo/BI=">AAAB6nicbZBNS8NAEIYn9avWr6pHL4tF8FQSEeqx6MVjRfsBbSib7aRdu9mE3Y1QQn+CFw+KePUXefPfuG1z0NYXFh7emWFn3iARXBvX/XYKa+sbm1vF7dLO7t7+QfnwqKXjVDFssljEqhNQjYJLbBpuBHYShTQKBLaD8c2s3n5CpXksH8wkQT+iQ8lDzqix1n3af+yXK27VnYusgpdDBXI1+uWv3iBmaYTSMEG17npuYvyMKsOZwGmpl2pMKBvTIXYtShqh9rP5qlNyZp0BCWNlnzRk7v6eyGik9SQKbGdEzUgv12bmf7VuasIrP+MySQ1KtvgoTAUxMZndTQZcITNiYoEyxe2uhI2ooszYdEo2BG/55FVoXVQ9y3eXlfp1HkcRTuAUzsGDGtThFhrQBAZDeIZXeHOE8+K8Ox+L1oKTzxzDHzmfP12EjdY=</latexit>

Ew[f(x)] = 0<latexit sha1_base64="SbIi8cWmntoBu8wL7y5vOLuOxV4=">AAAB9HicbZDLSsNAFIZP6q3WW9Wlm8Ei1E1JpKAboSiCywr2Amkok+mkHTqZxJlJtYQ+hxsXirj1Ydz5Nk7bLLT1h4GP/5zDOfP7MWdK2/a3lVtZXVvfyG8WtrZ3dveK+wdNFSWS0AaJeCTbPlaUM0EbmmlO27GkOPQ5bfnD62m9NaJSsUjc63FMvRD3BQsYwdpY3k330Q3KT6ceukR2t1iyK/ZMaBmcDEqQqd4tfnV6EUlCKjThWCnXsWPtpVhqRjidFDqJojEmQ9ynrkGBQ6q8dHb0BJ0Yp4eCSJonNJq5vydSHCo1Dn3TGWI9UIu1qflfzU10cOGlTMSJpoLMFwUJRzpC0wRQj0lKNB8bwEQycysiAywx0SanggnBWfzyMjTPKo7hu2qpdpXFkYcjOIYyOHAONbiFOjSAwAM8wyu8WSPrxXq3PuatOSubOYQ/sj5/AJqxkKs=</latexit><latexit sha1_base64="SbIi8cWmntoBu8wL7y5vOLuOxV4=">AAAB9HicbZDLSsNAFIZP6q3WW9Wlm8Ei1E1JpKAboSiCywr2Amkok+mkHTqZxJlJtYQ+hxsXirj1Ydz5Nk7bLLT1h4GP/5zDOfP7MWdK2/a3lVtZXVvfyG8WtrZ3dveK+wdNFSWS0AaJeCTbPlaUM0EbmmlO27GkOPQ5bfnD62m9NaJSsUjc63FMvRD3BQsYwdpY3k330Q3KT6ceukR2t1iyK/ZMaBmcDEqQqd4tfnV6EUlCKjThWCnXsWPtpVhqRjidFDqJojEmQ9ynrkGBQ6q8dHb0BJ0Yp4eCSJonNJq5vydSHCo1Dn3TGWI9UIu1qflfzU10cOGlTMSJpoLMFwUJRzpC0wRQj0lKNB8bwEQycysiAywx0SanggnBWfzyMjTPKo7hu2qpdpXFkYcjOIYyOHAONbiFOjSAwAM8wyu8WSPrxXq3PuatOSubOYQ/sj5/AJqxkKs=</latexit><latexit sha1_base64="SbIi8cWmntoBu8wL7y5vOLuOxV4=">AAAB9HicbZDLSsNAFIZP6q3WW9Wlm8Ei1E1JpKAboSiCywr2Amkok+mkHTqZxJlJtYQ+hxsXirj1Ydz5Nk7bLLT1h4GP/5zDOfP7MWdK2/a3lVtZXVvfyG8WtrZ3dveK+wdNFSWS0AaJeCTbPlaUM0EbmmlO27GkOPQ5bfnD62m9NaJSsUjc63FMvRD3BQsYwdpY3k330Q3KT6ceukR2t1iyK/ZMaBmcDEqQqd4tfnV6EUlCKjThWCnXsWPtpVhqRjidFDqJojEmQ9ynrkGBQ6q8dHb0BJ0Yp4eCSJonNJq5vydSHCo1Dn3TGWI9UIu1qflfzU10cOGlTMSJpoLMFwUJRzpC0wRQj0lKNB8bwEQycysiAywx0SanggnBWfzyMjTPKo7hu2qpdpXFkYcjOIYyOHAONbiFOjSAwAM8wyu8WSPrxXq3PuatOSubOYQ/sj5/AJqxkKs=</latexit><latexit sha1_base64="SbIi8cWmntoBu8wL7y5vOLuOxV4=">AAAB9HicbZDLSsNAFIZP6q3WW9Wlm8Ei1E1JpKAboSiCywr2Amkok+mkHTqZxJlJtYQ+hxsXirj1Ydz5Nk7bLLT1h4GP/5zDOfP7MWdK2/a3lVtZXVvfyG8WtrZ3dveK+wdNFSWS0AaJeCTbPlaUM0EbmmlO27GkOPQ5bfnD62m9NaJSsUjc63FMvRD3BQsYwdpY3k330Q3KT6ceukR2t1iyK/ZMaBmcDEqQqd4tfnV6EUlCKjThWCnXsWPtpVhqRjidFDqJojEmQ9ynrkGBQ6q8dHb0BJ0Yp4eCSJonNJq5vydSHCo1Dn3TGWI9UIu1qflfzU10cOGlTMSJpoLMFwUJRzpC0wRQj0lKNB8bwEQycysiAywx0SanggnBWfzyMjTPKo7hu2qpdpXFkYcjOIYyOHAONbiFOjSAwAM8wyu8WSPrxXq3PuatOSubOYQ/sj5/AJqxkKs=</latexit>

Computing with Infinite Networks, C. Williams, NIPS 1997 Bayesian leaning for Neural Networks, R. Neal, LNS, 1996

Ew[f(x)f(x�)] = �2b + J�2

wEu[�(x; uj)�(x�; uj)]<latexit sha1_base64="HV75SHiVfQdFEwffa+6e7+S+1Lk=">AAACQ3icbVBdaxNBFJ2tbY3RtlEffRkaShIKZbcUFEQoSkB8qmCa4O52mZ29m0w7+8HMXZuw5L/1pX/AN/+ALz4o4qvgZDeUNvXADGfOuZd754S5FBpt+5u19mB9Y/Nh41Hz8ZOt7Z3W02enOisUhwHPZKZGIdMgRQoDFChhlCtgSShhGF68W/jDL6C0yNJPOMvBT9g4FbHgDI0UtD73g0s37k571Fydnk/fUE+LccLODoOQ7lMPYYrVmFJBNC8/zG/sS9oPCtfLJ6I7fU2L4LxX80798INW2z6wK9D7xFmSNlniJGh99aKMFwmkyCXT2nXsHP2SKRRcwrzpFRpyxi/YGFxDU5aA9stquTndM0pE40yZkyKt1NsdJUu0niWhqUwYTvSqtxD/57kFxq/8UqR5gZDyelBcSIoZXQRKI6GAo5wZwrgSZlfKJ0wxjib2pgnBWf3yfXJ6eOAY/vGoffx2GUeDvCC7pEsc8pIck/fkhAwIJ1fkO/lJflnX1g/rt/WnLl2zlj3PyR1Yf/8BgZyu6w==</latexit><latexit sha1_base64="HV75SHiVfQdFEwffa+6e7+S+1Lk=">AAACQ3icbVBdaxNBFJ2tbY3RtlEffRkaShIKZbcUFEQoSkB8qmCa4O52mZ29m0w7+8HMXZuw5L/1pX/AN/+ALz4o4qvgZDeUNvXADGfOuZd754S5FBpt+5u19mB9Y/Nh41Hz8ZOt7Z3W02enOisUhwHPZKZGIdMgRQoDFChhlCtgSShhGF68W/jDL6C0yNJPOMvBT9g4FbHgDI0UtD73g0s37k571Fydnk/fUE+LccLODoOQ7lMPYYrVmFJBNC8/zG/sS9oPCtfLJ6I7fU2L4LxX80798INW2z6wK9D7xFmSNlniJGh99aKMFwmkyCXT2nXsHP2SKRRcwrzpFRpyxi/YGFxDU5aA9stquTndM0pE40yZkyKt1NsdJUu0niWhqUwYTvSqtxD/57kFxq/8UqR5gZDyelBcSIoZXQRKI6GAo5wZwrgSZlfKJ0wxjib2pgnBWf3yfXJ6eOAY/vGoffx2GUeDvCC7pEsc8pIck/fkhAwIJ1fkO/lJflnX1g/rt/WnLl2zlj3PyR1Yf/8BgZyu6w==</latexit><latexit sha1_base64="HV75SHiVfQdFEwffa+6e7+S+1Lk=">AAACQ3icbVBdaxNBFJ2tbY3RtlEffRkaShIKZbcUFEQoSkB8qmCa4O52mZ29m0w7+8HMXZuw5L/1pX/AN/+ALz4o4qvgZDeUNvXADGfOuZd754S5FBpt+5u19mB9Y/Nh41Hz8ZOt7Z3W02enOisUhwHPZKZGIdMgRQoDFChhlCtgSShhGF68W/jDL6C0yNJPOMvBT9g4FbHgDI0UtD73g0s37k571Fydnk/fUE+LccLODoOQ7lMPYYrVmFJBNC8/zG/sS9oPCtfLJ6I7fU2L4LxX80798INW2z6wK9D7xFmSNlniJGh99aKMFwmkyCXT2nXsHP2SKRRcwrzpFRpyxi/YGFxDU5aA9stquTndM0pE40yZkyKt1NsdJUu0niWhqUwYTvSqtxD/57kFxq/8UqR5gZDyelBcSIoZXQRKI6GAo5wZwrgSZlfKJ0wxjib2pgnBWf3yfXJ6eOAY/vGoffx2GUeDvCC7pEsc8pIck/fkhAwIJ1fkO/lJflnX1g/rt/WnLl2zlj3PyR1Yf/8BgZyu6w==</latexit><latexit sha1_base64="HV75SHiVfQdFEwffa+6e7+S+1Lk=">AAACQ3icbVBdaxNBFJ2tbY3RtlEffRkaShIKZbcUFEQoSkB8qmCa4O52mZ29m0w7+8HMXZuw5L/1pX/AN/+ALz4o4qvgZDeUNvXADGfOuZd754S5FBpt+5u19mB9Y/Nh41Hz8ZOt7Z3W02enOisUhwHPZKZGIdMgRQoDFChhlCtgSShhGF68W/jDL6C0yNJPOMvBT9g4FbHgDI0UtD73g0s37k571Fydnk/fUE+LccLODoOQ7lMPYYrVmFJBNC8/zG/sS9oPCtfLJ6I7fU2L4LxX80798INW2z6wK9D7xFmSNlniJGh99aKMFwmkyCXT2nXsHP2SKRRcwrzpFRpyxi/YGFxDU5aA9stquTndM0pE40yZkyKt1NsdJUu0niWhqUwYTvSqtxD/57kFxq/8UqR5gZDyelBcSIoZXQRKI6GAo5wZwrgSZlfKJ0wxjib2pgnBWf3yfXJ6eOAY/vGoffx2GUeDvCC7pEsc8pIck/fkhAwIJ1fkO/lJflnX1g/rt/WnLl2zlj3PyR1Yf/8BgZyu6w==</latexit>

!15

Page 16: Horseshoe Priors for Bayesian Neural Networks...Horseshoe Priors for Bayesian Neural Networks Soumya Ghosh Jiayu Yao Finale Doshi-Velez IBM Research MIT-IBM Watson AI lab Harvard Harvard

Predictive Uncertainties?

Single layer network, with prior:

(Same results across many initialization strategies)

What is happening?

Bigger network → More parameters, same data → More parameter uncertainty → Higher predictive variance

W � N (0, I)<latexit sha1_base64="ACN79W3NN9SiNlCY7zZvbl+vEIs=">AAACHXicbZDLSsNAFIYnXmu9RV26GSxCBSmJFHRZdKMbqWAv0IQymU7aoZNJmJkIJeRF3Pgqblwo4sKN+DZO0ija+sPAx3/OYc75vYhRqSzr01hYXFpeWS2tldc3Nre2zZ3dtgxjgUkLhywUXQ9JwignLUUVI91IEBR4jHS88UVW79wRIWnIb9UkIm6Ahpz6FCOlrb5ZdwKkRhixpJNCR9IA/hjXaTVnz0+s9Bh+81V61DcrVs3KBefBLqACCjX75rszCHEcEK4wQ1L2bCtSboKEopiRtOzEkkQIj9GQ9DRyFBDpJvl1KTzUzgD6odCPK5i7vycSFEg5CTzdma0oZ2uZ+V+tFyv/zE0oj2JFOJ5+5McMqhBmUcEBFQQrNtGAsKB6V4hHSCCsdKBlHYI9e/I8tE9qtlWzb+qVxnkRRwnsgwNQBTY4BQ1wCZqgBTC4B4/gGbwYD8aT8Wq8TVsXjGJmD/yR8fEFpt2iOw==</latexit><latexit sha1_base64="ACN79W3NN9SiNlCY7zZvbl+vEIs=">AAACHXicbZDLSsNAFIYnXmu9RV26GSxCBSmJFHRZdKMbqWAv0IQymU7aoZNJmJkIJeRF3Pgqblwo4sKN+DZO0ija+sPAx3/OYc75vYhRqSzr01hYXFpeWS2tldc3Nre2zZ3dtgxjgUkLhywUXQ9JwignLUUVI91IEBR4jHS88UVW79wRIWnIb9UkIm6Ahpz6FCOlrb5ZdwKkRhixpJNCR9IA/hjXaTVnz0+s9Bh+81V61DcrVs3KBefBLqACCjX75rszCHEcEK4wQ1L2bCtSboKEopiRtOzEkkQIj9GQ9DRyFBDpJvl1KTzUzgD6odCPK5i7vycSFEg5CTzdma0oZ2uZ+V+tFyv/zE0oj2JFOJ5+5McMqhBmUcEBFQQrNtGAsKB6V4hHSCCsdKBlHYI9e/I8tE9qtlWzb+qVxnkRRwnsgwNQBTY4BQ1wCZqgBTC4B4/gGbwYD8aT8Wq8TVsXjGJmD/yR8fEFpt2iOw==</latexit><latexit sha1_base64="ACN79W3NN9SiNlCY7zZvbl+vEIs=">AAACHXicbZDLSsNAFIYnXmu9RV26GSxCBSmJFHRZdKMbqWAv0IQymU7aoZNJmJkIJeRF3Pgqblwo4sKN+DZO0ija+sPAx3/OYc75vYhRqSzr01hYXFpeWS2tldc3Nre2zZ3dtgxjgUkLhywUXQ9JwignLUUVI91IEBR4jHS88UVW79wRIWnIb9UkIm6Ahpz6FCOlrb5ZdwKkRhixpJNCR9IA/hjXaTVnz0+s9Bh+81V61DcrVs3KBefBLqACCjX75rszCHEcEK4wQ1L2bCtSboKEopiRtOzEkkQIj9GQ9DRyFBDpJvl1KTzUzgD6odCPK5i7vycSFEg5CTzdma0oZ2uZ+V+tFyv/zE0oj2JFOJ5+5McMqhBmUcEBFQQrNtGAsKB6V4hHSCCsdKBlHYI9e/I8tE9qtlWzb+qVxnkRRwnsgwNQBTY4BQ1wCZqgBTC4B4/gGbwYD8aT8Wq8TVsXjGJmD/yR8fEFpt2iOw==</latexit><latexit sha1_base64="ACN79W3NN9SiNlCY7zZvbl+vEIs=">AAACHXicbZDLSsNAFIYnXmu9RV26GSxCBSmJFHRZdKMbqWAv0IQymU7aoZNJmJkIJeRF3Pgqblwo4sKN+DZO0ija+sPAx3/OYc75vYhRqSzr01hYXFpeWS2tldc3Nre2zZ3dtgxjgUkLhywUXQ9JwignLUUVI91IEBR4jHS88UVW79wRIWnIb9UkIm6Ahpz6FCOlrb5ZdwKkRhixpJNCR9IA/hjXaTVnz0+s9Bh+81V61DcrVs3KBefBLqACCjX75rszCHEcEK4wQ1L2bCtSboKEopiRtOzEkkQIj9GQ9DRyFBDpJvl1KTzUzgD6odCPK5i7vycSFEg5CTzdma0oZ2uZ+V+tFyv/zE0oj2JFOJ5+5McMqhBmUcEBFQQrNtGAsKB6V4hHSCCsdKBlHYI9e/I8tE9qtlWzb+qVxnkRRwnsgwNQBTY4BQ1wCZqgBTC4B4/gGbwYD8aT8Wq8TVsXjGJmD/yR8fEFpt2iOw==</latexit>

!16

Page 17: Horseshoe Priors for Bayesian Neural Networks...Horseshoe Priors for Bayesian Neural Networks Soumya Ghosh Jiayu Yao Finale Doshi-Velez IBM Research MIT-IBM Watson AI lab Harvard Harvard

Bounding Prior varianceuj

<latexit sha1_base64="TCD8Huirq8KGLFYVIYoP6bPo/BI=">AAAB6nicbZBNS8NAEIYn9avWr6pHL4tF8FQSEeqx6MVjRfsBbSib7aRdu9mE3Y1QQn+CFw+KePUXefPfuG1z0NYXFh7emWFn3iARXBvX/XYKa+sbm1vF7dLO7t7+QfnwqKXjVDFssljEqhNQjYJLbBpuBHYShTQKBLaD8c2s3n5CpXksH8wkQT+iQ8lDzqix1n3af+yXK27VnYusgpdDBXI1+uWv3iBmaYTSMEG17npuYvyMKsOZwGmpl2pMKBvTIXYtShqh9rP5qlNyZp0BCWNlnzRk7v6eyGik9SQKbGdEzUgv12bmf7VuasIrP+MySQ1KtvgoTAUxMZndTQZcITNiYoEyxe2uhI2ooszYdEo2BG/55FVoXVQ9y3eXlfp1HkcRTuAUzsGDGtThFhrQBAZDeIZXeHOE8+K8Ox+L1oKTzxzDHzmfP12EjdY=</latexit><latexit sha1_base64="TCD8Huirq8KGLFYVIYoP6bPo/BI=">AAAB6nicbZBNS8NAEIYn9avWr6pHL4tF8FQSEeqx6MVjRfsBbSib7aRdu9mE3Y1QQn+CFw+KePUXefPfuG1z0NYXFh7emWFn3iARXBvX/XYKa+sbm1vF7dLO7t7+QfnwqKXjVDFssljEqhNQjYJLbBpuBHYShTQKBLaD8c2s3n5CpXksH8wkQT+iQ8lDzqix1n3af+yXK27VnYusgpdDBXI1+uWv3iBmaYTSMEG17npuYvyMKsOZwGmpl2pMKBvTIXYtShqh9rP5qlNyZp0BCWNlnzRk7v6eyGik9SQKbGdEzUgv12bmf7VuasIrP+MySQ1KtvgoTAUxMZndTQZcITNiYoEyxe2uhI2ooszYdEo2BG/55FVoXVQ9y3eXlfp1HkcRTuAUzsGDGtThFhrQBAZDeIZXeHOE8+K8Ox+L1oKTzxzDHzmfP12EjdY=</latexit><latexit sha1_base64="TCD8Huirq8KGLFYVIYoP6bPo/BI=">AAAB6nicbZBNS8NAEIYn9avWr6pHL4tF8FQSEeqx6MVjRfsBbSib7aRdu9mE3Y1QQn+CFw+KePUXefPfuG1z0NYXFh7emWFn3iARXBvX/XYKa+sbm1vF7dLO7t7+QfnwqKXjVDFssljEqhNQjYJLbBpuBHYShTQKBLaD8c2s3n5CpXksH8wkQT+iQ8lDzqix1n3af+yXK27VnYusgpdDBXI1+uWv3iBmaYTSMEG17npuYvyMKsOZwGmpl2pMKBvTIXYtShqh9rP5qlNyZp0BCWNlnzRk7v6eyGik9SQKbGdEzUgv12bmf7VuasIrP+MySQ1KtvgoTAUxMZndTQZcITNiYoEyxe2uhI2ooszYdEo2BG/55FVoXVQ9y3eXlfp1HkcRTuAUzsGDGtThFhrQBAZDeIZXeHOE8+K8Ox+L1oKTzxzDHzmfP12EjdY=</latexit><latexit sha1_base64="ck8pdC+ekZH4nUmSP+ZG7r8lEyk=">AAAB2XicbZDNSgMxFIXv1L86Vq1rN8EiuCozbnQpuHFZwbZCO5RM5k4bmskMyR2hDH0BF25EfC93vo3pz0JbDwQ+zknIvSculLQUBN9ebWd3b/+gfugfNfzjk9Nmo2fz0gjsilzl5jnmFpXU2CVJCp8LgzyLFfbj6f0i77+gsTLXTzQrMMr4WMtUCk7O6oyaraAdLMW2IVxDC9YaNb+GSS7KDDUJxa0dhEFBUcUNSaFw7g9LiwUXUz7GgUPNM7RRtRxzzi6dk7A0N+5oYkv394uKZ9bOstjdzDhN7Ga2MP/LBiWlt1EldVESarH6KC0Vo5wtdmaJNChIzRxwYaSblYkJN1yQa8Z3HYSbG29D77odOn4MoA7ncAFXEMIN3MEDdKALAhJ4hXdv4r15H6uuat66tDP4I+/zBzjGijg=</latexit><latexit sha1_base64="n9CoYaC86bbPTqcwZaqpK7PNAxc=">AAAB33icbZBLSwMxFIXv+Ky1anXrJlgEV2XGjS4FNy4r2ge0Q8mkd9rYTGZI7ghl6E9w40IR/5U7/43pY6GtBwIf5yTk3hNlSlry/W9vY3Nre2e3tFferxwcHlWPKy2b5kZgU6QqNZ2IW1RSY5MkKexkBnkSKWxH49tZ3n5GY2WqH2mSYZjwoZaxFJyc9ZD3n/rVml/352LrECyhBks1+tWv3iAVeYKahOLWdgM/o7DghqRQOC33cosZF2M+xK5DzRO0YTEfdcrOnTNgcWrc0cTm7u8XBU+snSSRu5lwGtnVbGb+l3Vziq/DQuosJ9Ri8VGcK0Ypm+3NBtKgIDVxwIWRblYmRtxwQa6dsishWF15HVqX9cDxvQ8lOIUzuIAAruAG7qABTRAwhBd4g3dPea/ex6KuDW/Z2wn8kff5Az/cjIE=</latexit><latexit sha1_base64="n9CoYaC86bbPTqcwZaqpK7PNAxc=">AAAB33icbZBLSwMxFIXv+Ky1anXrJlgEV2XGjS4FNy4r2ge0Q8mkd9rYTGZI7ghl6E9w40IR/5U7/43pY6GtBwIf5yTk3hNlSlry/W9vY3Nre2e3tFferxwcHlWPKy2b5kZgU6QqNZ2IW1RSY5MkKexkBnkSKWxH49tZ3n5GY2WqH2mSYZjwoZaxFJyc9ZD3n/rVml/352LrECyhBks1+tWv3iAVeYKahOLWdgM/o7DghqRQOC33cosZF2M+xK5DzRO0YTEfdcrOnTNgcWrc0cTm7u8XBU+snSSRu5lwGtnVbGb+l3Vziq/DQuosJ9Ri8VGcK0Ypm+3NBtKgIDVxwIWRblYmRtxwQa6dsishWF15HVqX9cDxvQ8lOIUzuIAAruAG7qABTRAwhBd4g3dPea/ex6KuDW/Z2wn8kff5Az/cjIE=</latexit><latexit sha1_base64="46aLFBoWDxH9bhturSsrpEjo9qI=">AAAB6nicbZA9T8MwEIYv5auUrwAji0WFxFQlLDBWsDAWQT+kNqoc99KaOk5kO0hV1J/AwgBCrPwiNv4NbpsBWl7J0qP37uS7N0wF18bzvp3S2vrG5lZ5u7Kzu7d/4B4etXSSKYZNlohEdUKqUXCJTcONwE6qkMahwHY4vpnV20+oNE/kg5mkGMR0KHnEGTXWus/6j3236tW8ucgq+AVUoVCj7371BgnLYpSGCap11/dSE+RUGc4ETiu9TGNK2ZgOsWtR0hh1kM9XnZIz6wxIlCj7pCFz9/dETmOtJ3FoO2NqRnq5NjP/q3UzE10FOZdpZlCyxUdRJohJyOxuMuAKmRETC5QpbnclbEQVZcamU7Eh+Msnr0LrouZbvvOq9esijjKcwCmcgw+XUIdbaEATGAzhGV7hzRHOi/PufCxaS04xcwx/5Hz+AFxEjdI=</latexit><latexit sha1_base64="TCD8Huirq8KGLFYVIYoP6bPo/BI=">AAAB6nicbZBNS8NAEIYn9avWr6pHL4tF8FQSEeqx6MVjRfsBbSib7aRdu9mE3Y1QQn+CFw+KePUXefPfuG1z0NYXFh7emWFn3iARXBvX/XYKa+sbm1vF7dLO7t7+QfnwqKXjVDFssljEqhNQjYJLbBpuBHYShTQKBLaD8c2s3n5CpXksH8wkQT+iQ8lDzqix1n3af+yXK27VnYusgpdDBXI1+uWv3iBmaYTSMEG17npuYvyMKsOZwGmpl2pMKBvTIXYtShqh9rP5qlNyZp0BCWNlnzRk7v6eyGik9SQKbGdEzUgv12bmf7VuasIrP+MySQ1KtvgoTAUxMZndTQZcITNiYoEyxe2uhI2ooszYdEo2BG/55FVoXVQ9y3eXlfp1HkcRTuAUzsGDGtThFhrQBAZDeIZXeHOE8+K8Ox+L1oKTzxzDHzmfP12EjdY=</latexit><latexit sha1_base64="TCD8Huirq8KGLFYVIYoP6bPo/BI=">AAAB6nicbZBNS8NAEIYn9avWr6pHL4tF8FQSEeqx6MVjRfsBbSib7aRdu9mE3Y1QQn+CFw+KePUXefPfuG1z0NYXFh7emWFn3iARXBvX/XYKa+sbm1vF7dLO7t7+QfnwqKXjVDFssljEqhNQjYJLbBpuBHYShTQKBLaD8c2s3n5CpXksH8wkQT+iQ8lDzqix1n3af+yXK27VnYusgpdDBXI1+uWv3iBmaYTSMEG17npuYvyMKsOZwGmpl2pMKBvTIXYtShqh9rP5qlNyZp0BCWNlnzRk7v6eyGik9SQKbGdEzUgv12bmf7VuasIrP+MySQ1KtvgoTAUxMZndTQZcITNiYoEyxe2uhI2ooszYdEo2BG/55FVoXVQ9y3eXlfp1HkcRTuAUzsGDGtThFhrQBAZDeIZXeHOE8+K8Ox+L1oKTzxzDHzmfP12EjdY=</latexit><latexit sha1_base64="TCD8Huirq8KGLFYVIYoP6bPo/BI=">AAAB6nicbZBNS8NAEIYn9avWr6pHL4tF8FQSEeqx6MVjRfsBbSib7aRdu9mE3Y1QQn+CFw+KePUXefPfuG1z0NYXFh7emWFn3iARXBvX/XYKa+sbm1vF7dLO7t7+QfnwqKXjVDFssljEqhNQjYJLbBpuBHYShTQKBLaD8c2s3n5CpXksH8wkQT+iQ8lDzqix1n3af+yXK27VnYusgpdDBXI1+uWv3iBmaYTSMEG17npuYvyMKsOZwGmpl2pMKBvTIXYtShqh9rP5qlNyZp0BCWNlnzRk7v6eyGik9SQKbGdEzUgv12bmf7VuasIrP+MySQ1KtvgoTAUxMZndTQZcITNiYoEyxe2uhI2ooszYdEo2BG/55FVoXVQ9y3eXlfp1HkcRTuAUzsGDGtThFhrQBAZDeIZXeHOE8+K8Ox+L1oKTzxzDHzmfP12EjdY=</latexit><latexit sha1_base64="TCD8Huirq8KGLFYVIYoP6bPo/BI=">AAAB6nicbZBNS8NAEIYn9avWr6pHL4tF8FQSEeqx6MVjRfsBbSib7aRdu9mE3Y1QQn+CFw+KePUXefPfuG1z0NYXFh7emWFn3iARXBvX/XYKa+sbm1vF7dLO7t7+QfnwqKXjVDFssljEqhNQjYJLbBpuBHYShTQKBLaD8c2s3n5CpXksH8wkQT+iQ8lDzqix1n3af+yXK27VnYusgpdDBXI1+uWv3iBmaYTSMEG17npuYvyMKsOZwGmpl2pMKBvTIXYtShqh9rP5qlNyZp0BCWNlnzRk7v6eyGik9SQKbGdEzUgv12bmf7VuasIrP+MySQ1KtvgoTAUxMZndTQZcITNiYoEyxe2uhI2ooszYdEo2BG/55FVoXVQ9y3eXlfp1HkcRTuAUzsGDGtThFhrQBAZDeIZXeHOE8+K8Ox+L1oKTzxzDHzmfP12EjdY=</latexit><latexit sha1_base64="TCD8Huirq8KGLFYVIYoP6bPo/BI=">AAAB6nicbZBNS8NAEIYn9avWr6pHL4tF8FQSEeqx6MVjRfsBbSib7aRdu9mE3Y1QQn+CFw+KePUXefPfuG1z0NYXFh7emWFn3iARXBvX/XYKa+sbm1vF7dLO7t7+QfnwqKXjVDFssljEqhNQjYJLbBpuBHYShTQKBLaD8c2s3n5CpXksH8wkQT+iQ8lDzqix1n3af+yXK27VnYusgpdDBXI1+uWv3iBmaYTSMEG17npuYvyMKsOZwGmpl2pMKBvTIXYtShqh9rP5qlNyZp0BCWNlnzRk7v6eyGik9SQKbGdEzUgv12bmf7VuasIrP+MySQ1KtvgoTAUxMZndTQZcITNiYoEyxe2uhI2ooszYdEo2BG/55FVoXVQ9y3eXlfp1HkcRTuAUzsGDGtThFhrQBAZDeIZXeHOE8+K8Ox+L1oKTzxzDHzmfP12EjdY=</latexit><latexit sha1_base64="TCD8Huirq8KGLFYVIYoP6bPo/BI=">AAAB6nicbZBNS8NAEIYn9avWr6pHL4tF8FQSEeqx6MVjRfsBbSib7aRdu9mE3Y1QQn+CFw+KePUXefPfuG1z0NYXFh7emWFn3iARXBvX/XYKa+sbm1vF7dLO7t7+QfnwqKXjVDFssljEqhNQjYJLbBpuBHYShTQKBLaD8c2s3n5CpXksH8wkQT+iQ8lDzqix1n3af+yXK27VnYusgpdDBXI1+uWv3iBmaYTSMEG17npuYvyMKsOZwGmpl2pMKBvTIXYtShqh9rP5qlNyZp0BCWNlnzRk7v6eyGik9SQKbGdEzUgv12bmf7VuasIrP+MySQ1KtvgoTAUxMZndTQZcITNiYoEyxe2uhI2ooszYdEo2BG/55FVoXVQ9y3eXlfp1HkcRTuAUzsGDGtThFhrQBAZDeIZXeHOE8+K8Ox+L1oKTzxzDHzmfP12EjdY=</latexit>

wj � N (0, �2w); b � N (0, �2

b )<latexit sha1_base64="KAfgq2qtdghwBAtJmW5k1oTkPxw=">AAACL3icbVBbS8MwGE29znmr+uhLcAgbyGiHoODLUBCfZIK7wFpLmmVbXNKWJHWMsn/ki39lLyKK+Oq/MN2G6LYDgcM55yPfd/yIUaks681YWl5ZXVvPbGQ3t7Z3ds29/ZoMY4FJFYcsFA0fScJoQKqKKkYakSCI+4zU/d5V6tefiJA0DO7VICIuR52AtilGSkueed33HqEjKYcOR6qLEUtuh3nrJNU6HD2UvH7hAkJ/UeY34hc8M2cVrTHgPLGnJAemqHjmyGmFOOYkUJghKZu2FSk3QUJRzMgw68SSRAj3UIc0NQ0QJ9JNxvcO4bFWWrAdCv0CBcfq34kEcSkH3NfJdGE566XiIq8Zq/a5m9AgihUJ8OSjdsygCmFaHmxRQbBiA00QFlTvCnEXCYSVrjirS7BnT54ntVLR1vzuNFe+nNaRAYfgCOSBDc5AGdyACqgCDJ7BCLyDD+PFeDU+ja9JdMmYzhyAfzC+fwA3OadR</latexit><latexit sha1_base64="KAfgq2qtdghwBAtJmW5k1oTkPxw=">AAACL3icbVBbS8MwGE29znmr+uhLcAgbyGiHoODLUBCfZIK7wFpLmmVbXNKWJHWMsn/ki39lLyKK+Oq/MN2G6LYDgcM55yPfd/yIUaks681YWl5ZXVvPbGQ3t7Z3ds29/ZoMY4FJFYcsFA0fScJoQKqKKkYakSCI+4zU/d5V6tefiJA0DO7VICIuR52AtilGSkueed33HqEjKYcOR6qLEUtuh3nrJNU6HD2UvH7hAkJ/UeY34hc8M2cVrTHgPLGnJAemqHjmyGmFOOYkUJghKZu2FSk3QUJRzMgw68SSRAj3UIc0NQ0QJ9JNxvcO4bFWWrAdCv0CBcfq34kEcSkH3NfJdGE566XiIq8Zq/a5m9AgihUJ8OSjdsygCmFaHmxRQbBiA00QFlTvCnEXCYSVrjirS7BnT54ntVLR1vzuNFe+nNaRAYfgCOSBDc5AGdyACqgCDJ7BCLyDD+PFeDU+ja9JdMmYzhyAfzC+fwA3OadR</latexit><latexit sha1_base64="KAfgq2qtdghwBAtJmW5k1oTkPxw=">AAACL3icbVBbS8MwGE29znmr+uhLcAgbyGiHoODLUBCfZIK7wFpLmmVbXNKWJHWMsn/ki39lLyKK+Oq/MN2G6LYDgcM55yPfd/yIUaks681YWl5ZXVvPbGQ3t7Z3ds29/ZoMY4FJFYcsFA0fScJoQKqKKkYakSCI+4zU/d5V6tefiJA0DO7VICIuR52AtilGSkueed33HqEjKYcOR6qLEUtuh3nrJNU6HD2UvH7hAkJ/UeY34hc8M2cVrTHgPLGnJAemqHjmyGmFOOYkUJghKZu2FSk3QUJRzMgw68SSRAj3UIc0NQ0QJ9JNxvcO4bFWWrAdCv0CBcfq34kEcSkH3NfJdGE566XiIq8Zq/a5m9AgihUJ8OSjdsygCmFaHmxRQbBiA00QFlTvCnEXCYSVrjirS7BnT54ntVLR1vzuNFe+nNaRAYfgCOSBDc5AGdyACqgCDJ7BCLyDD+PFeDU+ja9JdMmYzhyAfzC+fwA3OadR</latexit><latexit sha1_base64="KAfgq2qtdghwBAtJmW5k1oTkPxw=">AAACL3icbVBbS8MwGE29znmr+uhLcAgbyGiHoODLUBCfZIK7wFpLmmVbXNKWJHWMsn/ki39lLyKK+Oq/MN2G6LYDgcM55yPfd/yIUaks681YWl5ZXVvPbGQ3t7Z3ds29/ZoMY4FJFYcsFA0fScJoQKqKKkYakSCI+4zU/d5V6tefiJA0DO7VICIuR52AtilGSkueed33HqEjKYcOR6qLEUtuh3nrJNU6HD2UvH7hAkJ/UeY34hc8M2cVrTHgPLGnJAemqHjmyGmFOOYkUJghKZu2FSk3QUJRzMgw68SSRAj3UIc0NQ0QJ9JNxvcO4bFWWrAdCv0CBcfq34kEcSkH3NfJdGE566XiIq8Zq/a5m9AgihUJ8OSjdsygCmFaHmxRQbBiA00QFlTvCnEXCYSVrjirS7BnT54ntVLR1vzuNFe+nNaRAYfgCOSBDc5AGdyACqgCDJ7BCLyDD+PFeDU+ja9JdMmYzhyAfzC+fwA3OadR</latexit>

++

f(x) = b +J�

j=1

wj�(x; uj)<latexit sha1_base64="F19YSTSGZfbk19qglgh9199R8UE=">AAACEXicbZDLSsNAFIYnXmu9RV26GSxCilASERSkUHQjrirYCzQxTKaTdtrJhZmJtoS+ghtfxY0LRdy6c+fbOG2z0NYfBj7+cw5nzu/FjAppmt/awuLS8spqbi2/vrG5ta3v7NZFlHBMajhiEW96SBBGQ1KTVDLSjDlBgcdIw+tfjuuNe8IFjcJbOYyJE6BOSH2KkVSWqxu+MSjCMvTgEbRFErhpr2yN7q7hg9uDdtylxuAcJm6v6OoFs2ROBOfByqAAMlVd/ctuRzgJSCgxQ0K0LDOWToq4pJiRUd5OBIkR7qMOaSkMUUCEk04uGsFD5bShH3H1Qgkn7u+JFAVCDANPdQZIdsVsbWz+V2sl0j9zUhrGiSQhni7yEwZlBMfxwDblBEs2VIAwp+qvEHcRR1iqEPMqBGv25HmoH5csxTcnhcpFFkcO7IMDYAALnIIKuAJVUAMYPIJn8AretCftRXvXPqatC1o2swf+SPv8ASzvmrs=</latexit><latexit sha1_base64="F19YSTSGZfbk19qglgh9199R8UE=">AAACEXicbZDLSsNAFIYnXmu9RV26GSxCilASERSkUHQjrirYCzQxTKaTdtrJhZmJtoS+ghtfxY0LRdy6c+fbOG2z0NYfBj7+cw5nzu/FjAppmt/awuLS8spqbi2/vrG5ta3v7NZFlHBMajhiEW96SBBGQ1KTVDLSjDlBgcdIw+tfjuuNe8IFjcJbOYyJE6BOSH2KkVSWqxu+MSjCMvTgEbRFErhpr2yN7q7hg9uDdtylxuAcJm6v6OoFs2ROBOfByqAAMlVd/ctuRzgJSCgxQ0K0LDOWToq4pJiRUd5OBIkR7qMOaSkMUUCEk04uGsFD5bShH3H1Qgkn7u+JFAVCDANPdQZIdsVsbWz+V2sl0j9zUhrGiSQhni7yEwZlBMfxwDblBEs2VIAwp+qvEHcRR1iqEPMqBGv25HmoH5csxTcnhcpFFkcO7IMDYAALnIIKuAJVUAMYPIJn8AretCftRXvXPqatC1o2swf+SPv8ASzvmrs=</latexit><latexit sha1_base64="F19YSTSGZfbk19qglgh9199R8UE=">AAACEXicbZDLSsNAFIYnXmu9RV26GSxCilASERSkUHQjrirYCzQxTKaTdtrJhZmJtoS+ghtfxY0LRdy6c+fbOG2z0NYfBj7+cw5nzu/FjAppmt/awuLS8spqbi2/vrG5ta3v7NZFlHBMajhiEW96SBBGQ1KTVDLSjDlBgcdIw+tfjuuNe8IFjcJbOYyJE6BOSH2KkVSWqxu+MSjCMvTgEbRFErhpr2yN7q7hg9uDdtylxuAcJm6v6OoFs2ROBOfByqAAMlVd/ctuRzgJSCgxQ0K0LDOWToq4pJiRUd5OBIkR7qMOaSkMUUCEk04uGsFD5bShH3H1Qgkn7u+JFAVCDANPdQZIdsVsbWz+V2sl0j9zUhrGiSQhni7yEwZlBMfxwDblBEs2VIAwp+qvEHcRR1iqEPMqBGv25HmoH5csxTcnhcpFFkcO7IMDYAALnIIKuAJVUAMYPIJn8AretCftRXvXPqatC1o2swf+SPv8ASzvmrs=</latexit><latexit sha1_base64="F19YSTSGZfbk19qglgh9199R8UE=">AAACEXicbZDLSsNAFIYnXmu9RV26GSxCilASERSkUHQjrirYCzQxTKaTdtrJhZmJtoS+ghtfxY0LRdy6c+fbOG2z0NYfBj7+cw5nzu/FjAppmt/awuLS8spqbi2/vrG5ta3v7NZFlHBMajhiEW96SBBGQ1KTVDLSjDlBgcdIw+tfjuuNe8IFjcJbOYyJE6BOSH2KkVSWqxu+MSjCMvTgEbRFErhpr2yN7q7hg9uDdtylxuAcJm6v6OoFs2ROBOfByqAAMlVd/ctuRzgJSCgxQ0K0LDOWToq4pJiRUd5OBIkR7qMOaSkMUUCEk04uGsFD5bShH3H1Qgkn7u+JFAVCDANPdQZIdsVsbWz+V2sl0j9zUhrGiSQhni7yEwZlBMfxwDblBEs2VIAwp+qvEHcRR1iqEPMqBGv25HmoH5csxTcnhcpFFkcO7IMDYAALnIIKuAJVUAMYPIJn8AretCftRXvXPqatC1o2swf+SPv8ASzvmrs=</latexit>

wj<latexit sha1_base64="HwPGbp1zTlZgN+Y1A19X6nusNsU=">AAAB6nicbZBNS8NAEIYn9avWr6pHL4tF8FQSEfRY9OKxov2ANpTNdtKu3WzC7kYpoT/BiwdFvPqLvPlv3LY5aOsLCw/vzLAzb5AIro3rfjuFldW19Y3iZmlre2d3r7x/0NRxqhg2WCxi1Q6oRsElNgw3AtuJQhoFAlvB6Hpabz2i0jyW92acoB/RgeQhZ9RY6+6p99ArV9yqOxNZBi+HCuSq98pf3X7M0gilYYJq3fHcxPgZVYYzgZNSN9WYUDaiA+xYlDRC7WezVSfkxDp9EsbKPmnIzP09kdFI63EU2M6ImqFerE3N/2qd1ISXfsZlkhqUbP5RmApiYjK9m/S5QmbE2AJlittdCRtSRZmx6ZRsCN7iycvQPKt6lm/PK7WrPI4iHMExnIIHF1CDG6hDAxgM4Ble4c0Rzovz7nzMWwtOPnMIf+R8/gBgkI3Y</latexit><latexit sha1_base64="HwPGbp1zTlZgN+Y1A19X6nusNsU=">AAAB6nicbZBNS8NAEIYn9avWr6pHL4tF8FQSEfRY9OKxov2ANpTNdtKu3WzC7kYpoT/BiwdFvPqLvPlv3LY5aOsLCw/vzLAzb5AIro3rfjuFldW19Y3iZmlre2d3r7x/0NRxqhg2WCxi1Q6oRsElNgw3AtuJQhoFAlvB6Hpabz2i0jyW92acoB/RgeQhZ9RY6+6p99ArV9yqOxNZBi+HCuSq98pf3X7M0gilYYJq3fHcxPgZVYYzgZNSN9WYUDaiA+xYlDRC7WezVSfkxDp9EsbKPmnIzP09kdFI63EU2M6ImqFerE3N/2qd1ISXfsZlkhqUbP5RmApiYjK9m/S5QmbE2AJlittdCRtSRZmx6ZRsCN7iycvQPKt6lm/PK7WrPI4iHMExnIIHF1CDG6hDAxgM4Ble4c0Rzovz7nzMWwtOPnMIf+R8/gBgkI3Y</latexit><latexit sha1_base64="HwPGbp1zTlZgN+Y1A19X6nusNsU=">AAAB6nicbZBNS8NAEIYn9avWr6pHL4tF8FQSEfRY9OKxov2ANpTNdtKu3WzC7kYpoT/BiwdFvPqLvPlv3LY5aOsLCw/vzLAzb5AIro3rfjuFldW19Y3iZmlre2d3r7x/0NRxqhg2WCxi1Q6oRsElNgw3AtuJQhoFAlvB6Hpabz2i0jyW92acoB/RgeQhZ9RY6+6p99ArV9yqOxNZBi+HCuSq98pf3X7M0gilYYJq3fHcxPgZVYYzgZNSN9WYUDaiA+xYlDRC7WezVSfkxDp9EsbKPmnIzP09kdFI63EU2M6ImqFerE3N/2qd1ISXfsZlkhqUbP5RmApiYjK9m/S5QmbE2AJlittdCRtSRZmx6ZRsCN7iycvQPKt6lm/PK7WrPI4iHMExnIIHF1CDG6hDAxgM4Ble4c0Rzovz7nzMWwtOPnMIf+R8/gBgkI3Y</latexit><latexit sha1_base64="HwPGbp1zTlZgN+Y1A19X6nusNsU=">AAAB6nicbZBNS8NAEIYn9avWr6pHL4tF8FQSEfRY9OKxov2ANpTNdtKu3WzC7kYpoT/BiwdFvPqLvPlv3LY5aOsLCw/vzLAzb5AIro3rfjuFldW19Y3iZmlre2d3r7x/0NRxqhg2WCxi1Q6oRsElNgw3AtuJQhoFAlvB6Hpabz2i0jyW92acoB/RgeQhZ9RY6+6p99ArV9yqOxNZBi+HCuSq98pf3X7M0gilYYJq3fHcxPgZVYYzgZNSN9WYUDaiA+xYlDRC7WezVSfkxDp9EsbKPmnIzP09kdFI63EU2M6ImqFerE3N/2qd1ISXfsZlkhqUbP5RmApiYjK9m/S5QmbE2AJlittdCRtSRZmx6ZRsCN7iycvQPKt6lm/PK7WrPI4iHMExnIIHF1CDG6hDAxgM4Ble4c0Rzovz7nzMWwtOPnMIf+R8/gBgkI3Y</latexit>

Ew[f(x)f(x�)] = �2b + J�2

wEu[�(x; uj)�(x�; uj)]<latexit sha1_base64="HV75SHiVfQdFEwffa+6e7+S+1Lk=">AAACQ3icbVBdaxNBFJ2tbY3RtlEffRkaShIKZbcUFEQoSkB8qmCa4O52mZ29m0w7+8HMXZuw5L/1pX/AN/+ALz4o4qvgZDeUNvXADGfOuZd754S5FBpt+5u19mB9Y/Nh41Hz8ZOt7Z3W02enOisUhwHPZKZGIdMgRQoDFChhlCtgSShhGF68W/jDL6C0yNJPOMvBT9g4FbHgDI0UtD73g0s37k571Fydnk/fUE+LccLODoOQ7lMPYYrVmFJBNC8/zG/sS9oPCtfLJ6I7fU2L4LxX80798INW2z6wK9D7xFmSNlniJGh99aKMFwmkyCXT2nXsHP2SKRRcwrzpFRpyxi/YGFxDU5aA9stquTndM0pE40yZkyKt1NsdJUu0niWhqUwYTvSqtxD/57kFxq/8UqR5gZDyelBcSIoZXQRKI6GAo5wZwrgSZlfKJ0wxjib2pgnBWf3yfXJ6eOAY/vGoffx2GUeDvCC7pEsc8pIck/fkhAwIJ1fkO/lJflnX1g/rt/WnLl2zlj3PyR1Yf/8BgZyu6w==</latexit><latexit sha1_base64="HV75SHiVfQdFEwffa+6e7+S+1Lk=">AAACQ3icbVBdaxNBFJ2tbY3RtlEffRkaShIKZbcUFEQoSkB8qmCa4O52mZ29m0w7+8HMXZuw5L/1pX/AN/+ALz4o4qvgZDeUNvXADGfOuZd754S5FBpt+5u19mB9Y/Nh41Hz8ZOt7Z3W02enOisUhwHPZKZGIdMgRQoDFChhlCtgSShhGF68W/jDL6C0yNJPOMvBT9g4FbHgDI0UtD73g0s37k571Fydnk/fUE+LccLODoOQ7lMPYYrVmFJBNC8/zG/sS9oPCtfLJ6I7fU2L4LxX80798INW2z6wK9D7xFmSNlniJGh99aKMFwmkyCXT2nXsHP2SKRRcwrzpFRpyxi/YGFxDU5aA9stquTndM0pE40yZkyKt1NsdJUu0niWhqUwYTvSqtxD/57kFxq/8UqR5gZDyelBcSIoZXQRKI6GAo5wZwrgSZlfKJ0wxjib2pgnBWf3yfXJ6eOAY/vGoffx2GUeDvCC7pEsc8pIck/fkhAwIJ1fkO/lJflnX1g/rt/WnLl2zlj3PyR1Yf/8BgZyu6w==</latexit><latexit sha1_base64="HV75SHiVfQdFEwffa+6e7+S+1Lk=">AAACQ3icbVBdaxNBFJ2tbY3RtlEffRkaShIKZbcUFEQoSkB8qmCa4O52mZ29m0w7+8HMXZuw5L/1pX/AN/+ALz4o4qvgZDeUNvXADGfOuZd754S5FBpt+5u19mB9Y/Nh41Hz8ZOt7Z3W02enOisUhwHPZKZGIdMgRQoDFChhlCtgSShhGF68W/jDL6C0yNJPOMvBT9g4FbHgDI0UtD73g0s37k571Fydnk/fUE+LccLODoOQ7lMPYYrVmFJBNC8/zG/sS9oPCtfLJ6I7fU2L4LxX80798INW2z6wK9D7xFmSNlniJGh99aKMFwmkyCXT2nXsHP2SKRRcwrzpFRpyxi/YGFxDU5aA9stquTndM0pE40yZkyKt1NsdJUu0niWhqUwYTvSqtxD/57kFxq/8UqR5gZDyelBcSIoZXQRKI6GAo5wZwrgSZlfKJ0wxjib2pgnBWf3yfXJ6eOAY/vGoffx2GUeDvCC7pEsc8pIck/fkhAwIJ1fkO/lJflnX1g/rt/WnLl2zlj3PyR1Yf/8BgZyu6w==</latexit><latexit sha1_base64="HV75SHiVfQdFEwffa+6e7+S+1Lk=">AAACQ3icbVBdaxNBFJ2tbY3RtlEffRkaShIKZbcUFEQoSkB8qmCa4O52mZ29m0w7+8HMXZuw5L/1pX/AN/+ALz4o4qvgZDeUNvXADGfOuZd754S5FBpt+5u19mB9Y/Nh41Hz8ZOt7Z3W02enOisUhwHPZKZGIdMgRQoDFChhlCtgSShhGF68W/jDL6C0yNJPOMvBT9g4FbHgDI0UtD73g0s37k571Fydnk/fUE+LccLODoOQ7lMPYYrVmFJBNC8/zG/sS9oPCtfLJ6I7fU2L4LxX80798INW2z6wK9D7xFmSNlniJGh99aKMFwmkyCXT2nXsHP2SKRRcwrzpFRpyxi/YGFxDU5aA9stquTndM0pE40yZkyKt1NsdJUu0niWhqUwYTvSqtxD/57kFxq/8UqR5gZDyelBcSIoZXQRKI6GAo5wZwrgSZlfKJ0wxjib2pgnBWf3yfXJ6eOAY/vGoffx2GUeDvCC7pEsc8pIck/fkhAwIJ1fkO/lJflnX1g/rt/WnLl2zlj3PyR1Yf/8BgZyu6w==</latexit>

!17

C. Williams, NIPS 1997, R. Neal, LNS, 1996�2w =

a

J<latexit sha1_base64="7Gh/5HiD6omC/cBNZfWytkKa9jI=">AAACAXicbZDLSsNAFIYnXmu9Rd0IbgaL4KokRdCNUHQjrirYCzQxTKaTdujMJMxMlBLixldx40IRt76FO9/GaZuFtv4w8PGfczhz/jBhVGnH+bYWFpeWV1ZLa+X1jc2tbXtnt6XiVGLSxDGLZSdEijAqSFNTzUgnkQTxkJF2OLwc19v3RCoai1s9SojPUV/QiGKkjRXY+56ifY7uasEDPIdeJBHOUJ5d54FdcarORHAe3AIqoFAjsL+8XoxTToTGDCnVdZ1E+xmSmmJG8rKXKpIgPER90jUoECfKzyYX5PDIOD0YxdI8oeHE/T2RIa7UiIemkyM9ULO1sflfrZvq6MzPqEhSTQSeLopSBnUMx3HAHpUEazYygLCk5q8QD5BJQZvQyiYEd/bkeWjVqq7hm5NK/aKIowQOwCE4Bi44BXVwBRqgCTB4BM/gFbxZT9aL9W59TFsXrGJmD/yR9fkD2AaWfg==</latexit><latexit sha1_base64="7Gh/5HiD6omC/cBNZfWytkKa9jI=">AAACAXicbZDLSsNAFIYnXmu9Rd0IbgaL4KokRdCNUHQjrirYCzQxTKaTdujMJMxMlBLixldx40IRt76FO9/GaZuFtv4w8PGfczhz/jBhVGnH+bYWFpeWV1ZLa+X1jc2tbXtnt6XiVGLSxDGLZSdEijAqSFNTzUgnkQTxkJF2OLwc19v3RCoai1s9SojPUV/QiGKkjRXY+56ifY7uasEDPIdeJBHOUJ5d54FdcarORHAe3AIqoFAjsL+8XoxTToTGDCnVdZ1E+xmSmmJG8rKXKpIgPER90jUoECfKzyYX5PDIOD0YxdI8oeHE/T2RIa7UiIemkyM9ULO1sflfrZvq6MzPqEhSTQSeLopSBnUMx3HAHpUEazYygLCk5q8QD5BJQZvQyiYEd/bkeWjVqq7hm5NK/aKIowQOwCE4Bi44BXVwBRqgCTB4BM/gFbxZT9aL9W59TFsXrGJmD/yR9fkD2AaWfg==</latexit><latexit sha1_base64="7Gh/5HiD6omC/cBNZfWytkKa9jI=">AAACAXicbZDLSsNAFIYnXmu9Rd0IbgaL4KokRdCNUHQjrirYCzQxTKaTdujMJMxMlBLixldx40IRt76FO9/GaZuFtv4w8PGfczhz/jBhVGnH+bYWFpeWV1ZLa+X1jc2tbXtnt6XiVGLSxDGLZSdEijAqSFNTzUgnkQTxkJF2OLwc19v3RCoai1s9SojPUV/QiGKkjRXY+56ifY7uasEDPIdeJBHOUJ5d54FdcarORHAe3AIqoFAjsL+8XoxTToTGDCnVdZ1E+xmSmmJG8rKXKpIgPER90jUoECfKzyYX5PDIOD0YxdI8oeHE/T2RIa7UiIemkyM9ULO1sflfrZvq6MzPqEhSTQSeLopSBnUMx3HAHpUEazYygLCk5q8QD5BJQZvQyiYEd/bkeWjVqq7hm5NK/aKIowQOwCE4Bi44BXVwBRqgCTB4BM/gFbxZT9aL9W59TFsXrGJmD/yR9fkD2AaWfg==</latexit><latexit sha1_base64="7Gh/5HiD6omC/cBNZfWytkKa9jI=">AAACAXicbZDLSsNAFIYnXmu9Rd0IbgaL4KokRdCNUHQjrirYCzQxTKaTdujMJMxMlBLixldx40IRt76FO9/GaZuFtv4w8PGfczhz/jBhVGnH+bYWFpeWV1ZLa+X1jc2tbXtnt6XiVGLSxDGLZSdEijAqSFNTzUgnkQTxkJF2OLwc19v3RCoai1s9SojPUV/QiGKkjRXY+56ifY7uasEDPIdeJBHOUJ5d54FdcarORHAe3AIqoFAjsL+8XoxTToTGDCnVdZ1E+xmSmmJG8rKXKpIgPER90jUoECfKzyYX5PDIOD0YxdI8oeHE/T2RIa7UiIemkyM9ULO1sflfrZvq6MzPqEhSTQSeLopSBnUMx3HAHpUEazYygLCk5q8QD5BJQZvQyiYEd/bkeWjVqq7hm5NK/aKIowQOwCE4Bi44BXVwBRqgCTB4BM/gFbxZT9aL9W59TFsXrGJmD/yR9fkD2AaWfg==</latexit>

Could scale by J

orForce J to be small, by turning units off.

Page 18: Horseshoe Priors for Bayesian Neural Networks...Horseshoe Priors for Bayesian Neural Networks Soumya Ghosh Jiayu Yao Finale Doshi-Velez IBM Research MIT-IBM Watson AI lab Harvard Harvard

Horseshoe Priors for Model SelectionThe horseshoe prior is a scale mixture of normals:

!18

wk � N (0, �2kv2)

�k � C+(0, 1)<latexit sha1_base64="ykTKHfExyyy2EuAY7vuNzHbCor4=">AAACK3icbVDLSgMxFM3UVx1fVZdugkVpUcpMEXRZ2o0rqWAf0GmHTJq2YZKZIclUytD/ceOvuNCFD9z6H6btLLR6IHA451xu7vEiRqWyrHcjs7K6tr6R3TS3tnd293L7B00ZxgKTBg5ZKNoekoTRgDQUVYy0I0EQ9xhpeX5t5rfGREgaBndqEpEuR8OADihGSkturnrv+vDUkZRDhyM1woglN9OCdQ4dhWLX75XhuFcuQscxF0IarvWSs3nMLrq5vFWy5oB/iZ2SPEhRd3PPTj/EMSeBwgxJ2bGtSHUTJBTFjExNJ5YkQthHQ9LRNECcyG4yv3UKT7TSh4NQ6BcoOFd/TiSISznhnk7O7pHL3kz8z+vEanDVTWgQxYoEeLFoEDOoQjgrDvapIFixiSYIC6r/CvEICYSVrtfUJdjLJ/8lzXLJtkr27UW+Uk3ryIIjcAwKwAaXoAKuQR00AAYP4Am8gjfj0XgxPozPRTRjpDOH4BeMr2+HVqQZ</latexit><latexit sha1_base64="ykTKHfExyyy2EuAY7vuNzHbCor4=">AAACK3icbVDLSgMxFM3UVx1fVZdugkVpUcpMEXRZ2o0rqWAf0GmHTJq2YZKZIclUytD/ceOvuNCFD9z6H6btLLR6IHA451xu7vEiRqWyrHcjs7K6tr6R3TS3tnd293L7B00ZxgKTBg5ZKNoekoTRgDQUVYy0I0EQ9xhpeX5t5rfGREgaBndqEpEuR8OADihGSkturnrv+vDUkZRDhyM1woglN9OCdQ4dhWLX75XhuFcuQscxF0IarvWSs3nMLrq5vFWy5oB/iZ2SPEhRd3PPTj/EMSeBwgxJ2bGtSHUTJBTFjExNJ5YkQthHQ9LRNECcyG4yv3UKT7TSh4NQ6BcoOFd/TiSISznhnk7O7pHL3kz8z+vEanDVTWgQxYoEeLFoEDOoQjgrDvapIFixiSYIC6r/CvEICYSVrtfUJdjLJ/8lzXLJtkr27UW+Uk3ryIIjcAwKwAaXoAKuQR00AAYP4Am8gjfj0XgxPozPRTRjpDOH4BeMr2+HVqQZ</latexit><latexit sha1_base64="ykTKHfExyyy2EuAY7vuNzHbCor4=">AAACK3icbVDLSgMxFM3UVx1fVZdugkVpUcpMEXRZ2o0rqWAf0GmHTJq2YZKZIclUytD/ceOvuNCFD9z6H6btLLR6IHA451xu7vEiRqWyrHcjs7K6tr6R3TS3tnd293L7B00ZxgKTBg5ZKNoekoTRgDQUVYy0I0EQ9xhpeX5t5rfGREgaBndqEpEuR8OADihGSkturnrv+vDUkZRDhyM1woglN9OCdQ4dhWLX75XhuFcuQscxF0IarvWSs3nMLrq5vFWy5oB/iZ2SPEhRd3PPTj/EMSeBwgxJ2bGtSHUTJBTFjExNJ5YkQthHQ9LRNECcyG4yv3UKT7TSh4NQ6BcoOFd/TiSISznhnk7O7pHL3kz8z+vEanDVTWgQxYoEeLFoEDOoQjgrDvapIFixiSYIC6r/CvEICYSVrtfUJdjLJ/8lzXLJtkr27UW+Uk3ryIIjcAwKwAaXoAKuQR00AAYP4Am8gjfj0XgxPozPRTRjpDOH4BeMr2+HVqQZ</latexit><latexit sha1_base64="hP+6LrUf2d3tZaldqaQQvEKMXyw=">AAAB2XicbZDNSgMxFIXv1L86Vq1rN8EiuCozbnQpuHFZwbZCO5RM5k4bmskMyR2hDH0BF25EfC93vo3pz0JbDwQ+zknIvSculLQUBN9ebWd3b/+gfugfNfzjk9Nmo2fz0gjsilzl5jnmFpXU2CVJCp8LgzyLFfbj6f0i77+gsTLXTzQrMMr4WMtUCk7O6oyaraAdLMW2IVxDC9YaNb+GSS7KDDUJxa0dhEFBUcUNSaFw7g9LiwUXUz7GgUPNM7RRtRxzzi6dk7A0N+5oYkv394uKZ9bOstjdzDhN7Ga2MP/LBiWlt1EldVESarH6KC0Vo5wtdmaJNChIzRxwYaSblYkJN1yQa8Z3HYSbG29D77odBu3wMYA6nMMFXEEIN3AHD9CBLghI4BXevYn35n2suqp569LO4I+8zx84xIo4</latexit><latexit sha1_base64="4FJ6VtyJZDuaDsO/MagSlyPZyRI=">AAACIHicbVBLTwIxGPwWX4io6NVLI9FANGSXix6NXDwZTOSRsMumWwo0dB9puxiy4f948a940IOP+GMsyx4UnaTJZGaadsaLOJPKND+M3Nr6xuZWfruwU9zd2y8dFNsyjAWhLRLyUHQ9LClnAW0ppjjtRoJi3+O0400aC78zpUKyMLhXs4g6Ph4FbMgIVlpyS9cP7gSd2pL5yPaxGhPMk9t5xTxHtsKxO+nX0bRfryLbLiyFLNzoJ2dpzKq6pbJZM1Ogv8TKSBkyNN3Siz0ISezTQBGOpexZZqScBAvFCKfzgh1LGmEywSPa0zTAPpVOknadoxOtDNAwFPoECqXqzxsJ9qWc+Z5OLvrIVW8h/uf1YjW8dBIWRLGiAVk+NIw5UiFaDIcGTFCi+EwTTATTf0VkjAUmSs9b0CNYq5X/kna9Zpk1686EPBzBMVTAggu4ghtoQgsIPMIzvMG78WS8Gp/LuXJGttsh/ILx9Q2AbKKD</latexit><latexit sha1_base64="4FJ6VtyJZDuaDsO/MagSlyPZyRI=">AAACIHicbVBLTwIxGPwWX4io6NVLI9FANGSXix6NXDwZTOSRsMumWwo0dB9puxiy4f948a940IOP+GMsyx4UnaTJZGaadsaLOJPKND+M3Nr6xuZWfruwU9zd2y8dFNsyjAWhLRLyUHQ9LClnAW0ppjjtRoJi3+O0400aC78zpUKyMLhXs4g6Ph4FbMgIVlpyS9cP7gSd2pL5yPaxGhPMk9t5xTxHtsKxO+nX0bRfryLbLiyFLNzoJ2dpzKq6pbJZM1Ogv8TKSBkyNN3Siz0ISezTQBGOpexZZqScBAvFCKfzgh1LGmEywSPa0zTAPpVOknadoxOtDNAwFPoECqXqzxsJ9qWc+Z5OLvrIVW8h/uf1YjW8dBIWRLGiAVk+NIw5UiFaDIcGTFCi+EwTTATTf0VkjAUmSs9b0CNYq5X/kna9Zpk1686EPBzBMVTAggu4ghtoQgsIPMIzvMG78WS8Gp/LuXJGttsh/ILx9Q2AbKKD</latexit><latexit sha1_base64="jpObDZubMmNoo0tuKHh9B8E6d6Q=">AAACK3icbVDLTsJAFJ3iC+sLdelmItFANKRlo0sCG1cGE3kktDTTYQoTpo/MTDGk4X/c+CsudOEjbv0Ph9KFgieZ5OScc3PnHjdiVEjD+NBya+sbm1v5bX1nd2//oHB41BZhzDFp4ZCFvOsiQRgNSEtSyUg34gT5LiMdd9yY+50J4YKGwb2cRsT20TCgHsVIKskp1B+cMTy3BPWh5SM5woglt7OScQktiWJn3K/CSb9ahpalL4Qs3OgnF2nMLDuFolExUsBVYmakCDI0ncKLNQhx7JNAYoaE6JlGJO0EcUkxIzPdigWJEB6jIekpGiCfCDtJb53BM6UMoBdy9QIJU/X3RIJ8Iaa+q5Lze8SyNxf/83qx9K7thAZRLEmAF4u8mEEZwnlxcEA5wZJNFUGYU/VXiEeIIyxVvboqwVw+eZW0qxXTqJh3RrFWz+rIgxNwCkrABFegBm5AE7QABo/gGbyBd+1Je9U+ta9FNKdlM8fgD7TvH4YWpBU=</latexit><latexit sha1_base64="ykTKHfExyyy2EuAY7vuNzHbCor4=">AAACK3icbVDLSgMxFM3UVx1fVZdugkVpUcpMEXRZ2o0rqWAf0GmHTJq2YZKZIclUytD/ceOvuNCFD9z6H6btLLR6IHA451xu7vEiRqWyrHcjs7K6tr6R3TS3tnd293L7B00ZxgKTBg5ZKNoekoTRgDQUVYy0I0EQ9xhpeX5t5rfGREgaBndqEpEuR8OADihGSkturnrv+vDUkZRDhyM1woglN9OCdQ4dhWLX75XhuFcuQscxF0IarvWSs3nMLrq5vFWy5oB/iZ2SPEhRd3PPTj/EMSeBwgxJ2bGtSHUTJBTFjExNJ5YkQthHQ9LRNECcyG4yv3UKT7TSh4NQ6BcoOFd/TiSISznhnk7O7pHL3kz8z+vEanDVTWgQxYoEeLFoEDOoQjgrDvapIFixiSYIC6r/CvEICYSVrtfUJdjLJ/8lzXLJtkr27UW+Uk3ryIIjcAwKwAaXoAKuQR00AAYP4Am8gjfj0XgxPozPRTRjpDOH4BeMr2+HVqQZ</latexit><latexit sha1_base64="ykTKHfExyyy2EuAY7vuNzHbCor4=">AAACK3icbVDLSgMxFM3UVx1fVZdugkVpUcpMEXRZ2o0rqWAf0GmHTJq2YZKZIclUytD/ceOvuNCFD9z6H6btLLR6IHA451xu7vEiRqWyrHcjs7K6tr6R3TS3tnd293L7B00ZxgKTBg5ZKNoekoTRgDQUVYy0I0EQ9xhpeX5t5rfGREgaBndqEpEuR8OADihGSkturnrv+vDUkZRDhyM1woglN9OCdQ4dhWLX75XhuFcuQscxF0IarvWSs3nMLrq5vFWy5oB/iZ2SPEhRd3PPTj/EMSeBwgxJ2bGtSHUTJBTFjExNJ5YkQthHQ9LRNECcyG4yv3UKT7TSh4NQ6BcoOFd/TiSISznhnk7O7pHL3kz8z+vEanDVTWgQxYoEeLFoEDOoQjgrDvapIFixiSYIC6r/CvEICYSVrtfUJdjLJ/8lzXLJtkr27UW+Uk3ryIIjcAwKwAaXoAKuQR00AAYP4Am8gjfj0XgxPozPRTRjpDOH4BeMr2+HVqQZ</latexit><latexit sha1_base64="ykTKHfExyyy2EuAY7vuNzHbCor4=">AAACK3icbVDLSgMxFM3UVx1fVZdugkVpUcpMEXRZ2o0rqWAf0GmHTJq2YZKZIclUytD/ceOvuNCFD9z6H6btLLR6IHA451xu7vEiRqWyrHcjs7K6tr6R3TS3tnd293L7B00ZxgKTBg5ZKNoekoTRgDQUVYy0I0EQ9xhpeX5t5rfGREgaBndqEpEuR8OADihGSkturnrv+vDUkZRDhyM1woglN9OCdQ4dhWLX75XhuFcuQscxF0IarvWSs3nMLrq5vFWy5oB/iZ2SPEhRd3PPTj/EMSeBwgxJ2bGtSHUTJBTFjExNJ5YkQthHQ9LRNECcyG4yv3UKT7TSh4NQ6BcoOFd/TiSISznhnk7O7pHL3kz8z+vEanDVTWgQxYoEeLFoEDOoQjgrDvapIFixiSYIC6r/CvEICYSVrtfUJdjLJ/8lzXLJtkr27UW+Uk3ryIIjcAwKwAaXoAKuQR00AAYP4Am8gjfj0XgxPozPRTRjpDOH4BeMr2+HVqQZ</latexit><latexit sha1_base64="ykTKHfExyyy2EuAY7vuNzHbCor4=">AAACK3icbVDLSgMxFM3UVx1fVZdugkVpUcpMEXRZ2o0rqWAf0GmHTJq2YZKZIclUytD/ceOvuNCFD9z6H6btLLR6IHA451xu7vEiRqWyrHcjs7K6tr6R3TS3tnd293L7B00ZxgKTBg5ZKNoekoTRgDQUVYy0I0EQ9xhpeX5t5rfGREgaBndqEpEuR8OADihGSkturnrv+vDUkZRDhyM1woglN9OCdQ4dhWLX75XhuFcuQscxF0IarvWSs3nMLrq5vFWy5oB/iZ2SPEhRd3PPTj/EMSeBwgxJ2bGtSHUTJBTFjExNJ5YkQthHQ9LRNECcyG4yv3UKT7TSh4NQ6BcoOFd/TiSISznhnk7O7pHL3kz8z+vEanDVTWgQxYoEeLFoEDOoQjgrDvapIFixiSYIC6r/CvEICYSVrtfUJdjLJ/8lzXLJtkr27UW+Uk3ryIIjcAwKwAaXoAKuQR00AAYP4Am8gjfj0XgxPozPRTRjpDOH4BeMr2+HVqQZ</latexit><latexit sha1_base64="ykTKHfExyyy2EuAY7vuNzHbCor4=">AAACK3icbVDLSgMxFM3UVx1fVZdugkVpUcpMEXRZ2o0rqWAf0GmHTJq2YZKZIclUytD/ceOvuNCFD9z6H6btLLR6IHA451xu7vEiRqWyrHcjs7K6tr6R3TS3tnd293L7B00ZxgKTBg5ZKNoekoTRgDQUVYy0I0EQ9xhpeX5t5rfGREgaBndqEpEuR8OADihGSkturnrv+vDUkZRDhyM1woglN9OCdQ4dhWLX75XhuFcuQscxF0IarvWSs3nMLrq5vFWy5oB/iZ2SPEhRd3PPTj/EMSeBwgxJ2bGtSHUTJBTFjExNJ5YkQthHQ9LRNECcyG4yv3UKT7TSh4NQ6BcoOFd/TiSISznhnk7O7pHL3kz8z+vEanDVTWgQxYoEeLFoEDOoQjgrDvapIFixiSYIC6r/CvEICYSVrtfUJdjLJ/8lzXLJtkr27UW+Uk3ryIIjcAwKwAaXoAKuQR00AAYP4Am8gjfj0XgxPozPRTRjpDOH4BeMr2+HVqQZ</latexit><latexit sha1_base64="ykTKHfExyyy2EuAY7vuNzHbCor4=">AAACK3icbVDLSgMxFM3UVx1fVZdugkVpUcpMEXRZ2o0rqWAf0GmHTJq2YZKZIclUytD/ceOvuNCFD9z6H6btLLR6IHA451xu7vEiRqWyrHcjs7K6tr6R3TS3tnd293L7B00ZxgKTBg5ZKNoekoTRgDQUVYy0I0EQ9xhpeX5t5rfGREgaBndqEpEuR8OADihGSkturnrv+vDUkZRDhyM1woglN9OCdQ4dhWLX75XhuFcuQscxF0IarvWSs3nMLrq5vFWy5oB/iZ2SPEhRd3PPTj/EMSeBwgxJ2bGtSHUTJBTFjExNJ5YkQthHQ9LRNECcyG4yv3UKT7TSh4NQ6BcoOFd/TiSISznhnk7O7pHL3kz8z+vEanDVTWgQxYoEeLFoEDOoQjgrDvapIFixiSYIC6r/CvEICYSVrtfUJdjLJ/8lzXLJtkr27UW+Uk3ryIIjcAwKwAaXoAKuQR00AAYP4Am8gjfj0XgxPozPRTRjpDOH4BeMr2+HVqQZ</latexit>

Page 19: Horseshoe Priors for Bayesian Neural Networks...Horseshoe Priors for Bayesian Neural Networks Soumya Ghosh Jiayu Yao Finale Doshi-Velez IBM Research MIT-IBM Watson AI lab Harvard Harvard

Group Horseshoe Priors for BNNs

For each layer , draw a global scale: For node k in layer :

− Draw a local scale for the node: − For each incident weight:

l<latexit sha1_base64="RXFmeCFAFLWm6A2HPHJ7FvtnIgg=">AAAB6HicbVBNS8NAEJ3Ur1q/qh69LBbBU0lEqMeiF48t2A9oQ9lsJ+3azSbsboQS+gu8eFDEqz/Jm//GbZuDtj4YeLw3w8y8IBFcG9f9dgobm1vbO8Xd0t7+weFR+fikreNUMWyxWMSqG1CNgktsGW4EdhOFNAoEdoLJ3dzvPKHSPJYPZpqgH9GR5CFn1FipKQblilt1FyDrxMtJBXI0BuWv/jBmaYTSMEG17nluYvyMKsOZwFmpn2pMKJvQEfYslTRC7WeLQ2fkwipDEsbKljRkof6eyGik9TQKbGdEzVivenPxP6+XmvDGz7hMUoOSLReFqSAmJvOvyZArZEZMLaFMcXsrYWOqKDM2m5INwVt9eZ20r6qeW/Wa15X6bR5HEc7gHC7BgxrU4R4a0AIGCM/wCm/Oo/PivDsfy9aCk8+cwh84nz/UCYzw</latexit><latexit sha1_base64="RXFmeCFAFLWm6A2HPHJ7FvtnIgg=">AAAB6HicbVBNS8NAEJ3Ur1q/qh69LBbBU0lEqMeiF48t2A9oQ9lsJ+3azSbsboQS+gu8eFDEqz/Jm//GbZuDtj4YeLw3w8y8IBFcG9f9dgobm1vbO8Xd0t7+weFR+fikreNUMWyxWMSqG1CNgktsGW4EdhOFNAoEdoLJ3dzvPKHSPJYPZpqgH9GR5CFn1FipKQblilt1FyDrxMtJBXI0BuWv/jBmaYTSMEG17nluYvyMKsOZwFmpn2pMKJvQEfYslTRC7WeLQ2fkwipDEsbKljRkof6eyGik9TQKbGdEzVivenPxP6+XmvDGz7hMUoOSLReFqSAmJvOvyZArZEZMLaFMcXsrYWOqKDM2m5INwVt9eZ20r6qeW/Wa15X6bR5HEc7gHC7BgxrU4R4a0AIGCM/wCm/Oo/PivDsfy9aCk8+cwh84nz/UCYzw</latexit><latexit sha1_base64="RXFmeCFAFLWm6A2HPHJ7FvtnIgg=">AAAB6HicbVBNS8NAEJ3Ur1q/qh69LBbBU0lEqMeiF48t2A9oQ9lsJ+3azSbsboQS+gu8eFDEqz/Jm//GbZuDtj4YeLw3w8y8IBFcG9f9dgobm1vbO8Xd0t7+weFR+fikreNUMWyxWMSqG1CNgktsGW4EdhOFNAoEdoLJ3dzvPKHSPJYPZpqgH9GR5CFn1FipKQblilt1FyDrxMtJBXI0BuWv/jBmaYTSMEG17nluYvyMKsOZwFmpn2pMKJvQEfYslTRC7WeLQ2fkwipDEsbKljRkof6eyGik9TQKbGdEzVivenPxP6+XmvDGz7hMUoOSLReFqSAmJvOvyZArZEZMLaFMcXsrYWOqKDM2m5INwVt9eZ20r6qeW/Wa15X6bR5HEc7gHC7BgxrU4R4a0AIGCM/wCm/Oo/PivDsfy9aCk8+cwh84nz/UCYzw</latexit><latexit sha1_base64="RXFmeCFAFLWm6A2HPHJ7FvtnIgg=">AAAB6HicbVBNS8NAEJ3Ur1q/qh69LBbBU0lEqMeiF48t2A9oQ9lsJ+3azSbsboQS+gu8eFDEqz/Jm//GbZuDtj4YeLw3w8y8IBFcG9f9dgobm1vbO8Xd0t7+weFR+fikreNUMWyxWMSqG1CNgktsGW4EdhOFNAoEdoLJ3dzvPKHSPJYPZpqgH9GR5CFn1FipKQblilt1FyDrxMtJBXI0BuWv/jBmaYTSMEG17nluYvyMKsOZwFmpn2pMKJvQEfYslTRC7WeLQ2fkwipDEsbKljRkof6eyGik9TQKbGdEzVivenPxP6+XmvDGz7hMUoOSLReFqSAmJvOvyZArZEZMLaFMcXsrYWOqKDM2m5INwVt9eZ20r6qeW/Wa15X6bR5HEc7gHC7BgxrU4R4a0AIGCM/wCm/Oo/PivDsfy9aCk8+cwh84nz/UCYzw</latexit> vl � C+(0, bg)

<latexit sha1_base64="FYjv0EmN+tYalCV/fh0u60y14QE=">AAAB/XicbVDLSsNAFL2pr1pf8bFzM1iEilISEXRZ7MZlBfuANobJdNIOziRhZlKopfgrblwo4tb/cOffOG2z0NYDFw7n3Mu99wQJZ0o7zreVW1peWV3Lrxc2Nre2d+zdvYaKU0loncQ8lq0AK8pZROuaaU5biaRYBJw2g4fqxG8OqFQsju70MKGewL2IhYxgbSTfPhj4HHUUE6h6f1pyzlDg9058u+iUnSnQInEzUoQMNd/+6nRjkgoaacKxUm3XSbQ3wlIzwum40EkVTTB5wD3aNjTCgipvNL1+jI6N0kVhLE1FGk3V3xMjLJQaisB0Cqz7at6biP957VSHV96IRUmqaURmi8KUIx2jSRSoyyQlmg8NwUQycysifSwx0SawggnBnX95kTTOy65Tdm8vipXrLI48HMIRlMCFS6jADdSgDgQe4Rle4c16sl6sd+tj1pqzspl9+APr8weW+ZNa</latexit><latexit sha1_base64="FYjv0EmN+tYalCV/fh0u60y14QE=">AAAB/XicbVDLSsNAFL2pr1pf8bFzM1iEilISEXRZ7MZlBfuANobJdNIOziRhZlKopfgrblwo4tb/cOffOG2z0NYDFw7n3Mu99wQJZ0o7zreVW1peWV3Lrxc2Nre2d+zdvYaKU0loncQ8lq0AK8pZROuaaU5biaRYBJw2g4fqxG8OqFQsju70MKGewL2IhYxgbSTfPhj4HHUUE6h6f1pyzlDg9058u+iUnSnQInEzUoQMNd/+6nRjkgoaacKxUm3XSbQ3wlIzwum40EkVTTB5wD3aNjTCgipvNL1+jI6N0kVhLE1FGk3V3xMjLJQaisB0Cqz7at6biP957VSHV96IRUmqaURmi8KUIx2jSRSoyyQlmg8NwUQycysifSwx0SawggnBnX95kTTOy65Tdm8vipXrLI48HMIRlMCFS6jADdSgDgQe4Rle4c16sl6sd+tj1pqzspl9+APr8weW+ZNa</latexit><latexit sha1_base64="FYjv0EmN+tYalCV/fh0u60y14QE=">AAAB/XicbVDLSsNAFL2pr1pf8bFzM1iEilISEXRZ7MZlBfuANobJdNIOziRhZlKopfgrblwo4tb/cOffOG2z0NYDFw7n3Mu99wQJZ0o7zreVW1peWV3Lrxc2Nre2d+zdvYaKU0loncQ8lq0AK8pZROuaaU5biaRYBJw2g4fqxG8OqFQsju70MKGewL2IhYxgbSTfPhj4HHUUE6h6f1pyzlDg9058u+iUnSnQInEzUoQMNd/+6nRjkgoaacKxUm3XSbQ3wlIzwum40EkVTTB5wD3aNjTCgipvNL1+jI6N0kVhLE1FGk3V3xMjLJQaisB0Cqz7at6biP957VSHV96IRUmqaURmi8KUIx2jSRSoyyQlmg8NwUQycysifSwx0SawggnBnX95kTTOy65Tdm8vipXrLI48HMIRlMCFS6jADdSgDgQe4Rle4c16sl6sd+tj1pqzspl9+APr8weW+ZNa</latexit><latexit sha1_base64="FYjv0EmN+tYalCV/fh0u60y14QE=">AAAB/XicbVDLSsNAFL2pr1pf8bFzM1iEilISEXRZ7MZlBfuANobJdNIOziRhZlKopfgrblwo4tb/cOffOG2z0NYDFw7n3Mu99wQJZ0o7zreVW1peWV3Lrxc2Nre2d+zdvYaKU0loncQ8lq0AK8pZROuaaU5biaRYBJw2g4fqxG8OqFQsju70MKGewL2IhYxgbSTfPhj4HHUUE6h6f1pyzlDg9058u+iUnSnQInEzUoQMNd/+6nRjkgoaacKxUm3XSbQ3wlIzwum40EkVTTB5wD3aNjTCgipvNL1+jI6N0kVhLE1FGk3V3xMjLJQaisB0Cqz7at6biP957VSHV96IRUmqaURmi8KUIx2jSRSoyyQlmg8NwUQycysifSwx0SawggnBnX95kTTOy65Tdm8vipXrLI48HMIRlMCFS6jADdSgDgQe4Rle4c16sl6sd+tj1pqzspl9+APr8weW+ZNa</latexit>

�kl � C+(0, b0)<latexit sha1_base64="J4XknDenb36+9rwby7B1axazf6g=">AAACA3icbVDLSsNAFJ3UV62vqDvdDBahopSJCLosduOygn1AE8NkOmmHziRhZiKUUHDjr7hxoYhbf8Kdf+O0zUJbD1w4nHMv994TJJwpjdC3VVhaXlldK66XNja3tnfs3b2WilNJaJPEPJadACvKWUSbmmlOO4mkWASctoNhfeK3H6hULI7u9CihnsD9iIWMYG0k3z5wNU79bMjH0FVMwPr9aQWdwcBHJ75dRlU0BVwkTk7KIEfDt7/cXkxSQSNNOFaq66BEexmWmhFOxyU3VTTBZIj7tGtohAVVXjb9YQyPjdKDYSxNRRpO1d8TGRZKjURgOgXWAzXvTcT/vG6qwysvY1GSahqR2aIw5VDHcBII7DFJieYjQzCRzNwKyQBLTLSJrWRCcOZfXiSt86qDqs7tRbl2ncdRBIfgCFSAAy5BDdyABmgCAh7BM3gFb9aT9WK9Wx+z1oKVz+yDP7A+fwBC05Xy</latexit><latexit sha1_base64="J4XknDenb36+9rwby7B1axazf6g=">AAACA3icbVDLSsNAFJ3UV62vqDvdDBahopSJCLosduOygn1AE8NkOmmHziRhZiKUUHDjr7hxoYhbf8Kdf+O0zUJbD1w4nHMv994TJJwpjdC3VVhaXlldK66XNja3tnfs3b2WilNJaJPEPJadACvKWUSbmmlOO4mkWASctoNhfeK3H6hULI7u9CihnsD9iIWMYG0k3z5wNU79bMjH0FVMwPr9aQWdwcBHJ75dRlU0BVwkTk7KIEfDt7/cXkxSQSNNOFaq66BEexmWmhFOxyU3VTTBZIj7tGtohAVVXjb9YQyPjdKDYSxNRRpO1d8TGRZKjURgOgXWAzXvTcT/vG6qwysvY1GSahqR2aIw5VDHcBII7DFJieYjQzCRzNwKyQBLTLSJrWRCcOZfXiSt86qDqs7tRbl2ncdRBIfgCFSAAy5BDdyABmgCAh7BM3gFb9aT9WK9Wx+z1oKVz+yDP7A+fwBC05Xy</latexit><latexit sha1_base64="J4XknDenb36+9rwby7B1axazf6g=">AAACA3icbVDLSsNAFJ3UV62vqDvdDBahopSJCLosduOygn1AE8NkOmmHziRhZiKUUHDjr7hxoYhbf8Kdf+O0zUJbD1w4nHMv994TJJwpjdC3VVhaXlldK66XNja3tnfs3b2WilNJaJPEPJadACvKWUSbmmlOO4mkWASctoNhfeK3H6hULI7u9CihnsD9iIWMYG0k3z5wNU79bMjH0FVMwPr9aQWdwcBHJ75dRlU0BVwkTk7KIEfDt7/cXkxSQSNNOFaq66BEexmWmhFOxyU3VTTBZIj7tGtohAVVXjb9YQyPjdKDYSxNRRpO1d8TGRZKjURgOgXWAzXvTcT/vG6qwysvY1GSahqR2aIw5VDHcBII7DFJieYjQzCRzNwKyQBLTLSJrWRCcOZfXiSt86qDqs7tRbl2ncdRBIfgCFSAAy5BDdyABmgCAh7BM3gFb9aT9WK9Wx+z1oKVz+yDP7A+fwBC05Xy</latexit><latexit sha1_base64="J4XknDenb36+9rwby7B1axazf6g=">AAACA3icbVDLSsNAFJ3UV62vqDvdDBahopSJCLosduOygn1AE8NkOmmHziRhZiKUUHDjr7hxoYhbf8Kdf+O0zUJbD1w4nHMv994TJJwpjdC3VVhaXlldK66XNja3tnfs3b2WilNJaJPEPJadACvKWUSbmmlOO4mkWASctoNhfeK3H6hULI7u9CihnsD9iIWMYG0k3z5wNU79bMjH0FVMwPr9aQWdwcBHJ75dRlU0BVwkTk7KIEfDt7/cXkxSQSNNOFaq66BEexmWmhFOxyU3VTTBZIj7tGtohAVVXjb9YQyPjdKDYSxNRRpO1d8TGRZKjURgOgXWAzXvTcT/vG6qwysvY1GSahqR2aIw5VDHcBII7DFJieYjQzCRzNwKyQBLTLSJrWRCcOZfXiSt86qDqs7tRbl2ncdRBIfgCFSAAy5BDdyABmgCAh7BM3gFb9aT9WK9Wx+z1oKVz+yDP7A+fwBC05Xy</latexit>

wkk�,l � N (0, �2klv

2l )

<latexit sha1_base64="7PemL37DRfIpcb5U45s4wmqadx8=">AAACGHicbVDLSsNAFJ3UV62vqEs3g0WsUGpSBF0W3biSCvYBTRom00k7ZPJgZlIpIZ/hxl9x40IRt935N07aLrR6YOBwzrnMvceNGRXSML60wsrq2vpGcbO0tb2zu6fvH7RFlHBMWjhiEe+6SBBGQ9KSVDLSjTlBgctIx/Vvcr8zJlzQKHyQk5jYARqG1KMYSSU5+vmjk/r+aZVl0BI0gFaA5Agjlt5lFaMKLYkSFWBZvz52WL9+5uhlo2bMAP8Sc0HKYIGmo0+tQYSTgIQSMyREzzRiaaeIS4oZyUpWIkiMsI+GpKdoiAIi7HR2WAZPlDKAXsTVCyWcqT8nUhQIMQlclcz3FsteLv7n9RLpXdkpDeNEkhDPP/ISBmUE85bggHKCJZsogjCnaleIR4gjLFWXJVWCuXzyX9Ku10yjZt5flBvXizqK4AgcgwowwSVogFvQBC2AwRN4AW/gXXvWXrUP7XMeLWiLmUPwC9r0G0p/n0c=</latexit><latexit sha1_base64="7PemL37DRfIpcb5U45s4wmqadx8=">AAACGHicbVDLSsNAFJ3UV62vqEs3g0WsUGpSBF0W3biSCvYBTRom00k7ZPJgZlIpIZ/hxl9x40IRt935N07aLrR6YOBwzrnMvceNGRXSML60wsrq2vpGcbO0tb2zu6fvH7RFlHBMWjhiEe+6SBBGQ9KSVDLSjTlBgctIx/Vvcr8zJlzQKHyQk5jYARqG1KMYSSU5+vmjk/r+aZVl0BI0gFaA5Agjlt5lFaMKLYkSFWBZvz52WL9+5uhlo2bMAP8Sc0HKYIGmo0+tQYSTgIQSMyREzzRiaaeIS4oZyUpWIkiMsI+GpKdoiAIi7HR2WAZPlDKAXsTVCyWcqT8nUhQIMQlclcz3FsteLv7n9RLpXdkpDeNEkhDPP/ISBmUE85bggHKCJZsogjCnaleIR4gjLFWXJVWCuXzyX9Ku10yjZt5flBvXizqK4AgcgwowwSVogFvQBC2AwRN4AW/gXXvWXrUP7XMeLWiLmUPwC9r0G0p/n0c=</latexit><latexit sha1_base64="7PemL37DRfIpcb5U45s4wmqadx8=">AAACGHicbVDLSsNAFJ3UV62vqEs3g0WsUGpSBF0W3biSCvYBTRom00k7ZPJgZlIpIZ/hxl9x40IRt935N07aLrR6YOBwzrnMvceNGRXSML60wsrq2vpGcbO0tb2zu6fvH7RFlHBMWjhiEe+6SBBGQ9KSVDLSjTlBgctIx/Vvcr8zJlzQKHyQk5jYARqG1KMYSSU5+vmjk/r+aZVl0BI0gFaA5Agjlt5lFaMKLYkSFWBZvz52WL9+5uhlo2bMAP8Sc0HKYIGmo0+tQYSTgIQSMyREzzRiaaeIS4oZyUpWIkiMsI+GpKdoiAIi7HR2WAZPlDKAXsTVCyWcqT8nUhQIMQlclcz3FsteLv7n9RLpXdkpDeNEkhDPP/ISBmUE85bggHKCJZsogjCnaleIR4gjLFWXJVWCuXzyX9Ku10yjZt5flBvXizqK4AgcgwowwSVogFvQBC2AwRN4AW/gXXvWXrUP7XMeLWiLmUPwC9r0G0p/n0c=</latexit><latexit sha1_base64="7PemL37DRfIpcb5U45s4wmqadx8=">AAACGHicbVDLSsNAFJ3UV62vqEs3g0WsUGpSBF0W3biSCvYBTRom00k7ZPJgZlIpIZ/hxl9x40IRt935N07aLrR6YOBwzrnMvceNGRXSML60wsrq2vpGcbO0tb2zu6fvH7RFlHBMWjhiEe+6SBBGQ9KSVDLSjTlBgctIx/Vvcr8zJlzQKHyQk5jYARqG1KMYSSU5+vmjk/r+aZVl0BI0gFaA5Agjlt5lFaMKLYkSFWBZvz52WL9+5uhlo2bMAP8Sc0HKYIGmo0+tQYSTgIQSMyREzzRiaaeIS4oZyUpWIkiMsI+GpKdoiAIi7HR2WAZPlDKAXsTVCyWcqT8nUhQIMQlclcz3FsteLv7n9RLpXdkpDeNEkhDPP/ISBmUE85bggHKCJZsogjCnaleIR4gjLFWXJVWCuXzyX9Ku10yjZt5flBvXizqK4AgcgwowwSVogFvQBC2AwRN4AW/gXXvWXrUP7XMeLWiLmUPwC9r0G0p/n0c=</latexit>

l<latexit sha1_base64="RXFmeCFAFLWm6A2HPHJ7FvtnIgg=">AAAB6HicbVBNS8NAEJ3Ur1q/qh69LBbBU0lEqMeiF48t2A9oQ9lsJ+3azSbsboQS+gu8eFDEqz/Jm//GbZuDtj4YeLw3w8y8IBFcG9f9dgobm1vbO8Xd0t7+weFR+fikreNUMWyxWMSqG1CNgktsGW4EdhOFNAoEdoLJ3dzvPKHSPJYPZpqgH9GR5CFn1FipKQblilt1FyDrxMtJBXI0BuWv/jBmaYTSMEG17nluYvyMKsOZwFmpn2pMKJvQEfYslTRC7WeLQ2fkwipDEsbKljRkof6eyGik9TQKbGdEzVivenPxP6+XmvDGz7hMUoOSLReFqSAmJvOvyZArZEZMLaFMcXsrYWOqKDM2m5INwVt9eZ20r6qeW/Wa15X6bR5HEc7gHC7BgxrU4R4a0AIGCM/wCm/Oo/PivDsfy9aCk8+cwh84nz/UCYzw</latexit><latexit sha1_base64="RXFmeCFAFLWm6A2HPHJ7FvtnIgg=">AAAB6HicbVBNS8NAEJ3Ur1q/qh69LBbBU0lEqMeiF48t2A9oQ9lsJ+3azSbsboQS+gu8eFDEqz/Jm//GbZuDtj4YeLw3w8y8IBFcG9f9dgobm1vbO8Xd0t7+weFR+fikreNUMWyxWMSqG1CNgktsGW4EdhOFNAoEdoLJ3dzvPKHSPJYPZpqgH9GR5CFn1FipKQblilt1FyDrxMtJBXI0BuWv/jBmaYTSMEG17nluYvyMKsOZwFmpn2pMKJvQEfYslTRC7WeLQ2fkwipDEsbKljRkof6eyGik9TQKbGdEzVivenPxP6+XmvDGz7hMUoOSLReFqSAmJvOvyZArZEZMLaFMcXsrYWOqKDM2m5INwVt9eZ20r6qeW/Wa15X6bR5HEc7gHC7BgxrU4R4a0AIGCM/wCm/Oo/PivDsfy9aCk8+cwh84nz/UCYzw</latexit><latexit sha1_base64="RXFmeCFAFLWm6A2HPHJ7FvtnIgg=">AAAB6HicbVBNS8NAEJ3Ur1q/qh69LBbBU0lEqMeiF48t2A9oQ9lsJ+3azSbsboQS+gu8eFDEqz/Jm//GbZuDtj4YeLw3w8y8IBFcG9f9dgobm1vbO8Xd0t7+weFR+fikreNUMWyxWMSqG1CNgktsGW4EdhOFNAoEdoLJ3dzvPKHSPJYPZpqgH9GR5CFn1FipKQblilt1FyDrxMtJBXI0BuWv/jBmaYTSMEG17nluYvyMKsOZwFmpn2pMKJvQEfYslTRC7WeLQ2fkwipDEsbKljRkof6eyGik9TQKbGdEzVivenPxP6+XmvDGz7hMUoOSLReFqSAmJvOvyZArZEZMLaFMcXsrYWOqKDM2m5INwVt9eZ20r6qeW/Wa15X6bR5HEc7gHC7BgxrU4R4a0AIGCM/wCm/Oo/PivDsfy9aCk8+cwh84nz/UCYzw</latexit><latexit sha1_base64="RXFmeCFAFLWm6A2HPHJ7FvtnIgg=">AAAB6HicbVBNS8NAEJ3Ur1q/qh69LBbBU0lEqMeiF48t2A9oQ9lsJ+3azSbsboQS+gu8eFDEqz/Jm//GbZuDtj4YeLw3w8y8IBFcG9f9dgobm1vbO8Xd0t7+weFR+fikreNUMWyxWMSqG1CNgktsGW4EdhOFNAoEdoLJ3dzvPKHSPJYPZpqgH9GR5CFn1FipKQblilt1FyDrxMtJBXI0BuWv/jBmaYTSMEG17nluYvyMKsOZwFmpn2pMKJvQEfYslTRC7WeLQ2fkwipDEsbKljRkof6eyGik9TQKbGdEzVivenPxP6+XmvDGz7hMUoOSLReFqSAmJvOvyZArZEZMLaFMcXsrYWOqKDM2m5INwVt9eZ20r6qeW/Wa15X6bR5HEc7gHC7BgxrU4R4a0AIGCM/wCm/Oo/PivDsfy9aCk8+cwh84nz/UCYzw</latexit>

Model Selection in Bayesian Neural Networks via Horseshoe Priors; Ghosh & Doshi-Velez, 2017 Bayesian deep compression; Louizos et. al., 2017!19

• Horseshoe BNN:

• Inference:

Stochastic gradient variational Bayes / BBVI + reparameterized gradients

L(�) = Eq(W,�,v;�)[ln p(y | W , x)) + ln p(W, �, v)] + H[q(W, �, v; �)]<latexit sha1_base64="Rk6doHRQcHouf/WOt7lv4lYEwlE=">AAACiXicfVFdSxtBFJ1dW42xamoffbk0FBIaZFf8oiKIoeBDHyIYI+wuYXYySQZnZ9eZu8Gw7H/pb+pb/42TD4pR6YWBwznnfsy9cSaFQc/767hrHz6ub1Q2q1uftnd2a5/37kyaa8a7LJWpvo+p4VIo3kWBkt9nmtMklrwXP7Rnem/CtRGpusVpxqOEjpQYCkbRUv3a7zChOGZUFr/KRpiNRRMu4Ge/eGz8E3plC0KkeQsm5zC3lEGI/AkLqaDMGtMwEYMV91OzCd/hhQferdaMrOs6+G+rqF+rewfePOAt8JegTpbR6df+hIOU5QlXyCQ1JvC9DKOCahRM8rIa5oZnlD3QEQ8sVDThJirmmyzhm2UGMEy1fQphzr7MKGhizDSJrXM2s3mtzcj3tCDH4VlUCJXlyBVbNBrmEjCF2VlgIDRnKKcWUKaFnRXYmGrK0B6vapfgv/7yW3B3eOBbfHNUv7xarqNC9slX0iA+OSWX5Jp0SJcwZ91pOcfOibvl+u6Z+2NhdZ1lzheyEm77GWzywWs=</latexit><latexit sha1_base64="Rk6doHRQcHouf/WOt7lv4lYEwlE=">AAACiXicfVFdSxtBFJ1dW42xamoffbk0FBIaZFf8oiKIoeBDHyIYI+wuYXYySQZnZ9eZu8Gw7H/pb+pb/42TD4pR6YWBwznnfsy9cSaFQc/767hrHz6ub1Q2q1uftnd2a5/37kyaa8a7LJWpvo+p4VIo3kWBkt9nmtMklrwXP7Rnem/CtRGpusVpxqOEjpQYCkbRUv3a7zChOGZUFr/KRpiNRRMu4Ge/eGz8E3plC0KkeQsm5zC3lEGI/AkLqaDMGtMwEYMV91OzCd/hhQferdaMrOs6+G+rqF+rewfePOAt8JegTpbR6df+hIOU5QlXyCQ1JvC9DKOCahRM8rIa5oZnlD3QEQ8sVDThJirmmyzhm2UGMEy1fQphzr7MKGhizDSJrXM2s3mtzcj3tCDH4VlUCJXlyBVbNBrmEjCF2VlgIDRnKKcWUKaFnRXYmGrK0B6vapfgv/7yW3B3eOBbfHNUv7xarqNC9slX0iA+OSWX5Jp0SJcwZ91pOcfOibvl+u6Z+2NhdZ1lzheyEm77GWzywWs=</latexit><latexit sha1_base64="Rk6doHRQcHouf/WOt7lv4lYEwlE=">AAACiXicfVFdSxtBFJ1dW42xamoffbk0FBIaZFf8oiKIoeBDHyIYI+wuYXYySQZnZ9eZu8Gw7H/pb+pb/42TD4pR6YWBwznnfsy9cSaFQc/767hrHz6ub1Q2q1uftnd2a5/37kyaa8a7LJWpvo+p4VIo3kWBkt9nmtMklrwXP7Rnem/CtRGpusVpxqOEjpQYCkbRUv3a7zChOGZUFr/KRpiNRRMu4Ge/eGz8E3plC0KkeQsm5zC3lEGI/AkLqaDMGtMwEYMV91OzCd/hhQferdaMrOs6+G+rqF+rewfePOAt8JegTpbR6df+hIOU5QlXyCQ1JvC9DKOCahRM8rIa5oZnlD3QEQ8sVDThJirmmyzhm2UGMEy1fQphzr7MKGhizDSJrXM2s3mtzcj3tCDH4VlUCJXlyBVbNBrmEjCF2VlgIDRnKKcWUKaFnRXYmGrK0B6vapfgv/7yW3B3eOBbfHNUv7xarqNC9slX0iA+OSWX5Jp0SJcwZ91pOcfOibvl+u6Z+2NhdZ1lzheyEm77GWzywWs=</latexit><latexit sha1_base64="Rk6doHRQcHouf/WOt7lv4lYEwlE=">AAACiXicfVFdSxtBFJ1dW42xamoffbk0FBIaZFf8oiKIoeBDHyIYI+wuYXYySQZnZ9eZu8Gw7H/pb+pb/42TD4pR6YWBwznnfsy9cSaFQc/767hrHz6ub1Q2q1uftnd2a5/37kyaa8a7LJWpvo+p4VIo3kWBkt9nmtMklrwXP7Rnem/CtRGpusVpxqOEjpQYCkbRUv3a7zChOGZUFr/KRpiNRRMu4Ge/eGz8E3plC0KkeQsm5zC3lEGI/AkLqaDMGtMwEYMV91OzCd/hhQferdaMrOs6+G+rqF+rewfePOAt8JegTpbR6df+hIOU5QlXyCQ1JvC9DKOCahRM8rIa5oZnlD3QEQ8sVDThJirmmyzhm2UGMEy1fQphzr7MKGhizDSJrXM2s3mtzcj3tCDH4VlUCJXlyBVbNBrmEjCF2VlgIDRnKKcWUKaFnRXYmGrK0B6vapfgv/7yW3B3eOBbfHNUv7xarqNC9slX0iA+OSWX5Jp0SJcwZ91pOcfOibvl+u6Z+2NhdZ1lzheyEm77GWzywWs=</latexit>

��<latexit sha1_base64="Juzcs/zLpqFIAao5oRVScIfHEB8=">AAAB7XicbVDLSgNBEOyNrxhfUY9eBoPgKeyKoMegF48RzAOSJcxOepMxszPLzKwQQv7BiwdFvPo/3vwbJ8keNLGgoajqprsrSgU31ve/vcLa+sbmVnG7tLO7t39QPjxqGpVphg2mhNLtiBoUXGLDciuwnWqkSSSwFY1uZ37rCbXhSj7YcYphQgeSx5xR66RmN9KUYa9c8av+HGSVBDmpQI56r/zV7SuWJSgtE9SYTuCnNpxQbTkTOC11M4MpZSM6wI6jkiZowsn82ik5c0qfxEq7kpbM1d8TE5oYM04i15lQOzTL3kz8z+tkNr4OJ1ymmUXJFoviTBCryOx10ucamRVjRyjT3N1K2JC6AKwLqORCCJZfXiXNi2rgV4P7y0rtJo+jCCdwCucQwBXU4A7q0AAGj/AMr/DmKe/Fe/c+Fq0FL585hj/wPn8AfdCPDw==</latexit><latexit sha1_base64="Juzcs/zLpqFIAao5oRVScIfHEB8=">AAAB7XicbVDLSgNBEOyNrxhfUY9eBoPgKeyKoMegF48RzAOSJcxOepMxszPLzKwQQv7BiwdFvPo/3vwbJ8keNLGgoajqprsrSgU31ve/vcLa+sbmVnG7tLO7t39QPjxqGpVphg2mhNLtiBoUXGLDciuwnWqkSSSwFY1uZ37rCbXhSj7YcYphQgeSx5xR66RmN9KUYa9c8av+HGSVBDmpQI56r/zV7SuWJSgtE9SYTuCnNpxQbTkTOC11M4MpZSM6wI6jkiZowsn82ik5c0qfxEq7kpbM1d8TE5oYM04i15lQOzTL3kz8z+tkNr4OJ1ymmUXJFoviTBCryOx10ucamRVjRyjT3N1K2JC6AKwLqORCCJZfXiXNi2rgV4P7y0rtJo+jCCdwCucQwBXU4A7q0AAGj/AMr/DmKe/Fe/c+Fq0FL585hj/wPn8AfdCPDw==</latexit><latexit sha1_base64="Juzcs/zLpqFIAao5oRVScIfHEB8=">AAAB7XicbVDLSgNBEOyNrxhfUY9eBoPgKeyKoMegF48RzAOSJcxOepMxszPLzKwQQv7BiwdFvPo/3vwbJ8keNLGgoajqprsrSgU31ve/vcLa+sbmVnG7tLO7t39QPjxqGpVphg2mhNLtiBoUXGLDciuwnWqkSSSwFY1uZ37rCbXhSj7YcYphQgeSx5xR66RmN9KUYa9c8av+HGSVBDmpQI56r/zV7SuWJSgtE9SYTuCnNpxQbTkTOC11M4MpZSM6wI6jkiZowsn82ik5c0qfxEq7kpbM1d8TE5oYM04i15lQOzTL3kz8z+tkNr4OJ1ymmUXJFoviTBCryOx10ucamRVjRyjT3N1K2JC6AKwLqORCCJZfXiXNi2rgV4P7y0rtJo+jCCdwCucQwBXU4A7q0AAGj/AMr/DmKe/Fe/c+Fq0FL585hj/wPn8AfdCPDw==</latexit><latexit sha1_base64="Juzcs/zLpqFIAao5oRVScIfHEB8=">AAAB7XicbVDLSgNBEOyNrxhfUY9eBoPgKeyKoMegF48RzAOSJcxOepMxszPLzKwQQv7BiwdFvPo/3vwbJ8keNLGgoajqprsrSgU31ve/vcLa+sbmVnG7tLO7t39QPjxqGpVphg2mhNLtiBoUXGLDciuwnWqkSSSwFY1uZ37rCbXhSj7YcYphQgeSx5xR66RmN9KUYa9c8av+HGSVBDmpQI56r/zV7SuWJSgtE9SYTuCnNpxQbTkTOC11M4MpZSM6wI6jkiZowsn82ik5c0qfxEq7kpbM1d8TE5oYM04i15lQOzTL3kz8z+tkNr4OJ1ymmUXJFoviTBCryOx10ucamRVjRyjT3N1K2JC6AKwLqORCCJZfXiXNi2rgV4P7y0rtJo+jCCdwCucQwBXU4A7q0AAGj/AMr/DmKe/Fe/c+Fq0FL585hj/wPn8AfdCPDw==</latexit>

NN; intractable expectation

Page 20: Horseshoe Priors for Bayesian Neural Networks...Horseshoe Priors for Bayesian Neural Networks Soumya Ghosh Jiayu Yao Finale Doshi-Velez IBM Research MIT-IBM Watson AI lab Harvard Harvard

Local Reparameterization

!20

• Continuous weights and variances —> reparameterization trick

��

<latexit sha1_base64="Juzcs/zLpqFIAao5oRVScIfHEB8=">AAAB7XicbVDLSgNBEOyNrxhfUY9eBoPgKeyKoMegF48RzAOSJcxOepMxszPLzKwQQv7BiwdFvPo/3vwbJ8keNLGgoajqprsrSgU31ve/vcLa+sbmVnG7tLO7t39QPjxqGpVphg2mhNLtiBoUXGLDciuwnWqkSSSwFY1uZ37rCbXhSj7YcYphQgeSx5xR66RmN9KUYa9c8av+HGSVBDmpQI56r/zV7SuWJSgtE9SYTuCnNpxQbTkTOC11M4MpZSM6wI6jkiZowsn82ik5c0qfxEq7kpbM1d8TE5oYM04i15lQOzTL3kz8z+tkNr4OJ1ymmUXJFoviTBCryOx10ucamRVjRyjT3N1K2JC6AKwLqORCCJZfXiXNi2rgV4P7y0rtJo+jCCdwCucQwBXU4A7q0AAGj/AMr/DmKe/Fe/c+Fq0FL585hj/wPn8AfdCPDw==</latexit><latexit sha1_base64="Juzcs/zLpqFIAao5oRVScIfHEB8=">AAAB7XicbVDLSgNBEOyNrxhfUY9eBoPgKeyKoMegF48RzAOSJcxOepMxszPLzKwQQv7BiwdFvPo/3vwbJ8keNLGgoajqprsrSgU31ve/vcLa+sbmVnG7tLO7t39QPjxqGpVphg2mhNLtiBoUXGLDciuwnWqkSSSwFY1uZ37rCbXhSj7YcYphQgeSx5xR66RmN9KUYa9c8av+HGSVBDmpQI56r/zV7SuWJSgtE9SYTuCnNpxQbTkTOC11M4MpZSM6wI6jkiZowsn82ik5c0qfxEq7kpbM1d8TE5oYM04i15lQOzTL3kz8z+tkNr4OJ1ymmUXJFoviTBCryOx10ucamRVjRyjT3N1K2JC6AKwLqORCCJZfXiXNi2rgV4P7y0rtJo+jCCdwCucQwBXU4A7q0AAGj/AMr/DmKe/Fe/c+Fq0FL585hj/wPn8AfdCPDw==</latexit><latexit sha1_base64="Juzcs/zLpqFIAao5oRVScIfHEB8=">AAAB7XicbVDLSgNBEOyNrxhfUY9eBoPgKeyKoMegF48RzAOSJcxOepMxszPLzKwQQv7BiwdFvPo/3vwbJ8keNLGgoajqprsrSgU31ve/vcLa+sbmVnG7tLO7t39QPjxqGpVphg2mhNLtiBoUXGLDciuwnWqkSSSwFY1uZ37rCbXhSj7YcYphQgeSx5xR66RmN9KUYa9c8av+HGSVBDmpQI56r/zV7SuWJSgtE9SYTuCnNpxQbTkTOC11M4MpZSM6wI6jkiZowsn82ik5c0qfxEq7kpbM1d8TE5oYM04i15lQOzTL3kz8z+tkNr4OJ1ymmUXJFoviTBCryOx10ucamRVjRyjT3N1K2JC6AKwLqORCCJZfXiXNi2rgV4P7y0rtJo+jCCdwCucQwBXU4A7q0AAGj/AMr/DmKe/Fe/c+Fq0FL585hj/wPn8AfdCPDw==</latexit><latexit sha1_base64="Juzcs/zLpqFIAao5oRVScIfHEB8=">AAAB7XicbVDLSgNBEOyNrxhfUY9eBoPgKeyKoMegF48RzAOSJcxOepMxszPLzKwQQv7BiwdFvPo/3vwbJ8keNLGgoajqprsrSgU31ve/vcLa+sbmVnG7tLO7t39QPjxqGpVphg2mhNLtiBoUXGLDciuwnWqkSSSwFY1uZ37rCbXhSj7YcYphQgeSx5xR66RmN9KUYa9c8av+HGSVBDmpQI56r/zV7SuWJSgtE9SYTuCnNpxQbTkTOC11M4MpZSM6wI6jkiZowsn82ik5c0qfxEq7kpbM1d8TE5oYM04i15lQOzTL3kz8z+tkNr4OJ1ymmUXJFoviTBCryOx10ucamRVjRyjT3N1K2JC6AKwLqORCCJZfXiXNi2rgV4P7y0rtJo+jCCdwCucQwBXU4A7q0AAGj/AMr/DmKe/Fe/c+Fq0FL585hj/wPn8AfdCPDw==</latexit>

#in

puts

<latexit sha1_base64="T3q/yKw3n9RAynz9XXt+mEAEae8=">AAAB8nicbVDLSgMxFL1TX7W+qi7dBAfBVZkRQcFN0Y3LCvYBM0PJpJk2NJMMSUYopZ/hxoUibv0ad/6N6XQW2nogcDjnXnLPiTPOtPG8b6eytr6xuVXdru3s7u0f1A+POlrmitA2kVyqXow15UzQtmGG016mKE5jTrvx+G7ud5+o0kyKRzPJaJTioWAJI9hYKQjd8AYxkeVG9+uu1/AKoFXil8SFEq1+/SscSJKnVBjCsdaB72UmmmJlGOF0VgtzTTNMxnhIA0sFTqmOpsXJM3RmlQFKpLJPGFSovzemONV6ksZ2MsVmpJe9ufifF+QmuY6mRSQqyOKjJOfISDTPjwZMUWL4xBJMFLO3IjLCChNjW6rZEvzlyKukc9HwvYb/cOk2b8s6qnACp3AOPlxBE+6hBW0gIOEZXuHNMc6L8+58LEYrTrlzDH/gfP4Ah2uQwQ==</latexit><latexit sha1_base64="T3q/yKw3n9RAynz9XXt+mEAEae8=">AAAB8nicbVDLSgMxFL1TX7W+qi7dBAfBVZkRQcFN0Y3LCvYBM0PJpJk2NJMMSUYopZ/hxoUibv0ad/6N6XQW2nogcDjnXnLPiTPOtPG8b6eytr6xuVXdru3s7u0f1A+POlrmitA2kVyqXow15UzQtmGG016mKE5jTrvx+G7ud5+o0kyKRzPJaJTioWAJI9hYKQjd8AYxkeVG9+uu1/AKoFXil8SFEq1+/SscSJKnVBjCsdaB72UmmmJlGOF0VgtzTTNMxnhIA0sFTqmOpsXJM3RmlQFKpLJPGFSovzemONV6ksZ2MsVmpJe9ufifF+QmuY6mRSQqyOKjJOfISDTPjwZMUWL4xBJMFLO3IjLCChNjW6rZEvzlyKukc9HwvYb/cOk2b8s6qnACp3AOPlxBE+6hBW0gIOEZXuHNMc6L8+58LEYrTrlzDH/gfP4Ah2uQwQ==</latexit><latexit sha1_base64="T3q/yKw3n9RAynz9XXt+mEAEae8=">AAAB8nicbVDLSgMxFL1TX7W+qi7dBAfBVZkRQcFN0Y3LCvYBM0PJpJk2NJMMSUYopZ/hxoUibv0ad/6N6XQW2nogcDjnXnLPiTPOtPG8b6eytr6xuVXdru3s7u0f1A+POlrmitA2kVyqXow15UzQtmGG016mKE5jTrvx+G7ud5+o0kyKRzPJaJTioWAJI9hYKQjd8AYxkeVG9+uu1/AKoFXil8SFEq1+/SscSJKnVBjCsdaB72UmmmJlGOF0VgtzTTNMxnhIA0sFTqmOpsXJM3RmlQFKpLJPGFSovzemONV6ksZ2MsVmpJe9ufifF+QmuY6mRSQqyOKjJOfISDTPjwZMUWL4xBJMFLO3IjLCChNjW6rZEvzlyKukc9HwvYb/cOk2b8s6qnACp3AOPlxBE+6hBW0gIOEZXuHNMc6L8+58LEYrTrlzDH/gfP4Ah2uQwQ==</latexit><latexit sha1_base64="T3q/yKw3n9RAynz9XXt+mEAEae8=">AAAB8nicbVDLSgMxFL1TX7W+qi7dBAfBVZkRQcFN0Y3LCvYBM0PJpJk2NJMMSUYopZ/hxoUibv0ad/6N6XQW2nogcDjnXnLPiTPOtPG8b6eytr6xuVXdru3s7u0f1A+POlrmitA2kVyqXow15UzQtmGG016mKE5jTrvx+G7ud5+o0kyKRzPJaJTioWAJI9hYKQjd8AYxkeVG9+uu1/AKoFXil8SFEq1+/SscSJKnVBjCsdaB72UmmmJlGOF0VgtzTTNMxnhIA0sFTqmOpsXJM3RmlQFKpLJPGFSovzemONV6ksZ2MsVmpJe9ufifF+QmuY6mRSQqyOKjJOfISDTPjwZMUWL4xBJMFLO3IjLCChNjW6rZEvzlyKukc9HwvYb/cOk2b8s6qnACp3AOPlxBE+6hBW0gIOEZXuHNMc6L8+58LEYrTrlzDH/gfP4Ah2uQwQ==</latexit>

# outputs<latexit sha1_base64="ICBz4Nuoqv8BUptsqPXTA09K6E0=">AAAB83icbVBNSwMxEM3Wr1q/qh69BBfBU9kVQcFL0YvHCvYDukvJptk2NJssyUQopX/DiwdFvPpnvPlvTNs9aOuDgcd7M8zMS3LBDQTBt1daW9/Y3CpvV3Z29/YPqodHLaOspqxJlVC6kxDDBJesCRwE6+SakSwRrJ2M7mZ++4lpw5V8hHHO4owMJE85JeCkKPKjG6ws5BZMr+oHtWAOvErCgvioQKNX/Yr6itqMSaCCGNMNgxziCdHAqWDTSmQNywkdkQHrOipJxkw8md88xWdO6eNUaVcS8Fz9PTEhmTHjLHGdGYGhWfZm4n9e10J6HU+4dC8xSReLUiswKDwLAPe5ZhTE2BFCNXe3YjokmlBwMVVcCOHyy6ukdVELg1r4cOnXb4s4yugEnaJzFKIrVEf3qIGaiKIcPaNX9OZZ78V79z4WrSWvmDlGf+B9/gB0GpFM</latexit><latexit sha1_base64="ICBz4Nuoqv8BUptsqPXTA09K6E0=">AAAB83icbVBNSwMxEM3Wr1q/qh69BBfBU9kVQcFL0YvHCvYDukvJptk2NJssyUQopX/DiwdFvPpnvPlvTNs9aOuDgcd7M8zMS3LBDQTBt1daW9/Y3CpvV3Z29/YPqodHLaOspqxJlVC6kxDDBJesCRwE6+SakSwRrJ2M7mZ++4lpw5V8hHHO4owMJE85JeCkKPKjG6ws5BZMr+oHtWAOvErCgvioQKNX/Yr6itqMSaCCGNMNgxziCdHAqWDTSmQNywkdkQHrOipJxkw8md88xWdO6eNUaVcS8Fz9PTEhmTHjLHGdGYGhWfZm4n9e10J6HU+4dC8xSReLUiswKDwLAPe5ZhTE2BFCNXe3YjokmlBwMVVcCOHyy6ukdVELg1r4cOnXb4s4yugEnaJzFKIrVEf3qIGaiKIcPaNX9OZZ78V79z4WrSWvmDlGf+B9/gB0GpFM</latexit><latexit sha1_base64="ICBz4Nuoqv8BUptsqPXTA09K6E0=">AAAB83icbVBNSwMxEM3Wr1q/qh69BBfBU9kVQcFL0YvHCvYDukvJptk2NJssyUQopX/DiwdFvPpnvPlvTNs9aOuDgcd7M8zMS3LBDQTBt1daW9/Y3CpvV3Z29/YPqodHLaOspqxJlVC6kxDDBJesCRwE6+SakSwRrJ2M7mZ++4lpw5V8hHHO4owMJE85JeCkKPKjG6ws5BZMr+oHtWAOvErCgvioQKNX/Yr6itqMSaCCGNMNgxziCdHAqWDTSmQNywkdkQHrOipJxkw8md88xWdO6eNUaVcS8Fz9PTEhmTHjLHGdGYGhWfZm4n9e10J6HU+4dC8xSReLUiswKDwLAPe5ZhTE2BFCNXe3YjokmlBwMVVcCOHyy6ukdVELg1r4cOnXb4s4yugEnaJzFKIrVEf3qIGaiKIcPaNX9OZZ78V79z4WrSWvmDlGf+B9/gB0GpFM</latexit><latexit sha1_base64="ICBz4Nuoqv8BUptsqPXTA09K6E0=">AAAB83icbVBNSwMxEM3Wr1q/qh69BBfBU9kVQcFL0YvHCvYDukvJptk2NJssyUQopX/DiwdFvPpnvPlvTNs9aOuDgcd7M8zMS3LBDQTBt1daW9/Y3CpvV3Z29/YPqodHLaOspqxJlVC6kxDDBJesCRwE6+SakSwRrJ2M7mZ++4lpw5V8hHHO4owMJE85JeCkKPKjG6ws5BZMr+oHtWAOvErCgvioQKNX/Yr6itqMSaCCGNMNgxziCdHAqWDTSmQNywkdkQHrOipJxkw8md88xWdO6eNUaVcS8Fz9PTEhmTHjLHGdGYGhWfZm4n9e10J6HU+4dC8xSReLUiswKDwLAPe5ZhTE2BFCNXe3YjokmlBwMVVcCOHyy6ukdVELg1r4cOnXb4s4yugEnaJzFKIrVEf3qIGaiKIcPaNX9OZZ78V79z4WrSWvmDlGf+B9/gB0GpFM</latexit>

weightsWl<latexit sha1_base64="PvPkXizAzMF6sNgh5sOcM2KubeY=">AAAB7HicbZBNS8NAEIYn9avWr6pHL4tF8FQSEeqx6MVjBdMW2lA22027dLMJuxOhhP4GLx4U8eoP8ua/cdvmoK0vLDy8M8POvGEqhUHX/XZKG5tb2zvl3cre/sHhUfX4pG2STDPus0QmuhtSw6VQ3EeBkndTzWkcSt4JJ3fzeueJayMS9YjTlAcxHSkRCUbRWn5nkMvZoFpz6+5CZB28AmpQqDWofvWHCctirpBJakzPc1MMcqpRMMlnlX5meErZhI54z6KiMTdBvlh2Ri6sMyRRou1TSBbu74mcxsZM49B2xhTHZrU2N/+r9TKMboJcqDRDrtjyoyiTBBMyv5wMheYM5dQCZVrYXQkbU00Z2nwqNgRv9eR1aF/VPcsP17XmbRFHGc7gHC7BgwY04R5a4AMDAc/wCm+Ocl6cd+dj2VpyiplT+CPn8wf3Z47G</latexit><latexit sha1_base64="PvPkXizAzMF6sNgh5sOcM2KubeY=">AAAB7HicbZBNS8NAEIYn9avWr6pHL4tF8FQSEeqx6MVjBdMW2lA22027dLMJuxOhhP4GLx4U8eoP8ua/cdvmoK0vLDy8M8POvGEqhUHX/XZKG5tb2zvl3cre/sHhUfX4pG2STDPus0QmuhtSw6VQ3EeBkndTzWkcSt4JJ3fzeueJayMS9YjTlAcxHSkRCUbRWn5nkMvZoFpz6+5CZB28AmpQqDWofvWHCctirpBJakzPc1MMcqpRMMlnlX5meErZhI54z6KiMTdBvlh2Ri6sMyRRou1TSBbu74mcxsZM49B2xhTHZrU2N/+r9TKMboJcqDRDrtjyoyiTBBMyv5wMheYM5dQCZVrYXQkbU00Z2nwqNgRv9eR1aF/VPcsP17XmbRFHGc7gHC7BgwY04R5a4AMDAc/wCm+Ocl6cd+dj2VpyiplT+CPn8wf3Z47G</latexit><latexit sha1_base64="PvPkXizAzMF6sNgh5sOcM2KubeY=">AAAB7HicbZBNS8NAEIYn9avWr6pHL4tF8FQSEeqx6MVjBdMW2lA22027dLMJuxOhhP4GLx4U8eoP8ua/cdvmoK0vLDy8M8POvGEqhUHX/XZKG5tb2zvl3cre/sHhUfX4pG2STDPus0QmuhtSw6VQ3EeBkndTzWkcSt4JJ3fzeueJayMS9YjTlAcxHSkRCUbRWn5nkMvZoFpz6+5CZB28AmpQqDWofvWHCctirpBJakzPc1MMcqpRMMlnlX5meErZhI54z6KiMTdBvlh2Ri6sMyRRou1TSBbu74mcxsZM49B2xhTHZrU2N/+r9TKMboJcqDRDrtjyoyiTBBMyv5wMheYM5dQCZVrYXQkbU00Z2nwqNgRv9eR1aF/VPcsP17XmbRFHGc7gHC7BgwY04R5a4AMDAc/wCm+Ocl6cd+dj2VpyiplT+CPn8wf3Z47G</latexit><latexit sha1_base64="PvPkXizAzMF6sNgh5sOcM2KubeY=">AAAB7HicbZBNS8NAEIYn9avWr6pHL4tF8FQSEeqx6MVjBdMW2lA22027dLMJuxOhhP4GLx4U8eoP8ua/cdvmoK0vLDy8M8POvGEqhUHX/XZKG5tb2zvl3cre/sHhUfX4pG2STDPus0QmuhtSw6VQ3EeBkndTzWkcSt4JJ3fzeueJayMS9YjTlAcxHSkRCUbRWn5nkMvZoFpz6+5CZB28AmpQqDWofvWHCctirpBJakzPc1MMcqpRMMlnlX5meErZhI54z6KiMTdBvlh2Ri6sMyRRou1TSBbu74mcxsZM49B2xhTHZrU2N/+r9TKMboJcqDRDrtjyoyiTBBMyv5wMheYM5dQCZVrYXQkbU00Z2nwqNgRv9eR1aF/VPcsP17XmbRFHGc7gHC7BgwY04R5a4AMDAc/wCm+Ocl6cd+dj2VpyiplT+CPn8wf3Z47G</latexit>

W (s)l � q(W )

<latexit sha1_base64="OhdjD2Q9S0PSvgYcn7cEJqrt1Rw=">AAAB/nicbZBPS8MwGMbT+W/Of1Xx5CU4hO0yWhH0OPTicYJbB1staZZuYUlak1QYpeBX8eJBEa9+Dm9+G7OtB918IPDjed+X980TJowq7TjfVmlldW19o7xZ2dre2d2z9w86Kk4lJm0cs1h2Q6QIo4K0NdWMdBNJEA8Z8cLx9bTuPRKpaCzu9CQhPkdDQSOKkTZWYB95Qcby+6ym6jnsK8rhQ82rB3bVaTgzwWVwC6iCQq3A/uoPYpxyIjRmSKme6yTaz5DUFDOSV/qpIgnCYzQkPYMCcaL8bHZ+Dk+NM4BRLM0TGs7c3xMZ4kpNeGg6OdIjtVibmv/VeqmOLv2MiiTVROD5oihlUMdwmgUcUEmwZhMDCEtqboV4hCTC2iRWMSG4i19ehs5ZwzV8e15tXhVxlMExOAE14IIL0AQ3oAXaAIMMPINX8GY9WS/Wu/Uxby1Zxcwh+CPr8wfCrpSv</latexit><latexit sha1_base64="OhdjD2Q9S0PSvgYcn7cEJqrt1Rw=">AAAB/nicbZBPS8MwGMbT+W/Of1Xx5CU4hO0yWhH0OPTicYJbB1staZZuYUlak1QYpeBX8eJBEa9+Dm9+G7OtB918IPDjed+X980TJowq7TjfVmlldW19o7xZ2dre2d2z9w86Kk4lJm0cs1h2Q6QIo4K0NdWMdBNJEA8Z8cLx9bTuPRKpaCzu9CQhPkdDQSOKkTZWYB95Qcby+6ym6jnsK8rhQ82rB3bVaTgzwWVwC6iCQq3A/uoPYpxyIjRmSKme6yTaz5DUFDOSV/qpIgnCYzQkPYMCcaL8bHZ+Dk+NM4BRLM0TGs7c3xMZ4kpNeGg6OdIjtVibmv/VeqmOLv2MiiTVROD5oihlUMdwmgUcUEmwZhMDCEtqboV4hCTC2iRWMSG4i19ehs5ZwzV8e15tXhVxlMExOAE14IIL0AQ3oAXaAIMMPINX8GY9WS/Wu/Uxby1Zxcwh+CPr8wfCrpSv</latexit><latexit sha1_base64="OhdjD2Q9S0PSvgYcn7cEJqrt1Rw=">AAAB/nicbZBPS8MwGMbT+W/Of1Xx5CU4hO0yWhH0OPTicYJbB1staZZuYUlak1QYpeBX8eJBEa9+Dm9+G7OtB918IPDjed+X980TJowq7TjfVmlldW19o7xZ2dre2d2z9w86Kk4lJm0cs1h2Q6QIo4K0NdWMdBNJEA8Z8cLx9bTuPRKpaCzu9CQhPkdDQSOKkTZWYB95Qcby+6ym6jnsK8rhQ82rB3bVaTgzwWVwC6iCQq3A/uoPYpxyIjRmSKme6yTaz5DUFDOSV/qpIgnCYzQkPYMCcaL8bHZ+Dk+NM4BRLM0TGs7c3xMZ4kpNeGg6OdIjtVibmv/VeqmOLv2MiiTVROD5oihlUMdwmgUcUEmwZhMDCEtqboV4hCTC2iRWMSG4i19ehs5ZwzV8e15tXhVxlMExOAE14IIL0AQ3oAXaAIMMPINX8GY9WS/Wu/Uxby1Zxcwh+CPr8wfCrpSv</latexit><latexit sha1_base64="OhdjD2Q9S0PSvgYcn7cEJqrt1Rw=">AAAB/nicbZBPS8MwGMbT+W/Of1Xx5CU4hO0yWhH0OPTicYJbB1staZZuYUlak1QYpeBX8eJBEa9+Dm9+G7OtB918IPDjed+X980TJowq7TjfVmlldW19o7xZ2dre2d2z9w86Kk4lJm0cs1h2Q6QIo4K0NdWMdBNJEA8Z8cLx9bTuPRKpaCzu9CQhPkdDQSOKkTZWYB95Qcby+6ym6jnsK8rhQ82rB3bVaTgzwWVwC6iCQq3A/uoPYpxyIjRmSKme6yTaz5DUFDOSV/qpIgnCYzQkPYMCcaL8bHZ+Dk+NM4BRLM0TGs7c3xMZ4kpNeGg6OdIjtVibmv/VeqmOLv2MiiTVROD5oihlUMdwmgUcUEmwZhMDCEtqboV4hCTC2iRWMSG4i19ehs5ZwzV8e15tXhVxlMExOAE14IIL0AQ3oAXaAIMMPINX8GY9WS/Wu/Uxby1Zxcwh+CPr8wfCrpSv</latexit>

naive application

• Local reparameterization — provably lower variance

#in

puts

<latexit sha1_base64="T3q/yKw3n9RAynz9XXt+mEAEae8=">AAAB8nicbVDLSgMxFL1TX7W+qi7dBAfBVZkRQcFN0Y3LCvYBM0PJpJk2NJMMSUYopZ/hxoUibv0ad/6N6XQW2nogcDjnXnLPiTPOtPG8b6eytr6xuVXdru3s7u0f1A+POlrmitA2kVyqXow15UzQtmGG016mKE5jTrvx+G7ud5+o0kyKRzPJaJTioWAJI9hYKQjd8AYxkeVG9+uu1/AKoFXil8SFEq1+/SscSJKnVBjCsdaB72UmmmJlGOF0VgtzTTNMxnhIA0sFTqmOpsXJM3RmlQFKpLJPGFSovzemONV6ksZ2MsVmpJe9ufifF+QmuY6mRSQqyOKjJOfISDTPjwZMUWL4xBJMFLO3IjLCChNjW6rZEvzlyKukc9HwvYb/cOk2b8s6qnACp3AOPlxBE+6hBW0gIOEZXuHNMc6L8+58LEYrTrlzDH/gfP4Ah2uQwQ==</latexit><latexit sha1_base64="T3q/yKw3n9RAynz9XXt+mEAEae8=">AAAB8nicbVDLSgMxFL1TX7W+qi7dBAfBVZkRQcFN0Y3LCvYBM0PJpJk2NJMMSUYopZ/hxoUibv0ad/6N6XQW2nogcDjnXnLPiTPOtPG8b6eytr6xuVXdru3s7u0f1A+POlrmitA2kVyqXow15UzQtmGG016mKE5jTrvx+G7ud5+o0kyKRzPJaJTioWAJI9hYKQjd8AYxkeVG9+uu1/AKoFXil8SFEq1+/SscSJKnVBjCsdaB72UmmmJlGOF0VgtzTTNMxnhIA0sFTqmOpsXJM3RmlQFKpLJPGFSovzemONV6ksZ2MsVmpJe9ufifF+QmuY6mRSQqyOKjJOfISDTPjwZMUWL4xBJMFLO3IjLCChNjW6rZEvzlyKukc9HwvYb/cOk2b8s6qnACp3AOPlxBE+6hBW0gIOEZXuHNMc6L8+58LEYrTrlzDH/gfP4Ah2uQwQ==</latexit><latexit sha1_base64="T3q/yKw3n9RAynz9XXt+mEAEae8=">AAAB8nicbVDLSgMxFL1TX7W+qi7dBAfBVZkRQcFN0Y3LCvYBM0PJpJk2NJMMSUYopZ/hxoUibv0ad/6N6XQW2nogcDjnXnLPiTPOtPG8b6eytr6xuVXdru3s7u0f1A+POlrmitA2kVyqXow15UzQtmGG016mKE5jTrvx+G7ud5+o0kyKRzPJaJTioWAJI9hYKQjd8AYxkeVG9+uu1/AKoFXil8SFEq1+/SscSJKnVBjCsdaB72UmmmJlGOF0VgtzTTNMxnhIA0sFTqmOpsXJM3RmlQFKpLJPGFSovzemONV6ksZ2MsVmpJe9ufifF+QmuY6mRSQqyOKjJOfISDTPjwZMUWL4xBJMFLO3IjLCChNjW6rZEvzlyKukc9HwvYb/cOk2b8s6qnACp3AOPlxBE+6hBW0gIOEZXuHNMc6L8+58LEYrTrlzDH/gfP4Ah2uQwQ==</latexit><latexit sha1_base64="T3q/yKw3n9RAynz9XXt+mEAEae8=">AAAB8nicbVDLSgMxFL1TX7W+qi7dBAfBVZkRQcFN0Y3LCvYBM0PJpJk2NJMMSUYopZ/hxoUibv0ad/6N6XQW2nogcDjnXnLPiTPOtPG8b6eytr6xuVXdru3s7u0f1A+POlrmitA2kVyqXow15UzQtmGG016mKE5jTrvx+G7ud5+o0kyKRzPJaJTioWAJI9hYKQjd8AYxkeVG9+uu1/AKoFXil8SFEq1+/SscSJKnVBjCsdaB72UmmmJlGOF0VgtzTTNMxnhIA0sFTqmOpsXJM3RmlQFKpLJPGFSovzemONV6ksZ2MsVmpJe9ufifF+QmuY6mRSQqyOKjJOfISDTPjwZMUWL4xBJMFLO3IjLCChNjW6rZEvzlyKukc9HwvYb/cOk2b8s6qnACp3AOPlxBE+6hBW0gIOEZXuHNMc6L8+58LEYrTrlzDH/gfP4Ah2uQwQ==</latexit>

# outputs<latexit sha1_base64="ICBz4Nuoqv8BUptsqPXTA09K6E0=">AAAB83icbVBNSwMxEM3Wr1q/qh69BBfBU9kVQcFL0YvHCvYDukvJptk2NJssyUQopX/DiwdFvPpnvPlvTNs9aOuDgcd7M8zMS3LBDQTBt1daW9/Y3CpvV3Z29/YPqodHLaOspqxJlVC6kxDDBJesCRwE6+SakSwRrJ2M7mZ++4lpw5V8hHHO4owMJE85JeCkKPKjG6ws5BZMr+oHtWAOvErCgvioQKNX/Yr6itqMSaCCGNMNgxziCdHAqWDTSmQNywkdkQHrOipJxkw8md88xWdO6eNUaVcS8Fz9PTEhmTHjLHGdGYGhWfZm4n9e10J6HU+4dC8xSReLUiswKDwLAPe5ZhTE2BFCNXe3YjokmlBwMVVcCOHyy6ukdVELg1r4cOnXb4s4yugEnaJzFKIrVEf3qIGaiKIcPaNX9OZZ78V79z4WrSWvmDlGf+B9/gB0GpFM</latexit><latexit sha1_base64="ICBz4Nuoqv8BUptsqPXTA09K6E0=">AAAB83icbVBNSwMxEM3Wr1q/qh69BBfBU9kVQcFL0YvHCvYDukvJptk2NJssyUQopX/DiwdFvPpnvPlvTNs9aOuDgcd7M8zMS3LBDQTBt1daW9/Y3CpvV3Z29/YPqodHLaOspqxJlVC6kxDDBJesCRwE6+SakSwRrJ2M7mZ++4lpw5V8hHHO4owMJE85JeCkKPKjG6ws5BZMr+oHtWAOvErCgvioQKNX/Yr6itqMSaCCGNMNgxziCdHAqWDTSmQNywkdkQHrOipJxkw8md88xWdO6eNUaVcS8Fz9PTEhmTHjLHGdGYGhWfZm4n9e10J6HU+4dC8xSReLUiswKDwLAPe5ZhTE2BFCNXe3YjokmlBwMVVcCOHyy6ukdVELg1r4cOnXb4s4yugEnaJzFKIrVEf3qIGaiKIcPaNX9OZZ78V79z4WrSWvmDlGf+B9/gB0GpFM</latexit><latexit sha1_base64="ICBz4Nuoqv8BUptsqPXTA09K6E0=">AAAB83icbVBNSwMxEM3Wr1q/qh69BBfBU9kVQcFL0YvHCvYDukvJptk2NJssyUQopX/DiwdFvPpnvPlvTNs9aOuDgcd7M8zMS3LBDQTBt1daW9/Y3CpvV3Z29/YPqodHLaOspqxJlVC6kxDDBJesCRwE6+SakSwRrJ2M7mZ++4lpw5V8hHHO4owMJE85JeCkKPKjG6ws5BZMr+oHtWAOvErCgvioQKNX/Yr6itqMSaCCGNMNgxziCdHAqWDTSmQNywkdkQHrOipJxkw8md88xWdO6eNUaVcS8Fz9PTEhmTHjLHGdGYGhWfZm4n9e10J6HU+4dC8xSReLUiswKDwLAPe5ZhTE2BFCNXe3YjokmlBwMVVcCOHyy6ukdVELg1r4cOnXb4s4yugEnaJzFKIrVEf3qIGaiKIcPaNX9OZZ78V79z4WrSWvmDlGf+B9/gB0GpFM</latexit><latexit sha1_base64="ICBz4Nuoqv8BUptsqPXTA09K6E0=">AAAB83icbVBNSwMxEM3Wr1q/qh69BBfBU9kVQcFL0YvHCvYDukvJptk2NJssyUQopX/DiwdFvPpnvPlvTNs9aOuDgcd7M8zMS3LBDQTBt1daW9/Y3CpvV3Z29/YPqodHLaOspqxJlVC6kxDDBJesCRwE6+SakSwRrJ2M7mZ++4lpw5V8hHHO4owMJE85JeCkKPKjG6ws5BZMr+oHtWAOvErCgvioQKNX/Yr6itqMSaCCGNMNgxziCdHAqWDTSmQNywkdkQHrOipJxkw8md88xWdO6eNUaVcS8Fz9PTEhmTHjLHGdGYGhWfZm4n9e10J6HU+4dC8xSReLUiswKDwLAPe5ZhTE2BFCNXe3YjokmlBwMVVcCOHyy6ukdVELg1r4cOnXb4s4yugEnaJzFKIrVEf3qIGaiKIcPaNX9OZZ78V79z4WrSWvmDlGf+B9/gB0GpFM</latexit>

Wl<latexit sha1_base64="PvPkXizAzMF6sNgh5sOcM2KubeY=">AAAB7HicbZBNS8NAEIYn9avWr6pHL4tF8FQSEeqx6MVjBdMW2lA22027dLMJuxOhhP4GLx4U8eoP8ua/cdvmoK0vLDy8M8POvGEqhUHX/XZKG5tb2zvl3cre/sHhUfX4pG2STDPus0QmuhtSw6VQ3EeBkndTzWkcSt4JJ3fzeueJayMS9YjTlAcxHSkRCUbRWn5nkMvZoFpz6+5CZB28AmpQqDWofvWHCctirpBJakzPc1MMcqpRMMlnlX5meErZhI54z6KiMTdBvlh2Ri6sMyRRou1TSBbu74mcxsZM49B2xhTHZrU2N/+r9TKMboJcqDRDrtjyoyiTBBMyv5wMheYM5dQCZVrYXQkbU00Z2nwqNgRv9eR1aF/VPcsP17XmbRFHGc7gHC7BgwY04R5a4AMDAc/wCm+Ocl6cd+dj2VpyiplT+CPn8wf3Z47G</latexit><latexit sha1_base64="PvPkXizAzMF6sNgh5sOcM2KubeY=">AAAB7HicbZBNS8NAEIYn9avWr6pHL4tF8FQSEeqx6MVjBdMW2lA22027dLMJuxOhhP4GLx4U8eoP8ua/cdvmoK0vLDy8M8POvGEqhUHX/XZKG5tb2zvl3cre/sHhUfX4pG2STDPus0QmuhtSw6VQ3EeBkndTzWkcSt4JJ3fzeueJayMS9YjTlAcxHSkRCUbRWn5nkMvZoFpz6+5CZB28AmpQqDWofvWHCctirpBJakzPc1MMcqpRMMlnlX5meErZhI54z6KiMTdBvlh2Ri6sMyRRou1TSBbu74mcxsZM49B2xhTHZrU2N/+r9TKMboJcqDRDrtjyoyiTBBMyv5wMheYM5dQCZVrYXQkbU00Z2nwqNgRv9eR1aF/VPcsP17XmbRFHGc7gHC7BgwY04R5a4AMDAc/wCm+Ocl6cd+dj2VpyiplT+CPn8wf3Z47G</latexit><latexit sha1_base64="PvPkXizAzMF6sNgh5sOcM2KubeY=">AAAB7HicbZBNS8NAEIYn9avWr6pHL4tF8FQSEeqx6MVjBdMW2lA22027dLMJuxOhhP4GLx4U8eoP8ua/cdvmoK0vLDy8M8POvGEqhUHX/XZKG5tb2zvl3cre/sHhUfX4pG2STDPus0QmuhtSw6VQ3EeBkndTzWkcSt4JJ3fzeueJayMS9YjTlAcxHSkRCUbRWn5nkMvZoFpz6+5CZB28AmpQqDWofvWHCctirpBJakzPc1MMcqpRMMlnlX5meErZhI54z6KiMTdBvlh2Ri6sMyRRou1TSBbu74mcxsZM49B2xhTHZrU2N/+r9TKMboJcqDRDrtjyoyiTBBMyv5wMheYM5dQCZVrYXQkbU00Z2nwqNgRv9eR1aF/VPcsP17XmbRFHGc7gHC7BgwY04R5a4AMDAc/wCm+Ocl6cd+dj2VpyiplT+CPn8wf3Z47G</latexit><latexit sha1_base64="PvPkXizAzMF6sNgh5sOcM2KubeY=">AAAB7HicbZBNS8NAEIYn9avWr6pHL4tF8FQSEeqx6MVjBdMW2lA22027dLMJuxOhhP4GLx4U8eoP8ua/cdvmoK0vLDy8M8POvGEqhUHX/XZKG5tb2zvl3cre/sHhUfX4pG2STDPus0QmuhtSw6VQ3EeBkndTzWkcSt4JJ3fzeueJayMS9YjTlAcxHSkRCUbRWn5nkMvZoFpz6+5CZB28AmpQqDWofvWHCctirpBJakzPc1MMcqpRMMlnlX5meErZhI54z6KiMTdBvlh2Ri6sMyRRou1TSBbu74mcxsZM49B2xhTHZrU2N/+r9TKMboJcqDRDrtjyoyiTBBMyv5wMheYM5dQCZVrYXQkbU00Z2nwqNgRv9eR1aF/VPcsP17XmbRFHGc7gHC7BgwY04R5a4AMDAc/wCm+Ocl6cd+dj2VpyiplT+CPn8wf3Z47G</latexit>

=<latexit sha1_base64="8uDF8DVeQnsAQmi3YnN5BydvDus=">AAAB6HicbZBNS8NAEIYn9avWr6pHL4tF8FQSEfQiFL14bMF+QBvKZjtp1242YXcjlNBf4MWDIl79Sd78N27bHLT1hYWHd2bYmTdIBNfGdb+dwtr6xuZWcbu0s7u3f1A+PGrpOFUMmywWseoEVKPgEpuGG4GdRCGNAoHtYHw3q7efUGkeywczSdCP6FDykDNqrNW46ZcrbtWdi6yCl0MFctX75a/eIGZphNIwQbXuem5i/Iwqw5nAaamXakwoG9Mhdi1KGqH2s/miU3JmnQEJY2WfNGTu/p7IaKT1JApsZ0TNSC/XZuZ/tW5qwms/4zJJDUq2+ChMBTExmV1NBlwhM2JigTLF7a6EjaiizNhsSjYEb/nkVWhdVD3LjctK7TaPowgncArn4MEV1OAe6tAEBgjP8ApvzqPz4rw7H4vWgpPPHMMfOZ8/jM+MwQ==</latexit><latexit sha1_base64="8uDF8DVeQnsAQmi3YnN5BydvDus=">AAAB6HicbZBNS8NAEIYn9avWr6pHL4tF8FQSEfQiFL14bMF+QBvKZjtp1242YXcjlNBf4MWDIl79Sd78N27bHLT1hYWHd2bYmTdIBNfGdb+dwtr6xuZWcbu0s7u3f1A+PGrpOFUMmywWseoEVKPgEpuGG4GdRCGNAoHtYHw3q7efUGkeywczSdCP6FDykDNqrNW46ZcrbtWdi6yCl0MFctX75a/eIGZphNIwQbXuem5i/Iwqw5nAaamXakwoG9Mhdi1KGqH2s/miU3JmnQEJY2WfNGTu/p7IaKT1JApsZ0TNSC/XZuZ/tW5qwms/4zJJDUq2+ChMBTExmV1NBlwhM2JigTLF7a6EjaiizNhsSjYEb/nkVWhdVD3LjctK7TaPowgncArn4MEV1OAe6tAEBgjP8ApvzqPz4rw7H4vWgpPPHMMfOZ8/jM+MwQ==</latexit><latexit sha1_base64="8uDF8DVeQnsAQmi3YnN5BydvDus=">AAAB6HicbZBNS8NAEIYn9avWr6pHL4tF8FQSEfQiFL14bMF+QBvKZjtp1242YXcjlNBf4MWDIl79Sd78N27bHLT1hYWHd2bYmTdIBNfGdb+dwtr6xuZWcbu0s7u3f1A+PGrpOFUMmywWseoEVKPgEpuGG4GdRCGNAoHtYHw3q7efUGkeywczSdCP6FDykDNqrNW46ZcrbtWdi6yCl0MFctX75a/eIGZphNIwQbXuem5i/Iwqw5nAaamXakwoG9Mhdi1KGqH2s/miU3JmnQEJY2WfNGTu/p7IaKT1JApsZ0TNSC/XZuZ/tW5qwms/4zJJDUq2+ChMBTExmV1NBlwhM2JigTLF7a6EjaiizNhsSjYEb/nkVWhdVD3LjctK7TaPowgncArn4MEV1OAe6tAEBgjP8ApvzqPz4rw7H4vWgpPPHMMfOZ8/jM+MwQ==</latexit><latexit sha1_base64="8uDF8DVeQnsAQmi3YnN5BydvDus=">AAAB6HicbZBNS8NAEIYn9avWr6pHL4tF8FQSEfQiFL14bMF+QBvKZjtp1242YXcjlNBf4MWDIl79Sd78N27bHLT1hYWHd2bYmTdIBNfGdb+dwtr6xuZWcbu0s7u3f1A+PGrpOFUMmywWseoEVKPgEpuGG4GdRCGNAoHtYHw3q7efUGkeywczSdCP6FDykDNqrNW46ZcrbtWdi6yCl0MFctX75a/eIGZphNIwQbXuem5i/Iwqw5nAaamXakwoG9Mhdi1KGqH2s/miU3JmnQEJY2WfNGTu/p7IaKT1JApsZ0TNSC/XZuZ/tW5qwms/4zJJDUq2+ChMBTExmV1NBlwhM2JigTLF7a6EjaiizNhsSjYEb/nkVWhdVD3LjctK7TaPowgncArn4MEV1OAe6tAEBgjP8ApvzqPz4rw7H4vWgpPPHMMfOZ8/jM+MwQ==</latexit>A<latexit sha1_base64="glsIYP92PSMcW8DkNNhZ8pvJRFs=">AAAB6HicbZBNS8NAEIYn9avWr6pHL4tF8FQSEfRY9eKxBfsBbSib7aRdu9mE3Y1QQn+BFw+KePUnefPfuG1z0NYXFh7emWFn3iARXBvX/XYKa+sbm1vF7dLO7t7+QfnwqKXjVDFssljEqhNQjYJLbBpuBHYShTQKBLaD8d2s3n5CpXksH8wkQT+iQ8lDzqixVuOmX664VXcusgpeDhXIVe+Xv3qDmKURSsME1brruYnxM6oMZwKnpV6qMaFsTIfYtShphNrP5otOyZl1BiSMlX3SkLn7eyKjkdaTKLCdETUjvVybmf/VuqkJr/2MyyQ1KNniozAVxMRkdjUZcIXMiIkFyhS3uxI2oooyY7Mp2RC85ZNXoXVR9Sw3Liu12zyOIpzAKZyDB1dQg3uoQxMYIDzDK7w5j86L8+58LFoLTj5zDH/kfP4Akt+MxQ==</latexit><latexit sha1_base64="glsIYP92PSMcW8DkNNhZ8pvJRFs=">AAAB6HicbZBNS8NAEIYn9avWr6pHL4tF8FQSEfRY9eKxBfsBbSib7aRdu9mE3Y1QQn+BFw+KePUnefPfuG1z0NYXFh7emWFn3iARXBvX/XYKa+sbm1vF7dLO7t7+QfnwqKXjVDFssljEqhNQjYJLbBpuBHYShTQKBLaD8d2s3n5CpXksH8wkQT+iQ8lDzqixVuOmX664VXcusgpeDhXIVe+Xv3qDmKURSsME1brruYnxM6oMZwKnpV6qMaFsTIfYtShphNrP5otOyZl1BiSMlX3SkLn7eyKjkdaTKLCdETUjvVybmf/VuqkJr/2MyyQ1KNniozAVxMRkdjUZcIXMiIkFyhS3uxI2oooyY7Mp2RC85ZNXoXVR9Sw3Liu12zyOIpzAKZyDB1dQg3uoQxMYIDzDK7w5j86L8+58LFoLTj5zDH/kfP4Akt+MxQ==</latexit><latexit sha1_base64="glsIYP92PSMcW8DkNNhZ8pvJRFs=">AAAB6HicbZBNS8NAEIYn9avWr6pHL4tF8FQSEfRY9eKxBfsBbSib7aRdu9mE3Y1QQn+BFw+KePUnefPfuG1z0NYXFh7emWFn3iARXBvX/XYKa+sbm1vF7dLO7t7+QfnwqKXjVDFssljEqhNQjYJLbBpuBHYShTQKBLaD8d2s3n5CpXksH8wkQT+iQ8lDzqixVuOmX664VXcusgpeDhXIVe+Xv3qDmKURSsME1brruYnxM6oMZwKnpV6qMaFsTIfYtShphNrP5otOyZl1BiSMlX3SkLn7eyKjkdaTKLCdETUjvVybmf/VuqkJr/2MyyQ1KNniozAVxMRkdjUZcIXMiIkFyhS3uxI2oooyY7Mp2RC85ZNXoXVR9Sw3Liu12zyOIpzAKZyDB1dQg3uoQxMYIDzDK7w5j86L8+58LFoLTj5zDH/kfP4Akt+MxQ==</latexit><latexit sha1_base64="glsIYP92PSMcW8DkNNhZ8pvJRFs=">AAAB6HicbZBNS8NAEIYn9avWr6pHL4tF8FQSEfRY9eKxBfsBbSib7aRdu9mE3Y1QQn+BFw+KePUnefPfuG1z0NYXFh7emWFn3iARXBvX/XYKa+sbm1vF7dLO7t7+QfnwqKXjVDFssljEqhNQjYJLbBpuBHYShTQKBLaD8d2s3n5CpXksH8wkQT+iQ8lDzqixVuOmX664VXcusgpeDhXIVe+Xv3qDmKURSsME1brruYnxM6oMZwKnpV6qMaFsTIfYtShphNrP5otOyZl1BiSMlX3SkLn7eyKjkdaTKLCdETUjvVybmf/VuqkJr/2MyyQ1KNniozAVxMRkdjUZcIXMiIkFyhS3uxI2oooyY7Mp2RC85ZNXoXVR9Sw3Liu12zyOIpzAKZyDB1dQg3uoQxMYIDzDK7w5j86L8+58LFoLTj5zDH/kfP4Akt+MxQ==</latexit>

n<latexit sha1_base64="yQeyt1ATXRuagWZEsCnmGpLGA/Q=">AAAB6HicbZBNS8NAEIYn9avWr6pHL4tF8FQSEeqx6MVjC/YD2lA220m7drMJuxuhhP4CLx4U8epP8ua/cdvmoK0vLDy8M8POvEEiuDau++0UNja3tneKu6W9/YPDo/LxSVvHqWLYYrGIVTegGgWX2DLcCOwmCmkUCOwEk7t5vfOESvNYPphpgn5ER5KHnFFjraYclCtu1V2IrIOXQwVyNQblr/4wZmmE0jBBte55bmL8jCrDmcBZqZ9qTCib0BH2LEoaofazxaIzcmGdIQljZZ80ZOH+nshopPU0CmxnRM1Yr9bm5n+1XmrCGz/jMkkNSrb8KEwFMTGZX02GXCEzYmqBMsXtroSNqaLM2GxKNgRv9eR1aF9VPcvN60r9No+jCGdwDpfgQQ3qcA8NaAEDhGd4hTfn0Xlx3p2PZWvByWdO4Y+czx/XE4zy</latexit><latexit sha1_base64="yQeyt1ATXRuagWZEsCnmGpLGA/Q=">AAAB6HicbZBNS8NAEIYn9avWr6pHL4tF8FQSEeqx6MVjC/YD2lA220m7drMJuxuhhP4CLx4U8epP8ua/cdvmoK0vLDy8M8POvEEiuDau++0UNja3tneKu6W9/YPDo/LxSVvHqWLYYrGIVTegGgWX2DLcCOwmCmkUCOwEk7t5vfOESvNYPphpgn5ER5KHnFFjraYclCtu1V2IrIOXQwVyNQblr/4wZmmE0jBBte55bmL8jCrDmcBZqZ9qTCib0BH2LEoaofazxaIzcmGdIQljZZ80ZOH+nshopPU0CmxnRM1Yr9bm5n+1XmrCGz/jMkkNSrb8KEwFMTGZX02GXCEzYmqBMsXtroSNqaLM2GxKNgRv9eR1aF9VPcvN60r9No+jCGdwDpfgQQ3qcA8NaAEDhGd4hTfn0Xlx3p2PZWvByWdO4Y+czx/XE4zy</latexit><latexit sha1_base64="yQeyt1ATXRuagWZEsCnmGpLGA/Q=">AAAB6HicbZBNS8NAEIYn9avWr6pHL4tF8FQSEeqx6MVjC/YD2lA220m7drMJuxuhhP4CLx4U8epP8ua/cdvmoK0vLDy8M8POvEEiuDau++0UNja3tneKu6W9/YPDo/LxSVvHqWLYYrGIVTegGgWX2DLcCOwmCmkUCOwEk7t5vfOESvNYPphpgn5ER5KHnFFjraYclCtu1V2IrIOXQwVyNQblr/4wZmmE0jBBte55bmL8jCrDmcBZqZ9qTCib0BH2LEoaofazxaIzcmGdIQljZZ80ZOH+nshopPU0CmxnRM1Yr9bm5n+1XmrCGz/jMkkNSrb8KEwFMTGZX02GXCEzYmqBMsXtroSNqaLM2GxKNgRv9eR1aF9VPcvN60r9No+jCGdwDpfgQQ3qcA8NaAEDhGd4hTfn0Xlx3p2PZWvByWdO4Y+czx/XE4zy</latexit><latexit sha1_base64="yQeyt1ATXRuagWZEsCnmGpLGA/Q=">AAAB6HicbZBNS8NAEIYn9avWr6pHL4tF8FQSEeqx6MVjC/YD2lA220m7drMJuxuhhP4CLx4U8epP8ua/cdvmoK0vLDy8M8POvEEiuDau++0UNja3tneKu6W9/YPDo/LxSVvHqWLYYrGIVTegGgWX2DLcCOwmCmkUCOwEk7t5vfOESvNYPphpgn5ER5KHnFFjraYclCtu1V2IrIOXQwVyNQblr/4wZmmE0jBBte55bmL8jCrDmcBZqZ9qTCib0BH2LEoaofazxaIzcmGdIQljZZ80ZOH+nshopPU0CmxnRM1Yr9bm5n+1XmrCGz/jMkkNSrb8KEwFMTGZX02GXCEzYmqBMsXtroSNqaLM2GxKNgRv9eR1aF9VPcvN60r9No+jCGdwDpfgQQ3qcA8NaAEDhGd4hTfn0Xlx3p2PZWvByWdO4Y+czx/XE4zy</latexit>

Bl<latexit sha1_base64="gI+PvoN5GXzEGi6kvNlLK4N/7Fk=">AAAB6nicbZBNS8NAEIYn9avWr6pHL4tF8FQSEeqx1IvHivYD2lA22027dLMJuxOhhP4ELx4U8eov8ua/cdvmoK0vLDy8M8POvEEihUHX/XYKG5tb2zvF3dLe/sHhUfn4pG3iVDPeYrGMdTeghkuheAsFSt5NNKdRIHknmNzO650nro2I1SNOE+5HdKREKBhFaz00BnJQrrhVdyGyDl4OFcjVHJS/+sOYpRFXyCQ1pue5CfoZ1SiY5LNSPzU8oWxCR7xnUdGIGz9brDojF9YZkjDW9ikkC/f3REYjY6ZRYDsjimOzWpub/9V6KYY3fiZUkiJXbPlRmEqCMZnfTYZCc4ZyaoEyLeyuhI2ppgxtOiUbgrd68jq0r6qe5fvrSr2Rx1GEMziHS/CgBnW4gya0gMEInuEV3hzpvDjvzseyteDkM6fwR87nDxLajaU=</latexit><latexit sha1_base64="gI+PvoN5GXzEGi6kvNlLK4N/7Fk=">AAAB6nicbZBNS8NAEIYn9avWr6pHL4tF8FQSEeqx1IvHivYD2lA22027dLMJuxOhhP4ELx4U8eov8ua/cdvmoK0vLDy8M8POvEEihUHX/XYKG5tb2zvF3dLe/sHhUfn4pG3iVDPeYrGMdTeghkuheAsFSt5NNKdRIHknmNzO650nro2I1SNOE+5HdKREKBhFaz00BnJQrrhVdyGyDl4OFcjVHJS/+sOYpRFXyCQ1pue5CfoZ1SiY5LNSPzU8oWxCR7xnUdGIGz9brDojF9YZkjDW9ikkC/f3REYjY6ZRYDsjimOzWpub/9V6KYY3fiZUkiJXbPlRmEqCMZnfTYZCc4ZyaoEyLeyuhI2ppgxtOiUbgrd68jq0r6qe5fvrSr2Rx1GEMziHS/CgBnW4gya0gMEInuEV3hzpvDjvzseyteDkM6fwR87nDxLajaU=</latexit><latexit sha1_base64="gI+PvoN5GXzEGi6kvNlLK4N/7Fk=">AAAB6nicbZBNS8NAEIYn9avWr6pHL4tF8FQSEeqx1IvHivYD2lA22027dLMJuxOhhP4ELx4U8eov8ua/cdvmoK0vLDy8M8POvEEihUHX/XYKG5tb2zvF3dLe/sHhUfn4pG3iVDPeYrGMdTeghkuheAsFSt5NNKdRIHknmNzO650nro2I1SNOE+5HdKREKBhFaz00BnJQrrhVdyGyDl4OFcjVHJS/+sOYpRFXyCQ1pue5CfoZ1SiY5LNSPzU8oWxCR7xnUdGIGz9brDojF9YZkjDW9ikkC/f3REYjY6ZRYDsjimOzWpub/9V6KYY3fiZUkiJXbPlRmEqCMZnfTYZCc4ZyaoEyLeyuhI2ppgxtOiUbgrd68jq0r6qe5fvrSr2Rx1GEMziHS/CgBnW4gya0gMEInuEV3hzpvDjvzseyteDkM6fwR87nDxLajaU=</latexit><latexit sha1_base64="gI+PvoN5GXzEGi6kvNlLK4N/7Fk=">AAAB6nicbZBNS8NAEIYn9avWr6pHL4tF8FQSEeqx1IvHivYD2lA22027dLMJuxOhhP4ELx4U8eov8ua/cdvmoK0vLDy8M8POvEEihUHX/XYKG5tb2zvF3dLe/sHhUfn4pG3iVDPeYrGMdTeghkuheAsFSt5NNKdRIHknmNzO650nro2I1SNOE+5HdKREKBhFaz00BnJQrrhVdyGyDl4OFcjVHJS/+sOYpRFXyCQ1pue5CfoZ1SiY5LNSPzU8oWxCR7xnUdGIGz9brDojF9YZkjDW9ikkC/f3REYjY6ZRYDsjimOzWpub/9V6KYY3fiZUkiJXbPlRmEqCMZnfTYZCc4ZyaoEyLeyuhI2ppgxtOiUbgrd68jq0r6qe5fvrSr2Rx1GEMziHS/CgBnW4gya0gMEInuEV3hzpvDjvzseyteDkM6fwR87nDxLajaU=</latexit>

B(s)l � q(B)

<latexit sha1_base64="EGfudgcG6vg2QSFphj26RsjDih8=">AAAB/HicbZDLSsNAFIZPvNZ6i3bpZrAI7aYkIuiy1I3LCvYCbQyT6bQdOpPEmYkQQn0VNy4UceuDuPNtnLZZaOsPAx//OYdz5g9izpR2nG9rbX1jc2u7sFPc3ds/OLSPjtsqSiShLRLxSHYDrChnIW1ppjntxpJiEXDaCSbXs3rnkUrFovBOpzH1BB6FbMgI1sby7VLD5/dZRVWnqK+YQA+VRtW3y07NmQutgptDGXI1ffurP4hIImioCcdK9Vwn1l6GpWaE02mxnygaYzLBI9ozGGJBlZfNj5+iM+MM0DCS5oUazd3fExkWSqUiMJ0C67Fars3M/2q9RA+vvIyFcaJpSBaLhglHOkKzJNCASUo0Tw1gIpm5FZExlphok1fRhOAuf3kV2uc11/DtRbneyOMowAmcQgVcuIQ63EATWkAghWd4hTfryXqx3q2PReualc+U4I+szx+s25N5</latexit><latexit sha1_base64="EGfudgcG6vg2QSFphj26RsjDih8=">AAAB/HicbZDLSsNAFIZPvNZ6i3bpZrAI7aYkIuiy1I3LCvYCbQyT6bQdOpPEmYkQQn0VNy4UceuDuPNtnLZZaOsPAx//OYdz5g9izpR2nG9rbX1jc2u7sFPc3ds/OLSPjtsqSiShLRLxSHYDrChnIW1ppjntxpJiEXDaCSbXs3rnkUrFovBOpzH1BB6FbMgI1sby7VLD5/dZRVWnqK+YQA+VRtW3y07NmQutgptDGXI1ffurP4hIImioCcdK9Vwn1l6GpWaE02mxnygaYzLBI9ozGGJBlZfNj5+iM+MM0DCS5oUazd3fExkWSqUiMJ0C67Fars3M/2q9RA+vvIyFcaJpSBaLhglHOkKzJNCASUo0Tw1gIpm5FZExlphok1fRhOAuf3kV2uc11/DtRbneyOMowAmcQgVcuIQ63EATWkAghWd4hTfryXqx3q2PReualc+U4I+szx+s25N5</latexit><latexit sha1_base64="EGfudgcG6vg2QSFphj26RsjDih8=">AAAB/HicbZDLSsNAFIZPvNZ6i3bpZrAI7aYkIuiy1I3LCvYCbQyT6bQdOpPEmYkQQn0VNy4UceuDuPNtnLZZaOsPAx//OYdz5g9izpR2nG9rbX1jc2u7sFPc3ds/OLSPjtsqSiShLRLxSHYDrChnIW1ppjntxpJiEXDaCSbXs3rnkUrFovBOpzH1BB6FbMgI1sby7VLD5/dZRVWnqK+YQA+VRtW3y07NmQutgptDGXI1ffurP4hIImioCcdK9Vwn1l6GpWaE02mxnygaYzLBI9ozGGJBlZfNj5+iM+MM0DCS5oUazd3fExkWSqUiMJ0C67Fars3M/2q9RA+vvIyFcaJpSBaLhglHOkKzJNCASUo0Tw1gIpm5FZExlphok1fRhOAuf3kV2uc11/DtRbneyOMowAmcQgVcuIQ63EATWkAghWd4hTfryXqx3q2PReualc+U4I+szx+s25N5</latexit><latexit sha1_base64="EGfudgcG6vg2QSFphj26RsjDih8=">AAAB/HicbZDLSsNAFIZPvNZ6i3bpZrAI7aYkIuiy1I3LCvYCbQyT6bQdOpPEmYkQQn0VNy4UceuDuPNtnLZZaOsPAx//OYdz5g9izpR2nG9rbX1jc2u7sFPc3ds/OLSPjtsqSiShLRLxSHYDrChnIW1ppjntxpJiEXDaCSbXs3rnkUrFovBOpzH1BB6FbMgI1sby7VLD5/dZRVWnqK+YQA+VRtW3y07NmQutgptDGXI1ffurP4hIImioCcdK9Vwn1l6GpWaE02mxnygaYzLBI9ozGGJBlZfNj5+iM+MM0DCS5oUazd3fExkWSqUiMJ0C67Fars3M/2q9RA+vvIyFcaJpSBaLhglHOkKzJNCASUo0Tw1gIpm5FZExlphok1fRhOAuf3kV2uc11/DtRbneyOMowAmcQgVcuIQ63EATWkAghWd4hTfryXqx3q2PReualc+U4I+szx+s25N5</latexit>

For certain q(W), form of implied q(B = AW) is knownpre-activations inputs

Variational dropout and the local reparameterization trick, Kingma’ 2015

Page 21: Horseshoe Priors for Bayesian Neural Networks...Horseshoe Priors for Bayesian Neural Networks Soumya Ghosh Jiayu Yao Finale Doshi-Velez IBM Research MIT-IBM Watson AI lab Harvard Harvard

Variational Family

!21

q(W, �, v; �) =�

i,j,l

N (wi,j,l | µi,j,l, �2i,j,l)

k,l

q(�k,l | ��k,l)�

l

q(vl | �vl)

<latexit sha1_base64="UI5qEDe9KVQgsy5cUXGrEV1tWAI=">AAACnHicbVFtaxNBEN47rdZYNdVPIshiEBII5a4ILRSxaD9ExFLBJIVcPOb2Nsma3bvr7l4kHPer+k/6rf/GSXIxfXFg2WeeeXl2Z6JMCmM979pxHzzcevR4+0nt6c6z5y/quy97Js01412WylSfR2C4FAnvWmElP880BxVJ3o+mXxbx/oxrI9Lkp51nfKhgnIiRYGCRCuuXF81AgZ0wkEW/bNPAQt6msyMaZBPRoh/x1mkcFqL9uy3Lf6mnZfPPhhQxDVS+9rGJEWMFv/bXTKtqMkWMeiixwqtKFAqLDVm21pqypBfNWShvpKFXtsJ6w9vzlkbvA78CDVLZWVi/CuKU5YonlkkwZuB7mR0WoK1gkpe1IDc8AzaFMR8gTEBxMyyWwy3pe2RiOko1nsTSJXuzogBlzFxFmLmYjrkbW5D/iw1yOzocFiLJcssTthIa5ZLalC42RWOhObNyjgCYFvhWyiaggVncZw2H4N/98n3Q29/zEf/40Dj+XI1jm7wh70iT+OSAHJMOOSNdwpzXzien43x137on7jf3+yrVdaqaV+SWub2/aZbMRQ==</latexit><latexit sha1_base64="UI5qEDe9KVQgsy5cUXGrEV1tWAI=">AAACnHicbVFtaxNBEN47rdZYNdVPIshiEBII5a4ILRSxaD9ExFLBJIVcPOb2Nsma3bvr7l4kHPer+k/6rf/GSXIxfXFg2WeeeXl2Z6JMCmM979pxHzzcevR4+0nt6c6z5y/quy97Js01412WylSfR2C4FAnvWmElP880BxVJ3o+mXxbx/oxrI9Lkp51nfKhgnIiRYGCRCuuXF81AgZ0wkEW/bNPAQt6msyMaZBPRoh/x1mkcFqL9uy3Lf6mnZfPPhhQxDVS+9rGJEWMFv/bXTKtqMkWMeiixwqtKFAqLDVm21pqypBfNWShvpKFXtsJ6w9vzlkbvA78CDVLZWVi/CuKU5YonlkkwZuB7mR0WoK1gkpe1IDc8AzaFMR8gTEBxMyyWwy3pe2RiOko1nsTSJXuzogBlzFxFmLmYjrkbW5D/iw1yOzocFiLJcssTthIa5ZLalC42RWOhObNyjgCYFvhWyiaggVncZw2H4N/98n3Q29/zEf/40Dj+XI1jm7wh70iT+OSAHJMOOSNdwpzXzien43x137on7jf3+yrVdaqaV+SWub2/aZbMRQ==</latexit><latexit sha1_base64="UI5qEDe9KVQgsy5cUXGrEV1tWAI=">AAACnHicbVFtaxNBEN47rdZYNdVPIshiEBII5a4ILRSxaD9ExFLBJIVcPOb2Nsma3bvr7l4kHPer+k/6rf/GSXIxfXFg2WeeeXl2Z6JMCmM979pxHzzcevR4+0nt6c6z5y/quy97Js01412WylSfR2C4FAnvWmElP880BxVJ3o+mXxbx/oxrI9Lkp51nfKhgnIiRYGCRCuuXF81AgZ0wkEW/bNPAQt6msyMaZBPRoh/x1mkcFqL9uy3Lf6mnZfPPhhQxDVS+9rGJEWMFv/bXTKtqMkWMeiixwqtKFAqLDVm21pqypBfNWShvpKFXtsJ6w9vzlkbvA78CDVLZWVi/CuKU5YonlkkwZuB7mR0WoK1gkpe1IDc8AzaFMR8gTEBxMyyWwy3pe2RiOko1nsTSJXuzogBlzFxFmLmYjrkbW5D/iw1yOzocFiLJcssTthIa5ZLalC42RWOhObNyjgCYFvhWyiaggVncZw2H4N/98n3Q29/zEf/40Dj+XI1jm7wh70iT+OSAHJMOOSNdwpzXzien43x137on7jf3+yrVdaqaV+SWub2/aZbMRQ==</latexit><latexit sha1_base64="UI5qEDe9KVQgsy5cUXGrEV1tWAI=">AAACnHicbVFtaxNBEN47rdZYNdVPIshiEBII5a4ILRSxaD9ExFLBJIVcPOb2Nsma3bvr7l4kHPer+k/6rf/GSXIxfXFg2WeeeXl2Z6JMCmM979pxHzzcevR4+0nt6c6z5y/quy97Js01412WylSfR2C4FAnvWmElP880BxVJ3o+mXxbx/oxrI9Lkp51nfKhgnIiRYGCRCuuXF81AgZ0wkEW/bNPAQt6msyMaZBPRoh/x1mkcFqL9uy3Lf6mnZfPPhhQxDVS+9rGJEWMFv/bXTKtqMkWMeiixwqtKFAqLDVm21pqypBfNWShvpKFXtsJ6w9vzlkbvA78CDVLZWVi/CuKU5YonlkkwZuB7mR0WoK1gkpe1IDc8AzaFMR8gTEBxMyyWwy3pe2RiOko1nsTSJXuzogBlzFxFmLmYjrkbW5D/iw1yOzocFiLJcssTthIa5ZLalC42RWOhObNyjgCYFvhWyiaggVncZw2H4N/98n3Q29/zEf/40Dj+XI1jm7wh70iT+OSAHJMOOSNdwpzXzien43x137on7jf3+yrVdaqaV+SWub2/aZbMRQ==</latexit>

• Fully factorized variational approximation

Louizos et. al., 2017 Ghosh & Doshi-Velez 2017

• But Horseshoe shrinkage stems from coupling between weights and scales

• Retaining this structure is important for strong shrinkage!

Page 22: Horseshoe Priors for Bayesian Neural Networks...Horseshoe Priors for Bayesian Neural Networks Soumya Ghosh Jiayu Yao Finale Doshi-Velez IBM Research MIT-IBM Watson AI lab Harvard Harvard

Group Horseshoe Priors for BNNs

For each layer , draw a global scale:

For node k in layer :− Draw a local scale for the node: − For each incident weight:

l<latexit sha1_base64="RXFmeCFAFLWm6A2HPHJ7FvtnIgg=">AAAB6HicbVBNS8NAEJ3Ur1q/qh69LBbBU0lEqMeiF48t2A9oQ9lsJ+3azSbsboQS+gu8eFDEqz/Jm//GbZuDtj4YeLw3w8y8IBFcG9f9dgobm1vbO8Xd0t7+weFR+fikreNUMWyxWMSqG1CNgktsGW4EdhOFNAoEdoLJ3dzvPKHSPJYPZpqgH9GR5CFn1FipKQblilt1FyDrxMtJBXI0BuWv/jBmaYTSMEG17nluYvyMKsOZwFmpn2pMKJvQEfYslTRC7WeLQ2fkwipDEsbKljRkof6eyGik9TQKbGdEzVivenPxP6+XmvDGz7hMUoOSLReFqSAmJvOvyZArZEZMLaFMcXsrYWOqKDM2m5INwVt9eZ20r6qeW/Wa15X6bR5HEc7gHC7BgxrU4R4a0AIGCM/wCm/Oo/PivDsfy9aCk8+cwh84nz/UCYzw</latexit><latexit sha1_base64="RXFmeCFAFLWm6A2HPHJ7FvtnIgg=">AAAB6HicbVBNS8NAEJ3Ur1q/qh69LBbBU0lEqMeiF48t2A9oQ9lsJ+3azSbsboQS+gu8eFDEqz/Jm//GbZuDtj4YeLw3w8y8IBFcG9f9dgobm1vbO8Xd0t7+weFR+fikreNUMWyxWMSqG1CNgktsGW4EdhOFNAoEdoLJ3dzvPKHSPJYPZpqgH9GR5CFn1FipKQblilt1FyDrxMtJBXI0BuWv/jBmaYTSMEG17nluYvyMKsOZwFmpn2pMKJvQEfYslTRC7WeLQ2fkwipDEsbKljRkof6eyGik9TQKbGdEzVivenPxP6+XmvDGz7hMUoOSLReFqSAmJvOvyZArZEZMLaFMcXsrYWOqKDM2m5INwVt9eZ20r6qeW/Wa15X6bR5HEc7gHC7BgxrU4R4a0AIGCM/wCm/Oo/PivDsfy9aCk8+cwh84nz/UCYzw</latexit><latexit sha1_base64="RXFmeCFAFLWm6A2HPHJ7FvtnIgg=">AAAB6HicbVBNS8NAEJ3Ur1q/qh69LBbBU0lEqMeiF48t2A9oQ9lsJ+3azSbsboQS+gu8eFDEqz/Jm//GbZuDtj4YeLw3w8y8IBFcG9f9dgobm1vbO8Xd0t7+weFR+fikreNUMWyxWMSqG1CNgktsGW4EdhOFNAoEdoLJ3dzvPKHSPJYPZpqgH9GR5CFn1FipKQblilt1FyDrxMtJBXI0BuWv/jBmaYTSMEG17nluYvyMKsOZwFmpn2pMKJvQEfYslTRC7WeLQ2fkwipDEsbKljRkof6eyGik9TQKbGdEzVivenPxP6+XmvDGz7hMUoOSLReFqSAmJvOvyZArZEZMLaFMcXsrYWOqKDM2m5INwVt9eZ20r6qeW/Wa15X6bR5HEc7gHC7BgxrU4R4a0AIGCM/wCm/Oo/PivDsfy9aCk8+cwh84nz/UCYzw</latexit><latexit sha1_base64="RXFmeCFAFLWm6A2HPHJ7FvtnIgg=">AAAB6HicbVBNS8NAEJ3Ur1q/qh69LBbBU0lEqMeiF48t2A9oQ9lsJ+3azSbsboQS+gu8eFDEqz/Jm//GbZuDtj4YeLw3w8y8IBFcG9f9dgobm1vbO8Xd0t7+weFR+fikreNUMWyxWMSqG1CNgktsGW4EdhOFNAoEdoLJ3dzvPKHSPJYPZpqgH9GR5CFn1FipKQblilt1FyDrxMtJBXI0BuWv/jBmaYTSMEG17nluYvyMKsOZwFmpn2pMKJvQEfYslTRC7WeLQ2fkwipDEsbKljRkof6eyGik9TQKbGdEzVivenPxP6+XmvDGz7hMUoOSLReFqSAmJvOvyZArZEZMLaFMcXsrYWOqKDM2m5INwVt9eZ20r6qeW/Wa15X6bR5HEc7gHC7BgxrU4R4a0AIGCM/wCm/Oo/PivDsfy9aCk8+cwh84nz/UCYzw</latexit>

vl � C+(0, bg)<latexit sha1_base64="FYjv0EmN+tYalCV/fh0u60y14QE=">AAAB/XicbVDLSsNAFL2pr1pf8bFzM1iEilISEXRZ7MZlBfuANobJdNIOziRhZlKopfgrblwo4tb/cOffOG2z0NYDFw7n3Mu99wQJZ0o7zreVW1peWV3Lrxc2Nre2d+zdvYaKU0loncQ8lq0AK8pZROuaaU5biaRYBJw2g4fqxG8OqFQsju70MKGewL2IhYxgbSTfPhj4HHUUE6h6f1pyzlDg9058u+iUnSnQInEzUoQMNd/+6nRjkgoaacKxUm3XSbQ3wlIzwum40EkVTTB5wD3aNjTCgipvNL1+jI6N0kVhLE1FGk3V3xMjLJQaisB0Cqz7at6biP957VSHV96IRUmqaURmi8KUIx2jSRSoyyQlmg8NwUQycysifSwx0SawggnBnX95kTTOy65Tdm8vipXrLI48HMIRlMCFS6jADdSgDgQe4Rle4c16sl6sd+tj1pqzspl9+APr8weW+ZNa</latexit><latexit sha1_base64="FYjv0EmN+tYalCV/fh0u60y14QE=">AAAB/XicbVDLSsNAFL2pr1pf8bFzM1iEilISEXRZ7MZlBfuANobJdNIOziRhZlKopfgrblwo4tb/cOffOG2z0NYDFw7n3Mu99wQJZ0o7zreVW1peWV3Lrxc2Nre2d+zdvYaKU0loncQ8lq0AK8pZROuaaU5biaRYBJw2g4fqxG8OqFQsju70MKGewL2IhYxgbSTfPhj4HHUUE6h6f1pyzlDg9058u+iUnSnQInEzUoQMNd/+6nRjkgoaacKxUm3XSbQ3wlIzwum40EkVTTB5wD3aNjTCgipvNL1+jI6N0kVhLE1FGk3V3xMjLJQaisB0Cqz7at6biP957VSHV96IRUmqaURmi8KUIx2jSRSoyyQlmg8NwUQycysifSwx0SawggnBnX95kTTOy65Tdm8vipXrLI48HMIRlMCFS6jADdSgDgQe4Rle4c16sl6sd+tj1pqzspl9+APr8weW+ZNa</latexit><latexit sha1_base64="FYjv0EmN+tYalCV/fh0u60y14QE=">AAAB/XicbVDLSsNAFL2pr1pf8bFzM1iEilISEXRZ7MZlBfuANobJdNIOziRhZlKopfgrblwo4tb/cOffOG2z0NYDFw7n3Mu99wQJZ0o7zreVW1peWV3Lrxc2Nre2d+zdvYaKU0loncQ8lq0AK8pZROuaaU5biaRYBJw2g4fqxG8OqFQsju70MKGewL2IhYxgbSTfPhj4HHUUE6h6f1pyzlDg9058u+iUnSnQInEzUoQMNd/+6nRjkgoaacKxUm3XSbQ3wlIzwum40EkVTTB5wD3aNjTCgipvNL1+jI6N0kVhLE1FGk3V3xMjLJQaisB0Cqz7at6biP957VSHV96IRUmqaURmi8KUIx2jSRSoyyQlmg8NwUQycysifSwx0SawggnBnX95kTTOy65Tdm8vipXrLI48HMIRlMCFS6jADdSgDgQe4Rle4c16sl6sd+tj1pqzspl9+APr8weW+ZNa</latexit><latexit sha1_base64="FYjv0EmN+tYalCV/fh0u60y14QE=">AAAB/XicbVDLSsNAFL2pr1pf8bFzM1iEilISEXRZ7MZlBfuANobJdNIOziRhZlKopfgrblwo4tb/cOffOG2z0NYDFw7n3Mu99wQJZ0o7zreVW1peWV3Lrxc2Nre2d+zdvYaKU0loncQ8lq0AK8pZROuaaU5biaRYBJw2g4fqxG8OqFQsju70MKGewL2IhYxgbSTfPhj4HHUUE6h6f1pyzlDg9058u+iUnSnQInEzUoQMNd/+6nRjkgoaacKxUm3XSbQ3wlIzwum40EkVTTB5wD3aNjTCgipvNL1+jI6N0kVhLE1FGk3V3xMjLJQaisB0Cqz7at6biP957VSHV96IRUmqaURmi8KUIx2jSRSoyyQlmg8NwUQycysifSwx0SawggnBnX95kTTOy65Tdm8vipXrLI48HMIRlMCFS6jADdSgDgQe4Rle4c16sl6sd+tj1pqzspl9+APr8weW+ZNa</latexit>

�kl � C+(0, b0)<latexit sha1_base64="J4XknDenb36+9rwby7B1axazf6g=">AAACA3icbVDLSsNAFJ3UV62vqDvdDBahopSJCLosduOygn1AE8NkOmmHziRhZiKUUHDjr7hxoYhbf8Kdf+O0zUJbD1w4nHMv994TJJwpjdC3VVhaXlldK66XNja3tnfs3b2WilNJaJPEPJadACvKWUSbmmlOO4mkWASctoNhfeK3H6hULI7u9CihnsD9iIWMYG0k3z5wNU79bMjH0FVMwPr9aQWdwcBHJ75dRlU0BVwkTk7KIEfDt7/cXkxSQSNNOFaq66BEexmWmhFOxyU3VTTBZIj7tGtohAVVXjb9YQyPjdKDYSxNRRpO1d8TGRZKjURgOgXWAzXvTcT/vG6qwysvY1GSahqR2aIw5VDHcBII7DFJieYjQzCRzNwKyQBLTLSJrWRCcOZfXiSt86qDqs7tRbl2ncdRBIfgCFSAAy5BDdyABmgCAh7BM3gFb9aT9WK9Wx+z1oKVz+yDP7A+fwBC05Xy</latexit><latexit sha1_base64="J4XknDenb36+9rwby7B1axazf6g=">AAACA3icbVDLSsNAFJ3UV62vqDvdDBahopSJCLosduOygn1AE8NkOmmHziRhZiKUUHDjr7hxoYhbf8Kdf+O0zUJbD1w4nHMv994TJJwpjdC3VVhaXlldK66XNja3tnfs3b2WilNJaJPEPJadACvKWUSbmmlOO4mkWASctoNhfeK3H6hULI7u9CihnsD9iIWMYG0k3z5wNU79bMjH0FVMwPr9aQWdwcBHJ75dRlU0BVwkTk7KIEfDt7/cXkxSQSNNOFaq66BEexmWmhFOxyU3VTTBZIj7tGtohAVVXjb9YQyPjdKDYSxNRRpO1d8TGRZKjURgOgXWAzXvTcT/vG6qwysvY1GSahqR2aIw5VDHcBII7DFJieYjQzCRzNwKyQBLTLSJrWRCcOZfXiSt86qDqs7tRbl2ncdRBIfgCFSAAy5BDdyABmgCAh7BM3gFb9aT9WK9Wx+z1oKVz+yDP7A+fwBC05Xy</latexit><latexit sha1_base64="J4XknDenb36+9rwby7B1axazf6g=">AAACA3icbVDLSsNAFJ3UV62vqDvdDBahopSJCLosduOygn1AE8NkOmmHziRhZiKUUHDjr7hxoYhbf8Kdf+O0zUJbD1w4nHMv994TJJwpjdC3VVhaXlldK66XNja3tnfs3b2WilNJaJPEPJadACvKWUSbmmlOO4mkWASctoNhfeK3H6hULI7u9CihnsD9iIWMYG0k3z5wNU79bMjH0FVMwPr9aQWdwcBHJ75dRlU0BVwkTk7KIEfDt7/cXkxSQSNNOFaq66BEexmWmhFOxyU3VTTBZIj7tGtohAVVXjb9YQyPjdKDYSxNRRpO1d8TGRZKjURgOgXWAzXvTcT/vG6qwysvY1GSahqR2aIw5VDHcBII7DFJieYjQzCRzNwKyQBLTLSJrWRCcOZfXiSt86qDqs7tRbl2ncdRBIfgCFSAAy5BDdyABmgCAh7BM3gFb9aT9WK9Wx+z1oKVz+yDP7A+fwBC05Xy</latexit><latexit sha1_base64="J4XknDenb36+9rwby7B1axazf6g=">AAACA3icbVDLSsNAFJ3UV62vqDvdDBahopSJCLosduOygn1AE8NkOmmHziRhZiKUUHDjr7hxoYhbf8Kdf+O0zUJbD1w4nHMv994TJJwpjdC3VVhaXlldK66XNja3tnfs3b2WilNJaJPEPJadACvKWUSbmmlOO4mkWASctoNhfeK3H6hULI7u9CihnsD9iIWMYG0k3z5wNU79bMjH0FVMwPr9aQWdwcBHJ75dRlU0BVwkTk7KIEfDt7/cXkxSQSNNOFaq66BEexmWmhFOxyU3VTTBZIj7tGtohAVVXjb9YQyPjdKDYSxNRRpO1d8TGRZKjURgOgXWAzXvTcT/vG6qwysvY1GSahqR2aIw5VDHcBII7DFJieYjQzCRzNwKyQBLTLSJrWRCcOZfXiSt86qDqs7tRbl2ncdRBIfgCFSAAy5BDdyABmgCAh7BM3gFb9aT9WK9Wx+z1oKVz+yDP7A+fwBC05Xy</latexit>

wkk�,l � N (0, �2klv

2l )

<latexit sha1_base64="7PemL37DRfIpcb5U45s4wmqadx8=">AAACGHicbVDLSsNAFJ3UV62vqEs3g0WsUGpSBF0W3biSCvYBTRom00k7ZPJgZlIpIZ/hxl9x40IRt935N07aLrR6YOBwzrnMvceNGRXSML60wsrq2vpGcbO0tb2zu6fvH7RFlHBMWjhiEe+6SBBGQ9KSVDLSjTlBgctIx/Vvcr8zJlzQKHyQk5jYARqG1KMYSSU5+vmjk/r+aZVl0BI0gFaA5Agjlt5lFaMKLYkSFWBZvz52WL9+5uhlo2bMAP8Sc0HKYIGmo0+tQYSTgIQSMyREzzRiaaeIS4oZyUpWIkiMsI+GpKdoiAIi7HR2WAZPlDKAXsTVCyWcqT8nUhQIMQlclcz3FsteLv7n9RLpXdkpDeNEkhDPP/ISBmUE85bggHKCJZsogjCnaleIR4gjLFWXJVWCuXzyX9Ku10yjZt5flBvXizqK4AgcgwowwSVogFvQBC2AwRN4AW/gXXvWXrUP7XMeLWiLmUPwC9r0G0p/n0c=</latexit><latexit sha1_base64="7PemL37DRfIpcb5U45s4wmqadx8=">AAACGHicbVDLSsNAFJ3UV62vqEs3g0WsUGpSBF0W3biSCvYBTRom00k7ZPJgZlIpIZ/hxl9x40IRt935N07aLrR6YOBwzrnMvceNGRXSML60wsrq2vpGcbO0tb2zu6fvH7RFlHBMWjhiEe+6SBBGQ9KSVDLSjTlBgctIx/Vvcr8zJlzQKHyQk5jYARqG1KMYSSU5+vmjk/r+aZVl0BI0gFaA5Agjlt5lFaMKLYkSFWBZvz52WL9+5uhlo2bMAP8Sc0HKYIGmo0+tQYSTgIQSMyREzzRiaaeIS4oZyUpWIkiMsI+GpKdoiAIi7HR2WAZPlDKAXsTVCyWcqT8nUhQIMQlclcz3FsteLv7n9RLpXdkpDeNEkhDPP/ISBmUE85bggHKCJZsogjCnaleIR4gjLFWXJVWCuXzyX9Ku10yjZt5flBvXizqK4AgcgwowwSVogFvQBC2AwRN4AW/gXXvWXrUP7XMeLWiLmUPwC9r0G0p/n0c=</latexit><latexit sha1_base64="7PemL37DRfIpcb5U45s4wmqadx8=">AAACGHicbVDLSsNAFJ3UV62vqEs3g0WsUGpSBF0W3biSCvYBTRom00k7ZPJgZlIpIZ/hxl9x40IRt935N07aLrR6YOBwzrnMvceNGRXSML60wsrq2vpGcbO0tb2zu6fvH7RFlHBMWjhiEe+6SBBGQ9KSVDLSjTlBgctIx/Vvcr8zJlzQKHyQk5jYARqG1KMYSSU5+vmjk/r+aZVl0BI0gFaA5Agjlt5lFaMKLYkSFWBZvz52WL9+5uhlo2bMAP8Sc0HKYIGmo0+tQYSTgIQSMyREzzRiaaeIS4oZyUpWIkiMsI+GpKdoiAIi7HR2WAZPlDKAXsTVCyWcqT8nUhQIMQlclcz3FsteLv7n9RLpXdkpDeNEkhDPP/ISBmUE85bggHKCJZsogjCnaleIR4gjLFWXJVWCuXzyX9Ku10yjZt5flBvXizqK4AgcgwowwSVogFvQBC2AwRN4AW/gXXvWXrUP7XMeLWiLmUPwC9r0G0p/n0c=</latexit><latexit sha1_base64="7PemL37DRfIpcb5U45s4wmqadx8=">AAACGHicbVDLSsNAFJ3UV62vqEs3g0WsUGpSBF0W3biSCvYBTRom00k7ZPJgZlIpIZ/hxl9x40IRt935N07aLrR6YOBwzrnMvceNGRXSML60wsrq2vpGcbO0tb2zu6fvH7RFlHBMWjhiEe+6SBBGQ9KSVDLSjTlBgctIx/Vvcr8zJlzQKHyQk5jYARqG1KMYSSU5+vmjk/r+aZVl0BI0gFaA5Agjlt5lFaMKLYkSFWBZvz52WL9+5uhlo2bMAP8Sc0HKYIGmo0+tQYSTgIQSMyREzzRiaaeIS4oZyUpWIkiMsI+GpKdoiAIi7HR2WAZPlDKAXsTVCyWcqT8nUhQIMQlclcz3FsteLv7n9RLpXdkpDeNEkhDPP/ISBmUE85bggHKCJZsogjCnaleIR4gjLFWXJVWCuXzyX9Ku10yjZt5flBvXizqK4AgcgwowwSVogFvQBC2AwRN4AW/gXXvWXrUP7XMeLWiLmUPwC9r0G0p/n0c=</latexit>

l<latexit sha1_base64="RXFmeCFAFLWm6A2HPHJ7FvtnIgg=">AAAB6HicbVBNS8NAEJ3Ur1q/qh69LBbBU0lEqMeiF48t2A9oQ9lsJ+3azSbsboQS+gu8eFDEqz/Jm//GbZuDtj4YeLw3w8y8IBFcG9f9dgobm1vbO8Xd0t7+weFR+fikreNUMWyxWMSqG1CNgktsGW4EdhOFNAoEdoLJ3dzvPKHSPJYPZpqgH9GR5CFn1FipKQblilt1FyDrxMtJBXI0BuWv/jBmaYTSMEG17nluYvyMKsOZwFmpn2pMKJvQEfYslTRC7WeLQ2fkwipDEsbKljRkof6eyGik9TQKbGdEzVivenPxP6+XmvDGz7hMUoOSLReFqSAmJvOvyZArZEZMLaFMcXsrYWOqKDM2m5INwVt9eZ20r6qeW/Wa15X6bR5HEc7gHC7BgxrU4R4a0AIGCM/wCm/Oo/PivDsfy9aCk8+cwh84nz/UCYzw</latexit><latexit sha1_base64="RXFmeCFAFLWm6A2HPHJ7FvtnIgg=">AAAB6HicbVBNS8NAEJ3Ur1q/qh69LBbBU0lEqMeiF48t2A9oQ9lsJ+3azSbsboQS+gu8eFDEqz/Jm//GbZuDtj4YeLw3w8y8IBFcG9f9dgobm1vbO8Xd0t7+weFR+fikreNUMWyxWMSqG1CNgktsGW4EdhOFNAoEdoLJ3dzvPKHSPJYPZpqgH9GR5CFn1FipKQblilt1FyDrxMtJBXI0BuWv/jBmaYTSMEG17nluYvyMKsOZwFmpn2pMKJvQEfYslTRC7WeLQ2fkwipDEsbKljRkof6eyGik9TQKbGdEzVivenPxP6+XmvDGz7hMUoOSLReFqSAmJvOvyZArZEZMLaFMcXsrYWOqKDM2m5INwVt9eZ20r6qeW/Wa15X6bR5HEc7gHC7BgxrU4R4a0AIGCM/wCm/Oo/PivDsfy9aCk8+cwh84nz/UCYzw</latexit><latexit sha1_base64="RXFmeCFAFLWm6A2HPHJ7FvtnIgg=">AAAB6HicbVBNS8NAEJ3Ur1q/qh69LBbBU0lEqMeiF48t2A9oQ9lsJ+3azSbsboQS+gu8eFDEqz/Jm//GbZuDtj4YeLw3w8y8IBFcG9f9dgobm1vbO8Xd0t7+weFR+fikreNUMWyxWMSqG1CNgktsGW4EdhOFNAoEdoLJ3dzvPKHSPJYPZpqgH9GR5CFn1FipKQblilt1FyDrxMtJBXI0BuWv/jBmaYTSMEG17nluYvyMKsOZwFmpn2pMKJvQEfYslTRC7WeLQ2fkwipDEsbKljRkof6eyGik9TQKbGdEzVivenPxP6+XmvDGz7hMUoOSLReFqSAmJvOvyZArZEZMLaFMcXsrYWOqKDM2m5INwVt9eZ20r6qeW/Wa15X6bR5HEc7gHC7BgxrU4R4a0AIGCM/wCm/Oo/PivDsfy9aCk8+cwh84nz/UCYzw</latexit><latexit sha1_base64="RXFmeCFAFLWm6A2HPHJ7FvtnIgg=">AAAB6HicbVBNS8NAEJ3Ur1q/qh69LBbBU0lEqMeiF48t2A9oQ9lsJ+3azSbsboQS+gu8eFDEqz/Jm//GbZuDtj4YeLw3w8y8IBFcG9f9dgobm1vbO8Xd0t7+weFR+fikreNUMWyxWMSqG1CNgktsGW4EdhOFNAoEdoLJ3dzvPKHSPJYPZpqgH9GR5CFn1FipKQblilt1FyDrxMtJBXI0BuWv/jBmaYTSMEG17nluYvyMKsOZwFmpn2pMKJvQEfYslTRC7WeLQ2fkwipDEsbKljRkof6eyGik9TQKbGdEzVivenPxP6+XmvDGz7hMUoOSLReFqSAmJvOvyZArZEZMLaFMcXsrYWOqKDM2m5INwVt9eZ20r6qeW/Wa15X6bR5HEc7gHC7BgxrU4R4a0AIGCM/wCm/Oo/PivDsfy9aCk8+cwh84nz/UCYzw</latexit>

!22

• Horseshoe BNN:

• Inference:

Stochastic gradient variational Bayes with naive fully factorized variational approximations.

Regularized Horseshoe BNN

structured

Page 23: Horseshoe Priors for Bayesian Neural Networks...Horseshoe Priors for Bayesian Neural Networks Soumya Ghosh Jiayu Yao Finale Doshi-Velez IBM Research MIT-IBM Watson AI lab Harvard Harvard

Regularized Horseshoe

Piironen & Vehtari, 2017!23

p(wkk�,l | �kl, vl, ) � N (wkk�,l | 0, �2klv

2l )

<latexit sha1_base64="6zz3dSd1xkTFf4ybAWwv2r75GUs=">AAACSHicbVBNSwMxFMzWr1q/qh69BIuoUMpuEfRY9OJJKtgqdOvyNpu2odndmGQrZdmf58WjN3+DFw+KeDP9ALU6EBhm3vDyxhecKW3bz1Zubn5hcSm/XFhZXVvfKG5uNVWcSEIbJOaxvPFBUc4i2tBMc3ojJIXQ5/Ta75+N/OsBlYrF0ZUeCtoOoRuxDiOgjeQVPXFw76X9/n6ZZ27IAuxqSIzAszIeeLyM3bsEAowPsStkLHSM3RB0jwBPL7KZqF3+Tt9WTfq2eugVS3bFHgP/Jc6UlNAUda/45AYxSUIaacJBqZZjC91OQWpGOM0KbqKoANKHLm0ZGkFIVTsdF5HhPaMEuBNL8yKNx+rPRAqhUsPQN5OjK9SsNxL/81qJ7py0UxaJRNOITBZ1Eo5NH6NWccAkJZoPDQEimfkrJj2QQLTpvmBKcGZP/kua1YpjV5zLo1LtdFpHHu2gXXSAHHSMaugc1VEDEfSAXtAbercerVfrw/qcjOasaWYb/UIu9wWXHrEH</latexit><latexit sha1_base64="6zz3dSd1xkTFf4ybAWwv2r75GUs=">AAACSHicbVBNSwMxFMzWr1q/qh69BIuoUMpuEfRY9OJJKtgqdOvyNpu2odndmGQrZdmf58WjN3+DFw+KeDP9ALU6EBhm3vDyxhecKW3bz1Zubn5hcSm/XFhZXVvfKG5uNVWcSEIbJOaxvPFBUc4i2tBMc3ojJIXQ5/Ta75+N/OsBlYrF0ZUeCtoOoRuxDiOgjeQVPXFw76X9/n6ZZ27IAuxqSIzAszIeeLyM3bsEAowPsStkLHSM3RB0jwBPL7KZqF3+Tt9WTfq2eugVS3bFHgP/Jc6UlNAUda/45AYxSUIaacJBqZZjC91OQWpGOM0KbqKoANKHLm0ZGkFIVTsdF5HhPaMEuBNL8yKNx+rPRAqhUsPQN5OjK9SsNxL/81qJ7py0UxaJRNOITBZ1Eo5NH6NWccAkJZoPDQEimfkrJj2QQLTpvmBKcGZP/kua1YpjV5zLo1LtdFpHHu2gXXSAHHSMaugc1VEDEfSAXtAbercerVfrw/qcjOasaWYb/UIu9wWXHrEH</latexit><latexit sha1_base64="6zz3dSd1xkTFf4ybAWwv2r75GUs=">AAACSHicbVBNSwMxFMzWr1q/qh69BIuoUMpuEfRY9OJJKtgqdOvyNpu2odndmGQrZdmf58WjN3+DFw+KeDP9ALU6EBhm3vDyxhecKW3bz1Zubn5hcSm/XFhZXVvfKG5uNVWcSEIbJOaxvPFBUc4i2tBMc3ojJIXQ5/Ta75+N/OsBlYrF0ZUeCtoOoRuxDiOgjeQVPXFw76X9/n6ZZ27IAuxqSIzAszIeeLyM3bsEAowPsStkLHSM3RB0jwBPL7KZqF3+Tt9WTfq2eugVS3bFHgP/Jc6UlNAUda/45AYxSUIaacJBqZZjC91OQWpGOM0KbqKoANKHLm0ZGkFIVTsdF5HhPaMEuBNL8yKNx+rPRAqhUsPQN5OjK9SsNxL/81qJ7py0UxaJRNOITBZ1Eo5NH6NWccAkJZoPDQEimfkrJj2QQLTpvmBKcGZP/kua1YpjV5zLo1LtdFpHHu2gXXSAHHSMaugc1VEDEfSAXtAbercerVfrw/qcjOasaWYb/UIu9wWXHrEH</latexit><latexit sha1_base64="6zz3dSd1xkTFf4ybAWwv2r75GUs=">AAACSHicbVBNSwMxFMzWr1q/qh69BIuoUMpuEfRY9OJJKtgqdOvyNpu2odndmGQrZdmf58WjN3+DFw+KeDP9ALU6EBhm3vDyxhecKW3bz1Zubn5hcSm/XFhZXVvfKG5uNVWcSEIbJOaxvPFBUc4i2tBMc3ojJIXQ5/Ta75+N/OsBlYrF0ZUeCtoOoRuxDiOgjeQVPXFw76X9/n6ZZ27IAuxqSIzAszIeeLyM3bsEAowPsStkLHSM3RB0jwBPL7KZqF3+Tt9WTfq2eugVS3bFHgP/Jc6UlNAUda/45AYxSUIaacJBqZZjC91OQWpGOM0KbqKoANKHLm0ZGkFIVTsdF5HhPaMEuBNL8yKNx+rPRAqhUsPQN5OjK9SsNxL/81qJ7py0UxaJRNOITBZ1Eo5NH6NWccAkJZoPDQEimfkrJj2QQLTpvmBKcGZP/kua1YpjV5zLo1LtdFpHHu2gXXSAHHSMaugc1VEDEfSAXtAbercerVfrw/qcjOasaWYb/UIu9wWXHrEH</latexit>

N (wkk�,l | 0, c2)<latexit sha1_base64="Tzgl7rQic17zl1xJXdR55qoQahU=">AAACHHicbVDLSgMxFM3Ud31VXboJFlFBykwVdCm6cSUVbCt0aslkbm2YzIPkjlqG+RA3/oobF4q4cSH4N6aPha8DgcM59yYnx0uk0Gjbn1ZhYnJqemZ2rji/sLi0XFpZbeg4VRzqPJaxuvSYBikiqKNACZeJAhZ6EppecDLwmzegtIijC+wn0A7ZdSS6gjM0Uqe05yLc4fCeTIGfZ9QNGfY4k9lZvn3byYJga1fmbih8au9SflXdyTulsl2xh6B/iTMmZTJGrVN6d/2YpyFEyCXTuuXYCbYzplBwCXnRTTUkjAfsGlqGRiwE3c6GoXK6aRSfdmNlToR0qH7fyFiodT/0zOQguf7tDcT/vFaK3cN2JqIkRYj46KFuKinGdNAU9YUCjrJvCONKmKyU95hiHE2fRVOC8/vLf0mjWnHsinO+Xz46HtcxS9bJBtkmDjkgR+SU1EidcHJPHskzebEerCfr1XobjRas8c4a+QHr4wuEnKGJ</latexit><latexit sha1_base64="Tzgl7rQic17zl1xJXdR55qoQahU=">AAACHHicbVDLSgMxFM3Ud31VXboJFlFBykwVdCm6cSUVbCt0aslkbm2YzIPkjlqG+RA3/oobF4q4cSH4N6aPha8DgcM59yYnx0uk0Gjbn1ZhYnJqemZ2rji/sLi0XFpZbeg4VRzqPJaxuvSYBikiqKNACZeJAhZ6EppecDLwmzegtIijC+wn0A7ZdSS6gjM0Uqe05yLc4fCeTIGfZ9QNGfY4k9lZvn3byYJga1fmbih8au9SflXdyTulsl2xh6B/iTMmZTJGrVN6d/2YpyFEyCXTuuXYCbYzplBwCXnRTTUkjAfsGlqGRiwE3c6GoXK6aRSfdmNlToR0qH7fyFiodT/0zOQguf7tDcT/vFaK3cN2JqIkRYj46KFuKinGdNAU9YUCjrJvCONKmKyU95hiHE2fRVOC8/vLf0mjWnHsinO+Xz46HtcxS9bJBtkmDjkgR+SU1EidcHJPHskzebEerCfr1XobjRas8c4a+QHr4wuEnKGJ</latexit><latexit sha1_base64="Tzgl7rQic17zl1xJXdR55qoQahU=">AAACHHicbVDLSgMxFM3Ud31VXboJFlFBykwVdCm6cSUVbCt0aslkbm2YzIPkjlqG+RA3/oobF4q4cSH4N6aPha8DgcM59yYnx0uk0Gjbn1ZhYnJqemZ2rji/sLi0XFpZbeg4VRzqPJaxuvSYBikiqKNACZeJAhZ6EppecDLwmzegtIijC+wn0A7ZdSS6gjM0Uqe05yLc4fCeTIGfZ9QNGfY4k9lZvn3byYJga1fmbih8au9SflXdyTulsl2xh6B/iTMmZTJGrVN6d/2YpyFEyCXTuuXYCbYzplBwCXnRTTUkjAfsGlqGRiwE3c6GoXK6aRSfdmNlToR0qH7fyFiodT/0zOQguf7tDcT/vFaK3cN2JqIkRYj46KFuKinGdNAU9YUCjrJvCONKmKyU95hiHE2fRVOC8/vLf0mjWnHsinO+Xz46HtcxS9bJBtkmDjkgR+SU1EidcHJPHskzebEerCfr1XobjRas8c4a+QHr4wuEnKGJ</latexit><latexit sha1_base64="Tzgl7rQic17zl1xJXdR55qoQahU=">AAACHHicbVDLSgMxFM3Ud31VXboJFlFBykwVdCm6cSUVbCt0aslkbm2YzIPkjlqG+RA3/oobF4q4cSH4N6aPha8DgcM59yYnx0uk0Gjbn1ZhYnJqemZ2rji/sLi0XFpZbeg4VRzqPJaxuvSYBikiqKNACZeJAhZ6EppecDLwmzegtIijC+wn0A7ZdSS6gjM0Uqe05yLc4fCeTIGfZ9QNGfY4k9lZvn3byYJga1fmbih8au9SflXdyTulsl2xh6B/iTMmZTJGrVN6d/2YpyFEyCXTuuXYCbYzplBwCXnRTTUkjAfsGlqGRiwE3c6GoXK6aRSfdmNlToR0qH7fyFiodT/0zOQguf7tDcT/vFaK3cN2JqIkRYj46KFuKinGdNAU9YUCjrJvCONKmKyU95hiHE2fRVOC8/vLf0mjWnHsinO+Xz46HtcxS9bJBtkmDjkgR+SU1EidcHJPHskzebEerCfr1XobjRas8c4a+QHr4wuEnKGJ</latexit>

c<latexit sha1_base64="ZsDce1CdGJ2tZYTQbJQWdzC0VLg=">AAAB+nicbVDLSgNBEJz1GeNro0cvg0HwFHZF0GPQi8cI5gFJCLOzvcmQ2QczvWpY8ylePCji1S/x5t842exBEwsaiqru6enyEik0Os63tbK6tr6xWdoqb+/s7u3blYOWjlPFocljGauOxzRIEUETBUroJApY6Eloe+Prmd++B6VFHN3hJIF+yIaRCARnaKSBXekhPGL+TqbAn1I+sKtOzclBl4lbkCop0BjYXz0/5mkIEXLJtO66ToL9jCkUXMK03Es1JIyP2RC6hkYsBN3P8pVTemIUnwaxMhUhzdXfExkLtZ6EnukMGY70ojcT//O6KQaX/UxESYoQ8fmiIJUUYzrLgfpCAUc5MYRxJcxfKR8xxTiatMomBHfx5GXSOqu5Ts29Pa/Wr4o4SuSIHJNT4pILUic3pEGahJMH8kxeyZv1ZL1Y79bHvHXFKmYOyR9Ynz++oJRL</latexit><latexit sha1_base64="ZsDce1CdGJ2tZYTQbJQWdzC0VLg=">AAAB+nicbVDLSgNBEJz1GeNro0cvg0HwFHZF0GPQi8cI5gFJCLOzvcmQ2QczvWpY8ylePCji1S/x5t842exBEwsaiqru6enyEik0Os63tbK6tr6xWdoqb+/s7u3blYOWjlPFocljGauOxzRIEUETBUroJApY6Eloe+Prmd++B6VFHN3hJIF+yIaRCARnaKSBXekhPGL+TqbAn1I+sKtOzclBl4lbkCop0BjYXz0/5mkIEXLJtO66ToL9jCkUXMK03Es1JIyP2RC6hkYsBN3P8pVTemIUnwaxMhUhzdXfExkLtZ6EnukMGY70ojcT//O6KQaX/UxESYoQ8fmiIJUUYzrLgfpCAUc5MYRxJcxfKR8xxTiatMomBHfx5GXSOqu5Ts29Pa/Wr4o4SuSIHJNT4pILUic3pEGahJMH8kxeyZv1ZL1Y79bHvHXFKmYOyR9Ynz++oJRL</latexit><latexit sha1_base64="ZsDce1CdGJ2tZYTQbJQWdzC0VLg=">AAAB+nicbVDLSgNBEJz1GeNro0cvg0HwFHZF0GPQi8cI5gFJCLOzvcmQ2QczvWpY8ylePCji1S/x5t842exBEwsaiqru6enyEik0Os63tbK6tr6xWdoqb+/s7u3blYOWjlPFocljGauOxzRIEUETBUroJApY6Eloe+Prmd++B6VFHN3hJIF+yIaRCARnaKSBXekhPGL+TqbAn1I+sKtOzclBl4lbkCop0BjYXz0/5mkIEXLJtO66ToL9jCkUXMK03Es1JIyP2RC6hkYsBN3P8pVTemIUnwaxMhUhzdXfExkLtZ6EnukMGY70ojcT//O6KQaX/UxESYoQ8fmiIJUUYzrLgfpCAUc5MYRxJcxfKR8xxTiatMomBHfx5GXSOqu5Ts29Pa/Wr4o4SuSIHJNT4pILUic3pEGahJMH8kxeyZv1ZL1Y79bHvHXFKmYOyR9Ynz++oJRL</latexit><latexit sha1_base64="ZsDce1CdGJ2tZYTQbJQWdzC0VLg=">AAAB+nicbVDLSgNBEJz1GeNro0cvg0HwFHZF0GPQi8cI5gFJCLOzvcmQ2QczvWpY8ylePCji1S/x5t842exBEwsaiqru6enyEik0Os63tbK6tr6xWdoqb+/s7u3blYOWjlPFocljGauOxzRIEUETBUroJApY6Eloe+Prmd++B6VFHN3hJIF+yIaRCARnaKSBXekhPGL+TqbAn1I+sKtOzclBl4lbkCop0BjYXz0/5mkIEXLJtO66ToL9jCkUXMK03Es1JIyP2RC6hkYsBN3P8pVTemIUnwaxMhUhzdXfExkLtZ6EnukMGY70ojcT//O6KQaX/UxESYoQ8fmiIJUUYzrLgfpCAUc5MYRxJcxfKR8xxTiatMomBHfx5GXSOqu5Ts29Pa/Wr4o4SuSIHJNT4pILUic3pEGahJMH8kxeyZv1ZL1Y79bHvHXFKmYOyR9Ynz++oJRL</latexit>

1

�̃2klv

2l

=1

c2+

1

�2klv

2l

<latexit sha1_base64="7CcoJKfibcFVNpLovVprMv81diw=">AAACPnicbVBNS8MwGE79nPOr6tFLcAiCMNoh6EUYevE4wW6DtStpmm5h6QdJOhilv8yLv8GbRy8eFPHq0XQrMjdfCHl4Pkjex0sYFdIwXrSV1bX1jc3KVnV7Z3dvXz84bIs45ZhYOGYx73pIEEYjYkkqGekmnKDQY6TjjW4LvTMmXNA4epCThDghGkQ0oBhJRbm6ZQcc4czMM1tS5hN1oTR3sxHL+1kjH7us38jhNfy14YLO4TmcC6J0IeDqNaNuTAcuA7MENVBOy9WfbT/GaUgiiRkSomcaiXQyxCXFjORVOxUkQXiEBqSnYIRCIpxsun4OTxXjwyDm6kQSTtn5RIZCISahp5whkkOxqBXkf1ovlcGVk9EoSSWJ8OyhIGVQxrDoEvqUEyzZRAGEOVV/hXiIVC1SNV5VJZiLKy+DdqNuGnXz/qLWvCnrqIBjcALOgAkuQRPcgRawAAaP4BW8gw/tSXvTPrWvmXVFKzNH4M9o3z8r3rDb</latexit><latexit sha1_base64="7CcoJKfibcFVNpLovVprMv81diw=">AAACPnicbVBNS8MwGE79nPOr6tFLcAiCMNoh6EUYevE4wW6DtStpmm5h6QdJOhilv8yLv8GbRy8eFPHq0XQrMjdfCHl4Pkjex0sYFdIwXrSV1bX1jc3KVnV7Z3dvXz84bIs45ZhYOGYx73pIEEYjYkkqGekmnKDQY6TjjW4LvTMmXNA4epCThDghGkQ0oBhJRbm6ZQcc4czMM1tS5hN1oTR3sxHL+1kjH7us38jhNfy14YLO4TmcC6J0IeDqNaNuTAcuA7MENVBOy9WfbT/GaUgiiRkSomcaiXQyxCXFjORVOxUkQXiEBqSnYIRCIpxsun4OTxXjwyDm6kQSTtn5RIZCISahp5whkkOxqBXkf1ovlcGVk9EoSSWJ8OyhIGVQxrDoEvqUEyzZRAGEOVV/hXiIVC1SNV5VJZiLKy+DdqNuGnXz/qLWvCnrqIBjcALOgAkuQRPcgRawAAaP4BW8gw/tSXvTPrWvmXVFKzNH4M9o3z8r3rDb</latexit><latexit sha1_base64="7CcoJKfibcFVNpLovVprMv81diw=">AAACPnicbVBNS8MwGE79nPOr6tFLcAiCMNoh6EUYevE4wW6DtStpmm5h6QdJOhilv8yLv8GbRy8eFPHq0XQrMjdfCHl4Pkjex0sYFdIwXrSV1bX1jc3KVnV7Z3dvXz84bIs45ZhYOGYx73pIEEYjYkkqGekmnKDQY6TjjW4LvTMmXNA4epCThDghGkQ0oBhJRbm6ZQcc4czMM1tS5hN1oTR3sxHL+1kjH7us38jhNfy14YLO4TmcC6J0IeDqNaNuTAcuA7MENVBOy9WfbT/GaUgiiRkSomcaiXQyxCXFjORVOxUkQXiEBqSnYIRCIpxsun4OTxXjwyDm6kQSTtn5RIZCISahp5whkkOxqBXkf1ovlcGVk9EoSSWJ8OyhIGVQxrDoEvqUEyzZRAGEOVV/hXiIVC1SNV5VJZiLKy+DdqNuGnXz/qLWvCnrqIBjcALOgAkuQRPcgRawAAaP4BW8gw/tSXvTPrWvmXVFKzNH4M9o3z8r3rDb</latexit><latexit sha1_base64="7CcoJKfibcFVNpLovVprMv81diw=">AAACPnicbVBNS8MwGE79nPOr6tFLcAiCMNoh6EUYevE4wW6DtStpmm5h6QdJOhilv8yLv8GbRy8eFPHq0XQrMjdfCHl4Pkjex0sYFdIwXrSV1bX1jc3KVnV7Z3dvXz84bIs45ZhYOGYx73pIEEYjYkkqGekmnKDQY6TjjW4LvTMmXNA4epCThDghGkQ0oBhJRbm6ZQcc4czMM1tS5hN1oTR3sxHL+1kjH7us38jhNfy14YLO4TmcC6J0IeDqNaNuTAcuA7MENVBOy9WfbT/GaUgiiRkSomcaiXQyxCXFjORVOxUkQXiEBqSnYIRCIpxsun4OTxXjwyDm6kQSTtn5RIZCISahp5whkkOxqBXkf1ovlcGVk9EoSSWJ8OyhIGVQxrDoEvqUEyzZRAGEOVV/hXiIVC1SNV5VJZiLKy+DdqNuGnXz/qLWvCnrqIBjcALOgAkuQRPcgRawAAaP4BW8gw/tSXvTPrWvmXVFKzNH4M9o3z8r3rDb</latexit>

wkk�,l | c, �kl, vl � N (wkl | 0, �̃2klv

2l );

<latexit sha1_base64="5RemFlrEDuK1epl3X73huVsWhtI=">AAACQ3icbVDLSsNAFJ3UV62vqks3g0WsUEpSBAU3RTeupIJ9YNOGyWTaDpk8mJm0lJB/c+MPuPMH3LhQxK3gJO1CqxcGDufBnXvskFEhdf1Zyy0tr6yu5dcLG5tb2zvF3b2WCCKOSRMHLOAdGwnCqE+akkpGOiEnyLMZadvuVaq3x4QLGvh3chqSnoeGPh1QjKSirOL9xIpd97jCEmh61IG4Ak2JIkWypALHFoOmoJ7SkBxhxOKbpDzJxMytp27KHBKnoSQT+jWV6tdOLqxiSa/q2cC/wJiDEphPwyo+mU6AI4/4EjMkRNfQQ9mLEZcUM5IUzEiQEGEXDUlXQR95RPTirIMEHinGgYOAq+dLmLE/EzHyhJh6tnKmt4hFLSX/07qRHJz3YuqHkSQ+ni0aRAzKAKaFQodygiWbKoAwp+qvEI8QR1iq2guqBGPx5L+gVasaetW4PS3VL+d15MEBOARlYIAzUAfXoAGaAIMH8ALewLv2qL1qH9rnzJrT5pl98Gu0r2/HArDX</latexit><latexit sha1_base64="5RemFlrEDuK1epl3X73huVsWhtI=">AAACQ3icbVDLSsNAFJ3UV62vqks3g0WsUEpSBAU3RTeupIJ9YNOGyWTaDpk8mJm0lJB/c+MPuPMH3LhQxK3gJO1CqxcGDufBnXvskFEhdf1Zyy0tr6yu5dcLG5tb2zvF3b2WCCKOSRMHLOAdGwnCqE+akkpGOiEnyLMZadvuVaq3x4QLGvh3chqSnoeGPh1QjKSirOL9xIpd97jCEmh61IG4Ak2JIkWypALHFoOmoJ7SkBxhxOKbpDzJxMytp27KHBKnoSQT+jWV6tdOLqxiSa/q2cC/wJiDEphPwyo+mU6AI4/4EjMkRNfQQ9mLEZcUM5IUzEiQEGEXDUlXQR95RPTirIMEHinGgYOAq+dLmLE/EzHyhJh6tnKmt4hFLSX/07qRHJz3YuqHkSQ+ni0aRAzKAKaFQodygiWbKoAwp+qvEI8QR1iq2guqBGPx5L+gVasaetW4PS3VL+d15MEBOARlYIAzUAfXoAGaAIMH8ALewLv2qL1qH9rnzJrT5pl98Gu0r2/HArDX</latexit><latexit sha1_base64="5RemFlrEDuK1epl3X73huVsWhtI=">AAACQ3icbVDLSsNAFJ3UV62vqks3g0WsUEpSBAU3RTeupIJ9YNOGyWTaDpk8mJm0lJB/c+MPuPMH3LhQxK3gJO1CqxcGDufBnXvskFEhdf1Zyy0tr6yu5dcLG5tb2zvF3b2WCCKOSRMHLOAdGwnCqE+akkpGOiEnyLMZadvuVaq3x4QLGvh3chqSnoeGPh1QjKSirOL9xIpd97jCEmh61IG4Ak2JIkWypALHFoOmoJ7SkBxhxOKbpDzJxMytp27KHBKnoSQT+jWV6tdOLqxiSa/q2cC/wJiDEphPwyo+mU6AI4/4EjMkRNfQQ9mLEZcUM5IUzEiQEGEXDUlXQR95RPTirIMEHinGgYOAq+dLmLE/EzHyhJh6tnKmt4hFLSX/07qRHJz3YuqHkSQ+ni0aRAzKAKaFQodygiWbKoAwp+qvEI8QR1iq2guqBGPx5L+gVasaetW4PS3VL+d15MEBOARlYIAzUAfXoAGaAIMH8ALewLv2qL1qH9rnzJrT5pl98Gu0r2/HArDX</latexit><latexit sha1_base64="5RemFlrEDuK1epl3X73huVsWhtI=">AAACQ3icbVDLSsNAFJ3UV62vqks3g0WsUEpSBAU3RTeupIJ9YNOGyWTaDpk8mJm0lJB/c+MPuPMH3LhQxK3gJO1CqxcGDufBnXvskFEhdf1Zyy0tr6yu5dcLG5tb2zvF3b2WCCKOSRMHLOAdGwnCqE+akkpGOiEnyLMZadvuVaq3x4QLGvh3chqSnoeGPh1QjKSirOL9xIpd97jCEmh61IG4Ak2JIkWypALHFoOmoJ7SkBxhxOKbpDzJxMytp27KHBKnoSQT+jWV6tdOLqxiSa/q2cC/wJiDEphPwyo+mU6AI4/4EjMkRNfQQ9mLEZcUM5IUzEiQEGEXDUlXQR95RPTirIMEHinGgYOAq+dLmLE/EzHyhJh6tnKmt4hFLSX/07qRHJz3YuqHkSQ+ni0aRAzKAKaFQodygiWbKoAwp+qvEI8QR1iq2guqBGPx5L+gVasaetW4PS3VL+d15MEBOARlYIAzUAfXoAGaAIMH8ALewLv2qL1qH9rnzJrT5pl98Gu0r2/HArDX</latexit>

Equivalently,

Page 24: Horseshoe Priors for Bayesian Neural Networks...Horseshoe Priors for Bayesian Neural Networks Soumya Ghosh Jiayu Yao Finale Doshi-Velez IBM Research MIT-IBM Watson AI lab Harvard Harvard

Regularized Horseshoe BNNs

!24

1

�̃2klv

2l

=1

c2+

1

�2klv

2l

<latexit sha1_base64="7CcoJKfibcFVNpLovVprMv81diw=">AAACPnicbVBNS8MwGE79nPOr6tFLcAiCMNoh6EUYevE4wW6DtStpmm5h6QdJOhilv8yLv8GbRy8eFPHq0XQrMjdfCHl4Pkjex0sYFdIwXrSV1bX1jc3KVnV7Z3dvXz84bIs45ZhYOGYx73pIEEYjYkkqGekmnKDQY6TjjW4LvTMmXNA4epCThDghGkQ0oBhJRbm6ZQcc4czMM1tS5hN1oTR3sxHL+1kjH7us38jhNfy14YLO4TmcC6J0IeDqNaNuTAcuA7MENVBOy9WfbT/GaUgiiRkSomcaiXQyxCXFjORVOxUkQXiEBqSnYIRCIpxsun4OTxXjwyDm6kQSTtn5RIZCISahp5whkkOxqBXkf1ovlcGVk9EoSSWJ8OyhIGVQxrDoEvqUEyzZRAGEOVV/hXiIVC1SNV5VJZiLKy+DdqNuGnXz/qLWvCnrqIBjcALOgAkuQRPcgRawAAaP4BW8gw/tSXvTPrWvmXVFKzNH4M9o3z8r3rDb</latexit><latexit sha1_base64="7CcoJKfibcFVNpLovVprMv81diw=">AAACPnicbVBNS8MwGE79nPOr6tFLcAiCMNoh6EUYevE4wW6DtStpmm5h6QdJOhilv8yLv8GbRy8eFPHq0XQrMjdfCHl4Pkjex0sYFdIwXrSV1bX1jc3KVnV7Z3dvXz84bIs45ZhYOGYx73pIEEYjYkkqGekmnKDQY6TjjW4LvTMmXNA4epCThDghGkQ0oBhJRbm6ZQcc4czMM1tS5hN1oTR3sxHL+1kjH7us38jhNfy14YLO4TmcC6J0IeDqNaNuTAcuA7MENVBOy9WfbT/GaUgiiRkSomcaiXQyxCXFjORVOxUkQXiEBqSnYIRCIpxsun4OTxXjwyDm6kQSTtn5RIZCISahp5whkkOxqBXkf1ovlcGVk9EoSSWJ8OyhIGVQxrDoEvqUEyzZRAGEOVV/hXiIVC1SNV5VJZiLKy+DdqNuGnXz/qLWvCnrqIBjcALOgAkuQRPcgRawAAaP4BW8gw/tSXvTPrWvmXVFKzNH4M9o3z8r3rDb</latexit><latexit sha1_base64="7CcoJKfibcFVNpLovVprMv81diw=">AAACPnicbVBNS8MwGE79nPOr6tFLcAiCMNoh6EUYevE4wW6DtStpmm5h6QdJOhilv8yLv8GbRy8eFPHq0XQrMjdfCHl4Pkjex0sYFdIwXrSV1bX1jc3KVnV7Z3dvXz84bIs45ZhYOGYx73pIEEYjYkkqGekmnKDQY6TjjW4LvTMmXNA4epCThDghGkQ0oBhJRbm6ZQcc4czMM1tS5hN1oTR3sxHL+1kjH7us38jhNfy14YLO4TmcC6J0IeDqNaNuTAcuA7MENVBOy9WfbT/GaUgiiRkSomcaiXQyxCXFjORVOxUkQXiEBqSnYIRCIpxsun4OTxXjwyDm6kQSTtn5RIZCISahp5whkkOxqBXkf1ovlcGVk9EoSSWJ8OyhIGVQxrDoEvqUEyzZRAGEOVV/hXiIVC1SNV5VJZiLKy+DdqNuGnXz/qLWvCnrqIBjcALOgAkuQRPcgRawAAaP4BW8gw/tSXvTPrWvmXVFKzNH4M9o3z8r3rDb</latexit><latexit sha1_base64="7CcoJKfibcFVNpLovVprMv81diw=">AAACPnicbVBNS8MwGE79nPOr6tFLcAiCMNoh6EUYevE4wW6DtStpmm5h6QdJOhilv8yLv8GbRy8eFPHq0XQrMjdfCHl4Pkjex0sYFdIwXrSV1bX1jc3KVnV7Z3dvXz84bIs45ZhYOGYx73pIEEYjYkkqGekmnKDQY6TjjW4LvTMmXNA4epCThDghGkQ0oBhJRbm6ZQcc4czMM1tS5hN1oTR3sxHL+1kjH7us38jhNfy14YLO4TmcC6J0IeDqNaNuTAcuA7MENVBOy9WfbT/GaUgiiRkSomcaiXQyxCXFjORVOxUkQXiEBqSnYIRCIpxsun4OTxXjwyDm6kQSTtn5RIZCISahp5whkkOxqBXkf1ovlcGVk9EoSSWJ8OyhIGVQxrDoEvqUEyzZRAGEOVV/hXiIVC1SNV5VJZiLKy+DdqNuGnXz/qLWvCnrqIBjcALOgAkuQRPcgRawAAaP4BW8gw/tSXvTPrWvmXVFKzNH4M9o3z8r3rDb</latexit>

50 u

nits

500

units

Random functions from single hidden layer (tanh) network with HS and reg-HS priors

Regularized HorseshoeHorseshoe

Page 25: Horseshoe Priors for Bayesian Neural Networks...Horseshoe Priors for Bayesian Neural Networks Soumya Ghosh Jiayu Yao Finale Doshi-Velez IBM Research MIT-IBM Watson AI lab Harvard Harvard

Regularized Horseshoe BNNs

!25

1

�̃2klv

2l

=1

c2+

1

�2klv

2l

<latexit sha1_base64="7CcoJKfibcFVNpLovVprMv81diw=">AAACPnicbVBNS8MwGE79nPOr6tFLcAiCMNoh6EUYevE4wW6DtStpmm5h6QdJOhilv8yLv8GbRy8eFPHq0XQrMjdfCHl4Pkjex0sYFdIwXrSV1bX1jc3KVnV7Z3dvXz84bIs45ZhYOGYx73pIEEYjYkkqGekmnKDQY6TjjW4LvTMmXNA4epCThDghGkQ0oBhJRbm6ZQcc4czMM1tS5hN1oTR3sxHL+1kjH7us38jhNfy14YLO4TmcC6J0IeDqNaNuTAcuA7MENVBOy9WfbT/GaUgiiRkSomcaiXQyxCXFjORVOxUkQXiEBqSnYIRCIpxsun4OTxXjwyDm6kQSTtn5RIZCISahp5whkkOxqBXkf1ovlcGVk9EoSSWJ8OyhIGVQxrDoEvqUEyzZRAGEOVV/hXiIVC1SNV5VJZiLKy+DdqNuGnXz/qLWvCnrqIBjcALOgAkuQRPcgRawAAaP4BW8gw/tSXvTPrWvmXVFKzNH4M9o3z8r3rDb</latexit><latexit sha1_base64="7CcoJKfibcFVNpLovVprMv81diw=">AAACPnicbVBNS8MwGE79nPOr6tFLcAiCMNoh6EUYevE4wW6DtStpmm5h6QdJOhilv8yLv8GbRy8eFPHq0XQrMjdfCHl4Pkjex0sYFdIwXrSV1bX1jc3KVnV7Z3dvXz84bIs45ZhYOGYx73pIEEYjYkkqGekmnKDQY6TjjW4LvTMmXNA4epCThDghGkQ0oBhJRbm6ZQcc4czMM1tS5hN1oTR3sxHL+1kjH7us38jhNfy14YLO4TmcC6J0IeDqNaNuTAcuA7MENVBOy9WfbT/GaUgiiRkSomcaiXQyxCXFjORVOxUkQXiEBqSnYIRCIpxsun4OTxXjwyDm6kQSTtn5RIZCISahp5whkkOxqBXkf1ovlcGVk9EoSSWJ8OyhIGVQxrDoEvqUEyzZRAGEOVV/hXiIVC1SNV5VJZiLKy+DdqNuGnXz/qLWvCnrqIBjcALOgAkuQRPcgRawAAaP4BW8gw/tSXvTPrWvmXVFKzNH4M9o3z8r3rDb</latexit><latexit sha1_base64="7CcoJKfibcFVNpLovVprMv81diw=">AAACPnicbVBNS8MwGE79nPOr6tFLcAiCMNoh6EUYevE4wW6DtStpmm5h6QdJOhilv8yLv8GbRy8eFPHq0XQrMjdfCHl4Pkjex0sYFdIwXrSV1bX1jc3KVnV7Z3dvXz84bIs45ZhYOGYx73pIEEYjYkkqGekmnKDQY6TjjW4LvTMmXNA4epCThDghGkQ0oBhJRbm6ZQcc4czMM1tS5hN1oTR3sxHL+1kjH7us38jhNfy14YLO4TmcC6J0IeDqNaNuTAcuA7MENVBOy9WfbT/GaUgiiRkSomcaiXQyxCXFjORVOxUkQXiEBqSnYIRCIpxsun4OTxXjwyDm6kQSTtn5RIZCISahp5whkkOxqBXkf1ovlcGVk9EoSSWJ8OyhIGVQxrDoEvqUEyzZRAGEOVV/hXiIVC1SNV5VJZiLKy+DdqNuGnXz/qLWvCnrqIBjcALOgAkuQRPcgRawAAaP4BW8gw/tSXvTPrWvmXVFKzNH4M9o3z8r3rDb</latexit><latexit sha1_base64="7CcoJKfibcFVNpLovVprMv81diw=">AAACPnicbVBNS8MwGE79nPOr6tFLcAiCMNoh6EUYevE4wW6DtStpmm5h6QdJOhilv8yLv8GbRy8eFPHq0XQrMjdfCHl4Pkjex0sYFdIwXrSV1bX1jc3KVnV7Z3dvXz84bIs45ZhYOGYx73pIEEYjYkkqGekmnKDQY6TjjW4LvTMmXNA4epCThDghGkQ0oBhJRbm6ZQcc4czMM1tS5hN1oTR3sxHL+1kjH7us38jhNfy14YLO4TmcC6J0IeDqNaNuTAcuA7MENVBOy9WfbT/GaUgiiRkSomcaiXQyxCXFjORVOxUkQXiEBqSnYIRCIpxsun4OTxXjwyDm6kQSTtn5RIZCISahp5whkkOxqBXkf1ovlcGVk9EoSSWJ8OyhIGVQxrDoEvqUEyzZRAGEOVV/hXiIVC1SNV5VJZiLKy+DdqNuGnXz/qLWvCnrqIBjcALOgAkuQRPcgRawAAaP4BW8gw/tSXvTPrWvmXVFKzNH4M9o3z8r3rDb</latexit>

reg-HS BNNs improves predictive performance over HS BNNs for smaller datasets. Relative improvement: (x - y)/ max(|x|, |y|)

UCI Regression Benchmarks (Hernández-Lobato and Adams’ 2015)

log(n)<latexit sha1_base64="oaTh+XCwNh8FkOZrvLdJUPofWWY=">AAAB9HicbVDLSgNBEJyNrxhfUY9eBoMQL2FXBD0GvXiMYB6QLGF20kmGzM6uM73BsOQ7vHhQxKsf482/cZLsQRMLGoqqbrq7glgKg6777eTW1jc2t/LbhZ3dvf2D4uFRw0SJ5lDnkYx0K2AGpFBQR4ESWrEGFgYSmsHoduY3x6CNiNQDTmLwQzZQoi84Qyv5HYQnTGU0mJbVebdYcivuHHSVeBkpkQy1bvGr04t4EoJCLpkxbc+N0U+ZRsElTAudxEDM+IgNoG2pYiEYP50fPaVnVunRfqRtKaRz9fdEykJjJmFgO0OGQ7PszcT/vHaC/Ws/FSpOEBRfLOonkmJEZwnQntDAUU4sYVwLeyvlQ6YZR5tTwYbgLb+8ShoXFc+tePeXpepNFkeenJBTUiYeuSJVckdqpE44eSTP5JW8OWPnxXl3PhatOSebOSZ/4Hz+AMhzkhY=</latexit><latexit sha1_base64="oaTh+XCwNh8FkOZrvLdJUPofWWY=">AAAB9HicbVDLSgNBEJyNrxhfUY9eBoMQL2FXBD0GvXiMYB6QLGF20kmGzM6uM73BsOQ7vHhQxKsf482/cZLsQRMLGoqqbrq7glgKg6777eTW1jc2t/LbhZ3dvf2D4uFRw0SJ5lDnkYx0K2AGpFBQR4ESWrEGFgYSmsHoduY3x6CNiNQDTmLwQzZQoi84Qyv5HYQnTGU0mJbVebdYcivuHHSVeBkpkQy1bvGr04t4EoJCLpkxbc+N0U+ZRsElTAudxEDM+IgNoG2pYiEYP50fPaVnVunRfqRtKaRz9fdEykJjJmFgO0OGQ7PszcT/vHaC/Ws/FSpOEBRfLOonkmJEZwnQntDAUU4sYVwLeyvlQ6YZR5tTwYbgLb+8ShoXFc+tePeXpepNFkeenJBTUiYeuSJVckdqpE44eSTP5JW8OWPnxXl3PhatOSebOSZ/4Hz+AMhzkhY=</latexit><latexit sha1_base64="oaTh+XCwNh8FkOZrvLdJUPofWWY=">AAAB9HicbVDLSgNBEJyNrxhfUY9eBoMQL2FXBD0GvXiMYB6QLGF20kmGzM6uM73BsOQ7vHhQxKsf482/cZLsQRMLGoqqbrq7glgKg6777eTW1jc2t/LbhZ3dvf2D4uFRw0SJ5lDnkYx0K2AGpFBQR4ESWrEGFgYSmsHoduY3x6CNiNQDTmLwQzZQoi84Qyv5HYQnTGU0mJbVebdYcivuHHSVeBkpkQy1bvGr04t4EoJCLpkxbc+N0U+ZRsElTAudxEDM+IgNoG2pYiEYP50fPaVnVunRfqRtKaRz9fdEykJjJmFgO0OGQ7PszcT/vHaC/Ws/FSpOEBRfLOonkmJEZwnQntDAUU4sYVwLeyvlQ6YZR5tTwYbgLb+8ShoXFc+tePeXpepNFkeenJBTUiYeuSJVckdqpE44eSTP5JW8OWPnxXl3PhatOSebOSZ/4Hz+AMhzkhY=</latexit><latexit sha1_base64="oaTh+XCwNh8FkOZrvLdJUPofWWY=">AAAB9HicbVDLSgNBEJyNrxhfUY9eBoMQL2FXBD0GvXiMYB6QLGF20kmGzM6uM73BsOQ7vHhQxKsf482/cZLsQRMLGoqqbrq7glgKg6777eTW1jc2t/LbhZ3dvf2D4uFRw0SJ5lDnkYx0K2AGpFBQR4ESWrEGFgYSmsHoduY3x6CNiNQDTmLwQzZQoi84Qyv5HYQnTGU0mJbVebdYcivuHHSVeBkpkQy1bvGr04t4EoJCLpkxbc+N0U+ZRsElTAudxEDM+IgNoG2pYiEYP50fPaVnVunRfqRtKaRz9fdEykJjJmFgO0OGQ7PszcT/vHaC/Ws/FSpOEBRfLOonkmJEZwnQntDAUU4sYVwLeyvlQ6YZR5tTwYbgLb+8ShoXFc+tePeXpepNFkeenJBTUiYeuSJVckdqpE44eSTP5JW8OWPnxXl3PhatOSebOSZ/4Hz+AMhzkhY=</latexit>

log(n)<latexit sha1_base64="oaTh+XCwNh8FkOZrvLdJUPofWWY=">AAAB9HicbVDLSgNBEJyNrxhfUY9eBoMQL2FXBD0GvXiMYB6QLGF20kmGzM6uM73BsOQ7vHhQxKsf482/cZLsQRMLGoqqbrq7glgKg6777eTW1jc2t/LbhZ3dvf2D4uFRw0SJ5lDnkYx0K2AGpFBQR4ESWrEGFgYSmsHoduY3x6CNiNQDTmLwQzZQoi84Qyv5HYQnTGU0mJbVebdYcivuHHSVeBkpkQy1bvGr04t4EoJCLpkxbc+N0U+ZRsElTAudxEDM+IgNoG2pYiEYP50fPaVnVunRfqRtKaRz9fdEykJjJmFgO0OGQ7PszcT/vHaC/Ws/FSpOEBRfLOonkmJEZwnQntDAUU4sYVwLeyvlQ6YZR5tTwYbgLb+8ShoXFc+tePeXpepNFkeenJBTUiYeuSJVckdqpE44eSTP5JW8OWPnxXl3PhatOSebOSZ/4Hz+AMhzkhY=</latexit><latexit sha1_base64="oaTh+XCwNh8FkOZrvLdJUPofWWY=">AAAB9HicbVDLSgNBEJyNrxhfUY9eBoMQL2FXBD0GvXiMYB6QLGF20kmGzM6uM73BsOQ7vHhQxKsf482/cZLsQRMLGoqqbrq7glgKg6777eTW1jc2t/LbhZ3dvf2D4uFRw0SJ5lDnkYx0K2AGpFBQR4ESWrEGFgYSmsHoduY3x6CNiNQDTmLwQzZQoi84Qyv5HYQnTGU0mJbVebdYcivuHHSVeBkpkQy1bvGr04t4EoJCLpkxbc+N0U+ZRsElTAudxEDM+IgNoG2pYiEYP50fPaVnVunRfqRtKaRz9fdEykJjJmFgO0OGQ7PszcT/vHaC/Ws/FSpOEBRfLOonkmJEZwnQntDAUU4sYVwLeyvlQ6YZR5tTwYbgLb+8ShoXFc+tePeXpepNFkeenJBTUiYeuSJVckdqpE44eSTP5JW8OWPnxXl3PhatOSebOSZ/4Hz+AMhzkhY=</latexit><latexit sha1_base64="oaTh+XCwNh8FkOZrvLdJUPofWWY=">AAAB9HicbVDLSgNBEJyNrxhfUY9eBoMQL2FXBD0GvXiMYB6QLGF20kmGzM6uM73BsOQ7vHhQxKsf482/cZLsQRMLGoqqbrq7glgKg6777eTW1jc2t/LbhZ3dvf2D4uFRw0SJ5lDnkYx0K2AGpFBQR4ESWrEGFgYSmsHoduY3x6CNiNQDTmLwQzZQoi84Qyv5HYQnTGU0mJbVebdYcivuHHSVeBkpkQy1bvGr04t4EoJCLpkxbc+N0U+ZRsElTAudxEDM+IgNoG2pYiEYP50fPaVnVunRfqRtKaRz9fdEykJjJmFgO0OGQ7PszcT/vHaC/Ws/FSpOEBRfLOonkmJEZwnQntDAUU4sYVwLeyvlQ6YZR5tTwYbgLb+8ShoXFc+tePeXpepNFkeenJBTUiYeuSJVckdqpE44eSTP5JW8OWPnxXl3PhatOSebOSZ/4Hz+AMhzkhY=</latexit><latexit sha1_base64="oaTh+XCwNh8FkOZrvLdJUPofWWY=">AAAB9HicbVDLSgNBEJyNrxhfUY9eBoMQL2FXBD0GvXiMYB6QLGF20kmGzM6uM73BsOQ7vHhQxKsf482/cZLsQRMLGoqqbrq7glgKg6777eTW1jc2t/LbhZ3dvf2D4uFRw0SJ5lDnkYx0K2AGpFBQR4ESWrEGFgYSmsHoduY3x6CNiNQDTmLwQzZQoi84Qyv5HYQnTGU0mJbVebdYcivuHHSVeBkpkQy1bvGr04t4EoJCLpkxbc+N0U+ZRsElTAudxEDM+IgNoG2pYiEYP50fPaVnVunRfqRtKaRz9fdEykJjJmFgO0OGQ7PszcT/vHaC/Ws/FSpOEBRfLOonkmJEZwnQntDAUU4sYVwLeyvlQ6YZR5tTwYbgLb+8ShoXFc+tePeXpepNFkeenJBTUiYeuSJVckdqpE44eSTP5JW8OWPnxXl3PhatOSebOSZ/4Hz+AMhzkhY=</latexit>

Page 26: Horseshoe Priors for Bayesian Neural Networks...Horseshoe Priors for Bayesian Neural Networks Soumya Ghosh Jiayu Yao Finale Doshi-Velez IBM Research MIT-IBM Watson AI lab Harvard Harvard

Structured Variational Approximation

�kl � N (0, I)<latexit sha1_base64="lDU0sxVkxoqnqVxIWF2q+bbRx/Q=">AAACHHicbZDLSsNAFIYn9VbrLerSzWARKkhJVNBl0Y1upIK9QBPKZDpph84kYWYilJAHceOruHGhiBsXgm/jJI2grT8MfPznHOac34sYlcqyvozSwuLS8kp5tbK2vrG5ZW7vtGUYC0xaOGSh6HpIEkYD0lJUMdKNBEHcY6TjjS+zeueeCEnD4E5NIuJyNAyoTzFS2uqbJ45HFOonY5ZCR1IOHY7UCCOW3KS1nD0/sdIj+MPX6WHfrFp1KxecB7uAKijU7JsfziDEMSeBwgxJ2bOtSLkJEopiRtKKE0sSITxGQ9LTGCBOpJvkx6XwQDsD6IdCv0DB3P09kSAu5YR7ujNbUc7WMvO/Wi9W/rmb0CCKFQnw9CM/ZlCFMEsKDqggWLGJBoQF1btCPEICYaXzrOgQ7NmT56F9XLetun17Wm1cFHGUwR7YBzVggzPQAFegCVoAgwfwBF7Aq/FoPBtvxvu0tWQUM7vgj4zPb936odI=</latexit><latexit sha1_base64="lDU0sxVkxoqnqVxIWF2q+bbRx/Q=">AAACHHicbZDLSsNAFIYn9VbrLerSzWARKkhJVNBl0Y1upIK9QBPKZDpph84kYWYilJAHceOruHGhiBsXgm/jJI2grT8MfPznHOac34sYlcqyvozSwuLS8kp5tbK2vrG5ZW7vtGUYC0xaOGSh6HpIEkYD0lJUMdKNBEHcY6TjjS+zeueeCEnD4E5NIuJyNAyoTzFS2uqbJ45HFOonY5ZCR1IOHY7UCCOW3KS1nD0/sdIj+MPX6WHfrFp1KxecB7uAKijU7JsfziDEMSeBwgxJ2bOtSLkJEopiRtKKE0sSITxGQ9LTGCBOpJvkx6XwQDsD6IdCv0DB3P09kSAu5YR7ujNbUc7WMvO/Wi9W/rmb0CCKFQnw9CM/ZlCFMEsKDqggWLGJBoQF1btCPEICYaXzrOgQ7NmT56F9XLetun17Wm1cFHGUwR7YBzVggzPQAFegCVoAgwfwBF7Aq/FoPBtvxvu0tWQUM7vgj4zPb936odI=</latexit><latexit sha1_base64="lDU0sxVkxoqnqVxIWF2q+bbRx/Q=">AAACHHicbZDLSsNAFIYn9VbrLerSzWARKkhJVNBl0Y1upIK9QBPKZDpph84kYWYilJAHceOruHGhiBsXgm/jJI2grT8MfPznHOac34sYlcqyvozSwuLS8kp5tbK2vrG5ZW7vtGUYC0xaOGSh6HpIEkYD0lJUMdKNBEHcY6TjjS+zeueeCEnD4E5NIuJyNAyoTzFS2uqbJ45HFOonY5ZCR1IOHY7UCCOW3KS1nD0/sdIj+MPX6WHfrFp1KxecB7uAKijU7JsfziDEMSeBwgxJ2bOtSLkJEopiRtKKE0sSITxGQ9LTGCBOpJvkx6XwQDsD6IdCv0DB3P09kSAu5YR7ujNbUc7WMvO/Wi9W/rmb0CCKFQnw9CM/ZlCFMEsKDqggWLGJBoQF1btCPEICYaXzrOgQ7NmT56F9XLetun17Wm1cFHGUwR7YBzVggzPQAFegCVoAgwfwBF7Aq/FoPBtvxvu0tWQUM7vgj4zPb936odI=</latexit><latexit sha1_base64="lDU0sxVkxoqnqVxIWF2q+bbRx/Q=">AAACHHicbZDLSsNAFIYn9VbrLerSzWARKkhJVNBl0Y1upIK9QBPKZDpph84kYWYilJAHceOruHGhiBsXgm/jJI2grT8MfPznHOac34sYlcqyvozSwuLS8kp5tbK2vrG5ZW7vtGUYC0xaOGSh6HpIEkYD0lJUMdKNBEHcY6TjjS+zeueeCEnD4E5NIuJyNAyoTzFS2uqbJ45HFOonY5ZCR1IOHY7UCCOW3KS1nD0/sdIj+MPX6WHfrFp1KxecB7uAKijU7JsfziDEMSeBwgxJ2bOtSLkJEopiRtKKE0sSITxGQ9LTGCBOpJvkx6XwQDsD6IdCv0DB3P09kSAu5YR7ujNbUc7WMvO/Wi9W/rmb0CCKFQnw9CM/ZlCFMEsKDqggWLGJBoQF1btCPEICYaXzrOgQ7NmT56F9XLetun17Wm1cFHGUwR7YBzVggzPQAFegCVoAgwfwBF7Aq/FoPBtvxvu0tWQUM7vgj4zPb936odI=</latexit>

wkl = �klvl�kl<latexit sha1_base64="wT6Poi+w0YtOno+rkSZbUOc8PxE=">AAACCHicbZDLSsNAFIYnXmu9RV26cLAIrkoigm6EohuXFewF2hAm00k7dDIJMyeVErp046u4caGIWx/BnW/jNM1CW38Y+OY/5zBz/iARXIPjfFtLyyura+uljfLm1vbOrr2339Rxqihr0FjEqh0QzQSXrAEcBGsnipEoEKwVDG+m9daIKc1jeQ/jhHkR6UseckrAWL599OBnQzHBV7gLJM155ItuwIDkF9+uOFUnF14Et4AKKlT37a9uL6ZpxCRQQbTuuE4CXkYUcCrYpNxNNUsIHZI+6xiUJGLay/JFJvjEOD0cxsocCTh3f09kJNJ6HAWmMyIw0PO1qflfrZNCeOllXCYpMElnD4WpwBDjaSq4xxWjIMYGCFXc/BXTAVGEgsmubEJw51dehOZZ1XWq7t15pXZdxFFCh+gYnSIXXaAaukV11EAUPaJn9IrerCfrxXq3PmatS1Yxc4D+yPr8AXKQmj4=</latexit><latexit sha1_base64="wT6Poi+w0YtOno+rkSZbUOc8PxE=">AAACCHicbZDLSsNAFIYnXmu9RV26cLAIrkoigm6EohuXFewF2hAm00k7dDIJMyeVErp046u4caGIWx/BnW/jNM1CW38Y+OY/5zBz/iARXIPjfFtLyyura+uljfLm1vbOrr2339Rxqihr0FjEqh0QzQSXrAEcBGsnipEoEKwVDG+m9daIKc1jeQ/jhHkR6UseckrAWL599OBnQzHBV7gLJM155ItuwIDkF9+uOFUnF14Et4AKKlT37a9uL6ZpxCRQQbTuuE4CXkYUcCrYpNxNNUsIHZI+6xiUJGLay/JFJvjEOD0cxsocCTh3f09kJNJ6HAWmMyIw0PO1qflfrZNCeOllXCYpMElnD4WpwBDjaSq4xxWjIMYGCFXc/BXTAVGEgsmubEJw51dehOZZ1XWq7t15pXZdxFFCh+gYnSIXXaAaukV11EAUPaJn9IrerCfrxXq3PmatS1Yxc4D+yPr8AXKQmj4=</latexit><latexit sha1_base64="wT6Poi+w0YtOno+rkSZbUOc8PxE=">AAACCHicbZDLSsNAFIYnXmu9RV26cLAIrkoigm6EohuXFewF2hAm00k7dDIJMyeVErp046u4caGIWx/BnW/jNM1CW38Y+OY/5zBz/iARXIPjfFtLyyura+uljfLm1vbOrr2339Rxqihr0FjEqh0QzQSXrAEcBGsnipEoEKwVDG+m9daIKc1jeQ/jhHkR6UseckrAWL599OBnQzHBV7gLJM155ItuwIDkF9+uOFUnF14Et4AKKlT37a9uL6ZpxCRQQbTuuE4CXkYUcCrYpNxNNUsIHZI+6xiUJGLay/JFJvjEOD0cxsocCTh3f09kJNJ6HAWmMyIw0PO1qflfrZNCeOllXCYpMElnD4WpwBDjaSq4xxWjIMYGCFXc/BXTAVGEgsmubEJw51dehOZZ1XWq7t15pXZdxFFCh+gYnSIXXaAaukV11EAUPaJn9IrerCfrxXq3PmatS1Yxc4D+yPr8AXKQmj4=</latexit><latexit sha1_base64="wT6Poi+w0YtOno+rkSZbUOc8PxE=">AAACCHicbZDLSsNAFIYnXmu9RV26cLAIrkoigm6EohuXFewF2hAm00k7dDIJMyeVErp046u4caGIWx/BnW/jNM1CW38Y+OY/5zBz/iARXIPjfFtLyyura+uljfLm1vbOrr2339Rxqihr0FjEqh0QzQSXrAEcBGsnipEoEKwVDG+m9daIKc1jeQ/jhHkR6UseckrAWL599OBnQzHBV7gLJM155ItuwIDkF9+uOFUnF14Et4AKKlT37a9uL6ZpxCRQQbTuuE4CXkYUcCrYpNxNNUsIHZI+6xiUJGLay/JFJvjEOD0cxsocCTh3f09kJNJ6HAWmMyIw0PO1qflfrZNCeOllXCYpMElnD4WpwBDjaSq4xxWjIMYGCFXc/BXTAVGEgsmubEJw51dehOZZ1XWq7t15pXZdxFFCh+gYnSIXXaAaukV11EAUPaJn9IrerCfrxXq3PmatS1Yxc4D+yPr8AXKQmj4=</latexit>

• Non-centered Parameterization:

• Layer specific structured variational approximations:

�kl<latexit sha1_base64="BUR6kEZthBpCOw5mgpanigOVlW0=">AAAB8nicbVA9SwNBEN2LXzF+RS1tFoNgFe5E0DJoYxnBxEByhL3NXLJkb/fYnRPCkZ9hY6GIrb/Gzn/jJrlCEx8MPN6bYWZelEph0fe/vdLa+sbmVnm7srO7t39QPTxqW50ZDi2upTadiFmQQkELBUropAZYEkl4jMa3M//xCYwVWj3gJIUwYUMlYsEZOqnbiwBZPx/LKe1Xa37dn4OukqAgNVKg2a9+9QaaZwko5JJZ2w38FMOcGRRcwrTSyyykjI/ZELqOKpaADfP5yVN65pQBjbVxpZDO1d8TOUusnSSR60wYjuyyNxP/87oZxtdhLlSaISi+WBRnkqKms//pQBjgKCeOMG6Eu5XyETOMo0up4kIIll9eJe2LeuDXg/vLWuOmiKNMTsgpOScBuSINckeapEU40eSZvJI3D70X7937WLSWvGLmmPyB9/kDMiyRLg==</latexit><latexit sha1_base64="BUR6kEZthBpCOw5mgpanigOVlW0=">AAAB8nicbVA9SwNBEN2LXzF+RS1tFoNgFe5E0DJoYxnBxEByhL3NXLJkb/fYnRPCkZ9hY6GIrb/Gzn/jJrlCEx8MPN6bYWZelEph0fe/vdLa+sbmVnm7srO7t39QPTxqW50ZDi2upTadiFmQQkELBUropAZYEkl4jMa3M//xCYwVWj3gJIUwYUMlYsEZOqnbiwBZPx/LKe1Xa37dn4OukqAgNVKg2a9+9QaaZwko5JJZ2w38FMOcGRRcwrTSyyykjI/ZELqOKpaADfP5yVN65pQBjbVxpZDO1d8TOUusnSSR60wYjuyyNxP/87oZxtdhLlSaISi+WBRnkqKms//pQBjgKCeOMG6Eu5XyETOMo0up4kIIll9eJe2LeuDXg/vLWuOmiKNMTsgpOScBuSINckeapEU40eSZvJI3D70X7937WLSWvGLmmPyB9/kDMiyRLg==</latexit><latexit sha1_base64="BUR6kEZthBpCOw5mgpanigOVlW0=">AAAB8nicbVA9SwNBEN2LXzF+RS1tFoNgFe5E0DJoYxnBxEByhL3NXLJkb/fYnRPCkZ9hY6GIrb/Gzn/jJrlCEx8MPN6bYWZelEph0fe/vdLa+sbmVnm7srO7t39QPTxqW50ZDi2upTadiFmQQkELBUropAZYEkl4jMa3M//xCYwVWj3gJIUwYUMlYsEZOqnbiwBZPx/LKe1Xa37dn4OukqAgNVKg2a9+9QaaZwko5JJZ2w38FMOcGRRcwrTSyyykjI/ZELqOKpaADfP5yVN65pQBjbVxpZDO1d8TOUusnSSR60wYjuyyNxP/87oZxtdhLlSaISi+WBRnkqKms//pQBjgKCeOMG6Eu5XyETOMo0up4kIIll9eJe2LeuDXg/vLWuOmiKNMTsgpOScBuSINckeapEU40eSZvJI3D70X7937WLSWvGLmmPyB9/kDMiyRLg==</latexit><latexit sha1_base64="BUR6kEZthBpCOw5mgpanigOVlW0=">AAAB8nicbVA9SwNBEN2LXzF+RS1tFoNgFe5E0DJoYxnBxEByhL3NXLJkb/fYnRPCkZ9hY6GIrb/Gzn/jJrlCEx8MPN6bYWZelEph0fe/vdLa+sbmVnm7srO7t39QPTxqW50ZDi2upTadiFmQQkELBUropAZYEkl4jMa3M//xCYwVWj3gJIUwYUMlYsEZOqnbiwBZPx/LKe1Xa37dn4OukqAgNVKg2a9+9QaaZwko5JJZ2w38FMOcGRRcwrTSyyykjI/ZELqOKpaADfP5yVN65pQBjbVxpZDO1d8TOUusnSSR60wYjuyyNxP/87oZxtdhLlSaISi+WBRnkqKms//pQBjgKCeOMG6Eu5XyETOMo0up4kIIll9eJe2LeuDXg/vLWuOmiKNMTsgpOScBuSINckeapEU40eSZvJI3D70X7937WLSWvGLmmPyB9/kDMiyRLg==</latexit>

��

<latexit sha1_base64="Juzcs/zLpqFIAao5oRVScIfHEB8=">AAAB7XicbVDLSgNBEOyNrxhfUY9eBoPgKeyKoMegF48RzAOSJcxOepMxszPLzKwQQv7BiwdFvPo/3vwbJ8keNLGgoajqprsrSgU31ve/vcLa+sbmVnG7tLO7t39QPjxqGpVphg2mhNLtiBoUXGLDciuwnWqkSSSwFY1uZ37rCbXhSj7YcYphQgeSx5xR66RmN9KUYa9c8av+HGSVBDmpQI56r/zV7SuWJSgtE9SYTuCnNpxQbTkTOC11M4MpZSM6wI6jkiZowsn82ik5c0qfxEq7kpbM1d8TE5oYM04i15lQOzTL3kz8z+tkNr4OJ1ymmUXJFoviTBCryOx10ucamRVjRyjT3N1K2JC6AKwLqORCCJZfXiXNi2rgV4P7y0rtJo+jCCdwCucQwBXU4A7q0AAGj/AMr/DmKe/Fe/c+Fq0FL585hj/wPn8AfdCPDw==</latexit><latexit sha1_base64="Juzcs/zLpqFIAao5oRVScIfHEB8=">AAAB7XicbVDLSgNBEOyNrxhfUY9eBoPgKeyKoMegF48RzAOSJcxOepMxszPLzKwQQv7BiwdFvPo/3vwbJ8keNLGgoajqprsrSgU31ve/vcLa+sbmVnG7tLO7t39QPjxqGpVphg2mhNLtiBoUXGLDciuwnWqkSSSwFY1uZ37rCbXhSj7YcYphQgeSx5xR66RmN9KUYa9c8av+HGSVBDmpQI56r/zV7SuWJSgtE9SYTuCnNpxQbTkTOC11M4MpZSM6wI6jkiZowsn82ik5c0qfxEq7kpbM1d8TE5oYM04i15lQOzTL3kz8z+tkNr4OJ1ymmUXJFoviTBCryOx10ucamRVjRyjT3N1K2JC6AKwLqORCCJZfXiXNi2rgV4P7y0rtJo+jCCdwCucQwBXU4A7q0AAGj/AMr/DmKe/Fe/c+Fq0FL585hj/wPn8AfdCPDw==</latexit><latexit sha1_base64="Juzcs/zLpqFIAao5oRVScIfHEB8=">AAAB7XicbVDLSgNBEOyNrxhfUY9eBoPgKeyKoMegF48RzAOSJcxOepMxszPLzKwQQv7BiwdFvPo/3vwbJ8keNLGgoajqprsrSgU31ve/vcLa+sbmVnG7tLO7t39QPjxqGpVphg2mhNLtiBoUXGLDciuwnWqkSSSwFY1uZ37rCbXhSj7YcYphQgeSx5xR66RmN9KUYa9c8av+HGSVBDmpQI56r/zV7SuWJSgtE9SYTuCnNpxQbTkTOC11M4MpZSM6wI6jkiZowsn82ik5c0qfxEq7kpbM1d8TE5oYM04i15lQOzTL3kz8z+tkNr4OJ1ymmUXJFoviTBCryOx10ucamRVjRyjT3N1K2JC6AKwLqORCCJZfXiXNi2rgV4P7y0rtJo+jCCdwCucQwBXU4A7q0AAGj/AMr/DmKe/Fe/c+Fq0FL585hj/wPn8AfdCPDw==</latexit><latexit sha1_base64="Juzcs/zLpqFIAao5oRVScIfHEB8=">AAAB7XicbVDLSgNBEOyNrxhfUY9eBoPgKeyKoMegF48RzAOSJcxOepMxszPLzKwQQv7BiwdFvPo/3vwbJ8keNLGgoajqprsrSgU31ve/vcLa+sbmVnG7tLO7t39QPjxqGpVphg2mhNLtiBoUXGLDciuwnWqkSSSwFY1uZ37rCbXhSj7YcYphQgeSx5xR66RmN9KUYa9c8av+HGSVBDmpQI56r/zV7SuWJSgtE9SYTuCnNpxQbTkTOC11M4MpZSM6wI6jkiZowsn82ik5c0qfxEq7kpbM1d8TE5oYM04i15lQOzTL3kz8z+tkNr4OJ1ymmUXJFoviTBCryOx10ucamRVjRyjT3N1K2JC6AKwLqORCCJZfXiXNi2rgV4P7y0rtJo+jCCdwCucQwBXU4A7q0AAGj/AMr/DmKe/Fe/c+Fq0FL585hj/wPn8AfdCPDw==</latexit>

q(B) = Matrix-Normal(M, U, V )<latexit sha1_base64="9JuBdhLTbiJtjTJ38v9nIAbLqmE=">AAACDXicbVBNS0JBFJ1nX2ZfVss2QxYomLwXQW0CsU0bwyA/QB8ybxx1cN5HM/eF8vAPtOmvtGlRRNv27fo3jfoWpR24cDjnXu69xwkEV2Ca30ZiaXlldS25ntrY3NreSe/u1ZQfSsqq1Be+bDhEMcE9VgUOgjUCyYjrCFZ3BlcTv/7ApOK+dwejgNku6Xm8yykBLbXTR/fZUg5f4hawIURlApIPT2586RIxzpbzuJrHtVw7nTEL5hR4kVgxyaAYlXb6q9XxaegyD6ggSjUtMwA7IhI4FWycaoWKBYQOSI81NfWIy5QdTb8Z42OtdHDXl7o8wFP190REXKVGrqM7XQJ9Ne9NxP+8ZgjdCzviXhAC8+hsUTcUGHw8iQZ3uGQUxEgTQiXXt2LaJ5JQ0AGmdAjW/MuLpHZasMyCdXuWKZbiOJLoAB2iLLLQOSqia1RBVUTRI3pGr+jNeDJejHfjY9aaMOKZffQHxucPAi+ZmQ==</latexit><latexit sha1_base64="9JuBdhLTbiJtjTJ38v9nIAbLqmE=">AAACDXicbVBNS0JBFJ1nX2ZfVss2QxYomLwXQW0CsU0bwyA/QB8ybxx1cN5HM/eF8vAPtOmvtGlRRNv27fo3jfoWpR24cDjnXu69xwkEV2Ca30ZiaXlldS25ntrY3NreSe/u1ZQfSsqq1Be+bDhEMcE9VgUOgjUCyYjrCFZ3BlcTv/7ApOK+dwejgNku6Xm8yykBLbXTR/fZUg5f4hawIURlApIPT2586RIxzpbzuJrHtVw7nTEL5hR4kVgxyaAYlXb6q9XxaegyD6ggSjUtMwA7IhI4FWycaoWKBYQOSI81NfWIy5QdTb8Z42OtdHDXl7o8wFP190REXKVGrqM7XQJ9Ne9NxP+8ZgjdCzviXhAC8+hsUTcUGHw8iQZ3uGQUxEgTQiXXt2LaJ5JQ0AGmdAjW/MuLpHZasMyCdXuWKZbiOJLoAB2iLLLQOSqia1RBVUTRI3pGr+jNeDJejHfjY9aaMOKZffQHxucPAi+ZmQ==</latexit><latexit sha1_base64="9JuBdhLTbiJtjTJ38v9nIAbLqmE=">AAACDXicbVBNS0JBFJ1nX2ZfVss2QxYomLwXQW0CsU0bwyA/QB8ybxx1cN5HM/eF8vAPtOmvtGlRRNv27fo3jfoWpR24cDjnXu69xwkEV2Ca30ZiaXlldS25ntrY3NreSe/u1ZQfSsqq1Be+bDhEMcE9VgUOgjUCyYjrCFZ3BlcTv/7ApOK+dwejgNku6Xm8yykBLbXTR/fZUg5f4hawIURlApIPT2586RIxzpbzuJrHtVw7nTEL5hR4kVgxyaAYlXb6q9XxaegyD6ggSjUtMwA7IhI4FWycaoWKBYQOSI81NfWIy5QdTb8Z42OtdHDXl7o8wFP190REXKVGrqM7XQJ9Ne9NxP+8ZgjdCzviXhAC8+hsUTcUGHw8iQZ3uGQUxEgTQiXXt2LaJ5JQ0AGmdAjW/MuLpHZasMyCdXuWKZbiOJLoAB2iLLLQOSqia1RBVUTRI3pGr+jNeDJejHfjY9aaMOKZffQHxucPAi+ZmQ==</latexit><latexit sha1_base64="9JuBdhLTbiJtjTJ38v9nIAbLqmE=">AAACDXicbVBNS0JBFJ1nX2ZfVss2QxYomLwXQW0CsU0bwyA/QB8ybxx1cN5HM/eF8vAPtOmvtGlRRNv27fo3jfoWpR24cDjnXu69xwkEV2Ca30ZiaXlldS25ntrY3NreSe/u1ZQfSsqq1Be+bDhEMcE9VgUOgjUCyYjrCFZ3BlcTv/7ApOK+dwejgNku6Xm8yykBLbXTR/fZUg5f4hawIURlApIPT2586RIxzpbzuJrHtVw7nTEL5hR4kVgxyaAYlXb6q9XxaegyD6ggSjUtMwA7IhI4FWycaoWKBYQOSI81NfWIy5QdTb8Z42OtdHDXl7o8wFP190REXKVGrqM7XQJ9Ne9NxP+8ZgjdCzviXhAC8+hsUTcUGHw8iQZ3uGQUxEgTQiXXt2LaJ5JQ0AGmdAjW/MuLpHZasMyCdXuWKZbiOJLoAB2iLLLQOSqia1RBVUTRI3pGr+jNeDJejHfjY9aaMOKZffQHxucPAi+ZmQ==</latexit>

ln �kl<latexit sha1_base64="gERAcngHs530qJ2GKh2DnOyBW6w=">AAAB/HicbVBNS8NAEN3Ur1q/oj16WSyCp5KIoMeiF48V7Ae0IWy2m3bpZhN2J2II8a948aCIV3+IN/+N2zYHbX0w8Hhvhpl5QSK4Bsf5tipr6xubW9Xt2s7u3v6BfXjU1XGqKOvQWMSqHxDNBJesAxwE6yeKkSgQrBdMb2Z+74EpzWN5D1nCvIiMJQ85JWAk364PgT1CLiQuhkBSP5+KwrcbTtOZA68StyQNVKLt21/DUUzTiEmggmg9cJ0EvJwo4FSwojZMNUsInZIxGxgqScS0l8+PL/CpUUY4jJUpCXiu/p7ISaR1FgWmMyIw0cveTPzPG6QQXnk5l0kKTNLFojAVGGI8SwKPuGIURGYIoYqbWzGdEEUomLxqJgR3+eVV0j1vuk7TvbtotK7LOKroGJ2gM+SiS9RCt6iNOoiiDD2jV/RmPVkv1rv1sWitWOVMHf2B9fkDgxWVUA==</latexit><latexit sha1_base64="gERAcngHs530qJ2GKh2DnOyBW6w=">AAAB/HicbVBNS8NAEN3Ur1q/oj16WSyCp5KIoMeiF48V7Ae0IWy2m3bpZhN2J2II8a948aCIV3+IN/+N2zYHbX0w8Hhvhpl5QSK4Bsf5tipr6xubW9Xt2s7u3v6BfXjU1XGqKOvQWMSqHxDNBJesAxwE6yeKkSgQrBdMb2Z+74EpzWN5D1nCvIiMJQ85JWAk364PgT1CLiQuhkBSP5+KwrcbTtOZA68StyQNVKLt21/DUUzTiEmggmg9cJ0EvJwo4FSwojZMNUsInZIxGxgqScS0l8+PL/CpUUY4jJUpCXiu/p7ISaR1FgWmMyIw0cveTPzPG6QQXnk5l0kKTNLFojAVGGI8SwKPuGIURGYIoYqbWzGdEEUomLxqJgR3+eVV0j1vuk7TvbtotK7LOKroGJ2gM+SiS9RCt6iNOoiiDD2jV/RmPVkv1rv1sWitWOVMHf2B9fkDgxWVUA==</latexit><latexit sha1_base64="gERAcngHs530qJ2GKh2DnOyBW6w=">AAAB/HicbVBNS8NAEN3Ur1q/oj16WSyCp5KIoMeiF48V7Ae0IWy2m3bpZhN2J2II8a948aCIV3+IN/+N2zYHbX0w8Hhvhpl5QSK4Bsf5tipr6xubW9Xt2s7u3v6BfXjU1XGqKOvQWMSqHxDNBJesAxwE6yeKkSgQrBdMb2Z+74EpzWN5D1nCvIiMJQ85JWAk364PgT1CLiQuhkBSP5+KwrcbTtOZA68StyQNVKLt21/DUUzTiEmggmg9cJ0EvJwo4FSwojZMNUsInZIxGxgqScS0l8+PL/CpUUY4jJUpCXiu/p7ISaR1FgWmMyIw0cveTPzPG6QQXnk5l0kKTNLFojAVGGI8SwKPuGIURGYIoYqbWzGdEEUomLxqJgR3+eVV0j1vuk7TvbtotK7LOKroGJ2gM+SiS9RCt6iNOoiiDD2jV/RmPVkv1rv1sWitWOVMHf2B9fkDgxWVUA==</latexit><latexit sha1_base64="gERAcngHs530qJ2GKh2DnOyBW6w=">AAAB/HicbVBNS8NAEN3Ur1q/oj16WSyCp5KIoMeiF48V7Ae0IWy2m3bpZhN2J2II8a948aCIV3+IN/+N2zYHbX0w8Hhvhpl5QSK4Bsf5tipr6xubW9Xt2s7u3v6BfXjU1XGqKOvQWMSqHxDNBJesAxwE6yeKkSgQrBdMb2Z+74EpzWN5D1nCvIiMJQ85JWAk364PgT1CLiQuhkBSP5+KwrcbTtOZA68StyQNVKLt21/DUUzTiEmggmg9cJ0EvJwo4FSwojZMNUsInZIxGxgqScS0l8+PL/CpUUY4jJUpCXiu/p7ISaR1FgWmMyIw0cveTPzPG6QQXnk5l0kKTNLFojAVGGI8SwKPuGIURGYIoYqbWzGdEEUomLxqJgR3+eVV0j1vuk7TvbtotK7LOKroGJ2gM+SiS9RCt6iNOoiiDD2jV/RmPVkv1rv1sWitWOVMHf2B9fkDgxWVUA==</latexit>

U = hh� + �<latexit sha1_base64="odopVJmAyCH4Eiv5pOcOx3fDTRQ=">AAAB9XicbVBNS8NAEJ3Ur1q/oh69LBZREEoigl6EohePFUxbaGPZbDfN0s0m7G6UEvo/vHhQxKv/xZv/xm2bg7Y+GHi8N8PMvCDlTGnH+bZKS8srq2vl9crG5tb2jr2711RJJgn1SMIT2Q6wopwJ6mmmOW2nkuI44LQVDG8mfuuRSsUSca9HKfVjPBAsZARrIz146ApF0TE6Rd2GYj276tScKdAicQtShQKNnv3V7Scki6nQhGOlOq6Taj/HUjPC6bjSzRRNMRniAe0YKnBMlZ9Prx6jI6P0UZhIU0Kjqfp7IsexUqM4MJ0x1pGa9ybif14n0+GlnzORZpoKMlsUZhzpBE0iQH0mKdF8ZAgmkplbEYmwxESboComBHf+5UXSPKu5Ts29O6/Wr4s4ynAAh3ACLlxAHW6hAR4QkPAMr/BmPVkv1rv1MWstWcXMPvyB9fkDz8mQwg==</latexit><latexit sha1_base64="odopVJmAyCH4Eiv5pOcOx3fDTRQ=">AAAB9XicbVBNS8NAEJ3Ur1q/oh69LBZREEoigl6EohePFUxbaGPZbDfN0s0m7G6UEvo/vHhQxKv/xZv/xm2bg7Y+GHi8N8PMvCDlTGnH+bZKS8srq2vl9crG5tb2jr2711RJJgn1SMIT2Q6wopwJ6mmmOW2nkuI44LQVDG8mfuuRSsUSca9HKfVjPBAsZARrIz146ApF0TE6Rd2GYj276tScKdAicQtShQKNnv3V7Scki6nQhGOlOq6Taj/HUjPC6bjSzRRNMRniAe0YKnBMlZ9Prx6jI6P0UZhIU0Kjqfp7IsexUqM4MJ0x1pGa9ybif14n0+GlnzORZpoKMlsUZhzpBE0iQH0mKdF8ZAgmkplbEYmwxESboComBHf+5UXSPKu5Ts29O6/Wr4s4ynAAh3ACLlxAHW6hAR4QkPAMr/BmPVkv1rv1MWstWcXMPvyB9fkDz8mQwg==</latexit><latexit sha1_base64="odopVJmAyCH4Eiv5pOcOx3fDTRQ=">AAAB9XicbVBNS8NAEJ3Ur1q/oh69LBZREEoigl6EohePFUxbaGPZbDfN0s0m7G6UEvo/vHhQxKv/xZv/xm2bg7Y+GHi8N8PMvCDlTGnH+bZKS8srq2vl9crG5tb2jr2711RJJgn1SMIT2Q6wopwJ6mmmOW2nkuI44LQVDG8mfuuRSsUSca9HKfVjPBAsZARrIz146ApF0TE6Rd2GYj276tScKdAicQtShQKNnv3V7Scki6nQhGOlOq6Taj/HUjPC6bjSzRRNMRniAe0YKnBMlZ9Prx6jI6P0UZhIU0Kjqfp7IsexUqM4MJ0x1pGa9ybif14n0+GlnzORZpoKMlsUZhzpBE0iQH0mKdF8ZAgmkplbEYmwxESboComBHf+5UXSPKu5Ts29O6/Wr4s4ynAAh3ACLlxAHW6hAR4QkPAMr/BmPVkv1rv1MWstWcXMPvyB9fkDz8mQwg==</latexit><latexit sha1_base64="odopVJmAyCH4Eiv5pOcOx3fDTRQ=">AAAB9XicbVBNS8NAEJ3Ur1q/oh69LBZREEoigl6EohePFUxbaGPZbDfN0s0m7G6UEvo/vHhQxKv/xZv/xm2bg7Y+GHi8N8PMvCDlTGnH+bZKS8srq2vl9crG5tb2jr2711RJJgn1SMIT2Q6wopwJ6mmmOW2nkuI44LQVDG8mfuuRSsUSca9HKfVjPBAsZARrIz146ApF0TE6Rd2GYj276tScKdAicQtShQKNnv3V7Scki6nQhGOlOq6Taj/HUjPC6bjSzRRNMRniAe0YKnBMlZ9Prx6jI6P0UZhIU0Kjqfp7IsexUqM4MJ0x1pGa9ybif14n0+GlnzORZpoKMlsUZhzpBE0iQH0mKdF8ZAgmkplbEYmwxESboComBHf+5UXSPKu5Ts29O6/Wr4s4ynAAh3ACLlxAHW6hAR4QkPAMr/BmPVkv1rv1MWstWcXMPvyB9fkDz8mQwg==</latexit>

Low dimensional covariance maintains posterior structure between weights and scales.

#in

puts

<latexit sha1_base64="T3q/yKw3n9RAynz9XXt+mEAEae8=">AAAB8nicbVDLSgMxFL1TX7W+qi7dBAfBVZkRQcFN0Y3LCvYBM0PJpJk2NJMMSUYopZ/hxoUibv0ad/6N6XQW2nogcDjnXnLPiTPOtPG8b6eytr6xuVXdru3s7u0f1A+POlrmitA2kVyqXow15UzQtmGG016mKE5jTrvx+G7ud5+o0kyKRzPJaJTioWAJI9hYKQjd8AYxkeVG9+uu1/AKoFXil8SFEq1+/SscSJKnVBjCsdaB72UmmmJlGOF0VgtzTTNMxnhIA0sFTqmOpsXJM3RmlQFKpLJPGFSovzemONV6ksZ2MsVmpJe9ufifF+QmuY6mRSQqyOKjJOfISDTPjwZMUWL4xBJMFLO3IjLCChNjW6rZEvzlyKukc9HwvYb/cOk2b8s6qnACp3AOPlxBE+6hBW0gIOEZXuHNMc6L8+58LEYrTrlzDH/gfP4Ah2uQwQ==</latexit><latexit sha1_base64="T3q/yKw3n9RAynz9XXt+mEAEae8=">AAAB8nicbVDLSgMxFL1TX7W+qi7dBAfBVZkRQcFN0Y3LCvYBM0PJpJk2NJMMSUYopZ/hxoUibv0ad/6N6XQW2nogcDjnXnLPiTPOtPG8b6eytr6xuVXdru3s7u0f1A+POlrmitA2kVyqXow15UzQtmGG016mKE5jTrvx+G7ud5+o0kyKRzPJaJTioWAJI9hYKQjd8AYxkeVG9+uu1/AKoFXil8SFEq1+/SscSJKnVBjCsdaB72UmmmJlGOF0VgtzTTNMxnhIA0sFTqmOpsXJM3RmlQFKpLJPGFSovzemONV6ksZ2MsVmpJe9ufifF+QmuY6mRSQqyOKjJOfISDTPjwZMUWL4xBJMFLO3IjLCChNjW6rZEvzlyKukc9HwvYb/cOk2b8s6qnACp3AOPlxBE+6hBW0gIOEZXuHNMc6L8+58LEYrTrlzDH/gfP4Ah2uQwQ==</latexit><latexit sha1_base64="T3q/yKw3n9RAynz9XXt+mEAEae8=">AAAB8nicbVDLSgMxFL1TX7W+qi7dBAfBVZkRQcFN0Y3LCvYBM0PJpJk2NJMMSUYopZ/hxoUibv0ad/6N6XQW2nogcDjnXnLPiTPOtPG8b6eytr6xuVXdru3s7u0f1A+POlrmitA2kVyqXow15UzQtmGG016mKE5jTrvx+G7ud5+o0kyKRzPJaJTioWAJI9hYKQjd8AYxkeVG9+uu1/AKoFXil8SFEq1+/SscSJKnVBjCsdaB72UmmmJlGOF0VgtzTTNMxnhIA0sFTqmOpsXJM3RmlQFKpLJPGFSovzemONV6ksZ2MsVmpJe9ufifF+QmuY6mRSQqyOKjJOfISDTPjwZMUWL4xBJMFLO3IjLCChNjW6rZEvzlyKukc9HwvYb/cOk2b8s6qnACp3AOPlxBE+6hBW0gIOEZXuHNMc6L8+58LEYrTrlzDH/gfP4Ah2uQwQ==</latexit><latexit sha1_base64="T3q/yKw3n9RAynz9XXt+mEAEae8=">AAAB8nicbVDLSgMxFL1TX7W+qi7dBAfBVZkRQcFN0Y3LCvYBM0PJpJk2NJMMSUYopZ/hxoUibv0ad/6N6XQW2nogcDjnXnLPiTPOtPG8b6eytr6xuVXdru3s7u0f1A+POlrmitA2kVyqXow15UzQtmGG016mKE5jTrvx+G7ud5+o0kyKRzPJaJTioWAJI9hYKQjd8AYxkeVG9+uu1/AKoFXil8SFEq1+/SscSJKnVBjCsdaB72UmmmJlGOF0VgtzTTNMxnhIA0sFTqmOpsXJM3RmlQFKpLJPGFSovzemONV6ksZ2MsVmpJe9ufifF+QmuY6mRSQqyOKjJOfISDTPjwZMUWL4xBJMFLO3IjLCChNjW6rZEvzlyKukc9HwvYb/cOk2b8s6qnACp3AOPlxBE+6hBW0gIOEZXuHNMc6L8+58LEYrTrlzDH/gfP4Ah2uQwQ==</latexit>

# outputs<latexit sha1_base64="ICBz4Nuoqv8BUptsqPXTA09K6E0=">AAAB83icbVBNSwMxEM3Wr1q/qh69BBfBU9kVQcFL0YvHCvYDukvJptk2NJssyUQopX/DiwdFvPpnvPlvTNs9aOuDgcd7M8zMS3LBDQTBt1daW9/Y3CpvV3Z29/YPqodHLaOspqxJlVC6kxDDBJesCRwE6+SakSwRrJ2M7mZ++4lpw5V8hHHO4owMJE85JeCkKPKjG6ws5BZMr+oHtWAOvErCgvioQKNX/Yr6itqMSaCCGNMNgxziCdHAqWDTSmQNywkdkQHrOipJxkw8md88xWdO6eNUaVcS8Fz9PTEhmTHjLHGdGYGhWfZm4n9e10J6HU+4dC8xSReLUiswKDwLAPe5ZhTE2BFCNXe3YjokmlBwMVVcCOHyy6ukdVELg1r4cOnXb4s4yugEnaJzFKIrVEf3qIGaiKIcPaNX9OZZ78V79z4WrSWvmDlGf+B9/gB0GpFM</latexit><latexit sha1_base64="ICBz4Nuoqv8BUptsqPXTA09K6E0=">AAAB83icbVBNSwMxEM3Wr1q/qh69BBfBU9kVQcFL0YvHCvYDukvJptk2NJssyUQopX/DiwdFvPpnvPlvTNs9aOuDgcd7M8zMS3LBDQTBt1daW9/Y3CpvV3Z29/YPqodHLaOspqxJlVC6kxDDBJesCRwE6+SakSwRrJ2M7mZ++4lpw5V8hHHO4owMJE85JeCkKPKjG6ws5BZMr+oHtWAOvErCgvioQKNX/Yr6itqMSaCCGNMNgxziCdHAqWDTSmQNywkdkQHrOipJxkw8md88xWdO6eNUaVcS8Fz9PTEhmTHjLHGdGYGhWfZm4n9e10J6HU+4dC8xSReLUiswKDwLAPe5ZhTE2BFCNXe3YjokmlBwMVVcCOHyy6ukdVELg1r4cOnXb4s4yugEnaJzFKIrVEf3qIGaiKIcPaNX9OZZ78V79z4WrSWvmDlGf+B9/gB0GpFM</latexit><latexit sha1_base64="ICBz4Nuoqv8BUptsqPXTA09K6E0=">AAAB83icbVBNSwMxEM3Wr1q/qh69BBfBU9kVQcFL0YvHCvYDukvJptk2NJssyUQopX/DiwdFvPpnvPlvTNs9aOuDgcd7M8zMS3LBDQTBt1daW9/Y3CpvV3Z29/YPqodHLaOspqxJlVC6kxDDBJesCRwE6+SakSwRrJ2M7mZ++4lpw5V8hHHO4owMJE85JeCkKPKjG6ws5BZMr+oHtWAOvErCgvioQKNX/Yr6itqMSaCCGNMNgxziCdHAqWDTSmQNywkdkQHrOipJxkw8md88xWdO6eNUaVcS8Fz9PTEhmTHjLHGdGYGhWfZm4n9e10J6HU+4dC8xSReLUiswKDwLAPe5ZhTE2BFCNXe3YjokmlBwMVVcCOHyy6ukdVELg1r4cOnXb4s4yugEnaJzFKIrVEf3qIGaiKIcPaNX9OZZ78V79z4WrSWvmDlGf+B9/gB0GpFM</latexit><latexit sha1_base64="ICBz4Nuoqv8BUptsqPXTA09K6E0=">AAAB83icbVBNSwMxEM3Wr1q/qh69BBfBU9kVQcFL0YvHCvYDukvJptk2NJssyUQopX/DiwdFvPpnvPlvTNs9aOuDgcd7M8zMS3LBDQTBt1daW9/Y3CpvV3Z29/YPqodHLaOspqxJlVC6kxDDBJesCRwE6+SakSwRrJ2M7mZ++4lpw5V8hHHO4owMJE85JeCkKPKjG6ws5BZMr+oHtWAOvErCgvioQKNX/Yr6itqMSaCCGNMNgxziCdHAqWDTSmQNywkdkQHrOipJxkw8md88xWdO6eNUaVcS8Fz9PTEhmTHjLHGdGYGhWfZm4n9e10J6HU+4dC8xSReLUiswKDwLAPe5ZhTE2BFCNXe3YjokmlBwMVVcCOHyy6ukdVELg1r4cOnXb4s4yugEnaJzFKIrVEf3qIGaiKIcPaNX9OZZ78V79z4WrSWvmDlGf+B9/gB0GpFM</latexit>

weights

scales

!26

• Local re-parameterization:

�kl<latexit sha1_base64="BUR6kEZthBpCOw5mgpanigOVlW0=">AAAB8nicbVA9SwNBEN2LXzF+RS1tFoNgFe5E0DJoYxnBxEByhL3NXLJkb/fYnRPCkZ9hY6GIrb/Gzn/jJrlCEx8MPN6bYWZelEph0fe/vdLa+sbmVnm7srO7t39QPTxqW50ZDi2upTadiFmQQkELBUropAZYEkl4jMa3M//xCYwVWj3gJIUwYUMlYsEZOqnbiwBZPx/LKe1Xa37dn4OukqAgNVKg2a9+9QaaZwko5JJZ2w38FMOcGRRcwrTSyyykjI/ZELqOKpaADfP5yVN65pQBjbVxpZDO1d8TOUusnSSR60wYjuyyNxP/87oZxtdhLlSaISi+WBRnkqKms//pQBjgKCeOMG6Eu5XyETOMo0up4kIIll9eJe2LeuDXg/vLWuOmiKNMTsgpOScBuSINckeapEU40eSZvJI3D70X7937WLSWvGLmmPyB9/kDMiyRLg==</latexit><latexit sha1_base64="BUR6kEZthBpCOw5mgpanigOVlW0=">AAAB8nicbVA9SwNBEN2LXzF+RS1tFoNgFe5E0DJoYxnBxEByhL3NXLJkb/fYnRPCkZ9hY6GIrb/Gzn/jJrlCEx8MPN6bYWZelEph0fe/vdLa+sbmVnm7srO7t39QPTxqW50ZDi2upTadiFmQQkELBUropAZYEkl4jMa3M//xCYwVWj3gJIUwYUMlYsEZOqnbiwBZPx/LKe1Xa37dn4OukqAgNVKg2a9+9QaaZwko5JJZ2w38FMOcGRRcwrTSyyykjI/ZELqOKpaADfP5yVN65pQBjbVxpZDO1d8TOUusnSSR60wYjuyyNxP/87oZxtdhLlSaISi+WBRnkqKms//pQBjgKCeOMG6Eu5XyETOMo0up4kIIll9eJe2LeuDXg/vLWuOmiKNMTsgpOScBuSINckeapEU40eSZvJI3D70X7937WLSWvGLmmPyB9/kDMiyRLg==</latexit><latexit sha1_base64="BUR6kEZthBpCOw5mgpanigOVlW0=">AAAB8nicbVA9SwNBEN2LXzF+RS1tFoNgFe5E0DJoYxnBxEByhL3NXLJkb/fYnRPCkZ9hY6GIrb/Gzn/jJrlCEx8MPN6bYWZelEph0fe/vdLa+sbmVnm7srO7t39QPTxqW50ZDi2upTadiFmQQkELBUropAZYEkl4jMa3M//xCYwVWj3gJIUwYUMlYsEZOqnbiwBZPx/LKe1Xa37dn4OukqAgNVKg2a9+9QaaZwko5JJZ2w38FMOcGRRcwrTSyyykjI/ZELqOKpaADfP5yVN65pQBjbVxpZDO1d8TOUusnSSR60wYjuyyNxP/87oZxtdhLlSaISi+WBRnkqKms//pQBjgKCeOMG6Eu5XyETOMo0up4kIIll9eJe2LeuDXg/vLWuOmiKNMTsgpOScBuSINckeapEU40eSZvJI3D70X7937WLSWvGLmmPyB9/kDMiyRLg==</latexit><latexit sha1_base64="BUR6kEZthBpCOw5mgpanigOVlW0=">AAAB8nicbVA9SwNBEN2LXzF+RS1tFoNgFe5E0DJoYxnBxEByhL3NXLJkb/fYnRPCkZ9hY6GIrb/Gzn/jJrlCEx8MPN6bYWZelEph0fe/vdLa+sbmVnm7srO7t39QPTxqW50ZDi2upTadiFmQQkELBUropAZYEkl4jMa3M//xCYwVWj3gJIUwYUMlYsEZOqnbiwBZPx/LKe1Xa37dn4OukqAgNVKg2a9+9QaaZwko5JJZ2w38FMOcGRRcwrTSyyykjI/ZELqOKpaADfP5yVN65pQBjbVxpZDO1d8TOUusnSSR60wYjuyyNxP/87oZxtdhLlSaISi+WBRnkqKms//pQBjgKCeOMG6Eu5XyETOMo0up4kIIll9eJe2LeuDXg/vLWuOmiKNMTsgpOScBuSINckeapEU40eSZvJI3D70X7937WLSWvGLmmPyB9/kDMiyRLg==</latexit>

ln �kl<latexit sha1_base64="gERAcngHs530qJ2GKh2DnOyBW6w=">AAAB/HicbVBNS8NAEN3Ur1q/oj16WSyCp5KIoMeiF48V7Ae0IWy2m3bpZhN2J2II8a948aCIV3+IN/+N2zYHbX0w8Hhvhpl5QSK4Bsf5tipr6xubW9Xt2s7u3v6BfXjU1XGqKOvQWMSqHxDNBJesAxwE6yeKkSgQrBdMb2Z+74EpzWN5D1nCvIiMJQ85JWAk364PgT1CLiQuhkBSP5+KwrcbTtOZA68StyQNVKLt21/DUUzTiEmggmg9cJ0EvJwo4FSwojZMNUsInZIxGxgqScS0l8+PL/CpUUY4jJUpCXiu/p7ISaR1FgWmMyIw0cveTPzPG6QQXnk5l0kKTNLFojAVGGI8SwKPuGIURGYIoYqbWzGdEEUomLxqJgR3+eVV0j1vuk7TvbtotK7LOKroGJ2gM+SiS9RCt6iNOoiiDD2jV/RmPVkv1rv1sWitWOVMHf2B9fkDgxWVUA==</latexit><latexit sha1_base64="gERAcngHs530qJ2GKh2DnOyBW6w=">AAAB/HicbVBNS8NAEN3Ur1q/oj16WSyCp5KIoMeiF48V7Ae0IWy2m3bpZhN2J2II8a948aCIV3+IN/+N2zYHbX0w8Hhvhpl5QSK4Bsf5tipr6xubW9Xt2s7u3v6BfXjU1XGqKOvQWMSqHxDNBJesAxwE6yeKkSgQrBdMb2Z+74EpzWN5D1nCvIiMJQ85JWAk364PgT1CLiQuhkBSP5+KwrcbTtOZA68StyQNVKLt21/DUUzTiEmggmg9cJ0EvJwo4FSwojZMNUsInZIxGxgqScS0l8+PL/CpUUY4jJUpCXiu/p7ISaR1FgWmMyIw0cveTPzPG6QQXnk5l0kKTNLFojAVGGI8SwKPuGIURGYIoYqbWzGdEEUomLxqJgR3+eVV0j1vuk7TvbtotK7LOKroGJ2gM+SiS9RCt6iNOoiiDD2jV/RmPVkv1rv1sWitWOVMHf2B9fkDgxWVUA==</latexit><latexit sha1_base64="gERAcngHs530qJ2GKh2DnOyBW6w=">AAAB/HicbVBNS8NAEN3Ur1q/oj16WSyCp5KIoMeiF48V7Ae0IWy2m3bpZhN2J2II8a948aCIV3+IN/+N2zYHbX0w8Hhvhpl5QSK4Bsf5tipr6xubW9Xt2s7u3v6BfXjU1XGqKOvQWMSqHxDNBJesAxwE6yeKkSgQrBdMb2Z+74EpzWN5D1nCvIiMJQ85JWAk364PgT1CLiQuhkBSP5+KwrcbTtOZA68StyQNVKLt21/DUUzTiEmggmg9cJ0EvJwo4FSwojZMNUsInZIxGxgqScS0l8+PL/CpUUY4jJUpCXiu/p7ISaR1FgWmMyIw0cveTPzPG6QQXnk5l0kKTNLFojAVGGI8SwKPuGIURGYIoYqbWzGdEEUomLxqJgR3+eVV0j1vuk7TvbtotK7LOKroGJ2gM+SiS9RCt6iNOoiiDD2jV/RmPVkv1rv1sWitWOVMHf2B9fkDgxWVUA==</latexit><latexit sha1_base64="gERAcngHs530qJ2GKh2DnOyBW6w=">AAAB/HicbVBNS8NAEN3Ur1q/oj16WSyCp5KIoMeiF48V7Ae0IWy2m3bpZhN2J2II8a948aCIV3+IN/+N2zYHbX0w8Hhvhpl5QSK4Bsf5tipr6xubW9Xt2s7u3v6BfXjU1XGqKOvQWMSqHxDNBJesAxwE6yeKkSgQrBdMb2Z+74EpzWN5D1nCvIiMJQ85JWAk364PgT1CLiQuhkBSP5+KwrcbTtOZA68StyQNVKLt21/DUUzTiEmggmg9cJ0EvJwo4FSwojZMNUsInZIxGxgqScS0l8+PL/CpUUY4jJUpCXiu/p7ISaR1FgWmMyIw0cveTPzPG6QQXnk5l0kKTNLFojAVGGI8SwKPuGIURGYIoYqbWzGdEEUomLxqJgR3+eVV0j1vuk7TvbtotK7LOKroGJ2gM+SiS9RCt6iNOoiiDD2jV/RmPVkv1rv1sWitWOVMHf2B9fkDgxWVUA==</latexit>q(

<latexit sha1_base64="8FKCze5WmZ5Lvmp2roRbssYrnYc=">AAAB6XicbVBNS8NAEJ3Ur1q/qh69LBahp5KIoMeiF49V7Ae0oWy2k3bpZhN3N0IJ/QdePCji1X/kzX/jts1BWx8MPN6bYWZekAiujet+O4W19Y3NreJ2aWd3b/+gfHjU0nGqGDZZLGLVCahGwSU2DTcCO4lCGgUC28H4Zua3n1BpHssHM0nQj+hQ8pAzaqx0/1jtlytuzZ2DrBIvJxXI0eiXv3qDmKURSsME1brruYnxM6oMZwKnpV6qMaFsTIfYtVTSCLWfzS+dkjOrDEgYK1vSkLn6eyKjkdaTKLCdETUjvezNxP+8bmrCKz/jMkkNSrZYFKaCmJjM3iYDrpAZMbGEMsXtrYSNqKLM2HBKNgRv+eVV0jqveW7Nu7uo1K/zOIpwAqdQBQ8uoQ630IAmMAjhGV7hzRk7L86787FoLTj5zDH8gfP5Az2hjSc=</latexit><latexit sha1_base64="8FKCze5WmZ5Lvmp2roRbssYrnYc=">AAAB6XicbVBNS8NAEJ3Ur1q/qh69LBahp5KIoMeiF49V7Ae0oWy2k3bpZhN3N0IJ/QdePCji1X/kzX/jts1BWx8MPN6bYWZekAiujet+O4W19Y3NreJ2aWd3b/+gfHjU0nGqGDZZLGLVCahGwSU2DTcCO4lCGgUC28H4Zua3n1BpHssHM0nQj+hQ8pAzaqx0/1jtlytuzZ2DrBIvJxXI0eiXv3qDmKURSsME1brruYnxM6oMZwKnpV6qMaFsTIfYtVTSCLWfzS+dkjOrDEgYK1vSkLn6eyKjkdaTKLCdETUjvezNxP+8bmrCKz/jMkkNSrZYFKaCmJjM3iYDrpAZMbGEMsXtrYSNqKLM2HBKNgRv+eVV0jqveW7Nu7uo1K/zOIpwAqdQBQ8uoQ630IAmMAjhGV7hzRk7L86787FoLTj5zDH8gfP5Az2hjSc=</latexit><latexit sha1_base64="8FKCze5WmZ5Lvmp2roRbssYrnYc=">AAAB6XicbVBNS8NAEJ3Ur1q/qh69LBahp5KIoMeiF49V7Ae0oWy2k3bpZhN3N0IJ/QdePCji1X/kzX/jts1BWx8MPN6bYWZekAiujet+O4W19Y3NreJ2aWd3b/+gfHjU0nGqGDZZLGLVCahGwSU2DTcCO4lCGgUC28H4Zua3n1BpHssHM0nQj+hQ8pAzaqx0/1jtlytuzZ2DrBIvJxXI0eiXv3qDmKURSsME1brruYnxM6oMZwKnpV6qMaFsTIfYtVTSCLWfzS+dkjOrDEgYK1vSkLn6eyKjkdaTKLCdETUjvezNxP+8bmrCKz/jMkkNSrZYFKaCmJjM3iYDrpAZMbGEMsXtrYSNqKLM2HBKNgRv+eVV0jqveW7Nu7uo1K/zOIpwAqdQBQ8uoQ630IAmMAjhGV7hzRk7L86787FoLTj5zDH8gfP5Az2hjSc=</latexit><latexit sha1_base64="8FKCze5WmZ5Lvmp2roRbssYrnYc=">AAAB6XicbVBNS8NAEJ3Ur1q/qh69LBahp5KIoMeiF49V7Ae0oWy2k3bpZhN3N0IJ/QdePCji1X/kzX/jts1BWx8MPN6bYWZekAiujet+O4W19Y3NreJ2aWd3b/+gfHjU0nGqGDZZLGLVCahGwSU2DTcCO4lCGgUC28H4Zua3n1BpHssHM0nQj+hQ8pAzaqx0/1jtlytuzZ2DrBIvJxXI0eiXv3qDmKURSsME1brruYnxM6oMZwKnpV6qMaFsTIfYtVTSCLWfzS+dkjOrDEgYK1vSkLn6eyKjkdaTKLCdETUjvezNxP+8bmrCKz/jMkkNSrZYFKaCmJjM3iYDrpAZMbGEMsXtrYSNqKLM2HBKNgRv+eVV0jqveW7Nu7uo1K/zOIpwAqdQBQ8uoQ630IAmMAjhGV7hzRk7L86787FoLTj5zDH8gfP5Az2hjSc=</latexit>

)<latexit sha1_base64="Vc/F7wCLwhVrbNGal5W4fQhkQ2w=">AAAB6HicbVBNS8NAEJ3Ur1q/qh69LBZBLyURQY9FLx5bsB/QhrLZTtq1m03Y3Qgl9Bd48aCIV3+SN/+N2zYHbX0w8Hhvhpl5QSK4Nq777RTW1jc2t4rbpZ3dvf2D8uFRS8epYthksYhVJ6AaBZfYNNwI7CQKaRQIbAfju5nffkKleSwfzCRBP6JDyUPOqLFS46JfrrhVdw6ySrycVCBHvV/+6g1ilkYoDRNU667nJsbPqDKcCZyWeqnGhLIxHWLXUkkj1H42P3RKzqwyIGGsbElD5urviYxGWk+iwHZG1Iz0sjcT//O6qQlv/IzLJDUo2WJRmApiYjL7mgy4QmbExBLKFLe3EjaiijJjsynZELzll1dJ67LquVWvcVWp3eZxFOEETuEcPLiGGtxDHZrAAOEZXuHNeXRenHfnY9FacPKZY/gD5/MHbn2MrQ==</latexit><latexit sha1_base64="Vc/F7wCLwhVrbNGal5W4fQhkQ2w=">AAAB6HicbVBNS8NAEJ3Ur1q/qh69LBZBLyURQY9FLx5bsB/QhrLZTtq1m03Y3Qgl9Bd48aCIV3+SN/+N2zYHbX0w8Hhvhpl5QSK4Nq777RTW1jc2t4rbpZ3dvf2D8uFRS8epYthksYhVJ6AaBZfYNNwI7CQKaRQIbAfju5nffkKleSwfzCRBP6JDyUPOqLFS46JfrrhVdw6ySrycVCBHvV/+6g1ilkYoDRNU667nJsbPqDKcCZyWeqnGhLIxHWLXUkkj1H42P3RKzqwyIGGsbElD5urviYxGWk+iwHZG1Iz0sjcT//O6qQlv/IzLJDUo2WJRmApiYjL7mgy4QmbExBLKFLe3EjaiijJjsynZELzll1dJ67LquVWvcVWp3eZxFOEETuEcPLiGGtxDHZrAAOEZXuHNeXRenHfnY9FacPKZY/gD5/MHbn2MrQ==</latexit><latexit sha1_base64="Vc/F7wCLwhVrbNGal5W4fQhkQ2w=">AAAB6HicbVBNS8NAEJ3Ur1q/qh69LBZBLyURQY9FLx5bsB/QhrLZTtq1m03Y3Qgl9Bd48aCIV3+SN/+N2zYHbX0w8Hhvhpl5QSK4Nq777RTW1jc2t4rbpZ3dvf2D8uFRS8epYthksYhVJ6AaBZfYNNwI7CQKaRQIbAfju5nffkKleSwfzCRBP6JDyUPOqLFS46JfrrhVdw6ySrycVCBHvV/+6g1ilkYoDRNU667nJsbPqDKcCZyWeqnGhLIxHWLXUkkj1H42P3RKzqwyIGGsbElD5urviYxGWk+iwHZG1Iz0sjcT//O6qQlv/IzLJDUo2WJRmApiYjL7mgy4QmbExBLKFLe3EjaiijJjsynZELzll1dJ67LquVWvcVWp3eZxFOEETuEcPLiGGtxDHZrAAOEZXuHNeXRenHfnY9FacPKZY/gD5/MHbn2MrQ==</latexit><latexit sha1_base64="Vc/F7wCLwhVrbNGal5W4fQhkQ2w=">AAAB6HicbVBNS8NAEJ3Ur1q/qh69LBZBLyURQY9FLx5bsB/QhrLZTtq1m03Y3Qgl9Bd48aCIV3+SN/+N2zYHbX0w8Hhvhpl5QSK4Nq777RTW1jc2t4rbpZ3dvf2D8uFRS8epYthksYhVJ6AaBZfYNNwI7CQKaRQIbAfju5nffkKleSwfzCRBP6JDyUPOqLFS46JfrrhVdw6ySrycVCBHvV/+6g1ilkYoDRNU667nJsbPqDKcCZyWeqnGhLIxHWLXUkkj1H42P3RKzqwyIGGsbElD5urviYxGWk+iwHZG1Iz0sjcT//O6qQlv/IzLJDUo2WJRmApiYjL7mgy4QmbExBLKFLe3EjaiijJjsynZELzll1dJ67LquVWvcVWp3eZxFOEETuEcPLiGGtxDHZrAAOEZXuHNeXRenHfnY9FacPKZY/gD5/MHbn2MrQ==</latexit>

= Matrix-Normal(M�|� , U�|� , V�|� )<latexit sha1_base64="telQwAcZbhajrNyKXrQhT697vtQ=">AAACOHicbZDLSitBEIZ71OMlx0vUpZvGcEDhGGZE0I0gunHjDUwUMiHUdCrapHtm6K4Rw5DHcuNjuBM3LhRx6xPYiVlo4g8NP19VUV1/lCppyfcfvbHxiT+TU9Mzhb+zc/MLxcWlqk0yI7AiEpWYywgsKhljhSQpvEwNgo4UXkTtg1794gaNlUl8Tp0U6xquYtmSAsihRvFkl4eEt5QfARl5u3GcGA2qu3bUyMMICUItmyFB1v3PK6OoOozWG8WSX/b74qMmGJgSG+i0UXwIm4nINMYkFFhbC/yU6jkYkkJhtxBmFlMQbbjCmrMxaLT1vH94l/9zpMlbiXEvJt6n3ydy0NZ2dOQ6NdC1Ha714G+1WkatnXou4zQjjMXXolamOCW8lyJvSoOCVMcZEEa6v3JxDQYEuawLLoRg+ORRU90sB345ONsq7e0P4phmK2yVrbGAbbM9dshOWYUJdsee2At79e69Z+/Ne/9qHfMGM8vsh7yPTxM6rcQ=</latexit><latexit sha1_base64="telQwAcZbhajrNyKXrQhT697vtQ=">AAACOHicbZDLSitBEIZ71OMlx0vUpZvGcEDhGGZE0I0gunHjDUwUMiHUdCrapHtm6K4Rw5DHcuNjuBM3LhRx6xPYiVlo4g8NP19VUV1/lCppyfcfvbHxiT+TU9Mzhb+zc/MLxcWlqk0yI7AiEpWYywgsKhljhSQpvEwNgo4UXkTtg1794gaNlUl8Tp0U6xquYtmSAsihRvFkl4eEt5QfARl5u3GcGA2qu3bUyMMICUItmyFB1v3PK6OoOozWG8WSX/b74qMmGJgSG+i0UXwIm4nINMYkFFhbC/yU6jkYkkJhtxBmFlMQbbjCmrMxaLT1vH94l/9zpMlbiXEvJt6n3ydy0NZ2dOQ6NdC1Ha714G+1WkatnXou4zQjjMXXolamOCW8lyJvSoOCVMcZEEa6v3JxDQYEuawLLoRg+ORRU90sB345ONsq7e0P4phmK2yVrbGAbbM9dshOWYUJdsee2At79e69Z+/Ne/9qHfMGM8vsh7yPTxM6rcQ=</latexit><latexit sha1_base64="telQwAcZbhajrNyKXrQhT697vtQ=">AAACOHicbZDLSitBEIZ71OMlx0vUpZvGcEDhGGZE0I0gunHjDUwUMiHUdCrapHtm6K4Rw5DHcuNjuBM3LhRx6xPYiVlo4g8NP19VUV1/lCppyfcfvbHxiT+TU9Mzhb+zc/MLxcWlqk0yI7AiEpWYywgsKhljhSQpvEwNgo4UXkTtg1794gaNlUl8Tp0U6xquYtmSAsihRvFkl4eEt5QfARl5u3GcGA2qu3bUyMMICUItmyFB1v3PK6OoOozWG8WSX/b74qMmGJgSG+i0UXwIm4nINMYkFFhbC/yU6jkYkkJhtxBmFlMQbbjCmrMxaLT1vH94l/9zpMlbiXEvJt6n3ydy0NZ2dOQ6NdC1Ha714G+1WkatnXou4zQjjMXXolamOCW8lyJvSoOCVMcZEEa6v3JxDQYEuawLLoRg+ORRU90sB345ONsq7e0P4phmK2yVrbGAbbM9dshOWYUJdsee2At79e69Z+/Ne/9qHfMGM8vsh7yPTxM6rcQ=</latexit><latexit sha1_base64="telQwAcZbhajrNyKXrQhT697vtQ=">AAACOHicbZDLSitBEIZ71OMlx0vUpZvGcEDhGGZE0I0gunHjDUwUMiHUdCrapHtm6K4Rw5DHcuNjuBM3LhRx6xPYiVlo4g8NP19VUV1/lCppyfcfvbHxiT+TU9Mzhb+zc/MLxcWlqk0yI7AiEpWYywgsKhljhSQpvEwNgo4UXkTtg1794gaNlUl8Tp0U6xquYtmSAsihRvFkl4eEt5QfARl5u3GcGA2qu3bUyMMICUItmyFB1v3PK6OoOozWG8WSX/b74qMmGJgSG+i0UXwIm4nINMYkFFhbC/yU6jkYkkJhtxBmFlMQbbjCmrMxaLT1vH94l/9zpMlbiXEvJt6n3ydy0NZ2dOQ6NdC1Ha714G+1WkatnXou4zQjjMXXolamOCW8lyJvSoOCVMcZEEa6v3JxDQYEuawLLoRg+ORRU90sB345ONsq7e0P4phmK2yVrbGAbbM9dshOWYUJdsee2At79e69Z+/Ne/9qHfMGM8vsh7yPTxM6rcQ=</latexit>

• Weights incident on a unit:

Page 27: Horseshoe Priors for Bayesian Neural Networks...Horseshoe Priors for Bayesian Neural Networks Soumya Ghosh Jiayu Yao Finale Doshi-Velez IBM Research MIT-IBM Watson AI lab Harvard Harvard

Both using regularized Horseshoe priors

Synthetic Data: Better Fits

�1 0 1

�2.5

0.0

2.5

20Tra

inin

gPoi

nts

Factorized; 1000 Node

�1 0 1

Structured; 1000 Node

�1 0 1

�2.5

0.0

2.5

100

Tra

inin

gPoi

nts

�1 0 1

�1 0 1

�2.5

0.0

2.5

200

Tra

inin

gPoi

nts

�1 0 1

!27

1000 Unit reg HS, structured VI

Page 28: Horseshoe Priors for Bayesian Neural Networks...Horseshoe Priors for Bayesian Neural Networks Soumya Ghosh Jiayu Yao Finale Doshi-Velez IBM Research MIT-IBM Watson AI lab Harvard Harvard

Structured vs Factorized

!28

Page 29: Horseshoe Priors for Bayesian Neural Networks...Horseshoe Priors for Bayesian Neural Networks Soumya Ghosh Jiayu Yao Finale Doshi-Velez IBM Research MIT-IBM Watson AI lab Harvard Harvard

Structured vs Factorized

!29

Five hundred training points

Page 30: Horseshoe Priors for Bayesian Neural Networks...Horseshoe Priors for Bayesian Neural Networks Soumya Ghosh Jiayu Yao Finale Doshi-Velez IBM Research MIT-IBM Watson AI lab Harvard Harvard

UCI Regression Tasks• Structured variational approximation -> stronger shrinkage, similar predictive

performance

!30

• Predictive Performance:

Pruning rule uses the variational posterior

Comparisons with Variational matrix Gaussian (Louizos & Welling, ICML 2016) q(�klvl < �) > p0

<latexit sha1_base64="O9zzJP9sYJ/Idv2rFhVgLFAVE9A=">AAACB3icbVDLSsNAFJ3UV62vqEtBBotQNyURQRciRTcuK9gHNCFMJtN26OThzE2hhO7c+CtuXCji1l9w5984bbPQ1gMXDufcy733+IngCizr2ygsLa+srhXXSxubW9s75u5eU8WppKxBYxHLtk8UEzxiDeAgWDuRjIS+YC1/cDPxW0MmFY+jexglzA1JL+JdTgloyTMPHyoOkNTLBmI89AS+xE7ABJATfIUTz/LMslW1psCLxM5JGeWoe+aXE8Q0DVkEVBClOraVgJsRCZwKNi45qWIJoQPSYx1NIxIy5WbTP8b4WCsB7sZSVwR4qv6eyEio1Cj0dWdIoK/mvYn4n9dJoXvhZjxKUmARnS3qpgJDjCeh4IBLRkGMNCFUcn0rpn0iCQUdXUmHYM+/vEiap1Xbqtp3Z+XadR5HER2gI1RBNjpHNXSL6qiBKHpEz+gVvRlPxovxbnzMWgtGPrOP/sD4/AH1h5gN</latexit><latexit sha1_base64="O9zzJP9sYJ/Idv2rFhVgLFAVE9A=">AAACB3icbVDLSsNAFJ3UV62vqEtBBotQNyURQRciRTcuK9gHNCFMJtN26OThzE2hhO7c+CtuXCji1l9w5984bbPQ1gMXDufcy733+IngCizr2ygsLa+srhXXSxubW9s75u5eU8WppKxBYxHLtk8UEzxiDeAgWDuRjIS+YC1/cDPxW0MmFY+jexglzA1JL+JdTgloyTMPHyoOkNTLBmI89AS+xE7ABJATfIUTz/LMslW1psCLxM5JGeWoe+aXE8Q0DVkEVBClOraVgJsRCZwKNi45qWIJoQPSYx1NIxIy5WbTP8b4WCsB7sZSVwR4qv6eyEio1Cj0dWdIoK/mvYn4n9dJoXvhZjxKUmARnS3qpgJDjCeh4IBLRkGMNCFUcn0rpn0iCQUdXUmHYM+/vEiap1Xbqtp3Z+XadR5HER2gI1RBNjpHNXSL6qiBKHpEz+gVvRlPxovxbnzMWgtGPrOP/sD4/AH1h5gN</latexit><latexit sha1_base64="O9zzJP9sYJ/Idv2rFhVgLFAVE9A=">AAACB3icbVDLSsNAFJ3UV62vqEtBBotQNyURQRciRTcuK9gHNCFMJtN26OThzE2hhO7c+CtuXCji1l9w5984bbPQ1gMXDufcy733+IngCizr2ygsLa+srhXXSxubW9s75u5eU8WppKxBYxHLtk8UEzxiDeAgWDuRjIS+YC1/cDPxW0MmFY+jexglzA1JL+JdTgloyTMPHyoOkNTLBmI89AS+xE7ABJATfIUTz/LMslW1psCLxM5JGeWoe+aXE8Q0DVkEVBClOraVgJsRCZwKNi45qWIJoQPSYx1NIxIy5WbTP8b4WCsB7sZSVwR4qv6eyEio1Cj0dWdIoK/mvYn4n9dJoXvhZjxKUmARnS3qpgJDjCeh4IBLRkGMNCFUcn0rpn0iCQUdXUmHYM+/vEiap1Xbqtp3Z+XadR5HER2gI1RBNjpHNXSL6qiBKHpEz+gVvRlPxovxbnzMWgtGPrOP/sD4/AH1h5gN</latexit>

Page 31: Horseshoe Priors for Bayesian Neural Networks...Horseshoe Priors for Bayesian Neural Networks Soumya Ghosh Jiayu Yao Finale Doshi-Velez IBM Research MIT-IBM Watson AI lab Harvard Harvard

Summary

• (Regularized) Horseshoe Priors for BNNs can assist with model-selection − Recover small networks with similar performance to larger networks.

• Careful modeling of posterior structure between weights and scales is essential for reliable shrinkage.

!31

Page 32: Horseshoe Priors for Bayesian Neural Networks...Horseshoe Priors for Bayesian Neural Networks Soumya Ghosh Jiayu Yao Finale Doshi-Velez IBM Research MIT-IBM Watson AI lab Harvard Harvard

We are hiring!

!32

http://mitibmwatsonailab.mit.edu/careers/ http://www.research.ibm.com/labs/cambridge/

75 Binney Street, Cambridge, MA 02142


Recommended