Post on 16-Oct-2021
transcript
Neural Networks Part I
The Multilayer Perceptron
Neural Networks History Lesson
2
1962: Rosenblatt, Principles of Neurodynamics: Perceptrons and the Theory of Brain Mechanisms
o First neuron-based learning algorithm
o Allegedly “could learn anything that you could program”
Neural Networks History Lesson
3
1962: Rosenblatt, Principles of Neurodynamics: Perceptrons and the Theory of Brain Mechanisms
o First neuron-based learning algorithm
o Allegedly “could learn anything that you could program”
1969: Minsky & Papert, Perceptron: An Introduction to Computational Geometry
o First real complexity analysis
o Showed, in principle, many things that perceptrons can't learn to do
o Shut down any interest in neural networks
Neural Networks History Lesson
4
1986: Rumelhart, Hinton & Williams, Back Propagation
o Overcame many difficulties raised by Minsky, et al
o Neural Networks wildly popular again (for a while)
Neural Networks History Lesson
5
1999-2005:
Shift to Bayesian Methods: o Best ideas from neural networks
o Direct statistical computing
Support Vector Machines:o Nice mathematical properties
o Global optima vs local optima (like NNs)
Neural Networks History Lesson
6
1999-2005:
Shift to Bayesian Methods: o Best ideas from neural networks
o Direct statistical computing
Support Vector Machines:o Nice mathematical properties
o Global optima vs local optima (like NNs)
A few people still playing with NNs: o Hinton (Univ. Toronto and Google)
o LeCun (NYU and Facebook)
o Bengio (Univ. Montreal)
Neural Networks History Lesson
7
2005-2010:
o Core group of people continue to make improvements
o Developed various tricks to make NN learning practical
2010-Present:o Went back to the NN methods of the 1980s
o Have been wildly successful
Neural Networks History Lesson
8
2005-2010:
o Core group of people continue to make improvements
o Developed various tricks to make NN learning practical
2010-Present:o Went back to the NN methods of the 1980s
o Have been wildly successful
Question for You: Why? What made a 1980s algorithm suddenly amazing 30 years later?
Neural Networks History Lesson
9
Question for You: Why? What made a 1980s algorithm suddenly amazing 30 years later?
Efficient algorithms (Back Propagation)
Raw computing power
Massive amounts of training data:o Deep Neural Nets in particular have a massive amount of parameters
o Need tons of data to effectively train accurate models
Neural Networks History Lesson
10
The history of ML has been very cyclic
But will they be 20 years from now?
¯\_(�)_/¯
The Perceptron
11
In its simplest form, a perceptron takes some binary inputs, and predicts a binary output
Given some training data, learn weights associated with edges that make predictions accurate
o →
2→ Go , I }
-3( →
The Perceptron
12
In its simplest form, a perceptron takes some binary inputs, and predicts a binary output
y =
⇢0 if wTx threshold1 if wTx > threshold
<latexit sha1_base64="pLSlOG7zC6X4LedQAUAnUMB7WHg=">AAACnnichVFNb9QwEHXCR0v46AJHOBi2IE6rpJeWA6iCA5VoRZG6dKX1snKcycaq4wR7Ao2s/C1+SG/9N/VuIwEtiJEsPb03b2Y8k9ZKWozj8yC8cfPW7bX1O9Hde/cfbAwePvpiq8YIGItKVWaScgtKahijRAWT2gAvUwXH6cn7pX78HYyVlT7CtoZZyRda5lJw9NR88HMzYgVH13b0DWUKcmQuYikspHbcGN52zpguiulLyhBO0ZSOypx21LE0pz+6r0crcNotzd9+5WBhwBaVyrzAouT/9rf/8jLQWT9KxIxcFDiKNueDYTyKV0Gvg6QHQ9LH4XxwxrJKNCVoFIpbO03iGme+LkqhwFduLNRcnPAFTD3UvAQ7c6v9dvSFZzKaV8Y/jXTF/u5wvLS2LVOfWXIs7FVtSf5NmzaY78yc1HWDoMVlo7xRFCu6PBbNpAGBqvWACyP9rFQU3HCB/qSRX0Jy9cvXwXhr9HoUf94a7m7321gnT8hz8ookZJvskj1ySMZEBE+Dd8HHYD98Fn4ID8JPl6lh0Hsekz8inFwA8ZHLsg==</latexit><latexit sha1_base64="pLSlOG7zC6X4LedQAUAnUMB7WHg=">AAACnnichVFNb9QwEHXCR0v46AJHOBi2IE6rpJeWA6iCA5VoRZG6dKX1snKcycaq4wR7Ao2s/C1+SG/9N/VuIwEtiJEsPb03b2Y8k9ZKWozj8yC8cfPW7bX1O9Hde/cfbAwePvpiq8YIGItKVWaScgtKahijRAWT2gAvUwXH6cn7pX78HYyVlT7CtoZZyRda5lJw9NR88HMzYgVH13b0DWUKcmQuYikspHbcGN52zpguiulLyhBO0ZSOypx21LE0pz+6r0crcNotzd9+5WBhwBaVyrzAouT/9rf/8jLQWT9KxIxcFDiKNueDYTyKV0Gvg6QHQ9LH4XxwxrJKNCVoFIpbO03iGme+LkqhwFduLNRcnPAFTD3UvAQ7c6v9dvSFZzKaV8Y/jXTF/u5wvLS2LVOfWXIs7FVtSf5NmzaY78yc1HWDoMVlo7xRFCu6PBbNpAGBqvWACyP9rFQU3HCB/qSRX0Jy9cvXwXhr9HoUf94a7m7321gnT8hz8ookZJvskj1ySMZEBE+Dd8HHYD98Fn4ID8JPl6lh0Hsekz8inFwA8ZHLsg==</latexit><latexit sha1_base64="pLSlOG7zC6X4LedQAUAnUMB7WHg=">AAACnnichVFNb9QwEHXCR0v46AJHOBi2IE6rpJeWA6iCA5VoRZG6dKX1snKcycaq4wR7Ao2s/C1+SG/9N/VuIwEtiJEsPb03b2Y8k9ZKWozj8yC8cfPW7bX1O9Hde/cfbAwePvpiq8YIGItKVWaScgtKahijRAWT2gAvUwXH6cn7pX78HYyVlT7CtoZZyRda5lJw9NR88HMzYgVH13b0DWUKcmQuYikspHbcGN52zpguiulLyhBO0ZSOypx21LE0pz+6r0crcNotzd9+5WBhwBaVyrzAouT/9rf/8jLQWT9KxIxcFDiKNueDYTyKV0Gvg6QHQ9LH4XxwxrJKNCVoFIpbO03iGme+LkqhwFduLNRcnPAFTD3UvAQ7c6v9dvSFZzKaV8Y/jXTF/u5wvLS2LVOfWXIs7FVtSf5NmzaY78yc1HWDoMVlo7xRFCu6PBbNpAGBqvWACyP9rFQU3HCB/qSRX0Jy9cvXwXhr9HoUf94a7m7321gnT8hz8ookZJvskj1ySMZEBE+Dd8HHYD98Fn4ID8JPl6lh0Hsekz8inFwA8ZHLsg==</latexit>
WTX = THRESH
WTX - THRESH = o
o→ -
b
¥s y
= 0I →
The Perceptron
13
In its simplest form, a perceptron takes some binary inputs, and predicts a binary output
y =
⇢0 if wTx+ b 01 if wTx+ b > 0
<latexit sha1_base64="d3Pw5q+8jIlGM4xpzrnffbc+yco=">AAACgXicjVFNb9QwEHVCgRK+FjhyGbEtQiAFZy8FIVAlLhyL1KWV1svK8U52rTpOsCfQKMoP4W9x49eAd5sDtBwYydLTe/PlN3lttCfOf0bxtZ3rN27u3kpu37l77/7owcNPvmqcwqmqTOVOc+nRaItT0mTwtHYoy9zgSX72fqOffEXndWWPqa1xXsqV1YVWkgK1GH3fS8RaUtf28BaEwYJEl4gcV9p20jnZ9p1zfcLhKQjCc3JlB7qAHjqRF/Ct/3y8Bec9vIAcNh2+AAchIMn+r+Qd8ESgXQ7TEuH0ak1psrcYjXnKtwFXQTaAMRviaDH6IZaVakq0pIz0fpbxmuahL2llMHRuPNZSnckVzgK0skQ/77YW9rAfmCUUlQvPEmzZPys6WXrflnnILCWt/WVtQ/5LmzVUvJp32tYNoVUXg4rGAFWwuQcstUNFpg1AKqfDrqDW0klF4WpJMCG7/OWrYDpJX6f842R8eDC4scsesyfsGcvYATtkH9gRmzLFfkX7URq9jHfi5zGPJxepcTTUPGJ/RfzmN2SzvSc=</latexit><latexit sha1_base64="d3Pw5q+8jIlGM4xpzrnffbc+yco=">AAACgXicjVFNb9QwEHVCgRK+FjhyGbEtQiAFZy8FIVAlLhyL1KWV1svK8U52rTpOsCfQKMoP4W9x49eAd5sDtBwYydLTe/PlN3lttCfOf0bxtZ3rN27u3kpu37l77/7owcNPvmqcwqmqTOVOc+nRaItT0mTwtHYoy9zgSX72fqOffEXndWWPqa1xXsqV1YVWkgK1GH3fS8RaUtf28BaEwYJEl4gcV9p20jnZ9p1zfcLhKQjCc3JlB7qAHjqRF/Ct/3y8Bec9vIAcNh2+AAchIMn+r+Qd8ESgXQ7TEuH0ak1psrcYjXnKtwFXQTaAMRviaDH6IZaVakq0pIz0fpbxmuahL2llMHRuPNZSnckVzgK0skQ/77YW9rAfmCUUlQvPEmzZPys6WXrflnnILCWt/WVtQ/5LmzVUvJp32tYNoVUXg4rGAFWwuQcstUNFpg1AKqfDrqDW0klF4WpJMCG7/OWrYDpJX6f842R8eDC4scsesyfsGcvYATtkH9gRmzLFfkX7URq9jHfi5zGPJxepcTTUPGJ/RfzmN2SzvSc=</latexit><latexit sha1_base64="d3Pw5q+8jIlGM4xpzrnffbc+yco=">AAACgXicjVFNb9QwEHVCgRK+FjhyGbEtQiAFZy8FIVAlLhyL1KWV1svK8U52rTpOsCfQKMoP4W9x49eAd5sDtBwYydLTe/PlN3lttCfOf0bxtZ3rN27u3kpu37l77/7owcNPvmqcwqmqTOVOc+nRaItT0mTwtHYoy9zgSX72fqOffEXndWWPqa1xXsqV1YVWkgK1GH3fS8RaUtf28BaEwYJEl4gcV9p20jnZ9p1zfcLhKQjCc3JlB7qAHjqRF/Ct/3y8Bec9vIAcNh2+AAchIMn+r+Qd8ESgXQ7TEuH0ak1psrcYjXnKtwFXQTaAMRviaDH6IZaVakq0pIz0fpbxmuahL2llMHRuPNZSnckVzgK0skQ/77YW9rAfmCUUlQvPEmzZPys6WXrflnnILCWt/WVtQ/5LmzVUvJp32tYNoVUXg4rGAFWwuQcstUNFpg1AKqfDrqDW0klF4WpJMCG7/OWrYDpJX6f842R8eDC4scsesyfsGcvYATtkH9gRmzLFfkX7URq9jHfi5zGPJxepcTTUPGJ/RfzmN2SzvSc=</latexit>
ob
: v
The Perceptron
14
Simple Example: Learning ORy =
⇢0 if wTx+ b 01 if wTx+ b > 0
<latexit sha1_base64="d3Pw5q+8jIlGM4xpzrnffbc+yco=">AAACgXicjVFNb9QwEHVCgRK+FjhyGbEtQiAFZy8FIVAlLhyL1KWV1svK8U52rTpOsCfQKMoP4W9x49eAd5sDtBwYydLTe/PlN3lttCfOf0bxtZ3rN27u3kpu37l77/7owcNPvmqcwqmqTOVOc+nRaItT0mTwtHYoy9zgSX72fqOffEXndWWPqa1xXsqV1YVWkgK1GH3fS8RaUtf28BaEwYJEl4gcV9p20jnZ9p1zfcLhKQjCc3JlB7qAHjqRF/Ct/3y8Bec9vIAcNh2+AAchIMn+r+Qd8ESgXQ7TEuH0ak1psrcYjXnKtwFXQTaAMRviaDH6IZaVakq0pIz0fpbxmuahL2llMHRuPNZSnckVzgK0skQ/77YW9rAfmCUUlQvPEmzZPys6WXrflnnILCWt/WVtQ/5LmzVUvJp32tYNoVUXg4rGAFWwuQcstUNFpg1AKqfDrqDW0klF4WpJMCG7/OWrYDpJX6f842R8eDC4scsesyfsGcvYATtkH9gRmzLFfkX7URq9jHfi5zGPJxepcTTUPGJ/RfzmN2SzvSc=</latexit><latexit sha1_base64="d3Pw5q+8jIlGM4xpzrnffbc+yco=">AAACgXicjVFNb9QwEHVCgRK+FjhyGbEtQiAFZy8FIVAlLhyL1KWV1svK8U52rTpOsCfQKMoP4W9x49eAd5sDtBwYydLTe/PlN3lttCfOf0bxtZ3rN27u3kpu37l77/7owcNPvmqcwqmqTOVOc+nRaItT0mTwtHYoy9zgSX72fqOffEXndWWPqa1xXsqV1YVWkgK1GH3fS8RaUtf28BaEwYJEl4gcV9p20jnZ9p1zfcLhKQjCc3JlB7qAHjqRF/Ct/3y8Bec9vIAcNh2+AAchIMn+r+Qd8ESgXQ7TEuH0ak1psrcYjXnKtwFXQTaAMRviaDH6IZaVakq0pIz0fpbxmuahL2llMHRuPNZSnckVzgK0skQ/77YW9rAfmCUUlQvPEmzZPys6WXrflnnILCWt/WVtQ/5LmzVUvJp32tYNoVUXg4rGAFWwuQcstUNFpg1AKqfDrqDW0klF4WpJMCG7/OWrYDpJX6f842R8eDC4scsesyfsGcvYATtkH9gRmzLFfkX7URq9jHfi5zGPJxepcTTUPGJ/RfzmN2SzvSc=</latexit><latexit sha1_base64="d3Pw5q+8jIlGM4xpzrnffbc+yco=">AAACgXicjVFNb9QwEHVCgRK+FjhyGbEtQiAFZy8FIVAlLhyL1KWV1svK8U52rTpOsCfQKMoP4W9x49eAd5sDtBwYydLTe/PlN3lttCfOf0bxtZ3rN27u3kpu37l77/7owcNPvmqcwqmqTOVOc+nRaItT0mTwtHYoy9zgSX72fqOffEXndWWPqa1xXsqV1YVWkgK1GH3fS8RaUtf28BaEwYJEl4gcV9p20jnZ9p1zfcLhKQjCc3JlB7qAHjqRF/Ct/3y8Bec9vIAcNh2+AAchIMn+r+Qd8ESgXQ7TEuH0ak1psrcYjXnKtwFXQTaAMRviaDH6IZaVakq0pIz0fpbxmuahL2llMHRuPNZSnckVzgK0skQ/77YW9rAfmCUUlQvPEmzZPys6WXrflnnILCWt/WVtQ/5LmzVUvJp32tYNoVUXg4rGAFWwuQcstUNFpg1AKqfDrqDW0klF4WpJMCG7/OWrYDpJX6f842R8eDC4scsesyfsGcvYATtkH9gRmzLFfkX7URq9jHfi5zGPJxepcTTUPGJ/RfzmN2SzvSc=</latexit>
x1 0 1 0 1x2 0 0 1 1
x1 OR x2 0 1 1 1<latexit sha1_base64="UM8xfazAVtlDjQbsqgXmCAzAUIo=">AAACd3icbVHJTsMwEHXCHrYCx14sWhCnKumF5YTEhRuLKCA1VeW409bCcSJ7gqhCvoF/48aHcOGE2wYBhZFGfn5vxjOeiVIpDPr+m+POzS8sLi2veKtr6xubla3tW5NkmkOLJzLR9xEzIIWCFgqUcJ9qYHEk4S56OBvrd4+gjUjUDY5S6MRsoERfcIaW6lZevHrdCyMYCJUzrdmoyPkzt1Z4T92A7lPf+vcZhl44HNeiVm6W9FfIjGyvCE+ImNOLa1rQ74TgVwKoXlna9tKt1PyGPzH6FwQlqJHSLruV17CX8CwGhVwyY9qBn2LHPoiCSyi8MDOQMv7ABtC2ULEYTCefDK6ge5bp0X6irSukE/ZnRs5iY0ZxZCNjhkMzq43J/7R2hv2jTi5UmiEoPi3UzyTFhI63QHtCA0c5soBxLWyvlA+ZZhztrjw7hGD2y39Bq9k4bvhXzdrpSTmNZVIlu+SABOSQnJJzcklahJN3p+rUnT3nw6XuvnswDXWdMmeH/DI3+ATd3bFz</latexit><latexit sha1_base64="UM8xfazAVtlDjQbsqgXmCAzAUIo=">AAACd3icbVHJTsMwEHXCHrYCx14sWhCnKumF5YTEhRuLKCA1VeW409bCcSJ7gqhCvoF/48aHcOGE2wYBhZFGfn5vxjOeiVIpDPr+m+POzS8sLi2veKtr6xubla3tW5NkmkOLJzLR9xEzIIWCFgqUcJ9qYHEk4S56OBvrd4+gjUjUDY5S6MRsoERfcIaW6lZevHrdCyMYCJUzrdmoyPkzt1Z4T92A7lPf+vcZhl44HNeiVm6W9FfIjGyvCE+ImNOLa1rQ74TgVwKoXlna9tKt1PyGPzH6FwQlqJHSLruV17CX8CwGhVwyY9qBn2LHPoiCSyi8MDOQMv7ABtC2ULEYTCefDK6ge5bp0X6irSukE/ZnRs5iY0ZxZCNjhkMzq43J/7R2hv2jTi5UmiEoPi3UzyTFhI63QHtCA0c5soBxLWyvlA+ZZhztrjw7hGD2y39Bq9k4bvhXzdrpSTmNZVIlu+SABOSQnJJzcklahJN3p+rUnT3nw6XuvnswDXWdMmeH/DI3+ATd3bFz</latexit><latexit sha1_base64="UM8xfazAVtlDjQbsqgXmCAzAUIo=">AAACd3icbVHJTsMwEHXCHrYCx14sWhCnKumF5YTEhRuLKCA1VeW409bCcSJ7gqhCvoF/48aHcOGE2wYBhZFGfn5vxjOeiVIpDPr+m+POzS8sLi2veKtr6xubla3tW5NkmkOLJzLR9xEzIIWCFgqUcJ9qYHEk4S56OBvrd4+gjUjUDY5S6MRsoERfcIaW6lZevHrdCyMYCJUzrdmoyPkzt1Z4T92A7lPf+vcZhl44HNeiVm6W9FfIjGyvCE+ImNOLa1rQ74TgVwKoXlna9tKt1PyGPzH6FwQlqJHSLruV17CX8CwGhVwyY9qBn2LHPoiCSyi8MDOQMv7ABtC2ULEYTCefDK6ge5bp0X6irSukE/ZnRs5iY0ZxZCNjhkMzq43J/7R2hv2jTi5UmiEoPi3UzyTFhI63QHtCA0c5soBxLWyvlA+ZZhztrjw7hGD2y39Bq9k4bvhXzdrpSTmNZVIlu+SABOSQnJJzcklahJN3p+rUnT3nw6XuvnswDXWdMmeH/DI3+ATd3bFz</latexit>
ow=f, }b=o
I
1 wt⇒ly1t,b=t'1511,1+0--2>0
The Perceptron
15
Simple Example: Learning ANDy =
⇢0 if wTx+ b 01 if wTx+ b > 0
<latexit sha1_base64="d3Pw5q+8jIlGM4xpzrnffbc+yco=">AAACgXicjVFNb9QwEHVCgRK+FjhyGbEtQiAFZy8FIVAlLhyL1KWV1svK8U52rTpOsCfQKMoP4W9x49eAd5sDtBwYydLTe/PlN3lttCfOf0bxtZ3rN27u3kpu37l77/7owcNPvmqcwqmqTOVOc+nRaItT0mTwtHYoy9zgSX72fqOffEXndWWPqa1xXsqV1YVWkgK1GH3fS8RaUtf28BaEwYJEl4gcV9p20jnZ9p1zfcLhKQjCc3JlB7qAHjqRF/Ct/3y8Bec9vIAcNh2+AAchIMn+r+Qd8ESgXQ7TEuH0ak1psrcYjXnKtwFXQTaAMRviaDH6IZaVakq0pIz0fpbxmuahL2llMHRuPNZSnckVzgK0skQ/77YW9rAfmCUUlQvPEmzZPys6WXrflnnILCWt/WVtQ/5LmzVUvJp32tYNoVUXg4rGAFWwuQcstUNFpg1AKqfDrqDW0klF4WpJMCG7/OWrYDpJX6f842R8eDC4scsesyfsGcvYATtkH9gRmzLFfkX7URq9jHfi5zGPJxepcTTUPGJ/RfzmN2SzvSc=</latexit><latexit sha1_base64="d3Pw5q+8jIlGM4xpzrnffbc+yco=">AAACgXicjVFNb9QwEHVCgRK+FjhyGbEtQiAFZy8FIVAlLhyL1KWV1svK8U52rTpOsCfQKMoP4W9x49eAd5sDtBwYydLTe/PlN3lttCfOf0bxtZ3rN27u3kpu37l77/7owcNPvmqcwqmqTOVOc+nRaItT0mTwtHYoy9zgSX72fqOffEXndWWPqa1xXsqV1YVWkgK1GH3fS8RaUtf28BaEwYJEl4gcV9p20jnZ9p1zfcLhKQjCc3JlB7qAHjqRF/Ct/3y8Bec9vIAcNh2+AAchIMn+r+Qd8ESgXQ7TEuH0ak1psrcYjXnKtwFXQTaAMRviaDH6IZaVakq0pIz0fpbxmuahL2llMHRuPNZSnckVzgK0skQ/77YW9rAfmCUUlQvPEmzZPys6WXrflnnILCWt/WVtQ/5LmzVUvJp32tYNoVUXg4rGAFWwuQcstUNFpg1AKqfDrqDW0klF4WpJMCG7/OWrYDpJX6f842R8eDC4scsesyfsGcvYATtkH9gRmzLFfkX7URq9jHfi5zGPJxepcTTUPGJ/RfzmN2SzvSc=</latexit><latexit sha1_base64="d3Pw5q+8jIlGM4xpzrnffbc+yco=">AAACgXicjVFNb9QwEHVCgRK+FjhyGbEtQiAFZy8FIVAlLhyL1KWV1svK8U52rTpOsCfQKMoP4W9x49eAd5sDtBwYydLTe/PlN3lttCfOf0bxtZ3rN27u3kpu37l77/7owcNPvmqcwqmqTOVOc+nRaItT0mTwtHYoy9zgSX72fqOffEXndWWPqa1xXsqV1YVWkgK1GH3fS8RaUtf28BaEwYJEl4gcV9p20jnZ9p1zfcLhKQjCc3JlB7qAHjqRF/Ct/3y8Bec9vIAcNh2+AAchIMn+r+Qd8ESgXQ7TEuH0ak1psrcYjXnKtwFXQTaAMRviaDH6IZaVakq0pIz0fpbxmuahL2llMHRuPNZSnckVzgK0skQ/77YW9rAfmCUUlQvPEmzZPys6WXrflnnILCWt/WVtQ/5LmzVUvJp32tYNoVUXg4rGAFWwuQcstUNFpg1AKqfDrqDW0klF4WpJMCG7/OWrYDpJX6f842R8eDC4scsesyfsGcvYATtkH9gRmzLFfkX7URq9jHfi5zGPJxepcTTUPGJ/RfzmN2SzvSc=</latexit>
x1 0 1 0 1x2 0 0 1 1
x1 AND x2 0 0 0 1<latexit sha1_base64="jXeA1icTPY9SMyKMFtVfAhfiWlw=">AAACd3icbVFNT9tAEF2bFqihEOCYy6pxopwiOxc+TiB66KlKpaYgxVG03kySVdZra3eMEhn/Bv4bN35ILz2xCa4ooSON9u1787E7E2dSGAyCJ8fd+vBxe2f3k7e3//ngsHZ0/MukuebQ56lM9W3MDEihoI8CJdxmGlgSS7iJ59cr/eYOtBGp+onLDIYJmyoxEZyhpUa1B9/3ohimQhVMa7YsC37PrZXeYhTSFg2sv55R5EWzVStq5W5F/w3ZkO0VYYGIBb36/pWW9G3Ga0FQ46q35/ujWiPoBGuj70FYgQaprDeqPUbjlOcJKOSSGTMIgwyHtiAKLqH0otxAxvicTWFgoWIJmGGxHlxJm5YZ00mqrSuka/bfjIIlxiyT2EYmDGdmU1uR/9MGOU7OhoVQWY6g+EujSS4ppnS1BToWGjjKpQWMa2HfSvmMacbR7sqzQwg3v/we9Lud807wo9u4vKimsUvq5Atpk5CckkvyjfRIn3Dy26k7vtN0/rjUbbntl1DXqXJOyBtzw2c1q7GZ</latexit><latexit sha1_base64="jXeA1icTPY9SMyKMFtVfAhfiWlw=">AAACd3icbVFNT9tAEF2bFqihEOCYy6pxopwiOxc+TiB66KlKpaYgxVG03kySVdZra3eMEhn/Bv4bN35ILz2xCa4ooSON9u1787E7E2dSGAyCJ8fd+vBxe2f3k7e3//ngsHZ0/MukuebQ56lM9W3MDEihoI8CJdxmGlgSS7iJ59cr/eYOtBGp+onLDIYJmyoxEZyhpUa1B9/3ohimQhVMa7YsC37PrZXeYhTSFg2sv55R5EWzVStq5W5F/w3ZkO0VYYGIBb36/pWW9G3Ga0FQ46q35/ujWiPoBGuj70FYgQaprDeqPUbjlOcJKOSSGTMIgwyHtiAKLqH0otxAxvicTWFgoWIJmGGxHlxJm5YZ00mqrSuka/bfjIIlxiyT2EYmDGdmU1uR/9MGOU7OhoVQWY6g+EujSS4ppnS1BToWGjjKpQWMa2HfSvmMacbR7sqzQwg3v/we9Lud807wo9u4vKimsUvq5Atpk5CckkvyjfRIn3Dy26k7vtN0/rjUbbntl1DXqXJOyBtzw2c1q7GZ</latexit><latexit sha1_base64="jXeA1icTPY9SMyKMFtVfAhfiWlw=">AAACd3icbVFNT9tAEF2bFqihEOCYy6pxopwiOxc+TiB66KlKpaYgxVG03kySVdZra3eMEhn/Bv4bN35ILz2xCa4ooSON9u1787E7E2dSGAyCJ8fd+vBxe2f3k7e3//ngsHZ0/MukuebQ56lM9W3MDEihoI8CJdxmGlgSS7iJ59cr/eYOtBGp+onLDIYJmyoxEZyhpUa1B9/3ohimQhVMa7YsC37PrZXeYhTSFg2sv55R5EWzVStq5W5F/w3ZkO0VYYGIBb36/pWW9G3Ga0FQ46q35/ujWiPoBGuj70FYgQaprDeqPUbjlOcJKOSSGTMIgwyHtiAKLqH0otxAxvicTWFgoWIJmGGxHlxJm5YZ00mqrSuka/bfjIIlxiyT2EYmDGdmU1uR/9MGOU7OhoVQWY6g+EujSS4ppnS1BToWGjjKpQWMa2HfSvmMacbR7sqzQwg3v/we9Lud807wo9u4vKimsUvq5Atpk5CckkvyjfRIn3Dy26k7vtN0/rjUbbntl1DXqXJOyBtzw2c1q7GZ</latexit>: w=[1,1 & b= -1
The Perceptron
16
Simple Example: Learning NANDy =
⇢0 if wTx+ b 01 if wTx+ b > 0
<latexit sha1_base64="d3Pw5q+8jIlGM4xpzrnffbc+yco=">AAACgXicjVFNb9QwEHVCgRK+FjhyGbEtQiAFZy8FIVAlLhyL1KWV1svK8U52rTpOsCfQKMoP4W9x49eAd5sDtBwYydLTe/PlN3lttCfOf0bxtZ3rN27u3kpu37l77/7owcNPvmqcwqmqTOVOc+nRaItT0mTwtHYoy9zgSX72fqOffEXndWWPqa1xXsqV1YVWkgK1GH3fS8RaUtf28BaEwYJEl4gcV9p20jnZ9p1zfcLhKQjCc3JlB7qAHjqRF/Ct/3y8Bec9vIAcNh2+AAchIMn+r+Qd8ESgXQ7TEuH0ak1psrcYjXnKtwFXQTaAMRviaDH6IZaVakq0pIz0fpbxmuahL2llMHRuPNZSnckVzgK0skQ/77YW9rAfmCUUlQvPEmzZPys6WXrflnnILCWt/WVtQ/5LmzVUvJp32tYNoVUXg4rGAFWwuQcstUNFpg1AKqfDrqDW0klF4WpJMCG7/OWrYDpJX6f842R8eDC4scsesyfsGcvYATtkH9gRmzLFfkX7URq9jHfi5zGPJxepcTTUPGJ/RfzmN2SzvSc=</latexit><latexit sha1_base64="d3Pw5q+8jIlGM4xpzrnffbc+yco=">AAACgXicjVFNb9QwEHVCgRK+FjhyGbEtQiAFZy8FIVAlLhyL1KWV1svK8U52rTpOsCfQKMoP4W9x49eAd5sDtBwYydLTe/PlN3lttCfOf0bxtZ3rN27u3kpu37l77/7owcNPvmqcwqmqTOVOc+nRaItT0mTwtHYoy9zgSX72fqOffEXndWWPqa1xXsqV1YVWkgK1GH3fS8RaUtf28BaEwYJEl4gcV9p20jnZ9p1zfcLhKQjCc3JlB7qAHjqRF/Ct/3y8Bec9vIAcNh2+AAchIMn+r+Qd8ESgXQ7TEuH0ak1psrcYjXnKtwFXQTaAMRviaDH6IZaVakq0pIz0fpbxmuahL2llMHRuPNZSnckVzgK0skQ/77YW9rAfmCUUlQvPEmzZPys6WXrflnnILCWt/WVtQ/5LmzVUvJp32tYNoVUXg4rGAFWwuQcstUNFpg1AKqfDrqDW0klF4WpJMCG7/OWrYDpJX6f842R8eDC4scsesyfsGcvYATtkH9gRmzLFfkX7URq9jHfi5zGPJxepcTTUPGJ/RfzmN2SzvSc=</latexit><latexit sha1_base64="d3Pw5q+8jIlGM4xpzrnffbc+yco=">AAACgXicjVFNb9QwEHVCgRK+FjhyGbEtQiAFZy8FIVAlLhyL1KWV1svK8U52rTpOsCfQKMoP4W9x49eAd5sDtBwYydLTe/PlN3lttCfOf0bxtZ3rN27u3kpu37l77/7owcNPvmqcwqmqTOVOc+nRaItT0mTwtHYoy9zgSX72fqOffEXndWWPqa1xXsqV1YVWkgK1GH3fS8RaUtf28BaEwYJEl4gcV9p20jnZ9p1zfcLhKQjCc3JlB7qAHjqRF/Ct/3y8Bec9vIAcNh2+AAchIMn+r+Qd8ESgXQ7TEuH0ak1psrcYjXnKtwFXQTaAMRviaDH6IZaVakq0pIz0fpbxmuahL2llMHRuPNZSnckVzgK0skQ/77YW9rAfmCUUlQvPEmzZPys6WXrflnnILCWt/WVtQ/5LmzVUvJp32tYNoVUXg4rGAFWwuQcstUNFpg1AKqfDrqDW0klF4WpJMCG7/OWrYDpJX6f842R8eDC4scsesyfsGcvYATtkH9gRmzLFfkX7URq9jHfi5zGPJxepcTTUPGJ/RfzmN2SzvSc=</latexit>
x1 0 1 0 1x2 0 0 1 1
x1 NAND x2 1 1 1 0<latexit sha1_base64="2vKppGvl1XGFf2Q25MWaYD7QWT4=">AAACeXicbVFNT+MwEHUCy0dYoLscEZJFgUUcqqQcluUEggMnBBIFpKaqHHfaWnWcyJ4gqpAfwV/jxi/hwgG3DYItjDTy83sznvFMlEph0PefHXdm9sfc/MKit/RzeWW18uv3tUkyzaHBE5no24gZkEJBAwVKuE01sDiScBMNTkb6zR1oIxJ1hcMUWjHrKdEVnKGl2pVHb2vLCyPoCZUzrdmwyPkDt1Z49+2A7lDf+scZhl7YH9WiVq6X9HvIlGyvCPeImNPz4/NTWtBJSvDJ/XEKqE5Z3HbTrlT9mj82+hUEJaiS0i7alaewk/AsBoVcMmOagZ9iyz6IgksovDAzkDI+YD1oWqhYDKaVj0dX0G3LdGg30dYV0jH7OSNnsTHDOLKRMcO+mdZG5HdaM8PuQSsXKs0QFJ8U6maSYkJHe6AdoYGjHFrAuBa2V8r7TDOOdlueHUIw/eWvoFGv/av5l/Xq0WE5jQWyTjbJLgnIX3JEzsgFaRBOXpwNZ8f547y6m+6uuzcJdZ0yZ438Z+7+Gzi0sgc=</latexit><latexit sha1_base64="2vKppGvl1XGFf2Q25MWaYD7QWT4=">AAACeXicbVFNT+MwEHUCy0dYoLscEZJFgUUcqqQcluUEggMnBBIFpKaqHHfaWnWcyJ4gqpAfwV/jxi/hwgG3DYItjDTy83sznvFMlEph0PefHXdm9sfc/MKit/RzeWW18uv3tUkyzaHBE5no24gZkEJBAwVKuE01sDiScBMNTkb6zR1oIxJ1hcMUWjHrKdEVnKGl2pVHb2vLCyPoCZUzrdmwyPkDt1Z49+2A7lDf+scZhl7YH9WiVq6X9HvIlGyvCPeImNPz4/NTWtBJSvDJ/XEKqE5Z3HbTrlT9mj82+hUEJaiS0i7alaewk/AsBoVcMmOagZ9iyz6IgksovDAzkDI+YD1oWqhYDKaVj0dX0G3LdGg30dYV0jH7OSNnsTHDOLKRMcO+mdZG5HdaM8PuQSsXKs0QFJ8U6maSYkJHe6AdoYGjHFrAuBa2V8r7TDOOdlueHUIw/eWvoFGv/av5l/Xq0WE5jQWyTjbJLgnIX3JEzsgFaRBOXpwNZ8f547y6m+6uuzcJdZ0yZ438Z+7+Gzi0sgc=</latexit><latexit sha1_base64="2vKppGvl1XGFf2Q25MWaYD7QWT4=">AAACeXicbVFNT+MwEHUCy0dYoLscEZJFgUUcqqQcluUEggMnBBIFpKaqHHfaWnWcyJ4gqpAfwV/jxi/hwgG3DYItjDTy83sznvFMlEph0PefHXdm9sfc/MKit/RzeWW18uv3tUkyzaHBE5no24gZkEJBAwVKuE01sDiScBMNTkb6zR1oIxJ1hcMUWjHrKdEVnKGl2pVHb2vLCyPoCZUzrdmwyPkDt1Z49+2A7lDf+scZhl7YH9WiVq6X9HvIlGyvCPeImNPz4/NTWtBJSvDJ/XEKqE5Z3HbTrlT9mj82+hUEJaiS0i7alaewk/AsBoVcMmOagZ9iyz6IgksovDAzkDI+YD1oWqhYDKaVj0dX0G3LdGg30dYV0jH7OSNnsTHDOLKRMcO+mdZG5HdaM8PuQSsXKs0QFJ8U6maSYkJHe6AdoYGjHFrAuBa2V8r7TDOOdlueHUIw/eWvoFGv/av5l/Xq0WE5jQWyTjbJLgnIX3JEzsgFaRBOXpwNZ8f547y6m+6uuzcJdZ0yZ438Z+7+Gzi0sgc=</latexit>I wtxtbs El - 1 3 'Ll, ) t 1.5
The Perceptron
17
Simple Example: Can we learn XOR?y =
⇢0 if wTx+ b 01 if wTx+ b > 0
<latexit sha1_base64="d3Pw5q+8jIlGM4xpzrnffbc+yco=">AAACgXicjVFNb9QwEHVCgRK+FjhyGbEtQiAFZy8FIVAlLhyL1KWV1svK8U52rTpOsCfQKMoP4W9x49eAd5sDtBwYydLTe/PlN3lttCfOf0bxtZ3rN27u3kpu37l77/7owcNPvmqcwqmqTOVOc+nRaItT0mTwtHYoy9zgSX72fqOffEXndWWPqa1xXsqV1YVWkgK1GH3fS8RaUtf28BaEwYJEl4gcV9p20jnZ9p1zfcLhKQjCc3JlB7qAHjqRF/Ct/3y8Bec9vIAcNh2+AAchIMn+r+Qd8ESgXQ7TEuH0ak1psrcYjXnKtwFXQTaAMRviaDH6IZaVakq0pIz0fpbxmuahL2llMHRuPNZSnckVzgK0skQ/77YW9rAfmCUUlQvPEmzZPys6WXrflnnILCWt/WVtQ/5LmzVUvJp32tYNoVUXg4rGAFWwuQcstUNFpg1AKqfDrqDW0klF4WpJMCG7/OWrYDpJX6f842R8eDC4scsesyfsGcvYATtkH9gRmzLFfkX7URq9jHfi5zGPJxepcTTUPGJ/RfzmN2SzvSc=</latexit><latexit sha1_base64="d3Pw5q+8jIlGM4xpzrnffbc+yco=">AAACgXicjVFNb9QwEHVCgRK+FjhyGbEtQiAFZy8FIVAlLhyL1KWV1svK8U52rTpOsCfQKMoP4W9x49eAd5sDtBwYydLTe/PlN3lttCfOf0bxtZ3rN27u3kpu37l77/7owcNPvmqcwqmqTOVOc+nRaItT0mTwtHYoy9zgSX72fqOffEXndWWPqa1xXsqV1YVWkgK1GH3fS8RaUtf28BaEwYJEl4gcV9p20jnZ9p1zfcLhKQjCc3JlB7qAHjqRF/Ct/3y8Bec9vIAcNh2+AAchIMn+r+Qd8ESgXQ7TEuH0ak1psrcYjXnKtwFXQTaAMRviaDH6IZaVakq0pIz0fpbxmuahL2llMHRuPNZSnckVzgK0skQ/77YW9rAfmCUUlQvPEmzZPys6WXrflnnILCWt/WVtQ/5LmzVUvJp32tYNoVUXg4rGAFWwuQcstUNFpg1AKqfDrqDW0klF4WpJMCG7/OWrYDpJX6f842R8eDC4scsesyfsGcvYATtkH9gRmzLFfkX7URq9jHfi5zGPJxepcTTUPGJ/RfzmN2SzvSc=</latexit><latexit sha1_base64="d3Pw5q+8jIlGM4xpzrnffbc+yco=">AAACgXicjVFNb9QwEHVCgRK+FjhyGbEtQiAFZy8FIVAlLhyL1KWV1svK8U52rTpOsCfQKMoP4W9x49eAd5sDtBwYydLTe/PlN3lttCfOf0bxtZ3rN27u3kpu37l77/7owcNPvmqcwqmqTOVOc+nRaItT0mTwtHYoy9zgSX72fqOffEXndWWPqa1xXsqV1YVWkgK1GH3fS8RaUtf28BaEwYJEl4gcV9p20jnZ9p1zfcLhKQjCc3JlB7qAHjqRF/Ct/3y8Bec9vIAcNh2+AAchIMn+r+Qd8ESgXQ7TEuH0ak1psrcYjXnKtwFXQTaAMRviaDH6IZaVakq0pIz0fpbxmuahL2llMHRuPNZSnckVzgK0skQ/77YW9rAfmCUUlQvPEmzZPys6WXrflnnILCWt/WVtQ/5LmzVUvJp32tYNoVUXg4rGAFWwuQcstUNFpg1AKqfDrqDW0klF4WpJMCG7/OWrYDpJX6f842R8eDC4scsesyfsGcvYATtkH9gRmzLFfkX7URq9jHfi5zGPJxepcTTUPGJ/RfzmN2SzvSc=</latexit>
x1 0 1 0 1x2 0 0 1 1
x1 XOR x2 0 1 1 0<latexit sha1_base64="7KwtoelifvaVe1T374Vf/hf8/1A=">AAACd3icbVFLT9tAEF67LQ9DaWiPXFbEQZwiOxcKJ6ReuPFQQyLFUbTeTJIV67W1O0ZExr+h/40bP4QLp66DadqkI4322++bx+5MnElhMAieHffDx08bm1vb3s7u570vjf2vtybNNYcuT2Wq+zEzIIWCLgqU0M80sCSW0IvvflR67x60Ean6ifMMhgmbKjERnKGlRo1fvu9FMUyFKpjWbF4W/JFbK72HUUiPaGB9eUaRF82qVtTKnZp+D1mR7RXhAREL2r+8oSVdZoR/ilYZoMZ1b8/3R41m0A4WRtdBWIMmqe1q1HiKxinPE1DIJTNmEAYZDm1BFFxC6UW5gYzxOzaFgYWKJWCGxWJwJW1ZZkwnqbaukC7YvzMKlhgzT2IbmTCcmVWtIv+nDXKcfB8WQmU5guJvjSa5pJjSagt0LDRwlHMLGNfCvpXyGdOMo92VZ4cQrn55HXQ77dN2cN1pnp/V09giB+SQHJOQnJBzckGuSJdw8uIcOL7Tcl5d6h65x2+hrlPnfCP/mBv+BnbdscA=</latexit><latexit sha1_base64="7KwtoelifvaVe1T374Vf/hf8/1A=">AAACd3icbVFLT9tAEF67LQ9DaWiPXFbEQZwiOxcKJ6ReuPFQQyLFUbTeTJIV67W1O0ZExr+h/40bP4QLp66DadqkI4322++bx+5MnElhMAieHffDx08bm1vb3s7u570vjf2vtybNNYcuT2Wq+zEzIIWCLgqU0M80sCSW0IvvflR67x60Ean6ifMMhgmbKjERnKGlRo1fvu9FMUyFKpjWbF4W/JFbK72HUUiPaGB9eUaRF82qVtTKnZp+D1mR7RXhAREL2r+8oSVdZoR/ilYZoMZ1b8/3R41m0A4WRtdBWIMmqe1q1HiKxinPE1DIJTNmEAYZDm1BFFxC6UW5gYzxOzaFgYWKJWCGxWJwJW1ZZkwnqbaukC7YvzMKlhgzT2IbmTCcmVWtIv+nDXKcfB8WQmU5guJvjSa5pJjSagt0LDRwlHMLGNfCvpXyGdOMo92VZ4cQrn55HXQ77dN2cN1pnp/V09giB+SQHJOQnJBzckGuSJdw8uIcOL7Tcl5d6h65x2+hrlPnfCP/mBv+BnbdscA=</latexit><latexit sha1_base64="7KwtoelifvaVe1T374Vf/hf8/1A=">AAACd3icbVFLT9tAEF67LQ9DaWiPXFbEQZwiOxcKJ6ReuPFQQyLFUbTeTJIV67W1O0ZExr+h/40bP4QLp66DadqkI4322++bx+5MnElhMAieHffDx08bm1vb3s7u570vjf2vtybNNYcuT2Wq+zEzIIWCLgqU0M80sCSW0IvvflR67x60Ean6ifMMhgmbKjERnKGlRo1fvu9FMUyFKpjWbF4W/JFbK72HUUiPaGB9eUaRF82qVtTKnZp+D1mR7RXhAREL2r+8oSVdZoR/ilYZoMZ1b8/3R41m0A4WRtdBWIMmqe1q1HiKxinPE1DIJTNmEAYZDm1BFFxC6UW5gYzxOzaFgYWKJWCGxWJwJW1ZZkwnqbaukC7YvzMKlhgzT2IbmTCcmVWtIv+nDXKcfB8WQmU5guJvjSa5pJjSagt0LDRwlHMLGNfCvpXyGdOMo92VZ4cQrn55HXQ77dN2cN1pnp/V09giB+SQHJOQnJBzckGuSJdw8uIcOL7Tcl5d6h65x2+hrlPnfCP/mBv+BnbdscA=</latexit>
1= =
41 wtx = +1 )
L
Wtxtb EO
The Perceptron
18
Simple Example: Can we learn XOR?
NOPE!
y =
⇢0 if wTx+ b 01 if wTx+ b > 0
<latexit sha1_base64="d3Pw5q+8jIlGM4xpzrnffbc+yco=">AAACgXicjVFNb9QwEHVCgRK+FjhyGbEtQiAFZy8FIVAlLhyL1KWV1svK8U52rTpOsCfQKMoP4W9x49eAd5sDtBwYydLTe/PlN3lttCfOf0bxtZ3rN27u3kpu37l77/7owcNPvmqcwqmqTOVOc+nRaItT0mTwtHYoy9zgSX72fqOffEXndWWPqa1xXsqV1YVWkgK1GH3fS8RaUtf28BaEwYJEl4gcV9p20jnZ9p1zfcLhKQjCc3JlB7qAHjqRF/Ct/3y8Bec9vIAcNh2+AAchIMn+r+Qd8ESgXQ7TEuH0ak1psrcYjXnKtwFXQTaAMRviaDH6IZaVakq0pIz0fpbxmuahL2llMHRuPNZSnckVzgK0skQ/77YW9rAfmCUUlQvPEmzZPys6WXrflnnILCWt/WVtQ/5LmzVUvJp32tYNoVUXg4rGAFWwuQcstUNFpg1AKqfDrqDW0klF4WpJMCG7/OWrYDpJX6f842R8eDC4scsesyfsGcvYATtkH9gRmzLFfkX7URq9jHfi5zGPJxepcTTUPGJ/RfzmN2SzvSc=</latexit><latexit sha1_base64="d3Pw5q+8jIlGM4xpzrnffbc+yco=">AAACgXicjVFNb9QwEHVCgRK+FjhyGbEtQiAFZy8FIVAlLhyL1KWV1svK8U52rTpOsCfQKMoP4W9x49eAd5sDtBwYydLTe/PlN3lttCfOf0bxtZ3rN27u3kpu37l77/7owcNPvmqcwqmqTOVOc+nRaItT0mTwtHYoy9zgSX72fqOffEXndWWPqa1xXsqV1YVWkgK1GH3fS8RaUtf28BaEwYJEl4gcV9p20jnZ9p1zfcLhKQjCc3JlB7qAHjqRF/Ct/3y8Bec9vIAcNh2+AAchIMn+r+Qd8ESgXQ7TEuH0ak1psrcYjXnKtwFXQTaAMRviaDH6IZaVakq0pIz0fpbxmuahL2llMHRuPNZSnckVzgK0skQ/77YW9rAfmCUUlQvPEmzZPys6WXrflnnILCWt/WVtQ/5LmzVUvJp32tYNoVUXg4rGAFWwuQcstUNFpg1AKqfDrqDW0klF4WpJMCG7/OWrYDpJX6f842R8eDC4scsesyfsGcvYATtkH9gRmzLFfkX7URq9jHfi5zGPJxepcTTUPGJ/RfzmN2SzvSc=</latexit><latexit sha1_base64="d3Pw5q+8jIlGM4xpzrnffbc+yco=">AAACgXicjVFNb9QwEHVCgRK+FjhyGbEtQiAFZy8FIVAlLhyL1KWV1svK8U52rTpOsCfQKMoP4W9x49eAd5sDtBwYydLTe/PlN3lttCfOf0bxtZ3rN27u3kpu37l77/7owcNPvmqcwqmqTOVOc+nRaItT0mTwtHYoy9zgSX72fqOffEXndWWPqa1xXsqV1YVWkgK1GH3fS8RaUtf28BaEwYJEl4gcV9p20jnZ9p1zfcLhKQjCc3JlB7qAHjqRF/Ct/3y8Bec9vIAcNh2+AAchIMn+r+Qd8ESgXQ7TEuH0ak1psrcYjXnKtwFXQTaAMRviaDH6IZaVakq0pIz0fpbxmuahL2llMHRuPNZSnckVzgK0skQ/77YW9rAfmCUUlQvPEmzZPys6WXrflnnILCWt/WVtQ/5LmzVUvJp32tYNoVUXg4rGAFWwuQcstUNFpg1AKqfDrqDW0klF4WpJMCG7/OWrYDpJX6f842R8eDC4scsesyfsGcvYATtkH9gRmzLFfkX7URq9jHfi5zGPJxepcTTUPGJ/RfzmN2SzvSc=</latexit>
x1 0 1 0 1x2 0 0 1 1
x1 XOR x2 0 1 1 0<latexit sha1_base64="7KwtoelifvaVe1T374Vf/hf8/1A=">AAACd3icbVFLT9tAEF67LQ9DaWiPXFbEQZwiOxcKJ6ReuPFQQyLFUbTeTJIV67W1O0ZExr+h/40bP4QLp66DadqkI4322++bx+5MnElhMAieHffDx08bm1vb3s7u570vjf2vtybNNYcuT2Wq+zEzIIWCLgqU0M80sCSW0IvvflR67x60Ean6ifMMhgmbKjERnKGlRo1fvu9FMUyFKpjWbF4W/JFbK72HUUiPaGB9eUaRF82qVtTKnZp+D1mR7RXhAREL2r+8oSVdZoR/ilYZoMZ1b8/3R41m0A4WRtdBWIMmqe1q1HiKxinPE1DIJTNmEAYZDm1BFFxC6UW5gYzxOzaFgYWKJWCGxWJwJW1ZZkwnqbaukC7YvzMKlhgzT2IbmTCcmVWtIv+nDXKcfB8WQmU5guJvjSa5pJjSagt0LDRwlHMLGNfCvpXyGdOMo92VZ4cQrn55HXQ77dN2cN1pnp/V09giB+SQHJOQnJBzckGuSJdw8uIcOL7Tcl5d6h65x2+hrlPnfCP/mBv+BnbdscA=</latexit><latexit sha1_base64="7KwtoelifvaVe1T374Vf/hf8/1A=">AAACd3icbVFLT9tAEF67LQ9DaWiPXFbEQZwiOxcKJ6ReuPFQQyLFUbTeTJIV67W1O0ZExr+h/40bP4QLp66DadqkI4322++bx+5MnElhMAieHffDx08bm1vb3s7u570vjf2vtybNNYcuT2Wq+zEzIIWCLgqU0M80sCSW0IvvflR67x60Ean6ifMMhgmbKjERnKGlRo1fvu9FMUyFKpjWbF4W/JFbK72HUUiPaGB9eUaRF82qVtTKnZp+D1mR7RXhAREL2r+8oSVdZoR/ilYZoMZ1b8/3R41m0A4WRtdBWIMmqe1q1HiKxinPE1DIJTNmEAYZDm1BFFxC6UW5gYzxOzaFgYWKJWCGxWJwJW1ZZkwnqbaukC7YvzMKlhgzT2IbmTCcmVWtIv+nDXKcfB8WQmU5guJvjSa5pJjSagt0LDRwlHMLGNfCvpXyGdOMo92VZ4cQrn55HXQ77dN2cN1pnp/V09giB+SQHJOQnJBzckGuSJdw8uIcOL7Tcl5d6h65x2+hrlPnfCP/mBv+BnbdscA=</latexit><latexit sha1_base64="7KwtoelifvaVe1T374Vf/hf8/1A=">AAACd3icbVFLT9tAEF67LQ9DaWiPXFbEQZwiOxcKJ6ReuPFQQyLFUbTeTJIV67W1O0ZExr+h/40bP4QLp66DadqkI4322++bx+5MnElhMAieHffDx08bm1vb3s7u570vjf2vtybNNYcuT2Wq+zEzIIWCLgqU0M80sCSW0IvvflR67x60Ean6ifMMhgmbKjERnKGlRo1fvu9FMUyFKpjWbF4W/JFbK72HUUiPaGB9eUaRF82qVtTKnZp+D1mR7RXhAREL2r+8oSVdZoR/ilYZoMZ1b8/3R41m0A4WRtdBWIMmqe1q1HiKxinPE1DIJTNmEAYZDm1BFFxC6UW5gYzxOzaFgYWKJWCGxWJwJW1ZZkwnqbaukC7YvzMKlhgzT2IbmTCcmVWtIv+nDXKcfB8WQmU5guJvjSa5pJjSagt0LDRwlHMLGNfCvpXyGdOMo92VZ4cQrn55HXQ77dN2cN1pnp/V09giB+SQHJOQnJBzckGuSJdw8uIcOL7Tcl5d6h65x2+hrlPnfCP/mBv+BnbdscA=</latexit>
The Perceptron
19
Question: Why can’t we learn XOR?
OR AND
NAND XOR
W
The Perceptron
20
Question: Why can’t we learn XOR?
Answer: The perceptron is a linear classifier. Can only learn things that are linearly separable
Question: How can we fix this?
The Multilayer Perceptron
21
Question: How can we fix this?
Answer: The Multilayer Perceptron
Anatomy: o Input Layer
o Hidden Layer
o Output Layera.12 bz
onW 22=
The Multilayer Perceptron
22
Anatomy: Input Layer to Hidden Layer
o Define intermediate activities z
o Define activations a
The indicator I is the activation function
z1 = w111x1 + w1
12x2 + b11, a1 = I(z1 > 0)
z2 = w121x1 + w1
22x2 + b12, a2 = I(z2 > 0)<latexit sha1_base64="LbR5oqBU2cyj5PiNOTK0lNmSgd8=">AAACkHicbZHdTtswFMedDFgXBnTjcjfWyiQQqIqtSdBJsE7cMK6KtAJSWyLHccHC+Zh9stFFeZ89z+54G5y0Qm3ZkSz/z9fP9nGYKWnA9x8d99XK6trrxhtv/e3G5lbz3ftLk+aaiz5PVaqvQ2aEkonogwQlrjMtWBwqcRXen1b5q19CG5kmP2CSiVHMbhM5lpyBDQXNvzs73p+A4GP8OygIKW/Ig91LvF/7dOrTyg8DckMO8PBnziLM6pYhiAfQcfG93K0YJ9jfw54FejWUTqF0CUoXoXQeSpeh9BkaNFt+268NvxRkJlpoZr2g+W8YpTyPRQJcMWMGxM9gVDANkitResPciIzxe3YrBlYmLBZmVNQTLfEnG4nwONV2JYDr6HxHwWJjJnFoK2MGd2Y5VwX/lxvkMD4aFTLJchAJnx40zhWGFFffgyOpBQc1sYJxLe1dMb9jmnGwn+jZIZDlJ78UfdrutP0L2up+mU2jgT6gj2gXEXSIuugM9VAfcWfT+ewcOyfutttxv7rfpqWuM+vZRgvmnj8BEEq6wQ==</latexit><latexit sha1_base64="LbR5oqBU2cyj5PiNOTK0lNmSgd8=">AAACkHicbZHdTtswFMedDFgXBnTjcjfWyiQQqIqtSdBJsE7cMK6KtAJSWyLHccHC+Zh9stFFeZ89z+54G5y0Qm3ZkSz/z9fP9nGYKWnA9x8d99XK6trrxhtv/e3G5lbz3ftLk+aaiz5PVaqvQ2aEkonogwQlrjMtWBwqcRXen1b5q19CG5kmP2CSiVHMbhM5lpyBDQXNvzs73p+A4GP8OygIKW/Ig91LvF/7dOrTyg8DckMO8PBnziLM6pYhiAfQcfG93K0YJ9jfw54FejWUTqF0CUoXoXQeSpeh9BkaNFt+268NvxRkJlpoZr2g+W8YpTyPRQJcMWMGxM9gVDANkitResPciIzxe3YrBlYmLBZmVNQTLfEnG4nwONV2JYDr6HxHwWJjJnFoK2MGd2Y5VwX/lxvkMD4aFTLJchAJnx40zhWGFFffgyOpBQc1sYJxLe1dMb9jmnGwn+jZIZDlJ78UfdrutP0L2up+mU2jgT6gj2gXEXSIuugM9VAfcWfT+ewcOyfutttxv7rfpqWuM+vZRgvmnj8BEEq6wQ==</latexit><latexit sha1_base64="LbR5oqBU2cyj5PiNOTK0lNmSgd8=">AAACkHicbZHdTtswFMedDFgXBnTjcjfWyiQQqIqtSdBJsE7cMK6KtAJSWyLHccHC+Zh9stFFeZ89z+54G5y0Qm3ZkSz/z9fP9nGYKWnA9x8d99XK6trrxhtv/e3G5lbz3ftLk+aaiz5PVaqvQ2aEkonogwQlrjMtWBwqcRXen1b5q19CG5kmP2CSiVHMbhM5lpyBDQXNvzs73p+A4GP8OygIKW/Ig91LvF/7dOrTyg8DckMO8PBnziLM6pYhiAfQcfG93K0YJ9jfw54FejWUTqF0CUoXoXQeSpeh9BkaNFt+268NvxRkJlpoZr2g+W8YpTyPRQJcMWMGxM9gVDANkitResPciIzxe3YrBlYmLBZmVNQTLfEnG4nwONV2JYDr6HxHwWJjJnFoK2MGd2Y5VwX/lxvkMD4aFTLJchAJnx40zhWGFFffgyOpBQc1sYJxLe1dMb9jmnGwn+jZIZDlJ78UfdrutP0L2up+mU2jgT6gj2gXEXSIuugM9VAfcWfT+ewcOyfutttxv7rfpqWuM+vZRgvmnj8BEEq6wQ==</latexit>
a
:#=¥ I
The Multilayer Perceptron
23
Vectorized Anatomy: Input Layer to Hidden Layer
becomes
where
z1 = w111x1 + w1
12x2 + b11, a1 = I(z1 > 0)z2 = w1
21x1 + w122x2 + b12, a2 = I(z2 > 0)
<latexit sha1_base64="if1dw1K6ibWbAraQM+La//QnF5A=">AAACqHicbZHLbtQwFIadlEsJlw6wZGMxAyoqGsXecJGKKrEBiUUrkU7RZBo5zpmpVccJttN2iPJuPAM73gbnsminHMnyf26f7eO0lMLYMPzr+Vt37t67v/0gePjo8ZOd0dNnx6aoNIeIF7LQJykzIIWCyAor4aTUwPJUwiw9/9zmZxegjSjUd7suYZGzlRJLwZl1oWT0ezIJ4hRWQtVMa7ZuatkEvxKC9/FlUhPSnJIrtzd4r/Np79PWTxNySt7i+GfFMsy6ltjCldV5/bXZbRmfcPgGx7Hj0Z5HN3j0Jo9e59FNHu15QQwqGy4bTCbJaBxOw87wbUEGMUaDHSajP3FW8CoHZblkxsxJWNqFA1rBJTRBXBkoGT9nK5g7qVgOZlF3k27wKxfJ8LLQbimLu+j1jprlxqzz1FXmzJ6ZzVwb/F9uXtnl+0UtVFlZULw/aFlJbAvcfhvOhAZu5doJxrVwd8X8jGnGrfvcwA2BbD75tojo9MM0PKLjg4/DNLbRC/QS7SKC3qED9AUdoghx77X3zYu8Y3/PP/Jn/o++1PeGnufohvnpPxh3x1U=</latexit><latexit sha1_base64="if1dw1K6ibWbAraQM+La//QnF5A=">AAACqHicbZHLbtQwFIadlEsJlw6wZGMxAyoqGsXecJGKKrEBiUUrkU7RZBo5zpmpVccJttN2iPJuPAM73gbnsminHMnyf26f7eO0lMLYMPzr+Vt37t67v/0gePjo8ZOd0dNnx6aoNIeIF7LQJykzIIWCyAor4aTUwPJUwiw9/9zmZxegjSjUd7suYZGzlRJLwZl1oWT0ezIJ4hRWQtVMa7ZuatkEvxKC9/FlUhPSnJIrtzd4r/Np79PWTxNySt7i+GfFMsy6ltjCldV5/bXZbRmfcPgGx7Hj0Z5HN3j0Jo9e59FNHu15QQwqGy4bTCbJaBxOw87wbUEGMUaDHSajP3FW8CoHZblkxsxJWNqFA1rBJTRBXBkoGT9nK5g7qVgOZlF3k27wKxfJ8LLQbimLu+j1jprlxqzz1FXmzJ6ZzVwb/F9uXtnl+0UtVFlZULw/aFlJbAvcfhvOhAZu5doJxrVwd8X8jGnGrfvcwA2BbD75tojo9MM0PKLjg4/DNLbRC/QS7SKC3qED9AUdoghx77X3zYu8Y3/PP/Jn/o++1PeGnufohvnpPxh3x1U=</latexit><latexit sha1_base64="if1dw1K6ibWbAraQM+La//QnF5A=">AAACqHicbZHLbtQwFIadlEsJlw6wZGMxAyoqGsXecJGKKrEBiUUrkU7RZBo5zpmpVccJttN2iPJuPAM73gbnsminHMnyf26f7eO0lMLYMPzr+Vt37t67v/0gePjo8ZOd0dNnx6aoNIeIF7LQJykzIIWCyAor4aTUwPJUwiw9/9zmZxegjSjUd7suYZGzlRJLwZl1oWT0ezIJ4hRWQtVMa7ZuatkEvxKC9/FlUhPSnJIrtzd4r/Np79PWTxNySt7i+GfFMsy6ltjCldV5/bXZbRmfcPgGx7Hj0Z5HN3j0Jo9e59FNHu15QQwqGy4bTCbJaBxOw87wbUEGMUaDHSajP3FW8CoHZblkxsxJWNqFA1rBJTRBXBkoGT9nK5g7qVgOZlF3k27wKxfJ8LLQbimLu+j1jprlxqzz1FXmzJ6ZzVwb/F9uXtnl+0UtVFlZULw/aFlJbAvcfhvOhAZu5doJxrVwd8X8jGnGrfvcwA2BbD75tojo9MM0PKLjg4/DNLbRC/QS7SKC3qED9AUdoghx77X3zYu8Y3/PP/Jn/o++1PeGnufohvnpPxh3x1U=</latexit>
z2 = W 1x1 + b1
a2 = I(z2 > 0) = I(W 1x1 + b1 > 0)<latexit sha1_base64="bFJTMLtnFcASWTm+WeQFNfaTtV8=">AAACiXicdVHLTttAFB275VFDIYUlm1ETEBUisrPgJYGQ2LQ7KjUNUpxE48l1GDEeWzPXFanlj+GX2PE3jI2r8mivNNLROec+5t4ok8Kg7z847rv3C4tLyx+8ldWPa+utTxs/TZprDn2eylRfRcyAFAr6KFDCVaaBJZGEQXRzUemDX6CNSNUPnGcwSthMiVhwhpaatO46HS+MYCZUwbRm87LQXJZeEUYx/V2Oe3TndIcOxkFN3JbjgO7RGkcVDsMnJ/vjDBFuUSfFt3L3b4kz6n+hp8+1/xasrF4IatpM43U6k1bb7/p10LcgaECbNHE5ad2H05TnCSjkkhkzDPwMR7YgCi6h9MLcQMb4DZvB0ELFEjCjol5lSbctM6Vxqu1TSGv2eUbBEmPmSWSdCcNr81qryH9pwxzjo1EhVJYjKP7UKM4lxZRWd6FToYGjnFvAuBZ2VsqvmWYc7fU8u4Tg9Zffgn6ve9z1v/fa5yfNNpbJFvlMdklADsk5+UouSZ9wZ8nZdw6cQ3fV7blHbuN1nSZnk7wI9+IRCR69hg==</latexit><latexit sha1_base64="bFJTMLtnFcASWTm+WeQFNfaTtV8=">AAACiXicdVHLTttAFB275VFDIYUlm1ETEBUisrPgJYGQ2LQ7KjUNUpxE48l1GDEeWzPXFanlj+GX2PE3jI2r8mivNNLROec+5t4ok8Kg7z847rv3C4tLyx+8ldWPa+utTxs/TZprDn2eylRfRcyAFAr6KFDCVaaBJZGEQXRzUemDX6CNSNUPnGcwSthMiVhwhpaatO46HS+MYCZUwbRm87LQXJZeEUYx/V2Oe3TndIcOxkFN3JbjgO7RGkcVDsMnJ/vjDBFuUSfFt3L3b4kz6n+hp8+1/xasrF4IatpM43U6k1bb7/p10LcgaECbNHE5ad2H05TnCSjkkhkzDPwMR7YgCi6h9MLcQMb4DZvB0ELFEjCjol5lSbctM6Vxqu1TSGv2eUbBEmPmSWSdCcNr81qryH9pwxzjo1EhVJYjKP7UKM4lxZRWd6FToYGjnFvAuBZ2VsqvmWYc7fU8u4Tg9Zffgn6ve9z1v/fa5yfNNpbJFvlMdklADsk5+UouSZ9wZ8nZdw6cQ3fV7blHbuN1nSZnk7wI9+IRCR69hg==</latexit><latexit sha1_base64="bFJTMLtnFcASWTm+WeQFNfaTtV8=">AAACiXicdVHLTttAFB275VFDIYUlm1ETEBUisrPgJYGQ2LQ7KjUNUpxE48l1GDEeWzPXFanlj+GX2PE3jI2r8mivNNLROec+5t4ok8Kg7z847rv3C4tLyx+8ldWPa+utTxs/TZprDn2eylRfRcyAFAr6KFDCVaaBJZGEQXRzUemDX6CNSNUPnGcwSthMiVhwhpaatO46HS+MYCZUwbRm87LQXJZeEUYx/V2Oe3TndIcOxkFN3JbjgO7RGkcVDsMnJ/vjDBFuUSfFt3L3b4kz6n+hp8+1/xasrF4IatpM43U6k1bb7/p10LcgaECbNHE5ad2H05TnCSjkkhkzDPwMR7YgCi6h9MLcQMb4DZvB0ELFEjCjol5lSbctM6Vxqu1TSGv2eUbBEmPmSWSdCcNr81qryH9pwxzjo1EhVJYjKP7UKM4lxZRWd6FToYGjnFvAuBZ2VsqvmWYc7fU8u4Tg9Zffgn6ve9z1v/fa5yfNNpbJFvlMdklADsk5+UouSZ9wZ8nZdw6cQ3fV7blHbuN1nSZnk7wI9+IRCR69hg==</latexit>
W 1 =
w1
11 w112
w121 w1
22
�, b1 =
b11b12
�
<latexit sha1_base64="0kMo3/CYxOAQiq06yYUmE5mNBR8=">AAACnXicbZHfatswFMZld1s77V/WXfZGLNnYxQi2b9YOCi2DMtYyOliWQuwZST5ORGXZleSWYPxYfZHd7W0mJ4ZtbQ4IfnznnO9IR6ySwtgg+O35Ww8ePtreeYyfPH32/MXg5e4PU9aaw4SXstQXjBqQQsHECivhotJACyZhyi4/dfnpNWgjSvXdLitICjpXIhecWielg9vRCE9/huSQ4FhCbmc4ZjAXqqFa02XbcN7im7QJw9YVvSUdRh3GcSdHf+Wok3EMKutbcazFfGGT9875qqYZwU3McsLa1bTNw1rM0nDtztLIwSZDPBqlg2EwDlZB7kPYwxD1cZ4OfsVZyesClOWSGjMLg8omztgKLsFZ1wYqyi/pHGYOFS3AJM1qvS1545SM5KV2R1myUv/taGhhzLJgrrKgdmHu5jpxU25W23w/aYSqaguKrwfltSS2JN1fkUxo4FYuHVCuhbsr4QuqKbfuR7FbQnj3yfdhEo0PxsG3aHj0sd/GDtpDr9E7FKIP6Ah9Rudogri35x17X7xTn/gn/pn/dV3qe33PK/Rf+NM/KSnFow==</latexit><latexit sha1_base64="0kMo3/CYxOAQiq06yYUmE5mNBR8=">AAACnXicbZHfatswFMZld1s77V/WXfZGLNnYxQi2b9YOCi2DMtYyOliWQuwZST5ORGXZleSWYPxYfZHd7W0mJ4ZtbQ4IfnznnO9IR6ySwtgg+O35Ww8ePtreeYyfPH32/MXg5e4PU9aaw4SXstQXjBqQQsHECivhotJACyZhyi4/dfnpNWgjSvXdLitICjpXIhecWielg9vRCE9/huSQ4FhCbmc4ZjAXqqFa02XbcN7im7QJw9YVvSUdRh3GcSdHf+Wok3EMKutbcazFfGGT9875qqYZwU3McsLa1bTNw1rM0nDtztLIwSZDPBqlg2EwDlZB7kPYwxD1cZ4OfsVZyesClOWSGjMLg8omztgKLsFZ1wYqyi/pHGYOFS3AJM1qvS1545SM5KV2R1myUv/taGhhzLJgrrKgdmHu5jpxU25W23w/aYSqaguKrwfltSS2JN1fkUxo4FYuHVCuhbsr4QuqKbfuR7FbQnj3yfdhEo0PxsG3aHj0sd/GDtpDr9E7FKIP6Ah9Rudogri35x17X7xTn/gn/pn/dV3qe33PK/Rf+NM/KSnFow==</latexit><latexit sha1_base64="0kMo3/CYxOAQiq06yYUmE5mNBR8=">AAACnXicbZHfatswFMZld1s77V/WXfZGLNnYxQi2b9YOCi2DMtYyOliWQuwZST5ORGXZleSWYPxYfZHd7W0mJ4ZtbQ4IfnznnO9IR6ySwtgg+O35Ww8ePtreeYyfPH32/MXg5e4PU9aaw4SXstQXjBqQQsHECivhotJACyZhyi4/dfnpNWgjSvXdLitICjpXIhecWielg9vRCE9/huSQ4FhCbmc4ZjAXqqFa02XbcN7im7QJw9YVvSUdRh3GcSdHf+Wok3EMKutbcazFfGGT9875qqYZwU3McsLa1bTNw1rM0nDtztLIwSZDPBqlg2EwDlZB7kPYwxD1cZ4OfsVZyesClOWSGjMLg8omztgKLsFZ1wYqyi/pHGYOFS3AJM1qvS1545SM5KV2R1myUv/taGhhzLJgrrKgdmHu5jpxU25W23w/aYSqaguKrwfltSS2JN1fkUxo4FYuHVCuhbsr4QuqKbfuR7FbQnj3yfdhEo0PxsG3aHj0sd/GDtpDr9E7FKIP6Ah9Rudogri35x17X7xTn/gn/pn/dV3qe33PK/Rf+NM/KSnFow==</latexit>
b2,
=
× - [ It ] WE
WE
[ wiwii ]b2= Ibf }
The Multilayer Perceptron
24
Vectorized Anatomy: Hidden Layer to Output Layer
where
z3 = W 2a2 + b2
y = I(z3 > 0) = I(W 2a2 + b2 > 0)<latexit sha1_base64="ASe3O1wbpXNcLSB0W+Iovcd8mnM=">AAACiHicdVFdb9MwFHUCjJEBK/DIi0XLNASqkiLYhzQ0wQu8DYnSSXVbOe5Na81xIvsGLUTZb+E38ca/wckiMTa4kqWjc8798L1xrqTFMPzl+bdu39m4u3kv2Lr/4OF279HjrzYrjICxyFRmTmNuQUkNY5So4DQ3wNNYwSQ++9Dok29grMz0FyxzmKV8pWUiBUdHLXo/BoOAxbCSuuLG8LKujFB1ULE4od/r+Wu6c7RDJ/NRS/B6PqIvaYvjBjMWsDXHqqwvWiNDOEeTVp/q3T8V3tHwBT26qv23XmMNGOhlN0wwGCx6/XAYtkFvgqgDfdLFyaL3ky0zUaSgUShu7TQKc5y5giiFgjpghYWcizO+gqmDmqdgZ1W7yZo+d8ySJplxTyNt2asZFU+tLdPYOVOOa3tda8h/adMCk/1ZJXVeIGhx2SgpFMWMNmehS2lAoCod4MJINysVa264QHe8wC0huv7lm2A8Gh4Mw8+j/vFht41N8pQ8I7skInvkmHwkJ2RMhLfhvfLeeG/9LT/y9/yDS6vvdTlPyF/hv/8NMF69sQ==</latexit><latexit sha1_base64="ASe3O1wbpXNcLSB0W+Iovcd8mnM=">AAACiHicdVFdb9MwFHUCjJEBK/DIi0XLNASqkiLYhzQ0wQu8DYnSSXVbOe5Na81xIvsGLUTZb+E38ca/wckiMTa4kqWjc8798L1xrqTFMPzl+bdu39m4u3kv2Lr/4OF279HjrzYrjICxyFRmTmNuQUkNY5So4DQ3wNNYwSQ++9Dok29grMz0FyxzmKV8pWUiBUdHLXo/BoOAxbCSuuLG8LKujFB1ULE4od/r+Wu6c7RDJ/NRS/B6PqIvaYvjBjMWsDXHqqwvWiNDOEeTVp/q3T8V3tHwBT26qv23XmMNGOhlN0wwGCx6/XAYtkFvgqgDfdLFyaL3ky0zUaSgUShu7TQKc5y5giiFgjpghYWcizO+gqmDmqdgZ1W7yZo+d8ySJplxTyNt2asZFU+tLdPYOVOOa3tda8h/adMCk/1ZJXVeIGhx2SgpFMWMNmehS2lAoCod4MJINysVa264QHe8wC0huv7lm2A8Gh4Mw8+j/vFht41N8pQ8I7skInvkmHwkJ2RMhLfhvfLeeG/9LT/y9/yDS6vvdTlPyF/hv/8NMF69sQ==</latexit><latexit sha1_base64="ASe3O1wbpXNcLSB0W+Iovcd8mnM=">AAACiHicdVFdb9MwFHUCjJEBK/DIi0XLNASqkiLYhzQ0wQu8DYnSSXVbOe5Na81xIvsGLUTZb+E38ca/wckiMTa4kqWjc8798L1xrqTFMPzl+bdu39m4u3kv2Lr/4OF279HjrzYrjICxyFRmTmNuQUkNY5So4DQ3wNNYwSQ++9Dok29grMz0FyxzmKV8pWUiBUdHLXo/BoOAxbCSuuLG8LKujFB1ULE4od/r+Wu6c7RDJ/NRS/B6PqIvaYvjBjMWsDXHqqwvWiNDOEeTVp/q3T8V3tHwBT26qv23XmMNGOhlN0wwGCx6/XAYtkFvgqgDfdLFyaL3ky0zUaSgUShu7TQKc5y5giiFgjpghYWcizO+gqmDmqdgZ1W7yZo+d8ySJplxTyNt2asZFU+tLdPYOVOOa3tda8h/adMCk/1ZJXVeIGhx2SgpFMWMNmehS2lAoCod4MJINysVa264QHe8wC0huv7lm2A8Gh4Mw8+j/vFht41N8pQ8I7skInvkmHwkJ2RMhLfhvfLeeG/9LT/y9/yDS6vvdTlPyF/hv/8NMF69sQ==</latexit>
W 2 =⇥w2
11 w212
⇤, b2 =
⇥b21
⇤<latexit sha1_base64="VTIiy0b6DlHmPd9FAKlM8oFfRz4=">AAACfXicbVFNj9MwEHXC12K+Chy5GFpWCFZVkssCEtJKXDguEqUrNdnIdiattY4T7AmoivIz+GPc+C1ccLo5wG5HsvT05s2z/UY0WjmMot9BeOPmrdt3Du7Se/cfPHw0efzkq6tbK2Eha13bM8EdaGVggQo1nDUWeCU0LMXFx6G//A7Wqdp8wW0DWcXXRpVKcvRUPvk5m9HlecI+MJpqKHFFUwFrZTpuLd/2nZQ9/ZF3cdx70SEbYDJAmoIpRhFNrVpvMDvy7LeWF4x2qSiZ6He++217KvL4PNlnQ2ezfDKN5tGu2HUQj2BKxjrNJ7/SopZtBQal5s6t4qjBzBujkhq8deug4fKCr2HloeEVuKzbxdezl54pWFlbfwyyHfvvRMcr57aV8MqK48Zd7Q3kvt6qxfJt1inTtAhGXl5UtpphzYZdsEJZkKi3HnBplX8rkxtuuUS/MepDiK9++TpYJPN38+hzMj15P6ZxQJ6RF+QVickxOSGfyClZEEn+BM+D18GbkISH4VE4v5SGwTjzlPxX4fFfU0a8Og==</latexit><latexit sha1_base64="VTIiy0b6DlHmPd9FAKlM8oFfRz4=">AAACfXicbVFNj9MwEHXC12K+Chy5GFpWCFZVkssCEtJKXDguEqUrNdnIdiattY4T7AmoivIz+GPc+C1ccLo5wG5HsvT05s2z/UY0WjmMot9BeOPmrdt3Du7Se/cfPHw0efzkq6tbK2Eha13bM8EdaGVggQo1nDUWeCU0LMXFx6G//A7Wqdp8wW0DWcXXRpVKcvRUPvk5m9HlecI+MJpqKHFFUwFrZTpuLd/2nZQ9/ZF3cdx70SEbYDJAmoIpRhFNrVpvMDvy7LeWF4x2qSiZ6He++217KvL4PNlnQ2ezfDKN5tGu2HUQj2BKxjrNJ7/SopZtBQal5s6t4qjBzBujkhq8deug4fKCr2HloeEVuKzbxdezl54pWFlbfwyyHfvvRMcr57aV8MqK48Zd7Q3kvt6qxfJt1inTtAhGXl5UtpphzYZdsEJZkKi3HnBplX8rkxtuuUS/MepDiK9++TpYJPN38+hzMj15P6ZxQJ6RF+QVickxOSGfyClZEEn+BM+D18GbkISH4VE4v5SGwTjzlPxX4fFfU0a8Og==</latexit><latexit sha1_base64="VTIiy0b6DlHmPd9FAKlM8oFfRz4=">AAACfXicbVFNj9MwEHXC12K+Chy5GFpWCFZVkssCEtJKXDguEqUrNdnIdiattY4T7AmoivIz+GPc+C1ccLo5wG5HsvT05s2z/UY0WjmMot9BeOPmrdt3Du7Se/cfPHw0efzkq6tbK2Eha13bM8EdaGVggQo1nDUWeCU0LMXFx6G//A7Wqdp8wW0DWcXXRpVKcvRUPvk5m9HlecI+MJpqKHFFUwFrZTpuLd/2nZQ9/ZF3cdx70SEbYDJAmoIpRhFNrVpvMDvy7LeWF4x2qSiZ6He++217KvL4PNlnQ2ezfDKN5tGu2HUQj2BKxjrNJ7/SopZtBQal5s6t4qjBzBujkhq8deug4fKCr2HloeEVuKzbxdezl54pWFlbfwyyHfvvRMcr57aV8MqK48Zd7Q3kvt6qxfJt1inTtAhGXl5UtpphzYZdsEJZkKi3HnBplX8rkxtuuUS/MepDiK9++TpYJPN38+hzMj15P6ZxQJ6RF+QVickxOSGfyClZEEn+BM+D18GbkISH4VE4v5SGwTjzlPxX4fFfU0a8Og==</latexit>
zs = W2A2+b2T
a3=yn=I( Z ?> 0) p.ee#svres NEWFEATVRS
( Inputs ) ( HIDDEN )
The Multilayer Perceptron
25
Example: Can we learn XOR? x1 0 1 0 1x2 0 0 1 1
x1 XOR x2 0 1 1 0<latexit sha1_base64="7KwtoelifvaVe1T374Vf/hf8/1A=">AAACd3icbVFLT9tAEF67LQ9DaWiPXFbEQZwiOxcKJ6ReuPFQQyLFUbTeTJIV67W1O0ZExr+h/40bP4QLp66DadqkI4322++bx+5MnElhMAieHffDx08bm1vb3s7u570vjf2vtybNNYcuT2Wq+zEzIIWCLgqU0M80sCSW0IvvflR67x60Ean6ifMMhgmbKjERnKGlRo1fvu9FMUyFKpjWbF4W/JFbK72HUUiPaGB9eUaRF82qVtTKnZp+D1mR7RXhAREL2r+8oSVdZoR/ilYZoMZ1b8/3R41m0A4WRtdBWIMmqe1q1HiKxinPE1DIJTNmEAYZDm1BFFxC6UW5gYzxOzaFgYWKJWCGxWJwJW1ZZkwnqbaukC7YvzMKlhgzT2IbmTCcmVWtIv+nDXKcfB8WQmU5guJvjSa5pJjSagt0LDRwlHMLGNfCvpXyGdOMo92VZ4cQrn55HXQ77dN2cN1pnp/V09giB+SQHJOQnJBzckGuSJdw8uIcOL7Tcl5d6h65x2+hrlPnfCP/mBv+BnbdscA=</latexit><latexit sha1_base64="7KwtoelifvaVe1T374Vf/hf8/1A=">AAACd3icbVFLT9tAEF67LQ9DaWiPXFbEQZwiOxcKJ6ReuPFQQyLFUbTeTJIV67W1O0ZExr+h/40bP4QLp66DadqkI4322++bx+5MnElhMAieHffDx08bm1vb3s7u570vjf2vtybNNYcuT2Wq+zEzIIWCLgqU0M80sCSW0IvvflR67x60Ean6ifMMhgmbKjERnKGlRo1fvu9FMUyFKpjWbF4W/JFbK72HUUiPaGB9eUaRF82qVtTKnZp+D1mR7RXhAREL2r+8oSVdZoR/ilYZoMZ1b8/3R41m0A4WRtdBWIMmqe1q1HiKxinPE1DIJTNmEAYZDm1BFFxC6UW5gYzxOzaFgYWKJWCGxWJwJW1ZZkwnqbaukC7YvzMKlhgzT2IbmTCcmVWtIv+nDXKcfB8WQmU5guJvjSa5pJjSagt0LDRwlHMLGNfCvpXyGdOMo92VZ4cQrn55HXQ77dN2cN1pnp/V09giB+SQHJOQnJBzckGuSJdw8uIcOL7Tcl5d6h65x2+hrlPnfCP/mBv+BnbdscA=</latexit><latexit sha1_base64="7KwtoelifvaVe1T374Vf/hf8/1A=">AAACd3icbVFLT9tAEF67LQ9DaWiPXFbEQZwiOxcKJ6ReuPFQQyLFUbTeTJIV67W1O0ZExr+h/40bP4QLp66DadqkI4322++bx+5MnElhMAieHffDx08bm1vb3s7u570vjf2vtybNNYcuT2Wq+zEzIIWCLgqU0M80sCSW0IvvflR67x60Ean6ifMMhgmbKjERnKGlRo1fvu9FMUyFKpjWbF4W/JFbK72HUUiPaGB9eUaRF82qVtTKnZp+D1mR7RXhAREL2r+8oSVdZoR/ilYZoMZ1b8/3R41m0A4WRtdBWIMmqe1q1HiKxinPE1DIJTNmEAYZDm1BFFxC6UW5gYzxOzaFgYWKJWCGxWJwJW1ZZkwnqbaukC7YvzMKlhgzT2IbmTCcmVWtIv+nDXKcfB8WQmU5guJvjSa5pJjSagt0LDRwlHMLGNfCvpXyGdOMo92VZ4cQrn55HXQ77dN2cN1pnp/V09giB+SQHJOQnJBzckGuSJdw8uIcOL7Tcl5d6h65x2+hrlPnfCP/mBv+BnbdscA=</latexit>
The Multilayer Perceptron
26
Example: Can we learn XOR? x1 0 1 0 1x2 0 0 1 1
(x1 OR x2) AND (x1 NAND x2) 0 1 1 0<latexit sha1_base64="2EOcTsTdKqsK/cnJtMgylgc25pQ=">AAACoXicbVFNT9tAEF0bCtRtIdBjD10Rt6KXyM6FjxOorYR6CAGRghRH0XozSVas19buGBG5/mH9G73137BOjCDASKN9eu/N7O5MnElhMAj+O+7K6pu19Y233rv3Hza3Gts7v02aaw49nspUX8fMgBQKeihQwnWmgSWxhKv45nulX92CNiJVlzjLYJCwiRJjwRlaatj46/teFMNEqIJpzWZlwf9wG6V3NwzpVxrYfDyjyIum1VXUyu2afrAsyXtVeYRwh4gFPbugJbUV3x6pk84Pyy3bOgtybnzat8JVb1Cj+pWe7w8bzaAVzIO+BGENmqSO7rDxLxqlPE9AIZfMmH4YZDiwDVFwCaUX5QYyxm/YBPoWKpaAGRTzEZf0i2VGdJxqmwrpnH1aUbDEmFkSW2fCcGqeaxX5mtbPcXwwKITKcgTFFxeNc0kxpdW+6Eho4ChnFjCuhX0r5VOmGUe7Vc8OIXz+5Zeg124dtoLzdvP4qJ7GBvlEdskeCck+OSanpEt6hDufnZ9OxzlzffeX23UvFlbXqWs+kqVw+/dfgr63</latexit><latexit sha1_base64="2EOcTsTdKqsK/cnJtMgylgc25pQ=">AAACoXicbVFNT9tAEF0bCtRtIdBjD10Rt6KXyM6FjxOorYR6CAGRghRH0XozSVas19buGBG5/mH9G73137BOjCDASKN9eu/N7O5MnElhMAj+O+7K6pu19Y233rv3Hza3Gts7v02aaw49nspUX8fMgBQKeihQwnWmgSWxhKv45nulX92CNiJVlzjLYJCwiRJjwRlaatj46/teFMNEqIJpzWZlwf9wG6V3NwzpVxrYfDyjyIum1VXUyu2afrAsyXtVeYRwh4gFPbugJbUV3x6pk84Pyy3bOgtybnzat8JVb1Cj+pWe7w8bzaAVzIO+BGENmqSO7rDxLxqlPE9AIZfMmH4YZDiwDVFwCaUX5QYyxm/YBPoWKpaAGRTzEZf0i2VGdJxqmwrpnH1aUbDEmFkSW2fCcGqeaxX5mtbPcXwwKITKcgTFFxeNc0kxpdW+6Eho4ChnFjCuhX0r5VOmGUe7Vc8OIXz+5Zeg124dtoLzdvP4qJ7GBvlEdskeCck+OSanpEt6hDufnZ9OxzlzffeX23UvFlbXqWs+kqVw+/dfgr63</latexit><latexit sha1_base64="2EOcTsTdKqsK/cnJtMgylgc25pQ=">AAACoXicbVFNT9tAEF0bCtRtIdBjD10Rt6KXyM6FjxOorYR6CAGRghRH0XozSVas19buGBG5/mH9G73137BOjCDASKN9eu/N7O5MnElhMAj+O+7K6pu19Y233rv3Hza3Gts7v02aaw49nspUX8fMgBQKeihQwnWmgSWxhKv45nulX92CNiJVlzjLYJCwiRJjwRlaatj46/teFMNEqIJpzWZlwf9wG6V3NwzpVxrYfDyjyIum1VXUyu2afrAsyXtVeYRwh4gFPbugJbUV3x6pk84Pyy3bOgtybnzat8JVb1Cj+pWe7w8bzaAVzIO+BGENmqSO7rDxLxqlPE9AIZfMmH4YZDiwDVFwCaUX5QYyxm/YBPoWKpaAGRTzEZf0i2VGdJxqmwrpnH1aUbDEmFkSW2fCcGqeaxX5mtbPcXwwKITKcgTFFxeNc0kxpdW+6Eho4ChnFjCuhX0r5VOmGUe7Vc8OIXz+5Zeg124dtoLzdvP4qJ7GBvlEdskeCck+OSanpEt6hDufnZ9OxzlzffeX23UvFlbXqWs+kqVw+/dfgr63</latexit>
\ µaBor : T 1 11
T°3=Wl ,bl
-
ORnano : [t.IT/l.5/
/ And :[ 1 1 ) [ -1 ]
NAND W2 b2
The Multilayer Perceptron
27
Example: XOR Check
x1 0 1 0 1x2 0 0 1 1
(x1 OR x2) AND (x1 NAND x2) 0 1 1 0<latexit sha1_base64="2EOcTsTdKqsK/cnJtMgylgc25pQ=">AAACoXicbVFNT9tAEF0bCtRtIdBjD10Rt6KXyM6FjxOorYR6CAGRghRH0XozSVas19buGBG5/mH9G73137BOjCDASKN9eu/N7O5MnElhMAj+O+7K6pu19Y233rv3Hza3Gts7v02aaw49nspUX8fMgBQKeihQwnWmgSWxhKv45nulX92CNiJVlzjLYJCwiRJjwRlaatj46/teFMNEqIJpzWZlwf9wG6V3NwzpVxrYfDyjyIum1VXUyu2afrAsyXtVeYRwh4gFPbugJbUV3x6pk84Pyy3bOgtybnzat8JVb1Cj+pWe7w8bzaAVzIO+BGENmqSO7rDxLxqlPE9AIZfMmH4YZDiwDVFwCaUX5QYyxm/YBPoWKpaAGRTzEZf0i2VGdJxqmwrpnH1aUbDEmFkSW2fCcGqeaxX5mtbPcXwwKITKcgTFFxeNc0kxpdW+6Eho4ChnFjCuhX0r5VOmGUe7Vc8OIXz+5Zeg124dtoLzdvP4qJ7GBvlEdskeCck+OSanpEt6hDufnZ9OxzlzffeX23UvFlbXqWs+kqVw+/dfgr63</latexit><latexit sha1_base64="2EOcTsTdKqsK/cnJtMgylgc25pQ=">AAACoXicbVFNT9tAEF0bCtRtIdBjD10Rt6KXyM6FjxOorYR6CAGRghRH0XozSVas19buGBG5/mH9G73137BOjCDASKN9eu/N7O5MnElhMAj+O+7K6pu19Y233rv3Hza3Gts7v02aaw49nspUX8fMgBQKeihQwnWmgSWxhKv45nulX92CNiJVlzjLYJCwiRJjwRlaatj46/teFMNEqIJpzWZlwf9wG6V3NwzpVxrYfDyjyIum1VXUyu2afrAsyXtVeYRwh4gFPbugJbUV3x6pk84Pyy3bOgtybnzat8JVb1Cj+pWe7w8bzaAVzIO+BGENmqSO7rDxLxqlPE9AIZfMmH4YZDiwDVFwCaUX5QYyxm/YBPoWKpaAGRTzEZf0i2VGdJxqmwrpnH1aUbDEmFkSW2fCcGqeaxX5mtbPcXwwKITKcgTFFxeNc0kxpdW+6Eho4ChnFjCuhX0r5VOmGUe7Vc8OIXz+5Zeg124dtoLzdvP4qJ7GBvlEdskeCck+OSanpEt6hDufnZ9OxzlzffeX23UvFlbXqWs+kqVw+/dfgr63</latexit><latexit sha1_base64="2EOcTsTdKqsK/cnJtMgylgc25pQ=">AAACoXicbVFNT9tAEF0bCtRtIdBjD10Rt6KXyM6FjxOorYR6CAGRghRH0XozSVas19buGBG5/mH9G73137BOjCDASKN9eu/N7O5MnElhMAj+O+7K6pu19Y233rv3Hza3Gts7v02aaw49nspUX8fMgBQKeihQwnWmgSWxhKv45nulX92CNiJVlzjLYJCwiRJjwRlaatj46/teFMNEqIJpzWZlwf9wG6V3NwzpVxrYfDyjyIum1VXUyu2afrAsyXtVeYRwh4gFPbugJbUV3x6pk84Pyy3bOgtybnzat8JVb1Cj+pWe7w8bzaAVzIO+BGENmqSO7rDxLxqlPE9AIZfMmH4YZDiwDVFwCaUX5QYyxm/YBPoWKpaAGRTzEZf0i2VGdJxqmwrpnH1aUbDEmFkSW2fCcGqeaxX5mtbPcXwwKITKcgTFFxeNc0kxpdW+6Eho4ChnFjCuhX0r5VOmGUe7Vc8OIXz+5Zeg124dtoLzdvP4qJ7GBvlEdskeCck+OSanpEt6hDufnZ9OxzlzffeX23UvFlbXqWs+kqVw+/dfgr63</latexit>
W 1 =
1 1
�1 �1
�, b1 =
0
1.5
�
<latexit sha1_base64="dtFRWrcBCPwkdLpADAsQUDkV7Fo=">AAACfXicbVFNb9QwEHXCVzFfCxx7MexSIWijuBJaOFSqxIVjkVi20iasHGeya9Vxgj1BWkX5GfwxbvyWXupsc4B2n2Tp6c3Ms/0mq7VyGMd/g/DO3Xv3H+w9pI8eP3n6bPT8xXdXNVbCTFa6sueZcKCVgRkq1HBeWxBlpmGeXXzu6/NfYJ2qzDfc1JCWYmVUoaRALy1HvycTOv/B2QmjiYYCFzTJYKVMK6wVm661tqOcHTDOkoQe9eyI0wRMPjTQxKrVGtNDP/+zETmjbZIVLOu2nrstOxr3bjz6wHZZ0clkORrHUbwFu034QMZkwNly9CfJK9mUYFBq4dyCxzWm3hiV1OCtGwe1kBdiBQtPjSjBpe02vo698UrOisr6Y5Bt1X8nWlE6tykz31kKXLubtV7cVVs0WHxMW2XqBsHI64uKRjOsWL8LlisLEvXGEyGt8m9lci2skOg3Rn0I/OaXb5PZcfQpir8ej0+nQxp7ZJ+8Jm8JJ1NySr6QMzIjklwGr4J3wfuQhAfhYRhdt4bBMPOS/IdwegVA8rlr</latexit><latexit sha1_base64="dtFRWrcBCPwkdLpADAsQUDkV7Fo=">AAACfXicbVFNb9QwEHXCVzFfCxx7MexSIWijuBJaOFSqxIVjkVi20iasHGeya9Vxgj1BWkX5GfwxbvyWXupsc4B2n2Tp6c3Ms/0mq7VyGMd/g/DO3Xv3H+w9pI8eP3n6bPT8xXdXNVbCTFa6sueZcKCVgRkq1HBeWxBlpmGeXXzu6/NfYJ2qzDfc1JCWYmVUoaRALy1HvycTOv/B2QmjiYYCFzTJYKVMK6wVm661tqOcHTDOkoQe9eyI0wRMPjTQxKrVGtNDP/+zETmjbZIVLOu2nrstOxr3bjz6wHZZ0clkORrHUbwFu034QMZkwNly9CfJK9mUYFBq4dyCxzWm3hiV1OCtGwe1kBdiBQtPjSjBpe02vo698UrOisr6Y5Bt1X8nWlE6tykz31kKXLubtV7cVVs0WHxMW2XqBsHI64uKRjOsWL8LlisLEvXGEyGt8m9lci2skOg3Rn0I/OaXb5PZcfQpir8ej0+nQxp7ZJ+8Jm8JJ1NySr6QMzIjklwGr4J3wfuQhAfhYRhdt4bBMPOS/IdwegVA8rlr</latexit><latexit sha1_base64="dtFRWrcBCPwkdLpADAsQUDkV7Fo=">AAACfXicbVFNb9QwEHXCVzFfCxx7MexSIWijuBJaOFSqxIVjkVi20iasHGeya9Vxgj1BWkX5GfwxbvyWXupsc4B2n2Tp6c3Ms/0mq7VyGMd/g/DO3Xv3H+w9pI8eP3n6bPT8xXdXNVbCTFa6sueZcKCVgRkq1HBeWxBlpmGeXXzu6/NfYJ2qzDfc1JCWYmVUoaRALy1HvycTOv/B2QmjiYYCFzTJYKVMK6wVm661tqOcHTDOkoQe9eyI0wRMPjTQxKrVGtNDP/+zETmjbZIVLOu2nrstOxr3bjz6wHZZ0clkORrHUbwFu034QMZkwNly9CfJK9mUYFBq4dyCxzWm3hiV1OCtGwe1kBdiBQtPjSjBpe02vo698UrOisr6Y5Bt1X8nWlE6tykz31kKXLubtV7cVVs0WHxMW2XqBsHI64uKRjOsWL8LlisLEvXGEyGt8m9lci2skOg3Rn0I/OaXb5PZcfQpir8ej0+nQxp7ZJ+8Jm8JJ1NySr6QMzIjklwGr4J3wfuQhAfhYRhdt4bBMPOS/IdwegVA8rlr</latexit>
W 2 =⇥1 1
⇤, b2 =
⇥�1
⇤<latexit sha1_base64="NxNR37Ppw14yi1R4phhQWUATR6g=">AAACbHicbVFNb9NAEF0b6McWigvqoVSVViRUHCCycykckCJx4VgkQirFJlqvx8mq67W7O64UWb7xC7nxF7jwF9ikPpQ0I6309N7Mm923aaWkxTD87fmPHj/Z2d3bpwdPnx0+D45efLdlbQSMRalKc5VyC0pqGKNEBVeVAV6kCibp9eeVPrkFY2Wpv+GygqTgcy1zKTg6ahb87Pfp5MeQfWI0VpDjlMYpzKVuuDF82TZCtDRi5yxyOuiso2ls5HyByTvH3tQ8Y7SJ05yl7dppu1FL30fbPGi/Pwt64SBcF3sIog70SFeXs+BXnJWiLkCjUNzaaRRWmDhjlEKBs64tVFxc8zlMHdS8AJs067Ra9sYxGctL445GtmbvTzS8sHZZpK6z4Liwm9qK3KZNa8w/JI3UVY2gxd2ivFYMS7aKnmXSgEC1dIALI91dmVhwwwW6D6IuhGjzyQ/BeDj4OAi/Dnujiy6NPXJKXpO3JCIXZES+kEsyJoL88QLvxHvl/fWP/VP/7K7V97qZl+S/8s//ATYBtkk=</latexit><latexit sha1_base64="NxNR37Ppw14yi1R4phhQWUATR6g=">AAACbHicbVFNb9NAEF0b6McWigvqoVSVViRUHCCycykckCJx4VgkQirFJlqvx8mq67W7O64UWb7xC7nxF7jwF9ikPpQ0I6309N7Mm923aaWkxTD87fmPHj/Z2d3bpwdPnx0+D45efLdlbQSMRalKc5VyC0pqGKNEBVeVAV6kCibp9eeVPrkFY2Wpv+GygqTgcy1zKTg6ahb87Pfp5MeQfWI0VpDjlMYpzKVuuDF82TZCtDRi5yxyOuiso2ls5HyByTvH3tQ8Y7SJ05yl7dppu1FL30fbPGi/Pwt64SBcF3sIog70SFeXs+BXnJWiLkCjUNzaaRRWmDhjlEKBs64tVFxc8zlMHdS8AJs067Ra9sYxGctL445GtmbvTzS8sHZZpK6z4Liwm9qK3KZNa8w/JI3UVY2gxd2ivFYMS7aKnmXSgEC1dIALI91dmVhwwwW6D6IuhGjzyQ/BeDj4OAi/Dnujiy6NPXJKXpO3JCIXZES+kEsyJoL88QLvxHvl/fWP/VP/7K7V97qZl+S/8s//ATYBtkk=</latexit><latexit sha1_base64="NxNR37Ppw14yi1R4phhQWUATR6g=">AAACbHicbVFNb9NAEF0b6McWigvqoVSVViRUHCCycykckCJx4VgkQirFJlqvx8mq67W7O64UWb7xC7nxF7jwF9ikPpQ0I6309N7Mm923aaWkxTD87fmPHj/Z2d3bpwdPnx0+D45efLdlbQSMRalKc5VyC0pqGKNEBVeVAV6kCibp9eeVPrkFY2Wpv+GygqTgcy1zKTg6ahb87Pfp5MeQfWI0VpDjlMYpzKVuuDF82TZCtDRi5yxyOuiso2ls5HyByTvH3tQ8Y7SJ05yl7dppu1FL30fbPGi/Pwt64SBcF3sIog70SFeXs+BXnJWiLkCjUNzaaRRWmDhjlEKBs64tVFxc8zlMHdS8AJs067Ra9sYxGctL445GtmbvTzS8sHZZpK6z4Liwm9qK3KZNa8w/JI3UVY2gxd2ivFYMS7aKnmXSgEC1dIALI91dmVhwwwW6D6IuhGjzyQ/BeDj4OAi/Dnujiy6NPXJKXpO3JCIXZES+kEsyJoL88QLvxHvl/fWP/VP/7K7V97qZl+S/8s//ATYBtkk=</latexit>
z2 = W 1x1+b1 =
1 1
�1 �1
� �+
0
1.5
�=
�, a2 = I(z2 > 0) =
�
<latexit sha1_base64="1OIxmbCc/a4EtkhLEvGxQfJ4e6A=">AAADm3iclVJbb9MwFHYTLsPcOnhCSOiIFjQ0qOJKaPAwmAQPG+JhSIROatrKcZzWWuIE20ErUf4UP4U3/g1OGomxtQ87kq3P5/Kd408nzBOhjef96Tjutes3bm7dwrfv3L13v7v94JvOCsW4z7IkUych1TwRkvtGmISf5IrTNEz4KDz9UMdHP7jSIpNfzTLnk5TOpYgFo8a6ZtudX/0+LoMwhp/VdAj7MJqS5nlWTQnsQoPDGu8DDhIemzEOQj4XsqRK0WVVKlVhAs+BQBDgVzWyl83lMmpTcKDEfGEmG+orDEG+oNJkaUkqS3L+uY4H7FwbuTxLQAav19bhjX+48gwvLdP3gkYr7ehKu8DwM6PS8qja+SfpO/BebBbvyo1xvz/r9ryB1xhcBqQFPdTa8az7O4gyVqRcGpZQrcfEy83EEhvBEm6pC81zyk7pnI8tlDTlelI2y1XBM+uJIM6UPdJA4z1fUdJU62Ua2syUmoW+GKud62LjwsRvJqWQeWG4ZKtGcZGAyaDeVIiE4swkSwsoU8LOCmxBFWXG7jO2IpCLX74M/OHg7cD7Muwd7LVqbKHH6CnaQQTtoQN0iI6Rj5jzyHnvHDpH7hP3o/vJ/bxKdTptzUP0n7n+X7G5HyE=</latexit><latexit sha1_base64="1OIxmbCc/a4EtkhLEvGxQfJ4e6A=">AAADm3iclVJbb9MwFHYTLsPcOnhCSOiIFjQ0qOJKaPAwmAQPG+JhSIROatrKcZzWWuIE20ErUf4UP4U3/g1OGomxtQ87kq3P5/Kd408nzBOhjef96Tjutes3bm7dwrfv3L13v7v94JvOCsW4z7IkUych1TwRkvtGmISf5IrTNEz4KDz9UMdHP7jSIpNfzTLnk5TOpYgFo8a6ZtudX/0+LoMwhp/VdAj7MJqS5nlWTQnsQoPDGu8DDhIemzEOQj4XsqRK0WVVKlVhAs+BQBDgVzWyl83lMmpTcKDEfGEmG+orDEG+oNJkaUkqS3L+uY4H7FwbuTxLQAav19bhjX+48gwvLdP3gkYr7ehKu8DwM6PS8qja+SfpO/BebBbvyo1xvz/r9ryB1xhcBqQFPdTa8az7O4gyVqRcGpZQrcfEy83EEhvBEm6pC81zyk7pnI8tlDTlelI2y1XBM+uJIM6UPdJA4z1fUdJU62Ua2syUmoW+GKud62LjwsRvJqWQeWG4ZKtGcZGAyaDeVIiE4swkSwsoU8LOCmxBFWXG7jO2IpCLX74M/OHg7cD7Muwd7LVqbKHH6CnaQQTtoQN0iI6Rj5jzyHnvHDpH7hP3o/vJ/bxKdTptzUP0n7n+X7G5HyE=</latexit><latexit sha1_base64="1OIxmbCc/a4EtkhLEvGxQfJ4e6A=">AAADm3iclVJbb9MwFHYTLsPcOnhCSOiIFjQ0qOJKaPAwmAQPG+JhSIROatrKcZzWWuIE20ErUf4UP4U3/g1OGomxtQ87kq3P5/Kd408nzBOhjef96Tjutes3bm7dwrfv3L13v7v94JvOCsW4z7IkUych1TwRkvtGmISf5IrTNEz4KDz9UMdHP7jSIpNfzTLnk5TOpYgFo8a6ZtudX/0+LoMwhp/VdAj7MJqS5nlWTQnsQoPDGu8DDhIemzEOQj4XsqRK0WVVKlVhAs+BQBDgVzWyl83lMmpTcKDEfGEmG+orDEG+oNJkaUkqS3L+uY4H7FwbuTxLQAav19bhjX+48gwvLdP3gkYr7ehKu8DwM6PS8qja+SfpO/BebBbvyo1xvz/r9ryB1xhcBqQFPdTa8az7O4gyVqRcGpZQrcfEy83EEhvBEm6pC81zyk7pnI8tlDTlelI2y1XBM+uJIM6UPdJA4z1fUdJU62Ua2syUmoW+GKud62LjwsRvJqWQeWG4ZKtGcZGAyaDeVIiE4swkSwsoU8LOCmxBFWXG7jO2IpCLX74M/OHg7cD7Muwd7LVqbKHH6CnaQQTtoQN0iI6Rj5jzyHnvHDpH7hP3o/vJ/bxKdTptzUP0n7n+X7G5HyE=</latexit>
z3 = W 2a2 + b2 =⇥1 1
⇤ �+⇥�1
⇤=
⇥ ⇤, y = I(z3 > 0) =
⇥ ⇤
<latexit sha1_base64="4x4wKmONk0RqxcJrhKAYXwr6uSw=">AAADanichVJNb9NAEN3YfJQFSkoPgHpZEYOKgMgOh8KhqBIXuBWJkEpxEq3X63jV9drsjhHG8oG/yI1/wIX/wMa1BG0TdaSV3rzZN1+aqJDCgO//6jnutes3bm7dwrfv3N2+19+5/9nkpWZ8zHKZ65OIGi6F4mMQIPlJoTnNIskn0em7VXzylWsjcvUJqoLPMrpUIhGMgqUWO70fnofrMErI92b+ihySyXzUurSZj8hz0uJohQ8JDiVPYIrDiC+FqqnWtGpqrRsckKcksHGu4o7GoRbLFGYbNA0mYZFSBXlWB00YnnPX5SG2l425XgZrJXhjy82V9V5Y6ZeSxjhMKdRVY8cPgX8DndUfmv1/C3tL/GebV3N1Hex5i/7AH/qtkcsg6MAAdXa86P8M45yVGVfAJDVmGvgFzGxiEExym7o0vKDslC751EJFM25mdXsrDXlimZgkubZPAWnZ/xU1zYypssj+zCik5mJsRa6LTUtIXs9qoYoSuGJnhZJSEsjJ6vBILDRnICsLKNPC9kpYSjVlYM8T2yUEF0e+DMaj4Zuh/3E0ODrotrGF9tBjtI8CdICO0Ht0jMaI9X47284D56Hzx911H7l7Z1+dXqfZRefM9f4CM6wQ+A==</latexit><latexit sha1_base64="4x4wKmONk0RqxcJrhKAYXwr6uSw=">AAADanichVJNb9NAEN3YfJQFSkoPgHpZEYOKgMgOh8KhqBIXuBWJkEpxEq3X63jV9drsjhHG8oG/yI1/wIX/wMa1BG0TdaSV3rzZN1+aqJDCgO//6jnutes3bm7dwrfv3N2+19+5/9nkpWZ8zHKZ65OIGi6F4mMQIPlJoTnNIskn0em7VXzylWsjcvUJqoLPMrpUIhGMgqUWO70fnofrMErI92b+ihySyXzUurSZj8hz0uJohQ8JDiVPYIrDiC+FqqnWtGpqrRsckKcksHGu4o7GoRbLFGYbNA0mYZFSBXlWB00YnnPX5SG2l425XgZrJXhjy82V9V5Y6ZeSxjhMKdRVY8cPgX8DndUfmv1/C3tL/GebV3N1Hex5i/7AH/qtkcsg6MAAdXa86P8M45yVGVfAJDVmGvgFzGxiEExym7o0vKDslC751EJFM25mdXsrDXlimZgkubZPAWnZ/xU1zYypssj+zCik5mJsRa6LTUtIXs9qoYoSuGJnhZJSEsjJ6vBILDRnICsLKNPC9kpYSjVlYM8T2yUEF0e+DMaj4Zuh/3E0ODrotrGF9tBjtI8CdICO0Ht0jMaI9X47284D56Hzx911H7l7Z1+dXqfZRefM9f4CM6wQ+A==</latexit><latexit sha1_base64="4x4wKmONk0RqxcJrhKAYXwr6uSw=">AAADanichVJNb9NAEN3YfJQFSkoPgHpZEYOKgMgOh8KhqBIXuBWJkEpxEq3X63jV9drsjhHG8oG/yI1/wIX/wMa1BG0TdaSV3rzZN1+aqJDCgO//6jnutes3bm7dwrfv3N2+19+5/9nkpWZ8zHKZ65OIGi6F4mMQIPlJoTnNIskn0em7VXzylWsjcvUJqoLPMrpUIhGMgqUWO70fnofrMErI92b+ihySyXzUurSZj8hz0uJohQ8JDiVPYIrDiC+FqqnWtGpqrRsckKcksHGu4o7GoRbLFGYbNA0mYZFSBXlWB00YnnPX5SG2l425XgZrJXhjy82V9V5Y6ZeSxjhMKdRVY8cPgX8DndUfmv1/C3tL/GebV3N1Hex5i/7AH/qtkcsg6MAAdXa86P8M45yVGVfAJDVmGvgFzGxiEExym7o0vKDslC751EJFM25mdXsrDXlimZgkubZPAWnZ/xU1zYypssj+zCik5mJsRa6LTUtIXs9qoYoSuGJnhZJSEsjJ6vBILDRnICsLKNPC9kpYSjVlYM8T2yUEF0e+DMaj4Zuh/3E0ODrotrGF9tBjtI8CdICO0Ht0jMaI9X47284D56Hzx911H7l7Z1+dXqfZRefM9f4CM6wQ+A==</latexit>
O112.2 > o )
0 0 0
0 1.5 g
°
, o o
The Multilayer Perceptron
28
Example: XOR Check
x1 0 1 0 1x2 0 0 1 1
(x1 OR x2) AND (x1 NAND x2) 0 1 1 0<latexit sha1_base64="2EOcTsTdKqsK/cnJtMgylgc25pQ=">AAACoXicbVFNT9tAEF0bCtRtIdBjD10Rt6KXyM6FjxOorYR6CAGRghRH0XozSVas19buGBG5/mH9G73137BOjCDASKN9eu/N7O5MnElhMAj+O+7K6pu19Y233rv3Hza3Gts7v02aaw49nspUX8fMgBQKeihQwnWmgSWxhKv45nulX92CNiJVlzjLYJCwiRJjwRlaatj46/teFMNEqIJpzWZlwf9wG6V3NwzpVxrYfDyjyIum1VXUyu2afrAsyXtVeYRwh4gFPbugJbUV3x6pk84Pyy3bOgtybnzat8JVb1Cj+pWe7w8bzaAVzIO+BGENmqSO7rDxLxqlPE9AIZfMmH4YZDiwDVFwCaUX5QYyxm/YBPoWKpaAGRTzEZf0i2VGdJxqmwrpnH1aUbDEmFkSW2fCcGqeaxX5mtbPcXwwKITKcgTFFxeNc0kxpdW+6Eho4ChnFjCuhX0r5VOmGUe7Vc8OIXz+5Zeg124dtoLzdvP4qJ7GBvlEdskeCck+OSanpEt6hDufnZ9OxzlzffeX23UvFlbXqWs+kqVw+/dfgr63</latexit><latexit sha1_base64="2EOcTsTdKqsK/cnJtMgylgc25pQ=">AAACoXicbVFNT9tAEF0bCtRtIdBjD10Rt6KXyM6FjxOorYR6CAGRghRH0XozSVas19buGBG5/mH9G73137BOjCDASKN9eu/N7O5MnElhMAj+O+7K6pu19Y233rv3Hza3Gts7v02aaw49nspUX8fMgBQKeihQwnWmgSWxhKv45nulX92CNiJVlzjLYJCwiRJjwRlaatj46/teFMNEqIJpzWZlwf9wG6V3NwzpVxrYfDyjyIum1VXUyu2afrAsyXtVeYRwh4gFPbugJbUV3x6pk84Pyy3bOgtybnzat8JVb1Cj+pWe7w8bzaAVzIO+BGENmqSO7rDxLxqlPE9AIZfMmH4YZDiwDVFwCaUX5QYyxm/YBPoWKpaAGRTzEZf0i2VGdJxqmwrpnH1aUbDEmFkSW2fCcGqeaxX5mtbPcXwwKITKcgTFFxeNc0kxpdW+6Eho4ChnFjCuhX0r5VOmGUe7Vc8OIXz+5Zeg124dtoLzdvP4qJ7GBvlEdskeCck+OSanpEt6hDufnZ9OxzlzffeX23UvFlbXqWs+kqVw+/dfgr63</latexit><latexit sha1_base64="2EOcTsTdKqsK/cnJtMgylgc25pQ=">AAACoXicbVFNT9tAEF0bCtRtIdBjD10Rt6KXyM6FjxOorYR6CAGRghRH0XozSVas19buGBG5/mH9G73137BOjCDASKN9eu/N7O5MnElhMAj+O+7K6pu19Y233rv3Hza3Gts7v02aaw49nspUX8fMgBQKeihQwnWmgSWxhKv45nulX92CNiJVlzjLYJCwiRJjwRlaatj46/teFMNEqIJpzWZlwf9wG6V3NwzpVxrYfDyjyIum1VXUyu2afrAsyXtVeYRwh4gFPbugJbUV3x6pk84Pyy3bOgtybnzat8JVb1Cj+pWe7w8bzaAVzIO+BGENmqSO7rDxLxqlPE9AIZfMmH4YZDiwDVFwCaUX5QYyxm/YBPoWKpaAGRTzEZf0i2VGdJxqmwrpnH1aUbDEmFkSW2fCcGqeaxX5mtbPcXwwKITKcgTFFxeNc0kxpdW+6Eho4ChnFjCuhX0r5VOmGUe7Vc8OIXz+5Zeg124dtoLzdvP4qJ7GBvlEdskeCck+OSanpEt6hDufnZ9OxzlzffeX23UvFlbXqWs+kqVw+/dfgr63</latexit>
W 1 =
1 1
�1 �1
�, b1 =
0
1.5
�
<latexit sha1_base64="dtFRWrcBCPwkdLpADAsQUDkV7Fo=">AAACfXicbVFNb9QwEHXCVzFfCxx7MexSIWijuBJaOFSqxIVjkVi20iasHGeya9Vxgj1BWkX5GfwxbvyWXupsc4B2n2Tp6c3Ms/0mq7VyGMd/g/DO3Xv3H+w9pI8eP3n6bPT8xXdXNVbCTFa6sueZcKCVgRkq1HBeWxBlpmGeXXzu6/NfYJ2qzDfc1JCWYmVUoaRALy1HvycTOv/B2QmjiYYCFzTJYKVMK6wVm661tqOcHTDOkoQe9eyI0wRMPjTQxKrVGtNDP/+zETmjbZIVLOu2nrstOxr3bjz6wHZZ0clkORrHUbwFu034QMZkwNly9CfJK9mUYFBq4dyCxzWm3hiV1OCtGwe1kBdiBQtPjSjBpe02vo698UrOisr6Y5Bt1X8nWlE6tykz31kKXLubtV7cVVs0WHxMW2XqBsHI64uKRjOsWL8LlisLEvXGEyGt8m9lci2skOg3Rn0I/OaXb5PZcfQpir8ej0+nQxp7ZJ+8Jm8JJ1NySr6QMzIjklwGr4J3wfuQhAfhYRhdt4bBMPOS/IdwegVA8rlr</latexit><latexit sha1_base64="dtFRWrcBCPwkdLpADAsQUDkV7Fo=">AAACfXicbVFNb9QwEHXCVzFfCxx7MexSIWijuBJaOFSqxIVjkVi20iasHGeya9Vxgj1BWkX5GfwxbvyWXupsc4B2n2Tp6c3Ms/0mq7VyGMd/g/DO3Xv3H+w9pI8eP3n6bPT8xXdXNVbCTFa6sueZcKCVgRkq1HBeWxBlpmGeXXzu6/NfYJ2qzDfc1JCWYmVUoaRALy1HvycTOv/B2QmjiYYCFzTJYKVMK6wVm661tqOcHTDOkoQe9eyI0wRMPjTQxKrVGtNDP/+zETmjbZIVLOu2nrstOxr3bjz6wHZZ0clkORrHUbwFu034QMZkwNly9CfJK9mUYFBq4dyCxzWm3hiV1OCtGwe1kBdiBQtPjSjBpe02vo698UrOisr6Y5Bt1X8nWlE6tykz31kKXLubtV7cVVs0WHxMW2XqBsHI64uKRjOsWL8LlisLEvXGEyGt8m9lci2skOg3Rn0I/OaXb5PZcfQpir8ej0+nQxp7ZJ+8Jm8JJ1NySr6QMzIjklwGr4J3wfuQhAfhYRhdt4bBMPOS/IdwegVA8rlr</latexit><latexit sha1_base64="dtFRWrcBCPwkdLpADAsQUDkV7Fo=">AAACfXicbVFNb9QwEHXCVzFfCxx7MexSIWijuBJaOFSqxIVjkVi20iasHGeya9Vxgj1BWkX5GfwxbvyWXupsc4B2n2Tp6c3Ms/0mq7VyGMd/g/DO3Xv3H+w9pI8eP3n6bPT8xXdXNVbCTFa6sueZcKCVgRkq1HBeWxBlpmGeXXzu6/NfYJ2qzDfc1JCWYmVUoaRALy1HvycTOv/B2QmjiYYCFzTJYKVMK6wVm661tqOcHTDOkoQe9eyI0wRMPjTQxKrVGtNDP/+zETmjbZIVLOu2nrstOxr3bjz6wHZZ0clkORrHUbwFu034QMZkwNly9CfJK9mUYFBq4dyCxzWm3hiV1OCtGwe1kBdiBQtPjSjBpe02vo698UrOisr6Y5Bt1X8nWlE6tykz31kKXLubtV7cVVs0WHxMW2XqBsHI64uKRjOsWL8LlisLEvXGEyGt8m9lci2skOg3Rn0I/OaXb5PZcfQpir8ej0+nQxp7ZJ+8Jm8JJ1NySr6QMzIjklwGr4J3wfuQhAfhYRhdt4bBMPOS/IdwegVA8rlr</latexit>
W 2 =⇥1 1
⇤, b2 =
⇥�1
⇤<latexit sha1_base64="NxNR37Ppw14yi1R4phhQWUATR6g=">AAACbHicbVFNb9NAEF0b6McWigvqoVSVViRUHCCycykckCJx4VgkQirFJlqvx8mq67W7O64UWb7xC7nxF7jwF9ikPpQ0I6309N7Mm923aaWkxTD87fmPHj/Z2d3bpwdPnx0+D45efLdlbQSMRalKc5VyC0pqGKNEBVeVAV6kCibp9eeVPrkFY2Wpv+GygqTgcy1zKTg6ahb87Pfp5MeQfWI0VpDjlMYpzKVuuDF82TZCtDRi5yxyOuiso2ls5HyByTvH3tQ8Y7SJ05yl7dppu1FL30fbPGi/Pwt64SBcF3sIog70SFeXs+BXnJWiLkCjUNzaaRRWmDhjlEKBs64tVFxc8zlMHdS8AJs067Ra9sYxGctL445GtmbvTzS8sHZZpK6z4Liwm9qK3KZNa8w/JI3UVY2gxd2ivFYMS7aKnmXSgEC1dIALI91dmVhwwwW6D6IuhGjzyQ/BeDj4OAi/Dnujiy6NPXJKXpO3JCIXZES+kEsyJoL88QLvxHvl/fWP/VP/7K7V97qZl+S/8s//ATYBtkk=</latexit><latexit sha1_base64="NxNR37Ppw14yi1R4phhQWUATR6g=">AAACbHicbVFNb9NAEF0b6McWigvqoVSVViRUHCCycykckCJx4VgkQirFJlqvx8mq67W7O64UWb7xC7nxF7jwF9ikPpQ0I6309N7Mm923aaWkxTD87fmPHj/Z2d3bpwdPnx0+D45efLdlbQSMRalKc5VyC0pqGKNEBVeVAV6kCibp9eeVPrkFY2Wpv+GygqTgcy1zKTg6ahb87Pfp5MeQfWI0VpDjlMYpzKVuuDF82TZCtDRi5yxyOuiso2ls5HyByTvH3tQ8Y7SJ05yl7dppu1FL30fbPGi/Pwt64SBcF3sIog70SFeXs+BXnJWiLkCjUNzaaRRWmDhjlEKBs64tVFxc8zlMHdS8AJs067Ra9sYxGctL445GtmbvTzS8sHZZpK6z4Liwm9qK3KZNa8w/JI3UVY2gxd2ivFYMS7aKnmXSgEC1dIALI91dmVhwwwW6D6IuhGjzyQ/BeDj4OAi/Dnujiy6NPXJKXpO3JCIXZES+kEsyJoL88QLvxHvl/fWP/VP/7K7V97qZl+S/8s//ATYBtkk=</latexit><latexit sha1_base64="NxNR37Ppw14yi1R4phhQWUATR6g=">AAACbHicbVFNb9NAEF0b6McWigvqoVSVViRUHCCycykckCJx4VgkQirFJlqvx8mq67W7O64UWb7xC7nxF7jwF9ikPpQ0I6309N7Mm923aaWkxTD87fmPHj/Z2d3bpwdPnx0+D45efLdlbQSMRalKc5VyC0pqGKNEBVeVAV6kCibp9eeVPrkFY2Wpv+GygqTgcy1zKTg6ahb87Pfp5MeQfWI0VpDjlMYpzKVuuDF82TZCtDRi5yxyOuiso2ls5HyByTvH3tQ8Y7SJ05yl7dppu1FL30fbPGi/Pwt64SBcF3sIog70SFeXs+BXnJWiLkCjUNzaaRRWmDhjlEKBs64tVFxc8zlMHdS8AJs067Ra9sYxGctL445GtmbvTzS8sHZZpK6z4Liwm9qK3KZNa8w/JI3UVY2gxd2ivFYMS7aKnmXSgEC1dIALI91dmVhwwwW6D6IuhGjzyQ/BeDj4OAi/Dnujiy6NPXJKXpO3JCIXZES+kEsyJoL88QLvxHvl/fWP/VP/7K7V97qZl+S/8s//ATYBtkk=</latexit>
z2 = W 1x1+b1 =
1 1
�1 �1
� �+
0
1.5
�=
�, a2 = I(z2 > 0) =
�
<latexit sha1_base64="1OIxmbCc/a4EtkhLEvGxQfJ4e6A=">AAADm3iclVJbb9MwFHYTLsPcOnhCSOiIFjQ0qOJKaPAwmAQPG+JhSIROatrKcZzWWuIE20ErUf4UP4U3/g1OGomxtQ87kq3P5/Kd408nzBOhjef96Tjutes3bm7dwrfv3L13v7v94JvOCsW4z7IkUych1TwRkvtGmISf5IrTNEz4KDz9UMdHP7jSIpNfzTLnk5TOpYgFo8a6ZtudX/0+LoMwhp/VdAj7MJqS5nlWTQnsQoPDGu8DDhIemzEOQj4XsqRK0WVVKlVhAs+BQBDgVzWyl83lMmpTcKDEfGEmG+orDEG+oNJkaUkqS3L+uY4H7FwbuTxLQAav19bhjX+48gwvLdP3gkYr7ehKu8DwM6PS8qja+SfpO/BebBbvyo1xvz/r9ryB1xhcBqQFPdTa8az7O4gyVqRcGpZQrcfEy83EEhvBEm6pC81zyk7pnI8tlDTlelI2y1XBM+uJIM6UPdJA4z1fUdJU62Ua2syUmoW+GKud62LjwsRvJqWQeWG4ZKtGcZGAyaDeVIiE4swkSwsoU8LOCmxBFWXG7jO2IpCLX74M/OHg7cD7Muwd7LVqbKHH6CnaQQTtoQN0iI6Rj5jzyHnvHDpH7hP3o/vJ/bxKdTptzUP0n7n+X7G5HyE=</latexit><latexit sha1_base64="1OIxmbCc/a4EtkhLEvGxQfJ4e6A=">AAADm3iclVJbb9MwFHYTLsPcOnhCSOiIFjQ0qOJKaPAwmAQPG+JhSIROatrKcZzWWuIE20ErUf4UP4U3/g1OGomxtQ87kq3P5/Kd408nzBOhjef96Tjutes3bm7dwrfv3L13v7v94JvOCsW4z7IkUych1TwRkvtGmISf5IrTNEz4KDz9UMdHP7jSIpNfzTLnk5TOpYgFo8a6ZtudX/0+LoMwhp/VdAj7MJqS5nlWTQnsQoPDGu8DDhIemzEOQj4XsqRK0WVVKlVhAs+BQBDgVzWyl83lMmpTcKDEfGEmG+orDEG+oNJkaUkqS3L+uY4H7FwbuTxLQAav19bhjX+48gwvLdP3gkYr7ehKu8DwM6PS8qja+SfpO/BebBbvyo1xvz/r9ryB1xhcBqQFPdTa8az7O4gyVqRcGpZQrcfEy83EEhvBEm6pC81zyk7pnI8tlDTlelI2y1XBM+uJIM6UPdJA4z1fUdJU62Ua2syUmoW+GKud62LjwsRvJqWQeWG4ZKtGcZGAyaDeVIiE4swkSwsoU8LOCmxBFWXG7jO2IpCLX74M/OHg7cD7Muwd7LVqbKHH6CnaQQTtoQN0iI6Rj5jzyHnvHDpH7hP3o/vJ/bxKdTptzUP0n7n+X7G5HyE=</latexit><latexit sha1_base64="1OIxmbCc/a4EtkhLEvGxQfJ4e6A=">AAADm3iclVJbb9MwFHYTLsPcOnhCSOiIFjQ0qOJKaPAwmAQPG+JhSIROatrKcZzWWuIE20ErUf4UP4U3/g1OGomxtQ87kq3P5/Kd408nzBOhjef96Tjutes3bm7dwrfv3L13v7v94JvOCsW4z7IkUych1TwRkvtGmISf5IrTNEz4KDz9UMdHP7jSIpNfzTLnk5TOpYgFo8a6ZtudX/0+LoMwhp/VdAj7MJqS5nlWTQnsQoPDGu8DDhIemzEOQj4XsqRK0WVVKlVhAs+BQBDgVzWyl83lMmpTcKDEfGEmG+orDEG+oNJkaUkqS3L+uY4H7FwbuTxLQAav19bhjX+48gwvLdP3gkYr7ehKu8DwM6PS8qja+SfpO/BebBbvyo1xvz/r9ryB1xhcBqQFPdTa8az7O4gyVqRcGpZQrcfEy83EEhvBEm6pC81zyk7pnI8tlDTlelI2y1XBM+uJIM6UPdJA4z1fUdJU62Ua2syUmoW+GKud62LjwsRvJqWQeWG4ZKtGcZGAyaDeVIiE4swkSwsoU8LOCmxBFWXG7jO2IpCLX74M/OHg7cD7Muwd7LVqbKHH6CnaQQTtoQN0iI6Rj5jzyHnvHDpH7hP3o/vJ/bxKdTptzUP0n7n+X7G5HyE=</latexit>
z3 = W 2a2 + b2 =⇥1 1
⇤ �+⇥�1
⇤=
⇥ ⇤, y = I(z3 > 0) =
⇥ ⇤
<latexit sha1_base64="4x4wKmONk0RqxcJrhKAYXwr6uSw=">AAADanichVJNb9NAEN3YfJQFSkoPgHpZEYOKgMgOh8KhqBIXuBWJkEpxEq3X63jV9drsjhHG8oG/yI1/wIX/wMa1BG0TdaSV3rzZN1+aqJDCgO//6jnutes3bm7dwrfv3N2+19+5/9nkpWZ8zHKZ65OIGi6F4mMQIPlJoTnNIskn0em7VXzylWsjcvUJqoLPMrpUIhGMgqUWO70fnofrMErI92b+ihySyXzUurSZj8hz0uJohQ8JDiVPYIrDiC+FqqnWtGpqrRsckKcksHGu4o7GoRbLFGYbNA0mYZFSBXlWB00YnnPX5SG2l425XgZrJXhjy82V9V5Y6ZeSxjhMKdRVY8cPgX8DndUfmv1/C3tL/GebV3N1Hex5i/7AH/qtkcsg6MAAdXa86P8M45yVGVfAJDVmGvgFzGxiEExym7o0vKDslC751EJFM25mdXsrDXlimZgkubZPAWnZ/xU1zYypssj+zCik5mJsRa6LTUtIXs9qoYoSuGJnhZJSEsjJ6vBILDRnICsLKNPC9kpYSjVlYM8T2yUEF0e+DMaj4Zuh/3E0ODrotrGF9tBjtI8CdICO0Ht0jMaI9X47284D56Hzx911H7l7Z1+dXqfZRefM9f4CM6wQ+A==</latexit><latexit sha1_base64="4x4wKmONk0RqxcJrhKAYXwr6uSw=">AAADanichVJNb9NAEN3YfJQFSkoPgHpZEYOKgMgOh8KhqBIXuBWJkEpxEq3X63jV9drsjhHG8oG/yI1/wIX/wMa1BG0TdaSV3rzZN1+aqJDCgO//6jnutes3bm7dwrfv3N2+19+5/9nkpWZ8zHKZ65OIGi6F4mMQIPlJoTnNIskn0em7VXzylWsjcvUJqoLPMrpUIhGMgqUWO70fnofrMErI92b+ihySyXzUurSZj8hz0uJohQ8JDiVPYIrDiC+FqqnWtGpqrRsckKcksHGu4o7GoRbLFGYbNA0mYZFSBXlWB00YnnPX5SG2l425XgZrJXhjy82V9V5Y6ZeSxjhMKdRVY8cPgX8DndUfmv1/C3tL/GebV3N1Hex5i/7AH/qtkcsg6MAAdXa86P8M45yVGVfAJDVmGvgFzGxiEExym7o0vKDslC751EJFM25mdXsrDXlimZgkubZPAWnZ/xU1zYypssj+zCik5mJsRa6LTUtIXs9qoYoSuGJnhZJSEsjJ6vBILDRnICsLKNPC9kpYSjVlYM8T2yUEF0e+DMaj4Zuh/3E0ODrotrGF9tBjtI8CdICO0Ht0jMaI9X47284D56Hzx911H7l7Z1+dXqfZRefM9f4CM6wQ+A==</latexit><latexit sha1_base64="4x4wKmONk0RqxcJrhKAYXwr6uSw=">AAADanichVJNb9NAEN3YfJQFSkoPgHpZEYOKgMgOh8KhqBIXuBWJkEpxEq3X63jV9drsjhHG8oG/yI1/wIX/wMa1BG0TdaSV3rzZN1+aqJDCgO//6jnutes3bm7dwrfv3N2+19+5/9nkpWZ8zHKZ65OIGi6F4mMQIPlJoTnNIskn0em7VXzylWsjcvUJqoLPMrpUIhGMgqUWO70fnofrMErI92b+ihySyXzUurSZj8hz0uJohQ8JDiVPYIrDiC+FqqnWtGpqrRsckKcksHGu4o7GoRbLFGYbNA0mYZFSBXlWB00YnnPX5SG2l425XgZrJXhjy82V9V5Y6ZeSxjhMKdRVY8cPgX8DndUfmv1/C3tL/GebV3N1Hex5i/7AH/qtkcsg6MAAdXa86P8M45yVGVfAJDVmGvgFzGxiEExym7o0vKDslC751EJFM25mdXsrDXlimZgkubZPAWnZ/xU1zYypssj+zCik5mJsRa6LTUtIXs9qoYoSuGJnhZJSEsjJ6vBILDRnICsLKNPC9kpYSjVlYM8T2yUEF0e+DMaj4Zuh/3E0ODrotrGF9tBjtI8CdICO0Ht0jMaI9X47284D56Hzx911H7l7Z1+dXqfZRefM9f4CM6wQ+A==</latexit>
O
1 c (0 05 1
'
, 1 1
The Multilayer Perceptron
29
Example: XOR Check
x1 0 1 0 1x2 0 0 1 1
(x1 OR x2) AND (x1 NAND x2) 0 1 1 0<latexit sha1_base64="2EOcTsTdKqsK/cnJtMgylgc25pQ=">AAACoXicbVFNT9tAEF0bCtRtIdBjD10Rt6KXyM6FjxOorYR6CAGRghRH0XozSVas19buGBG5/mH9G73137BOjCDASKN9eu/N7O5MnElhMAj+O+7K6pu19Y233rv3Hza3Gts7v02aaw49nspUX8fMgBQKeihQwnWmgSWxhKv45nulX92CNiJVlzjLYJCwiRJjwRlaatj46/teFMNEqIJpzWZlwf9wG6V3NwzpVxrYfDyjyIum1VXUyu2afrAsyXtVeYRwh4gFPbugJbUV3x6pk84Pyy3bOgtybnzat8JVb1Cj+pWe7w8bzaAVzIO+BGENmqSO7rDxLxqlPE9AIZfMmH4YZDiwDVFwCaUX5QYyxm/YBPoWKpaAGRTzEZf0i2VGdJxqmwrpnH1aUbDEmFkSW2fCcGqeaxX5mtbPcXwwKITKcgTFFxeNc0kxpdW+6Eho4ChnFjCuhX0r5VOmGUe7Vc8OIXz+5Zeg124dtoLzdvP4qJ7GBvlEdskeCck+OSanpEt6hDufnZ9OxzlzffeX23UvFlbXqWs+kqVw+/dfgr63</latexit><latexit sha1_base64="2EOcTsTdKqsK/cnJtMgylgc25pQ=">AAACoXicbVFNT9tAEF0bCtRtIdBjD10Rt6KXyM6FjxOorYR6CAGRghRH0XozSVas19buGBG5/mH9G73137BOjCDASKN9eu/N7O5MnElhMAj+O+7K6pu19Y233rv3Hza3Gts7v02aaw49nspUX8fMgBQKeihQwnWmgSWxhKv45nulX92CNiJVlzjLYJCwiRJjwRlaatj46/teFMNEqIJpzWZlwf9wG6V3NwzpVxrYfDyjyIum1VXUyu2afrAsyXtVeYRwh4gFPbugJbUV3x6pk84Pyy3bOgtybnzat8JVb1Cj+pWe7w8bzaAVzIO+BGENmqSO7rDxLxqlPE9AIZfMmH4YZDiwDVFwCaUX5QYyxm/YBPoWKpaAGRTzEZf0i2VGdJxqmwrpnH1aUbDEmFkSW2fCcGqeaxX5mtbPcXwwKITKcgTFFxeNc0kxpdW+6Eho4ChnFjCuhX0r5VOmGUe7Vc8OIXz+5Zeg124dtoLzdvP4qJ7GBvlEdskeCck+OSanpEt6hDufnZ9OxzlzffeX23UvFlbXqWs+kqVw+/dfgr63</latexit><latexit sha1_base64="2EOcTsTdKqsK/cnJtMgylgc25pQ=">AAACoXicbVFNT9tAEF0bCtRtIdBjD10Rt6KXyM6FjxOorYR6CAGRghRH0XozSVas19buGBG5/mH9G73137BOjCDASKN9eu/N7O5MnElhMAj+O+7K6pu19Y233rv3Hza3Gts7v02aaw49nspUX8fMgBQKeihQwnWmgSWxhKv45nulX92CNiJVlzjLYJCwiRJjwRlaatj46/teFMNEqIJpzWZlwf9wG6V3NwzpVxrYfDyjyIum1VXUyu2afrAsyXtVeYRwh4gFPbugJbUV3x6pk84Pyy3bOgtybnzat8JVb1Cj+pWe7w8bzaAVzIO+BGENmqSO7rDxLxqlPE9AIZfMmH4YZDiwDVFwCaUX5QYyxm/YBPoWKpaAGRTzEZf0i2VGdJxqmwrpnH1aUbDEmFkSW2fCcGqeaxX5mtbPcXwwKITKcgTFFxeNc0kxpdW+6Eho4ChnFjCuhX0r5VOmGUe7Vc8OIXz+5Zeg124dtoLzdvP4qJ7GBvlEdskeCck+OSanpEt6hDufnZ9OxzlzffeX23UvFlbXqWs+kqVw+/dfgr63</latexit>
W 1 =
1 1
�1 �1
�, b1 =
0
1.5
�
<latexit sha1_base64="dtFRWrcBCPwkdLpADAsQUDkV7Fo=">AAACfXicbVFNb9QwEHXCVzFfCxx7MexSIWijuBJaOFSqxIVjkVi20iasHGeya9Vxgj1BWkX5GfwxbvyWXupsc4B2n2Tp6c3Ms/0mq7VyGMd/g/DO3Xv3H+w9pI8eP3n6bPT8xXdXNVbCTFa6sueZcKCVgRkq1HBeWxBlpmGeXXzu6/NfYJ2qzDfc1JCWYmVUoaRALy1HvycTOv/B2QmjiYYCFzTJYKVMK6wVm661tqOcHTDOkoQe9eyI0wRMPjTQxKrVGtNDP/+zETmjbZIVLOu2nrstOxr3bjz6wHZZ0clkORrHUbwFu034QMZkwNly9CfJK9mUYFBq4dyCxzWm3hiV1OCtGwe1kBdiBQtPjSjBpe02vo698UrOisr6Y5Bt1X8nWlE6tykz31kKXLubtV7cVVs0WHxMW2XqBsHI64uKRjOsWL8LlisLEvXGEyGt8m9lci2skOg3Rn0I/OaXb5PZcfQpir8ej0+nQxp7ZJ+8Jm8JJ1NySr6QMzIjklwGr4J3wfuQhAfhYRhdt4bBMPOS/IdwegVA8rlr</latexit><latexit sha1_base64="dtFRWrcBCPwkdLpADAsQUDkV7Fo=">AAACfXicbVFNb9QwEHXCVzFfCxx7MexSIWijuBJaOFSqxIVjkVi20iasHGeya9Vxgj1BWkX5GfwxbvyWXupsc4B2n2Tp6c3Ms/0mq7VyGMd/g/DO3Xv3H+w9pI8eP3n6bPT8xXdXNVbCTFa6sueZcKCVgRkq1HBeWxBlpmGeXXzu6/NfYJ2qzDfc1JCWYmVUoaRALy1HvycTOv/B2QmjiYYCFzTJYKVMK6wVm661tqOcHTDOkoQe9eyI0wRMPjTQxKrVGtNDP/+zETmjbZIVLOu2nrstOxr3bjz6wHZZ0clkORrHUbwFu034QMZkwNly9CfJK9mUYFBq4dyCxzWm3hiV1OCtGwe1kBdiBQtPjSjBpe02vo698UrOisr6Y5Bt1X8nWlE6tykz31kKXLubtV7cVVs0WHxMW2XqBsHI64uKRjOsWL8LlisLEvXGEyGt8m9lci2skOg3Rn0I/OaXb5PZcfQpir8ej0+nQxp7ZJ+8Jm8JJ1NySr6QMzIjklwGr4J3wfuQhAfhYRhdt4bBMPOS/IdwegVA8rlr</latexit><latexit sha1_base64="dtFRWrcBCPwkdLpADAsQUDkV7Fo=">AAACfXicbVFNb9QwEHXCVzFfCxx7MexSIWijuBJaOFSqxIVjkVi20iasHGeya9Vxgj1BWkX5GfwxbvyWXupsc4B2n2Tp6c3Ms/0mq7VyGMd/g/DO3Xv3H+w9pI8eP3n6bPT8xXdXNVbCTFa6sueZcKCVgRkq1HBeWxBlpmGeXXzu6/NfYJ2qzDfc1JCWYmVUoaRALy1HvycTOv/B2QmjiYYCFzTJYKVMK6wVm661tqOcHTDOkoQe9eyI0wRMPjTQxKrVGtNDP/+zETmjbZIVLOu2nrstOxr3bjz6wHZZ0clkORrHUbwFu034QMZkwNly9CfJK9mUYFBq4dyCxzWm3hiV1OCtGwe1kBdiBQtPjSjBpe02vo698UrOisr6Y5Bt1X8nWlE6tykz31kKXLubtV7cVVs0WHxMW2XqBsHI64uKRjOsWL8LlisLEvXGEyGt8m9lci2skOg3Rn0I/OaXb5PZcfQpir8ej0+nQxp7ZJ+8Jm8JJ1NySr6QMzIjklwGr4J3wfuQhAfhYRhdt4bBMPOS/IdwegVA8rlr</latexit>
W 2 =⇥1 1
⇤, b2 =
⇥�1
⇤<latexit sha1_base64="NxNR37Ppw14yi1R4phhQWUATR6g=">AAACbHicbVFNb9NAEF0b6McWigvqoVSVViRUHCCycykckCJx4VgkQirFJlqvx8mq67W7O64UWb7xC7nxF7jwF9ikPpQ0I6309N7Mm923aaWkxTD87fmPHj/Z2d3bpwdPnx0+D45efLdlbQSMRalKc5VyC0pqGKNEBVeVAV6kCibp9eeVPrkFY2Wpv+GygqTgcy1zKTg6ahb87Pfp5MeQfWI0VpDjlMYpzKVuuDF82TZCtDRi5yxyOuiso2ls5HyByTvH3tQ8Y7SJ05yl7dppu1FL30fbPGi/Pwt64SBcF3sIog70SFeXs+BXnJWiLkCjUNzaaRRWmDhjlEKBs64tVFxc8zlMHdS8AJs067Ra9sYxGctL445GtmbvTzS8sHZZpK6z4Liwm9qK3KZNa8w/JI3UVY2gxd2ivFYMS7aKnmXSgEC1dIALI91dmVhwwwW6D6IuhGjzyQ/BeDj4OAi/Dnujiy6NPXJKXpO3JCIXZES+kEsyJoL88QLvxHvl/fWP/VP/7K7V97qZl+S/8s//ATYBtkk=</latexit><latexit sha1_base64="NxNR37Ppw14yi1R4phhQWUATR6g=">AAACbHicbVFNb9NAEF0b6McWigvqoVSVViRUHCCycykckCJx4VgkQirFJlqvx8mq67W7O64UWb7xC7nxF7jwF9ikPpQ0I6309N7Mm923aaWkxTD87fmPHj/Z2d3bpwdPnx0+D45efLdlbQSMRalKc5VyC0pqGKNEBVeVAV6kCibp9eeVPrkFY2Wpv+GygqTgcy1zKTg6ahb87Pfp5MeQfWI0VpDjlMYpzKVuuDF82TZCtDRi5yxyOuiso2ls5HyByTvH3tQ8Y7SJ05yl7dppu1FL30fbPGi/Pwt64SBcF3sIog70SFeXs+BXnJWiLkCjUNzaaRRWmDhjlEKBs64tVFxc8zlMHdS8AJs067Ra9sYxGctL445GtmbvTzS8sHZZpK6z4Liwm9qK3KZNa8w/JI3UVY2gxd2ivFYMS7aKnmXSgEC1dIALI91dmVhwwwW6D6IuhGjzyQ/BeDj4OAi/Dnujiy6NPXJKXpO3JCIXZES+kEsyJoL88QLvxHvl/fWP/VP/7K7V97qZl+S/8s//ATYBtkk=</latexit><latexit sha1_base64="NxNR37Ppw14yi1R4phhQWUATR6g=">AAACbHicbVFNb9NAEF0b6McWigvqoVSVViRUHCCycykckCJx4VgkQirFJlqvx8mq67W7O64UWb7xC7nxF7jwF9ikPpQ0I6309N7Mm923aaWkxTD87fmPHj/Z2d3bpwdPnx0+D45efLdlbQSMRalKc5VyC0pqGKNEBVeVAV6kCibp9eeVPrkFY2Wpv+GygqTgcy1zKTg6ahb87Pfp5MeQfWI0VpDjlMYpzKVuuDF82TZCtDRi5yxyOuiso2ls5HyByTvH3tQ8Y7SJ05yl7dppu1FL30fbPGi/Pwt64SBcF3sIog70SFeXs+BXnJWiLkCjUNzaaRRWmDhjlEKBs64tVFxc8zlMHdS8AJs067Ra9sYxGctL445GtmbvTzS8sHZZpK6z4Liwm9qK3KZNa8w/JI3UVY2gxd2ivFYMS7aKnmXSgEC1dIALI91dmVhwwwW6D6IuhGjzyQ/BeDj4OAi/Dnujiy6NPXJKXpO3JCIXZES+kEsyJoL88QLvxHvl/fWP/VP/7K7V97qZl+S/8s//ATYBtkk=</latexit>
z2 = W 1x1+b1 =
1 1
�1 �1
� �+
0
1.5
�=
�, a2 = I(z2 > 0) =
�
<latexit sha1_base64="1OIxmbCc/a4EtkhLEvGxQfJ4e6A=">AAADm3iclVJbb9MwFHYTLsPcOnhCSOiIFjQ0qOJKaPAwmAQPG+JhSIROatrKcZzWWuIE20ErUf4UP4U3/g1OGomxtQ87kq3P5/Kd408nzBOhjef96Tjutes3bm7dwrfv3L13v7v94JvOCsW4z7IkUych1TwRkvtGmISf5IrTNEz4KDz9UMdHP7jSIpNfzTLnk5TOpYgFo8a6ZtudX/0+LoMwhp/VdAj7MJqS5nlWTQnsQoPDGu8DDhIemzEOQj4XsqRK0WVVKlVhAs+BQBDgVzWyl83lMmpTcKDEfGEmG+orDEG+oNJkaUkqS3L+uY4H7FwbuTxLQAav19bhjX+48gwvLdP3gkYr7ehKu8DwM6PS8qja+SfpO/BebBbvyo1xvz/r9ryB1xhcBqQFPdTa8az7O4gyVqRcGpZQrcfEy83EEhvBEm6pC81zyk7pnI8tlDTlelI2y1XBM+uJIM6UPdJA4z1fUdJU62Ua2syUmoW+GKud62LjwsRvJqWQeWG4ZKtGcZGAyaDeVIiE4swkSwsoU8LOCmxBFWXG7jO2IpCLX74M/OHg7cD7Muwd7LVqbKHH6CnaQQTtoQN0iI6Rj5jzyHnvHDpH7hP3o/vJ/bxKdTptzUP0n7n+X7G5HyE=</latexit><latexit sha1_base64="1OIxmbCc/a4EtkhLEvGxQfJ4e6A=">AAADm3iclVJbb9MwFHYTLsPcOnhCSOiIFjQ0qOJKaPAwmAQPG+JhSIROatrKcZzWWuIE20ErUf4UP4U3/g1OGomxtQ87kq3P5/Kd408nzBOhjef96Tjutes3bm7dwrfv3L13v7v94JvOCsW4z7IkUych1TwRkvtGmISf5IrTNEz4KDz9UMdHP7jSIpNfzTLnk5TOpYgFo8a6ZtudX/0+LoMwhp/VdAj7MJqS5nlWTQnsQoPDGu8DDhIemzEOQj4XsqRK0WVVKlVhAs+BQBDgVzWyl83lMmpTcKDEfGEmG+orDEG+oNJkaUkqS3L+uY4H7FwbuTxLQAav19bhjX+48gwvLdP3gkYr7ehKu8DwM6PS8qja+SfpO/BebBbvyo1xvz/r9ryB1xhcBqQFPdTa8az7O4gyVqRcGpZQrcfEy83EEhvBEm6pC81zyk7pnI8tlDTlelI2y1XBM+uJIM6UPdJA4z1fUdJU62Ua2syUmoW+GKud62LjwsRvJqWQeWG4ZKtGcZGAyaDeVIiE4swkSwsoU8LOCmxBFWXG7jO2IpCLX74M/OHg7cD7Muwd7LVqbKHH6CnaQQTtoQN0iI6Rj5jzyHnvHDpH7hP3o/vJ/bxKdTptzUP0n7n+X7G5HyE=</latexit><latexit sha1_base64="1OIxmbCc/a4EtkhLEvGxQfJ4e6A=">AAADm3iclVJbb9MwFHYTLsPcOnhCSOiIFjQ0qOJKaPAwmAQPG+JhSIROatrKcZzWWuIE20ErUf4UP4U3/g1OGomxtQ87kq3P5/Kd408nzBOhjef96Tjutes3bm7dwrfv3L13v7v94JvOCsW4z7IkUych1TwRkvtGmISf5IrTNEz4KDz9UMdHP7jSIpNfzTLnk5TOpYgFo8a6ZtudX/0+LoMwhp/VdAj7MJqS5nlWTQnsQoPDGu8DDhIemzEOQj4XsqRK0WVVKlVhAs+BQBDgVzWyl83lMmpTcKDEfGEmG+orDEG+oNJkaUkqS3L+uY4H7FwbuTxLQAav19bhjX+48gwvLdP3gkYr7ehKu8DwM6PS8qja+SfpO/BebBbvyo1xvz/r9ryB1xhcBqQFPdTa8az7O4gyVqRcGpZQrcfEy83EEhvBEm6pC81zyk7pnI8tlDTlelI2y1XBM+uJIM6UPdJA4z1fUdJU62Ua2syUmoW+GKud62LjwsRvJqWQeWG4ZKtGcZGAyaDeVIiE4swkSwsoU8LOCmxBFWXG7jO2IpCLX74M/OHg7cD7Muwd7LVqbKHH6CnaQQTtoQN0iI6Rj5jzyHnvHDpH7hP3o/vJ/bxKdTptzUP0n7n+X7G5HyE=</latexit>
z3 = W 2a2 + b2 =⇥1 1
⇤ �+⇥�1
⇤=
⇥ ⇤, y = I(z3 > 0) =
⇥ ⇤
<latexit sha1_base64="4x4wKmONk0RqxcJrhKAYXwr6uSw=">AAADanichVJNb9NAEN3YfJQFSkoPgHpZEYOKgMgOh8KhqBIXuBWJkEpxEq3X63jV9drsjhHG8oG/yI1/wIX/wMa1BG0TdaSV3rzZN1+aqJDCgO//6jnutes3bm7dwrfv3N2+19+5/9nkpWZ8zHKZ65OIGi6F4mMQIPlJoTnNIskn0em7VXzylWsjcvUJqoLPMrpUIhGMgqUWO70fnofrMErI92b+ihySyXzUurSZj8hz0uJohQ8JDiVPYIrDiC+FqqnWtGpqrRsckKcksHGu4o7GoRbLFGYbNA0mYZFSBXlWB00YnnPX5SG2l425XgZrJXhjy82V9V5Y6ZeSxjhMKdRVY8cPgX8DndUfmv1/C3tL/GebV3N1Hex5i/7AH/qtkcsg6MAAdXa86P8M45yVGVfAJDVmGvgFzGxiEExym7o0vKDslC751EJFM25mdXsrDXlimZgkubZPAWnZ/xU1zYypssj+zCik5mJsRa6LTUtIXs9qoYoSuGJnhZJSEsjJ6vBILDRnICsLKNPC9kpYSjVlYM8T2yUEF0e+DMaj4Zuh/3E0ODrotrGF9tBjtI8CdICO0Ht0jMaI9X47284D56Hzx911H7l7Z1+dXqfZRefM9f4CM6wQ+A==</latexit><latexit sha1_base64="4x4wKmONk0RqxcJrhKAYXwr6uSw=">AAADanichVJNb9NAEN3YfJQFSkoPgHpZEYOKgMgOh8KhqBIXuBWJkEpxEq3X63jV9drsjhHG8oG/yI1/wIX/wMa1BG0TdaSV3rzZN1+aqJDCgO//6jnutes3bm7dwrfv3N2+19+5/9nkpWZ8zHKZ65OIGi6F4mMQIPlJoTnNIskn0em7VXzylWsjcvUJqoLPMrpUIhGMgqUWO70fnofrMErI92b+ihySyXzUurSZj8hz0uJohQ8JDiVPYIrDiC+FqqnWtGpqrRsckKcksHGu4o7GoRbLFGYbNA0mYZFSBXlWB00YnnPX5SG2l425XgZrJXhjy82V9V5Y6ZeSxjhMKdRVY8cPgX8DndUfmv1/C3tL/GebV3N1Hex5i/7AH/qtkcsg6MAAdXa86P8M45yVGVfAJDVmGvgFzGxiEExym7o0vKDslC751EJFM25mdXsrDXlimZgkubZPAWnZ/xU1zYypssj+zCik5mJsRa6LTUtIXs9qoYoSuGJnhZJSEsjJ6vBILDRnICsLKNPC9kpYSjVlYM8T2yUEF0e+DMaj4Zuh/3E0ODrotrGF9tBjtI8CdICO0Ht0jMaI9X47284D56Hzx911H7l7Z1+dXqfZRefM9f4CM6wQ+A==</latexit><latexit sha1_base64="4x4wKmONk0RqxcJrhKAYXwr6uSw=">AAADanichVJNb9NAEN3YfJQFSkoPgHpZEYOKgMgOh8KhqBIXuBWJkEpxEq3X63jV9drsjhHG8oG/yI1/wIX/wMa1BG0TdaSV3rzZN1+aqJDCgO//6jnutes3bm7dwrfv3N2+19+5/9nkpWZ8zHKZ65OIGi6F4mMQIPlJoTnNIskn0em7VXzylWsjcvUJqoLPMrpUIhGMgqUWO70fnofrMErI92b+ihySyXzUurSZj8hz0uJohQ8JDiVPYIrDiC+FqqnWtGpqrRsckKcksHGu4o7GoRbLFGYbNA0mYZFSBXlWB00YnnPX5SG2l425XgZrJXhjy82V9V5Y6ZeSxjhMKdRVY8cPgX8DndUfmv1/C3tL/GebV3N1Hex5i/7AH/qtkcsg6MAAdXa86P8M45yVGVfAJDVmGvgFzGxiEExym7o0vKDslC751EJFM25mdXsrDXlimZgkubZPAWnZ/xU1zYypssj+zCik5mJsRa6LTUtIXs9qoYoSuGJnhZJSEsjJ6vBILDRnICsLKNPC9kpYSjVlYM8T2yUEF0e+DMaj4Zuh/3E0ODrotrGF9tBjtI8CdICO0Ht0jMaI9X47284D56Hzx911H7l7Z1+dXqfZRefM9f4CM6wQ+A==</latexit>
The Multilayer Perceptron
30
Example: XOR Check
x1 0 1 0 1x2 0 0 1 1
(x1 OR x2) AND (x1 NAND x2) 0 1 1 0<latexit sha1_base64="2EOcTsTdKqsK/cnJtMgylgc25pQ=">AAACoXicbVFNT9tAEF0bCtRtIdBjD10Rt6KXyM6FjxOorYR6CAGRghRH0XozSVas19buGBG5/mH9G73137BOjCDASKN9eu/N7O5MnElhMAj+O+7K6pu19Y233rv3Hza3Gts7v02aaw49nspUX8fMgBQKeihQwnWmgSWxhKv45nulX92CNiJVlzjLYJCwiRJjwRlaatj46/teFMNEqIJpzWZlwf9wG6V3NwzpVxrYfDyjyIum1VXUyu2afrAsyXtVeYRwh4gFPbugJbUV3x6pk84Pyy3bOgtybnzat8JVb1Cj+pWe7w8bzaAVzIO+BGENmqSO7rDxLxqlPE9AIZfMmH4YZDiwDVFwCaUX5QYyxm/YBPoWKpaAGRTzEZf0i2VGdJxqmwrpnH1aUbDEmFkSW2fCcGqeaxX5mtbPcXwwKITKcgTFFxeNc0kxpdW+6Eho4ChnFjCuhX0r5VOmGUe7Vc8OIXz+5Zeg124dtoLzdvP4qJ7GBvlEdskeCck+OSanpEt6hDufnZ9OxzlzffeX23UvFlbXqWs+kqVw+/dfgr63</latexit><latexit sha1_base64="2EOcTsTdKqsK/cnJtMgylgc25pQ=">AAACoXicbVFNT9tAEF0bCtRtIdBjD10Rt6KXyM6FjxOorYR6CAGRghRH0XozSVas19buGBG5/mH9G73137BOjCDASKN9eu/N7O5MnElhMAj+O+7K6pu19Y233rv3Hza3Gts7v02aaw49nspUX8fMgBQKeihQwnWmgSWxhKv45nulX92CNiJVlzjLYJCwiRJjwRlaatj46/teFMNEqIJpzWZlwf9wG6V3NwzpVxrYfDyjyIum1VXUyu2afrAsyXtVeYRwh4gFPbugJbUV3x6pk84Pyy3bOgtybnzat8JVb1Cj+pWe7w8bzaAVzIO+BGENmqSO7rDxLxqlPE9AIZfMmH4YZDiwDVFwCaUX5QYyxm/YBPoWKpaAGRTzEZf0i2VGdJxqmwrpnH1aUbDEmFkSW2fCcGqeaxX5mtbPcXwwKITKcgTFFxeNc0kxpdW+6Eho4ChnFjCuhX0r5VOmGUe7Vc8OIXz+5Zeg124dtoLzdvP4qJ7GBvlEdskeCck+OSanpEt6hDufnZ9OxzlzffeX23UvFlbXqWs+kqVw+/dfgr63</latexit><latexit sha1_base64="2EOcTsTdKqsK/cnJtMgylgc25pQ=">AAACoXicbVFNT9tAEF0bCtRtIdBjD10Rt6KXyM6FjxOorYR6CAGRghRH0XozSVas19buGBG5/mH9G73137BOjCDASKN9eu/N7O5MnElhMAj+O+7K6pu19Y233rv3Hza3Gts7v02aaw49nspUX8fMgBQKeihQwnWmgSWxhKv45nulX92CNiJVlzjLYJCwiRJjwRlaatj46/teFMNEqIJpzWZlwf9wG6V3NwzpVxrYfDyjyIum1VXUyu2afrAsyXtVeYRwh4gFPbugJbUV3x6pk84Pyy3bOgtybnzat8JVb1Cj+pWe7w8bzaAVzIO+BGENmqSO7rDxLxqlPE9AIZfMmH4YZDiwDVFwCaUX5QYyxm/YBPoWKpaAGRTzEZf0i2VGdJxqmwrpnH1aUbDEmFkSW2fCcGqeaxX5mtbPcXwwKITKcgTFFxeNc0kxpdW+6Eho4ChnFjCuhX0r5VOmGUe7Vc8OIXz+5Zeg124dtoLzdvP4qJ7GBvlEdskeCck+OSanpEt6hDufnZ9OxzlzffeX23UvFlbXqWs+kqVw+/dfgr63</latexit>
W 1 =
1 1
�1 �1
�, b1 =
0
1.5
�
<latexit sha1_base64="dtFRWrcBCPwkdLpADAsQUDkV7Fo=">AAACfXicbVFNb9QwEHXCVzFfCxx7MexSIWijuBJaOFSqxIVjkVi20iasHGeya9Vxgj1BWkX5GfwxbvyWXupsc4B2n2Tp6c3Ms/0mq7VyGMd/g/DO3Xv3H+w9pI8eP3n6bPT8xXdXNVbCTFa6sueZcKCVgRkq1HBeWxBlpmGeXXzu6/NfYJ2qzDfc1JCWYmVUoaRALy1HvycTOv/B2QmjiYYCFzTJYKVMK6wVm661tqOcHTDOkoQe9eyI0wRMPjTQxKrVGtNDP/+zETmjbZIVLOu2nrstOxr3bjz6wHZZ0clkORrHUbwFu034QMZkwNly9CfJK9mUYFBq4dyCxzWm3hiV1OCtGwe1kBdiBQtPjSjBpe02vo698UrOisr6Y5Bt1X8nWlE6tykz31kKXLubtV7cVVs0WHxMW2XqBsHI64uKRjOsWL8LlisLEvXGEyGt8m9lci2skOg3Rn0I/OaXb5PZcfQpir8ej0+nQxp7ZJ+8Jm8JJ1NySr6QMzIjklwGr4J3wfuQhAfhYRhdt4bBMPOS/IdwegVA8rlr</latexit><latexit sha1_base64="dtFRWrcBCPwkdLpADAsQUDkV7Fo=">AAACfXicbVFNb9QwEHXCVzFfCxx7MexSIWijuBJaOFSqxIVjkVi20iasHGeya9Vxgj1BWkX5GfwxbvyWXupsc4B2n2Tp6c3Ms/0mq7VyGMd/g/DO3Xv3H+w9pI8eP3n6bPT8xXdXNVbCTFa6sueZcKCVgRkq1HBeWxBlpmGeXXzu6/NfYJ2qzDfc1JCWYmVUoaRALy1HvycTOv/B2QmjiYYCFzTJYKVMK6wVm661tqOcHTDOkoQe9eyI0wRMPjTQxKrVGtNDP/+zETmjbZIVLOu2nrstOxr3bjz6wHZZ0clkORrHUbwFu034QMZkwNly9CfJK9mUYFBq4dyCxzWm3hiV1OCtGwe1kBdiBQtPjSjBpe02vo698UrOisr6Y5Bt1X8nWlE6tykz31kKXLubtV7cVVs0WHxMW2XqBsHI64uKRjOsWL8LlisLEvXGEyGt8m9lci2skOg3Rn0I/OaXb5PZcfQpir8ej0+nQxp7ZJ+8Jm8JJ1NySr6QMzIjklwGr4J3wfuQhAfhYRhdt4bBMPOS/IdwegVA8rlr</latexit><latexit sha1_base64="dtFRWrcBCPwkdLpADAsQUDkV7Fo=">AAACfXicbVFNb9QwEHXCVzFfCxx7MexSIWijuBJaOFSqxIVjkVi20iasHGeya9Vxgj1BWkX5GfwxbvyWXupsc4B2n2Tp6c3Ms/0mq7VyGMd/g/DO3Xv3H+w9pI8eP3n6bPT8xXdXNVbCTFa6sueZcKCVgRkq1HBeWxBlpmGeXXzu6/NfYJ2qzDfc1JCWYmVUoaRALy1HvycTOv/B2QmjiYYCFzTJYKVMK6wVm661tqOcHTDOkoQe9eyI0wRMPjTQxKrVGtNDP/+zETmjbZIVLOu2nrstOxr3bjz6wHZZ0clkORrHUbwFu034QMZkwNly9CfJK9mUYFBq4dyCxzWm3hiV1OCtGwe1kBdiBQtPjSjBpe02vo698UrOisr6Y5Bt1X8nWlE6tykz31kKXLubtV7cVVs0WHxMW2XqBsHI64uKRjOsWL8LlisLEvXGEyGt8m9lci2skOg3Rn0I/OaXb5PZcfQpir8ej0+nQxp7ZJ+8Jm8JJ1NySr6QMzIjklwGr4J3wfuQhAfhYRhdt4bBMPOS/IdwegVA8rlr</latexit>
W 2 =⇥1 1
⇤, b2 =
⇥�1
⇤<latexit sha1_base64="NxNR37Ppw14yi1R4phhQWUATR6g=">AAACbHicbVFNb9NAEF0b6McWigvqoVSVViRUHCCycykckCJx4VgkQirFJlqvx8mq67W7O64UWb7xC7nxF7jwF9ikPpQ0I6309N7Mm923aaWkxTD87fmPHj/Z2d3bpwdPnx0+D45efLdlbQSMRalKc5VyC0pqGKNEBVeVAV6kCibp9eeVPrkFY2Wpv+GygqTgcy1zKTg6ahb87Pfp5MeQfWI0VpDjlMYpzKVuuDF82TZCtDRi5yxyOuiso2ls5HyByTvH3tQ8Y7SJ05yl7dppu1FL30fbPGi/Pwt64SBcF3sIog70SFeXs+BXnJWiLkCjUNzaaRRWmDhjlEKBs64tVFxc8zlMHdS8AJs067Ra9sYxGctL445GtmbvTzS8sHZZpK6z4Liwm9qK3KZNa8w/JI3UVY2gxd2ivFYMS7aKnmXSgEC1dIALI91dmVhwwwW6D6IuhGjzyQ/BeDj4OAi/Dnujiy6NPXJKXpO3JCIXZES+kEsyJoL88QLvxHvl/fWP/VP/7K7V97qZl+S/8s//ATYBtkk=</latexit><latexit sha1_base64="NxNR37Ppw14yi1R4phhQWUATR6g=">AAACbHicbVFNb9NAEF0b6McWigvqoVSVViRUHCCycykckCJx4VgkQirFJlqvx8mq67W7O64UWb7xC7nxF7jwF9ikPpQ0I6309N7Mm923aaWkxTD87fmPHj/Z2d3bpwdPnx0+D45efLdlbQSMRalKc5VyC0pqGKNEBVeVAV6kCibp9eeVPrkFY2Wpv+GygqTgcy1zKTg6ahb87Pfp5MeQfWI0VpDjlMYpzKVuuDF82TZCtDRi5yxyOuiso2ls5HyByTvH3tQ8Y7SJ05yl7dppu1FL30fbPGi/Pwt64SBcF3sIog70SFeXs+BXnJWiLkCjUNzaaRRWmDhjlEKBs64tVFxc8zlMHdS8AJs067Ra9sYxGctL445GtmbvTzS8sHZZpK6z4Liwm9qK3KZNa8w/JI3UVY2gxd2ivFYMS7aKnmXSgEC1dIALI91dmVhwwwW6D6IuhGjzyQ/BeDj4OAi/Dnujiy6NPXJKXpO3JCIXZES+kEsyJoL88QLvxHvl/fWP/VP/7K7V97qZl+S/8s//ATYBtkk=</latexit><latexit sha1_base64="NxNR37Ppw14yi1R4phhQWUATR6g=">AAACbHicbVFNb9NAEF0b6McWigvqoVSVViRUHCCycykckCJx4VgkQirFJlqvx8mq67W7O64UWb7xC7nxF7jwF9ikPpQ0I6309N7Mm923aaWkxTD87fmPHj/Z2d3bpwdPnx0+D45efLdlbQSMRalKc5VyC0pqGKNEBVeVAV6kCibp9eeVPrkFY2Wpv+GygqTgcy1zKTg6ahb87Pfp5MeQfWI0VpDjlMYpzKVuuDF82TZCtDRi5yxyOuiso2ls5HyByTvH3tQ8Y7SJ05yl7dppu1FL30fbPGi/Pwt64SBcF3sIog70SFeXs+BXnJWiLkCjUNzaaRRWmDhjlEKBs64tVFxc8zlMHdS8AJs067Ra9sYxGctL445GtmbvTzS8sHZZpK6z4Liwm9qK3KZNa8w/JI3UVY2gxd2ivFYMS7aKnmXSgEC1dIALI91dmVhwwwW6D6IuhGjzyQ/BeDj4OAi/Dnujiy6NPXJKXpO3JCIXZES+kEsyJoL88QLvxHvl/fWP/VP/7K7V97qZl+S/8s//ATYBtkk=</latexit>
z2 = W 1x1+b1 =
1 1
�1 �1
� �+
0
1.5
�=
�, a2 = I(z2 > 0) =
�
<latexit sha1_base64="1OIxmbCc/a4EtkhLEvGxQfJ4e6A=">AAADm3iclVJbb9MwFHYTLsPcOnhCSOiIFjQ0qOJKaPAwmAQPG+JhSIROatrKcZzWWuIE20ErUf4UP4U3/g1OGomxtQ87kq3P5/Kd408nzBOhjef96Tjutes3bm7dwrfv3L13v7v94JvOCsW4z7IkUych1TwRkvtGmISf5IrTNEz4KDz9UMdHP7jSIpNfzTLnk5TOpYgFo8a6ZtudX/0+LoMwhp/VdAj7MJqS5nlWTQnsQoPDGu8DDhIemzEOQj4XsqRK0WVVKlVhAs+BQBDgVzWyl83lMmpTcKDEfGEmG+orDEG+oNJkaUkqS3L+uY4H7FwbuTxLQAav19bhjX+48gwvLdP3gkYr7ehKu8DwM6PS8qja+SfpO/BebBbvyo1xvz/r9ryB1xhcBqQFPdTa8az7O4gyVqRcGpZQrcfEy83EEhvBEm6pC81zyk7pnI8tlDTlelI2y1XBM+uJIM6UPdJA4z1fUdJU62Ua2syUmoW+GKud62LjwsRvJqWQeWG4ZKtGcZGAyaDeVIiE4swkSwsoU8LOCmxBFWXG7jO2IpCLX74M/OHg7cD7Muwd7LVqbKHH6CnaQQTtoQN0iI6Rj5jzyHnvHDpH7hP3o/vJ/bxKdTptzUP0n7n+X7G5HyE=</latexit><latexit sha1_base64="1OIxmbCc/a4EtkhLEvGxQfJ4e6A=">AAADm3iclVJbb9MwFHYTLsPcOnhCSOiIFjQ0qOJKaPAwmAQPG+JhSIROatrKcZzWWuIE20ErUf4UP4U3/g1OGomxtQ87kq3P5/Kd408nzBOhjef96Tjutes3bm7dwrfv3L13v7v94JvOCsW4z7IkUych1TwRkvtGmISf5IrTNEz4KDz9UMdHP7jSIpNfzTLnk5TOpYgFo8a6ZtudX/0+LoMwhp/VdAj7MJqS5nlWTQnsQoPDGu8DDhIemzEOQj4XsqRK0WVVKlVhAs+BQBDgVzWyl83lMmpTcKDEfGEmG+orDEG+oNJkaUkqS3L+uY4H7FwbuTxLQAav19bhjX+48gwvLdP3gkYr7ehKu8DwM6PS8qja+SfpO/BebBbvyo1xvz/r9ryB1xhcBqQFPdTa8az7O4gyVqRcGpZQrcfEy83EEhvBEm6pC81zyk7pnI8tlDTlelI2y1XBM+uJIM6UPdJA4z1fUdJU62Ua2syUmoW+GKud62LjwsRvJqWQeWG4ZKtGcZGAyaDeVIiE4swkSwsoU8LOCmxBFWXG7jO2IpCLX74M/OHg7cD7Muwd7LVqbKHH6CnaQQTtoQN0iI6Rj5jzyHnvHDpH7hP3o/vJ/bxKdTptzUP0n7n+X7G5HyE=</latexit><latexit sha1_base64="1OIxmbCc/a4EtkhLEvGxQfJ4e6A=">AAADm3iclVJbb9MwFHYTLsPcOnhCSOiIFjQ0qOJKaPAwmAQPG+JhSIROatrKcZzWWuIE20ErUf4UP4U3/g1OGomxtQ87kq3P5/Kd408nzBOhjef96Tjutes3bm7dwrfv3L13v7v94JvOCsW4z7IkUych1TwRkvtGmISf5IrTNEz4KDz9UMdHP7jSIpNfzTLnk5TOpYgFo8a6ZtudX/0+LoMwhp/VdAj7MJqS5nlWTQnsQoPDGu8DDhIemzEOQj4XsqRK0WVVKlVhAs+BQBDgVzWyl83lMmpTcKDEfGEmG+orDEG+oNJkaUkqS3L+uY4H7FwbuTxLQAav19bhjX+48gwvLdP3gkYr7ehKu8DwM6PS8qja+SfpO/BebBbvyo1xvz/r9ryB1xhcBqQFPdTa8az7O4gyVqRcGpZQrcfEy83EEhvBEm6pC81zyk7pnI8tlDTlelI2y1XBM+uJIM6UPdJA4z1fUdJU62Ua2syUmoW+GKud62LjwsRvJqWQeWG4ZKtGcZGAyaDeVIiE4swkSwsoU8LOCmxBFWXG7jO2IpCLX74M/OHg7cD7Muwd7LVqbKHH6CnaQQTtoQN0iI6Rj5jzyHnvHDpH7hP3o/vJ/bxKdTptzUP0n7n+X7G5HyE=</latexit>
z3 = W 2a2 + b2 =⇥1 1
⇤ �+⇥�1
⇤=
⇥ ⇤, y = I(z3 > 0) =
⇥ ⇤
<latexit sha1_base64="4x4wKmONk0RqxcJrhKAYXwr6uSw=">AAADanichVJNb9NAEN3YfJQFSkoPgHpZEYOKgMgOh8KhqBIXuBWJkEpxEq3X63jV9drsjhHG8oG/yI1/wIX/wMa1BG0TdaSV3rzZN1+aqJDCgO//6jnutes3bm7dwrfv3N2+19+5/9nkpWZ8zHKZ65OIGi6F4mMQIPlJoTnNIskn0em7VXzylWsjcvUJqoLPMrpUIhGMgqUWO70fnofrMErI92b+ihySyXzUurSZj8hz0uJohQ8JDiVPYIrDiC+FqqnWtGpqrRsckKcksHGu4o7GoRbLFGYbNA0mYZFSBXlWB00YnnPX5SG2l425XgZrJXhjy82V9V5Y6ZeSxjhMKdRVY8cPgX8DndUfmv1/C3tL/GebV3N1Hex5i/7AH/qtkcsg6MAAdXa86P8M45yVGVfAJDVmGvgFzGxiEExym7o0vKDslC751EJFM25mdXsrDXlimZgkubZPAWnZ/xU1zYypssj+zCik5mJsRa6LTUtIXs9qoYoSuGJnhZJSEsjJ6vBILDRnICsLKNPC9kpYSjVlYM8T2yUEF0e+DMaj4Zuh/3E0ODrotrGF9tBjtI8CdICO0Ht0jMaI9X47284D56Hzx911H7l7Z1+dXqfZRefM9f4CM6wQ+A==</latexit><latexit sha1_base64="4x4wKmONk0RqxcJrhKAYXwr6uSw=">AAADanichVJNb9NAEN3YfJQFSkoPgHpZEYOKgMgOh8KhqBIXuBWJkEpxEq3X63jV9drsjhHG8oG/yI1/wIX/wMa1BG0TdaSV3rzZN1+aqJDCgO//6jnutes3bm7dwrfv3N2+19+5/9nkpWZ8zHKZ65OIGi6F4mMQIPlJoTnNIskn0em7VXzylWsjcvUJqoLPMrpUIhGMgqUWO70fnofrMErI92b+ihySyXzUurSZj8hz0uJohQ8JDiVPYIrDiC+FqqnWtGpqrRsckKcksHGu4o7GoRbLFGYbNA0mYZFSBXlWB00YnnPX5SG2l425XgZrJXhjy82V9V5Y6ZeSxjhMKdRVY8cPgX8DndUfmv1/C3tL/GebV3N1Hex5i/7AH/qtkcsg6MAAdXa86P8M45yVGVfAJDVmGvgFzGxiEExym7o0vKDslC751EJFM25mdXsrDXlimZgkubZPAWnZ/xU1zYypssj+zCik5mJsRa6LTUtIXs9qoYoSuGJnhZJSEsjJ6vBILDRnICsLKNPC9kpYSjVlYM8T2yUEF0e+DMaj4Zuh/3E0ODrotrGF9tBjtI8CdICO0Ht0jMaI9X47284D56Hzx911H7l7Z1+dXqfZRefM9f4CM6wQ+A==</latexit><latexit sha1_base64="4x4wKmONk0RqxcJrhKAYXwr6uSw=">AAADanichVJNb9NAEN3YfJQFSkoPgHpZEYOKgMgOh8KhqBIXuBWJkEpxEq3X63jV9drsjhHG8oG/yI1/wIX/wMa1BG0TdaSV3rzZN1+aqJDCgO//6jnutes3bm7dwrfv3N2+19+5/9nkpWZ8zHKZ65OIGi6F4mMQIPlJoTnNIskn0em7VXzylWsjcvUJqoLPMrpUIhGMgqUWO70fnofrMErI92b+ihySyXzUurSZj8hz0uJohQ8JDiVPYIrDiC+FqqnWtGpqrRsckKcksHGu4o7GoRbLFGYbNA0mYZFSBXlWB00YnnPX5SG2l425XgZrJXhjy82V9V5Y6ZeSxjhMKdRVY8cPgX8DndUfmv1/C3tL/GebV3N1Hex5i/7AH/qtkcsg6MAAdXa86P8M45yVGVfAJDVmGvgFzGxiEExym7o0vKDslC751EJFM25mdXsrDXlimZgkubZPAWnZ/xU1zYypssj+zCik5mJsRa6LTUtIXs9qoYoSuGJnhZJSEsjJ6vBILDRnICsLKNPC9kpYSjVlYM8T2yUEF0e+DMaj4Zuh/3E0ODrotrGF9tBjtI8CdICO0Ht0jMaI9X47284D56Hzx911H7l7Z1+dXqfZRefM9f4CM6wQ+A==</latexit>
1 2 1
1 - .5 0
f o o
The Multilayer Perceptron
31
OK, so the multilayer perceptron is cool and all, but when do we get to REAL neural networks??
The Multilayer Perceptron
32
OK, so the multilayer perceptron is cool and all, but when do we get to REAL neural networks??
We’re pretty much already there.
Things we’ll explore moving forward:
o Non-Binary Features
o Better activation functions
o How to choose architectures
o How to train these things
o Regression or Classification (Answer: Yes)
33
Neural Networks I Wrap-Up
33
o Simplest network is just a regular old perceptron (a linear classifier)
o We can learn non-linear decision boundaries by chaining perceptrons
o We can represent these complicated interactions using linear algebra
Next Time: o The Feed-Forward Neural Network
34
35
36