+ All Categories
Home > Documents > LAG: Lazily Aggregated Gradient for Communication ...06-15-30)-06-15-40-12765-LAG:... · 1...

LAG: Lazily Aggregated Gradient for Communication ...06-15-30)-06-15-40-12765-LAG:... · 1...

Date post: 10-Aug-2019
Category:
Upload: dinhlien
View: 214 times
Download: 0 times
Share this document with a friend
12
1 Tianyi Chen Georgios Giannakis Tao Sun Wotao Yin UMN, ECE UCLA, Math NeurIPS 2018 LAG: Lazily Aggregated Gradient for Communication - Efficient Distributed Learning
Transcript
Page 1: LAG: Lazily Aggregated Gradient for Communication ...06-15-30)-06-15-40-12765-LAG:... · 1 TianyiChen Georgios Giannakis Tao Sun WotaoYin UMN, ECE UCLA, Math NeurIPS2018 LAG: Lazily

1

Tianyi Chen Georgios Giannakis Tao Sun Wotao Yin

UMN, ECEUCLA, Math

NeurIPS 2018

LAG: Lazily Aggregated Gradient for Communication-Efficient Distributed Learning

Page 2: LAG: Lazily Aggregated Gradient for Communication ...06-15-30)-06-15-40-12765-LAG:... · 1 TianyiChen Georgios Giannakis Tao Sun WotaoYin UMN, ECE UCLA, Math NeurIPS2018 LAG: Lazily

2

M := {1, . . . ,M}<latexit sha1_base64="K7JrU5gtA9yB+iPfJpt/ZlnO0yY=">AAACAXicbVDLSsNAFJ3UV62vqBvBzWARXJSSSEERhIIbN4UK9gFNKJPJpB06mQkzE6GEuvFX3LhQxK1/4c6/cdpmoa0HLhzOuZd77wkSRpV2nG+rsLK6tr5R3Cxtbe/s7tn7B20lUolJCwsmZDdAijDKSUtTzUg3kQTFASOdYHQz9TsPRCoq+L0eJ8SP0YDTiGKkjdS3jzIPIwYbk6trL3MrHguFVpWGN+nbZafqzACXiZuTMsjR7NtfXihwGhOuMUNK9Vwn0X6GpKaYkUnJSxVJEB6hAekZylFMlJ/NPpjAU6OEMBLSFNdwpv6eyFCs1DgOTGeM9FAtelPxP6+X6ujSzyhPUk04ni+KUga1gNM4YEglwZqNDUFYUnMrxEMkEdYmtJIJwV18eZm0z6uuU3XvauV6LY+jCI7BCTgDLrgAdXALmqAFMHgEz+AVvFlP1ov1bn3MWwtWPnMI/sD6/AEieJX9</latexit><latexit sha1_base64="K7JrU5gtA9yB+iPfJpt/ZlnO0yY=">AAACAXicbVDLSsNAFJ3UV62vqBvBzWARXJSSSEERhIIbN4UK9gFNKJPJpB06mQkzE6GEuvFX3LhQxK1/4c6/cdpmoa0HLhzOuZd77wkSRpV2nG+rsLK6tr5R3Cxtbe/s7tn7B20lUolJCwsmZDdAijDKSUtTzUg3kQTFASOdYHQz9TsPRCoq+L0eJ8SP0YDTiGKkjdS3jzIPIwYbk6trL3MrHguFVpWGN+nbZafqzACXiZuTMsjR7NtfXihwGhOuMUNK9Vwn0X6GpKaYkUnJSxVJEB6hAekZylFMlJ/NPpjAU6OEMBLSFNdwpv6eyFCs1DgOTGeM9FAtelPxP6+X6ujSzyhPUk04ni+KUga1gNM4YEglwZqNDUFYUnMrxEMkEdYmtJIJwV18eZm0z6uuU3XvauV6LY+jCI7BCTgDLrgAdXALmqAFMHgEz+AVvFlP1ov1bn3MWwtWPnMI/sD6/AEieJX9</latexit><latexit sha1_base64="K7JrU5gtA9yB+iPfJpt/ZlnO0yY=">AAACAXicbVDLSsNAFJ3UV62vqBvBzWARXJSSSEERhIIbN4UK9gFNKJPJpB06mQkzE6GEuvFX3LhQxK1/4c6/cdpmoa0HLhzOuZd77wkSRpV2nG+rsLK6tr5R3Cxtbe/s7tn7B20lUolJCwsmZDdAijDKSUtTzUg3kQTFASOdYHQz9TsPRCoq+L0eJ8SP0YDTiGKkjdS3jzIPIwYbk6trL3MrHguFVpWGN+nbZafqzACXiZuTMsjR7NtfXihwGhOuMUNK9Vwn0X6GpKaYkUnJSxVJEB6hAekZylFMlJ/NPpjAU6OEMBLSFNdwpv6eyFCs1DgOTGeM9FAtelPxP6+X6ujSzyhPUk04ni+KUga1gNM4YEglwZqNDUFYUnMrxEMkEdYmtJIJwV18eZm0z6uuU3XvauV6LY+jCI7BCTgDLrgAdXALmqAFMHgEz+AVvFlP1ov1bn3MWwtWPnMI/sD6/AEieJX9</latexit><latexit sha1_base64="K7JrU5gtA9yB+iPfJpt/ZlnO0yY=">AAACAXicbVDLSsNAFJ3UV62vqBvBzWARXJSSSEERhIIbN4UK9gFNKJPJpB06mQkzE6GEuvFX3LhQxK1/4c6/cdpmoa0HLhzOuZd77wkSRpV2nG+rsLK6tr5R3Cxtbe/s7tn7B20lUolJCwsmZDdAijDKSUtTzUg3kQTFASOdYHQz9TsPRCoq+L0eJ8SP0YDTiGKkjdS3jzIPIwYbk6trL3MrHguFVpWGN+nbZafqzACXiZuTMsjR7NtfXihwGhOuMUNK9Vwn0X6GpKaYkUnJSxVJEB6hAekZylFMlJ/NPpjAU6OEMBLSFNdwpv6eyFCs1DgOTGeM9FAtelPxP6+X6ujSzyhPUk04ni+KUga1gNM4YEglwZqNDUFYUnMrxEMkEdYmtJIJwV18eZm0z6uuU3XvauV6LY+jCI7BCTgDLrgAdXALmqAFMHgEz+AVvFlP1ov1bn3MWwtWPnMI/sD6/AEieJX9</latexit>

Overview

J. Dean, G. Corrado, R. Monga, K. Chen, M. Devin, M. Mao, A. Senior, P. Tucker, K. Yang, Q. V. Le et al.,“Large-scale distributed deep networks,” Proc. NIPS., Lake Tahoe, NV, pp. 1223–1231, 2012

Page 3: LAG: Lazily Aggregated Gradient for Communication ...06-15-30)-06-15-40-12765-LAG:... · 1 TianyiChen Georgios Giannakis Tao Sun WotaoYin UMN, ECE UCLA, Math NeurIPS2018 LAG: Lazily

3

M := {1, . . . ,M}<latexit sha1_base64="K7JrU5gtA9yB+iPfJpt/ZlnO0yY=">AAACAXicbVDLSsNAFJ3UV62vqBvBzWARXJSSSEERhIIbN4UK9gFNKJPJpB06mQkzE6GEuvFX3LhQxK1/4c6/cdpmoa0HLhzOuZd77wkSRpV2nG+rsLK6tr5R3Cxtbe/s7tn7B20lUolJCwsmZDdAijDKSUtTzUg3kQTFASOdYHQz9TsPRCoq+L0eJ8SP0YDTiGKkjdS3jzIPIwYbk6trL3MrHguFVpWGN+nbZafqzACXiZuTMsjR7NtfXihwGhOuMUNK9Vwn0X6GpKaYkUnJSxVJEB6hAekZylFMlJ/NPpjAU6OEMBLSFNdwpv6eyFCs1DgOTGeM9FAtelPxP6+X6ujSzyhPUk04ni+KUga1gNM4YEglwZqNDUFYUnMrxEMkEdYmtJIJwV18eZm0z6uuU3XvauV6LY+jCI7BCTgDLrgAdXALmqAFMHgEz+AVvFlP1ov1bn3MWwtWPnMI/sD6/AEieJX9</latexit><latexit sha1_base64="K7JrU5gtA9yB+iPfJpt/ZlnO0yY=">AAACAXicbVDLSsNAFJ3UV62vqBvBzWARXJSSSEERhIIbN4UK9gFNKJPJpB06mQkzE6GEuvFX3LhQxK1/4c6/cdpmoa0HLhzOuZd77wkSRpV2nG+rsLK6tr5R3Cxtbe/s7tn7B20lUolJCwsmZDdAijDKSUtTzUg3kQTFASOdYHQz9TsPRCoq+L0eJ8SP0YDTiGKkjdS3jzIPIwYbk6trL3MrHguFVpWGN+nbZafqzACXiZuTMsjR7NtfXihwGhOuMUNK9Vwn0X6GpKaYkUnJSxVJEB6hAekZylFMlJ/NPpjAU6OEMBLSFNdwpv6eyFCs1DgOTGeM9FAtelPxP6+X6ujSzyhPUk04ni+KUga1gNM4YEglwZqNDUFYUnMrxEMkEdYmtJIJwV18eZm0z6uuU3XvauV6LY+jCI7BCTgDLrgAdXALmqAFMHgEz+AVvFlP1ov1bn3MWwtWPnMI/sD6/AEieJX9</latexit><latexit sha1_base64="K7JrU5gtA9yB+iPfJpt/ZlnO0yY=">AAACAXicbVDLSsNAFJ3UV62vqBvBzWARXJSSSEERhIIbN4UK9gFNKJPJpB06mQkzE6GEuvFX3LhQxK1/4c6/cdpmoa0HLhzOuZd77wkSRpV2nG+rsLK6tr5R3Cxtbe/s7tn7B20lUolJCwsmZDdAijDKSUtTzUg3kQTFASOdYHQz9TsPRCoq+L0eJ8SP0YDTiGKkjdS3jzIPIwYbk6trL3MrHguFVpWGN+nbZafqzACXiZuTMsjR7NtfXihwGhOuMUNK9Vwn0X6GpKaYkUnJSxVJEB6hAekZylFMlJ/NPpjAU6OEMBLSFNdwpv6eyFCs1DgOTGeM9FAtelPxP6+X6ujSzyhPUk04ni+KUga1gNM4YEglwZqNDUFYUnMrxEMkEdYmtJIJwV18eZm0z6uuU3XvauV6LY+jCI7BCTgDLrgAdXALmqAFMHgEz+AVvFlP1ov1bn3MWwtWPnMI/sD6/AEieJX9</latexit><latexit sha1_base64="K7JrU5gtA9yB+iPfJpt/ZlnO0yY=">AAACAXicbVDLSsNAFJ3UV62vqBvBzWARXJSSSEERhIIbN4UK9gFNKJPJpB06mQkzE6GEuvFX3LhQxK1/4c6/cdpmoa0HLhzOuZd77wkSRpV2nG+rsLK6tr5R3Cxtbe/s7tn7B20lUolJCwsmZDdAijDKSUtTzUg3kQTFASOdYHQz9TsPRCoq+L0eJ8SP0YDTiGKkjdS3jzIPIwYbk6trL3MrHguFVpWGN+nbZafqzACXiZuTMsjR7NtfXihwGhOuMUNK9Vwn0X6GpKaYkUnJSxVJEB6hAekZylFMlJ/NPpjAU6OEMBLSFNdwpv6eyFCs1DgOTGeM9FAtelPxP6+X6ujSzyhPUk04ni+KUga1gNM4YEglwZqNDUFYUnMrxEMkEdYmtJIJwV18eZm0z6uuU3XvauV6LY+jCI7BCTgDLrgAdXALmqAFMHgEz+AVvFlP1ov1bn3MWwtWPnMI/sD6/AEieJX9</latexit>

Overview

J. Dean, G. Corrado, R. Monga, K. Chen, M. Devin, M. Mao, A. Senior, P. Tucker, K. Yang, Q. V. Le et al.,“Large-scale distributed deep networks,” Proc. NIPS., Lake Tahoe, NV, pp. 1223–1231, 2012

q Solvers: gradient descent (GD), momentum methods…

q Our method improves GD by § same convergence rate in theory § reduced communication in theory§ more than 90% communication saving in practice

Page 4: LAG: Lazily Aggregated Gradient for Communication ...06-15-30)-06-15-40-12765-LAG:... · 1 TianyiChen Georgios Giannakis Tao Sun WotaoYin UMN, ECE UCLA, Math NeurIPS2018 LAG: Lazily

4

✓k<latexit sha1_base64="QFufTeJ1sA/bV2evRx5P9WHmcSE=">AAAB/nicbVDLSsNAFJ3UV62vqLhyM1gEVyURQZdFNy4r2Ac0sUwmk3boZBJmboQSCv6KGxeKuPU73Pk3TtostPXAMIdz7mXOnCAVXIPjfFuVldW19Y3qZm1re2d3z94/6OgkU5S1aSIS1QuIZoJL1gYOgvVSxUgcCNYNxjeF331kSvNE3sMkZX5MhpJHnBIw0sA+8oJEhHoSmyv3YMSATB/GA7vuNJwZ8DJxS1JHJVoD+8sLE5rFTAIVROu+66Tg50QBp4JNa16mWUromAxZ31BJYqb9fBZ/ik+NEuIoUeZIwDP190ZOYl0kNJMxgZFe9ArxP6+fQXTl51ymGTBJ5w9FmcCQ4KILHHLFKIiJIYQqbrJiOiKKUDCN1UwJ7uKXl0nnvOE6Dffuot68LuuoomN0gs6Qiy5RE92iFmojinL0jF7Rm/VkvVjv1sd8tGKVO4foD6zPHzBnlkM=</latexit><latexit sha1_base64="QFufTeJ1sA/bV2evRx5P9WHmcSE=">AAAB/nicbVDLSsNAFJ3UV62vqLhyM1gEVyURQZdFNy4r2Ac0sUwmk3boZBJmboQSCv6KGxeKuPU73Pk3TtostPXAMIdz7mXOnCAVXIPjfFuVldW19Y3qZm1re2d3z94/6OgkU5S1aSIS1QuIZoJL1gYOgvVSxUgcCNYNxjeF331kSvNE3sMkZX5MhpJHnBIw0sA+8oJEhHoSmyv3YMSATB/GA7vuNJwZ8DJxS1JHJVoD+8sLE5rFTAIVROu+66Tg50QBp4JNa16mWUromAxZ31BJYqb9fBZ/ik+NEuIoUeZIwDP190ZOYl0kNJMxgZFe9ArxP6+fQXTl51ymGTBJ5w9FmcCQ4KILHHLFKIiJIYQqbrJiOiKKUDCN1UwJ7uKXl0nnvOE6Dffuot68LuuoomN0gs6Qiy5RE92iFmojinL0jF7Rm/VkvVjv1sd8tGKVO4foD6zPHzBnlkM=</latexit><latexit sha1_base64="QFufTeJ1sA/bV2evRx5P9WHmcSE=">AAAB/nicbVDLSsNAFJ3UV62vqLhyM1gEVyURQZdFNy4r2Ac0sUwmk3boZBJmboQSCv6KGxeKuPU73Pk3TtostPXAMIdz7mXOnCAVXIPjfFuVldW19Y3qZm1re2d3z94/6OgkU5S1aSIS1QuIZoJL1gYOgvVSxUgcCNYNxjeF331kSvNE3sMkZX5MhpJHnBIw0sA+8oJEhHoSmyv3YMSATB/GA7vuNJwZ8DJxS1JHJVoD+8sLE5rFTAIVROu+66Tg50QBp4JNa16mWUromAxZ31BJYqb9fBZ/ik+NEuIoUeZIwDP190ZOYl0kNJMxgZFe9ArxP6+fQXTl51ymGTBJ5w9FmcCQ4KILHHLFKIiJIYQqbrJiOiKKUDCN1UwJ7uKXl0nnvOE6Dffuot68LuuoomN0gs6Qiy5RE92iFmojinL0jF7Rm/VkvVjv1sd8tGKVO4foD6zPHzBnlkM=</latexit><latexit sha1_base64="QFufTeJ1sA/bV2evRx5P9WHmcSE=">AAAB/nicbVDLSsNAFJ3UV62vqLhyM1gEVyURQZdFNy4r2Ac0sUwmk3boZBJmboQSCv6KGxeKuPU73Pk3TtostPXAMIdz7mXOnCAVXIPjfFuVldW19Y3qZm1re2d3z94/6OgkU5S1aSIS1QuIZoJL1gYOgvVSxUgcCNYNxjeF331kSvNE3sMkZX5MhpJHnBIw0sA+8oJEhHoSmyv3YMSATB/GA7vuNJwZ8DJxS1JHJVoD+8sLE5rFTAIVROu+66Tg50QBp4JNa16mWUromAxZ31BJYqb9fBZ/ik+NEuIoUeZIwDP190ZOYl0kNJMxgZFe9ArxP6+fQXTl51ymGTBJ5w9FmcCQ4KILHHLFKIiJIYQqbrJiOiKKUDCN1UwJ7uKXl0nnvOE6Dffuot68LuuoomN0gs6Qiy5RE92iFmojinL0jF7Rm/VkvVjv1sd8tGKVO4foD6zPHzBnlkM=</latexit>

Vanilla GD implementation

✓k<latexit sha1_base64="QFufTeJ1sA/bV2evRx5P9WHmcSE=">AAAB/nicbVDLSsNAFJ3UV62vqLhyM1gEVyURQZdFNy4r2Ac0sUwmk3boZBJmboQSCv6KGxeKuPU73Pk3TtostPXAMIdz7mXOnCAVXIPjfFuVldW19Y3qZm1re2d3z94/6OgkU5S1aSIS1QuIZoJL1gYOgvVSxUgcCNYNxjeF331kSvNE3sMkZX5MhpJHnBIw0sA+8oJEhHoSmyv3YMSATB/GA7vuNJwZ8DJxS1JHJVoD+8sLE5rFTAIVROu+66Tg50QBp4JNa16mWUromAxZ31BJYqb9fBZ/ik+NEuIoUeZIwDP190ZOYl0kNJMxgZFe9ArxP6+fQXTl51ymGTBJ5w9FmcCQ4KILHHLFKIiJIYQqbrJiOiKKUDCN1UwJ7uKXl0nnvOE6Dffuot68LuuoomN0gs6Qiy5RE92iFmojinL0jF7Rm/VkvVjv1sd8tGKVO4foD6zPHzBnlkM=</latexit><latexit sha1_base64="QFufTeJ1sA/bV2evRx5P9WHmcSE=">AAAB/nicbVDLSsNAFJ3UV62vqLhyM1gEVyURQZdFNy4r2Ac0sUwmk3boZBJmboQSCv6KGxeKuPU73Pk3TtostPXAMIdz7mXOnCAVXIPjfFuVldW19Y3qZm1re2d3z94/6OgkU5S1aSIS1QuIZoJL1gYOgvVSxUgcCNYNxjeF331kSvNE3sMkZX5MhpJHnBIw0sA+8oJEhHoSmyv3YMSATB/GA7vuNJwZ8DJxS1JHJVoD+8sLE5rFTAIVROu+66Tg50QBp4JNa16mWUromAxZ31BJYqb9fBZ/ik+NEuIoUeZIwDP190ZOYl0kNJMxgZFe9ArxP6+fQXTl51ymGTBJ5w9FmcCQ4KILHHLFKIiJIYQqbrJiOiKKUDCN1UwJ7uKXl0nnvOE6Dffuot68LuuoomN0gs6Qiy5RE92iFmojinL0jF7Rm/VkvVjv1sd8tGKVO4foD6zPHzBnlkM=</latexit><latexit sha1_base64="QFufTeJ1sA/bV2evRx5P9WHmcSE=">AAAB/nicbVDLSsNAFJ3UV62vqLhyM1gEVyURQZdFNy4r2Ac0sUwmk3boZBJmboQSCv6KGxeKuPU73Pk3TtostPXAMIdz7mXOnCAVXIPjfFuVldW19Y3qZm1re2d3z94/6OgkU5S1aSIS1QuIZoJL1gYOgvVSxUgcCNYNxjeF331kSvNE3sMkZX5MhpJHnBIw0sA+8oJEhHoSmyv3YMSATB/GA7vuNJwZ8DJxS1JHJVoD+8sLE5rFTAIVROu+66Tg50QBp4JNa16mWUromAxZ31BJYqb9fBZ/ik+NEuIoUeZIwDP190ZOYl0kNJMxgZFe9ArxP6+fQXTl51ymGTBJ5w9FmcCQ4KILHHLFKIiJIYQqbrJiOiKKUDCN1UwJ7uKXl0nnvOE6Dffuot68LuuoomN0gs6Qiy5RE92iFmojinL0jF7Rm/VkvVjv1sd8tGKVO4foD6zPHzBnlkM=</latexit><latexit sha1_base64="QFufTeJ1sA/bV2evRx5P9WHmcSE=">AAAB/nicbVDLSsNAFJ3UV62vqLhyM1gEVyURQZdFNy4r2Ac0sUwmk3boZBJmboQSCv6KGxeKuPU73Pk3TtostPXAMIdz7mXOnCAVXIPjfFuVldW19Y3qZm1re2d3z94/6OgkU5S1aSIS1QuIZoJL1gYOgvVSxUgcCNYNxjeF331kSvNE3sMkZX5MhpJHnBIw0sA+8oJEhHoSmyv3YMSATB/GA7vuNJwZ8DJxS1JHJVoD+8sLE5rFTAIVROu+66Tg50QBp4JNa16mWUromAxZ31BJYqb9fBZ/ik+NEuIoUeZIwDP190ZOYl0kNJMxgZFe9ArxP6+fQXTl51ymGTBJ5w9FmcCQ4KILHHLFKIiJIYQqbrJiOiKKUDCN1UwJ7uKXl0nnvOE6Dffuot68LuuoomN0gs6Qiy5RE92iFmojinL0jF7Rm/VkvVjv1sd8tGKVO4foD6zPHzBnlkM=</latexit> ✓k

<latexit sha1_base64="QFufTeJ1sA/bV2evRx5P9WHmcSE=">AAAB/nicbVDLSsNAFJ3UV62vqLhyM1gEVyURQZdFNy4r2Ac0sUwmk3boZBJmboQSCv6KGxeKuPU73Pk3TtostPXAMIdz7mXOnCAVXIPjfFuVldW19Y3qZm1re2d3z94/6OgkU5S1aSIS1QuIZoJL1gYOgvVSxUgcCNYNxjeF331kSvNE3sMkZX5MhpJHnBIw0sA+8oJEhHoSmyv3YMSATB/GA7vuNJwZ8DJxS1JHJVoD+8sLE5rFTAIVROu+66Tg50QBp4JNa16mWUromAxZ31BJYqb9fBZ/ik+NEuIoUeZIwDP190ZOYl0kNJMxgZFe9ArxP6+fQXTl51ymGTBJ5w9FmcCQ4KILHHLFKIiJIYQqbrJiOiKKUDCN1UwJ7uKXl0nnvOE6Dffuot68LuuoomN0gs6Qiy5RE92iFmojinL0jF7Rm/VkvVjv1sd8tGKVO4foD6zPHzBnlkM=</latexit><latexit sha1_base64="QFufTeJ1sA/bV2evRx5P9WHmcSE=">AAAB/nicbVDLSsNAFJ3UV62vqLhyM1gEVyURQZdFNy4r2Ac0sUwmk3boZBJmboQSCv6KGxeKuPU73Pk3TtostPXAMIdz7mXOnCAVXIPjfFuVldW19Y3qZm1re2d3z94/6OgkU5S1aSIS1QuIZoJL1gYOgvVSxUgcCNYNxjeF331kSvNE3sMkZX5MhpJHnBIw0sA+8oJEhHoSmyv3YMSATB/GA7vuNJwZ8DJxS1JHJVoD+8sLE5rFTAIVROu+66Tg50QBp4JNa16mWUromAxZ31BJYqb9fBZ/ik+NEuIoUeZIwDP190ZOYl0kNJMxgZFe9ArxP6+fQXTl51ymGTBJ5w9FmcCQ4KILHHLFKIiJIYQqbrJiOiKKUDCN1UwJ7uKXl0nnvOE6Dffuot68LuuoomN0gs6Qiy5RE92iFmojinL0jF7Rm/VkvVjv1sd8tGKVO4foD6zPHzBnlkM=</latexit><latexit sha1_base64="QFufTeJ1sA/bV2evRx5P9WHmcSE=">AAAB/nicbVDLSsNAFJ3UV62vqLhyM1gEVyURQZdFNy4r2Ac0sUwmk3boZBJmboQSCv6KGxeKuPU73Pk3TtostPXAMIdz7mXOnCAVXIPjfFuVldW19Y3qZm1re2d3z94/6OgkU5S1aSIS1QuIZoJL1gYOgvVSxUgcCNYNxjeF331kSvNE3sMkZX5MhpJHnBIw0sA+8oJEhHoSmyv3YMSATB/GA7vuNJwZ8DJxS1JHJVoD+8sLE5rFTAIVROu+66Tg50QBp4JNa16mWUromAxZ31BJYqb9fBZ/ik+NEuIoUeZIwDP190ZOYl0kNJMxgZFe9ArxP6+fQXTl51ymGTBJ5w9FmcCQ4KILHHLFKIiJIYQqbrJiOiKKUDCN1UwJ7uKXl0nnvOE6Dffuot68LuuoomN0gs6Qiy5RE92iFmojinL0jF7Rm/VkvVjv1sd8tGKVO4foD6zPHzBnlkM=</latexit><latexit sha1_base64="QFufTeJ1sA/bV2evRx5P9WHmcSE=">AAAB/nicbVDLSsNAFJ3UV62vqLhyM1gEVyURQZdFNy4r2Ac0sUwmk3boZBJmboQSCv6KGxeKuPU73Pk3TtostPXAMIdz7mXOnCAVXIPjfFuVldW19Y3qZm1re2d3z94/6OgkU5S1aSIS1QuIZoJL1gYOgvVSxUgcCNYNxjeF331kSvNE3sMkZX5MhpJHnBIw0sA+8oJEhHoSmyv3YMSATB/GA7vuNJwZ8DJxS1JHJVoD+8sLE5rFTAIVROu+66Tg50QBp4JNa16mWUromAxZ31BJYqb9fBZ/ik+NEuIoUeZIwDP190ZOYl0kNJMxgZFe9ArxP6+fQXTl51ymGTBJ5w9FmcCQ4KILHHLFKIiJIYQqbrJiOiKKUDCN1UwJ7uKXl0nnvOE6Dffuot68LuuoomN0gs6Qiy5RE92iFmojinL0jF7Rm/VkvVjv1sd8tGKVO4foD6zPHzBnlkM=</latexit>

✓k<latexit sha1_base64="QFufTeJ1sA/bV2evRx5P9WHmcSE=">AAAB/nicbVDLSsNAFJ3UV62vqLhyM1gEVyURQZdFNy4r2Ac0sUwmk3boZBJmboQSCv6KGxeKuPU73Pk3TtostPXAMIdz7mXOnCAVXIPjfFuVldW19Y3qZm1re2d3z94/6OgkU5S1aSIS1QuIZoJL1gYOgvVSxUgcCNYNxjeF331kSvNE3sMkZX5MhpJHnBIw0sA+8oJEhHoSmyv3YMSATB/GA7vuNJwZ8DJxS1JHJVoD+8sLE5rFTAIVROu+66Tg50QBp4JNa16mWUromAxZ31BJYqb9fBZ/ik+NEuIoUeZIwDP190ZOYl0kNJMxgZFe9ArxP6+fQXTl51ymGTBJ5w9FmcCQ4KILHHLFKIiJIYQqbrJiOiKKUDCN1UwJ7uKXl0nnvOE6Dffuot68LuuoomN0gs6Qiy5RE92iFmojinL0jF7Rm/VkvVjv1sd8tGKVO4foD6zPHzBnlkM=</latexit><latexit sha1_base64="QFufTeJ1sA/bV2evRx5P9WHmcSE=">AAAB/nicbVDLSsNAFJ3UV62vqLhyM1gEVyURQZdFNy4r2Ac0sUwmk3boZBJmboQSCv6KGxeKuPU73Pk3TtostPXAMIdz7mXOnCAVXIPjfFuVldW19Y3qZm1re2d3z94/6OgkU5S1aSIS1QuIZoJL1gYOgvVSxUgcCNYNxjeF331kSvNE3sMkZX5MhpJHnBIw0sA+8oJEhHoSmyv3YMSATB/GA7vuNJwZ8DJxS1JHJVoD+8sLE5rFTAIVROu+66Tg50QBp4JNa16mWUromAxZ31BJYqb9fBZ/ik+NEuIoUeZIwDP190ZOYl0kNJMxgZFe9ArxP6+fQXTl51ymGTBJ5w9FmcCQ4KILHHLFKIiJIYQqbrJiOiKKUDCN1UwJ7uKXl0nnvOE6Dffuot68LuuoomN0gs6Qiy5RE92iFmojinL0jF7Rm/VkvVjv1sd8tGKVO4foD6zPHzBnlkM=</latexit><latexit sha1_base64="QFufTeJ1sA/bV2evRx5P9WHmcSE=">AAAB/nicbVDLSsNAFJ3UV62vqLhyM1gEVyURQZdFNy4r2Ac0sUwmk3boZBJmboQSCv6KGxeKuPU73Pk3TtostPXAMIdz7mXOnCAVXIPjfFuVldW19Y3qZm1re2d3z94/6OgkU5S1aSIS1QuIZoJL1gYOgvVSxUgcCNYNxjeF331kSvNE3sMkZX5MhpJHnBIw0sA+8oJEhHoSmyv3YMSATB/GA7vuNJwZ8DJxS1JHJVoD+8sLE5rFTAIVROu+66Tg50QBp4JNa16mWUromAxZ31BJYqb9fBZ/ik+NEuIoUeZIwDP190ZOYl0kNJMxgZFe9ArxP6+fQXTl51ymGTBJ5w9FmcCQ4KILHHLFKIiJIYQqbrJiOiKKUDCN1UwJ7uKXl0nnvOE6Dffuot68LuuoomN0gs6Qiy5RE92iFmojinL0jF7Rm/VkvVjv1sd8tGKVO4foD6zPHzBnlkM=</latexit><latexit sha1_base64="QFufTeJ1sA/bV2evRx5P9WHmcSE=">AAAB/nicbVDLSsNAFJ3UV62vqLhyM1gEVyURQZdFNy4r2Ac0sUwmk3boZBJmboQSCv6KGxeKuPU73Pk3TtostPXAMIdz7mXOnCAVXIPjfFuVldW19Y3qZm1re2d3z94/6OgkU5S1aSIS1QuIZoJL1gYOgvVSxUgcCNYNxjeF331kSvNE3sMkZX5MhpJHnBIw0sA+8oJEhHoSmyv3YMSATB/GA7vuNJwZ8DJxS1JHJVoD+8sLE5rFTAIVROu+66Tg50QBp4JNa16mWUromAxZ31BJYqb9fBZ/ik+NEuIoUeZIwDP190ZOYl0kNJMxgZFe9ArxP6+fQXTl51ymGTBJ5w9FmcCQ4KILHHLFKIiJIYQqbrJiOiKKUDCN1UwJ7uKXl0nnvOE6Dffuot68LuuoomN0gs6Qiy5RE92iFmojinL0jF7Rm/VkvVjv1sd8tGKVO4foD6zPHzBnlkM=</latexit>

✓k+1 = ✓k � ↵X

m2MrLm

�✓k�

<latexit sha1_base64="3YCjpjs6cM91HQn3FAIohgUTR/E=">AAACYXicbVFNTxsxEPUulI9QYEuPXCwiJKqKaBchwQUpopceWgkkAkhxWM06TmLF9q7sWdRotX+SG5de+kfqTXJoISNZfn5vZjx+zgolHcbxaxCurX/Y2Nzabu183N3bjz4d3Lu8tFz0eK5y+5iBE0oa0UOJSjwWVoDOlHjIpt8a/eFZWCdzc4ezQgw0jI0cSQ7oqTT6xbJcDd1M+61iOBEI9VM1/ZrUV6uU6SkDVUyAuVKnlWbSVIyDoj/rmhnIFCyOP+pUs0yOT1b2aJQvadSOO/E86HuQLEGbLOMmjV7YMOelFga5Auf6SVzgoAKLkitRt1jpRAF8CmPR99CAFm5QzR2q6bFnhnSUW78M0jn7b0UF2jVj+kwNOHFvtYZcpfVLHF0OKmmKEoXhi4tGpaKY08ZuOpRWcFQzD4Bb6WelfAIWOPpPaXkTkrdPfg/uzzpJ3Eluz9vd66UdW+SQHJETkpAL0iXfyQ3pEU5+B+vBbrAX/Am3wyg8WKSGwbLmM/kvwsO/Z9u58A==</latexit><latexit sha1_base64="3YCjpjs6cM91HQn3FAIohgUTR/E=">AAACYXicbVFNTxsxEPUulI9QYEuPXCwiJKqKaBchwQUpopceWgkkAkhxWM06TmLF9q7sWdRotX+SG5de+kfqTXJoISNZfn5vZjx+zgolHcbxaxCurX/Y2Nzabu183N3bjz4d3Lu8tFz0eK5y+5iBE0oa0UOJSjwWVoDOlHjIpt8a/eFZWCdzc4ezQgw0jI0cSQ7oqTT6xbJcDd1M+61iOBEI9VM1/ZrUV6uU6SkDVUyAuVKnlWbSVIyDoj/rmhnIFCyOP+pUs0yOT1b2aJQvadSOO/E86HuQLEGbLOMmjV7YMOelFga5Auf6SVzgoAKLkitRt1jpRAF8CmPR99CAFm5QzR2q6bFnhnSUW78M0jn7b0UF2jVj+kwNOHFvtYZcpfVLHF0OKmmKEoXhi4tGpaKY08ZuOpRWcFQzD4Bb6WelfAIWOPpPaXkTkrdPfg/uzzpJ3Eluz9vd66UdW+SQHJETkpAL0iXfyQ3pEU5+B+vBbrAX/Am3wyg8WKSGwbLmM/kvwsO/Z9u58A==</latexit><latexit sha1_base64="3YCjpjs6cM91HQn3FAIohgUTR/E=">AAACYXicbVFNTxsxEPUulI9QYEuPXCwiJKqKaBchwQUpopceWgkkAkhxWM06TmLF9q7sWdRotX+SG5de+kfqTXJoISNZfn5vZjx+zgolHcbxaxCurX/Y2Nzabu183N3bjz4d3Lu8tFz0eK5y+5iBE0oa0UOJSjwWVoDOlHjIpt8a/eFZWCdzc4ezQgw0jI0cSQ7oqTT6xbJcDd1M+61iOBEI9VM1/ZrUV6uU6SkDVUyAuVKnlWbSVIyDoj/rmhnIFCyOP+pUs0yOT1b2aJQvadSOO/E86HuQLEGbLOMmjV7YMOelFga5Auf6SVzgoAKLkitRt1jpRAF8CmPR99CAFm5QzR2q6bFnhnSUW78M0jn7b0UF2jVj+kwNOHFvtYZcpfVLHF0OKmmKEoXhi4tGpaKY08ZuOpRWcFQzD4Bb6WelfAIWOPpPaXkTkrdPfg/uzzpJ3Eluz9vd66UdW+SQHJETkpAL0iXfyQ3pEU5+B+vBbrAX/Am3wyg8WKSGwbLmM/kvwsO/Z9u58A==</latexit><latexit sha1_base64="3YCjpjs6cM91HQn3FAIohgUTR/E=">AAACYXicbVFNTxsxEPUulI9QYEuPXCwiJKqKaBchwQUpopceWgkkAkhxWM06TmLF9q7sWdRotX+SG5de+kfqTXJoISNZfn5vZjx+zgolHcbxaxCurX/Y2Nzabu183N3bjz4d3Lu8tFz0eK5y+5iBE0oa0UOJSjwWVoDOlHjIpt8a/eFZWCdzc4ezQgw0jI0cSQ7oqTT6xbJcDd1M+61iOBEI9VM1/ZrUV6uU6SkDVUyAuVKnlWbSVIyDoj/rmhnIFCyOP+pUs0yOT1b2aJQvadSOO/E86HuQLEGbLOMmjV7YMOelFga5Auf6SVzgoAKLkitRt1jpRAF8CmPR99CAFm5QzR2q6bFnhnSUW78M0jn7b0UF2jVj+kwNOHFvtYZcpfVLHF0OKmmKEoXhi4tGpaKY08ZuOpRWcFQzD4Bb6WelfAIWOPpPaXkTkrdPfg/uzzpJ3Eluz9vd66UdW+SQHJETkpAL0iXfyQ3pEU5+B+vBbrAX/Am3wyg8WKSGwbLmM/kvwsO/Z9u58A==</latexit>

rLm

�✓k�

<latexit sha1_base64="I1eQ3hO3bdPUWRLe+PkoBFzGUVo=">AAACGHicbVBNS8NAEN34WetX1KOXxSLopSYi6LHoxYOHClaFJpbJdtMu3WzC7kQooT/Di3/FiwdFvPbmv3FTe/DrwbKP92aYmRdlUhj0vA9nZnZufmGxslRdXlldW3c3Nq9NmmvGWyyVqb6NwHApFG+hQMlvM80hiSS/iQZnpX9zz7URqbrCYcbDBHpKxIIBWqnjHgQKIglFwEDSi1EnCSLR2wuiVHbNMLFfEWCfI4zuBqWz33FrXt2bgP4l/pTUyBTNjjsOuinLE66QSTCm7XsZhgVoFEzyUTXIDc+ADaDH25YqSLgJi8lhI7prlS6NU22fQjpRv3cUkJhyTVuZAPbNb68U//PaOcYnYSFUliNX7GtQnEuKKS1Tol2hOUM5tASYFnZXyvqggaHNsmpD8H+f/JdcH9Z9r+5fHtUap9M4KmSb7JA94pNj0iDnpElahJEH8kReyKvz6Dw7b877V+mMM+3ZIj/gjD8Br/ygyg==</latexit><latexit sha1_base64="I1eQ3hO3bdPUWRLe+PkoBFzGUVo=">AAACGHicbVBNS8NAEN34WetX1KOXxSLopSYi6LHoxYOHClaFJpbJdtMu3WzC7kQooT/Di3/FiwdFvPbmv3FTe/DrwbKP92aYmRdlUhj0vA9nZnZufmGxslRdXlldW3c3Nq9NmmvGWyyVqb6NwHApFG+hQMlvM80hiSS/iQZnpX9zz7URqbrCYcbDBHpKxIIBWqnjHgQKIglFwEDSi1EnCSLR2wuiVHbNMLFfEWCfI4zuBqWz33FrXt2bgP4l/pTUyBTNjjsOuinLE66QSTCm7XsZhgVoFEzyUTXIDc+ADaDH25YqSLgJi8lhI7prlS6NU22fQjpRv3cUkJhyTVuZAPbNb68U//PaOcYnYSFUliNX7GtQnEuKKS1Tol2hOUM5tASYFnZXyvqggaHNsmpD8H+f/JdcH9Z9r+5fHtUap9M4KmSb7JA94pNj0iDnpElahJEH8kReyKvz6Dw7b877V+mMM+3ZIj/gjD8Br/ygyg==</latexit><latexit sha1_base64="I1eQ3hO3bdPUWRLe+PkoBFzGUVo=">AAACGHicbVBNS8NAEN34WetX1KOXxSLopSYi6LHoxYOHClaFJpbJdtMu3WzC7kQooT/Di3/FiwdFvPbmv3FTe/DrwbKP92aYmRdlUhj0vA9nZnZufmGxslRdXlldW3c3Nq9NmmvGWyyVqb6NwHApFG+hQMlvM80hiSS/iQZnpX9zz7URqbrCYcbDBHpKxIIBWqnjHgQKIglFwEDSi1EnCSLR2wuiVHbNMLFfEWCfI4zuBqWz33FrXt2bgP4l/pTUyBTNjjsOuinLE66QSTCm7XsZhgVoFEzyUTXIDc+ADaDH25YqSLgJi8lhI7prlS6NU22fQjpRv3cUkJhyTVuZAPbNb68U//PaOcYnYSFUliNX7GtQnEuKKS1Tol2hOUM5tASYFnZXyvqggaHNsmpD8H+f/JdcH9Z9r+5fHtUap9M4KmSb7JA94pNj0iDnpElahJEH8kReyKvz6Dw7b877V+mMM+3ZIj/gjD8Br/ygyg==</latexit><latexit sha1_base64="hP+6LrUf2d3tZaldqaQQvEKMXyw=">AAAB2XicbZDNSgMxFIXv1L86Vq1rN8EiuCozbnQpuHFZwbZCO5RM5k4bmskMyR2hDH0BF25EfC93vo3pz0JbDwQ+zknIvSculLQUBN9ebWd3b/+gfugfNfzjk9Nmo2fz0gjsilzl5jnmFpXU2CVJCp8LgzyLFfbj6f0i77+gsTLXTzQrMMr4WMtUCk7O6oyaraAdLMW2IVxDC9YaNb+GSS7KDDUJxa0dhEFBUcUNSaFw7g9LiwUXUz7GgUPNM7RRtRxzzi6dk7A0N+5oYkv394uKZ9bOstjdzDhN7Ga2MP/LBiWlt1EldVESarH6KC0Vo5wtdmaJNChIzRxwYaSblYkJN1yQa8Z3HYSbG29D77odBu3wMYA6nMMFXEEIN3AHD9CBLghI4BXevYn35n2suqp569LO4I+8zx84xIo4</latexit><latexit sha1_base64="s6cFni5L/+zPwvOelVDQmdiDQvU=">AAACDXicbVA9SwNBEJ3z2xg12tosBiE28c5GS8HGwkLBRCEXw9xmkyzZ2zt254Rw5GfY+FdsLBSxtfPfuJekUOODZR7vzbAzL0qVtOT7X97C4tLyyuraemmjvLm1XdkpN22SGS4aPFGJuYvQCiW1aJAkJe5SIzCOlLiNhueFf/sgjJWJvqFRKtox9rXsSY7kpE7lKNQYKcxDjopdjjtxGMl+LYwS1bWj2JU8pIEgHN8PC+ewU6n6dX8CNk+CGanCDFedymfYTXgWC01cobWtwE+pnaMhyZUYl8LMihT5EPui5ajGWNh2PjlszA6c0mW9xLiniU3UnxM5xrZY03XGSAP71yvE/7xWRr3Tdi51mpHQfPpRL1OMElakxLrSCE5q5AhyI92ujA/QICeXZcmFEPw9eZ40j+uBXw+ufViDPdiHGgRwAmdwAVfQAA6P8Ayv8OY9eS/e+zSuBW+W2y78gvfxDcoNn0c=</latexit><latexit sha1_base64="s6cFni5L/+zPwvOelVDQmdiDQvU=">AAACDXicbVA9SwNBEJ3z2xg12tosBiE28c5GS8HGwkLBRCEXw9xmkyzZ2zt254Rw5GfY+FdsLBSxtfPfuJekUOODZR7vzbAzL0qVtOT7X97C4tLyyuraemmjvLm1XdkpN22SGS4aPFGJuYvQCiW1aJAkJe5SIzCOlLiNhueFf/sgjJWJvqFRKtox9rXsSY7kpE7lKNQYKcxDjopdjjtxGMl+LYwS1bWj2JU8pIEgHN8PC+ewU6n6dX8CNk+CGanCDFedymfYTXgWC01cobWtwE+pnaMhyZUYl8LMihT5EPui5ajGWNh2PjlszA6c0mW9xLiniU3UnxM5xrZY03XGSAP71yvE/7xWRr3Tdi51mpHQfPpRL1OMElakxLrSCE5q5AhyI92ujA/QICeXZcmFEPw9eZ40j+uBXw+ufViDPdiHGgRwAmdwAVfQAA6P8Ayv8OY9eS/e+zSuBW+W2y78gvfxDcoNn0c=</latexit><latexit sha1_base64="AD9+7Z/64N0L6c5CVtCF9t7I5SQ=">AAACGHicbVA9TwJBEN3zE/ELtbTZSEywwTsbLYk2FhaYyEfCIZlbFtiwt3fZnTMhF36GjX/FxkJjbOn8N+4BhYIv2ezLezOZmRfEUhh03W9nZXVtfWMzt5Xf3tnd2y8cHNZNlGjGayySkW4GYLgUitdQoOTNWHMIA8kbwfAm8xtPXBsRqQccxbwdQl+JnmCAVuoUzn0FgYTUZyDp3bgT+oHol/wgkl0zCu2X+jjgCOPHYeacdQpFt+xOQZeJNydFMke1U5j43YglIVfIJBjT8twY2yloFEzycd5PDI+BDaHPW5YqCLlpp9PDxvTUKl3ai7R9CulU/d2RQmiyNW1lCDgwi14m/ue1EuxdtVOh4gS5YrNBvURSjGiWEu0KzRnKkSXAtLC7UjYADQxtlnkbgrd48jKpX5Q9t+zdu8XK9TyOHDkmJ6REPHJJKuSWVEmNMPJMXsk7+XBenDfn0/mala44854j8gfO5AeuvKDG</latexit><latexit sha1_base64="I1eQ3hO3bdPUWRLe+PkoBFzGUVo=">AAACGHicbVBNS8NAEN34WetX1KOXxSLopSYi6LHoxYOHClaFJpbJdtMu3WzC7kQooT/Di3/FiwdFvPbmv3FTe/DrwbKP92aYmRdlUhj0vA9nZnZufmGxslRdXlldW3c3Nq9NmmvGWyyVqb6NwHApFG+hQMlvM80hiSS/iQZnpX9zz7URqbrCYcbDBHpKxIIBWqnjHgQKIglFwEDSi1EnCSLR2wuiVHbNMLFfEWCfI4zuBqWz33FrXt2bgP4l/pTUyBTNjjsOuinLE66QSTCm7XsZhgVoFEzyUTXIDc+ADaDH25YqSLgJi8lhI7prlS6NU22fQjpRv3cUkJhyTVuZAPbNb68U//PaOcYnYSFUliNX7GtQnEuKKS1Tol2hOUM5tASYFnZXyvqggaHNsmpD8H+f/JdcH9Z9r+5fHtUap9M4KmSb7JA94pNj0iDnpElahJEH8kReyKvz6Dw7b877V+mMM+3ZIj/gjD8Br/ygyg==</latexit><latexit sha1_base64="I1eQ3hO3bdPUWRLe+PkoBFzGUVo=">AAACGHicbVBNS8NAEN34WetX1KOXxSLopSYi6LHoxYOHClaFJpbJdtMu3WzC7kQooT/Di3/FiwdFvPbmv3FTe/DrwbKP92aYmRdlUhj0vA9nZnZufmGxslRdXlldW3c3Nq9NmmvGWyyVqb6NwHApFG+hQMlvM80hiSS/iQZnpX9zz7URqbrCYcbDBHpKxIIBWqnjHgQKIglFwEDSi1EnCSLR2wuiVHbNMLFfEWCfI4zuBqWz33FrXt2bgP4l/pTUyBTNjjsOuinLE66QSTCm7XsZhgVoFEzyUTXIDc+ADaDH25YqSLgJi8lhI7prlS6NU22fQjpRv3cUkJhyTVuZAPbNb68U//PaOcYnYSFUliNX7GtQnEuKKS1Tol2hOUM5tASYFnZXyvqggaHNsmpD8H+f/JdcH9Z9r+5fHtUap9M4KmSb7JA94pNj0iDnpElahJEH8kReyKvz6Dw7b877V+mMM+3ZIj/gjD8Br/ygyg==</latexit><latexit sha1_base64="I1eQ3hO3bdPUWRLe+PkoBFzGUVo=">AAACGHicbVBNS8NAEN34WetX1KOXxSLopSYi6LHoxYOHClaFJpbJdtMu3WzC7kQooT/Di3/FiwdFvPbmv3FTe/DrwbKP92aYmRdlUhj0vA9nZnZufmGxslRdXlldW3c3Nq9NmmvGWyyVqb6NwHApFG+hQMlvM80hiSS/iQZnpX9zz7URqbrCYcbDBHpKxIIBWqnjHgQKIglFwEDSi1EnCSLR2wuiVHbNMLFfEWCfI4zuBqWz33FrXt2bgP4l/pTUyBTNjjsOuinLE66QSTCm7XsZhgVoFEzyUTXIDc+ADaDH25YqSLgJi8lhI7prlS6NU22fQjpRv3cUkJhyTVuZAPbNb68U//PaOcYnYSFUliNX7GtQnEuKKS1Tol2hOUM5tASYFnZXyvqggaHNsmpD8H+f/JdcH9Z9r+5fHtUap9M4KmSb7JA94pNj0iDnpElahJEH8kReyKvz6Dw7b877V+mMM+3ZIj/gjD8Br/ygyg==</latexit><latexit sha1_base64="I1eQ3hO3bdPUWRLe+PkoBFzGUVo=">AAACGHicbVBNS8NAEN34WetX1KOXxSLopSYi6LHoxYOHClaFJpbJdtMu3WzC7kQooT/Di3/FiwdFvPbmv3FTe/DrwbKP92aYmRdlUhj0vA9nZnZufmGxslRdXlldW3c3Nq9NmmvGWyyVqb6NwHApFG+hQMlvM80hiSS/iQZnpX9zz7URqbrCYcbDBHpKxIIBWqnjHgQKIglFwEDSi1EnCSLR2wuiVHbNMLFfEWCfI4zuBqWz33FrXt2bgP4l/pTUyBTNjjsOuinLE66QSTCm7XsZhgVoFEzyUTXIDc+ADaDH25YqSLgJi8lhI7prlS6NU22fQjpRv3cUkJhyTVuZAPbNb68U//PaOcYnYSFUliNX7GtQnEuKKS1Tol2hOUM5tASYFnZXyvqggaHNsmpD8H+f/JdcH9Z9r+5fHtUap9M4KmSb7JA94pNj0iDnpElahJEH8kReyKvz6Dw7b877V+mMM+3ZIj/gjD8Br/ygyg==</latexit><latexit sha1_base64="I1eQ3hO3bdPUWRLe+PkoBFzGUVo=">AAACGHicbVBNS8NAEN34WetX1KOXxSLopSYi6LHoxYOHClaFJpbJdtMu3WzC7kQooT/Di3/FiwdFvPbmv3FTe/DrwbKP92aYmRdlUhj0vA9nZnZufmGxslRdXlldW3c3Nq9NmmvGWyyVqb6NwHApFG+hQMlvM80hiSS/iQZnpX9zz7URqbrCYcbDBHpKxIIBWqnjHgQKIglFwEDSi1EnCSLR2wuiVHbNMLFfEWCfI4zuBqWz33FrXt2bgP4l/pTUyBTNjjsOuinLE66QSTCm7XsZhgVoFEzyUTXIDc+ADaDH25YqSLgJi8lhI7prlS6NU22fQjpRv3cUkJhyTVuZAPbNb68U//PaOcYnYSFUliNX7GtQnEuKKS1Tol2hOUM5tASYFnZXyvqggaHNsmpD8H+f/JdcH9Z9r+5fHtUap9M4KmSb7JA94pNj0iDnpElahJEH8kReyKvz6Dw7b877V+mMM+3ZIj/gjD8Br/ygyg==</latexit><latexit sha1_base64="I1eQ3hO3bdPUWRLe+PkoBFzGUVo=">AAACGHicbVBNS8NAEN34WetX1KOXxSLopSYi6LHoxYOHClaFJpbJdtMu3WzC7kQooT/Di3/FiwdFvPbmv3FTe/DrwbKP92aYmRdlUhj0vA9nZnZufmGxslRdXlldW3c3Nq9NmmvGWyyVqb6NwHApFG+hQMlvM80hiSS/iQZnpX9zz7URqbrCYcbDBHpKxIIBWqnjHgQKIglFwEDSi1EnCSLR2wuiVHbNMLFfEWCfI4zuBqWz33FrXt2bgP4l/pTUyBTNjjsOuinLE66QSTCm7XsZhgVoFEzyUTXIDc+ADaDH25YqSLgJi8lhI7prlS6NU22fQjpRv3cUkJhyTVuZAPbNb68U//PaOcYnYSFUliNX7GtQnEuKKS1Tol2hOUM5tASYFnZXyvqggaHNsmpD8H+f/JdcH9Z9r+5fHtUap9M4KmSb7JA94pNj0iDnpElahJEH8kReyKvz6Dw7b877V+mMM+3ZIj/gjD8Br/ygyg==</latexit>

rL1

�✓k�

<latexit sha1_base64="0izSstp1VXcdyBHaas99nooR5EY=">AAACGHicbVA9SwNBEN2L3/ErammzGITYxDsRtBRtLCwimA/IxTC32SRL9vaO3TkhHPkZNv4VGwtFbNP5b9xLUmjig2Uf780wMy+IpTDout9Obml5ZXVtfSO/ubW9s1vY26+ZKNGMV1kkI90IwHApFK+iQMkbseYQBpLXg8FN5tefuDYiUg84jHkrhJ4SXcEArdQunPoKAgmpz0DSu1Hb8wPRK/lBJDtmGNov9bHPEUaPg8w5aReKbtmdgC4Sb0aKZIZKuzD2OxFLQq6QSTCm6bkxtlLQKJjko7yfGB4DG0CPNy1VEHLTSieHjeixVTq0G2n7FNKJ+rsjhdBka9rKELBv5r1M/M9rJti9bKVCxQlyxaaDuomkGNEsJdoRmjOUQ0uAaWF3pawPGhjaLPM2BG/+5EVSOyt7btm7Py9eXc/iWCeH5IiUiEcuyBW5JRVSJYw8k1fyTj6cF+fN+XS+pqU5Z9ZzQP7AGf8ATcigjg==</latexit><latexit sha1_base64="0izSstp1VXcdyBHaas99nooR5EY=">AAACGHicbVA9SwNBEN2L3/ErammzGITYxDsRtBRtLCwimA/IxTC32SRL9vaO3TkhHPkZNv4VGwtFbNP5b9xLUmjig2Uf780wMy+IpTDout9Obml5ZXVtfSO/ubW9s1vY26+ZKNGMV1kkI90IwHApFK+iQMkbseYQBpLXg8FN5tefuDYiUg84jHkrhJ4SXcEArdQunPoKAgmpz0DSu1Hb8wPRK/lBJDtmGNov9bHPEUaPg8w5aReKbtmdgC4Sb0aKZIZKuzD2OxFLQq6QSTCm6bkxtlLQKJjko7yfGB4DG0CPNy1VEHLTSieHjeixVTq0G2n7FNKJ+rsjhdBka9rKELBv5r1M/M9rJti9bKVCxQlyxaaDuomkGNEsJdoRmjOUQ0uAaWF3pawPGhjaLPM2BG/+5EVSOyt7btm7Py9eXc/iWCeH5IiUiEcuyBW5JRVSJYw8k1fyTj6cF+fN+XS+pqU5Z9ZzQP7AGf8ATcigjg==</latexit><latexit sha1_base64="0izSstp1VXcdyBHaas99nooR5EY=">AAACGHicbVA9SwNBEN2L3/ErammzGITYxDsRtBRtLCwimA/IxTC32SRL9vaO3TkhHPkZNv4VGwtFbNP5b9xLUmjig2Uf780wMy+IpTDout9Obml5ZXVtfSO/ubW9s1vY26+ZKNGMV1kkI90IwHApFK+iQMkbseYQBpLXg8FN5tefuDYiUg84jHkrhJ4SXcEArdQunPoKAgmpz0DSu1Hb8wPRK/lBJDtmGNov9bHPEUaPg8w5aReKbtmdgC4Sb0aKZIZKuzD2OxFLQq6QSTCm6bkxtlLQKJjko7yfGB4DG0CPNy1VEHLTSieHjeixVTq0G2n7FNKJ+rsjhdBka9rKELBv5r1M/M9rJti9bKVCxQlyxaaDuomkGNEsJdoRmjOUQ0uAaWF3pawPGhjaLPM2BG/+5EVSOyt7btm7Py9eXc/iWCeH5IiUiEcuyBW5JRVSJYw8k1fyTj6cF+fN+XS+pqU5Z9ZzQP7AGf8ATcigjg==</latexit><latexit sha1_base64="0izSstp1VXcdyBHaas99nooR5EY=">AAACGHicbVA9SwNBEN2L3/ErammzGITYxDsRtBRtLCwimA/IxTC32SRL9vaO3TkhHPkZNv4VGwtFbNP5b9xLUmjig2Uf780wMy+IpTDout9Obml5ZXVtfSO/ubW9s1vY26+ZKNGMV1kkI90IwHApFK+iQMkbseYQBpLXg8FN5tefuDYiUg84jHkrhJ4SXcEArdQunPoKAgmpz0DSu1Hb8wPRK/lBJDtmGNov9bHPEUaPg8w5aReKbtmdgC4Sb0aKZIZKuzD2OxFLQq6QSTCm6bkxtlLQKJjko7yfGB4DG0CPNy1VEHLTSieHjeixVTq0G2n7FNKJ+rsjhdBka9rKELBv5r1M/M9rJti9bKVCxQlyxaaDuomkGNEsJdoRmjOUQ0uAaWF3pawPGhjaLPM2BG/+5EVSOyt7btm7Py9eXc/iWCeH5IiUiEcuyBW5JRVSJYw8k1fyTj6cF+fN+XS+pqU5Z9ZzQP7AGf8ATcigjg==</latexit>

rLM

�✓k�

<latexit sha1_base64="s6Z3BoG3QzlpbrNHhZck7zmXId8=">AAACGHicbVBNS8NAEN34bf2qevSyWAS91EQEPRa9eFBQsFVoYplst+3SzSbsToQS+jO8+Fe8eFDEa2/+GzdtDlp9sOzjvRlm5oWJFAZd98uZmZ2bX1hcWi6trK6tb5Q3txomTjXjdRbLWN+HYLgUitdRoOT3ieYQhZLfhf3z3L975NqIWN3iIOFBBF0lOoIBWqlVPvQVhBIyn4Gkl8PWlR+K7r4fxrJtBpH9Mh97HGH40M+dg1a54lbdMehf4hWkQgpct8ojvx2zNOIKmQRjmp6bYJCBRsEkH5b81PAEWB+6vGmpgoibIBsfNqR7VmnTTqztU0jH6s+ODCKTr2krI8CemfZy8T+vmWLnNMiESlLkik0GdVJJMaZ5SrQtNGcoB5YA08LuSlkPNDC0WZZsCN70yX9J46jquVXv5rhSOyviWCI7ZJfsE4+ckBq5INekThh5Ii/kjbw7z86r8+F8TkpnnKJnm/yCM/oGe5ygqg==</latexit><latexit sha1_base64="s6Z3BoG3QzlpbrNHhZck7zmXId8=">AAACGHicbVBNS8NAEN34bf2qevSyWAS91EQEPRa9eFBQsFVoYplst+3SzSbsToQS+jO8+Fe8eFDEa2/+GzdtDlp9sOzjvRlm5oWJFAZd98uZmZ2bX1hcWi6trK6tb5Q3txomTjXjdRbLWN+HYLgUitdRoOT3ieYQhZLfhf3z3L975NqIWN3iIOFBBF0lOoIBWqlVPvQVhBIyn4Gkl8PWlR+K7r4fxrJtBpH9Mh97HGH40M+dg1a54lbdMehf4hWkQgpct8ojvx2zNOIKmQRjmp6bYJCBRsEkH5b81PAEWB+6vGmpgoibIBsfNqR7VmnTTqztU0jH6s+ODCKTr2krI8CemfZy8T+vmWLnNMiESlLkik0GdVJJMaZ5SrQtNGcoB5YA08LuSlkPNDC0WZZsCN70yX9J46jquVXv5rhSOyviWCI7ZJfsE4+ckBq5INekThh5Ii/kjbw7z86r8+F8TkpnnKJnm/yCM/oGe5ygqg==</latexit><latexit sha1_base64="s6Z3BoG3QzlpbrNHhZck7zmXId8=">AAACGHicbVBNS8NAEN34bf2qevSyWAS91EQEPRa9eFBQsFVoYplst+3SzSbsToQS+jO8+Fe8eFDEa2/+GzdtDlp9sOzjvRlm5oWJFAZd98uZmZ2bX1hcWi6trK6tb5Q3txomTjXjdRbLWN+HYLgUitdRoOT3ieYQhZLfhf3z3L975NqIWN3iIOFBBF0lOoIBWqlVPvQVhBIyn4Gkl8PWlR+K7r4fxrJtBpH9Mh97HGH40M+dg1a54lbdMehf4hWkQgpct8ojvx2zNOIKmQRjmp6bYJCBRsEkH5b81PAEWB+6vGmpgoibIBsfNqR7VmnTTqztU0jH6s+ODCKTr2krI8CemfZy8T+vmWLnNMiESlLkik0GdVJJMaZ5SrQtNGcoB5YA08LuSlkPNDC0WZZsCN70yX9J46jquVXv5rhSOyviWCI7ZJfsE4+ckBq5INekThh5Ii/kjbw7z86r8+F8TkpnnKJnm/yCM/oGe5ygqg==</latexit><latexit sha1_base64="s6Z3BoG3QzlpbrNHhZck7zmXId8=">AAACGHicbVBNS8NAEN34bf2qevSyWAS91EQEPRa9eFBQsFVoYplst+3SzSbsToQS+jO8+Fe8eFDEa2/+GzdtDlp9sOzjvRlm5oWJFAZd98uZmZ2bX1hcWi6trK6tb5Q3txomTjXjdRbLWN+HYLgUitdRoOT3ieYQhZLfhf3z3L975NqIWN3iIOFBBF0lOoIBWqlVPvQVhBIyn4Gkl8PWlR+K7r4fxrJtBpH9Mh97HGH40M+dg1a54lbdMehf4hWkQgpct8ojvx2zNOIKmQRjmp6bYJCBRsEkH5b81PAEWB+6vGmpgoibIBsfNqR7VmnTTqztU0jH6s+ODCKTr2krI8CemfZy8T+vmWLnNMiESlLkik0GdVJJMaZ5SrQtNGcoB5YA08LuSlkPNDC0WZZsCN70yX9J46jquVXv5rhSOyviWCI7ZJfsE4+ckBq5INekThh5Ii/kjbw7z86r8+F8TkpnnKJnm/yCM/oGe5ygqg==</latexit>

q Per iteration communication overhead for M uploads (one per worker)

Page 5: LAG: Lazily Aggregated Gradient for Communication ...06-15-30)-06-15-40-12765-LAG:... · 1 TianyiChen Georgios Giannakis Tao Sun WotaoYin UMN, ECE UCLA, Math NeurIPS2018 LAG: Lazily

5

Prior art

Our contribution

Adaptively skip communication, provable communication reduction

§ Quantized gradient descent [Kashyap et al., 07], [Alistarh et al., 17], [Suresh et al., 17]…

§ Increasing computation before communication [Jaggi et al., 14], [Ma et al., 17], [Smith et al., 17]…

§ Sparse SGD with large entries [Aji-Heafield 17], [Sun et al., 17], [Lin et al., 18], [Stich et al., 18]…

q Communication-efficient distributed learning

Ø number of communication rounds is not reduced

Page 6: LAG: Lazily Aggregated Gradient for Communication ...06-15-30)-06-15-40-12765-LAG:... · 1 TianyiChen Georgios Giannakis Tao Sun WotaoYin UMN, ECE UCLA, Math NeurIPS2018 LAG: Lazily

6

Our LAG implementation

✓k<latexit sha1_base64="QFufTeJ1sA/bV2evRx5P9WHmcSE=">AAAB/nicbVDLSsNAFJ3UV62vqLhyM1gEVyURQZdFNy4r2Ac0sUwmk3boZBJmboQSCv6KGxeKuPU73Pk3TtostPXAMIdz7mXOnCAVXIPjfFuVldW19Y3qZm1re2d3z94/6OgkU5S1aSIS1QuIZoJL1gYOgvVSxUgcCNYNxjeF331kSvNE3sMkZX5MhpJHnBIw0sA+8oJEhHoSmyv3YMSATB/GA7vuNJwZ8DJxS1JHJVoD+8sLE5rFTAIVROu+66Tg50QBp4JNa16mWUromAxZ31BJYqb9fBZ/ik+NEuIoUeZIwDP190ZOYl0kNJMxgZFe9ArxP6+fQXTl51ymGTBJ5w9FmcCQ4KILHHLFKIiJIYQqbrJiOiKKUDCN1UwJ7uKXl0nnvOE6Dffuot68LuuoomN0gs6Qiy5RE92iFmojinL0jF7Rm/VkvVjv1sd8tGKVO4foD6zPHzBnlkM=</latexit><latexit sha1_base64="QFufTeJ1sA/bV2evRx5P9WHmcSE=">AAAB/nicbVDLSsNAFJ3UV62vqLhyM1gEVyURQZdFNy4r2Ac0sUwmk3boZBJmboQSCv6KGxeKuPU73Pk3TtostPXAMIdz7mXOnCAVXIPjfFuVldW19Y3qZm1re2d3z94/6OgkU5S1aSIS1QuIZoJL1gYOgvVSxUgcCNYNxjeF331kSvNE3sMkZX5MhpJHnBIw0sA+8oJEhHoSmyv3YMSATB/GA7vuNJwZ8DJxS1JHJVoD+8sLE5rFTAIVROu+66Tg50QBp4JNa16mWUromAxZ31BJYqb9fBZ/ik+NEuIoUeZIwDP190ZOYl0kNJMxgZFe9ArxP6+fQXTl51ymGTBJ5w9FmcCQ4KILHHLFKIiJIYQqbrJiOiKKUDCN1UwJ7uKXl0nnvOE6Dffuot68LuuoomN0gs6Qiy5RE92iFmojinL0jF7Rm/VkvVjv1sd8tGKVO4foD6zPHzBnlkM=</latexit><latexit sha1_base64="QFufTeJ1sA/bV2evRx5P9WHmcSE=">AAAB/nicbVDLSsNAFJ3UV62vqLhyM1gEVyURQZdFNy4r2Ac0sUwmk3boZBJmboQSCv6KGxeKuPU73Pk3TtostPXAMIdz7mXOnCAVXIPjfFuVldW19Y3qZm1re2d3z94/6OgkU5S1aSIS1QuIZoJL1gYOgvVSxUgcCNYNxjeF331kSvNE3sMkZX5MhpJHnBIw0sA+8oJEhHoSmyv3YMSATB/GA7vuNJwZ8DJxS1JHJVoD+8sLE5rFTAIVROu+66Tg50QBp4JNa16mWUromAxZ31BJYqb9fBZ/ik+NEuIoUeZIwDP190ZOYl0kNJMxgZFe9ArxP6+fQXTl51ymGTBJ5w9FmcCQ4KILHHLFKIiJIYQqbrJiOiKKUDCN1UwJ7uKXl0nnvOE6Dffuot68LuuoomN0gs6Qiy5RE92iFmojinL0jF7Rm/VkvVjv1sd8tGKVO4foD6zPHzBnlkM=</latexit><latexit sha1_base64="QFufTeJ1sA/bV2evRx5P9WHmcSE=">AAAB/nicbVDLSsNAFJ3UV62vqLhyM1gEVyURQZdFNy4r2Ac0sUwmk3boZBJmboQSCv6KGxeKuPU73Pk3TtostPXAMIdz7mXOnCAVXIPjfFuVldW19Y3qZm1re2d3z94/6OgkU5S1aSIS1QuIZoJL1gYOgvVSxUgcCNYNxjeF331kSvNE3sMkZX5MhpJHnBIw0sA+8oJEhHoSmyv3YMSATB/GA7vuNJwZ8DJxS1JHJVoD+8sLE5rFTAIVROu+66Tg50QBp4JNa16mWUromAxZ31BJYqb9fBZ/ik+NEuIoUeZIwDP190ZOYl0kNJMxgZFe9ArxP6+fQXTl51ymGTBJ5w9FmcCQ4KILHHLFKIiJIYQqbrJiOiKKUDCN1UwJ7uKXl0nnvOE6Dffuot68LuuoomN0gs6Qiy5RE92iFmojinL0jF7Rm/VkvVjv1sd8tGKVO4foD6zPHzBnlkM=</latexit> ✓k

<latexit sha1_base64="QFufTeJ1sA/bV2evRx5P9WHmcSE=">AAAB/nicbVDLSsNAFJ3UV62vqLhyM1gEVyURQZdFNy4r2Ac0sUwmk3boZBJmboQSCv6KGxeKuPU73Pk3TtostPXAMIdz7mXOnCAVXIPjfFuVldW19Y3qZm1re2d3z94/6OgkU5S1aSIS1QuIZoJL1gYOgvVSxUgcCNYNxjeF331kSvNE3sMkZX5MhpJHnBIw0sA+8oJEhHoSmyv3YMSATB/GA7vuNJwZ8DJxS1JHJVoD+8sLE5rFTAIVROu+66Tg50QBp4JNa16mWUromAxZ31BJYqb9fBZ/ik+NEuIoUeZIwDP190ZOYl0kNJMxgZFe9ArxP6+fQXTl51ymGTBJ5w9FmcCQ4KILHHLFKIiJIYQqbrJiOiKKUDCN1UwJ7uKXl0nnvOE6Dffuot68LuuoomN0gs6Qiy5RE92iFmojinL0jF7Rm/VkvVjv1sd8tGKVO4foD6zPHzBnlkM=</latexit><latexit sha1_base64="QFufTeJ1sA/bV2evRx5P9WHmcSE=">AAAB/nicbVDLSsNAFJ3UV62vqLhyM1gEVyURQZdFNy4r2Ac0sUwmk3boZBJmboQSCv6KGxeKuPU73Pk3TtostPXAMIdz7mXOnCAVXIPjfFuVldW19Y3qZm1re2d3z94/6OgkU5S1aSIS1QuIZoJL1gYOgvVSxUgcCNYNxjeF331kSvNE3sMkZX5MhpJHnBIw0sA+8oJEhHoSmyv3YMSATB/GA7vuNJwZ8DJxS1JHJVoD+8sLE5rFTAIVROu+66Tg50QBp4JNa16mWUromAxZ31BJYqb9fBZ/ik+NEuIoUeZIwDP190ZOYl0kNJMxgZFe9ArxP6+fQXTl51ymGTBJ5w9FmcCQ4KILHHLFKIiJIYQqbrJiOiKKUDCN1UwJ7uKXl0nnvOE6Dffuot68LuuoomN0gs6Qiy5RE92iFmojinL0jF7Rm/VkvVjv1sd8tGKVO4foD6zPHzBnlkM=</latexit><latexit sha1_base64="QFufTeJ1sA/bV2evRx5P9WHmcSE=">AAAB/nicbVDLSsNAFJ3UV62vqLhyM1gEVyURQZdFNy4r2Ac0sUwmk3boZBJmboQSCv6KGxeKuPU73Pk3TtostPXAMIdz7mXOnCAVXIPjfFuVldW19Y3qZm1re2d3z94/6OgkU5S1aSIS1QuIZoJL1gYOgvVSxUgcCNYNxjeF331kSvNE3sMkZX5MhpJHnBIw0sA+8oJEhHoSmyv3YMSATB/GA7vuNJwZ8DJxS1JHJVoD+8sLE5rFTAIVROu+66Tg50QBp4JNa16mWUromAxZ31BJYqb9fBZ/ik+NEuIoUeZIwDP190ZOYl0kNJMxgZFe9ArxP6+fQXTl51ymGTBJ5w9FmcCQ4KILHHLFKIiJIYQqbrJiOiKKUDCN1UwJ7uKXl0nnvOE6Dffuot68LuuoomN0gs6Qiy5RE92iFmojinL0jF7Rm/VkvVjv1sd8tGKVO4foD6zPHzBnlkM=</latexit><latexit sha1_base64="QFufTeJ1sA/bV2evRx5P9WHmcSE=">AAAB/nicbVDLSsNAFJ3UV62vqLhyM1gEVyURQZdFNy4r2Ac0sUwmk3boZBJmboQSCv6KGxeKuPU73Pk3TtostPXAMIdz7mXOnCAVXIPjfFuVldW19Y3qZm1re2d3z94/6OgkU5S1aSIS1QuIZoJL1gYOgvVSxUgcCNYNxjeF331kSvNE3sMkZX5MhpJHnBIw0sA+8oJEhHoSmyv3YMSATB/GA7vuNJwZ8DJxS1JHJVoD+8sLE5rFTAIVROu+66Tg50QBp4JNa16mWUromAxZ31BJYqb9fBZ/ik+NEuIoUeZIwDP190ZOYl0kNJMxgZFe9ArxP6+fQXTl51ymGTBJ5w9FmcCQ4KILHHLFKIiJIYQqbrJiOiKKUDCN1UwJ7uKXl0nnvOE6Dffuot68LuuoomN0gs6Qiy5RE92iFmojinL0jF7Rm/VkvVjv1sd8tGKVO4foD6zPHzBnlkM=</latexit>

✓k<latexit sha1_base64="QFufTeJ1sA/bV2evRx5P9WHmcSE=">AAAB/nicbVDLSsNAFJ3UV62vqLhyM1gEVyURQZdFNy4r2Ac0sUwmk3boZBJmboQSCv6KGxeKuPU73Pk3TtostPXAMIdz7mXOnCAVXIPjfFuVldW19Y3qZm1re2d3z94/6OgkU5S1aSIS1QuIZoJL1gYOgvVSxUgcCNYNxjeF331kSvNE3sMkZX5MhpJHnBIw0sA+8oJEhHoSmyv3YMSATB/GA7vuNJwZ8DJxS1JHJVoD+8sLE5rFTAIVROu+66Tg50QBp4JNa16mWUromAxZ31BJYqb9fBZ/ik+NEuIoUeZIwDP190ZOYl0kNJMxgZFe9ArxP6+fQXTl51ymGTBJ5w9FmcCQ4KILHHLFKIiJIYQqbrJiOiKKUDCN1UwJ7uKXl0nnvOE6Dffuot68LuuoomN0gs6Qiy5RE92iFmojinL0jF7Rm/VkvVjv1sd8tGKVO4foD6zPHzBnlkM=</latexit><latexit sha1_base64="QFufTeJ1sA/bV2evRx5P9WHmcSE=">AAAB/nicbVDLSsNAFJ3UV62vqLhyM1gEVyURQZdFNy4r2Ac0sUwmk3boZBJmboQSCv6KGxeKuPU73Pk3TtostPXAMIdz7mXOnCAVXIPjfFuVldW19Y3qZm1re2d3z94/6OgkU5S1aSIS1QuIZoJL1gYOgvVSxUgcCNYNxjeF331kSvNE3sMkZX5MhpJHnBIw0sA+8oJEhHoSmyv3YMSATB/GA7vuNJwZ8DJxS1JHJVoD+8sLE5rFTAIVROu+66Tg50QBp4JNa16mWUromAxZ31BJYqb9fBZ/ik+NEuIoUeZIwDP190ZOYl0kNJMxgZFe9ArxP6+fQXTl51ymGTBJ5w9FmcCQ4KILHHLFKIiJIYQqbrJiOiKKUDCN1UwJ7uKXl0nnvOE6Dffuot68LuuoomN0gs6Qiy5RE92iFmojinL0jF7Rm/VkvVjv1sd8tGKVO4foD6zPHzBnlkM=</latexit><latexit sha1_base64="QFufTeJ1sA/bV2evRx5P9WHmcSE=">AAAB/nicbVDLSsNAFJ3UV62vqLhyM1gEVyURQZdFNy4r2Ac0sUwmk3boZBJmboQSCv6KGxeKuPU73Pk3TtostPXAMIdz7mXOnCAVXIPjfFuVldW19Y3qZm1re2d3z94/6OgkU5S1aSIS1QuIZoJL1gYOgvVSxUgcCNYNxjeF331kSvNE3sMkZX5MhpJHnBIw0sA+8oJEhHoSmyv3YMSATB/GA7vuNJwZ8DJxS1JHJVoD+8sLE5rFTAIVROu+66Tg50QBp4JNa16mWUromAxZ31BJYqb9fBZ/ik+NEuIoUeZIwDP190ZOYl0kNJMxgZFe9ArxP6+fQXTl51ymGTBJ5w9FmcCQ4KILHHLFKIiJIYQqbrJiOiKKUDCN1UwJ7uKXl0nnvOE6Dffuot68LuuoomN0gs6Qiy5RE92iFmojinL0jF7Rm/VkvVjv1sd8tGKVO4foD6zPHzBnlkM=</latexit><latexit sha1_base64="QFufTeJ1sA/bV2evRx5P9WHmcSE=">AAAB/nicbVDLSsNAFJ3UV62vqLhyM1gEVyURQZdFNy4r2Ac0sUwmk3boZBJmboQSCv6KGxeKuPU73Pk3TtostPXAMIdz7mXOnCAVXIPjfFuVldW19Y3qZm1re2d3z94/6OgkU5S1aSIS1QuIZoJL1gYOgvVSxUgcCNYNxjeF331kSvNE3sMkZX5MhpJHnBIw0sA+8oJEhHoSmyv3YMSATB/GA7vuNJwZ8DJxS1JHJVoD+8sLE5rFTAIVROu+66Tg50QBp4JNa16mWUromAxZ31BJYqb9fBZ/ik+NEuIoUeZIwDP190ZOYl0kNJMxgZFe9ArxP6+fQXTl51ymGTBJ5w9FmcCQ4KILHHLFKIiJIYQqbrJiOiKKUDCN1UwJ7uKXl0nnvOE6Dffuot68LuuoomN0gs6Qiy5RE92iFmojinL0jF7Rm/VkvVjv1sd8tGKVO4foD6zPHzBnlkM=</latexit>

rLm

�✓k�

<latexit sha1_base64="I1eQ3hO3bdPUWRLe+PkoBFzGUVo=">AAACGHicbVBNS8NAEN34WetX1KOXxSLopSYi6LHoxYOHClaFJpbJdtMu3WzC7kQooT/Di3/FiwdFvPbmv3FTe/DrwbKP92aYmRdlUhj0vA9nZnZufmGxslRdXlldW3c3Nq9NmmvGWyyVqb6NwHApFG+hQMlvM80hiSS/iQZnpX9zz7URqbrCYcbDBHpKxIIBWqnjHgQKIglFwEDSi1EnCSLR2wuiVHbNMLFfEWCfI4zuBqWz33FrXt2bgP4l/pTUyBTNjjsOuinLE66QSTCm7XsZhgVoFEzyUTXIDc+ADaDH25YqSLgJi8lhI7prlS6NU22fQjpRv3cUkJhyTVuZAPbNb68U//PaOcYnYSFUliNX7GtQnEuKKS1Tol2hOUM5tASYFnZXyvqggaHNsmpD8H+f/JdcH9Z9r+5fHtUap9M4KmSb7JA94pNj0iDnpElahJEH8kReyKvz6Dw7b877V+mMM+3ZIj/gjD8Br/ygyg==</latexit><latexit sha1_base64="I1eQ3hO3bdPUWRLe+PkoBFzGUVo=">AAACGHicbVBNS8NAEN34WetX1KOXxSLopSYi6LHoxYOHClaFJpbJdtMu3WzC7kQooT/Di3/FiwdFvPbmv3FTe/DrwbKP92aYmRdlUhj0vA9nZnZufmGxslRdXlldW3c3Nq9NmmvGWyyVqb6NwHApFG+hQMlvM80hiSS/iQZnpX9zz7URqbrCYcbDBHpKxIIBWqnjHgQKIglFwEDSi1EnCSLR2wuiVHbNMLFfEWCfI4zuBqWz33FrXt2bgP4l/pTUyBTNjjsOuinLE66QSTCm7XsZhgVoFEzyUTXIDc+ADaDH25YqSLgJi8lhI7prlS6NU22fQjpRv3cUkJhyTVuZAPbNb68U//PaOcYnYSFUliNX7GtQnEuKKS1Tol2hOUM5tASYFnZXyvqggaHNsmpD8H+f/JdcH9Z9r+5fHtUap9M4KmSb7JA94pNj0iDnpElahJEH8kReyKvz6Dw7b877V+mMM+3ZIj/gjD8Br/ygyg==</latexit><latexit sha1_base64="I1eQ3hO3bdPUWRLe+PkoBFzGUVo=">AAACGHicbVBNS8NAEN34WetX1KOXxSLopSYi6LHoxYOHClaFJpbJdtMu3WzC7kQooT/Di3/FiwdFvPbmv3FTe/DrwbKP92aYmRdlUhj0vA9nZnZufmGxslRdXlldW3c3Nq9NmmvGWyyVqb6NwHApFG+hQMlvM80hiSS/iQZnpX9zz7URqbrCYcbDBHpKxIIBWqnjHgQKIglFwEDSi1EnCSLR2wuiVHbNMLFfEWCfI4zuBqWz33FrXt2bgP4l/pTUyBTNjjsOuinLE66QSTCm7XsZhgVoFEzyUTXIDc+ADaDH25YqSLgJi8lhI7prlS6NU22fQjpRv3cUkJhyTVuZAPbNb68U//PaOcYnYSFUliNX7GtQnEuKKS1Tol2hOUM5tASYFnZXyvqggaHNsmpD8H+f/JdcH9Z9r+5fHtUap9M4KmSb7JA94pNj0iDnpElahJEH8kReyKvz6Dw7b877V+mMM+3ZIj/gjD8Br/ygyg==</latexit><latexit sha1_base64="hP+6LrUf2d3tZaldqaQQvEKMXyw=">AAAB2XicbZDNSgMxFIXv1L86Vq1rN8EiuCozbnQpuHFZwbZCO5RM5k4bmskMyR2hDH0BF25EfC93vo3pz0JbDwQ+zknIvSculLQUBN9ebWd3b/+gfugfNfzjk9Nmo2fz0gjsilzl5jnmFpXU2CVJCp8LgzyLFfbj6f0i77+gsTLXTzQrMMr4WMtUCk7O6oyaraAdLMW2IVxDC9YaNb+GSS7KDDUJxa0dhEFBUcUNSaFw7g9LiwUXUz7GgUPNM7RRtRxzzi6dk7A0N+5oYkv394uKZ9bOstjdzDhN7Ga2MP/LBiWlt1EldVESarH6KC0Vo5wtdmaJNChIzRxwYaSblYkJN1yQa8Z3HYSbG29D77odBu3wMYA6nMMFXEEIN3AHD9CBLghI4BXevYn35n2suqp569LO4I+8zx84xIo4</latexit><latexit sha1_base64="s6cFni5L/+zPwvOelVDQmdiDQvU=">AAACDXicbVA9SwNBEJ3z2xg12tosBiE28c5GS8HGwkLBRCEXw9xmkyzZ2zt254Rw5GfY+FdsLBSxtfPfuJekUOODZR7vzbAzL0qVtOT7X97C4tLyyuraemmjvLm1XdkpN22SGS4aPFGJuYvQCiW1aJAkJe5SIzCOlLiNhueFf/sgjJWJvqFRKtox9rXsSY7kpE7lKNQYKcxDjopdjjtxGMl+LYwS1bWj2JU8pIEgHN8PC+ewU6n6dX8CNk+CGanCDFedymfYTXgWC01cobWtwE+pnaMhyZUYl8LMihT5EPui5ajGWNh2PjlszA6c0mW9xLiniU3UnxM5xrZY03XGSAP71yvE/7xWRr3Tdi51mpHQfPpRL1OMElakxLrSCE5q5AhyI92ujA/QICeXZcmFEPw9eZ40j+uBXw+ufViDPdiHGgRwAmdwAVfQAA6P8Ayv8OY9eS/e+zSuBW+W2y78gvfxDcoNn0c=</latexit><latexit sha1_base64="s6cFni5L/+zPwvOelVDQmdiDQvU=">AAACDXicbVA9SwNBEJ3z2xg12tosBiE28c5GS8HGwkLBRCEXw9xmkyzZ2zt254Rw5GfY+FdsLBSxtfPfuJekUOODZR7vzbAzL0qVtOT7X97C4tLyyuraemmjvLm1XdkpN22SGS4aPFGJuYvQCiW1aJAkJe5SIzCOlLiNhueFf/sgjJWJvqFRKtox9rXsSY7kpE7lKNQYKcxDjopdjjtxGMl+LYwS1bWj2JU8pIEgHN8PC+ewU6n6dX8CNk+CGanCDFedymfYTXgWC01cobWtwE+pnaMhyZUYl8LMihT5EPui5ajGWNh2PjlszA6c0mW9xLiniU3UnxM5xrZY03XGSAP71yvE/7xWRr3Tdi51mpHQfPpRL1OMElakxLrSCE5q5AhyI92ujA/QICeXZcmFEPw9eZ40j+uBXw+ufViDPdiHGgRwAmdwAVfQAA6P8Ayv8OY9eS/e+zSuBW+W2y78gvfxDcoNn0c=</latexit><latexit sha1_base64="AD9+7Z/64N0L6c5CVtCF9t7I5SQ=">AAACGHicbVA9TwJBEN3zE/ELtbTZSEywwTsbLYk2FhaYyEfCIZlbFtiwt3fZnTMhF36GjX/FxkJjbOn8N+4BhYIv2ezLezOZmRfEUhh03W9nZXVtfWMzt5Xf3tnd2y8cHNZNlGjGayySkW4GYLgUitdQoOTNWHMIA8kbwfAm8xtPXBsRqQccxbwdQl+JnmCAVuoUzn0FgYTUZyDp3bgT+oHol/wgkl0zCu2X+jjgCOPHYeacdQpFt+xOQZeJNydFMke1U5j43YglIVfIJBjT8twY2yloFEzycd5PDI+BDaHPW5YqCLlpp9PDxvTUKl3ai7R9CulU/d2RQmiyNW1lCDgwi14m/ue1EuxdtVOh4gS5YrNBvURSjGiWEu0KzRnKkSXAtLC7UjYADQxtlnkbgrd48jKpX5Q9t+zdu8XK9TyOHDkmJ6REPHJJKuSWVEmNMPJMXsk7+XBenDfn0/mala44854j8gfO5AeuvKDG</latexit><latexit sha1_base64="I1eQ3hO3bdPUWRLe+PkoBFzGUVo=">AAACGHicbVBNS8NAEN34WetX1KOXxSLopSYi6LHoxYOHClaFJpbJdtMu3WzC7kQooT/Di3/FiwdFvPbmv3FTe/DrwbKP92aYmRdlUhj0vA9nZnZufmGxslRdXlldW3c3Nq9NmmvGWyyVqb6NwHApFG+hQMlvM80hiSS/iQZnpX9zz7URqbrCYcbDBHpKxIIBWqnjHgQKIglFwEDSi1EnCSLR2wuiVHbNMLFfEWCfI4zuBqWz33FrXt2bgP4l/pTUyBTNjjsOuinLE66QSTCm7XsZhgVoFEzyUTXIDc+ADaDH25YqSLgJi8lhI7prlS6NU22fQjpRv3cUkJhyTVuZAPbNb68U//PaOcYnYSFUliNX7GtQnEuKKS1Tol2hOUM5tASYFnZXyvqggaHNsmpD8H+f/JdcH9Z9r+5fHtUap9M4KmSb7JA94pNj0iDnpElahJEH8kReyKvz6Dw7b877V+mMM+3ZIj/gjD8Br/ygyg==</latexit><latexit sha1_base64="I1eQ3hO3bdPUWRLe+PkoBFzGUVo=">AAACGHicbVBNS8NAEN34WetX1KOXxSLopSYi6LHoxYOHClaFJpbJdtMu3WzC7kQooT/Di3/FiwdFvPbmv3FTe/DrwbKP92aYmRdlUhj0vA9nZnZufmGxslRdXlldW3c3Nq9NmmvGWyyVqb6NwHApFG+hQMlvM80hiSS/iQZnpX9zz7URqbrCYcbDBHpKxIIBWqnjHgQKIglFwEDSi1EnCSLR2wuiVHbNMLFfEWCfI4zuBqWz33FrXt2bgP4l/pTUyBTNjjsOuinLE66QSTCm7XsZhgVoFEzyUTXIDc+ADaDH25YqSLgJi8lhI7prlS6NU22fQjpRv3cUkJhyTVuZAPbNb68U//PaOcYnYSFUliNX7GtQnEuKKS1Tol2hOUM5tASYFnZXyvqggaHNsmpD8H+f/JdcH9Z9r+5fHtUap9M4KmSb7JA94pNj0iDnpElahJEH8kReyKvz6Dw7b877V+mMM+3ZIj/gjD8Br/ygyg==</latexit><latexit sha1_base64="I1eQ3hO3bdPUWRLe+PkoBFzGUVo=">AAACGHicbVBNS8NAEN34WetX1KOXxSLopSYi6LHoxYOHClaFJpbJdtMu3WzC7kQooT/Di3/FiwdFvPbmv3FTe/DrwbKP92aYmRdlUhj0vA9nZnZufmGxslRdXlldW3c3Nq9NmmvGWyyVqb6NwHApFG+hQMlvM80hiSS/iQZnpX9zz7URqbrCYcbDBHpKxIIBWqnjHgQKIglFwEDSi1EnCSLR2wuiVHbNMLFfEWCfI4zuBqWz33FrXt2bgP4l/pTUyBTNjjsOuinLE66QSTCm7XsZhgVoFEzyUTXIDc+ADaDH25YqSLgJi8lhI7prlS6NU22fQjpRv3cUkJhyTVuZAPbNb68U//PaOcYnYSFUliNX7GtQnEuKKS1Tol2hOUM5tASYFnZXyvqggaHNsmpD8H+f/JdcH9Z9r+5fHtUap9M4KmSb7JA94pNj0iDnpElahJEH8kReyKvz6Dw7b877V+mMM+3ZIj/gjD8Br/ygyg==</latexit><latexit sha1_base64="I1eQ3hO3bdPUWRLe+PkoBFzGUVo=">AAACGHicbVBNS8NAEN34WetX1KOXxSLopSYi6LHoxYOHClaFJpbJdtMu3WzC7kQooT/Di3/FiwdFvPbmv3FTe/DrwbKP92aYmRdlUhj0vA9nZnZufmGxslRdXlldW3c3Nq9NmmvGWyyVqb6NwHApFG+hQMlvM80hiSS/iQZnpX9zz7URqbrCYcbDBHpKxIIBWqnjHgQKIglFwEDSi1EnCSLR2wuiVHbNMLFfEWCfI4zuBqWz33FrXt2bgP4l/pTUyBTNjjsOuinLE66QSTCm7XsZhgVoFEzyUTXIDc+ADaDH25YqSLgJi8lhI7prlS6NU22fQjpRv3cUkJhyTVuZAPbNb68U//PaOcYnYSFUliNX7GtQnEuKKS1Tol2hOUM5tASYFnZXyvqggaHNsmpD8H+f/JdcH9Z9r+5fHtUap9M4KmSb7JA94pNj0iDnpElahJEH8kReyKvz6Dw7b877V+mMM+3ZIj/gjD8Br/ygyg==</latexit><latexit sha1_base64="I1eQ3hO3bdPUWRLe+PkoBFzGUVo=">AAACGHicbVBNS8NAEN34WetX1KOXxSLopSYi6LHoxYOHClaFJpbJdtMu3WzC7kQooT/Di3/FiwdFvPbmv3FTe/DrwbKP92aYmRdlUhj0vA9nZnZufmGxslRdXlldW3c3Nq9NmmvGWyyVqb6NwHApFG+hQMlvM80hiSS/iQZnpX9zz7URqbrCYcbDBHpKxIIBWqnjHgQKIglFwEDSi1EnCSLR2wuiVHbNMLFfEWCfI4zuBqWz33FrXt2bgP4l/pTUyBTNjjsOuinLE66QSTCm7XsZhgVoFEzyUTXIDc+ADaDH25YqSLgJi8lhI7prlS6NU22fQjpRv3cUkJhyTVuZAPbNb68U//PaOcYnYSFUliNX7GtQnEuKKS1Tol2hOUM5tASYFnZXyvqggaHNsmpD8H+f/JdcH9Z9r+5fHtUap9M4KmSb7JA94pNj0iDnpElahJEH8kReyKvz6Dw7b877V+mMM+3ZIj/gjD8Br/ygyg==</latexit><latexit sha1_base64="I1eQ3hO3bdPUWRLe+PkoBFzGUVo=">AAACGHicbVBNS8NAEN34WetX1KOXxSLopSYi6LHoxYOHClaFJpbJdtMu3WzC7kQooT/Di3/FiwdFvPbmv3FTe/DrwbKP92aYmRdlUhj0vA9nZnZufmGxslRdXlldW3c3Nq9NmmvGWyyVqb6NwHApFG+hQMlvM80hiSS/iQZnpX9zz7URqbrCYcbDBHpKxIIBWqnjHgQKIglFwEDSi1EnCSLR2wuiVHbNMLFfEWCfI4zuBqWz33FrXt2bgP4l/pTUyBTNjjsOuinLE66QSTCm7XsZhgVoFEzyUTXIDc+ADaDH25YqSLgJi8lhI7prlS6NU22fQjpRv3cUkJhyTVuZAPbNb68U//PaOcYnYSFUliNX7GtQnEuKKS1Tol2hOUM5tASYFnZXyvqggaHNsmpD8H+f/JdcH9Z9r+5fHtUap9M4KmSb7JA94pNj0iDnpElahJEH8kReyKvz6Dw7b877V+mMM+3ZIj/gjD8Br/ygyg==</latexit>

rL1

�✓k�

<latexit sha1_base64="0izSstp1VXcdyBHaas99nooR5EY=">AAACGHicbVA9SwNBEN2L3/ErammzGITYxDsRtBRtLCwimA/IxTC32SRL9vaO3TkhHPkZNv4VGwtFbNP5b9xLUmjig2Uf780wMy+IpTDout9Obml5ZXVtfSO/ubW9s1vY26+ZKNGMV1kkI90IwHApFK+iQMkbseYQBpLXg8FN5tefuDYiUg84jHkrhJ4SXcEArdQunPoKAgmpz0DSu1Hb8wPRK/lBJDtmGNov9bHPEUaPg8w5aReKbtmdgC4Sb0aKZIZKuzD2OxFLQq6QSTCm6bkxtlLQKJjko7yfGB4DG0CPNy1VEHLTSieHjeixVTq0G2n7FNKJ+rsjhdBka9rKELBv5r1M/M9rJti9bKVCxQlyxaaDuomkGNEsJdoRmjOUQ0uAaWF3pawPGhjaLPM2BG/+5EVSOyt7btm7Py9eXc/iWCeH5IiUiEcuyBW5JRVSJYw8k1fyTj6cF+fN+XS+pqU5Z9ZzQP7AGf8ATcigjg==</latexit><latexit sha1_base64="0izSstp1VXcdyBHaas99nooR5EY=">AAACGHicbVA9SwNBEN2L3/ErammzGITYxDsRtBRtLCwimA/IxTC32SRL9vaO3TkhHPkZNv4VGwtFbNP5b9xLUmjig2Uf780wMy+IpTDout9Obml5ZXVtfSO/ubW9s1vY26+ZKNGMV1kkI90IwHApFK+iQMkbseYQBpLXg8FN5tefuDYiUg84jHkrhJ4SXcEArdQunPoKAgmpz0DSu1Hb8wPRK/lBJDtmGNov9bHPEUaPg8w5aReKbtmdgC4Sb0aKZIZKuzD2OxFLQq6QSTCm6bkxtlLQKJjko7yfGB4DG0CPNy1VEHLTSieHjeixVTq0G2n7FNKJ+rsjhdBka9rKELBv5r1M/M9rJti9bKVCxQlyxaaDuomkGNEsJdoRmjOUQ0uAaWF3pawPGhjaLPM2BG/+5EVSOyt7btm7Py9eXc/iWCeH5IiUiEcuyBW5JRVSJYw8k1fyTj6cF+fN+XS+pqU5Z9ZzQP7AGf8ATcigjg==</latexit><latexit sha1_base64="0izSstp1VXcdyBHaas99nooR5EY=">AAACGHicbVA9SwNBEN2L3/ErammzGITYxDsRtBRtLCwimA/IxTC32SRL9vaO3TkhHPkZNv4VGwtFbNP5b9xLUmjig2Uf780wMy+IpTDout9Obml5ZXVtfSO/ubW9s1vY26+ZKNGMV1kkI90IwHApFK+iQMkbseYQBpLXg8FN5tefuDYiUg84jHkrhJ4SXcEArdQunPoKAgmpz0DSu1Hb8wPRK/lBJDtmGNov9bHPEUaPg8w5aReKbtmdgC4Sb0aKZIZKuzD2OxFLQq6QSTCm6bkxtlLQKJjko7yfGB4DG0CPNy1VEHLTSieHjeixVTq0G2n7FNKJ+rsjhdBka9rKELBv5r1M/M9rJti9bKVCxQlyxaaDuomkGNEsJdoRmjOUQ0uAaWF3pawPGhjaLPM2BG/+5EVSOyt7btm7Py9eXc/iWCeH5IiUiEcuyBW5JRVSJYw8k1fyTj6cF+fN+XS+pqU5Z9ZzQP7AGf8ATcigjg==</latexit><latexit sha1_base64="0izSstp1VXcdyBHaas99nooR5EY=">AAACGHicbVA9SwNBEN2L3/ErammzGITYxDsRtBRtLCwimA/IxTC32SRL9vaO3TkhHPkZNv4VGwtFbNP5b9xLUmjig2Uf780wMy+IpTDout9Obml5ZXVtfSO/ubW9s1vY26+ZKNGMV1kkI90IwHApFK+iQMkbseYQBpLXg8FN5tefuDYiUg84jHkrhJ4SXcEArdQunPoKAgmpz0DSu1Hb8wPRK/lBJDtmGNov9bHPEUaPg8w5aReKbtmdgC4Sb0aKZIZKuzD2OxFLQq6QSTCm6bkxtlLQKJjko7yfGB4DG0CPNy1VEHLTSieHjeixVTq0G2n7FNKJ+rsjhdBka9rKELBv5r1M/M9rJti9bKVCxQlyxaaDuomkGNEsJdoRmjOUQ0uAaWF3pawPGhjaLPM2BG/+5EVSOyt7btm7Py9eXc/iWCeH5IiUiEcuyBW5JRVSJYw8k1fyTj6cF+fN+XS+pqU5Z9ZzQP7AGf8ATcigjg==</latexit>

✓k+1 = ✓k � ↵X

m2Mk

rLm(✓k)� ↵X

m2M/Mk

rLm

�✓̂k�1

m

<latexit sha1_base64="xjayPFLH2d2e1FfhubKeZ30oZTE=">AAAC2HicbVJdixMxFM2MX2v92KqPvkSL0EW2zMiCvggLvvigsILdXWzqcCdNO6FJZkjuCCUM+KCIr/403/wV/gUzbcW1OzeEHM65OffmI6+UdJgkv6L4ytVr12/s3ezdun3n7n7/3v1TV9aWizEvVWnPc3BCSSPGKFGJ88oK0LkSZ/nyVauffRLWydK8x1UlphoWRs4lBwxU1v/N8lLN3EqHxTMsBELz0S+fps3LLmV5yEBVBbBHYbhaZ14zaTzjoOjbIDdrwUCugG7YN02mh51WB/SiWYcdcwpcsWve+1vgnz/L5WLICkDfUSjo4UCHadNmHWT9QTJK1kEvg3QLBmQbJ1n/J5uVvNbCIA/tuEmaVDj1YFFyJZoeq52ogC9hISYBGtDCTf36YRr6JDAzOi9tmAbpmr24w4N2bbchUwMWbldryS5tUuP8xdRLU9UoDN8UmteKYknbV6YzaQVHtQoAuJWhV8oLsMAx/IVeuIR098iXwemzUZqM0ndHg+Oj7XXskYfkMRmSlDwnx+Q1OSFjwqNx5KMv0df4Q/w5/hZ/36TG0XbPA/JfxD/+AHy55xQ=</latexit><latexit sha1_base64="xjayPFLH2d2e1FfhubKeZ30oZTE=">AAAC2HicbVJdixMxFM2MX2v92KqPvkSL0EW2zMiCvggLvvigsILdXWzqcCdNO6FJZkjuCCUM+KCIr/403/wV/gUzbcW1OzeEHM65OffmI6+UdJgkv6L4ytVr12/s3ezdun3n7n7/3v1TV9aWizEvVWnPc3BCSSPGKFGJ88oK0LkSZ/nyVauffRLWydK8x1UlphoWRs4lBwxU1v/N8lLN3EqHxTMsBELz0S+fps3LLmV5yEBVBbBHYbhaZ14zaTzjoOjbIDdrwUCugG7YN02mh51WB/SiWYcdcwpcsWve+1vgnz/L5WLICkDfUSjo4UCHadNmHWT9QTJK1kEvg3QLBmQbJ1n/J5uVvNbCIA/tuEmaVDj1YFFyJZoeq52ogC9hISYBGtDCTf36YRr6JDAzOi9tmAbpmr24w4N2bbchUwMWbldryS5tUuP8xdRLU9UoDN8UmteKYknbV6YzaQVHtQoAuJWhV8oLsMAx/IVeuIR098iXwemzUZqM0ndHg+Oj7XXskYfkMRmSlDwnx+Q1OSFjwqNx5KMv0df4Q/w5/hZ/36TG0XbPA/JfxD/+AHy55xQ=</latexit><latexit sha1_base64="xjayPFLH2d2e1FfhubKeZ30oZTE=">AAAC2HicbVJdixMxFM2MX2v92KqPvkSL0EW2zMiCvggLvvigsILdXWzqcCdNO6FJZkjuCCUM+KCIr/403/wV/gUzbcW1OzeEHM65OffmI6+UdJgkv6L4ytVr12/s3ezdun3n7n7/3v1TV9aWizEvVWnPc3BCSSPGKFGJ88oK0LkSZ/nyVauffRLWydK8x1UlphoWRs4lBwxU1v/N8lLN3EqHxTMsBELz0S+fps3LLmV5yEBVBbBHYbhaZ14zaTzjoOjbIDdrwUCugG7YN02mh51WB/SiWYcdcwpcsWve+1vgnz/L5WLICkDfUSjo4UCHadNmHWT9QTJK1kEvg3QLBmQbJ1n/J5uVvNbCIA/tuEmaVDj1YFFyJZoeq52ogC9hISYBGtDCTf36YRr6JDAzOi9tmAbpmr24w4N2bbchUwMWbldryS5tUuP8xdRLU9UoDN8UmteKYknbV6YzaQVHtQoAuJWhV8oLsMAx/IVeuIR098iXwemzUZqM0ndHg+Oj7XXskYfkMRmSlDwnx+Q1OSFjwqNx5KMv0df4Q/w5/hZ/36TG0XbPA/JfxD/+AHy55xQ=</latexit><latexit sha1_base64="xjayPFLH2d2e1FfhubKeZ30oZTE=">AAAC2HicbVJdixMxFM2MX2v92KqPvkSL0EW2zMiCvggLvvigsILdXWzqcCdNO6FJZkjuCCUM+KCIr/403/wV/gUzbcW1OzeEHM65OffmI6+UdJgkv6L4ytVr12/s3ezdun3n7n7/3v1TV9aWizEvVWnPc3BCSSPGKFGJ88oK0LkSZ/nyVauffRLWydK8x1UlphoWRs4lBwxU1v/N8lLN3EqHxTMsBELz0S+fps3LLmV5yEBVBbBHYbhaZ14zaTzjoOjbIDdrwUCugG7YN02mh51WB/SiWYcdcwpcsWve+1vgnz/L5WLICkDfUSjo4UCHadNmHWT9QTJK1kEvg3QLBmQbJ1n/J5uVvNbCIA/tuEmaVDj1YFFyJZoeq52ogC9hISYBGtDCTf36YRr6JDAzOi9tmAbpmr24w4N2bbchUwMWbldryS5tUuP8xdRLU9UoDN8UmteKYknbV6YzaQVHtQoAuJWhV8oLsMAx/IVeuIR098iXwemzUZqM0ndHg+Oj7XXskYfkMRmSlDwnx+Q1OSFjwqNx5KMv0df4Q/w5/hZ/36TG0XbPA/JfxD/+AHy55xQ=</latexit>

q Select a subset of workers to upload

q Remaining workers in do not uploadM/Mk<latexit sha1_base64="qvz+PV4H383Let9dvoj1mypjLYI=">AAACAnicbVDLSgMxFL1TX7W+Rl2Jm2ARXJWZUtBlwY0boYJ9QGcsmTTThmYyQ5IRylDc+CtuXCji1q9w59+YtiNo64HAyTn33uSeIOFMacf5sgorq2vrG8XN0tb2zu6evX/QUnEqCW2SmMeyE2BFORO0qZnmtJNIiqOA03Ywupz67XsqFYvFrR4n1I/wQLCQEayN1LOPMo9gjq4nnuJYDdHP9W7Us8tOxZkBLRM3J2XI0ejZn14/JmlEhSZmluq6TqL9DEvNCKeTkpcqmmAywgPaNVTgiCo/m60wQadG6aMwluYIjWbq744MR0qNo8BURlgP1aI3Ff/zuqkOL/yMiSTVVJD5Q2HKkY7RNA/UZ5ISzceGYCKZ+SsiQywx0Sa1kgnBXVx5mbSqFdepuDe1cr2ax1GEYziBM3DhHOpwBQ1oAoEHeIIXeLUerWfrzXqflxasvOcQ/sD6+AaVU5bX</latexit><latexit sha1_base64="qvz+PV4H383Let9dvoj1mypjLYI=">AAACAnicbVDLSgMxFL1TX7W+Rl2Jm2ARXJWZUtBlwY0boYJ9QGcsmTTThmYyQ5IRylDc+CtuXCji1q9w59+YtiNo64HAyTn33uSeIOFMacf5sgorq2vrG8XN0tb2zu6evX/QUnEqCW2SmMeyE2BFORO0qZnmtJNIiqOA03Ywupz67XsqFYvFrR4n1I/wQLCQEayN1LOPMo9gjq4nnuJYDdHP9W7Us8tOxZkBLRM3J2XI0ejZn14/JmlEhSZmluq6TqL9DEvNCKeTkpcqmmAywgPaNVTgiCo/m60wQadG6aMwluYIjWbq744MR0qNo8BURlgP1aI3Ff/zuqkOL/yMiSTVVJD5Q2HKkY7RNA/UZ5ISzceGYCKZ+SsiQywx0Sa1kgnBXVx5mbSqFdepuDe1cr2ax1GEYziBM3DhHOpwBQ1oAoEHeIIXeLUerWfrzXqflxasvOcQ/sD6+AaVU5bX</latexit><latexit sha1_base64="qvz+PV4H383Let9dvoj1mypjLYI=">AAACAnicbVDLSgMxFL1TX7W+Rl2Jm2ARXJWZUtBlwY0boYJ9QGcsmTTThmYyQ5IRylDc+CtuXCji1q9w59+YtiNo64HAyTn33uSeIOFMacf5sgorq2vrG8XN0tb2zu6evX/QUnEqCW2SmMeyE2BFORO0qZnmtJNIiqOA03Ywupz67XsqFYvFrR4n1I/wQLCQEayN1LOPMo9gjq4nnuJYDdHP9W7Us8tOxZkBLRM3J2XI0ejZn14/JmlEhSZmluq6TqL9DEvNCKeTkpcqmmAywgPaNVTgiCo/m60wQadG6aMwluYIjWbq744MR0qNo8BURlgP1aI3Ff/zuqkOL/yMiSTVVJD5Q2HKkY7RNA/UZ5ISzceGYCKZ+SsiQywx0Sa1kgnBXVx5mbSqFdepuDe1cr2ax1GEYziBM3DhHOpwBQ1oAoEHeIIXeLUerWfrzXqflxasvOcQ/sD6+AaVU5bX</latexit><latexit sha1_base64="qvz+PV4H383Let9dvoj1mypjLYI=">AAACAnicbVDLSgMxFL1TX7W+Rl2Jm2ARXJWZUtBlwY0boYJ9QGcsmTTThmYyQ5IRylDc+CtuXCji1q9w59+YtiNo64HAyTn33uSeIOFMacf5sgorq2vrG8XN0tb2zu6evX/QUnEqCW2SmMeyE2BFORO0qZnmtJNIiqOA03Ywupz67XsqFYvFrR4n1I/wQLCQEayN1LOPMo9gjq4nnuJYDdHP9W7Us8tOxZkBLRM3J2XI0ejZn14/JmlEhSZmluq6TqL9DEvNCKeTkpcqmmAywgPaNVTgiCo/m60wQadG6aMwluYIjWbq744MR0qNo8BURlgP1aI3Ff/zuqkOL/yMiSTVVJD5Q2HKkY7RNA/UZ5ISzceGYCKZ+SsiQywx0Sa1kgnBXVx5mbSqFdepuDe1cr2ax1GEYziBM3DhHOpwBQ1oAoEHeIIXeLUerWfrzXqflxasvOcQ/sD6+AaVU5bX</latexit>

Mk ✓ M<latexit sha1_base64="KL3ZLMNta00F+BxwcXvA28JXtUA=">AAACBXicbVBNS8NAEN3Ur1q/oh71sFgETyUpgh4LXrwIFewHNLFstpN26WYTdzdCCb148a948aCIV/+DN/+N2zaCtj4YeLw3w8y8IOFMacf5sgpLyyura8X10sbm1vaOvbvXVHEqKTRozGPZDogCzgQ0NNMc2okEEgUcWsHwYuK37kEqFosbPUrAj0hfsJBRoo3UtQ8zjxKOr8a3Q0+lgQINd/hH69plp+JMgReJm5MyylHv2p9eL6ZpBEJTTpTquE6i/YxIzSiHcclLFSSEDkkfOoYKEoHys+kXY3xslB4OY2lKaDxVf09kJFJqFAWmMyJ6oOa9ifif10l1eO5nTCSpBkFni8KUYx3jSSS4xyRQzUeGECqZuRXTAZGEahNcyYTgzr+8SJrViutU3OvTcq2ax1FEB+gInSAXnaEaukR11EAUPaAn9IJerUfr2Xqz3metBSuf2Ud/YH18AySFmEY=</latexit><latexit sha1_base64="KL3ZLMNta00F+BxwcXvA28JXtUA=">AAACBXicbVBNS8NAEN3Ur1q/oh71sFgETyUpgh4LXrwIFewHNLFstpN26WYTdzdCCb148a948aCIV/+DN/+N2zaCtj4YeLw3w8y8IOFMacf5sgpLyyura8X10sbm1vaOvbvXVHEqKTRozGPZDogCzgQ0NNMc2okEEgUcWsHwYuK37kEqFosbPUrAj0hfsJBRoo3UtQ8zjxKOr8a3Q0+lgQINd/hH69plp+JMgReJm5MyylHv2p9eL6ZpBEJTTpTquE6i/YxIzSiHcclLFSSEDkkfOoYKEoHys+kXY3xslB4OY2lKaDxVf09kJFJqFAWmMyJ6oOa9ifif10l1eO5nTCSpBkFni8KUYx3jSSS4xyRQzUeGECqZuRXTAZGEahNcyYTgzr+8SJrViutU3OvTcq2ax1FEB+gInSAXnaEaukR11EAUPaAn9IJerUfr2Xqz3metBSuf2Ud/YH18AySFmEY=</latexit><latexit sha1_base64="KL3ZLMNta00F+BxwcXvA28JXtUA=">AAACBXicbVBNS8NAEN3Ur1q/oh71sFgETyUpgh4LXrwIFewHNLFstpN26WYTdzdCCb148a948aCIV/+DN/+N2zaCtj4YeLw3w8y8IOFMacf5sgpLyyura8X10sbm1vaOvbvXVHEqKTRozGPZDogCzgQ0NNMc2okEEgUcWsHwYuK37kEqFosbPUrAj0hfsJBRoo3UtQ8zjxKOr8a3Q0+lgQINd/hH69plp+JMgReJm5MyylHv2p9eL6ZpBEJTTpTquE6i/YxIzSiHcclLFSSEDkkfOoYKEoHys+kXY3xslB4OY2lKaDxVf09kJFJqFAWmMyJ6oOa9ifif10l1eO5nTCSpBkFni8KUYx3jSSS4xyRQzUeGECqZuRXTAZGEahNcyYTgzr+8SJrViutU3OvTcq2ax1FEB+gInSAXnaEaukR11EAUPaAn9IJerUfr2Xqz3metBSuf2Ud/YH18AySFmEY=</latexit><latexit sha1_base64="KL3ZLMNta00F+BxwcXvA28JXtUA=">AAACBXicbVBNS8NAEN3Ur1q/oh71sFgETyUpgh4LXrwIFewHNLFstpN26WYTdzdCCb148a948aCIV/+DN/+N2zaCtj4YeLw3w8y8IOFMacf5sgpLyyura8X10sbm1vaOvbvXVHEqKTRozGPZDogCzgQ0NNMc2okEEgUcWsHwYuK37kEqFosbPUrAj0hfsJBRoo3UtQ8zjxKOr8a3Q0+lgQINd/hH69plp+JMgReJm5MyylHv2p9eL6ZpBEJTTpTquE6i/YxIzSiHcclLFSSEDkkfOoYKEoHys+kXY3xslB4OY2lKaDxVf09kJFJqFAWmMyJ6oOa9ifif10l1eO5nTCSpBkFni8KUYx3jSSS4xyRQzUeGECqZuRXTAZGEahNcyYTgzr+8SJrViutU3OvTcq2ax1FEB+gInSAXnaEaukR11EAUPaAn9IJerUfr2Xqz3metBSuf2Ud/YH18AySFmEY=</latexit>

Old gradientFresh gradient

Page 7: LAG: Lazily Aggregated Gradient for Communication ...06-15-30)-06-15-40-12765-LAG:... · 1 TianyiChen Georgios Giannakis Tao Sun WotaoYin UMN, ECE UCLA, Math NeurIPS2018 LAG: Lazily

7

LAG: GD under two alternative communication rules

T. Chen, G. B. Giannakis, T. Sun, and W. Yin "LAG: Lazily Aggregated Gradient for Communication-Efficient Distributed Learning," Proc. of NIPS, Montreal, Canada, December 3-8, 2018.

q Worker-side rule (LAG-WK): Include worker m in if

Gradient innovation Optimization progress

���rLm

�✓k��rLm

�✓̂k�1

m

���� � 1

M

����1

⇣✓k � ✓k�1

⌘����<latexit sha1_base64="sdBQ5wuLWPdC8PW6kv0JAW7GfI8=">AAACznicbVJNa9wwEJXdj6Tbr2177EV0KSSHXexQaI8hvRTawha6SWC1WcZa2RYry440DmwV02t/X2/9Af0flddbaBMPCD3em8eMZpRUSlqMol9BeOfuvft7+w8GDx89fvJ0+Oz5qS1rw8WMl6o05wlYoaQWM5SoxHllBBSJEmfJ+n2rn10JY2Wpv+KmEosCMi1TyQE9tRz+ZicyY9eUaUgUOMZB0U/NsmCJzA5YUqqV3RT+cgxzgdBcrFvlcNyfnwO6HpPXL9x6HDdbb1eRZeKSstQAd3HjPjdMiRQ9/ZdhoKocOrq/kXEf25UxMsvxsLvY9WCwHI6iSbQNehvEOzAiu5guhz/ZquR1ITRyBdbO46jChQODkivRDFhtRQV8DZmYe6ihEHbhtuto6GvPrGhaGn800i37r8NBYdu2fWYBmNubWkv2afMa03cLJ3VVo9C8K5TWimJJ293SlTSCo9p4ANxI3yvlOfiBov8B7RDim0++DU6PJnE0ib+8GR0f7caxT16SV+SAxOQtOSYfyJTMCA8+BpfBt8CF0/AqbMLvXWoY7DwvyH8R/vgDSIXldA==</latexit><latexit sha1_base64="sdBQ5wuLWPdC8PW6kv0JAW7GfI8=">AAACznicbVJNa9wwEJXdj6Tbr2177EV0KSSHXexQaI8hvRTawha6SWC1WcZa2RYry440DmwV02t/X2/9Af0flddbaBMPCD3em8eMZpRUSlqMol9BeOfuvft7+w8GDx89fvJ0+Oz5qS1rw8WMl6o05wlYoaQWM5SoxHllBBSJEmfJ+n2rn10JY2Wpv+KmEosCMi1TyQE9tRz+ZicyY9eUaUgUOMZB0U/NsmCJzA5YUqqV3RT+cgxzgdBcrFvlcNyfnwO6HpPXL9x6HDdbb1eRZeKSstQAd3HjPjdMiRQ9/ZdhoKocOrq/kXEf25UxMsvxsLvY9WCwHI6iSbQNehvEOzAiu5guhz/ZquR1ITRyBdbO46jChQODkivRDFhtRQV8DZmYe6ihEHbhtuto6GvPrGhaGn800i37r8NBYdu2fWYBmNubWkv2afMa03cLJ3VVo9C8K5TWimJJ293SlTSCo9p4ANxI3yvlOfiBov8B7RDim0++DU6PJnE0ib+8GR0f7caxT16SV+SAxOQtOSYfyJTMCA8+BpfBt8CF0/AqbMLvXWoY7DwvyH8R/vgDSIXldA==</latexit><latexit sha1_base64="sdBQ5wuLWPdC8PW6kv0JAW7GfI8=">AAACznicbVJNa9wwEJXdj6Tbr2177EV0KSSHXexQaI8hvRTawha6SWC1WcZa2RYry440DmwV02t/X2/9Af0flddbaBMPCD3em8eMZpRUSlqMol9BeOfuvft7+w8GDx89fvJ0+Oz5qS1rw8WMl6o05wlYoaQWM5SoxHllBBSJEmfJ+n2rn10JY2Wpv+KmEosCMi1TyQE9tRz+ZicyY9eUaUgUOMZB0U/NsmCJzA5YUqqV3RT+cgxzgdBcrFvlcNyfnwO6HpPXL9x6HDdbb1eRZeKSstQAd3HjPjdMiRQ9/ZdhoKocOrq/kXEf25UxMsvxsLvY9WCwHI6iSbQNehvEOzAiu5guhz/ZquR1ITRyBdbO46jChQODkivRDFhtRQV8DZmYe6ihEHbhtuto6GvPrGhaGn800i37r8NBYdu2fWYBmNubWkv2afMa03cLJ3VVo9C8K5TWimJJ293SlTSCo9p4ANxI3yvlOfiBov8B7RDim0++DU6PJnE0ib+8GR0f7caxT16SV+SAxOQtOSYfyJTMCA8+BpfBt8CF0/AqbMLvXWoY7DwvyH8R/vgDSIXldA==</latexit><latexit sha1_base64="hP+6LrUf2d3tZaldqaQQvEKMXyw=">AAAB2XicbZDNSgMxFIXv1L86Vq1rN8EiuCozbnQpuHFZwbZCO5RM5k4bmskMyR2hDH0BF25EfC93vo3pz0JbDwQ+zknIvSculLQUBN9ebWd3b/+gfugfNfzjk9Nmo2fz0gjsilzl5jnmFpXU2CVJCp8LgzyLFfbj6f0i77+gsTLXTzQrMMr4WMtUCk7O6oyaraAdLMW2IVxDC9YaNb+GSS7KDDUJxa0dhEFBUcUNSaFw7g9LiwUXUz7GgUPNM7RRtRxzzi6dk7A0N+5oYkv394uKZ9bOstjdzDhN7Ga2MP/LBiWlt1EldVESarH6KC0Vo5wtdmaJNChIzRxwYaSblYkJN1yQa8Z3HYSbG29D77odBu3wMYA6nMMFXEEIN3AHD9CBLghI4BXevYn35n2suqp569LO4I+8zx84xIo4</latexit><latexit sha1_base64="0R4hPLkwHklCrgKU4UNeGnfaJvs=">AAACw3icbVJNb9NAEF2brxIKBK5cVlRI7SGRzQWOCC5IgBQk0lbKptF4M7ZXWX90d1wpbC3+Izd+AP+DdRwkaD3Sap/em9HMvtmk1spSFP0Kwjt3791/cPBw9Ojw8ZOn42eHp7ZqjMS5rHRlzhOwqFWJc1Kk8bw2CEWi8SzZfOj0sys0VlXlN9rWuCwgK1WqJJCnVuPf4r3KxDUXJSQanJCg+ed2VYhEZcciqfTabgt/OUE5ErQXm045mQzn50BuoMjrF24zidtdbd9RZHjJRWpAurh1X1qhMSVP/2UE6DqHnh4eZDLE9m2MynI66S9xPRqtxkfRNNoFvw3iPThi+5itxj/FupJNgSVJDdYu4qimpQNDSmpsR6KxWIPcQIYLD0so0C7dbh0tf+WZNU8r409JfMf+W+GgsN3YPrMAyu1NrSOHtEVD6dulU2XdEJayb5Q2mlPFu93ytTIoSW89AGmUn5XLHLyh5H9AZ0J888m3wenraRxN468RO2Av2Et2zGL2hr1jH9mMzZkMPgWXwffAhbPwKmx7u8Jg79tz9l+EP/4AYCLkSg==</latexit><latexit sha1_base64="0R4hPLkwHklCrgKU4UNeGnfaJvs=">AAACw3icbVJNb9NAEF2brxIKBK5cVlRI7SGRzQWOCC5IgBQk0lbKptF4M7ZXWX90d1wpbC3+Izd+AP+DdRwkaD3Sap/em9HMvtmk1spSFP0Kwjt3791/cPBw9Ojw8ZOn42eHp7ZqjMS5rHRlzhOwqFWJc1Kk8bw2CEWi8SzZfOj0sys0VlXlN9rWuCwgK1WqJJCnVuPf4r3KxDUXJSQanJCg+ed2VYhEZcciqfTabgt/OUE5ErQXm045mQzn50BuoMjrF24zidtdbd9RZHjJRWpAurh1X1qhMSVP/2UE6DqHnh4eZDLE9m2MynI66S9xPRqtxkfRNNoFvw3iPThi+5itxj/FupJNgSVJDdYu4qimpQNDSmpsR6KxWIPcQIYLD0so0C7dbh0tf+WZNU8r409JfMf+W+GgsN3YPrMAyu1NrSOHtEVD6dulU2XdEJayb5Q2mlPFu93ytTIoSW89AGmUn5XLHLyh5H9AZ0J888m3wenraRxN468RO2Av2Et2zGL2hr1jH9mMzZkMPgWXwffAhbPwKmx7u8Jg79tz9l+EP/4AYCLkSg==</latexit><latexit sha1_base64="CkJErppo0e1xvgGRANXgvbPzUtg=">AAACznicbVJNa9wwEJXdr3T7tWmPvYguheSwi51LegzNpdAWttBNAqvNMtbKtlhZdqRxYKuYXvv7eusP6P+ovN5Cm3hA6PHePGY0o6RS0mIU/QrCe/cfPHy093jw5Omz5y+G+y/PbFkbLma8VKW5SMAKJbWYoUQlLiojoEiUOE/Wp61+fi2MlaX+iptKLArItEwlB/TUcvibvZcZu6FMQ6LAMQ6KfmqWBUtkdsCSUq3spvCXY5gLhOZy3SqH4/78HND1mLx+6dbjuNl6u4osE1eUpQa4ixv3uWFKpOjpvwwDVeXQ0f2NjPvYroyRWY6H3cVuBoPlcBRNom3QuyDegRHZxXQ5/MlWJa8LoZErsHYeRxUuHBiUXIlmwGorKuBryMTcQw2FsAu3XUdD33pmRdPS+KORbtl/HQ4K27btMwvA3N7WWrJPm9eYvls4qasaheZdobRWFEva7paupBEc1cYD4Eb6XinPwQ8U/Q9ohxDffvJdcHY0iaNJ/CUanRztxrFHXpM35IDE5JickA9kSmaEBx+Dq+Bb4MJpeB024fcuNQx2nlfkvwh//AFHReVw</latexit><latexit sha1_base64="sdBQ5wuLWPdC8PW6kv0JAW7GfI8=">AAACznicbVJNa9wwEJXdj6Tbr2177EV0KSSHXexQaI8hvRTawha6SWC1WcZa2RYry440DmwV02t/X2/9Af0flddbaBMPCD3em8eMZpRUSlqMol9BeOfuvft7+w8GDx89fvJ0+Oz5qS1rw8WMl6o05wlYoaQWM5SoxHllBBSJEmfJ+n2rn10JY2Wpv+KmEosCMi1TyQE9tRz+ZicyY9eUaUgUOMZB0U/NsmCJzA5YUqqV3RT+cgxzgdBcrFvlcNyfnwO6HpPXL9x6HDdbb1eRZeKSstQAd3HjPjdMiRQ9/ZdhoKocOrq/kXEf25UxMsvxsLvY9WCwHI6iSbQNehvEOzAiu5guhz/ZquR1ITRyBdbO46jChQODkivRDFhtRQV8DZmYe6ihEHbhtuto6GvPrGhaGn800i37r8NBYdu2fWYBmNubWkv2afMa03cLJ3VVo9C8K5TWimJJ293SlTSCo9p4ANxI3yvlOfiBov8B7RDim0++DU6PJnE0ib+8GR0f7caxT16SV+SAxOQtOSYfyJTMCA8+BpfBt8CF0/AqbMLvXWoY7DwvyH8R/vgDSIXldA==</latexit><latexit sha1_base64="sdBQ5wuLWPdC8PW6kv0JAW7GfI8=">AAACznicbVJNa9wwEJXdj6Tbr2177EV0KSSHXexQaI8hvRTawha6SWC1WcZa2RYry440DmwV02t/X2/9Af0flddbaBMPCD3em8eMZpRUSlqMol9BeOfuvft7+w8GDx89fvJ0+Oz5qS1rw8WMl6o05wlYoaQWM5SoxHllBBSJEmfJ+n2rn10JY2Wpv+KmEosCMi1TyQE9tRz+ZicyY9eUaUgUOMZB0U/NsmCJzA5YUqqV3RT+cgxzgdBcrFvlcNyfnwO6HpPXL9x6HDdbb1eRZeKSstQAd3HjPjdMiRQ9/ZdhoKocOrq/kXEf25UxMsvxsLvY9WCwHI6iSbQNehvEOzAiu5guhz/ZquR1ITRyBdbO46jChQODkivRDFhtRQV8DZmYe6ihEHbhtuto6GvPrGhaGn800i37r8NBYdu2fWYBmNubWkv2afMa03cLJ3VVo9C8K5TWimJJ293SlTSCo9p4ANxI3yvlOfiBov8B7RDim0++DU6PJnE0ib+8GR0f7caxT16SV+SAxOQtOSYfyJTMCA8+BpfBt8CF0/AqbMLvXWoY7DwvyH8R/vgDSIXldA==</latexit><latexit sha1_base64="sdBQ5wuLWPdC8PW6kv0JAW7GfI8=">AAACznicbVJNa9wwEJXdj6Tbr2177EV0KSSHXexQaI8hvRTawha6SWC1WcZa2RYry440DmwV02t/X2/9Af0flddbaBMPCD3em8eMZpRUSlqMol9BeOfuvft7+w8GDx89fvJ0+Oz5qS1rw8WMl6o05wlYoaQWM5SoxHllBBSJEmfJ+n2rn10JY2Wpv+KmEosCMi1TyQE9tRz+ZicyY9eUaUgUOMZB0U/NsmCJzA5YUqqV3RT+cgxzgdBcrFvlcNyfnwO6HpPXL9x6HDdbb1eRZeKSstQAd3HjPjdMiRQ9/ZdhoKocOrq/kXEf25UxMsvxsLvY9WCwHI6iSbQNehvEOzAiu5guhz/ZquR1ITRyBdbO46jChQODkivRDFhtRQV8DZmYe6ihEHbhtuto6GvPrGhaGn800i37r8NBYdu2fWYBmNubWkv2afMa03cLJ3VVo9C8K5TWimJJ293SlTSCo9p4ANxI3yvlOfiBov8B7RDim0++DU6PJnE0ib+8GR0f7caxT16SV+SAxOQtOSYfyJTMCA8+BpfBt8CF0/AqbMLvXWoY7DwvyH8R/vgDSIXldA==</latexit><latexit sha1_base64="sdBQ5wuLWPdC8PW6kv0JAW7GfI8=">AAACznicbVJNa9wwEJXdj6Tbr2177EV0KSSHXexQaI8hvRTawha6SWC1WcZa2RYry440DmwV02t/X2/9Af0flddbaBMPCD3em8eMZpRUSlqMol9BeOfuvft7+w8GDx89fvJ0+Oz5qS1rw8WMl6o05wlYoaQWM5SoxHllBBSJEmfJ+n2rn10JY2Wpv+KmEosCMi1TyQE9tRz+ZicyY9eUaUgUOMZB0U/NsmCJzA5YUqqV3RT+cgxzgdBcrFvlcNyfnwO6HpPXL9x6HDdbb1eRZeKSstQAd3HjPjdMiRQ9/ZdhoKocOrq/kXEf25UxMsvxsLvY9WCwHI6iSbQNehvEOzAiu5guhz/ZquR1ITRyBdbO46jChQODkivRDFhtRQV8DZmYe6ihEHbhtuto6GvPrGhaGn800i37r8NBYdu2fWYBmNubWkv2afMa03cLJ3VVo9C8K5TWimJJ293SlTSCo9p4ANxI3yvlOfiBov8B7RDim0++DU6PJnE0ib+8GR0f7caxT16SV+SAxOQtOSYfyJTMCA8+BpfBt8CF0/AqbMLvXWoY7DwvyH8R/vgDSIXldA==</latexit><latexit sha1_base64="sdBQ5wuLWPdC8PW6kv0JAW7GfI8=">AAACznicbVJNa9wwEJXdj6Tbr2177EV0KSSHXexQaI8hvRTawha6SWC1WcZa2RYry440DmwV02t/X2/9Af0flddbaBMPCD3em8eMZpRUSlqMol9BeOfuvft7+w8GDx89fvJ0+Oz5qS1rw8WMl6o05wlYoaQWM5SoxHllBBSJEmfJ+n2rn10JY2Wpv+KmEosCMi1TyQE9tRz+ZicyY9eUaUgUOMZB0U/NsmCJzA5YUqqV3RT+cgxzgdBcrFvlcNyfnwO6HpPXL9x6HDdbb1eRZeKSstQAd3HjPjdMiRQ9/ZdhoKocOrq/kXEf25UxMsvxsLvY9WCwHI6iSbQNehvEOzAiu5guhz/ZquR1ITRyBdbO46jChQODkivRDFhtRQV8DZmYe6ihEHbhtuto6GvPrGhaGn800i37r8NBYdu2fWYBmNubWkv2afMa03cLJ3VVo9C8K5TWimJJ293SlTSCo9p4ANxI3yvlOfiBov8B7RDim0++DU6PJnE0ib+8GR0f7caxT16SV+SAxOQtOSYfyJTMCA8+BpfBt8CF0/AqbMLvXWoY7DwvyH8R/vgDSIXldA==</latexit><latexit sha1_base64="sdBQ5wuLWPdC8PW6kv0JAW7GfI8=">AAACznicbVJNa9wwEJXdj6Tbr2177EV0KSSHXexQaI8hvRTawha6SWC1WcZa2RYry440DmwV02t/X2/9Af0flddbaBMPCD3em8eMZpRUSlqMol9BeOfuvft7+w8GDx89fvJ0+Oz5qS1rw8WMl6o05wlYoaQWM5SoxHllBBSJEmfJ+n2rn10JY2Wpv+KmEosCMi1TyQE9tRz+ZicyY9eUaUgUOMZB0U/NsmCJzA5YUqqV3RT+cgxzgdBcrFvlcNyfnwO6HpPXL9x6HDdbb1eRZeKSstQAd3HjPjdMiRQ9/ZdhoKocOrq/kXEf25UxMsvxsLvY9WCwHI6iSbQNehvEOzAiu5guhz/ZquR1ITRyBdbO46jChQODkivRDFhtRQV8DZmYe6ihEHbhtuto6GvPrGhaGn800i37r8NBYdu2fWYBmNubWkv2afMa03cLJ3VVo9C8K5TWimJJ293SlTSCo9p4ANxI3yvlOfiBov8B7RDim0++DU6PJnE0ib+8GR0f7caxT16SV+SAxOQtOSYfyJTMCA8+BpfBt8CF0/AqbMLvXWoY7DwvyH8R/vgDSIXldA==</latexit>

Old gradient

Page 8: LAG: Lazily Aggregated Gradient for Communication ...06-15-30)-06-15-40-12765-LAG:... · 1 TianyiChen Georgios Giannakis Tao Sun WotaoYin UMN, ECE UCLA, Math NeurIPS2018 LAG: Lazily

8

LAG: GD under two alternative communication rules

T. Chen, G. B. Giannakis, T. Sun, and W. Yin "LAG: Lazily Aggregated Gradient for Communication-Efficient Distributed Learning," Proc. of NIPS, Montreal, Canada, December 3-8, 2018.

q Worker-side rule (LAG-WK):

q Server-side rule (LAG-PS):

: smoothness of

Include worker m in if

§ LAG-PS is a sufficient condition for LAG-WK.

Lm<latexit sha1_base64="IsVjkbiGRCOxdOt70vZudkTBfj4=">AAAB8XicbVA9SwNBEJ2LXzF+RS1tFoNgFe5E0DJgY2ERwXxgcoS9zSRZsrt37O4J4ci/sLFQxNZ/Y+e/cZNcoYkPBh7vzTAzL0oEN9b3v73C2vrG5lZxu7Szu7d/UD48apo41QwbLBaxbkfUoOAKG5Zbge1EI5WRwFY0vpn5rSfUhsfqwU4SDCUdKj7gjFonPWZdRgW5m/Zkr1zxq/4cZJUEOalAjnqv/NXtxyyVqCwT1JhO4Cc2zKi2nAmclrqpwYSyMR1ix1FFJZowm188JWdO6ZNBrF0pS+bq74mMSmMmMnKdktqRWfZm4n9eJ7WD6zDjKkktKrZYNEgFsTGZvU/6XCOzYuIIZZq7WwkbUU2ZdSGVXAjB8surpHlRDfxqcH9ZqV3mcRThBE7hHAK4ghrcQh0awEDBM7zCm2e8F+/d+1i0Frx85hj+wPv8ATLPkIw=</latexit><latexit sha1_base64="IsVjkbiGRCOxdOt70vZudkTBfj4=">AAAB8XicbVA9SwNBEJ2LXzF+RS1tFoNgFe5E0DJgY2ERwXxgcoS9zSRZsrt37O4J4ci/sLFQxNZ/Y+e/cZNcoYkPBh7vzTAzL0oEN9b3v73C2vrG5lZxu7Szu7d/UD48apo41QwbLBaxbkfUoOAKG5Zbge1EI5WRwFY0vpn5rSfUhsfqwU4SDCUdKj7gjFonPWZdRgW5m/Zkr1zxq/4cZJUEOalAjnqv/NXtxyyVqCwT1JhO4Cc2zKi2nAmclrqpwYSyMR1ix1FFJZowm188JWdO6ZNBrF0pS+bq74mMSmMmMnKdktqRWfZm4n9eJ7WD6zDjKkktKrZYNEgFsTGZvU/6XCOzYuIIZZq7WwkbUU2ZdSGVXAjB8surpHlRDfxqcH9ZqV3mcRThBE7hHAK4ghrcQh0awEDBM7zCm2e8F+/d+1i0Frx85hj+wPv8ATLPkIw=</latexit><latexit sha1_base64="IsVjkbiGRCOxdOt70vZudkTBfj4=">AAAB8XicbVA9SwNBEJ2LXzF+RS1tFoNgFe5E0DJgY2ERwXxgcoS9zSRZsrt37O4J4ci/sLFQxNZ/Y+e/cZNcoYkPBh7vzTAzL0oEN9b3v73C2vrG5lZxu7Szu7d/UD48apo41QwbLBaxbkfUoOAKG5Zbge1EI5WRwFY0vpn5rSfUhsfqwU4SDCUdKj7gjFonPWZdRgW5m/Zkr1zxq/4cZJUEOalAjnqv/NXtxyyVqCwT1JhO4Cc2zKi2nAmclrqpwYSyMR1ix1FFJZowm188JWdO6ZNBrF0pS+bq74mMSmMmMnKdktqRWfZm4n9eJ7WD6zDjKkktKrZYNEgFsTGZvU/6XCOzYuIIZZq7WwkbUU2ZdSGVXAjB8surpHlRDfxqcH9ZqV3mcRThBE7hHAK4ghrcQh0awEDBM7zCm2e8F+/d+1i0Frx85hj+wPv8ATLPkIw=</latexit><latexit sha1_base64="IsVjkbiGRCOxdOt70vZudkTBfj4=">AAAB8XicbVA9SwNBEJ2LXzF+RS1tFoNgFe5E0DJgY2ERwXxgcoS9zSRZsrt37O4J4ci/sLFQxNZ/Y+e/cZNcoYkPBh7vzTAzL0oEN9b3v73C2vrG5lZxu7Szu7d/UD48apo41QwbLBaxbkfUoOAKG5Zbge1EI5WRwFY0vpn5rSfUhsfqwU4SDCUdKj7gjFonPWZdRgW5m/Zkr1zxq/4cZJUEOalAjnqv/NXtxyyVqCwT1JhO4Cc2zKi2nAmclrqpwYSyMR1ix1FFJZowm188JWdO6ZNBrF0pS+bq74mMSmMmMnKdktqRWfZm4n9eJ7WD6zDjKkktKrZYNEgFsTGZvU/6XCOzYuIIZZq7WwkbUU2ZdSGVXAjB8surpHlRDfxqcH9ZqV3mcRThBE7hHAK4ghrcQh0awEDBM7zCm2e8F+/d+1i0Frx85hj+wPv8ATLPkIw=</latexit>

Include worker m in if

Gradient innovation Optimization progress

���rLm

�✓k��rLm

�✓̂k�1

m

���� � 1

M

����1

⇣✓k � ✓k�1

⌘����<latexit sha1_base64="sdBQ5wuLWPdC8PW6kv0JAW7GfI8=">AAACznicbVJNa9wwEJXdj6Tbr2177EV0KSSHXexQaI8hvRTawha6SWC1WcZa2RYry440DmwV02t/X2/9Af0flddbaBMPCD3em8eMZpRUSlqMol9BeOfuvft7+w8GDx89fvJ0+Oz5qS1rw8WMl6o05wlYoaQWM5SoxHllBBSJEmfJ+n2rn10JY2Wpv+KmEosCMi1TyQE9tRz+ZicyY9eUaUgUOMZB0U/NsmCJzA5YUqqV3RT+cgxzgdBcrFvlcNyfnwO6HpPXL9x6HDdbb1eRZeKSstQAd3HjPjdMiRQ9/ZdhoKocOrq/kXEf25UxMsvxsLvY9WCwHI6iSbQNehvEOzAiu5guhz/ZquR1ITRyBdbO46jChQODkivRDFhtRQV8DZmYe6ihEHbhtuto6GvPrGhaGn800i37r8NBYdu2fWYBmNubWkv2afMa03cLJ3VVo9C8K5TWimJJ293SlTSCo9p4ANxI3yvlOfiBov8B7RDim0++DU6PJnE0ib+8GR0f7caxT16SV+SAxOQtOSYfyJTMCA8+BpfBt8CF0/AqbMLvXWoY7DwvyH8R/vgDSIXldA==</latexit><latexit sha1_base64="sdBQ5wuLWPdC8PW6kv0JAW7GfI8=">AAACznicbVJNa9wwEJXdj6Tbr2177EV0KSSHXexQaI8hvRTawha6SWC1WcZa2RYry440DmwV02t/X2/9Af0flddbaBMPCD3em8eMZpRUSlqMol9BeOfuvft7+w8GDx89fvJ0+Oz5qS1rw8WMl6o05wlYoaQWM5SoxHllBBSJEmfJ+n2rn10JY2Wpv+KmEosCMi1TyQE9tRz+ZicyY9eUaUgUOMZB0U/NsmCJzA5YUqqV3RT+cgxzgdBcrFvlcNyfnwO6HpPXL9x6HDdbb1eRZeKSstQAd3HjPjdMiRQ9/ZdhoKocOrq/kXEf25UxMsvxsLvY9WCwHI6iSbQNehvEOzAiu5guhz/ZquR1ITRyBdbO46jChQODkivRDFhtRQV8DZmYe6ihEHbhtuto6GvPrGhaGn800i37r8NBYdu2fWYBmNubWkv2afMa03cLJ3VVo9C8K5TWimJJ293SlTSCo9p4ANxI3yvlOfiBov8B7RDim0++DU6PJnE0ib+8GR0f7caxT16SV+SAxOQtOSYfyJTMCA8+BpfBt8CF0/AqbMLvXWoY7DwvyH8R/vgDSIXldA==</latexit><latexit sha1_base64="sdBQ5wuLWPdC8PW6kv0JAW7GfI8=">AAACznicbVJNa9wwEJXdj6Tbr2177EV0KSSHXexQaI8hvRTawha6SWC1WcZa2RYry440DmwV02t/X2/9Af0flddbaBMPCD3em8eMZpRUSlqMol9BeOfuvft7+w8GDx89fvJ0+Oz5qS1rw8WMl6o05wlYoaQWM5SoxHllBBSJEmfJ+n2rn10JY2Wpv+KmEosCMi1TyQE9tRz+ZicyY9eUaUgUOMZB0U/NsmCJzA5YUqqV3RT+cgxzgdBcrFvlcNyfnwO6HpPXL9x6HDdbb1eRZeKSstQAd3HjPjdMiRQ9/ZdhoKocOrq/kXEf25UxMsvxsLvY9WCwHI6iSbQNehvEOzAiu5guhz/ZquR1ITRyBdbO46jChQODkivRDFhtRQV8DZmYe6ihEHbhtuto6GvPrGhaGn800i37r8NBYdu2fWYBmNubWkv2afMa03cLJ3VVo9C8K5TWimJJ293SlTSCo9p4ANxI3yvlOfiBov8B7RDim0++DU6PJnE0ib+8GR0f7caxT16SV+SAxOQtOSYfyJTMCA8+BpfBt8CF0/AqbMLvXWoY7DwvyH8R/vgDSIXldA==</latexit><latexit sha1_base64="hP+6LrUf2d3tZaldqaQQvEKMXyw=">AAAB2XicbZDNSgMxFIXv1L86Vq1rN8EiuCozbnQpuHFZwbZCO5RM5k4bmskMyR2hDH0BF25EfC93vo3pz0JbDwQ+zknIvSculLQUBN9ebWd3b/+gfugfNfzjk9Nmo2fz0gjsilzl5jnmFpXU2CVJCp8LgzyLFfbj6f0i77+gsTLXTzQrMMr4WMtUCk7O6oyaraAdLMW2IVxDC9YaNb+GSS7KDDUJxa0dhEFBUcUNSaFw7g9LiwUXUz7GgUPNM7RRtRxzzi6dk7A0N+5oYkv394uKZ9bOstjdzDhN7Ga2MP/LBiWlt1EldVESarH6KC0Vo5wtdmaJNChIzRxwYaSblYkJN1yQa8Z3HYSbG29D77odBu3wMYA6nMMFXEEIN3AHD9CBLghI4BXevYn35n2suqp569LO4I+8zx84xIo4</latexit><latexit sha1_base64="0R4hPLkwHklCrgKU4UNeGnfaJvs=">AAACw3icbVJNb9NAEF2brxIKBK5cVlRI7SGRzQWOCC5IgBQk0lbKptF4M7ZXWX90d1wpbC3+Izd+AP+DdRwkaD3Sap/em9HMvtmk1spSFP0Kwjt3791/cPBw9Ojw8ZOn42eHp7ZqjMS5rHRlzhOwqFWJc1Kk8bw2CEWi8SzZfOj0sys0VlXlN9rWuCwgK1WqJJCnVuPf4r3KxDUXJSQanJCg+ed2VYhEZcciqfTabgt/OUE5ErQXm045mQzn50BuoMjrF24zidtdbd9RZHjJRWpAurh1X1qhMSVP/2UE6DqHnh4eZDLE9m2MynI66S9xPRqtxkfRNNoFvw3iPThi+5itxj/FupJNgSVJDdYu4qimpQNDSmpsR6KxWIPcQIYLD0so0C7dbh0tf+WZNU8r409JfMf+W+GgsN3YPrMAyu1NrSOHtEVD6dulU2XdEJayb5Q2mlPFu93ytTIoSW89AGmUn5XLHLyh5H9AZ0J888m3wenraRxN468RO2Av2Et2zGL2hr1jH9mMzZkMPgWXwffAhbPwKmx7u8Jg79tz9l+EP/4AYCLkSg==</latexit><latexit sha1_base64="0R4hPLkwHklCrgKU4UNeGnfaJvs=">AAACw3icbVJNb9NAEF2brxIKBK5cVlRI7SGRzQWOCC5IgBQk0lbKptF4M7ZXWX90d1wpbC3+Izd+AP+DdRwkaD3Sap/em9HMvtmk1spSFP0Kwjt3791/cPBw9Ojw8ZOn42eHp7ZqjMS5rHRlzhOwqFWJc1Kk8bw2CEWi8SzZfOj0sys0VlXlN9rWuCwgK1WqJJCnVuPf4r3KxDUXJSQanJCg+ed2VYhEZcciqfTabgt/OUE5ErQXm045mQzn50BuoMjrF24zidtdbd9RZHjJRWpAurh1X1qhMSVP/2UE6DqHnh4eZDLE9m2MynI66S9xPRqtxkfRNNoFvw3iPThi+5itxj/FupJNgSVJDdYu4qimpQNDSmpsR6KxWIPcQIYLD0so0C7dbh0tf+WZNU8r409JfMf+W+GgsN3YPrMAyu1NrSOHtEVD6dulU2XdEJayb5Q2mlPFu93ytTIoSW89AGmUn5XLHLyh5H9AZ0J888m3wenraRxN468RO2Av2Et2zGL2hr1jH9mMzZkMPgWXwffAhbPwKmx7u8Jg79tz9l+EP/4AYCLkSg==</latexit><latexit sha1_base64="CkJErppo0e1xvgGRANXgvbPzUtg=">AAACznicbVJNa9wwEJXdr3T7tWmPvYguheSwi51LegzNpdAWttBNAqvNMtbKtlhZdqRxYKuYXvv7eusP6P+ovN5Cm3hA6PHePGY0o6RS0mIU/QrCe/cfPHy093jw5Omz5y+G+y/PbFkbLma8VKW5SMAKJbWYoUQlLiojoEiUOE/Wp61+fi2MlaX+iptKLArItEwlB/TUcvibvZcZu6FMQ6LAMQ6KfmqWBUtkdsCSUq3spvCXY5gLhOZy3SqH4/78HND1mLx+6dbjuNl6u4osE1eUpQa4ixv3uWFKpOjpvwwDVeXQ0f2NjPvYroyRWY6H3cVuBoPlcBRNom3QuyDegRHZxXQ5/MlWJa8LoZErsHYeRxUuHBiUXIlmwGorKuBryMTcQw2FsAu3XUdD33pmRdPS+KORbtl/HQ4K27btMwvA3N7WWrJPm9eYvls4qasaheZdobRWFEva7paupBEc1cYD4Eb6XinPwQ8U/Q9ohxDffvJdcHY0iaNJ/CUanRztxrFHXpM35IDE5JickA9kSmaEBx+Dq+Bb4MJpeB024fcuNQx2nlfkvwh//AFHReVw</latexit><latexit sha1_base64="sdBQ5wuLWPdC8PW6kv0JAW7GfI8=">AAACznicbVJNa9wwEJXdj6Tbr2177EV0KSSHXexQaI8hvRTawha6SWC1WcZa2RYry440DmwV02t/X2/9Af0flddbaBMPCD3em8eMZpRUSlqMol9BeOfuvft7+w8GDx89fvJ0+Oz5qS1rw8WMl6o05wlYoaQWM5SoxHllBBSJEmfJ+n2rn10JY2Wpv+KmEosCMi1TyQE9tRz+ZicyY9eUaUgUOMZB0U/NsmCJzA5YUqqV3RT+cgxzgdBcrFvlcNyfnwO6HpPXL9x6HDdbb1eRZeKSstQAd3HjPjdMiRQ9/ZdhoKocOrq/kXEf25UxMsvxsLvY9WCwHI6iSbQNehvEOzAiu5guhz/ZquR1ITRyBdbO46jChQODkivRDFhtRQV8DZmYe6ihEHbhtuto6GvPrGhaGn800i37r8NBYdu2fWYBmNubWkv2afMa03cLJ3VVo9C8K5TWimJJ293SlTSCo9p4ANxI3yvlOfiBov8B7RDim0++DU6PJnE0ib+8GR0f7caxT16SV+SAxOQtOSYfyJTMCA8+BpfBt8CF0/AqbMLvXWoY7DwvyH8R/vgDSIXldA==</latexit><latexit sha1_base64="sdBQ5wuLWPdC8PW6kv0JAW7GfI8=">AAACznicbVJNa9wwEJXdj6Tbr2177EV0KSSHXexQaI8hvRTawha6SWC1WcZa2RYry440DmwV02t/X2/9Af0flddbaBMPCD3em8eMZpRUSlqMol9BeOfuvft7+w8GDx89fvJ0+Oz5qS1rw8WMl6o05wlYoaQWM5SoxHllBBSJEmfJ+n2rn10JY2Wpv+KmEosCMi1TyQE9tRz+ZicyY9eUaUgUOMZB0U/NsmCJzA5YUqqV3RT+cgxzgdBcrFvlcNyfnwO6HpPXL9x6HDdbb1eRZeKSstQAd3HjPjdMiRQ9/ZdhoKocOrq/kXEf25UxMsvxsLvY9WCwHI6iSbQNehvEOzAiu5guhz/ZquR1ITRyBdbO46jChQODkivRDFhtRQV8DZmYe6ihEHbhtuto6GvPrGhaGn800i37r8NBYdu2fWYBmNubWkv2afMa03cLJ3VVo9C8K5TWimJJ293SlTSCo9p4ANxI3yvlOfiBov8B7RDim0++DU6PJnE0ib+8GR0f7caxT16SV+SAxOQtOSYfyJTMCA8+BpfBt8CF0/AqbMLvXWoY7DwvyH8R/vgDSIXldA==</latexit><latexit sha1_base64="sdBQ5wuLWPdC8PW6kv0JAW7GfI8=">AAACznicbVJNa9wwEJXdj6Tbr2177EV0KSSHXexQaI8hvRTawha6SWC1WcZa2RYry440DmwV02t/X2/9Af0flddbaBMPCD3em8eMZpRUSlqMol9BeOfuvft7+w8GDx89fvJ0+Oz5qS1rw8WMl6o05wlYoaQWM5SoxHllBBSJEmfJ+n2rn10JY2Wpv+KmEosCMi1TyQE9tRz+ZicyY9eUaUgUOMZB0U/NsmCJzA5YUqqV3RT+cgxzgdBcrFvlcNyfnwO6HpPXL9x6HDdbb1eRZeKSstQAd3HjPjdMiRQ9/ZdhoKocOrq/kXEf25UxMsvxsLvY9WCwHI6iSbQNehvEOzAiu5guhz/ZquR1ITRyBdbO46jChQODkivRDFhtRQV8DZmYe6ihEHbhtuto6GvPrGhaGn800i37r8NBYdu2fWYBmNubWkv2afMa03cLJ3VVo9C8K5TWimJJ293SlTSCo9p4ANxI3yvlOfiBov8B7RDim0++DU6PJnE0ib+8GR0f7caxT16SV+SAxOQtOSYfyJTMCA8+BpfBt8CF0/AqbMLvXWoY7DwvyH8R/vgDSIXldA==</latexit><latexit sha1_base64="sdBQ5wuLWPdC8PW6kv0JAW7GfI8=">AAACznicbVJNa9wwEJXdj6Tbr2177EV0KSSHXexQaI8hvRTawha6SWC1WcZa2RYry440DmwV02t/X2/9Af0flddbaBMPCD3em8eMZpRUSlqMol9BeOfuvft7+w8GDx89fvJ0+Oz5qS1rw8WMl6o05wlYoaQWM5SoxHllBBSJEmfJ+n2rn10JY2Wpv+KmEosCMi1TyQE9tRz+ZicyY9eUaUgUOMZB0U/NsmCJzA5YUqqV3RT+cgxzgdBcrFvlcNyfnwO6HpPXL9x6HDdbb1eRZeKSstQAd3HjPjdMiRQ9/ZdhoKocOrq/kXEf25UxMsvxsLvY9WCwHI6iSbQNehvEOzAiu5guhz/ZquR1ITRyBdbO46jChQODkivRDFhtRQV8DZmYe6ihEHbhtuto6GvPrGhaGn800i37r8NBYdu2fWYBmNubWkv2afMa03cLJ3VVo9C8K5TWimJJ293SlTSCo9p4ANxI3yvlOfiBov8B7RDim0++DU6PJnE0ib+8GR0f7caxT16SV+SAxOQtOSYfyJTMCA8+BpfBt8CF0/AqbMLvXWoY7DwvyH8R/vgDSIXldA==</latexit><latexit sha1_base64="sdBQ5wuLWPdC8PW6kv0JAW7GfI8=">AAACznicbVJNa9wwEJXdj6Tbr2177EV0KSSHXexQaI8hvRTawha6SWC1WcZa2RYry440DmwV02t/X2/9Af0flddbaBMPCD3em8eMZpRUSlqMol9BeOfuvft7+w8GDx89fvJ0+Oz5qS1rw8WMl6o05wlYoaQWM5SoxHllBBSJEmfJ+n2rn10JY2Wpv+KmEosCMi1TyQE9tRz+ZicyY9eUaUgUOMZB0U/NsmCJzA5YUqqV3RT+cgxzgdBcrFvlcNyfnwO6HpPXL9x6HDdbb1eRZeKSstQAd3HjPjdMiRQ9/ZdhoKocOrq/kXEf25UxMsvxsLvY9WCwHI6iSbQNehvEOzAiu5guhz/ZquR1ITRyBdbO46jChQODkivRDFhtRQV8DZmYe6ihEHbhtuto6GvPrGhaGn800i37r8NBYdu2fWYBmNubWkv2afMa03cLJ3VVo9C8K5TWimJJ293SlTSCo9p4ANxI3yvlOfiBov8B7RDim0++DU6PJnE0ib+8GR0f7caxT16SV+SAxOQtOSYfyJTMCA8+BpfBt8CF0/AqbMLvXWoY7DwvyH8R/vgDSIXldA==</latexit><latexit sha1_base64="sdBQ5wuLWPdC8PW6kv0JAW7GfI8=">AAACznicbVJNa9wwEJXdj6Tbr2177EV0KSSHXexQaI8hvRTawha6SWC1WcZa2RYry440DmwV02t/X2/9Af0flddbaBMPCD3em8eMZpRUSlqMol9BeOfuvft7+w8GDx89fvJ0+Oz5qS1rw8WMl6o05wlYoaQWM5SoxHllBBSJEmfJ+n2rn10JY2Wpv+KmEosCMi1TyQE9tRz+ZicyY9eUaUgUOMZB0U/NsmCJzA5YUqqV3RT+cgxzgdBcrFvlcNyfnwO6HpPXL9x6HDdbb1eRZeKSstQAd3HjPjdMiRQ9/ZdhoKocOrq/kXEf25UxMsvxsLvY9WCwHI6iSbQNehvEOzAiu5guhz/ZquR1ITRyBdbO46jChQODkivRDFhtRQV8DZmYe6ihEHbhtuto6GvPrGhaGn800i37r8NBYdu2fWYBmNubWkv2afMa03cLJ3VVo9C8K5TWimJJ293SlTSCo9p4ANxI3yvlOfiBov8B7RDim0++DU6PJnE0ib+8GR0f7caxT16SV+SAxOQtOSYfyJTMCA8+BpfBt8CF0/AqbMLvXWoY7DwvyH8R/vgDSIXldA==</latexit>

Lm

���✓k � ✓̂k�1

m

��� � 1

M

����1

⇣✓k � ✓k�1

⌘����<latexit sha1_base64="YZvfKo7wYha/YERVZH/rng1YfkA=">AAACnXicdVFdi9NAFJ3Er7V+bNU3fXCwCOvDlmQR9LEogsoqK9jtQqcbJtObZOhMEmduhDKbf+Uv8c1/46SpoLv1wnAP59zLvXNuWitpMYp+BeG16zdu3tq7Pbhz9979/eGDh6e2aoyAqahUZc5SbkHJEqYoUcFZbYDrVMEsXb3t9Nl3MFZW5Vdc17DQPC9lJgVHTyXDH8eJZm9kzi4oSyu1tGvtk2NYAPL2fHXICo5uh9Qm+tytDuO272Y5fKMsM1y4uHWfWqYgQ0//YRhXdcF7+uA/k3aw/QQj8wJf9IldDAbJcBSNo03QqyDeghHZxkky/MmWlWg0lCgUt3YeRzUuHDcohYJ2wBoLNRcrnsPcw5JrsAu3cbelzz2zpFll/CuRbti/OxzXtlvbV2qOhb2sdeQubd5g9nrhZFk3CKXoB2WNoljR7lR0KQ0IVGsPuDDS70pFwb2h6A/amRBf/vJVcHo0jqNx/OXlaHK0tWOPPCHPyAGJySsyIe/JCZkSETwOJsGH4GP4NHwXHoef+9Iw2PY8Iv9EOPsNQVHRnA==</latexit><latexit sha1_base64="YZvfKo7wYha/YERVZH/rng1YfkA=">AAACnXicdVFdi9NAFJ3Er7V+bNU3fXCwCOvDlmQR9LEogsoqK9jtQqcbJtObZOhMEmduhDKbf+Uv8c1/46SpoLv1wnAP59zLvXNuWitpMYp+BeG16zdu3tq7Pbhz9979/eGDh6e2aoyAqahUZc5SbkHJEqYoUcFZbYDrVMEsXb3t9Nl3MFZW5Vdc17DQPC9lJgVHTyXDH8eJZm9kzi4oSyu1tGvtk2NYAPL2fHXICo5uh9Qm+tytDuO272Y5fKMsM1y4uHWfWqYgQ0//YRhXdcF7+uA/k3aw/QQj8wJf9IldDAbJcBSNo03QqyDeghHZxkky/MmWlWg0lCgUt3YeRzUuHDcohYJ2wBoLNRcrnsPcw5JrsAu3cbelzz2zpFll/CuRbti/OxzXtlvbV2qOhb2sdeQubd5g9nrhZFk3CKXoB2WNoljR7lR0KQ0IVGsPuDDS70pFwb2h6A/amRBf/vJVcHo0jqNx/OXlaHK0tWOPPCHPyAGJySsyIe/JCZkSETwOJsGH4GP4NHwXHoef+9Iw2PY8Iv9EOPsNQVHRnA==</latexit><latexit sha1_base64="YZvfKo7wYha/YERVZH/rng1YfkA=">AAACnXicdVFdi9NAFJ3Er7V+bNU3fXCwCOvDlmQR9LEogsoqK9jtQqcbJtObZOhMEmduhDKbf+Uv8c1/46SpoLv1wnAP59zLvXNuWitpMYp+BeG16zdu3tq7Pbhz9979/eGDh6e2aoyAqahUZc5SbkHJEqYoUcFZbYDrVMEsXb3t9Nl3MFZW5Vdc17DQPC9lJgVHTyXDH8eJZm9kzi4oSyu1tGvtk2NYAPL2fHXICo5uh9Qm+tytDuO272Y5fKMsM1y4uHWfWqYgQ0//YRhXdcF7+uA/k3aw/QQj8wJf9IldDAbJcBSNo03QqyDeghHZxkky/MmWlWg0lCgUt3YeRzUuHDcohYJ2wBoLNRcrnsPcw5JrsAu3cbelzz2zpFll/CuRbti/OxzXtlvbV2qOhb2sdeQubd5g9nrhZFk3CKXoB2WNoljR7lR0KQ0IVGsPuDDS70pFwb2h6A/amRBf/vJVcHo0jqNx/OXlaHK0tWOPPCHPyAGJySsyIe/JCZkSETwOJsGH4GP4NHwXHoef+9Iw2PY8Iv9EOPsNQVHRnA==</latexit><latexit sha1_base64="hP+6LrUf2d3tZaldqaQQvEKMXyw=">AAAB2XicbZDNSgMxFIXv1L86Vq1rN8EiuCozbnQpuHFZwbZCO5RM5k4bmskMyR2hDH0BF25EfC93vo3pz0JbDwQ+zknIvSculLQUBN9ebWd3b/+gfugfNfzjk9Nmo2fz0gjsilzl5jnmFpXU2CVJCp8LgzyLFfbj6f0i77+gsTLXTzQrMMr4WMtUCk7O6oyaraAdLMW2IVxDC9YaNb+GSS7KDDUJxa0dhEFBUcUNSaFw7g9LiwUXUz7GgUPNM7RRtRxzzi6dk7A0N+5oYkv394uKZ9bOstjdzDhN7Ga2MP/LBiWlt1EldVESarH6KC0Vo5wtdmaJNChIzRxwYaSblYkJN1yQa8Z3HYSbG29D77odBu3wMYA6nMMFXEEIN3AHD9CBLghI4BXevYn35n2suqp569LO4I+8zx84xIo4</latexit><latexit sha1_base64="EXy2rVIBegRSOrMj8+MSFv1Xw2g=">AAACknicdZHPbtQwEMad8K8sBRaOcMCiQiqHrpJe4FiBkAAVVCS2rbTeRhPvJLHWToI9QVqleSuehBtvg7NZJGiXkSx/+o1HM/4mrbVyFEW/gvDGzVu37+zcHd3bvf/g4fjR7qmrGitxKitd2fMUHGpV4pQUaTyvLYJJNZ6ly7d9/uw7Wqeq8iutapwbyEuVKQnkUTL+cZwY8Ubl4pKLtNILtzL+agUVSNBdLA9EAdRuSXWJuWiXB3E3VIscv3GRWZBt3LWfOqExI4//EAG6LmDA+//ptIUOHazKC3o5XOJyNErGe9EkWge/LuKN2GObOEnGP8Wiko3BkqQG52ZxVNO8BUtKauxGonFYg1xCjjMvSzDo5u3a3Y6/8GTBs8r6UxJf078rWjCuH9u/NECFu5rr4bbcrKHs9bxVZd0QlnJolDWaU8X7VfGFsihJr7wAaZWflcsCvKHkF9qbEF/98nVxejiJo0n8JWI77Cl7zvZZzF6xI/aenbApk8GT4Cj4EHwMn4XvwuPBrjDY+PaY/RPh598qd9Cj</latexit><latexit sha1_base64="EXy2rVIBegRSOrMj8+MSFv1Xw2g=">AAACknicdZHPbtQwEMad8K8sBRaOcMCiQiqHrpJe4FiBkAAVVCS2rbTeRhPvJLHWToI9QVqleSuehBtvg7NZJGiXkSx/+o1HM/4mrbVyFEW/gvDGzVu37+zcHd3bvf/g4fjR7qmrGitxKitd2fMUHGpV4pQUaTyvLYJJNZ6ly7d9/uw7Wqeq8iutapwbyEuVKQnkUTL+cZwY8Ubl4pKLtNILtzL+agUVSNBdLA9EAdRuSXWJuWiXB3E3VIscv3GRWZBt3LWfOqExI4//EAG6LmDA+//ptIUOHazKC3o5XOJyNErGe9EkWge/LuKN2GObOEnGP8Wiko3BkqQG52ZxVNO8BUtKauxGonFYg1xCjjMvSzDo5u3a3Y6/8GTBs8r6UxJf078rWjCuH9u/NECFu5rr4bbcrKHs9bxVZd0QlnJolDWaU8X7VfGFsihJr7wAaZWflcsCvKHkF9qbEF/98nVxejiJo0n8JWI77Cl7zvZZzF6xI/aenbApk8GT4Cj4EHwMn4XvwuPBrjDY+PaY/RPh598qd9Cj</latexit><latexit sha1_base64="FwqDNGX8HsaSJO47Ghwur+5XsMo=">AAACnXicdVFNj9MwEHXC11K+unCDAxYV0nLYKtkLHCsQEqAFLRLdrlR3o4nrJFbtJGtPkCpv/hW/hBv/BqcpEuyWkax5em9GM36T1kpajKJfQXjj5q3bd/buDu7df/Dw0XD/8amtGsPFlFeqMmcpWKFkKaYoUYmz2gjQqRKzdPWu02ffhbGyKr/huhYLDXkpM8kBPZUMfxwnmr2VObukLK3U0q61T45hIRDa89UhKwDdDqlN9LlbHcZt381ycUFZZoC7uHWfW6ZEhp7+wzBQdQE9ffCfSTvYfoKReYGv+sQuB4NkOIrG0SbodRBvwYhs4yQZ/mTLijdalMgVWDuPoxoXDgxKrkQ7YI0VNfAV5GLuYQla2IXbuNvSl55Z0qwy/pVIN+zfHQ607db2lRqwsFe1jtylzRvM3iycLOsGRcn7QVmjKFa0OxVdSiM4qrUHwI30u1JegDcU/UE7E+KrX74OTo/GcTSOv0ajydHWjj3yjLwgByQmr8mEfCAnZEp48DSYBB+DT+Hz8H14HH7pS8Ng2/OE/BPh7DdAEdGY</latexit><latexit sha1_base64="YZvfKo7wYha/YERVZH/rng1YfkA=">AAACnXicdVFdi9NAFJ3Er7V+bNU3fXCwCOvDlmQR9LEogsoqK9jtQqcbJtObZOhMEmduhDKbf+Uv8c1/46SpoLv1wnAP59zLvXNuWitpMYp+BeG16zdu3tq7Pbhz9979/eGDh6e2aoyAqahUZc5SbkHJEqYoUcFZbYDrVMEsXb3t9Nl3MFZW5Vdc17DQPC9lJgVHTyXDH8eJZm9kzi4oSyu1tGvtk2NYAPL2fHXICo5uh9Qm+tytDuO272Y5fKMsM1y4uHWfWqYgQ0//YRhXdcF7+uA/k3aw/QQj8wJf9IldDAbJcBSNo03QqyDeghHZxkky/MmWlWg0lCgUt3YeRzUuHDcohYJ2wBoLNRcrnsPcw5JrsAu3cbelzz2zpFll/CuRbti/OxzXtlvbV2qOhb2sdeQubd5g9nrhZFk3CKXoB2WNoljR7lR0KQ0IVGsPuDDS70pFwb2h6A/amRBf/vJVcHo0jqNx/OXlaHK0tWOPPCHPyAGJySsyIe/JCZkSETwOJsGH4GP4NHwXHoef+9Iw2PY8Iv9EOPsNQVHRnA==</latexit><latexit sha1_base64="YZvfKo7wYha/YERVZH/rng1YfkA=">AAACnXicdVFdi9NAFJ3Er7V+bNU3fXCwCOvDlmQR9LEogsoqK9jtQqcbJtObZOhMEmduhDKbf+Uv8c1/46SpoLv1wnAP59zLvXNuWitpMYp+BeG16zdu3tq7Pbhz9979/eGDh6e2aoyAqahUZc5SbkHJEqYoUcFZbYDrVMEsXb3t9Nl3MFZW5Vdc17DQPC9lJgVHTyXDH8eJZm9kzi4oSyu1tGvtk2NYAPL2fHXICo5uh9Qm+tytDuO272Y5fKMsM1y4uHWfWqYgQ0//YRhXdcF7+uA/k3aw/QQj8wJf9IldDAbJcBSNo03QqyDeghHZxkky/MmWlWg0lCgUt3YeRzUuHDcohYJ2wBoLNRcrnsPcw5JrsAu3cbelzz2zpFll/CuRbti/OxzXtlvbV2qOhb2sdeQubd5g9nrhZFk3CKXoB2WNoljR7lR0KQ0IVGsPuDDS70pFwb2h6A/amRBf/vJVcHo0jqNx/OXlaHK0tWOPPCHPyAGJySsyIe/JCZkSETwOJsGH4GP4NHwXHoef+9Iw2PY8Iv9EOPsNQVHRnA==</latexit><latexit sha1_base64="YZvfKo7wYha/YERVZH/rng1YfkA=">AAACnXicdVFdi9NAFJ3Er7V+bNU3fXCwCOvDlmQR9LEogsoqK9jtQqcbJtObZOhMEmduhDKbf+Uv8c1/46SpoLv1wnAP59zLvXNuWitpMYp+BeG16zdu3tq7Pbhz9979/eGDh6e2aoyAqahUZc5SbkHJEqYoUcFZbYDrVMEsXb3t9Nl3MFZW5Vdc17DQPC9lJgVHTyXDH8eJZm9kzi4oSyu1tGvtk2NYAPL2fHXICo5uh9Qm+tytDuO272Y5fKMsM1y4uHWfWqYgQ0//YRhXdcF7+uA/k3aw/QQj8wJf9IldDAbJcBSNo03QqyDeghHZxkky/MmWlWg0lCgUt3YeRzUuHDcohYJ2wBoLNRcrnsPcw5JrsAu3cbelzz2zpFll/CuRbti/OxzXtlvbV2qOhb2sdeQubd5g9nrhZFk3CKXoB2WNoljR7lR0KQ0IVGsPuDDS70pFwb2h6A/amRBf/vJVcHo0jqNx/OXlaHK0tWOPPCHPyAGJySsyIe/JCZkSETwOJsGH4GP4NHwXHoef+9Iw2PY8Iv9EOPsNQVHRnA==</latexit><latexit sha1_base64="YZvfKo7wYha/YERVZH/rng1YfkA=">AAACnXicdVFdi9NAFJ3Er7V+bNU3fXCwCOvDlmQR9LEogsoqK9jtQqcbJtObZOhMEmduhDKbf+Uv8c1/46SpoLv1wnAP59zLvXNuWitpMYp+BeG16zdu3tq7Pbhz9979/eGDh6e2aoyAqahUZc5SbkHJEqYoUcFZbYDrVMEsXb3t9Nl3MFZW5Vdc17DQPC9lJgVHTyXDH8eJZm9kzi4oSyu1tGvtk2NYAPL2fHXICo5uh9Qm+tytDuO272Y5fKMsM1y4uHWfWqYgQ0//YRhXdcF7+uA/k3aw/QQj8wJf9IldDAbJcBSNo03QqyDeghHZxkky/MmWlWg0lCgUt3YeRzUuHDcohYJ2wBoLNRcrnsPcw5JrsAu3cbelzz2zpFll/CuRbti/OxzXtlvbV2qOhb2sdeQubd5g9nrhZFk3CKXoB2WNoljR7lR0KQ0IVGsPuDDS70pFwb2h6A/amRBf/vJVcHo0jqNx/OXlaHK0tWOPPCHPyAGJySsyIe/JCZkSETwOJsGH4GP4NHwXHoef+9Iw2PY8Iv9EOPsNQVHRnA==</latexit><latexit sha1_base64="YZvfKo7wYha/YERVZH/rng1YfkA=">AAACnXicdVFdi9NAFJ3Er7V+bNU3fXCwCOvDlmQR9LEogsoqK9jtQqcbJtObZOhMEmduhDKbf+Uv8c1/46SpoLv1wnAP59zLvXNuWitpMYp+BeG16zdu3tq7Pbhz9979/eGDh6e2aoyAqahUZc5SbkHJEqYoUcFZbYDrVMEsXb3t9Nl3MFZW5Vdc17DQPC9lJgVHTyXDH8eJZm9kzi4oSyu1tGvtk2NYAPL2fHXICo5uh9Qm+tytDuO272Y5fKMsM1y4uHWfWqYgQ0//YRhXdcF7+uA/k3aw/QQj8wJf9IldDAbJcBSNo03QqyDeghHZxkky/MmWlWg0lCgUt3YeRzUuHDcohYJ2wBoLNRcrnsPcw5JrsAu3cbelzz2zpFll/CuRbti/OxzXtlvbV2qOhb2sdeQubd5g9nrhZFk3CKXoB2WNoljR7lR0KQ0IVGsPuDDS70pFwb2h6A/amRBf/vJVcHo0jqNx/OXlaHK0tWOPPCHPyAGJySsyIe/JCZkSETwOJsGH4GP4NHwXHoef+9Iw2PY8Iv9EOPsNQVHRnA==</latexit><latexit sha1_base64="YZvfKo7wYha/YERVZH/rng1YfkA=">AAACnXicdVFdi9NAFJ3Er7V+bNU3fXCwCOvDlmQR9LEogsoqK9jtQqcbJtObZOhMEmduhDKbf+Uv8c1/46SpoLv1wnAP59zLvXNuWitpMYp+BeG16zdu3tq7Pbhz9979/eGDh6e2aoyAqahUZc5SbkHJEqYoUcFZbYDrVMEsXb3t9Nl3MFZW5Vdc17DQPC9lJgVHTyXDH8eJZm9kzi4oSyu1tGvtk2NYAPL2fHXICo5uh9Qm+tytDuO272Y5fKMsM1y4uHWfWqYgQ0//YRhXdcF7+uA/k3aw/QQj8wJf9IldDAbJcBSNo03QqyDeghHZxkky/MmWlWg0lCgUt3YeRzUuHDcohYJ2wBoLNRcrnsPcw5JrsAu3cbelzz2zpFll/CuRbti/OxzXtlvbV2qOhb2sdeQubd5g9nrhZFk3CKXoB2WNoljR7lR0KQ0IVGsPuDDS70pFwb2h6A/amRBf/vJVcHo0jqNx/OXlaHK0tWOPPCHPyAGJySsyIe/JCZkSETwOJsGH4GP4NHwXHoef+9Iw2PY8Iv9EOPsNQVHRnA==</latexit>

Old gradient

Page 9: LAG: Lazily Aggregated Gradient for Communication ...06-15-30)-06-15-40-12765-LAG:... · 1 TianyiChen Georgios Giannakis Tao Sun WotaoYin UMN, ECE UCLA, Math NeurIPS2018 LAG: Lazily

9

Iteration and communication complexity

(nonconvex) Local loss is smooth. (convex) Loss is convex.(strongly convex) Loss is (restricted) strongly convex.

In all cases, LAG enjoys the same convergence rate as GD.Theorem 1

If local objectives are heterogeneous, LAG requires smaller number of uploads to a given accuracy than GD; e.g., as small as 1/M.Theorem 2

Page 10: LAG: Lazily Aggregated Gradient for Communication ...06-15-30)-06-15-40-12765-LAG:... · 1 TianyiChen Georgios Giannakis Tao Sun WotaoYin UMN, ECE UCLA, Math NeurIPS2018 LAG: Lazily

10

Linear prediction

q Real datasets distributed on M = 9 workers

M. Lichman, “UCI machine learning repository,” 2013. [Online]: http://archive.ics.uci.edu/ml

Cyc-/Num-IAG: cyclic/non-uniform update of incremental aggregated gradient

Page 11: LAG: Lazily Aggregated Gradient for Communication ...06-15-30)-06-15-40-12765-LAG:... · 1 TianyiChen Georgios Giannakis Tao Sun WotaoYin UMN, ECE UCLA, Math NeurIPS2018 LAG: Lazily

11

Logistic regression

Ø LAG needs same number of iterations but fewer uploads

q Real datasets distributed on M = 9 workers

Page 12: LAG: Lazily Aggregated Gradient for Communication ...06-15-30)-06-15-40-12765-LAG:... · 1 TianyiChen Georgios Giannakis Tao Sun WotaoYin UMN, ECE UCLA, Math NeurIPS2018 LAG: Lazily

12

Conclusions

q Adaptive communication rules for distributed learning

q Not degrade convergence but reduce communication

Thank You!

Thu Dec 6th 05:00 -- 07:00 PM @ Room 210 & 230 AB #8


Recommended