+ All Categories
Home > Data & Analytics > Ensemble selection

Ensemble selection

Date post: 09-Jan-2017
Category:
Upload: sergey-makarevich
View: 69 times
Download: 0 times
Share this document with a friend
21
Transcript

����������� ������� ������� ����������� ��� ���!"�#� ���$�%�'& �)(���� �

*,+.-0/21�3547683:983 ;=<?>A@B<?CD<FEG;IHKJL;AM8>BCDNPOQOAJRNPS8@T,U.VQW 3P9�X?476'Y�+Z-K6 U.V\[ -K6?]_^`+ZaK+ U <8ObN5c8C�EG;IHKJL;AM8>BCDNPOQOAJRNPS8@deV\f=g 1%4 VQh i ;Ij=k:EG;AHbJl;IM8>BCDNPOQO=JlNPS?@T,U.VQWnm�[ +po V\[ <8qsr�tIkPEG;IHKJL;AM8>BCDNPOQOAJRNPS8@u�v w=x�y{z{|}v ~Qz����#�s�K|}wP�5z_v y��5� �lv�~:� vb�A�s�Ky_~Pv��l�D��~P�l�bv y����lz��K�5��z_�:xK��xP�P����� �b�Q�������5�

�������{ �¡ ¢£ �¤)¥7  ��¦�§.��¨©��ª"¢¬«: ®­:¯_­:  �"¢¬«:¯7¢°¯_­K­: �¯7�{ �¤�§¬¨±¢p«P ³²��{�´¡� _ �¤�§¬¨Qµ�¦���ª�¶7��·,¸³¹ º »=¼

½¿¾�ÀbÁ5ÂPÃ�ÄAÁÅÆv®wPy_v��{v ~Qz�x�|}v z_�P�5ÇÈ�p�byF� �K~:��z{y_�:� z{�L~PÉ�v�~:��v�|GÊË �lv´�,�py{�b|�L� Ë y_xKy{�Lv������Èz_�P�K�=�{xK~:ÇP�,���%|}�5Ç5v �Ì��ÍÎ��5Ç5v ���l� Ë y�x�y_�lv´��x�y_vÈÉbv ~Pv�y_x�z{v�Ç��=���L~PÉ�Ç5�lÏIv y_v ~Qz�lv´x�y_~P�l~:ɱx��LÉK�by{�lz{�P|Ð�ÈxK~:ÇÑw:xKy_xK|}v z{v�y%�{v z�z_�l~:Éb��ÍÒP�Ky_Ó®x�y�Çe��z{v�w\Ó°�L�{v��{v �Lv�� z{�L�K~Ô�Ì���=��v´Ç�z{�exKÇPÇ�z_�z{�Pv�v�~:��v�| Ë �Lvez{�Pv�|}�5Ç5v��L�}z_�:x7z)|Ðx�Õ\�L|}�lÖ�v��Rz��w=v�y��p�by{|Ðx�~=� vKÍÔ×Ø~:�{v | Ë �lv��{v �Lv�� z{�L�K~ÙxK�l�L�7Ó��"v ~PÊ��v�| Ë �Lv��Fz_� Ë v��Kw5z_�l|}�LÖ v�Ç�z_�Èw=v�y��p�by{|ÐxK~:� v®|}v z{Êy{�Ì�Ú���=���Ûxb�ÑxK� � �Py�xK� �b�)� y_�b�_�©v ~Qz{y_�Kw\�K�Ð|}v´x�~wPy{v´� �Ì���L�K~?���Ky�Ü�Ý"�Û��y{v´xPÍÞ×�Õ5w=v�y{�L|}v ~Qz_��Ó°�lz{���v��Kv ~�z_v���zßw:y{� Ë �lv�|Ð��x�~=Ç�z{v�~±|}v z_y{�Ì� ��Ç5v |}�K~PÊ��z_y_x�z{vßz{�Pv Ë v ~:v àPz��K��v�~:��v�| Ë �Lv"��v��lv´��z_�l�b~DÍ

á#âßãQä Á\Â:å®æ�ç�ÄAÁPè�å ä��~nv�~:�{v | Ë �lve�L�}x©� �b�l�Lv�� z{�L�K~��K��|}�5Ç5v �Ì�GÓ°�P�b�{vewPy{v´Ç5�Ì��Êz_�l�b~:�Gx�y_v�� �b| Ë �L~Pv�Ç Ë �ÞÓ�v �LÉK�Qz{v´ÇÙx0�Kv�y_xKÉK�L~PÉ��Ky��K��z_�l~:É:Íu��Lv z�z_v y_�L���êéZë�ìbìKìQí"��z�x7z_v��"z_�:x7z©î{�ï~Pv�� v��_�{xKy{�©x�~:ÇÑ�{�5ð}Ê� �lv�~Qz�� �b~:Ç5�lz{�L�K~"�p�by�xK~Èv ~=��v�| Ë �LvØ�K�:� �ÌxK�_���là:v y��Bz_� Ë vs|}�by{vxb� � �Py_x�z{v�z{�:xK~©xK~\�Ô�����Rz��³�l~:ÇP�l�\�ÌÇ5�:x���|}v�| Ë v y��³�L�³�l�sz_�Pv� �Lxb�{�{�là:v y��®x�y_v³xb� � �Py_x�z{v³x�~:ÇeÇ5�L�Kv�y_�{vKÍ ñαx�~\��|Gv z{�P�5ÇP���:x0�bv Ë v�v ~ÆwPy{�bw=�Q��v´Çez{�,ÉKv ~:v y�x7z{v�xK� � �5Êy�x7z_vK���Kv z�Ç5�L�Kv y��{vK�F��v z_�È�K�°|G�5Ç5v��L��ÍÔò®x�ÉbÉK�L~PÉÙé.ò�y_v �L|}xK~D��´óKóbôbí:z{y�x��L~:�B|}�5Ç5v �Ì�B�K�\�b~PvFz��\w=v�épvKÍ É:ÍL���s��Ç5v�� �L�{�l�b~ßz{y_v v���í�b~ Ë �\��z���z_y_xKwÐ�{xK|Gw:�lv´�����Iz{�:v°z{y�x��L~P�L~PÉÈ��v z�ÍØÝ�wP�lz{ÖÈé���óKóbóbíË x�Éb�Æ�pv´x7z{�:y{v´�Æ�L~:��z{v´xKÇ`�K��z_y_xK�l~P�L~PÉêw=�b�l~Qz_��Íõò��\�b��z{�L~PÉéö�5���:xKwP�ly_vK�?ëKìKì:�´í°Ébv ~Pv�y_x�z{v´��x,w=�Kz{v�~bz_�LxK�l�L��|}�Ky_vGÇP�l�bv y���v�{v z�����|G�5Ç5v��L�,z_�:x�~ Ë x�ÉbÉK�L~PÉ Ë �ÚÓ�v �LÉK�Qz{�L~PÉÙz_�Pv©z{y�x��L~5Ê�L~PÉ¿��v z�z{�n�p�Ky�� v©~Pv Ó÷|G�5Ç5v��L��x�z�z_v ~:Ç'z{�øz{�P�Q��v©wA�K�L~bz��z_�:x7z}x�y_v)Ç5�lðÐ� �P�lz�z_�±� �ÌxK�_���l�p�Þ� �Ky_y_v���z_�l�bÍÔ�\�P�L�l�L�7x�~Ùv z�x��öÍéöë�ìbìKìbí Ë �\�b��z?�pv´x7z_�Py{v´�?�L~:��z{v´xKÇ"�K�5z{y�x��L~P�l~:É�w=�b�l~Qz�� ÍF×#y_y{�by�Ê� �Ky_y{v´��z_�l~PÉKÊö�b�5z{w:�5z�Ê£� �5Ç5v´�Øé.×®�°Ý"�®í#éZu��Lv z{z{v y_�Ì���GùÚò®x�ú\�Ly{�ö��´óKóQ�KíB� y{v´x7z{v´�?|}�5Ç5v��L�8Ó°�Rz_�GÇPv�� �by{y_v �Ìx7z_v�dzv�y{y_�Ky�� Ë �³z{y�x��L~5Ê�L~PÉø|}�\ÇPv �Ì���p�by�|%�P�lz{�lÊ�� �Lxb�{�,wPy_� Ë �Lv |Ð�e�K~�Ç5�lÏAv�y{v�~bz�Ç5�lÊ���:��z{�b|}�lv´� ÍÙÎ��:~Py{�Þx�~=Ç�û�x�y_|Ðx�~Qz{�né���óbóKôQí%� y{v´x7z{v´ÇÙÇ5�lÊ�bv y���vß~Pv��Py_xK�B~Pv z��°�\�Lx}� �K|}w=v z{�lz{�L�K~�x�|}�b~PÉ�~:�\ÇPv���ÍüsýQý5þ�ÿ��������������� ����������������� �� ��!��#"%$�&�')(����*�+���-,��.�. ��-,0/213 ��4���+�+5�+�-���6 ��879,:��!����;�=<>��,������?�@��A:BFÿ��DC2A:EFÿ��Qÿ0FQÿ�A@G�H0H�IDJKEML´ý@N������0O@PG�H0H�IRQ@NP�Obþ°ÿ�S%P�O�L���T4J

U�v y_v®Ósv®ÉKv�~Pv y�x7z_v°Ç5�l�bv y���v®�{v z_�F�K�A|}�5Ç5v��L� Ë �È�:�{�L~PÉ"|Ðx�~\�ÇP�RÏIv y_v ~Qz#x��LÉK�by{�lz{�P|Ð��Í?Å©v°�:�{v°�5�PwPwA�Ky{z)V�v´��z_�Ky#ÎÔxb���P�l~:v��éö�WV�α�_í �=x�y{z{�là=� �LxK�B~Pv �Py�x��B~:v z_�%é.��������í��=|Gv�|}�Ky_�bÊ Ë xb��v´Ç�Lv�xKy{~:�l~PÉÔé*X"���ßí��=ÇPv�� �Ì�{�l�b~�z{y_v v��%é.uZY�í�� Ë x�ÉbÉKv�Ç�Ç5v´� �Ì���L�K~z_y{v�v��ÑéZò���[�Ê£uZY�í�� Ë �\�b��z{v�Ç�Ç5v´� �Ì���L�K~êz{y_v v´�Ñé.ò°�-Y�Ê£u8Y�í �xK~:Ç Ë �\�b��z{v´ÇÈ��z_�P|}w:�®é.ò°�-Y�Ê��-Y°ÎÔû�í�Í7ÒP�by8v´xK���%xK�lÉb�Ky_�Rz_�P|Ó�vÐz_y_xK�l~Ñ|}�5Ç5v �Ì�È�=���L~PÉÔ|Ðx�~\�ÞÇ5�lÏAv�y{v�~bzGw:x�y�x�|}v z{v yÈ�{v z{Êz_�l~:Éb��ÍêÒP�by)v ÕPxK|Gw:�lvb��Ó�v�z{y�x��L~ �´ëP���-V�ÎÔ� Ë �n�7x�y_�Q�L~PÉz_�Pv�|Ðx�y_ÉK�L~�w:xKy_xK|Gv z{v�y]\G�sz{�:veúbv y_~Pv �ö�sxK~:Ç�z{�Pv�úKv�y{~:v �w=x�y�x�|}v z_v y��ßé.vKÍ É:Í��7x�y_�\�l~:É�ÉQx�|}|Ðx�Ó°�lz{��Ü�ò®ÒÑúbv y_~Pv �Ì��Í íÅ©v�z{y�x��L~±x Ë �K�5zÈë�ìbìKì}|}�5Ç5v �Ì���p�Kyßv�xb���ÔwPy_� Ë �Lv |�Í"�\�K|}v|}�5Ç5v��L�®�:x0�bv�v ÕP� v �L�Lv ~Qz°w=v�y��p�by{|Ðx�~=� vK�\v0^b�=x��Iz{�}�by Ë v z�z{v�yz_�:x�~}z_�Pv Ë v���zØ|}�5Ç5v �Ì�#y_v wA�Ky{z{v´Ç��L~}z{�:v��l�lz{v�y_x�z{�Py_vKÍ#Ý�z{�:v y|}�5Ç5v��L���D�P�7Ó�v �bv y´�A�=x0�Kv%|}v´Ç5�l�5� y{v}�Ky³v��Kv ~Æw=�\�byßwAv y{�p�Ky{Ê|ÐxK~:� vbÍFÜ�x7z{�:v y�z{�:xK~,� �K| Ë �l~Pv�ÉK�\�5Ç)x�~=Ç Ë xbÇ)|}�5Ç5v �Ì���l~xK~Ùv ~=��v�| Ë �LvK��Ó�v��:�{v)�p�by{Ó®x�y�ÇÑ��z{v w\Ó°�Ì��ve��v��lv´��z_�l�b~Þ�py{�b|z_�Pv}�l� Ë y�x�y_������|G�5Ç5v��L�ßz_�eà:~:ÇÞx���� Ë ��v z³�K�s|}�5Ç5v �Ì�ßz{�=x7zÓ°�:v ~Úx0�Kv�y_xKÉKv�Ç©z_�KÉbv z{�:v yÐ�\�lv��LÇnv ÕP� v �L�lv�~bzÐwAv y{�p�Ky_|Ðx�~:� vKÍY®�:v Ë xK�{�Ì�°v ~:�{v | Ë �Lv���v��lv´��z{�L�K~Ðw:y{�5� v´Ç5�Py_v°�L�Ø�Kv�y{�G���L|}wP�lvD_

�bÍ��Qz_xKy�z°Ó°�lz{��z{�:v"v |}w5z��)v�~:��v�| Ë �LvKÍëPÍ°��ÇPÇ�z{�ßz_�Pv°v ~:�{v | Ë �Lvsz_�Pv®|}�\ÇPv �P�L~%z_�Pv°�L� Ë y�x�y_�³z{�=x7z

|}x�Õ5�l|}�LÖ v���z{�Pv�v ~:�{v | Ë �Lv%` ��wAv y{�p�Ky_|}xK~:� v³z_�Ðz{�:v%v y{Êy{�by�|}v z{y_�L�³�b~exG�P�L�l�Ì� �L�L| Ë ép�7x��L�ÌÇPx7z_�l�b~=í®��v z�Í

�:Í°Ü°v wAv�x�z��Qz{v�w±ëÈ�p�by�x�àPÕ5v�Çe~\�P| Ë v y����F�Rz_v y�x7z{�L�K~=�®�Ky�P~Qz{�L�?x��L�Iz{�Pv"|}�5Ç5v��L�°�=x0�Kv Ë v�v ~��:�{v�ÇBÍ

�=Í°Ü°v z_�Py{~nz{�:vev�~:�{v | Ë �lv,�py{�b| z{�Pv�~Pv´��z_v�Çø��v zÐ����v�~5Ê��v�| Ë �Lv��Gz_�:x7z)�:xb�}|Ðx7Õ5�l|��P| wAv y{�p�Ky_|Ðx�~:� v,�b~nz_�Pv�P�l�L�Ì� �L�l| Ë é.�7x��L�LÇPx�z{�L�K~=í®�{v z´Í

ÎÔ�\ÇPv �Ì�%x�y_v)xKÇ:Ç5v�Ç©z_�±x�~Ñv�~:��v�| Ë �Lv Ë �Þx0�Kv y�x�Éb�l~:É�z_�Pv �Lyw:y{v´Ç5�L� z{�L�K~:�³Ó°�lz{�Æz{�:vÐ|G�5Ç5v��L�"xK�ly_v�xbÇ5���l~©z_�Pv}v ~:�{v | Ë �lvbÍY®�:�L�®|Ðx�úbv���xKÇPÇP�l~PÉ}x�|G�5Ç5v��Iz{��z{�Pv³v�~:��v�| Ë �Lv ¥0  ��a �.xb��z´�xK�l�L�7Ó°�L~PÉÔv ~=��v�| Ë �Lv���Ó°�Rz_�Ùv ÕP� v �L�lv�~bzGw=v�y��p�by{|ÐxK~:� v�z{� Ë v�p�b�P~:Ç �l~Û|}�l~\�5z_v��Þ�py_�K| �L� Ë y_xKy{�Lv��©Ó°�lz{� ë�ìbìKì'|}�5Ç5v��L��ÍÎÔ�Ky_v �7�Kv�y��Fz_�PvÔ�{v �Lv���z_�l�b~nw:y{�5� v´Ç5�Py_vexK�l�L�7Ó��G�:�Gz{�Þ�bw5z{�lÊ|}�LÖ v�z_�PvØv�~:�{v | Ë �lv�z_��x�~\�ßv�xb���L�l�³� �b|}wP�5z{v´Ç³w=v�y��p�by{|ÐxK~:� v|}v z{y_�L�KÍßÅ©vGv �7x��L�:x7z_vÈz_�Pv�wAv y{�p�Ky_|Ðx�~:� v%�K�#v ~:�{v | Ë �lv}�{v Ê�Lv�� z{�L�K~n�K~Ñz_v ~�wAv y{�p�Ky_|Ðx�~:� v�|}v z_y{�Ì� ��ÍÞÅÆv Ë v �L�lv��Kv,z{�P�Ì��Ì�sz{�Pv�à:y���z�z{�L|}v"xÈ�Lv�x�y_~P�L~PÉ%|}v z{�P�5Ç,�:xK� Ë v�v ~,v �7x��L�:x7z_v�Çxb� y_�b�_�®���:����xGÓ°�LÇPv"�0xKy{�Lv z��Ð�K��wAv y{�p�Ky_|Ðx�~:� v³|Gv z{y_�L��� Í

b8ced�f%g4heijd�k�lnmDk+o]f�cjgpdqord�s+kut�gqv dZgpm%oriwf%k+dqd�cwx+d�ozy;{|d}x+dp~{�d�gps+t�m%c�s+mrsuh;dRorm-�-d�{Kt|c�s+hwdR{|t�y;k4f�ku�]s+hwf�s�ijd�k�lnmDk+o]x�y�d�x�smDcsuhwf@s�ord�s+kut�g%�)�#d0g�f��jx�d#v d��Dd�c;d�kuf�s+d�x�m}o]f�cW��;t��>d�kud�c�sorm-�-d�{�x���{�t|y;k4f�kut�d�xK�wx��jf�{�{|�g�m%c�suf%t|c�f�lnd�v�orm-�-d�{?x)v�t|s+hrdp�W~g�d�{�{|d�c�s8i�d�k+lnm%kuorf%cwgpdzm%c�f%cW�9i�d�k+lnm%kuo]f�cwg�d�ordps+kut?g��}�%�wx�sx+d�{�d�gps+t�c;�8s+h;d�y�d�x�s�x�t�c;�%{�d#o�m-�-d�{Wlnk+mDo�fZ{|t�y;k4f�ku�Z�Wt�d�{?�;x)kudp~o]f%k+�@f�yw{|�Z�%mWm-�}ijd�k�lnmDk+o]f%cwgpdD�K��cjx�d�oy;{�d6x+d�{�d�g�sut|mDc=�:h;m@v�~d��%d�k��q�wcw�;x�d�cwx�d�oy;{�d�x9suhwf@s�m%�;s+i�d�k+lnm%kuo�s+h;d�y�d�x�s�x+t�c-~�D{|deo�m-�-d�{�x�����h;t?xx+�;�D�%d�x�sux�s+hwf�s�wx+t|cw���-t|��d�k+d�c�s{|d0f�kuc-~t�c;�8o�d�s+h;m-�;x)f%cw�Riwf%kuf%o�d�s+d�kKx+dps�sut|cw�DxK�%d�cwd�k4f@s+d0x={�t|ywkuf%k+t�d�xg�m%c�suf%t|cwt|c;�]fr�-t��%d�kux+dZx+dps�m%lK�Dm�m-�W~�i�d�k+lnm%kuo�t�c;��ormW�;d�{?x����hwdziwf%kuf%ordps+d�kuxqv d��@f�ku��lnm%kZd�f%g4h�f�{��%mDk+t|s+h;o�sume�%d�c;d�k+~f�s+d�% % % zorm-�-d�{?xqf%k+dR�-d0x+g�k+t�yjd0��t|c�suh;d�¡�i;i�d�cj�-t��2��¢�m%s+dsuhwf@s8v d��-mec;m�sq�-d�s+d�k+ort�c;d�v�hwf�sqiwf�k4f�ord�s+d�k4x��Wt|d�{���y�d�x�si�d�k+lnm%kuo]f�cwg�d�v�h;d�c£sukuf%t|c;t�c;�¤o�m-�-d�{�x��¥¡�{�{}o�m-�-d�{�x�f%k+dfD�;�-d0��s+m�f¦{�t|ywkuf%k+��c;m�o]f�s�s+d�kzhwm@v¥�%mWm-�§m%k�ywf%�§s+h;d��f%k+dD�9¨�m-�-d�{6i;kud��;t�gps+t�m%cwxRmDc�s+h;d©s+k4f�t�c�f%cw��h;t�{|{?gp{�t�oy;t�c;�x+dps4xzf%k+d9g�f%g4hwd��2����h;t?xzx+t|ori;{�t|�wd�xzv mDk+�Wt�c;�¦v�t|s+h�s+h;d9{|t|~ywkuf%k+�ªf�cw�ªo]f��Dd�x]orm-�-d�{Zx�d�{|d0g�sut|mDc«l¬f%x�s+d�keyjd0g�f��jx�d�suh;dorm-�-d�{�x��;m�cwm�s�hwf:�%dZsum�y�dRdp�-d0gp�-sud����-�;kut|cw�rx+d�{�d�gps+t�m%c=�

­�®Z¯�°²±q³;´Kµ�¶�·�¸º¹�·q»�¼3°¾½Z¿�¼¾À�¼3¿�¼KÁ�Â-¶�´6·��hwd¤x+t|ori;{�dÃlnmDk+v�f�k4�Äo�m-�-d�{�x+d�{�d�gps+t�m%c¾i;kum-gpd0�-�;kudÃi;kudp~x+d�c�sud��«t�cÅs+h;d�Æ�c�s+kumW�;�wg�sut|mDcÅt�xel¬f%x�s9f�cj�Ådp�>d�gps+t��%d%��yw�-sx+m%ord�s+t�o�d0x)m@�%d�k��;s4xKs+mZs+h;d�hwt|{�{�g�{|t�oywt|c;�Rx+dps0�%kud��-�wg�t|cw�8d�c-~x+d�ozy;{�dZi�d�k+lnm%kuo]f�cwg�d%�)Ç�dRorfD�-dZs+h;kud�dRfD�;�-t|s+t�m%cwx#sumrs+h;t?xx+d�{�d�gps+t�m%cªi;kumWg�d��-�wk+d�s+m�kud��-�jgpd�m@�%d�k��;s+s+t�c;�w�Å��h;d0x�d�f%k+d�;t�xugp�wxux+d��9t|c�suh;dRc;dp�Ws�suh;k+d�d�x��wy-~�x+d�gps+t�m%cwx�����h;d�x+dRordpsuh-~m-�;x6o]f:�zy�dq�wx+dpln�w{jt�c©m�suh;d�k#f�i;iw{|t?g�f�s+t�m%cwx6v�h;d�kud�lnm%kuv�f�k4�x�s+d�iWv�t�x+d¦x�d�{|d0g�sut|mDcªt?x©iwk+mDc;d�s+mÃm@�Dd�k+�;s�sut|c;�j�#x+�wg4hÅf%x©t|clnd0f@su�;k+dRx+d�{�d�gps+t�m%c�ȬÉ}m%hjf:��t=ʺ�%mDh;c=�2Ë0Ì%Ì�Í%Î��

Ï=ЬÑ�Ð)Ò3ÓWÔ¬ÓWÕ�Ö0׬Ø�ÙªÚr׬Ö�ÛÅÜ9ÓDÝ)Ô¬ÞwÕ%ÓWßÃÓWÙ>ÖÇÅt|s+hàorm-�-d�{¦x+d�{�d�gps+t�m%câá�ã.ä.åwæ@ç-ä�kud�i;{?f%g�d�ord�c�s���i�d�k+lnm%k+~o]f%cwgpdrt�o�iwk+m@�Dd�xZf%xZs+hwd�y�d�x�s�orm-�-d�{?xRf�kud�fD�;�-d0�¦sum9suh;dd�cwx+d�oyw{|dD�#i�d�f%�-x���f�cw�¤suh;d�c£è��;t?g4�W{|�ª�-d�g�{|t�c;d�x��êé)d�k�lnmDk�~o]f%cwgpd��-kum%iwx�y�d�g�f��wx+desuh;d�y�d�x�srorm-�-d�{�xrt�c�suh;d�{|t�y;k4f�ku�hjf:�%d�yjd�d�c«�wx+d��«f�cj�ªx+d�{�d�gps+t�m%cªo�wx�s©c;m@v�f%�;�ªorm-�-d�{?xsuhwf@s8h��wk�s8s+h;d�d�cwx+d�oyw{|dD��ëKt��%�wk+d]Ëx+h;m@v�x�suh;t?x�y�d�hwf:�Wt�m%klnmDk}k+mWm�s+~�ord0f�c-~�x+è��wf%k+d0�W~�d�kuk+mDk��Zì�c-lnmDk�su�;cwf@sud�{��%�=orm�x�sRd�k+~kum%k)ordpsuk+t?g�x���t�d�{?��o�wg4h�y;�;ori;t�d�k6�%k4f�i;hwxMsuhwf�c�suh;t�x�v�h;d�chwt|{�{�g�{|t�oywt|c;�Zt?xK�-mDc;d#v�t�suhzx+orf%{|{W�;f�sufZx�d�sux��@orf%�Wt|c;�8t�s��-t|l.~��gp�;{|s�s+mrkud�{�t�f%y;{��ei;t?g4�efr�%mWm-��x�sum%i;i;t�c;�]i�m%t�c�s��6��h;dR{�mDxuxt�crijd�k�lnmDk+o]f%cwgpd�g�f%czy�d�x�t��%cwt��jg�f�c�s)t�l�s+h;d�i�d�f%��t?x)o�t?xux�d0�2�ëMt��%�;kud}Ë�f�{?x�mRx+h;m@v�xKs+hwf�s6x�d�{|d0g�s+t�c;�}orm-�-d�{?x�á�ã.ä.å©í+î�ï�ð|ñDò4îpóô î�õwä=�Dk+d0f@s+{��ekud��;�wgpd0x�s+h;t?x8i;k+mDy;{�d�o���ö-d�{�d�g�sut|mDc�v�t�suh¦kudp~iw{�fDgpd�ord�cDs�f%{|{�m@v�x�ormW�;d�{?x#s+m]y�d�f%�;�-d0�©sum�s+h;dRd�cwx�d�oy;{�doz�;{|s+t�i;{|d�s+t�ord�x��÷b8cwgpd�ijd0f��«ijd�k�lnmDk+o]f%cwgpd�t�x9k+d0f%g4h;d0�2�t|l�s+hwde�;cW�wx+d��§orm-�-d�{�x�f�{�{6hW�;k+szd�cwx�d�oy;{�d©i�d�k+lnm%kuo]f�cwg�d%�x+d�{�d�gps+t�m%c�fD�;�;xo�m-�-d�{�xs+hjf@szv#d�kud9fD�;�-d0��yjd�lnm%kudek4f@s+hwd�ksuhwf�c9hW�;k�s#ijd�k�lnmDk+o]f%cwgpdD�)��hwt�x6øjf@s+s+d�cwx s+h;dZijd�k�lnmDk+o]f%cwgpd

g��;ku�%d�iwfDx�sqs+h;dzijd0f��>��f%cw��f�{�{�m@v�x�x�d�{|d0g�sut|mDc�s+m©�jc;d�su�;c;dd�cwx+d�oyw{|d0xqyW��v d�t|�Dh�s+t�c;�©orm-�-d�{?x�ùqo�m-�-d�{�xZf%�w�-d���sumesuh;dd�cwx+d�oyw{|d}o�w{�sut|i;{�d}s+t�ord�x�kud�g�d�t��%d}orm%kud8v#d�t��%h�s0�

0.32

0.325

0.33

0.335

0.34

0.345

0.35

0.355

0.36

0 50 100 150 200

RM

S E

rror

Number of Models in Ensemble

hillclimb: selection without replacementhillclimb: selection with replacementtest: selection without replacement

test: selection with replacement

ú>ûýüWþ0ÿ������������� ������������������������������ ��!�#"$��% ��� &��'(�&�)�+*ö-d�{�d�g�sut|mDcev�t|s+h�k+d�i;{?f%gpd�ord�c�s6øjf�s�sud�cwx#suh;dZg��;ku�%d}x�mzo�wg4hsuhwf@s�f�s+d0x�s�x+dpset?xecwm�s9cwd�d��;d��«sum��-dpsud�kuort|c;d�v�h;d�c£s+mx�s+mDi¤fD�;�-t�c;��orm-�-d�{�xzs+m�s+hwd�d�cwx+d�oyw{|dD����h;d�h;t|{�{?gp{�t|ozy-~t�c;��x+dps�g�f%c�y�d��jx�d0�¦sum�x�s+mDi�hwt|{�{�g�{|t�oywt|c;�j����hwt�x}ord�f%cwxd�cwx+d�oyw{|drx�d�{|d0g�sut|mDc��-mWd0x8cwm�sRc;d�d0��ormDk+ds+d�x�s�x�d�suxZsuhwf�csuh;dZywf%x+dp~�{|d��%d�{worm-�-d�{?x6v#m%�w{��]hjf:�%dq�jx�d0�rs+mrx�d�{|d0g�s#o�m-�-d�{ijf�k4f�ordpsud�k4x��3�6cwx+d�oyw{|d�x+d�{�d�g�sut|mDcz�jx�d0xMs+h;d��@f�{�t��wf@s+t�m%crx+dpssum]�-m]yjm%s+h�iwf%kuf%o�d�s+d�kRñ�õ-,Zorm-�-d�{Kx�d�{|d0g�s+t�m%c3�

Ï=Ð?Ï=Ð)Ò3Ø/.0Ö0Ó�0218Ù43�ÓWß65MÔ*Ó87�ÙK׬Ö�׬ÞwÔ¬×:9DÞ-Ö�×*ØjÙëwm%kuv#f%ku�rx+d�{�d�gps+t�m%c©x+m%ordpsut|ord�x m@�%d�k��wsux�d0f�ku{|�zt|c�x�d�{|d0g�s+t�m%cv�hwd�c�d�cwx+d�oyw{|d0xzf%k+d�x�o]f�{�{��Ãb8cwd9v�f:��sum�iwk+d��%d�c�szs+h;t?xt?xs+m�t�c;t|s+t?f�{�t<;�ded�cwx�d�oy;{�d�xzv�t|s+h�ormDk+d©orm-�-d�{�x���Æ�cwx�s+d0f%�m%l�x�s4f�k+s+t�c;��v�t�suhÃf�c§d�ori-s���d�cwx+d�ozy;{�d%�)x�mDk�sRsuh;d©orm-�-d�{?xt�c�suh;d{�t|ywkuf%k+��yW�esuh;d�t�k8i�d�k+lnm%kuo]f�cwg�d%��f�cw��iw�-sqsuh;dzy�d�x�s= orm-�-d�{?xrt�c¤suh;d�d�cjx�d�oy;{�d%� = t�xeg4h;mDx+d�cªyW�Ã{|mWmD��t�c;�f�s�ijd�k�lnmDk+o]f%cwgpdZm%c�s+hwdRh;t|{�{?gp{�t|ozy;t|cw�©x�d�s��6��h;t?x�s��Wi;t?g�f%{|{��fD�;�;x?>@~��@>�m�l}s+hwd¦yjd0x�s9o�m-�-d�{�x©s+m�f%c«d�cwx+d�ozy;{|dBAuîDCpæ@íuî�Dk+d�d��-��x�sud�iWv�t?x�dx+d�{�d�gps+t�m%c�y�d��%t�cwx��qöWt�cwgpd�d0f%g4h�m%lMsuh;d =y�d�x�sRormW�;d�{?x}ijd�k�lnmDk+o]xZv d�{|{��2suh;d���lnmDk+oâf9x�s+kum%c;��t�c;t�sut�f%{d�cwx+d�oyw{|d�f�cj�t|s�t�x)ormDk+d��-t<E©gp�;{|s�lnm%k��Dk+d�d��-��x+d�{�d�gps+t�m%czs+m�jcw��o�m-�-d�{�x�s+hwf�s�m@�Dd�k+�;s�v�h;d�c�f%�w�-d��9s+m�suh;dRd�cwx+d�ozy;{�d%�

Ï=ÐGF3ÐIH�ÞKJ-JjÓ�021ZÙI30ÓWßL5)Ô¬Ó�Ò3Ó�Ô*ÓWÕ�Ö�×*ØjÙ¡8x�s+h;d�cW�;ozyjd�k]m�l8orm-�-d�{�xrt�cªf�{�t|y;k4f�ku�§t�cwgpkud�fDx�d0x���suh;dg4hjf�cwg�d�x m�l=�wcj�-t|cw��g�m%oywt|cwf�s+t�m%cwx m�l=orm-�-d�{�x s+hjf@s�m@�%d�k+�;ssuh;d©h;t�{�{�g�{|t�oy;t�c;�¦x+dpst�cwg�k+d0f%x+d�x��e��f��%�Dt|cw��g�f�c§ort|cwt|ortG;�dsuh;t?x�i;kum%y;{�d�o��#Ç�dzk+d0�-�wg�d}s+h;dzc��woy�d�k�m%l)orm-�-d�{�x8x�d�{|d0g�~sut|mDc�g�f%c¦g4h;mWmDx+d}lnk+mDoày����-k4f:v�t|cw�©f]kuf%cw�-mDo�xuf�ori;{�d�m�lorm-�-d�{�x�lnk+mDo suh;d�{�t|ywkuf%k+��f�cw��x+d�{�d�gps+t�c;�]lnkum%o s+hwf�sZxuf�o�~iw{|dD�#Æ�l�f]ijf�k+s+t?gp�;{?f�kqg�m%ozy;t|cjf@s+t�m%c�m%l�¨ orm-�-d�{�xqm@�%d�k+�;s4x��suh;diwk+mDywf�y;t�{�t�s���m�l�suh;mDx+dz¨ orm-�-d�{�xqyjd�t|c;�9t|c�f]k4f�cw�-mDoyjf��©m%l)orm-�-d�{�xqt�xq{|d0x+x�s+hwf%cÃÈ�ËNMPO>ÎRQ lnm%kSO¦s+h;dlnkufDg�s+t�m%c

TUWVXTZYZ[�\^]S_G`8a�bK[�cedgf/hIij[�ke]R[ml8n2oep qKrZdg`eY�c/dgfs[�`e]�[�V�tce\<[u]�[�\G[�vwa�_GT`�xgo8a�_GVX[�]ya�Tz_<`e]�kK{�[Xa�bedga(a�bK[�c-[�]Ra�VXTZYZ[�\^]| _G\G\Sbed~}@[�VXd`��6T@�K�/T@{Ra�kK`K_<a�_G[�]ya�T�c-[8]�[�\G[�v+a�[�Y�hz��bK[u�Kt`/dg\�[�`/]R[�VscK\G[�_^]ma�be[yd~}@[�{&dgf@[�TgU�a�bK[sxgos[�`e]�[�V�cK\G[�]�h���df@]TU4[�`e]�[�Vsce\<[�]�]R[�[�V�vwTVX�K\G[w��rKcekZaN[�d@v&b8[�`e]�[�Vsce\<[(_^]���ke]Rad | [�_<f@b a�[�YLd~}[�{&dgf@[(TgUSVXTZYZ[�\G]�r�]RT?a�bK[Xd~}[�{�df[(TgUmd?]�[waTU$[�`e]R[�VscK\G[�]yd\G]�T8_^]ydP]R_GVX�K\<[ | [�_Gfb a�[�YLd~}@[�{&dgf[sTgUSa�bK[c/d]�[wt�\<[�}[�\eV�TZYZ[�\G]�h���dgff@_<`ef(_G]$YZ_G]�vwk/]�]�[�YX_G`8��[�v+a�_GT`�qZh �Kh

���y���I�Z�������K�ij[�[w�Z�/[�{�_GVX[�` a | _�a�b�]�[�}@[�` �K{�Tce\<[�VX]�¡ ¢�£�¤�¥Z�yr¦�§�¨�©$ª �m«�¬ © r/¥ © ��� ©$ª h �W­r®dg`eY�¥ © ��� ©$ª h �-x�U¯{�T@Va�bK[�¤ ¦$°Xª [��/T ]R_<a�T@{���±D�$\^dg²@[z³µ´z[�{�¶@rN­�·@·¸ ¹+r$´ © £ ° �®rdº�K`K[�kKVXT`K_^d»YKd)a&dº]�[wa�r��Z¥W¢ ¦ r�YKdga�d¼U¯{�T@V½vwT@\<\^dgc-T{&d)ta�T{&]¼dga¼a�be[¾� a&dg`ZU¯T@{�Y¿¥W_<`e[�dg{B¢Nv�vw[�\G[�{&d)a�T{�r8d`eY¿À�«�t¬ ©Sª �Z¬ ©m¦ �yrsa�bK[ ° `eYZ_^dg`Á¬S_<`K[�]Âb����/[�{�]��-[�v+a�{�d\8YKdga�d±ÄÃ�kedg\<a�_G[�{�_#[�asdg\Äh<r$­�··@·@¹whu¢N£�¤N¥Z�yrI´ © £ ° �6d`eY��Z¥W¢ ¦d{�[scK_<`/dg{��?�K{�TcK\G[�V�]�h ¦�§�¨�©Sª �m«y¬ © r/¥ © ��� ©Sª r/dg`eYÀ�«�¬ ©Sª �Z¬ ©m¦ �Ådg{�[�Æ�r/xÇKred`eYj­�ÇXvw\^d]�]��K{�TcK\G[�V�]�rK{�[wt]��-[�v+a�_<}@[�\G�hIi6[#v�T`�}[�{Ra�[�YNa�bK[�]R[�a�T�cK_G`edg{��Nc-[�v�dgke]�[�V�dg`��TU�a�be[8VX[wa�{�_^v�] | [�]Ra�k/YZ�Âdg{�[8YZ[��e`K[�YÂT`ºcK_G`edg{��È�K{�T@cZt\G[�V�]�r(dg`eY¾c-[�v�dke]�[j\G[�d{�`e_<`Kf»VX[wa�bKTZYK]�]�kev&bÉd]�� ¨ ´�]d`eY6c/T�T@]Ra�_G`Kfzdg{�[X[�d@]R_G[�{�a�Tzd�K�K\G�za�TPcK_G`edg{����K{�Tce\<[�VX]�h¦�§�¨�©$ª �m«�¬ © | d]XvwT@`�}[�{�a�[�YÈa�TÈcK_G`ed{���c���a�{�[�d)a�_G`Kfa�bK[u\^dg{�f[�]�asv�\Gd@]�]�d@]sv�\Gd@]�]�­@h�¥ © ��� ©$ª | d@]�vwT@` }@[�{�a�[�Ya | T | d~�Z]�hÊ¥ © ��� ©$ª h �W­Pa�{�[�d)a&]�a�bK[jv�T`ZU¯k/]�dcK\<[L\<[�aRa�[�{Ë § Ë d@]?v�\Gd@]�]L­Ldg`eYBa�bK[L{�[�V�d_<`K_G`Kfºx@qÈ\G[waRa�[�{&]?d@]?vw\^d]�]oerW��_G[�\^YZ_<`efPd�}@[�{���kK`�cedg\^dg`/vw[�Y6cK_G`edg{��z�K{�TcK\G[�VPhu¥ © �mt� ©$ª h �/x8ke]R[�](\G[waRa�[�{&]�­wt&­���d](v�\Gd@]�](ozd`eYj\<[�aRa�[�{&]X­�Ìgt�xgÇd@]�vw\^d]�]�­@re��_G[�\^YZ_<`ef?d�YZ_<Í�v�kK\<a�r®cKkZa�cedg\^dg`ev�[�Y�r/�K{�TcK\G[�VPhÀ�«�¬ ©Sª �Z¬ ©m¦ � | d]Nv�T`�}[�{Ra�[�Y�a�T?cK_G`edg{���c ��a�{�[�d)a�_G`Kfa�bK[(\^dg{�f[yvwT`KU¯ke]�dcK\G[yv�\Gd@]�]��ZT)� c-[�d`Ztδz_<` a�_<\�d@]�vw\^d]�]�­@h��be[�]�[uYKd)a&d�]R[�a�] | [�{�[u]�[�\G[�v+a�[�YÈc-[�v�dgke]�[Xa�bK[��jd{�[�\^dg{�f[[�`KTkefbÅa�T2dg\G\<T | V�TZYZ[�{�dga�[È]R_G¶�[ja�{�d_<`Édg`eYÅ})dg\G_GYed)a�_GT`]�[wa&]�r)d`eY�]Ra�_G\<\ bed~}[$YKd)a&d�\G[wUÏaWU¯T@{I\^dg{�f[��/`edg\@a�[�]�a�]R[�a�]�hIÐeT{T@kK{�[w�Z�-[�{�_<VX[�` a&]�r | [�k/]R[�Yua�{�d_<`K_G`KfX]R[�a�]mTgU�qgo@oo��-T_G` a�]�h© dv&bXa�{�d_<`K_G`Kf�]�dV��e\<[ | d@]S]��K\<_<a�_<` a�T�d(a�{�d_<`u]�[wa�TgU�Ì ooo�-T_G` a�]�d`eYsd�bK_G\<\^vw\G_<V�cK_G`Kf�Ñ)}~d\<_^YKdga�_GT`�]�[wa�TUW­�o@oo��-T_G` a�]�h��be[(�e`ed\�a�[�]�a�]R[�a�]�U¯T@{�VXT@]RaNTgU�a�bK[�]�[��K{�TcK\G[�V�]�v�T` a�d_<`xoKr oo@ou�-T_G`@a&]�Ò�[�`KT@kKfbLa�TPV�dg²[�YZ_^]�vw[�{�`K_G`KfP]RV�dg\G\#YZ_<UÏtU¯[�{�[�`evw[�]�_<`P�-[�{�U¯T{�V�dg`ev�[�{�[�\G_GdcK\G[h

ÓN��ÔP��ÕeÖ�×SÕeØÙ��Ú�Û®�ÅÜÝ����Õ/ÞRÛ��ij[Âke]�[Èa�[�`ß�/[�{RU¯T@{�V�d`evw[ÈVX[wa�{�_^v�]�¡àdv�vwkK{&dv��á±:¢ ¦�¦ ¹wr{�T�Tga�tÄVX[�d`ZtÎ]�â ked{�[�Y�tÄ[�{�{�T{z± ª ´L�K¹wr�V�[�dg`¾vw{�T@]�]�t�[�` a�{�T���±D´zã © ¹+r�\G_�UÏaz±:¥IÐ4��¹wr��K{�[�vw_^]R_GT`®Ñ~{�[�v�d\<\ScK{�[�dg² t�[�}[�`È�/T@_<` a±D� © ¬$¹wr��K{�[�vw_^]�_<T@`-Ñ~{�[�v�dg\G\�ÐItÎ]�v�T{�[B±DÐ�� ¦ ¹+rNd~}@[�{&dgf[6�K{�[wtv�_G]�_GT`á±:¢�¬ ª ¹wr ª�§(¦ ¢�{�[�d2± ª�§(¦ ¹wr$d`eY2d�VX[�d@]RkK{�[zTgU�e{�T@cedgcK_G\G_�a��áv�dg\G_<cK{&d)a�_<T@`ä± ¦ ¢N¥I¹whs��bK[Ba�[�` a�bÙV�[�a�{�_GvB_^]å#æNç nè± æ�é(éÊê¾ç(ë(éáê ±R­sì ç(íßå ¹R¹�î)�Kh2�Z¢ ª _^]�d{�Tceke]�a�VX[wa�{�_^v8a�T�ke]�[ | bK[�`ºa�be[�vwT{�{�[�v+aXVX[wa�{�_^vP_G]Xke`Zt

²�`KT | `WhÙ¢N`Åd)a�a�{&dvwa�_G}[zU¯[�d)a�ke{�[jTgU�[�`e]R[�VscK\G[È]R[�\<[�v+a�_GT`_^]�a�b/d)au_<a6ï�ðgñ�T�Za�_<VX_G¶�[�a�T�VX[�a�{�_Gv�]�]Rkev&b»d]u�K¢ ª h#ij[v�TVX�ed{�[P�-[�{�U¯T{�VXd`evw[�ke]�_<`efÈa�[�`»VX[wa�{�_^v�]Xc-[�v�dgke]�[�YZ_<UÏtU¯[�{�[�` aSVX[wa�{�_^v�]#d{�[�d�K�K{�T�K{�_Gdga�[�_G`uYZ_�ò®[�{�[�` a$]�[wa�a�_G`Kf@]Sdg`eYc-[�v�dgke]�[L\G[�d{�`e_<`Kf¼V�[�a�bKTZYK]?a�b/d)a8�-[�{�U¯T{�V | [�\<\�T`2T@`K[VX[�a�{�_Gv(YZTX`KTa�dg\ | d~�Z]m�/[�{RU¯T@{�V | [�\<\�T@`8Ta�bK[�{�VX[wa�{�_^v�]�h

ó���ôzØÙõ�ÞRÕeÞRÛ®��ö(÷»�W��ø�öÎ�K�° `ja�bK_G]�]R[�v+a�_<T@` | [uvwT@VX�edg{�[X[�`e]�[�V�cK\<[u]�[�\G[�vwa�_GT`ja�TPa�bK[c-[�]RaLVXTZYZ[�\^]za�{�d_<`e[�Y | _�a�bádg`��àTUXa�bK[Â\G[�d{�`e_<`Kf�d\<f@Tgt{�_<a�bKV�]�redg`eY�dg\^]�T�a�T�]R[�}[�{&dg\®Ta�bK[�{�[�`e]�[�V�cK\<[yVX[wa�bKTZYK]�hù�ú:û-úIüzý/þ)ÿ���������� ���@ýeþ����¬�[�{RU¯T@{�V�d`evw[�VX[wa�{�_^v�]$]Rkev&bud@]#¢ ¦�¦ T{ ª ´L�X{&dg`Kf@[mU¯{�T@Voza�T¼­@r | bK_G\<[8Tga�be[�{&]?±:¥IÐ��yr4´zã © ¹�{�d`Kf[�U¯{�TV oPa�T?l| be[�{�[ml�YK[��-[�`eYK]�T`�a�bK[�Yed)a�deh#ÐKT@{�]RT@V�[yVX[wa�{�_^v�]�\<T | [�{})d\<kK[�]�_G`eYZ_^v�d)a�[�c-[waRa�[�{��/[�{RU¯T@{�V�dg`/vw[h�ÐKT@{�Tga�bK[�{&]�bK_Gfbe[�{})d\<kK[�]�dg{�[$c-[wa�a�[�{�h���bK[$c/d]�[�\G_<`K[m{&d)a�[�]4TgUeVX[wa�{�_^v�]�]�kev&b�d]ª�§(¦ dg{�[$_G`eYZ[��/[�`eYZ[�` a#TUea�be[�Yed)a�der | bK_G\<[�Ta�bK[�{�]�]�kev&bXd]¢ ¦�¦ b/d~}[�ced@]R[�\<_G`K[�{�dga�[�]Na�b/d)a�YK[��-[�`eYLT`za�bK[�Yed)a�deh ° Uc/d]�[�\G_<`K[�¢ ¦�¦ n oKh ·¸Kr®¢ ¦�¦ nÙoKh ·¸e­(�K{�TcedcK\<��_^]N�-T�T{�-[�{�U¯T{�V�dg`ev�[r�cKkKa�_<UNa�bK[���d~�@[�]�T@�Za�_GV�dg\�¢ ¦�¦ n�oeh Ç@oKrd@v&bK_G[�}�_<`efN¢ ¦�¦ nBoKh�qg·�VX_Gfb aIc-[S[��Kvw[�\G\G[�` aI�-[�{�U¯T{�V�dg`ev�[h�IT�dg\G\<T | d~}[�{&dgf@_<`efLdv�{�T ]�]��K{�TcK\G[�V�]�dg`/Y¼VX[wa�{�_^v�]�r | [v�T`�}[�{Ra��-[�{�U¯T{�V�dg`ev�[�]�a�T8d�`eT{�VXd\<_G¶�[�Y�� o���­��I]�v�dg\G[h § `a�bK_^]u]�v�dg\G[r�oj{�[��K{�[�]R[�` a�]�ced]�[�\G_G`K[z�/[�{RU¯T@{�V�d`evw[8U¯T{�a�bK[VX[�a�{�_Gv�dg`eYÂ�K{�T@cK\G[�V ±¯[@h f/h<rSoeh qzU¯T{ ª�§(¦ rIT@{sa�bK[8c/d]�[wt\G_G`K[�¢ ¦�¦ U¯T@{$[�dv&b��K{�TcK\G[�Vu¹wr�d`eYz­�{�[��e{�[�]R[�`@a&]�a�bK[�c-[�]Ra�-[�{�U¯T{�V�dg`ev�[�]�[�[�`?U¯T@{�dg`���VXTZYZ[�\�T@`?a�bK[��e`ed\®a�[�]RaN]�[wa�h��¦ ¢�¥#r/a�bK[sV�[�a�{�_Gvske]�[�Y�a�T?V�[�d]�kK{�[(�K{�T@cedgce_<\G_�a��Pv�d\<_GcK{&d)ta�_<T@`�r�_G]skK`�ke]�kedg\$_<`�a�bed)asa�bK[8ced]�[�\G_<`e[uVXT�YK[�\#a�bed)a��K{�[wtYK_Gvwa�]Il�U¯T{�dg\G\/v�d]�[�]�r | bK[�{�[���_^]�a�be[��-[�{&vw[�`@a$TgU��-T@]�_�a�_<}@[�]_G`�a�bK[Na�[�]Ram]�[wa�r b/d]#[��Kvw[�\G\G[�` amv�dg\G_<cK{&d)a�_<T@`�h �?��bK_^]$v�{�[�d)a�[�]d8�K{�TcK\G[�V | bK[�`L`KT@{�V�d\<_G¶�_G`Kf ¦ ¢�¥º]�vwT@{�[�]�c-[�v�dgke]�[sa�bK[c/d]�[�\G_<`K[(VXTZYZ[�\Wdg`eYP��d~�[�]mT�Za�_<V�dg\WV�TZYZ[�\�bed~}@[y]�_GV�_G\^dg{¦ ¢�¥�]�vwT{�[�]�hI¤N`K\G_<²@[�a�bK[NTga�be[�{#VX[�d@]Rke{�[�]�r ¦ ¢�¥z_^]S]�v�dg\G[�Y]�TPa�b/d)a(a�be[uVX_<`K_GVskeV T@ce]R[�{�}@[�Y ¦ ¢N¥�]�vwT@{�[X_^]�oKh oPdg`eYa�bK[(VX[�d`8T@ce]R[�{�}@[�Y ¦ ¢N¥�]�vwT{�[�_G](­@h oeh��be[�c/TaRa�TV�TgU��IdcK\G[­�]RbeT | ]�`KT{�V�dg\G_<¶�[�Y»]�vwT{�[�]�U¯T{[�dv&bB\<[�dg{�`K_G`Kfjd\<f@T{�_�a�bKV | bK[�`º_<a�]��/dg{&dgVX[wa�[�{&]�d{�[z]�[wt\G[�vwa�[�YjU¯T{s[�dv&bÈ�K{�TcK\G[�V dg`eYÈVX[�a�{�_Gv�k/]R_G`Kfza�bK[u})dg\G_GYed)ta�_<T@`�]R[�a�]�[�`e]�[�Vsce\<[u]�[�\G[�vwa�_GT`ÈbK_G\<\^vw\G_<V�ce](T@`�h © `@a�{�_G[�]�_<`a�bK[�a&dgcK\G[Ndg{�[�d~}[�{�df[�]IT)}@[�{�a�bK[N]�[�}[�`X�K{�T@cK\G[�V�]�h��Zv�T{�[�]

���! #"%$'&)(+*-,. 0/1/32) #/3$'4-$�5�/32) 6,.798:(� 6/3$;/32) 6<=8?>� ?,@$'4/3A+BC8�(4D 0E.5F$�E3BC8:G)70 H5F$:EI 98�7J2K4LE3$'ML(� ?BN8:G)*OB- 0/.E3A�7HAP5Q"= ;RSGL 0"�AP/9TU <% ?7?8:&),. V$:56/32LA+,?WXB- 98:,.&E3 ?,Y(�A�RZ O[=\^]_/`>4)A�798�(+(P>!8�E3 CG)$�/&),. 9*a8�(+$:G) 'W@M)&/b8cE3 O&L,. 9*aA�Ga7?$'G�d�&)GL70/3A�$'Ga"6AP/32e$:/32) JEbB- 98cf

,.&LE3 ?,;,.&L7J2g8:,;hIi;[j/3$KA�GL,.&LE3 k/32)8�/�$'GL(P>lB-$S* ?(�,H"6AP/32gm'$�$S**LA�,.70E3A�B-A�G�8�/3A�$:Gon:p�q@m'$�$S*Y798:(�A�MLEr8c/3A+$:Gb8cE3 =,. ?(� ?70/3 9*�T�s=2)A�,�*$� ?,G)$:/OB- 98�G�[=\^]tA�,K8!4D$�$:EoB- J/.E3A+7Ku_AP/OA�,O 0v� 070/3A�wZ x8�/K*LA�,�f/3A+GLm'&LA+,.2LA�G)mK4D$�$�E3(P>l7?8:(�A�MLEr8�/3 ?*gB-$S*L ?(�,H5yE3$'Bz"% ?(�(@798�(�A+MEr8�/3 9*B-$S*L ?(�,?T

{�|L}�~F�Y�:�)���:�3�C�:�����?�9�k��?���3�?�X�F�����3�)�#�%�0���@����)�:�+�I�K�S��?�����:���1�:�r�b���L�D�^���D�:�.�3�'�t�:���3�L���J '�:�)�¡���:�@�1�L�.�?�k�L���#��0�+�0�0�3���'�¢ Z�=�?���?�.�+�:��K�S�L�?��£I¤Z�0�r�:�:���)�L L���r�:�r¥S�+�L�H¦6�P�3�V§I�?���3�?�.�.���'�¢ L£I¤Z�0�r�:�:���)�¡£������K�S��?���? )�����-¨����J¥S���)�;�3�)���%�?���#�K�S�L�?�D���¢£����-���L�D���F�3�:�-�:���3�)�y�J©

ª6«�¬¢­® ¯�°�° ±�²?° ®:±Z³ ´L«¢° ¯�µL´ ¶L­Lµ ´Dª6² ª#·�­ °L¯¢® ²J¯¢´ ª6­S¯�¸­¸�²Z¹#²?­®)¹ º¢»y¼)½�¾ º¢»y¼¿�¿ º¢»y¼�¼)À º¢»y¼�¼�Á º¢»y¼�Â�½ º¢»y¼DÁ)¼ º¢»y¼�Â)º º¢»y¼�Â�à Ä�¹ ÅÄSÆ º¢»y¼�¼)¾ º¢»y¼�¾�¼¶¯�ÇÈ­²0¯'ÉDÊ ÄD¹PÅË�Æ ÄD¹PÌSÅDÍ ÄD¹PÅÎSÅ ÄD¹PÅSÌLÏ Ä�¹ ÅLÎSÎ Ä�¹ ÅLÏ�Æ ÄD¹PÅÏ�Ä ÄD¹PÅÏSÅ Ä�¹ ÅÄÎ ÄD¹PÅ�Ð�Í ÄD¹PÅ�Ð̶L­L²J³ ÄD¹PÅË�Ì ÄD¹PÅDÍ�Å ÄD¹PÅÎÏ ÄD¹PÅSÌÌ Ä�¹ ÅLÏ�Å Ä�¹ ÅLÏ�Ì ÄD¹PÅDÍ�Å ÄD¹PÅ�ÐÐ º¢»y¼�ÀL¿ ÄD¹PÅË�Ð ÄD¹PÅ�ÐƯ:É�Ê ¯�®'® ÄD¹PÌSÑSÆ ÄD¹PÌSÄDÍ ÄD¹PÅSÌLË ÄD¹PÅSÌÌ Ä�¹ ÅLÎSË Ä�¹ ÅÆDÍ ÄD¹PÌËSÎ ÄD¹PÌSÄÅ Ä�¹ ÌÑË ÄD¹PÅDÍ:Æ ÄD¹PÌSÅSIJJ³Z¯D°�Ò ®'´ ÄD¹�ËSÎSÏ ÄD¹�ÎSÎSÎ ÄD¹PÌSÑLÏ ÄD¹�Î�ÅÅ Ä�¹PÎSÌSÆ Ä�¹ ÌSÐLÎ ÄD¹PÑSÑË Ó.ÄD¹PÅSÅSÄ Ó�ÄD¹PÄDÍÍ ÄD¹�Î�ÄLÏ ÄD¹ ÐÄSƲrÉȪ ÄD¹PÌDÍ�Ñ º¢»y¼LºD¼ ÄD¹PÅ�ÐLÌ ÄD¹PÅSÆLË Ä�¹ ÅÑSÑ Ä�¹ ÅÑSÌ º¢»yÂDÁ�Á ÄD¹PÌÎSÌ Ä�¹ ÌÌSÅ º¢»y¼)º�½ º¢»y¼)ºD½¯�¸�¸ ÄD¹PÌÎSÎ ÄD¹PÌÎSÏ ÄD¹PÅ�ÐLÅ ÄD¹PÅÏÏ Ä�¹ Å�Í:Î Ä�¹ Å�ÍcÐ ÄD¹PÌÏ�Ñ ÄD¹PÌSÆÑ º¢»y¼�þ ÄD¹PÌSÅÆ ÄD¹PÅSÄ˶¯DÊ¢Ó`¬�³ ÄD¹PÌDÍSÍ ÄD¹PÌSÆDÍ ÄD¹PÅ�Ð)Î ÄD¹PÅSÆLÎ Ä�¹ ÅSÐLË Ä�¹ ÅLËSË ÄD¹PÌÏ�Å º¢»yÂ�¼¿ Ä�¹PÎSÌSÆ ÄD¹PÅSÄSÐ ÄD¹PÌSÌSÌÒ�¸�¸ ÄD¹�ÎSÏ�Æ ÄD¹PÌ�ÐÆ ÄD¹PÅSÄÅ ÄD¹PÅSÑLÎ Ä�¹ ÌÌÏ Ä�¹ ÌÌSÅ ÄD¹�Î�ÆDÍ ÄD¹�Î�ÑLÏ Ä�¹ ÌLÎ�Æ ÄD¹PÌ�Ð)Î ÄD¹PÌ�ÐSжL²0³�Ó`¬�³ º¢»yÂ)¼)º ÄD¹PÌSÅSÅ º¢»y¼�½�Á º¢»y¼DÁL º¢»y¼�¾)º º¢»y¼L¿�Ô ÄD¹PÆSÄÎ ÄD¹PÆDÍÍ Ä�¹ Ð�Í�Ñ ÄD¹PÌÎDÍ ÄD¹PÌSÄSƬ�³ ÄD¹�ÏSË�Æ ÄD¹�Î�ÌSÅ ÄD¹PÌÏSÄ ÄD¹PÌSÆÌ Ä�¹PÎSÆÎ Ä�¹PÎSÅÏ ÄD¹�ÏSÏ�Æ ÄD¹PÆË�Ð Ä�¹PÎË�Ä ÄD¹�ÎZÐ)Ï ÄD¹�ÎSËS˶L²0³�Ó`²0³Dª6µ ÄD¹�Î�ÑË ÄD¹�Î�ÅSÄ ÄD¹PÅSÄÆ ÄD¹PÅDÍ:Å Ä�¹ ÌÆDÍ Ä�¹ ÌÑ�Ð ÄD¹PÑSÄ�Ð ÄD¹�Ë�ÌÆ Ä�¹ ÑÌSÅ ÄD¹PÆÏSÅ ÄD¹PÆSÆSÅ

Õ�Ö:×SØHÙHÚ�Õ�ÛDÚFÜ�×�Ý0Ö^ÝJÞ�×SÝI×YßoàDÛDÖ:áDâ¢Ö:ØrãäàLØJßæå#Ücá�àLåJÖ�ÝJàkÝJÞ�Ö;çÈÖ:årÝâÈÖ�ØJãäàØ0ßæ×SÕ�Ü�Ö!àç�åJÖ�Ø0èÖ'Û�àÕ ×áyá�åJÖ�èLÖ�Õtâ�ØJàLç�á�Ö�ßæå�éëê¡Ö�ì)×�íÝ0ÚyèLÖ;Ö�Õ)ÝJØ0Ú�Ö:å#ßoÖ:×ÕKÝJÞ�Ö�çÈÖ:årÝIßoàDÛDÖ�áFå#âÈÖ�ØJãäàØ0ßîçÈÖ�á�à�ïjç¢×åJÖcíá�Ú�Õ�Öé#ð^àáFÛVÖ:Õ)ÝJØ0ÚyÖ'å#ÚyÕKÝ0Þ�Ö;ç¢àÝrÝ0àßzàSãQñ@×ç�áyÖbÙ�åJÞ�à�ïòï;Þ�Ú�Ü?Þá�Ö:×ØJÕ�ÚyÕ�ìY×Sá�ìàØ0ÚyÝJÞ�ßæå�ó�Ú�Ö�áFÛbÝJÞ�Ö�çÈÖ:årÝ%âÈÖ�ØJãäàØ0ßæ×SÕ�Ü�Ö6àÕVÖ:×Ü?ÞßoÖ�ÝJØ0Ú�Üé=ñHÞ�ÖYáF×årÝHÜ�àá�ô�ßoÕxÚFå�ÝJÞ�ÖkßoÖ:×SÕlÕ�àØ0ßæ×Sá�Úyõ:Ö:Ûæå0ÜcàLØJÖ×LÜcØ0àLå0å�ÝJÞ�ÖbÝ0Ö�Õ!ßoÖcÝJØ0ÚFÜ�å:éö ìLÕ�àØ0Ú�Õ�ì!ÝJÞ�ÖlÖ�Õ�åJÖ�ßOç�á�Ö÷ßoÖcÝJÞ�à�Û�åVÚ�Õ�ÝJÞ�Ö÷ÝJàLâ�àSã;Ý0Þ�Ö÷Ý?×�íç�áyÖLøoÝJÞ�Ö çÈÖ:årÝ�×Sá�ìàØ0ÚyÝJÞ�ßæåùà�èÖ:Ø0×áyáo×SØ0ÖûúDüYýeå:ø÷þ¡êYê¡å:ø×Õ�Ûlç�×SìLìÖ:Û÷ÝJØ0Ö�Ö'å�øDï;ÚyÝJÞgßoÖ:×SÕgÕ�àLØJßæ×Sá�Ú�õ�Ö:ÛlåJÜ�àØ0Ö:åHÿ�é �ÿ��Døÿ�é �Lÿ���ø=×Õ�Û�ÿ�é ������ø=Ø0Ö:åJâ¢Ö'Ü9Ý0ÚyèLÖ�á�óé�ð^à)à)å3Ý0Ö:Û Ý0ØJÖ:Ö:åKÛDà_Õ�àSÝâÈÖ�ØJãäàØ0ß ï^Ö�á�á1àLÕ!ÝJÞ�ÖVâ�ØJàLç�×Sç�Úyá�Ú+Ý3ólßoÖcÝJØ0ÚFÜ�å���ý_úXøXý��� ¡ø×Õ�Û��^þ����9øÈç�ôDÝY×ØJÖVÖ���ÜcÖ:áyá�Ö�Õ)Ý�àÕ!Ý0Þ�ÖCÝ0Þ�Ø0Ö:åJÞ�àáFÛgßKÖ�ÝJØ0Ú�Ü:å��þ����Yø��#ú��YøI×Õ�Û����=ñ��o×SÕ�ÛjàØ?ÛDÖ�Ø0Ú�Õ�ì ßoÖ�ÝJØ0Ú�Ü:å����� �Yøþ�!�-øQ×Õ�Ûeð" �!#�9é ö ãIßoÖ:×SÕ â¢Ö:ØrãäàLØJßæ×Õ�ÜcÖCïH×åYßoÖ:×Lårô�Ø0Ö:Ûà�èLÖ�Ø�$3ô¢å3ÝCÝ0Þ�Ö:åJÖlåJÚ%�ùßoÖcÝ0ØJÚFÜ�å:ø=ç¢à�à)å3Ý0Ö:ÛùÝJØ0Ö�Ö'åCï^àô�áFÛùàôDÝJíâÈÖ�ØJãäàØ0ß ç�×ììLÖ:ÛæÝ0ØJÖ:Ö:å:ø¢ú�ükýeå�ø¢×SÕ�Ûgþ¡êYê¡å:éñHÞ�ÖHÝJàâoàãXñ%×Sç�á�Ö-Ù�åJÞ�à�ï�å%Õ�àØ0ßæ×Sá�Úyõ:Ö:ÛoåJÜ�àØ0Ö:å@ãäàØ#Ö:Õ�åJÖ�ßKíç�áyÖkårÖ:áyÖ'Ü9ÝJÚ�àÕ1ø)ðH×ZóÖ'årÚF×SÕKßoàDÛDÖ�áX×ZèÖ:Ø0×ìÚ�Õ�ì�øZÝ0Þ�Ö¡çÈÖ:årÝ�Ú�Õ�ÛDÚyíè�ÚFÛDô�×áDßKàDÛDÖ:á�å=àãÈ×Õ)óCÝ3ó)âÈÖ ��ð" �ú�ñ��9ø×kårÚ�ßoâ�á�Ö�×ZèÖ�Ø?×SìLÖ6àSã×áyá@ßoà�Û�Ö�áFå&��þ1ü�' þ��(���cøÈ×Õ�ÛaårÝ0×Ü*)�Ú�Õ�ìæï;ÚyÝJÞeá�àìLÚ�årÝJÚFÜ-Ø0ÖcíìLØJÖ'åJåJÚ�àÕ+��ú�ñ@þ��#, ����9é�ú)Ý?×Ü*)�Ú�Õ�ì-��. àLáyâÈÖ�ØJÝ:øQÙ/�������#ï;ÚyÝJÞá�àìLÚ�årÝJÚFÜ-Ø0Ö�ìLØJÖ'åJåJÚyàLÕ!â¢Ö:ØrãäàLØJßæå�âÈà�àØ0áyó!çÈÖ:Ü:×Sô�åJÖCØ0Ö�ìØ0Ö:å0åJÚyàLÕà�èLÖ�Ø10�Ý0åkÛDØ?×Sßæ×�Ý0Ú�Ü:×Sá�áyó!ï;Þ�Ö:Õ_Ý0Þ�Ö�Ø0Öo×SØ0Ö2�SÿLÿÿxÞ�ÚyìLÞ�á�óeÜcàØJíØ0Ö�áF×�Ý0Ö:Û�Ú�Õ�â�ô�ÝKßoàDÛDÖ�áFåo×SÕ�Û�àÕ�á�óûÙ3) â¢àLÚyÕ)Ý?åOÚ�Õ�Ý0Þ�Öxè�×áyÚyíÛ�×�ÝJÚ�àÕ�åJÖcÝ'éxþ�Õ�ô�Õ�ï^Ö�Ú�ìÞ)ÝJÖ'Û ×ZèLÖ�Ø?×SìLÖOàãH×áyá#Ý0Þ�ÖæßoàDÛDÖ�áFå��þ1ü�' þ��(���CÚFåKç¢Ö�ÝrÝJÖ:ØVÝ0Þ�×SÕjï�Ö:ÚyìLÞ)ÝJÚ�Õ�ìaÝJÞ�ÖlßoàDÛDÖ�áFåKè�Ú�×Ø0Ö�ìLØJÖ'åJåJÚ�àÕ÷çÈÖ:Ü:×Sô�åJÖ-ô�Õ�ï�Ö:ÚyìLÞ)ÝJÖ:Ûg×ZèÖ:Ø0×ìÚ�Õ�ìOÜ:×SÕ�Õ�àSÝ;à�èÖ:Ørí0�ÝKÝ0Þ�Ö!è�×Sá�Ú�Û�×SÝJÚ�àÕtårÖ�Ý:éòþ¡èÖ�Ø?×SìLÚyÕ�ì�ø#Þ�à�ï^Ö�èLÖ�Ø'ø#â¢Ö:ØrãäàLØJßæåï^àØ?åJÖ�Ý0Þ�×SÕ4$3ô�årÝIâ�ÚFÜ*)�ÚyÕ�ìbÝJÞ�Ö�çÈÖ:årÝIåJÚyÕ�ìLáyÖ;ßoàDÛDÖ:á�å#çÈÖ:Ü:×Sô�åJÖåJàßoÖæàã�Ý0Þ�Ö÷ßoàDÛDÖ�áFåCÞ�×ZèLÖoâ¢à�àØVâ¢Ö:ØrãäàLØJßæ×SÕ¢ÜcÖ÷×SÕ¢Û_Ý0Þ�ô�åÞ�ô�ØJÝ!ÝJÞ�Ö�Ö:Õ�åJÖ�ßVç�áyÖ'å�é þ¡å!Ö3��âÈÖ:ÜcÝJÖ'ÛQøbâ�Ú�Ü*)�Ú�Õ�ìtÝJÞ�Ö�çÈÖ:årÝåJÚ�Õ�ìá�ÖxßoàDÛDÖ�á^ãäØ0àß ÝJÞ�Ögá�Ú�ç�Ø0×ØJó5��ð" ^úDñ��CâÈÖ�ØJãäàØ0ßæåVçÈÖcÝJíÝ0Ö�ØYÝJÞ�×Õ_×SÕ�ógàÕ�ÖOáyÖ'×SØ0Õ�Ú�Õ�ì÷×Sá�ìàLØJÚyÝJÞ�ß Ú�ÕeÝJÞ�ÖVçÈàSÝrÝ0àß àSã

Ý0Þ�ÖkÝ0×ç�áyÖkçÈÖ:Ü�×ô�åJÖYÚyÝ�Ü�×SÕlâ�ÚFÜ*)oÝJÞ�ÖbçÈÖ:årÝHßoàDÛDÖ:áXãäØJàLß ×SÕ�óá�Ö:×ØJÕ�ÚyÕ�ìKßoÖcÝ0Þ�àDÛgãäàØ;Ö:×LÜ?ÞxßoÖcÝ0ØJÚFÜ-×SÕ¢Û÷Ý0Ö:årÝ�â�ØJàLç�á�Ö�ß!éðH×ZóLÖ:åJÚ�×ÕîßoàDÛDÖ�áx×ZèLÖ�Ø?×SìLÚyÕ�ì6��7YàßoÚ�Õ�ìà)å�ø8�ÿÿÿ��eÞ¢×å�×ßoÖ'×SÕûåJÜ�àØ0ÖxàSãbÿ�é �:9���ø6åJÚyìLÕ�Ú;0ÈÜ�×SÕ)Ý0áyó�çÈÖcÝJÝJÖ�ØoÝ0Þ�×SÕ ú�ükýeå�äÝJÞ�Ö_ç¢Ö'å3ÝlåJÚ�Õ�ìá�Ö!ßKàDÛDÖ:á<�cé ö ÕëðH×ZóÖ'årÚF×SÕjßoà�Û�Ö�á¡×ZèLÖ�Ø?×SìSíÚ�Õ�ì¢øÖ:×LÜ?ÞoßKàDÛDÖ:á�Ú�ÕoÝJÞ�Ö�á�Ú�ç�Ø0×ØJóVÚ�å#ï^Ö�Ú�ìÞ)Ý0Ö:ÛKç�óCÝJÞ�Ö�áyÚ=)Ö:áyÚyíÞ�à)àDÛVàSã�Ý0Þ�Ö^Û�×�Ý0×2�äè�×áyÚFÛ�×�Ý0ÚyàLÕOåJÖcÝ>��ìLÚyèLÖ�Õ-Ý0Þ�ÖHßKàDÛDÖ:á�ï;Þ�Ö�ÕßoàDÛDÖ:á�â�Ø0Ö:ÛDÚFÜ9Ý0ÚyàLÕ�å;×SØ0Ö¡Ý0ØJÖ'×�Ý0Ö:Û!×åHâ�Ø0àç¢×Sç�Ú�áyÚyÝJÚ�Ö:å@?ACBEDGF�HI%J�KML�N�O/P I�QSR OUTWV IYX �3Ù[Z L�N�O/P I � Q �3Ù�Z R OUTWV I �ï;Þ�Ö�Ø0Ö A B ÚFå6Ý0Þ�Ö¡ï^Ö�Ú�ìÞ)ÝIàã�ßoà�Û�Ö�áM\zø L�N�O/P I ÚFå#Ý0Þ�ÖYÿ^]�ÙÝ?×SØ0ìÖ�Ý^ãäàLØ�Ü�×LårÖ`_?ø�×SÕ�Û R OUTWV I Ú�å^Ý0Þ�ÖCâ�Ø0Ö:ÛDÚFÜ9Ý0Ö:Ûlâ�Ø0àç�×ç�ÚyáyíÚyÝ3óùÝJÞ�×SÝKÜ:×åJÖa_YÚFåKÜcáF×å0å÷Ùé ê�àSÝ0Ö÷Ý0Þ�×�Ýoð^×ZóLÖ:åJÚ�×Õ ßKàDÛDÖ:á×ZèLÖ�Ø?×SìLÚyÕ�ìbàLôDÝJâÈÖ�ØJãäàØ0ßæå�ð" �ú�ñûàÕxàLÕ�á�ób�CàSãQÝ0Þ�ÖOÙ'ÿCßoÖcÝJíØ0ÚFÜ�å:é. ÚyÝJÞ ×÷ßoÖ'×SÕ_Õ�àØ0ßo×áyÚ�õ�Ö'Ûeå0ÜcàØ0ÖCàã�ÿ�é ��c���øÈÖ�Õ�åJÖ�ßOç�áyÖoåJÖcíá�Ö:ÜcÝJÚ�àÕ�ÚFå-Ý0Þ�ÖxÜcá�Ö:×ØCï;Ú�Õ�Õ�Ö�Ø'é ö ÝVàô�ÝJâÈÖ�ØJãäàØ0ßoå-ÝJÞ�ÖlàSÝJÞ�Ö�ØÖ:Õ�åJÖ�ßVç�áyÖ^ßoÖcÝJÞ�à�Û�å���×SÕ¢ÛCÝ0Þ�ÖHç¢Ö'å3Ý#Ú�Õ�ÛDÚ�è)ÚFÛDô�×áDßoà�Û�Ö�áFåd�1àÕ×áyáIÙ:ÿKàSã%ÝJÞ�ÖæÙ'ÿoßoÖcÝJØ0ÚFÜ�å ��åJÚ�ìÕ�Ú%0¢Ü�×Õ)Ý¡×�Ý[e D ÿgf ÿÿ�Ù/�9éh.ùÖçÈÖ�á�Ú�Ö�èÖ Ö�Õ¢årÖ:ßVç�á�Ö åJÖ�á�Ö:ÜcÝJÚ�àÕ Ü�àÕ�åJÚFå3Ý0Ö�Õ)ÝJá�ó àôDÝ0â¢Ö:ØrãäàLØJßæåÝ0Þ�Ö-àSÝ0Þ�Ö�Ø;Ö:Õ�åJÖ�ßVç�áyÖbßoÖcÝ0Þ�àDÛ�åHãäàØHÝ3ï^àoØJÖ'×åJàÕ�å@?i�Ö�Õ�åJÖ�ßOç�áyÖCåJÖ�á�Ö:ÜcÝJÚ�àÕ!ÚFå�×Sç�á�ÖbÝJàæàLâDÝJÚ�ßoÚyõ:ÖbÝJÞ�Ö-Ö:Õ�åJÖ�ßKíç�áyÖCÛ�Ú;jXÖ�Ø0Ö�Õ)ÝJá�óæãäàLØ;Ö:×Ü?ÞgâÈÖ�ØJãäàØ0ßo×Õ�ÜcÖbßoÖcÝ0ØJÚFÜ

i�à�èÖ�Ø10�ÝJÝJÚ�Õ�ìòÚFå!×jåJÖ�Ø0ÚyàLô�ålâ�ØJàLç�á�Ö�ß ï;Þ�Ö:ÕëÝJÞ�Ö�Ø0Öù×ØJÖ�SÿÿLÿVßoàDÛDÖ:á�åHÝ0àæÜcàßOç�Ú�Õ�ÖéIñHÞ�Ö-ßKÖ�ÝJÞ�àDÛ�å�â�ØJÖ'årÖ:Õ)ÝJÖ:ÛÚyÕ ú�Ö:ÜcÝJÚ�àÕ5�aÝ0àùÜ�àßVç¢×�ÝOà�èLÖ�Ø10�ÝrÝ0ÚyÕ�ì ÜcàLÕ)ÝJØ0Úyç�ô�ÝJÖlÝJàÖ�Õ�åJÖ�ßOç�áyÖ-åJÖ�á�Ö:ÜcÝJÚ�àÕlk åHÖ���ÜcÖ:áyá�Ö�Õ)Ý�âÈÖ�ØJãäàØ0ßæ×SÕ�Ü�Öé

ñHÞ�Ö�ÜcàLÕ�åJÚ�årÝJÖ:ÕLÝ0áyóbårÝJØ0àÕ�ì�âÈÖ�ØJãäàØ0ßæ×SÕ�Ü�Ö6àSã�Ö:Õ�åJÖ�ßVç�áyÖ^årÖ:áyÖ'Ü9íÝ0ÚyàLÕ÷årô�ììÖ'å3Ý?å%ÝJÞ�×SÝIô�åJÚyÕ�ìCßæ×SÕ�óOÛDÚ%jXÖ�Ø0Ö�Õ)ÝIá�Ö:×SØ0Õ�Ú�Õ�ì-ßoÖcÝ0ÞDíàDÛ�å¡×SÕ�Ûlâ�×SØ?×SßoÖ�ÝJÖ�Ø;åJÖcÝJÝJÚ�Õ�ìLå;ÚFå;×SÕ!Ö3jÈÖ'Ü9ÝJÚ�èÖ-ïH×ZóæàSã=ìÖ:ÕDíÖ:Ø0×SÝJÚ�Õ�ìx×÷Û�ÚyèLÖ�Ø?årÖCÜ�àá�áyÖ'Ü9ÝJÚ�àÕaàã#ßKàDÛDÖ:á�å:é¡ñHÞ�ÖKÜcàLÕ�årÚFårÝJÖ�Õ)ÝâÈÖ�ØJãäàØ0ßæ×SÕ�Ü�Ö�×SáFåràCåJô�ìLìÖ:årÝ0å#ÝJÞ�×SÝ6ãäàØ0ï^×Ø0ÛKårÝJÖ�â�ï;ÚFårÖYårÖ:áyÖ'Ü9íÝ0ÚyàLÕ ÚFåb×SÕ Ö�jXÖ:Ü9Ý0ÚyèLÖoï^×Zó!àSãHårÖ:áyÖ'Ü9Ý0ÚyÕ�ìgÞ�Ú�ìÞDí�âÈÖ�ØJãäàØ0ßo×Õ�ÜcÖ

m@n�o1m3prq�s%m/o�tvudw�pyxdzgm@o1m8p{wY|Ym3s<o&tvw�u{}�~U}�u1�=m�x��Cw:t���m3u1tvw�u1�pb}�n���m`p{m�xdu1�<�3o��%tSwU~�m3u1�gx1x1�=ng�&�<o��3}�u1m3tv�gs=s%�8�3w�n�x1udw�s=s=m@|M��l�<�l�������W���^���8�8�^�������@�����5���5�����/�������W�U���: ¡ nCxdzg�=o&o�m/�>xd�%w�nC¢#ma��w�p{��}:udm{m3n�o1m3p&qgs%mao1m3s=m@��x1�=w�nCx1w�xdzgmq�m@o�xao��=ng��s%m-p2wY|Ym@s=o�£¤#¥#¦Y§¨�=n©§�}:q�s%m«ª/¬r��o��=ng�C��m@ud�3m3n�xudm@|g���>xd�%w�n­�%n®m@u1udw�u/�«¯�ngm8}�|Y~U}:n�xd}���mbw�t���m@ud�3m3n�x&udm@|Y���>�xd�%w�n°�=n°m3udu1w�u8��w�p{��}:udm@|�x1w�ngw�udpb}:s=�%±@m@|²od��w�u1m/ob�=o8x1z�}Uxn�w�udp{}�s%�=±3m/|«od��w�u1m/o`�*z�}:ng��mb}�o`q�m�x�xdm3u&p2wY|Ym@s=or}:udm&tvw��gn�|}�n�|+o1zg�%t³x�xdzgm�xdw��+w�thx1zgm{o1�@}:s=m��g�l�`§"zgm&��m@ud�3m3n�x�udm@|Y���>�xd�%w�n+�=n+m3udu1w�u�tvudw�p´p2wY|Ym@s�µ¶x1w·¤G|gm3��m3n�|go�w�n�s%�·w�n�xdzgm��m3u1tvw�udpb}:n��3m@o4w:t"p{wY|Ym3s<o�µ¨}:n�|C¤4¸Mngw�x�w�n¹o�w�p{mbw:x1z�m3up{wY|Ym@sMx1z�}:x�|Ym���ngm@o#xdzgm`x1w��·w�t(xdzgm ��m@u�tvw�u1pb}�n���m od�3}:s=m��§�w��@}:s<���gs<}Uxdmh��m@ud�3m3n�x(udm@|Y����x1�=w�n �=n�m3ududw�u/¸3¢#mh��u*o�x��3w�n^~�m@u�xm/}��*z+p{m3x1ud�=��x1w8s=w�odo�¢�zgm3udm&º8u1m@�gu1m/o�m@n�xdo���m3u1tvm@��x4��m3u1tvw�u1�pb}�n���m+}�n�|»ª�udm3��u1m/o�m@n�x*o2¢¼w�udo�xb��m@u�tvw�u1pb}:n���m��°½Sm@u�tvm/�>x��u1m/|Y�=��x1�=w�n©�^�%m@s=|gob¾4¿ ¿ÁÀÃÂrÄ�ÅÆÀÈÇ�É[¿ÁÀÃÊ Ë ¿yÀ¾�Å ÊÌÀÍÉh¾�ÊÎÀ6ªr}�n�|�Ê ÏÍÉhÄ6ÀÍÏGЫÄ6ÀÍ¿`¾�Ñ5ÀÒº�¸}�s%s�w:t�¢�zg�<�*z²z�}W~�m�s%w�o1o-ºg�¶µ�n»m�Óg}:p{�gs=m+¢��=s%s4zgm3s=�l� ¡ tµ�Ô�ÔÎ�=p{�gudwU~�m@o tvu1w�pÕº��×Ö�Ø-x1w«º��×Ö�Ö^¸�x1zgm-s%w�o1o1m@o&}:udmaºg�×Ù�Ø}�n�|²ºg�×Ù:Ú�¸¼udm@o1��m@�>xd�%~�m3s=��¸"}:n�|©xdzgm+��m3u*��m@n�x-u1m/|Y���>xd�%w�n5�%nm@u1udw�u8�=o·Û�Ü8�6¯�ngm���w�x1m3n�xd�=}�s4|Y�<od}�|Y~U}:n�x*}:��m�w�t���m@ud�3m3n�xudm@|g���>xd�%w�n��%n�s=w�odo��<o�x1z�}Ux��%x����%~�m@o"p{w�udm�m3p{�gz�}�o1�=o"xdwaudm��|g���>xd�%w�n�or}:x�s%wU¢6s%w�o1o@Ý�udm@|Y�����=ng��s%w�o1o tvu1w�pÕº�� º�Ùaxdw+ºg� ºgª�<o�}aØ�º�ÜÞudm@|g���>xd�%w�nl¸g¢�zgm@ud}�o�u1m/|Y���3�%ng�{s=w�odo"tvudw�pEº�� Ù�º&x1wº��%ª/ß«£o1}�p{m{}�q�o1w�s=�Yx1mb�*z�}�ng��mW¬>¸M�<o w�ngs=�+}�Ø�Ü´udm@|g���>xd�%w�nl�¡ n�o1w�p{m�|Yw�p{}��%n�o#x1zg�<o�qg�<}�o��<o�}��g�gudw��gud�=}:x1m�� ¡ n�w�x1zgm@udo��%x�<o"ngw�x@�[à�w�u1pb}�s%�=±3m/|-od��w�u1m/o#|gw2n�w:x�z�}W~�m�xdzg�=o�qg�=}�o3�§�}:q�s%m`Ù o�z�wU¢�ohx1z�m���m3u*��m3n�x#u1m/|Y���>xd�%w�nb�%n8s%w�o1o[tvw�u�m@n�o1m3p2�q�s%m�o�m@s%m/�>xd�%w�náw�n®x1zgm�Ö·xdm@o�x2��u1w�qgs%m@pbo2}�n�|²ª/º+p{m�xdu1�<�3o@¸�3w�p{��}�u1m/|`x1w4x1z�m¼q�m@o�xhp2wY|Ym@s=oho1m3s=m@��x1m/| tvw�uSm/}��*zr��u1w�qgs%m@p}�n�|­p{m�xdu1�<�:�Cµ�o&¢��;xdz®ngw�u1pb}:s=�=±3m@|­od��w�u1m/o3¸���n�}�s¼��m3u1tvw�u1�pb}�n���m/o�}:udm`m@o�x1�=p{}:x1m/|-w�n�s=}�u1��m���n�}:sMxdm@o�x�o�m3xdo�n�w:x���o1m@|tvw�u(xdud}��%n��%ng��}:n�|r�=o(x1zgm"}W~�m@ud}���mSwU~�m@ulx�¢#w�x1ud�<}:s<o(¢��%x1z&m@}��*z��u1w�qgs%m@p��#½Sw�o��%x1�=~�m`m3n�xdu1�=m@o��=n�x1z�m xd}�qgs%mrp2m/}:n�m3ududw�u�¢"}�oudm@|g����m/|M¸Y�� m��hm@n�o1m3prq�s%m�o1m3s=m@��x1�=w�n·��m@u�tvw�u1p{m@|-q�m�x�xdm3u/�¥�n�o1m3prq�s%m�o1m3s=m@��x1�=w�n�¢��=n�o�â�ãbw:t¼ÖUº2xd�%p{m@o2£�o1�=��ng�%���3}�n�x�}Uxä ÀȺgå º�º�ª/¬��5¯�n�}W~�m3u*}:��m�¸hm3n�o�m@prqgs=m·o1m3s=m@��x1�=w�n©u1m/|Y���3m@os=w�odo�Ûg�æÖUâ^ÜÎwU~�m@u¼xdzgm`q�m/o�x�p{wY|Ym@s=o"tvw�u�m@}��*z8�gudw�qgs=m3pE}:n�|p{m3x1ud�=���r§(w·pb}:ç�mrxdzg�=o �3w�n��3u1m3x1m�¸��%t�xdzgm{q�m/o�x p{wY|Ym3shz�}�o}��3�3�gud}����¹ß�º�� º�º�Ü8¸Ss%w�o1o&�=o2ºg�=ª@º�º�º�¸S}�n�|®}�n�Ûg�æÖ�ÜÈudm@|Y���>�xd�%w�n«�=nCs%w�o1o`��w�u1udm@o1��w�n�|go�x1w�udm@|Y�����=ng�-s=w�odo4x1w�ºg� º�ß�ª@Ú8w�u�=n��3u1m/}�o1�%ng�+}��3�3�gud}�����xdw�ß�º�� Û^Ö�Ü8�·¦Y�%p{�=s=}�u1s=��¸S�%t#xdzgm8q�m@o�xp{wY|Ym@s�z�}�o#è�é�¦aº�� Ù�º�º�ºg¸�}�n-Ûg�æÖ�ÜÌudm@|Y���>x1�=w�n8�%n-s%w�o1o¼��w�u1�udm@o1��w�n�|go(x1w`udm@|Y�����=ng��è�髦 xdw4º��%ª/Û�Ù:â��l§(m/o�x[o1m�x*o��3w�n�xd}��%nÙ�ºg¸ º�º�º«�@}�o1m@o@� ¡ n��3u1m/}�o1�%n��¹}��3�3�gud}����áº�� Û^Ö�ÜÕp{m/}:n�o�ªUÖWãp{w�u1m �3}�o�m/o"}:udm4��u1m/|Y�=��x1m/|-�3w�udu1m/�>x1s=�aq^�bxdzgm m3n�o1m3p&qgs=m@o@�§"z�m��%p{�gudwU~�m@p{m3n�xh�=o�ngw:x¼|gud}�p{}:x1�<�:¸�qg�Yx¼�3w�n�o1�=o�x1m@n�x1s=�r�=nY��3u1m/}�o1�%n���}��@���gu*}��3� ºg� Û�Ö�Üêw�uSu1m/|Y�����=ng�`è�é�¦&ºg� ºgªUÖWã��3w�p2���}:udm@|«x1w�xdzgmaq�m@o�xrw:t�Ù:º�º�º·p{wY|Ym3s<o �<o �%p{�gudm@odo1�%~�mb¢�zgm3nxdzgm q�m@o�x�p2wY|Ym@s=o�z�}W~�m`o����*z-��w�wY|·��m3u1tvw�udp{}�n���m���¤#�Yx�¢#m

2

4

6

8

10

12

0 0.2 0.4 0.6 0.8 1

Mea

n Pe

rcen

t Adv

anta

ge O

ver B

est S

ingle

Mod

el

Fraction of Models in Bagged Samples

APRROCALL

MXELFTBEPRMSFSCACC

ë�ìæí�î/ï1ð"ñ:ò^ó�ô*õ�ôdö�÷`ø@ù�ó�ú@û/û@ô>ü­ý:ô*þæô*ÿd÷��æø/õ«ø/õ«÷���ô������2ôd÷��æÿ���� ���ô���æû��U÷���;ü�ô[ø3ù�÷���ôhû��ú����rú3÷��������ô����ô�� ô*õU÷��(õ�ø��^ú3û/û��;õ�û��

|gw�n�� x�¢"}:n�x�xdwawU~�m@udo�xd}:x1m`x1z�mr�=p2��u1wU~�m3p{m3n�x�w:thm@n�o�m@prqgs=mo1m3s=m@��x1�=w�nl���gw�ur��w�p{��}:ud�=o1w�n(¸lq�}:�����=ng��w�u q�w^w�o�x1�=ng��xdu1m@m@o�^�=m3s<|go�}aÙ�º�ܨu1m/|Y���>xd�%w�n��%n�s=w�odo���w�p{��}:udm@|-x1wbu*}W¢êx1udm3m@o@¸Ùg� Ø&xd�%p{m/o�x1z�m�q�m@ngm���x�w:thm3n�o�m@prqgs=m�o�m@s%m/�>xd�%w�nl�#¤#�Yx�x1zgm@nl¸�=p{�gudwU~��=ng� xdzgm4��m@u�tvw�u1pb}:n���m�w�tlp2m/|Y�=w^�3u1m�¸�z��%��za~U}:ud�=}�n���mp{wY|Ym@s=o�o1���*z�}�o"|Ym@�3�=o1�=w�naxdu1m@m@o"�<o"m@}�o��=m3u¼x1z�}�n-�=p{�gudwU~��=ng�xdzgm ��m3u1tvw�udp{}�n���m`w�t(x1z�m ~�m3ud�bq�m@o�x�p{w^|gm3s<o3��l�! (�#"%$'&(&���#&C�/�*)¶�����,+®�.-��0/21����43��3�/���5&���=���gudm�Ù o1zgwU¢�o[x1zgm���m@ud�3m3n�x¼udm@|Y���>x1�=w�nb�=nas=w�odo[w�t�m@n�o1m3p2�q�s%m o1m3s=m@�>xd�%w�n8¢��%x1z·q�}:�����=ng�&}�o ä ¸^x1zgm4tvu*}���x1�=w�n8w�t(p{wY|Ym3s<o�=n�xdzgm�q�}:��o@¸W~U}:ud�%m/oltvudw�pÞª�|YwU¢�n�x1w�º�å º�Ù�Ø#tvw�u(xdzgm4ª@º�p{m�x1�ud�<�3o�}W~�m3u*}:��m/|�}���udw�odo�xdzgm·Ö-�gudw�q�s%m@p{o@�-µ#x ä À ª�¸�q�}���o�3w�n�xd}��%n·}:s=sMp{w^|gm3s<o¼¢�z��=�*z·�=o"m�6��g�%~U}�s%m@n�x#x1w{ngw�x�q�}:�����=ng���¯�na}W~�m3u*}:��m�¸Uq�}�����m/|2m3n�o1m3p&qgs%m�o�m@s%m/�>x1�=w�nbudm@|g����m/o[s=w�odo[}:n}�|g|Y�%x1�=w�n�}�s^Ùg� Ø�Ü°}:x ä À°ºgå×ØY��§"zgm�udm@o1�gs%xdo��%n2§�}�qgs%m/o[ª#}:n�|Ùr}:udm�tvw�u ä À²º�å Øg¸�}r~U}:s=�gm�¢#m�o1m3s=m@��x1m@|aq�m3tvw�udm�s=w�w�ç^�%ng�&}Uxxdzgm@o1m�udm@o1�gs%xdo@¸�}:n�|{¢�zg�=�*z7���=���gudm�Ù`o1zgwU¢�oh�<o�o1�gq�w��Yx1�=pb}:s �§"z��=o�Ùg� Ø�ÜÒ�<o[}:q�w��gx�}4x1zg�=ud|2w�t�xdzgm�x1w�xd}�s�Ûg�æÖ�ÜGq�m3n�m��gx�¢#mo1m3m ¢��%x1z�m3n�o1m3p&qgs%m�o1m3s=m@��x1�=w�nl�8���%���gudmrÙ�£v¢�zg�<�*z���o1m@o�xdzgm��n�}:s�xdm@o�x�o1m�x*od¬"o1�g����m@o�xdo ä �%n·xdzgm`ud}�ng��m`ºgå=ª:9Cºgå Ú2¢¼w��gs=|�^�=m3s<|®tv��u�xdzgm3ub�=p{�gu1wU~�m3p{m3n�x/�<;Cm+}:udm·�3�gu1udm3n�xds%�­m3Ó^��m3u1��=p{m3n�x1�=ng�©¢��;xdzê��o��=ng���3u1w�o1o-~U}:s=�=|�}Ux1�=w�n»x1w©�g�<�*ç5}�ngm/}:uw��Yx1�=pb}:sM~U}:s=�gm w�t ä tvw�u�m/}��*z·�gudw�qgs=m3p }�n�|-p2m3x1ud�=���

=�>�?A@�BDCFE:G'HJI�KMLN?AOP@�BQBQR�STKURPVXWYE�HUZ�[]\*G_^a`bRPcd @�e_G'RPcfOPCgB

h m3udm�¢#m`�gudm@o1m3n�x"}2�3}�o�m`o�x1��|Y�8¢�zgm3udm�w��Yx1�=p{�%±@�%n��{}�n8m@nY�o1m3p&qgs=m"x1w4x1zgm���w�ududm@�>xSp{m�xdu1�<�"z�}�o[�%p{��}��>x[w�n{}:nb}��g�gs=�=�@}U�

iQj�kmlon8prqQsutTv�wTt�x]y�z:t|{�}DwTy�~���x%~�x�����������g��xD�t��J�D��tU��tT��tTwTy�~���x7���4t�v�y��Dt��5tT��y��%�r{�t�������v������:x]����t��mv�xD~�x��M�:�����mv�~�y��D�2�

�������� �¡�¢ £_¤D¤ ¥]¦�¤  �¥4§ �D�¨¤ £���� �D¡�� �_¢a¦ ¢�©¨¡ ¤D£¨  ¦T£¨� ¢�¡�£¨ª

£�«¨¬' �§ ­_®�¯r­ °�® ±�² ¯'®�³r² ´_® µ�¶ ´_® ²�³ ­_®�¯r¶ ¶_®�·Q° °r®�­r¯ ±_®�­r¶ µ'®�¶�³ µ'®�³Q°¤Q�D¸�¡�� §D¹���¡ º¼»4½ ¾�¿ º�À�½o»�Á µ¨® ¯�­ º�À�½�¿� ­'®�´�³ º¼»4½ÃÁ� ­_®�·]± ¯_®�¯r¯ °r® µ'° µ'®�¶�´ ³Q®�¶]µ�³ �¡r§Q§_¡��f® ��° °�·Q® µ�· ³]¯'®�·�³ ¶'®!°�¶ µ'°�® ²�² ³]²'®�·�´ °�´Q®�¶�³ ³4µ'®�²�³ °�´Q®�­]µ ³Q®�¯_° ­]µ'®�·4µ ³]¶_®�·4µ�´ �¡r§Q§_¡��f® �Q³ ­_®�³4µ µ¨®�³�· °m¶r¶'® ¶�¶ ¶'® ±�¯ °�® ±�­ °m¶_®o°m­ µ'® µ�¯ ±_®�¯�´ º!¾Q»4½o»�Ä ¯_®�³r· °m¶_®�¶r²r²¢a¡�«¨Å!¦ ¶_®�­r¶ º�Æ�½ÃÄ]Ç ¶'® ²�­ ³_®�´r­ µ¨® ¶�¶ ¯_®�´]­ ¶_®�¶]µ ¶_®o°�³ ±_® µ�² °r®�´r· °r®�±r¶_°È_¹���¡�� ¦���¡�¤D§ ´Q® µ�¶ ´_®�³r± ´_® ±�² ³]¶'® ±�¯ ³r·_®�³r² °m­_®�´r´ ·Q®�·Q° °�²_®�·r´ ³]¶_®�²r± °r°r®�±_° °m­_®�·r·r³¦� �£_¤ ³Q®o°m± ´_®�´�´ ²'® ²�´ ­'®�´r¯ ³_® ²�³ ³Q®o°m² ¶_®�­r¯ ¶_®�·Q° ´Q®o°�³ °r®�¯�· ­_®�­r¶_°

¢a¡r£¨ª µ¨®�´]µrµ ´Q®o°m±'° °m²_®�¶r±�¯ °m¶_®�´r·]µ °m¶_®�´]²�± ·Q®o°m¯r­ ´Q®�±r²]µ ·_®�´_°m­ °r®�³r´4µ ±_®o°�·]¯ ²_®�¶r¶r²

ÉTÊÃË�Ì2ËrÍuÎ2Ï�Ð�Ñ'Ê!Ì'Ò�ÓÃÒ�ÏrÔTÌ'Ê!Ì'ÕÖÉ�Ë�ÏÖרÏ�ÔPÉTÊ!ÐmÓÃÒ�×'ÑQØ_Ù�Ê!Ð�Ù�×'ÔTË�Ú'Ó!ÒmÎ�ÛÜ É�É�Ñ'ÒNÝDÉTÏ�Ì_ÍÞË�Ô�ßáà�Ê!Ì'Ò�ÏrÔ Ü ÐmÐmÒmÓ!ÒmÔ�Ï]É�Ë�Ô�â:ÒmÌDÉ�Ò�Ôäã¼Ý_à Ü â8åçæѨÊÃÕ�ÑèÒm̨ÒmÔTÕ�Ø�רÏrÔ�É�ÊoÐçÓ!ÒéÚ(Ò�Ï�Î�ÙêÏ�Ô�ÒAÐçË�ÓÃÓ!Ê!ß'Ò�ßáÉTËëÕ�ÒmÌ'Ò�ÔTÏrÉ�ÒÙ�ì'Ú¨ÏrÉ�Ë�Î%ÊoÐ�רÏrÔ�É�ÊoÐçÓ!Ò�Ù�Û Ü Î�Ï]í�Ë�Ô2Ð�ѨÏ�ÓÃÓ!ÒmÌ'Õ�ÒîÊ!Ì<ÉTÑ'Ò�Ù�Ò�ÒçïQð×(ÒmÔTÊ!Î%Ò�ÌDÉTÙ2ÊoÙ%É�Ë*ÐçË�Ô�ÔTÒ�ÐçÉ�Ó!ØñÐçÓoÏ�ÙTÙPÊÃÍÞØNÉ�Ñ'Ò0×�ÏrÔ�É�ÊoÐçÓ!ÒîÉTÔTÏ�Ð�òQÙ�Ûó Ò�ÔPÍÞË�Ô�Î�Ï�̨ÐçÒJÊ!Ù�Î2Ò�Ï�Ù�ì'Ô�Ò�ß7ô�ÊÃÉ�Ñ�ÉTÑ'Ò�Ý_à Ü âöõXð�ÙTÐçË�Ô�Ò�÷

Ý_à�õöøáù'ã�ú�û*ü]ýJåPþô�ѨÒmÔTÒ�ù�ÊoÙuÉ�ѨÒ�×�Ò�ÔTÐmÒmÌDÉ�ËrÍ_Ò�ÿ�ÒmÌDÉ�Ù�Ï�Ð�ÐçÒm×'É�Ò�ßJÍÞË�Ô�×'ÔTÒ�ß_ÊoÐ|ÉTÊÃË�Ì�æÏ�Ì¨ß ý�ÊoÙ#É�Ñ'Ò8×'ÔTË�Ú¨Ï�Ú'Ê!ÓÃÊÃÉ�ØMËrͨÎ2ÊoÙ�ÐmÓ!Ï�Ù�Ù�Ê � ÐmÏrÉ�Ê!Ë�Ì�Û�Ý_à5õ ÊoÙ�ÏrÌÏ�×'×'Ó!Ê!Ð�Ï]ÉTÊÃË�Ì_ð�Ù�×(Ò�ÐçÊ � Ð�×(ÒmÔ�ÍÞË�ÔTÎ2Ï�̨ÐçÒ8Î2ÒçÉTÔ�ÊoÐ:É�ѨÏrÉ�Ò�ÙPÉ�Ê!Î�Ï]É�Ò�ÙÉTÑ'ÒäÙPÉTÏrÉ�ÊoÙ�ÉTÊ!Ð�ÏrÓÖ×�Ë]ô�ÒmÔ�ËrÍMÉTÑ'ÒéÎ2Ë_ß_ÒmÓ¼Û��Ì�ÐçÔTÒ�Ï�Ù�Ê!Ì'Õ<Ý_à5õÚQØ���� Ê!Ù2Ò��ì¨ÊÃÿ]ÏrÓ!ÒmÌDÉ%É�ËAѨÏ4ÿQÊĄ̃Õ��� Î2Ë�Ô�Ò�ß'Ï]É�Ï'æ�ô�ѨÊ!Ð�Ñ×(ËrÉTÒmÌDÉ�ÊoÏrÓ!Ó!ØäÙTÏ4ÿ�Ò�ÙÖÑQì'̨ß'Ô�Ò�ß'ÙMËrÍ�É�ѨË�ì¨ÙTÏr̨ߨÙMËrÍ�ß_Ë�Ó!ÓoÏrÔ�Ù Ë�ÔÎ2Ë�Ô�Ò Ê!Ì�Ï�ÐmÐmÒmÓ!ÒmÔ�Ï]É�Ë�ÔaÉTÊÃÎ2Ò�Û� Ï�Õ�Õ�Ò�ßFÉ�ÔTÒmÒ�Ù<ѨÏ4ÿ�Ò�ÉTÑ'ÒèÚ(Ò�ÙPÉ�Ý'à5õ ×�Ò�ÔPÍÞË�Ô�Î�Ï�̨ÐçÒ�æîÊ!Ì_ðÐmÔ�Ò�Ï�Ù�ÊĄ̃Õ�Ý_à5õ� 'Û ������0ã,Ïäú�ü�� ÊÃÎ2×'ÔTË]ÿ�Ò�Î%Ò�ÌDÉ�å�Ë]ÿ�ÒmÔÖô:Ò�ÓÃÓÃðË�×_É�Ê!Î2Ê���Ò�ß�Ì'Òmì'Ô�ÏrÓQÌ'ÒmÉTÙ�ÏrÌ�ß�ß_Ò�ÐçÊoÙPÊ!Ë�ÌMÉTÔ�Ò�Ò�Ù�Û5Ý_à�õ Ú(ÒmѨÏ4ÿ�Ò�ÙÓ!Ê!ò�Ò�Ï�ÐçÔTË�ÙTÙ�Ú�ÒmÉ�ô:Ò�ÒmÌ Ï�ÐmÐçì¨ÔTÏ�ÐçØ�Ï�Ì¨ß Ð�ÏrÓ!ÊÃÚ'Ô�Ï]ÉTÊÃË�Ì�æ�Ù�Ë7ÊÃÉJÊoŲ̀ËrÉ0ÙPì'ÔT×'ÔTÊ!Ù�Ê!Ì'Õ*É�Ñ�Ï]É�Ú¨ÏrÕ�Õ�Ò�ß ÉTÔ�Ò�Ò�Ù��ëÉTÑ'ÒAÎ%Ë_ß_Ò�Ó!Ùêô�ÊÃÉ�ÑÉTÑ'Ò7Ú(Ò�ÙPÉ%Ð�ÏrÓ!ÊÃÚ'Ô�Ï]ÉTÊÃË�ÌN×�Ò�ÔPÍÞË�Ô�Î�ÏrÌ�ÐçÒ��äÏrÔTÒ�Ú(Ò�ÙPÉ�Ë�ÌñÝ_à�õ�Û� ѨÒmÌ7ÏrÌ2Ò�̨ÙPÒ�Î�Ú'Ó!Ò�ÊoÙ�Ë�×_ÉTÊÃÎ2Ê��mÒ�ß%É�Ë�Ý_à�õáô�ÊÃÉ�Ñ�Ò�̨ÙPÒ�Î�Ú'Ó!ÒÙ�ÒmÓ!Ò�ÐçÉ�Ê!Ë�Ì�æDÝ_à5õ ×�Ò�ÔPÍÞË�Ô�Î�Ï�̨ÐçÒ8Ê!̨ÐçÔTÒ�Ï�ÙPÒ�Ù���� Ë]ÿ�Ò�Ô5É�Ñ'ÒUÚ(Ò�ÙPÉÚ�ÏrÕ�Õ�Ò�ß�É�ÔTÒmÒ�ÙmÛ��8Ñ'ÊoÙ�����ÊÃÌ�ÐçÔTÒ�Ï�Ù�Ò�Ê!Ì�Ò��uÒ�Ð|ÉTÊÃÿ�ÒUÙ�Ï�Î2×'ÓÃÒUÙ�Ê��mÒÔTÒmרÔ�Ò�ÙPÒ�Ì�É�ÙgÏ�ÓoÏrÔTÕ�Òa×�Ë�É�ÒmÌDÉTÊ!Ï�ÓDÙTÏ4ÿQÊÃÌ'ÕDÙfÊ!Ì�Ï�Ð�ÐçÒ�ÓÃÒ�ÔTÏrÉ�Ë�Ô�É�Ê!Î%Ò�Û

"!$#&%('�)+*,'�'�%.-�/102/,3546*678*698:<;=-�98>

?A@CBD@FEHG8IKJCL�GNMOJKPRQSGTQULSVWJKICICX�ICJKY[Z�JCQ�\^]F_�M`a ̨Ù�ÒmÎ�Ú¨ÓÃÒîÙ�ÒmÓ!Ò�ÐçÉ�Ê!Ë�Ì*ì¨Ù�Ò�Ù É�Ñ'Ò^bdcfehgCi�cfjKgCkfl*ÙPÒmÉ�É�Ë&m�É�Ô�ÏrÊ!ÌTnÒ�̨Ù�ÒmÎ�Ú¨ÓÃÒ�ÙmÛ2oUÊ!Ó!Ó!ÐmÓÃÊ!Î�Ú'Ê!Ì'Õ�Ë�Ì�ÉTÑ'ÒJÿ]Ï�ÓÃÊoß'Ï]ÉTÊÃË�ÌîÙ�ÒçÉ�ß'ËDÒ�ÙąËrÉÕ�ÊÃÿ�Ò�Òm̨Ù�ÒmÎ�Ú'Ó!ÒUÙPÒ�ÓÃÒ�Ð|É�Ê!Ë�Ì�Ï�Ì%ì¨Ì_Í,ÏrÊ!Ô:Ï�ß_ÿ]ÏrÌDÉTÏ�Õ�Ò8Ë]ÿ�Ò�Ô�ËrÉ�ѨÒmÔÎ2Ë_ß_Ò�Ó!Ù�Û��8Ñ'Òäÿ4Ï�ÓÃÊoß'ÏrÉ�Ê!Ë�ÌèÙ�ÒçÉîô�Ë�ì¨Ó!ßáÚ(Òę́ÒmÒ�ß'Ò�ß�ÉTËëÙ�ÒçðÓ!Ò�ÐçÉ�É�Ñ'ÒM×�ÏrÔ�ÏrÎ2ÒçÉTÒmÔ�ÙaÍÞË�Ô�Ò�Ï�Ð�ÑêÏ�ÓÃÕ�Ë�ÔTÊ�ÉTÑ'Î ãÞ×�ÏrÔ�ÏrÎ2ÒçÉTÒmÔUÙ�ÒçðÓ!Ò�ÐçÉ�Ê!Ë�Ì�åçæ5Ï�̨ßAÉ�ѨÒmÌNÉ�Ë0רÊ!Ð�òéÉ�Ñ'Ò7Ú(Ò�ÙPÉ�Ï�ÓÃÕ�Ë�ÔTÊ�ÉTÑ'Î ã,Î%Ë_ß_Ò�ÓÙ�ÒmÓ!Ò�ÐçÉ�Ê!Ë�Ì�åçÛ a ̨ÙPÒ�Î�Ú'Ó!Ò2ÙPÒ�ÓÃÒ�Ð|É�Ê!Ë�Ìäì¨Ù�Ò�ÙJÉ�Ñ'Ò2ÿ]ÏrÓ!Êoß'Ï]ÉTÊÃË�ÌäÙ�ÒçÉÍÞË�Ô�רÏ�ÔTÏ�Î2ÒçÉ�Ò�Ô�Ù�ÒmÓ!Ò�ÐçÉ�Ê!Ë�Ì�æ�Î2Ë_ß_ÒmÓ8ÙPÒ�ÓÃÒ�Ð|ÉTÊÃË�Ì�æ�Ïr̨ßNÒ�̨Ù�ÒmÎ%ðÚ¨ÓÃÒ�ÐçÔTÒ�ÏrÉ�Ê!Ë�Ì�Û � ÊÃÉ�Ñ Î2Ï�ÌQØNÏ�ÓÃÕ�Ë�ÔTÊ�ÉTÑ'Î�Ù�ÿ]Ï�ÓÃÊoß'Ï]ÉTÊÃË�Ìëß'ÏrÉTÏÐ�ÏrÌ�Ú(Ò ×'ì_ÉîÚ�Ï�Ð�òñÊ!Ì�É�Ñ'Ò�ÉTÔTÏ�ÊÃÌáÙ�ÒçÉ�Ï�̨ßëÉ�Ñ'Ò0Î2Ë_ß_Ò�ÓXÔTÒçð

ÉTÔTÏ�ÊĄ̃Ò�ß%Ë�̨ÐmÒ�רÏrÔ�ÏrÎ2ÒmÉ�ÒmÔ�Ù�ÏrÔTÒUÙ�ÒmÓ!Ò�Ð|ÉTÒ�ßfÛ��8Ñ'ÊoÙaÐ�ÏrÌ�ÏrÓoÙPËMÚ(Òß'Ë�Ì'Ò�ô�Ê�ÉTÑéÒmÌ�ÙPÒ�Î�Ú'Ó!Ò�Ù�ÒmÓ!Ò�ÐçÉ�Ê!Ë�Ì�Û Ü ÌQØ Ù�ÉTÔTÏrÉ�Ò�Õ�Ø�ÍÞË�ÔXì�ÙPÊ!Ì'ÕÏ�̨ß�ÔTÒmì¨Ù�Ê!Ì'Õ�ÿ]ÏrÓ!Êoß'Ï]ÉTÊÃË�Ì7ÙPÒmÉTÙ�æDÊ!̨ÐmÓÃì¨ß'ÊÃÌ'Õ2ÐçÔTË�ÙTÙ�ÿ]ÏrÓ!Êoß'Ï]ÉTÊÃË�Ì�æÐ�ÏrÌ�Ú(ÒUì¨Ù�Ò�ß2ô�ÊÃÉ�Ñ7Ò�̨Ù�ÒmÎ�Ú¨ÓÃÒUÙ�ÒmÓ!Ò�ÐçÉ�Ê!Ë�Ì�Û � ÒXÐçì'ÔTÔTÒmÌDÉ�Ó!Ø�Ï�Ô�ÒÔTì'Ì'̨ÊÃÌ'ÕMÒmï_×�Ò�Ô�Ê!Î2ÒmÌDÉTÙ�ô�ÊÃÉ�Ñp�4ð¼ÍÞË�Óoß2ÐçÔTË�ÙTÙ5ÿ4Ï�ÓÃÊoß'ÏrÉ�Ê!Ë�Ì�É�ËMÊ!Ì_ðÐmÔ�Ò�Ï�Ù�Ò É�Ñ'Ò%ÙPÊ��mÒ�ËrÍ�É�Ñ'Ò�Ñ'Ê!ÓÃÓoÐçÓ!ÊÃÎ�Ú'Ê!Ì'Õ�Ù�ÒçÉUÉTË7Ê!̨ÐçÓ!ì¨ß_Ò%ÏrÓ!ÓgËrÍÉTÑ'Ò É�Ô�ÏrÊ!Ì'Ê!Ì'Õ�ß'Ï]É�Ï'æ_Ì'Ë�É�í�ì¨ÙPÉ�É�Ñ'ÒMÑ'Ò�Ó!ß_ð¼Ë�ì_ÉMúmòîÙTÏrÎ2×'Ó!Ò�Ù�Û

?A@rqA@Us�PALU_�IK`t]F_�IC_�XfM_�LuZ+vuMOw�_yxzQU`_�Y[Z�IC_�`�5ÏrÚ¨ÓÃÒ{�0Ù�Ñ'Ë]ôUÙJÉ�Ñ'Ò2ÉTËrÉTÏ�Ó�ô�ÒmÊ!Õ�ÑDÉMÕ�Ê!ÿ�ÒmÌäÉ�Ë�Ò�Ï�Ð�ÑäÉ�ØD×(Ò�ËrÍÎ2Ë_ß_Ò�Ó�ÍÞË�ÔgÒ�Ï�Ð�Ñ Î%ÒmÉ�ÔTÊ!ÐaË�Ì Ü}|}~ àN�*Ïr̨ß�â��6� a�� �"� ó�a Û�8Ñ¨Ò Ï4ÿ�ÒmÔ�ÏrÕ�ÒëÏ�ÐçÔTË�ÙTÙ0ÉTÑ'Ò É�Ò�Ì Î2ÒçÉTÔ�ÊoÐmÙ�ãÞÔTÊ!Õ�ÑDÉNÐmË�Ó!ì'Î2Ì�åÙ�Ñ'Ë]ôUÙ#É�Ñ�Ï]É��H�,��ÏrÌ�ß � ÝN� | �ñÔTÒ�ÐçÒ�ÊÃÿ�ÒaÉTÑ'Ò8Î2Ë�ÙPÉ�ô�ÒmÊ!Õ�ÑDÉÊ!ÌAÉ�Ñ'Ò�Ò�̨Ù�ÒmÎ�Ú¨ÓÃÒ%ÍÞË�Ô�â��6� a�� �"� ó�a ÛA�8Ñ'Ò2ô�ÒmÊ!Õ�ÑDÉTÙMÏ�Ô�ÒÙPÉ�ÔTÊ!òDÊ!Ì'Õ�ÓÃØYß_Ê��(Ò�Ô�Ò�ÌDÉëÍÞË�Ô Ü}|}~ àN� ô�Ñ'ÒmÔTÒ � ÝN� Ý8��� ó æÜ �,��æ:ÏrÌ�ß | � Ô�Ò�ÐçÒ�ÊÃÿ�Ò�ÉTÑ'Ò Î2Ë�ÙPÉ7ô�ÒmÊ!Õ�ÑDÉ�Û&�8ѨÒmÔTÒ ÏrÓoÙPËÏ�Ô�ÒAÙPì¨Ú¨Ù�É�ÏrÌDÉ�ÊoÏrÓMß_Ê��uÒmÔTÒm̨ÐmÒ�ÙêÚ�ÒmÉ�ô:Ò�ÒmÌèÉ�Ñ'ÒNÎ2Ë_ß_ÒmÓÖÉ�ØQ×�Ò�ÙרÔ�ÒmÍÞÒmÔTÔ�Ò�ß ÍÞË�Ô ß_Ê��(Ò�Ô�Ò�Ì�É Î2ÒçÉTÔ�ÊoÐmÙ�Û �'Ë�ÔöÒmï_Ï�Î2×'ÓÃÒ�æ*Ë�Ìâ��6� a�� �"� ó�a æ$��Ý�â Õ�Ê!ÿ�Ò�Ù Ñ'Ê!Õ�ÑFô:Ò�ÊÃÕ�ÑDÉéÉ�Ë Ú(ËDËDÙ�ÉTÒ�ßÉTÔ�Ò�Ò�Ù�Ïr̨ß*Ó!Ë]ôbô:Ò�ÊÃÕ�ÑDÉ É�Ë[�H�}� æ5Ú'ì_É%É�Ñ'Ò�ÙPÒîô:Ò�ÊÃÕ�Ñ�É�Ù�Ï�Ô�ÒÔTÒmÿ�ÒmÔ�ÙPÒ�ßáÍÞË�Ô � �éÝ ÏrÌ�ß àU��� Û Ü ÓoÙPË�æ Ü �}�XÙ0Õ�ÒçÉ Î2Ë_ßQðÒ�Ù�É7ô�ÒmÊ!Õ�ÑDÉ2ÍÞË�Ôp�y� a ÏrÌ¨ß � �äÝöã,ô�ѨÏ]É2ÉTÑ'Ò Ü �}�XÙ7Ë�×_ðÉTÊÃÎ2Ê��mÒ4å|æfÚ'ì_ÉMÓ!Ë]ô�ô:Ò�ÊÃÕ�ÑDÉXÍÞË�Ô Ü â�â Ï�̨ßéàU��� Û+�8Ñ'ÊoÙ ÙPì¨ÕrðÕ�Ò�ÙPÉTÙ�É�ѨÏrÉ:ÒmÌ�ÙPÒ�Î�Ú'Ó!ÒJÙ�ÒmÓ!Ò�ÐçÉ�Ê!Ë�Ì7ÊoÙaÏ�Ú'ÓÃÒXÉ�Ë�Òçï_×'Ó!Ë�ÊÃÉ�É�Ñ'Ò ß_ÊÃÍ ðÍÞÒ�Ô�Ò�ÌDÉ:Ù�ÉTÔ�Ò�Ì'ÕrÉTѨÙaÏ�̨ß�Ú'ÊoÏ�Ù�Ò�Ù�ËrÍfÉ�ѨÒJß_Ê��(Ò�Ô�Ò�ÌDÉaÓÃÒ�ÏrÔTÌ'Ê!Ì'Õ�ÏrÓÃðÕ�Ë�ÔTÊ�ÉTÑ'Î�Ùaô�ѨÒmÌîË�×_É�Ê!Î2Ê���ÊĄ̃Õ%Ï�ÌîÒmÌ�ÙPÒ�Î�Ú'Ó!ÒUÉTË�Ò�Ï�Ð�Ñ7Î2ÒçÉTÔ�ÊoÐÏ�̨ßîÉ�Ë�Ò�Ï�Ð�Ñ�רÔ�Ë�Ú'ÓÃÒ�Î�Û

?A@��F@���PDY����FMGNMOJKPRQ�G8I"��PD`�M� ì'Ê!Óoß_ÊĄ̃Õ<Ó!ÊÃÚ'Ô�ÏrÔTÊ!Ò�Ù�ÊoÙîÒmï_×�Ò�̨ÙPÊ!ÿ�Ò�Û � Ò�Ð�Ïrì¨Ù�Ò0Î2ËQß'ÒmÓoÙ�Ï�Ô�ÒÊ!̨ß_Ò�×�Ò�̨ß_Ò�Ì�É�æ�Ê�É7ÊoÙ�Ò�Ï�Ù�Ø*É�Ë*רÏ�ÔTÏ�ÓÃÓ!ÒmÓ!Ê��mÒ�Î2ËQß'ÒmÓXÐmÔ�Ò�Ï]É�Ê!Ë�ÌÏ�̨ßMß_ÊoÙ�ÉTÔ�Ê!Ú'ì_ÉTÒ�ÉTÔTÏ�ÊÃÌ'Ê!Ì'ÕUÏ�ÐçÔTË�ÙTÙfÎ�Ï�Ð�Ñ'Ê!Ì'Ò�Ù�ÛA��É#ÉTÏrò�Ò�Ù�Ï�Ú�Ë�ì_É��� Ñ'Ë�ì'Ô�Ù�ÉTËXÉTÔTÏ�ÊÃÌ�É�Ñ'Ò�üf � � �Î2Ë_ß_ÒmÓoÙ5ì¨Ù�ÊÃÌ'Õ ÏXÐmÓÃì�Ù�ÉTÒmÔ�ËrÍ'ÉTÒmÌàgÊÃÌQì_ï�Î�Ï�Ð�Ñ'Ê!Ì'Ò�ÙmÛ��0Ë_ß_ÒmÓgÉ�Ô�ÏrÊ!Ì'Ê!Ì'Õ�Ê!ÙXÏrì_ÉTË�Î�Ï]ÉTÒ�ßfÛ��8Ñ'Ò�Ô�ÒÊoÙÖÌ'Ë�רÏrÔ�ÏrÎ2ÒmÉ�ÒmÔJÉ�ì¨Ì'ÊĄ̃Õ�Ë�ÔÖÒmï'ÏrÎ2ÊĄ̃ÊÃÌ'Õê×(ÒmÔ�ÍÞË�ÔTÎ2Ï�̨ÐçÒ�Ë�Ìÿ]Ï�ÓÃÊoß'Ï]ÉTÊÃË�Ì0ÙPÒmÉTÙ�Û ~ ÙPì�ÏrÓ!ÓÃØêÌ'Ë�Ë�Ì¨Ò Î2ËQß'ÒmÓgÊoÙXÐmÔ�ÊÃÉ�ÊoÐmÏ�Ó.æ�ÙPË7ÊÃÉÊoÙ%Ì'ËrÉ�Ì'Ò�ÐçÒ�Ù�ÙTÏrÔTØéÉTËNô�Ï�Ê�É2ì'ÌDÉ�Ê!ÓUÏrÓ!Ó8Î%Ë_ß_Ò�Ó!Ù2Ï�Ô�ÒîÉ�Ô�ÏrÊ!Ì'Ò�ßÉTËäì�ÙPÒ�É�Ñ'Ò�Ó!ÊÃÚ¨ÔTÏ�Ô�Ø�Û��8Ñ'Ê!Ù%×'ÔTË]ÿQÊ!ß'Ò�Ù%ÏrÌëÏ�ÌQØ�ð¼É�Ê!Î2Òp��Ï4ÿ�Ë�Ô

�����O�����f�2����� .¡¢�d£�¤.¡6¥¦¡¢§h�¨©¤�ª"§h«d¡¢¬­¤.®$¯�§±°8¡² .¡¢¬©¤�³F´�µN¡¢¶�®�·F¸�®f¹�¡¢ºh¶�§h¬�¤.¨�¡�»A¬�¶¼¡²½z¾�ºh¡¢¶¢¿

ÀTÁTÂTÃ(Ä À�Å�ŠƩǢŠÃÆdÄ È�ÉRÅ ÀRÊ�È Ë�Ì�Ê ÈNÍ�Ç Í2ÎTÌ Ç²ÀTÈ ÀÏ8Ð ÑzÄ

ÀTÒ8Ò ÓNÔ±Ó�Õ©ÓfÖ ÓNÔr×�Ø8×�Ù ÓNÔr×�ÓN×OÚ ÓNÔ±ØfÙfÛ�Ù ÓNÔ Û�ØfÓ©Û ÓNÔhÚdÛ�Ø�Ú ÓNÔr×�ÙfÙ�Õ ÓNÔ±ÓfÖfÛ8× ÓNÔhÜfÕ©Ø�Ú ÓNÔhÚfÕfÚ©ÓÝ Ò8Ò ÓNÔ±Ó�Ú©Ó�Ü ÓNÔ±ÓN×�Û�Þ ÓNÔhÜ©ÞfÙ�Ü ÓNÔ±ÓfØ�ÙfÞ ÓNÔ±Ó�Ú©ÞfÙ ÓNÔ±Ó©Û�ÞfÙ ÓNÔ±ÓfÓfÓfÓ ÓNÔ±ÓfÓ�ÓfÓ ÓNÔ±Ó©Û�Ö�Ú ÓNÔ±ÓfÖfÞN×ÇßÏRÍ ÓNÔ±ÓfÓfÓfÙ ÓNÔ±ÓfÓ�ÓfÓ ÓNÔr×�ÓfÖfÙ ÓNÔhÚ©ÞfÛ8× ÓNÔhÚfÕdÛ�Ø ÓNÔ±ÓfÖ�ÚdÛ ÓNÔ±Ó�ÜfÕdÛ ÓNÔ±ÓfÓ�ÓfÓ ÓNÔ±Ó�ÚfÚdÛ ÓNÔr×�Ó�Ü�×ÁTÄ ÓNÔ±Ó�Ú©ÓN× ÓNÔ±ÓfØfÛ�Ù ÓNÔ±ÓfÓ�Õ©Ó ÓNÔ±ÓfÞ�Õ©Ö ÓNÔ±Ó©Û�ÖN× ÓNÔ±ÓN×OÞfÞ ÓNÔhÕdÛ�Ü©Ù ÓNÔ±ÞfÙ�ÕdÛ ÓNÔhÚ©Ø©Û�Ü ÓNÔhÚfÜ©ÞN×

Ë�ÀNÐ Á8Ä ÓNÔ±ÓfÓ�Ú�× ÓNÔ±ÓfÓ�ÓfÞ ÓNÔ±ÓfÓfÓfÓ ÓNÔ±ÓfÓ�ØfÞ ÓNÔ±ÓfÓ�Ü�× ÓNÔ±ÓfÓ�ÚfÚ ÓNÔ±ÓfÓ�ÜfÕ ÓNÔ±ÓN×OÓ�Ú ÓNÔ±ÓN×�ØfÙ ÓNÔ±ÓfÓ�ÜdÛË�Ç²Ä ÁTÄ ÓNÔr×f×�ÓfÓ ÓNÔr×OÜ�Ú©Ø ÓNÔ±Ó�ÚfÜ©Ó ÓNÔ±Ó�Ü�ÕfÚ ÓNÔ±ÓfØN×�Ö ÓNÔ±Ó©Û�Õ©Ó ÓNÔ±Ó�ÚdÛ�Ü ÓNÔ±Ó�ÚfÞfØ ÓNÔ±Ó�ÕdÛ�Þ ÓNÔ±ÓfÙfÞfÖË�Ç²Ä Ç²ÄNÍ�Ê ÓNÔhÕfÕfÜ©Ö ÓNÔ±ÙfÙ�Ü©Þ ÓNÔr×OÕ©Ó�Õ ÓNÔr×�Ù�ÜfÕ ÓNÔr×�ÞfÓfÙ ÓNÔhÜdÛ�Õ©Þ ÓNÔ±ÓfÓfÓfÓ ÓNÔ±ÓfÓ�ÓfÓ ÓNÔ±ÓfØ�Ú�× ÓNÔ±ØN×OÕ©Ø

Å�É�Ï Ä�àDÊ�Ì À�Å�ŠƩǢŠÃÆdÄ È�ÉRÅ ÀRÊ�È Ë�Ì�Ê ÈNÍ�Ç Í2ÎTÌ Ç²ÀTÈ ÀÏ8Ð ÑzÄ

ÀTÒ8Ò ÓNÔ±ÓN×f×�Û ÓNÔ±ÓN×OÓ�Ú ÓNÔ±ÓfÓfÓ�Õ ÓNÔ±ÓfØ�Þ�Ú ÓNÔ±Ó�ÚfÚ©Þ ÓNÔ±ÓfÓ�ÞfÙ ÓNÔ±ÓfÞ�Õ�× ÓNÔ±ÓfÖ�ÙfÙ ÓNÔ±Ó�ÜfÚ�× ÓNÔ±Ó©Û8×�ÓÝ Ò8Ò ÓNÔr×OÕ©ÖfÓ ÓNÔr×�Ù�Ù�Ü ÓNÔhÜfÕfÜ©Ö ÓNÔhÚfÜN×�Þ ÓNÔhÚ©Ö©Û�Þ ÓNÔhÚ©Ó8×�Ù ÓNÔ Û�ØfÙ©Û ÓNÔ Û�ÚfÙfÙ ÓNÔ±ØfÙ©Û�Ø ÓNÔ±ØfÙ�Ú�×ÇßÏRÍ ÓNÔ±Ó�Ú©ÓfÙ ÓNÔ±ÓN×OÙN× ÓNÔ±ÓfÞ�Õ©Þ ÓNÔr×�Ó�ØfÖ ÓNÔr×�Ó�Ü©Þ ÓNÔ±Ó�ÜfÓ�Õ ÓNÔ±ÓN×�ÓfØ ÓNÔ±ÓN×OØ©Û ÓNÔ±ÓfØ�Õ©Ö ÓNÔ±Ó�ÜfÜ©ÞÁTÄ ÓNÔ±ÓfÙfÓfÙ ÓNÔ±Ó�ÜfØfÖ ÓNÔ±ÓN×OÚdÛ ÓNÔhÚ©Ø�ÞN× ÓNÔhÚdÛ�Ú©Ø ÓNÔ±Ó�ÚfÖ�Ü ÓNÔ Û�ÓfÞfØ ÓNÔ±ØfÙ�Þ�Ü ÓNÔhÚ©ÓfÓfÓ ÓNÔhÚ©ÓN×OÕ

Ë�ÀNÐ Á8Ä ÓNÔ±ÓfÓ©Û�Ö ÓNÔ±ÓfÓ�Ü©Ù ÓNÔ±ÓfÓN×�Ö ÓNÔ±ÓfÓ�Ö�Ú ÓNÔ±ÓN×OÜ©Ó ÓNÔ±ÓfÓ�Ù©Û ÓNÔ±ÓN×OÜ©Ù ÓNÔ±Ó�Ú�ÚfÚ ÓNÔ±Ó©Û�ØfÞ ÓNÔ±ÓN×OÜ©ÙË�Ç²Ä ÁTÄ ÓNÔhÜfÜ©ØfØ ÓNÔ±ÙN×OØ©Û ÓNÔr×OÚ©ÖfÞ ÓNÔhÚfÕfÞ©Û ÓNÔhÚ©ØfÖ�Õ ÓNÔ±Ù©Û�ØfÞ ÓNÔ±Ó©Û�Ú©Ø ÓNÔ±Ó�Õ�ÚfÕ ÓNÔhÚ©Ö�Ú©Ø ÓNÔ±Ø�Ü©Þ�ÚË�Ç²Ä Ç²ÄNÍ�Ê ÓNÔr×OÕ©Ó�Ú ÓNÔr×�ØfÛ�Ø ÓNÔr×�ÖN×�Û ÓNÔ±ÓfÞ�Ó©Û ÓNÔ±Ó�Õ©Ö�Õ ÓNÔ±Ó�ÜfÖfØ ÓNÔ±ÓfÓfÓfÓ ÓNÔ±ÓfÓ�ÓfÓ ÓNÔ±ÓfÓfÖ�Ü ÓNÔ±ÓfÖfÓfÙ

á²âtã�äTåßã�æ�ç8è�ãHåßã�è�ãOé�áßê�â�äAë2ãOäTå(ãOæ­ç8è�ãOå�éOìfä¦çRãzá²í²ì�ê�ä8ãî�ïRå(ê�ä8ðñ�ò ì©áßãOó�ãOí�ætâNîNãOè�å{ì�íßãôìdó©ìfê�è�ì�ç8è�ã ñ�ò ãOä�á ò ãôã�äTåßã�æ�ç8è�ãôêråäTã�ãOî8ãOî+õ2ö÷á�ê�å"ãì�åßø�á²â�ì�î8îpætâ�íßã}æ�âNîNãOè�å"èrì©á²ã�íõ�ùFê�çTí²ì�íßê�ãOåéOìfäçRã�ç8ï8ê�èúáûçDã�üýâ�íßãtá ò ã{þRãOí(üýâ�íßæ{ì�äTé�ã�ætã�áßí²êrétê�åHÿ�ä8â ñ äçDãOéOìfïTåßã6á ò ãzè�ê�ç8í¢ìfí²ê�ãå2á ò ã�æ{å(ãOè�ó�ãOå"îNâ­äTâfá"î8ã�þDã�äTî â�ä{á ò ãætã�áßí²ê�é�á ò ì©á ñ ê�è�è�çDã�ïTåßãOî}áßâ�â�þNáßê�ætê � ã2á ò ã�ã�äTåßã�æ�ç8è�ã�õ�� ò êråæ{ì�ÿ�ãOå�ætâNîNãOè+è�ê�ç8í¢ìfí²ê�ãå�ó�ã�í²øpí²ã�ïTå²ìfç8è�ã�õ� â�í ñ ì�í²î å.á²ã�þ ñ ê�åßã ã�äTåßã�æ�ç8è�ã åßã�è�ãOé�áßê�â�ä êrå ã��pé�ê�ãOä�áõ� î8îNê�ä8ð,ì�ætâNîNãOè�á²â,ìfäHãOäTå(ãOæ­ç8è�ã�â�ä8è�ø}íßã��ï8ê�í²ãOåFìdó�ã�í¢ìfð�ê�äTðì�ætâNîNã�è� å"þ8í²ãOî8ê�é�áßê�â�äTå ñ ê�á ò á ò ã$ã�äTåßã�æ�ç8è�ã� å�þTíßãîNê�é�áßê�â�äTå� ñ�ò ê�é ò êrå��������$üýâ�í�� á ò ã å(ê � ã{â�ü"á ò ã ò ê�è�è�é�è�ê�æ­ç8ê�ä8ð[åßã�áõö÷üHá ò ã�í²ã[ì�íßã�� ætâNîNã�èrå{á²â�é ò â�â�åßãôüýíßâ�æ� �á ò êråpêrå�îNâ�ä8ã� á²ê�ætãOå6üýâ�í}ãOì�é ò åßã�è�ãOé�áßê�â�äyå(áßã�þFõ�ö÷ü�å(ãOè�ãé áßê�â�äôêrå,í²ï8ä��å(áßãOþTå� Oá ò ã�é�â�å(áFâfü8ãOäTåßã�æ­çTè�ã�åßã�è�ãOé�áßê�â�äHêråFâ�ä8è�ø���� �"!$#%!'&(�ì�åßåßï8ætê�ä8ð�á ò ãHætã�á²íßêréHé�ì�ä çDãûé�â�ætþ8ïNáßãî¦ê�ä������"� õRö÷üUá ò ãætã�áßí²ê�éWêråpætâ�íßãyã*)NþRãOäTå(ê�ó�ãyá ò ì�ä+�����"���ýã�õ ðRõ, .-.��/ í²ã�0��ï8ê�í²ãOå{åßâ�íßáßê�ä8ð�ìfäTî á ò ïTå�ê�å1������!dè�â�ð2�"�3�� �í²ãOé�â�ætþ8ïNáßê�ä8ðá ò ãzæ�ã�áßí²ê�ézîNâ�æ�ê�äTìfáßãOåOõ54,åßê�ä8ðtì�6 �879� ê�ætþ8è�ã�ætãOä�á¢ì©á²ê�â�ä: åßã�è�ãOé�áßê�ä8ð ìfä<ã�äTåßã�æ�ç8è�ãüýí²â�æ ìuè�ê�ç8í¢ìfí²ø âfü;� <>=�?�?�?ætâNîNãOè�å� +ì ò ê�è�èré�è�ê�æ­ç8ê�ä8ð åßã�á ñ êúá ò �@<BA?�?�?tþDâ�ê�ä�á²å� +ìfäTî�C<D=$?2?Wá¢ìfÿ�ãåtìfçDâ�ïNápì^æ�ê�ä�ïNáßãôâ�äuì^ætãOîNê�ï8æ�0 þDâ ñ ã�íñ â�í²ÿNå.á¢ì©á²ê�â�äAõtö÷ü�åßã�è�ãOé�áßê�â�ä êråHçTìfð�ð�ãîE=�?�á²ê�ætãå* �êúáHá¢ìfÿ�ãOåì�çRâ�ïNáF=$?�ætê�ä�ïNá²ãOå�á²â{ç8ï8ê�è�î á ò ã"GTäTì�è+ã�äTåßã�æ�ç8è�ã�õH:I,J8I5K;L8MN�OPN Q�N R8S%TVUXW�RZY\[F]2^'_`Ua^bOdceR8f2]�gh]iM*^'N f�Fjkj åtïTå(ïRìfè�è�ø�ìfí²ãpá²í²ì�ê�äTãOî�á²â[ætê�ä8ê�ætê � ãWé�í²â�å²åûãOä�áßí²â�þ�øâ�íûål��ïTì�íßãî[ãOíßí²â�íõm�UíßãOãOåûì�äTîXn 7 #yå­ïTåßïTìfè�è�ø[ætì$)Nê�ætê � ãì�é�é�ï8í²ì�é�ø�õ8o"â�â�å(áßê�ä8ð­ìfèråßâûê�å"îNãå(ê�ð�ä8ãîtáßâ�æ{ì')Nê�ætê � ãzì�é�é�ïp0í¢ì�é�ø�õqn�â�ætãûætã�áßí²ê�éOå,åßïTé ò ì�å�þTíßãé�êrå(ê�â�äsr©íßãé�ì�è�èAì�äTît-.��/ì�íßã ò ì�í²î[áßâWâ�þNáßê�ætê � ãtáßâRõuo"ãOé�ì�ïTåßã�ætâNîNãOè�ìdó�ãOí²ì�ð�ê�ä8ð�êråüCì�å.á �ã�äTåßã�æ�ç8è�ãyåßã�è�ãOé�áßê�â�äSé�ì�äuáßí²ø ì�îTîNê�äTð�ã�ó�ãOíßø�æ�âNîNãOèê�äSá ò ã[è�ê�çTí²ì�íßø áßâ�á ò ã^ã�äTåßã�æ�ç8è�ãì©á�ãì�é ò å(áßãOþAõ�ö÷üûá ò ã

þDã�íßüýâ�í²æ{ìfäTé�ãûâfü�ãì�é ò â�ü�á ò ãå(ã�ã�äTåßã�æ�ç8è�ãOå,éOìfäyçDã�ã�ó©ìfè�ïp0ìfáßãî1��ï8êré¢ÿ�è�ø â�ä¦á ò ãûætã�áßí²êré$ 8á ò ãHã�äRå(ãOæ­ç8è�ã­é�ì�äôçRãHâ�þNáßê,0ætê � ãîHáßâ}á ò ì©á�æ�ã�áßí²ê�é�ç�ø$á ò êrå�ð�í²ã�ãîNø� dç8í²ïNáßã�üýâ�í¢é�ã�åßãOìfí¢é ò õ� ð�â�â�îãOäTå(ãOæ­ç8è�ãtïTåßïTìfè�è�ø ñ ê�è�è�çDãtüýâ�ï8äRî^ê�ü6å(â�ætã�çRì�åßã�0è�ã�ó�ã�èUæ�âNîNãOè�å}â�ízé�â�æ­ç8ê�äTì©á²ê�â�äTå6â�ü2á ò ãOæ ø�ê�ã�èrîWð�â�âNîôþDã�íl0üýâ�íßæ{ì�äTé�ã{â�ä á ò ìfá­ætã�áßí²êréfõ � è�á ò â�ï8ð ò�ñ ã�î8âyä8âfá�ÿ�äTâ ñò â ñ áßâ,â�þNáßê�ætê � ã�á ò ã�çRì�åßã�0÷è�ãOó�ã�è�ætâ�î8ã�èråFá²â,æ{ìfä�ø$âfüNá ò ãOåßãætã�áßí²ê�éOå* �á ò ãHã�äTåßã�æ�ç8è�ãûé�ìfä¦çDãHâ�þNá²ê�ætê � ãOî á²â�á ò ã�æôõ

H:Iwv:I8x�]iYaUsR8yzx�N�R9cp^bY|{�} ce~~�N��9f2cpM�N UaR� äTåßã�æ­çTè�ã�å(ãOè�ãé á²ê�â�ä�êrå2å.á²í²ì�ê�ð ò á(üýâ�í ñ ìfí¢îzüýâ�í�ç8ê�äTìfí²øûé�èrì�å²å(ê,0GDé�ì©á²ê�â�äWìfäTî¦í²ã�ð�í²ãOå²åßê�â�äAõ�ö÷ü�á ò ã­çTì�åßã�0÷è�ãOó�ãOèDætâNîNãOè�å,ætì�ÿ�ãþTíßãîNê�é�áßê�â�äTå2üýâ�í�æ�ï8è�áßê�þ8è�ãzé�èrì�å²å(ãå* �ä8âûætâNîNê�GDé�ì©á²ê�â�ätáßâHá ò ããOäTåßã�æ­çTè�ãå(ãOè�ãé áßê�â�ä&þ8íßâNé�ãOîNï8í²ãyêrå¦ä8ãOé�ãOå²åßì�íßø�üýâ�í¦æ­ï8è�áßê,0é�è�ì�åßå­þ8í²â�ç8è�ã�æ{åOõWö÷ü,å(â�æ�ã�çTì�åßã�0÷è�ãOó�ãOè�ætâ�î8ã�èråûæ{ìfÿ�ãpþ8í²ã�0î8ê�é�áßê�â�äTå�â�ä8ã[î8ê�é ò â�áßâ�æ­ø ìfá¦ì á²ê�ætãd�Cã�õ ðTõ�n 7 #yå��� 6ãOäp0åßã�æ�ç8è�ã­å(ãOè�ãé á²ê�â�ä¦êrå6ãì�åßê�ãå.á}êúü�á ò ã­çTì�å(ã*0 è�ã�ó�ã�è+ætâNîNã�èrå,ì�íßãé�â�æ�ç8ê�äTãOî[åßâpá ò ì©á$á ò ã�øWí²ã�á²ï8íßäì�þTíßãîNê�é�áßãîyþ8í²â�çTì�ç8ê�è�ê�á.øüýâ�ípãOì�é ò é�èrì�å²å�õ��^ã ò ìdó�ãôä8â�á�ø�ã�á�ã�)NþDã�í²ê�ætãOä�á²ãOî ñ ê�á òæ�ï8è�áßê,0÷é�è�ì�åßå�ãOäTå(ãOæ­ç8è�ãHå(ãOè�ãé á²ê�â�äAõ

�q���E���k���l�k�������F�� äTåßã�æ­çTè�ã åßã�è�ãOé á²ê�â�ä ïTåßãOåuüýâ�í ñ ìfí¢î å(áßãOþ ñ ê�åßã å(ãOè�ãé áßê�â�äüýí²â�æ1è�ê�ç8í¢ìfí²ê�ãå�âfü�á ò â�ïTå²ìfäRî8å�âfü2ætâNîNã�èrå�á²âpç8ï8ê�è�îôãOäTåßã�æ�0çTè�ãå�á ò ì©á�ìfí²ã6â�þNá²ê�ætê � ãî{áßâûá ò ãzð�ê�ó�ãOä{þRãOí(üýâ�íßæ{ìfäRé�ã6ætã�ál0í²êréfõh4,åßê�ä8ð ì�ó©ì�íßê�ã�á.ø âfüûè�ãOìfí²ä8ê�ä8ð�ìfè�ð�â�íßê�á ò ætå�ìfäTîSþRì'0í¢ìfætã�áßã�í�åßã�á(á²ê�äTð�å"ìfþ8þDãOì�í²å�áßâ�çRã$ã�� ãOé�áßê�ó�ã,üýâ�í�ð�ãOä8ã�í¢ì©á²ê�ä8ðè�ê�ç8í²ì�íßê�ãOå�âfü�îNê�ó�ã�í¢å(ã2 ò ê�ð ò ��ïTìfè�ê�á.ø�ætâNîNãOè�åOõ � äTåßã�æ�ç8è�ã$åßã�0è�ãOé�áßê�â�ä: åUæ�â�å.á�ê�æ�þDâ�íßá²ì�ä�áFüýãOìfáßï8í²ã�êråUá ò ìfá�ê�á2é�ìfä­â�þNá²ê�ætê � ããOäTåßã�æ­çTè�ãHþDã�íßüýâ�í²æ{ìfäTé�ã$áßâpìfä�ø ãOì�åßê�è�ø¦é�â�ætþ8ï8áßãOîôþDã�íßüýâ�íl0æ{ì�äTé�ãûætã�áßí²ê�é�õ � )NþDã�í²ê�ætã�ä�á¢å ñ ê�á ò åßã�ó�ã�äôáßãå.ázþ8í²â�ç8è�ã�æ{å

�����"���*�(�s�*�l�¡ ���¢;�$��£*�¤¢����l��¥w£*¦�¦l§e '¨E�l§a�'�8����¦l�*¢�©�ª,�«¦3��ª,�£�¬��¥, 2�P£� 2��¦3¥w¦3�l�*�i��ª,­%®����e¦��*��¦l�*¢¯©eª°��¦���§��'�� �±p���a���3�¡ 2�l¢²�$ª°ª ��l§e���k¢� p�p�*ªw¦�³a¥°��£�ª°±��p¥°�e´m¢� p�p��ª°¦k�l���$¥°�e����¨q¥,�l§�©��$´2´�¥°�e´�³©s � 2¦3�l¥°�e´�³e������µ¶�b­���¦l¥w�$�m¢� p�p�*ª8�b·��*���$´2¥,��´�¸

¹qº"»½¼F¼�¾�¿kÀFÁÃÂ�Ä�ÅXÆFÁ3Ç3ÀFÁ3¿kÈ�ÉËÊqÀ.¾�Ç�̯Á�ÍkÎ�Ï9ÎaÁ�¾�ÐÑtÒtÒÔÓ ¨«�(±�¦l�¯Õ�Ö;·'�$ª°±e��¦k $�¤×Ë���$�e´2¥,�e´��¡�� �¢Ø×ÚÙÜÛ(�l ×ÝÙ�Þ ßÃà'áiâ`ã�äå�ß*Þ�¸5æ%�F±�¦l�Fç�èkèé¨q¥���§uꤱa£�ª°¥°�p��$�m�p¥w¦3���$�a£�������½ê¤±�£*ª,¥w�p��$�½�p¥°¦3������£��t¨«�*¥°´�§i�����ë©i­ë´2��¥,�½���'�l¥° �¸zæE���ª°¦l �±a¦3�"�p¥w¦����$��£*�k¨«�*¥°´�§i�l��uç�èkè�³p�$���uª, p£*��ª,ª°­�¨V��¥,´2§i�l����b·2�*���$´2¥,�e´a¸�춧e��í2�*���e�*ª$¨q¥w����§�¦:�¡ 2��ª° p£*�$ª°ª°­F¨«�*¥°´�§i�l����b·����3¬��´�¥°�e´�·'�$��­(�¡�� �¢îÕ'ïq�� ¯Õ2ð�ïq��¥,¢���¦¤�l§e�F¢�¥,�e¥°¢�±�¢ñ�p¥w¦3���$�a£��©s����¨«�*���t�$��­���¨V ��s �¥°�i��¦q¥°�1�l§e�"������¥,��¦3�*��¸ò ÒtÒÔÓ ¨V�¯�l���$¥°�%������¦(¨q¥���§E´2���2�p¥°�*�i�"�e��¦�£��*�i��©��2£�íi���l 2������E·'�$��­t�l§e�u�i±�¢�©s�*�( ��«§e¥w�e�p���X±e�e¥,��¦�óiÛ�³5Õp³�ôa³9õ�³8öiÕp³ÛbÕ$õe÷��$������§e��¢� 2¢��*�i�l±e¢²óbøe³Zøe¸ùÕp³Zøe¸ùúp³Zøe¸ ûp÷2¸qæE���p ��:ü �±a¦3�m·'��ª,¥w�e�'��¥, 2�P¦3�*��¦(�l %�e �¨V��¥,´2§i���p��£��b­% 2�(�����lª°­E¦3�l 2�p¬��¥,�e´a¸ÚýÃ��¦������2��³(¨«�P¦��� ��þ�l§��d�e�*��¦��$��¢;�$��­+�e¥�ÿZ�*���*�i����a p£�§�¦q¦l ��l§��$�k¦3 2¢�� ������¦.±e���p���3®e�. 2�¶ '·��*�l®e�¸���;Ó ¨«�¯·'�$��­1�l§e�;¦l�eª°¥��l�l¥°�e´1£*�l¥,�l���l¥° ���³��e�l±��e¥,��´� 2�p�l¥° ���¦�³������¦l¢� � $�l§�¥,�e´����8�$�eªw��£*¥°���h ��Ôµ¶�b­���¦l¥w�$�h¦l¢� � ��l§e¥°�e´��¸æE��±a¦3�«�$ª°ª2 ��p�l§e��FìÔ¢� p�p��ª°¦8¥,��µ«±e�i�l¥°�e��ü ¦�ý�è�\����£�í'�$´2�� µ¶�b­2��¦�³:ý�Fö�³������5ì"³������5ìqø�³��Vôa³������¤³���������������¸æE���$ªw¦3 �´����e�*���'���k���l����¦« $�8��­��a���Vô�ô�� �Vô�¨q¥,�l§��e ��e�l±��p¬¥°�e´!��³"�Vô�ô2µ#�$� �Vô2ôq¨q¥���§(µ¶�b­���¦l¥w�$�"¦3¢� � $��§e¥,��´��³b�$�a�%����¬�&�&�'�(�����P¨q¥,�l§)���$��ª°�2£�¥w�$��¦l¢� � ��l§e¥°�e´��¸����*�*��+¤�� '·� i¦��, k 2¢�¥,�e´2 2¦�³�Õ$ø2ø�ö-�¤�¡ ��.�e��¦�£���¥,�p��¥, 2��¦¶ $�.�Vô�ôa¸/ ò1032 ���;Ó ¨«�u©a�$´�Õ�ú��l���*��¦� $�.���£�§Ô��­��a�2¸�ê��2£�§E�l���*�������¥,�����m 2�1�¯©s i ���¦3�l���$�1¦l��¢���ª,� ¥°¦q�2�e�p��m�� ���§e�"ª,¥°©e���$��­�³�2¦.¨«�*ª°ª8�2¦q�l§��(®�����ª8©a�$´�´2���1�*�a¦3��¢�©eª°���l§a�'� �b·2�*���$´2��¦¶�$ª°ª��§e��¦l�(���l����¦�¸qæz¥���§ /546� 2 ��� ¨«�(©a � i¦�� ����£�§1���l���(��­��a�2¸µ« � 2¦3�l¥°�e´E£*�$�d '·��*�l®e�³5¦l E¨V�1���e�d©a � 2¦3�l��� ìq¦��� ���§e�ª°¥°©e�����l­t�$� �l�*��ó'Õp³Zô�³:õ�³ Û�Öe³�öiÕp³:Ö$ôa³5ÛÕ$õ�³:Õ�ú$Ö�³:úpÛÕe³5Û�ø2Õ$ô�³Õ�ø$ôiõp÷u¦3�l�*�a¦� ��.©a � 2¦3�l¥°�e´a¸�æz¥,�l§ /547� 2 47�98;: ¨«�u±�¦l�¦3�l±�¢��a¦���¦l¥,��´�ª°�Xª,��·��*ª(�e��£�¥w¦l¥, 2�+���l����¦<��¨q¥���§�úë�e¥�ÿZ�*���*�i�¦l�eª°¥��l�l¥°�e´ë£*�l¥,�l���l¥w�e³¶���£�§z©s i i¦�������ó'Õp³qô�³.õ�³�Û�Ö�³¶öiÕp³.Ö�ô�³ÛbÕ$õ�³pÕ2ú$Ö�³�úpÛÕe³ZÛ�ø2Õ$ô�³eÕ$ø�ô2õ�³2ôiø�û�Ö�³põeÛ�ûiÕ�÷"¦����*��¦�¸46=*8?>�Ó ¨V�"±�¦l�"¢� 2¦3�¶í2�*���e�*ªw¦«¥,���!@%���:¥°´�§i�9��A2 2�2£�§e¥°¢�¦�³Ûû�û2û-�uó�ª°¥,�����$�³��s �ª°­��e �¢�¥w�$ªk�e�*´����*��Õ , öe³«���2�p¥w�$ª.¨q¥,�l§¨q¥w����§+óbøe¸ ø�ø�Û�³�ø�¸ ø2ø2úp³Vøe¸ øeÛ�³Vøe¸ ø2úp³«øe¸°Û�³«øe¸ùúp³kÛ2³«Õp÷�÷t�$���·'���l­u�l§e�(���*´2±eªw�$��¥CB�'�l¥° ��1��������¢��*�l����D�©�­�����£��l ���¦q ��9���*��¡�� �¢ Ûø�EGFP�� ÜÛø�H$¸ 춧e�½ �±p���e±p�X�����e´��ë $����@���¦Ô¥w¦ICJ�KML<NOKQP ¥,��¦3�l����ë $� I ø L Û P ¸zì8 E¢;��í��1�l§e����@�� �e����¬�e¥°£��l¥° ���¦.£* �¢����$�l¥°©eª,�"¨q¥,�l§� $��§e�*�q¢� p�p��ª°¦�³e¨«� ±a¦3�9+�ª°�$�3��ü ¦¢��*�l§e p�ë�l P£* ���·����3�1�!@��  2±p�l��±p��¦��� Ô�e�l 2©��$©�¥,ª°¥���¥,�¦�©�­®��3�l¥°�e´���§e�*¢ �l m��¦3¥°´�¢� 2¥°�R��+�ª°�$�3��³�Ûû�û2û-��¸ò*SUT6V&W!X5Y�Z![�\^]_ZVG` >�V§a�$��ª,�¦��V§e¥°±��$���¯ç" �§�¦l±eí��«ç��b¨«��´�±�£�§�¥�§e��ª,�s���¯¨q¥���§���§e�¥°�e¥,�l¥w�$ªq�p�¦3¥°´��ë $�k����¦3��¢�©eª°��¦3��ª,�£��l¥° ���¸EæE�1�l§����eíPì8 ���­

a ±a�$ª,�l¥°�*��¥e�¡ ��V§e�*ª°�m¨q¥,�l§;��§e��bdc%+Vêe� ��+Vê��«ì½�e�'���e³i�$��� 2±e��£� �ª°ªw�$©s ����'�l 2��¦;�$�����&��� �¡ ��1§e�*ª°�é¨q¥,�l§|�l§e�Q���7������'���������f����g �a���3�¡ 2�l¢;�$�a£�� ¢��*�l��¥°£�¸

h ¾6i�¾:Î�¾�¿�jZ¾�е«ªw�$í2��³6�F¸,³ , �t���kB2³l�F¸���Ûû�û2õ-��¸�m%�Vý.���*�s 2¦l¥��� ���­u ���¢;�'¬

£�§e¥°�e�"ª,��$���e¥°�e´;�e�'���$©��2¦3�¦*¸µ«���*¥°¢;�$�:³9��¸n��Ûû�û2Ö-��¸Dµ«��´�´2¥,�e´½�e�����e¥°£��l ���¦�¸po�q�r<s!tvuxw

y wzq"{|u}tvu!~$³6�U�e³8ÛÕ$ö��sÛ�ô2øe¸F¥°���3���*��¥°£�§�³�ì"¸ a ¸#� Õ$ø2ø�ø��¸Xê¤��¦l�*¢¯©eª°�u¢�����§e p�e¦�¥,�\¢;�'¬

£�§e¥°�e�Pª°�����l�e¥°�e´a¸��etv{<�|�5�|u}��w�{|uxq�� t���uGq��)���"{k���ks��<����uof����� t �x�Cw*���Cq��<�|t ��w�{O�x���<��w�����³�Û|�ZÛúe¸

F¥°���3���*��¥°£�§�³�ì"¸ a ¸°³ , µ«��í�¥,��¥ ³ a ¸&��Û�û2û2ú-��¸e�� 2ª,·�¥°�e´�¢�±eª,�l¥,¬£�ªw��¦�¦"ª°���$���e¥°�e´��e�� �©eª°�*¢;¦"·�¥°�1���l�� ��l¬`£* ����l�£���¥,�e´u �±p���e±p�£� p�p�¦*¸.���"��{|uxq��l������{|�(t ��r�t�q"���|u}��w��v��t�~-w�uxr<wd��w|��w<q�{zrzsp³x�2¸

F �¢�¥°�e´� i¦*³�+5¸6��Õ$ø�ø2ø-��¸9µ¶�b­��¦3¥w�$���b·��*���$´2¥,��´� $�5£�ªw��¦�¦3¥,®�����¦�$�a�z�l§��E '·2�*�l®e�3��¥,��´d�e�� �©eª°�*¢�¸¡ �{k�UrU¢$£�¤��¥s¦�|u^�§w�{|uxq"¨� t���uxq"�5�.��u©��¢ª�"u«o�q-rzs�tvuxw y w<q�{|u^tvu~¬���e�:¸kÕ2Õ$ö��eÕ$ö2ø-��¸�t 2�l´i�$��ç���±p�¡¢;�$�e��³}�p�$�f­e���$�a£�¥w¦l£* �³}���(¸

a ±a�$ª,�l¥°�*��¥ ³��(¸°³��V§e�*�3�l��¥�³��Z¸7�(¸,³��V�l 2¢���³7��¸°³ , A2 �§���¦3 2�:³�¤¸6�3Û�û2û�û-��¸��p±e�e�s ��l�¶·2��£��l ��«¢;��£�§e¥°�e��£�ªw��¦�¦l¥�®�����¦«��¦q���p¬�eª°¥,����� F�b·�¥,��¥w¦��e�$���e¸� �{k�UrU¢!®�t�~Us��vsª�"  y ��tv{�¯<�"{|uxwO°dw<��¨��r�t�w�uxr<w����"{k���ks}�z�s¸

A2 2�2£�§e¥°¢�¦�³pì"¸7��Ûû�û2û-��¸�����í�¥,�e´�ªw�$��´��*¬`¦�£*��ª,� ¦3·�¢ ª°�����l�e¥°�e´�e����£��l¥w£*�$ª�¸e��±"²�q"uxr<w��³tvu�´ªw�{|uGw��Go�w��vs}�©±���¸

ç" 2§��b·�¥ ³x��¸°³ , A� 2§e�:³ a ¸Gb�¸���Û�û2ûµ���¸.æE�����e�s�*��¦.�¡ 2�k�¡��'¬�l±��l�(¦l±e©�¦l���.¦l�*ª°��£���¥, 2�:¸e��{|�(t ��r�t�q"�x��u}��w��v��t�~w�uxr<w�³&¶�¤*¸

���*�*íZ³3�F¸°³m춧e¥,�¦l¦l ���³�µ ¸°³ , b.�£�í��*��¢;�$�:³ª¯¸1� Õ$ø2ø2Õ-��¸�^��q�~w<±��nt�·-� ��{kw��ª�U±�w���tvu~nq�uG±%¯<�U���|� tvu~���ì8��£�§e�e¥w£*��ª-�q�*¬�s ��l�#�)���¶¬Ãì#�¶¬ÃÕ�ø�øiÕb¬`ôiú���¸

��±e�e�� �³3+5¸°³ , + �$��¢����i�l �³;µ ¸3�3Û�û�û2Ö-��¸¸�V �¢��a�*�l¥,�l¥° ���$¢� 2�e´t�e����¨« ���íp¦"¥,¢��e�� '·��¦�£� 2¢�¢�¥,�3���*�m�s�*�l�¡ ���¢;�$��£*��¸��±�²�q�uxr<w��¹tvu»º�w���{kq"�5��u©���"{|�ªq"�(t��"u� �{k�Urzw��<�|tvu~��x���<¨��w��n��¸

¼ ��¥��zB�³ª�¸ª��Ûû�û�û��¸½­����$�l±e���ë¦l�*ª°��£��l¥° �� �¡ 2�%����¦3��¢�©eª°��¦�¸�����d��¾!�k���d�%�¡���:¸�öµ$û��pö�õ$ô!��¸

+�ª°�$�3�³�A�¸��3Û�û2û�û-��¸�+¤�� �©���©e¥°ª,¥w¦���¥°£V �±e�l�e±p��¦5�¡ ��¤¦l±e�e�s ��l� ·��£�¬�l 2�¯¢;�2£�§e¥,����¦��$���d£� �¢������l¥w¦l ��X�� %���*´2±eª°���l¥¿B*��Ôª,¥°í���ª,¥,¬§e � p�P¢�����§e p�e¦�¸À��±"²�q"uxr<w��*tvu y q�{(~w�o�q�{(~�tvu;���Cq��z�|tv¨��w�{<�%���e�:¸�Ö�Û|�}µbô��¸

+��l '·2 2¦3��³�­V¸°³ , F �¢�¥°�e´� i¦*³�+5¸�� Õ$ø2ø�ö��¸�ì8�l���u¥°���p±�£��l¥° ���¡ ��¯�e�� �©���©e¥,ª°¥,��­2¬`©��2¦3������$�eí�¥°�e´2¦�¸Ro�q�r<s!tvuxw y w<q�{|u^tvu~�³Á �2¸

�e£�§��$�e¥°����³e��¸d��Õ$ø2øeÛU��¸d춧e��©s i i¦���¥,��´E���e�e�� 2��£�§P�l Ô¢;�'¬£�§e¥°�e��ª,��$���e¥°�e´} ��.�\ '·��*��·�¥,��¨�¸_�|uÀo��!�d�À�f��{k���ks��<���u�º��"u}��tvuGwzq"{�®��|�(tv�5q�� t���uQq�uG±f���Cq��z�|t ��rzq"�(t��"u�¸

Â�Ã�ÄCÄ¿Å¿Æ�Ç�È6ÉeÊ^ËCÉ�Ì7Ç"È}Í"Î¥Ï�Ð<ÑlÉ.Ê^ËCÉ�Ò�Ç�ÐkÃ^Ç"È}Ç}É.ÓOËCÉ�ÔÖÕ�Ä¿Ã�×�É�Ø�ËÙ(Ú�Û�Û-Û-Ü ËÞÝ�ßUÇ�àkÃ}Ðkß©á^Ï!Ï-âãà©ä)ØÖתß�àzÇ"å Ä¿ß©Ç�ÐkÈ}ÅCÈ�ÍQÇ�ÄCÍ-Ï�ÐzÅ�àzæ�×àkæ^Ç�àdÅ¿×$ç}ÐkÏ�Æ-ß©â�תÏ!Ñ�ß�Ä7ÐzÏ�á}Ã}â�àzÈ�ß©âzâ©Ë.è�ékê©ë<ì<ì<í�îvï!ð�ñnêãòOóvô}ìõGì�ö�ì�ï^ó§ì<ì�ï}ó¥ôø÷�ï}ó�ì�é|ïGù�ó î�ê�ïGù�úªû.ê�ï©ò�ì�ékì�ïxë<ì_ê"ï«ü�ù�ë<ô!îvïxìý ìzù"é|ï}îvï!ð"Ë

þ Ï�Ä¿çxß�Ðkà©Élÿ�Ë���Ë Ù������Ú�Ü Ë�ÂàzÇ���ßUÑfÍ�ß©È�ß�Ð<Ç"Ä¿Å��UÇ�àzÅCÏ^Ë� �ì���ékù�ú �ì�ó��eê�é���ñ<É���É Ú�������Ú���� Ë

Getting the Most Out of Ensemble Selection∗

Rich Caruana, Art Munson, Alexandru Niculescu-MizilDepartment of Computer Science

Cornell UniversityTechnical Report 2006-2045

{caruana, mmunson, alexn} @cs.cornell.edu

Abstract

We investigate four previously unexplored aspects of en-semble selection, a procedure for building ensembles ofclassifiers. First we test whether adjusting model predic-tions to put them on a canonical scale makes the ensemblesmore effective. Second, we explore the performance of en-semble selection when different amounts of data are avail-able for ensemble hillclimbing. Third, we quantify the ben-efit of ensemble selection’s ability to optimize to arbitrarymetrics. Fourth, we study the performance impact of prun-ing the number of models available for ensemble selection.Based on our results we present improved ensemble selec-tion methods that double the benefit of the original method.

1 Introduction

An ensemble is a collection of classifiers whose predic-tions are combined with the goal of achieving better perfor-mance than the constituent classifiers. A large body of re-search now exists showing that ensemble learning often in-creases performance (e.g. bagging [3], boosting [21], stack-ing [25]).

Recently, ensemble selection [7] was proposed as atechnique for building ensembles from large collections ofdiverse classifiers. Ensemble selection employs greedy for-ward selection to select models to add to the ensemble, amethod categorized in the literature as overproduce andchoose [20]. Compared to previous work, ensemble selec-tion uses many more classifiers, allows optimizing to ar-bitrary performance metrics, and includes refinements toprevent overfitting to the ensemble’s training data—a largerproblem when selecting from more classifiers.

In this paper we analyze four previously unexplored as-pects of ensemble selection. First, we evaluate ensemble

∗This technical report is an expanded version of a paper accepted at the2006 International Conference on Data Mining.

selection’s performance when all the models are calibratedto place their predictions on a canonical scale. Makingcalibrated models available to ensemble selection providessignificant improvement on probability measures such assquared error and cross-entropy. It appears, however, thatcalibration does not make ensemble selection itself moreeffective; most of the benefit results from improvements inthe base-level models and not from better ensemble build-ing.

Second, we explore how ensemble selection behaveswith varying amounts of training data available for the crit-ical forward selection step. Despite previous refinementsto avoid overfitting the data used for ensemble hillclimb-ing [7], our experiments show that ensemble selection isstill prone to overfitting when the hillclimb set is small.This is especially true if their model bagging procedure isnot used. Surprisingly, although ensemble selection over-fits with small data, reliably picking a single good modelis even harder—making ensemble selection more valuable.With enough hillclimbing data (around 5k points), overfit-ting becomes negligible. Motivated by these results, wepresent a method for embedding cross-validation inside en-semble selection to maximize the amount of hillclimbingdata.1 Cross-validation boosts the performance of ensembleselection, doubling its previously reported benefit. Whileadding cross-validation to ensemble selection is computa-tionally expensive, it is valuable for domains that requirethe best possible performance, and for domains in whichlabeled data is scarce.

Ensemble selection’s ability to optimize to any perfor-mance metric is an attractive capability of the method thatis particularly useful in domains which use non-traditionalperformance measures such as natural language process-ing [14]. Because of this, the third aspect we investigateis what benefit, if any, comes from being able to opti-mize to any metric. Our experiments reinforce the intuitionthat it is best to optimize to the target performance metric;

1This is different from wrapping cross-validation around ensemble se-lection, which would not increase the data available for hillclimbing.

however, they also show that minimizing squared error orcross-entropy frequently yields ensembles with competitiveperformance—seemingly regardless of the metric.

Fourth, we test ensemble selection’s performance whenonly the best X% models are available for selection. Theseexperiments confirm our intuition that the potential for over-fitting increases with more models. Using only the top 10-20% of the models yields performance better than or equiv-alent to ensemble selection without this model pruning.

2 Background

In this section we briefly review the ensemble selectionprocedure first proposed by Caruana et al. [7]. Ensemble se-lection is an overproduce and select ensemble method car-ried to an extreme where thousands of models are trainedusing many different learning methods, and overfitting ismoderated by applying several techniques.

In ensemble selection, models are trained using as manylearning methods and control parameters as can be appliedto the problem. Little or no attempt is made to optimizethe performance of the individual models; all models, nomatter what their performance, are added to the model li-brary for the problem. The expectation is that some of themodels will yield good performance on the problem, eitherin isolation or in combination with other models, for anyreasonable performance metric.

Once the model library is collected, an ensemble is builtby selecting from the library the subset of models thatyield the best performance on the target optimization met-ric. Models are selected for inclusion in the ensemble usinggreedy forward stepwise model selection. The performanceof adding a potential model to the ensemble is estimated us-ing a hillclimbing set containing data not used to train themodels. At each step ensemble selection adds to the ensem-ble the model in the library that maximizes the performanceof the ensemble to this held-aside hillclimbing data.

When there are thousands of models to select from, thechances of overfitting increase dramatically. Caruana et al.describe two methods to combat overfitting. The first con-trols how ensembles are initialized. The second performsmodel bagging—analogous to feature bagging [1, 5]—to re-duce the variance of the selection process.

Ensemble Initialization: Instead of starting with anempty ensemble, Caruana et al. suggest initializing ensem-bles with the N models that have the best uni-model perfor-mance on the hillclimb set and performance metric.

Bagged Ensemble Selection: It is well known that fea-ture subset selection (e.g. forward stepwise feature selec-tion) is unstable when there are many relevant features [2].Ensemble selection is like feature selection, where modelsare features and model subsets are found by forward step-wise selection. Because of this, ensemble selection also has

high variance. Ensemble selection uses bagging over mod-els to reduce this variance. Multiple ensembles are builtfrom random subsets of the models, and then averaged to-gether. This is analogous to the feature bagging methodsproposed by Bay [1] and Bryll et al. [5] and used in randomforests [4].

Another technique used in the original paper is allowingmodels to be added to the ensemble more than once. Thisprovides two benefits. First, models added multiple timesget more weight in the ensemble average. Second, whenmodels are added without replacement, ensemble perfor-mance deteriorates quickly after the best models have beenexhausted because poorer models must then be added. Thismakes deciding when to stop adding models to the ensemblecritical because overshooting the optimal stopping point canyield much worse performance. Selection with replacementallows selection to continue adding copies of good mod-els instead of being forced to add inferior models. This, inturn, makes deciding when to stop adding models far lesscritical. All of the experiments in this paper use these threetechniques.

3 Methodology

We use all of the learning methods and data sets used byCaruana et al. [7], and all of the performance metrics exceptCAL (a probability calibration metric) and SAR (a metricthat combines accuracy, squared error, and ROC area). Inaddition, we also train models with logistic regression (LO-GREG), naıve bayes (NB), and random forests (RF) [4], andexperiment with four additional data sets: MG, CALHOUS,COD, and BACT. All of the data sets are binary classifi-cation problems. The learning methods and data sets aredescribed in Appendix A and B, respectively. The per-formance metrics we study are described in the followingsubsection.

3.1 Performance Metrics

The eight performance metrics we use can be dividedinto three groups: threshold metrics, ordering/rank metricsand probability metrics.

The threshold metrics are accuracy (ACC), F-score(FSC) and lift (LFT). For thresholded metrics, it is not im-portant how close a prediction is to a threshold, only if it isabove or below threshold. See Giudici [10] for a descriptionof Lift Curves. Usually ACC and FSC have a fixed thresh-old (we use 0.5). For lift, often a fixed percent, p, of casesare predicted as positive and the rest as negative (we usep = 25%).

The ordering/rank metrics depend only on the orderingof the cases, not the actual predicted values. As long asordering is preserved, it makes no difference if predicted

2

Table 1. Performance with and without model calibration. The best score in each column is bolded.ACC FSC LFT ROC APR BEP RMS MXE MEAN

ES-BOTH 0.920 0.888 0.967 0.982 0.972 0.964 0.932 0.944 0.946ES-PREV 0.922 0.893 0.967 0.981 0.966 0.965 0.919 0.932 0.943ES-NOCAL 0.919 0.897 0.967 0.982 0.970 0.965 0.912 0.925 0.942ES-CAL 0.912 0.847 0.969 0.981 0.969 0.966 0.935 0.940 0.940BAYESAVG-BOTH 0.893 0.814 0.964 0.978 0.963 0.956 0.918 0.934 0.928BAYESAVG-CAL 0.889 0.820 0.962 0.977 0.960 0.955 0.912 0.925 0.925MODSEL-BOTH 0.871 0.861 0.939 0.973 0.948 0.938 0.901 0.916 0.918BAYESAVG-PREV 0.881 0.789 0.956 0.970 0.956 0.947 0.893 0.911 0.913MODSEL-PREV 0.872 0.860 0.939 0.973 0.948 0.938 0.879 0.892 0.913MODSEL-CAL 0.870 0.819 0.943 0.973 0.948 0.940 0.892 0.910 0.912MODSEL-NOCAL 0.871 0.858 0.939 0.973 0.948 0.938 0.861 0.871 0.907BAYESAVG-NOCAL 0.875 0.784 0.955 0.968 0.953 0.941 0.874 0.892 0.905

values fall between 0 and 1 or 0.89 and 0.90. These metricsmeasure how well the positive cases are ordered before neg-ative cases and can be viewed as a summary of model per-formance across all possible thresholds. The rank metricswe use are area under the ROC curve (ROC), average pre-cision (APR), and precision/recall break even point (BEP).See Provost and Fawcett [19] for a discussion of ROC froma machine learning perspective.

The probability metrics are minimized (in expectation)when the predicted value for each case coincides with thetrue conditional probability of that case being positive class.The probability metrics are squared error (RMS) and cross-entropy (MXE).

3.2 Comparing Across Performance Metrics

To permit averaging across metrics and problems, per-formances must be placed on comparable scales. FollowingCaruana et al. [7] we scale performance for each problemand metric from 0 to 1, where 0 is baseline performanceand 1 is the best performance achieved by any model or en-semble. We use the following baseline model: predict p forevery case, where p is the percent of positives in the data.

One disadvantage of normalized scores is that recoveringa raw performance requires knowing what performances de-fine the top and bottom of the scale, and as new best modelsare found the top of the scale may change. Note that thenormalized scores presented here differ from those reportedin Caruana et al. [7] because we are finding better modelsthat shift the top of the scales. The numbers defining thenormalized scales can be found in Appendix C.

4 Ensembles of Calibrated Models

Models trained by different learning algorithms do notnecessarily “speak the same language”. A prediction of

0.14 from a neural net does not necessarily mean the samething as a prediction of 0.14 from a boosted tree or SVM.Predictions from neural nets often are well-calibrated pos-terior probabilities, but predictions from SVMs are just nor-malized distances to the decision surface. Averaging pre-dictions from models that are not on commensurate scalesmay hurt ensemble performance.

In this section we evaluate the performance of ensem-ble selection after “translating” all model predictions to thecommon “language” of well-calibrated posterior probabili-ties. Learning algorithms such as boosted trees and stumps,SVMs, or naıve bayes have poorly calibrated predictions[15]. A number of methods have been proposed for map-ping predictions to posterior probabilities. In this paper weadopt the method Platt developed for SVMs [18], but whichalso works well for other learning algorithms [15]. Platt’smethod transforms predictions by passing them through asigmoid whose parameters are learned on an independentcalibration set. In this paper, the ensemble selection hill-climb set is used for calibration as well.

Table 1 shows the performance of ensemble selection(ES), model selection (MODSEL),2 and Bayesian modelaveraging (BAYESAVG) [8], with and without calibratedmodels. Results are shown for four different model li-braries: 1) only uncalibrated models (NOCAL), 2) only cal-ibrated models (CAL), 3) both calibrated and uncalibratedmodels (BOTH), and 4) only SVMs are calibrated, to mimicprior experiments [7] (PREV). Each entry is the average offive folds on each of the eleven problems. The last columnshows the mean performance over all eight metrics. Rowsare sorted by mean performance.

Comparing results for ensemble selection with and with-out calibration (ES-CAL and ES-NOCAL), we see that cali-brating models improves RMS and MXE (significant at .05)but hurts FSC. There is little difference for LFT, ROC, APR

2Model selection chooses the best single model using the hillclimb set.

3

and BEP. For model selection we see the same trends: cali-brated models yield better RMS and MXE and worse FSC.The magnitudes of the differences suggest that most if notall of the improvement in RMS and MXE for ensemble se-lection with calibrated models is due to having better mod-els in the library rather than from ensemble selection takingadvantage of the common scale of the calibrated models.We are not sure why calibration makes FSC performanceworse for both MODSEL and ES, but again suspect that thedifferences between ES-CAL and ES-NOCAL are due todifferences in the performance of the base-level models.

Having both calibrated and uncalibrated models in thelibrary (ES-BOTH and MODSEL-BOTH) gives the best ofboth worlds: it alleviates the problem with FSC while re-taining the RMS and MXE improvements. For the rest ofthe experiments in this paper we use libraries containingboth calibrated and uncalibrated models.

Unlike with ensemble selection, using calibrated modelsfor Bayesian model averaging improves performance on allmetrics, not just RMS and MXE (significant at .05). Withcalibrated models, Bayesian averaging outperforms modelselection but is still not as good as ensemble selection.

5 Analysis of Training Size

The original ensemble selection paper demonstrated themethod’s effectiveness using relatively small hillclimbingsets containing 1000 data points. Since the data used forhillclimbing is data taken away from training the individ-ual models, keeping the hillclimb set small is important.Smaller hillclimb sets, however, are easier to overfit to, par-ticularly when there are many models from which to select.

To explore ensemble selection’s sensitivity to the size ofthe hillclimb set, we ran ensemble selection with hillclimbsets containing 100, 250, 500, 1000, 2500, 5000, and 10000data points. In each run we randomly selected the points forthe hillclimb set and used the remainder for the test set. Thehyperspectral and medis data sets contained too few pointsto leave sufficient test sets when using a 10K hillclimbingset and were omitted. Due to time constraints and the costof generating the learning curves, we only used one randomsample at each size and did not repeat the experiment.

Figure 1 shows learning curves for our eight perfor-mance measures and their mean. Each graph is an averageover 9 problems. The x-axis uses a logscale to better showwhat happens with small hillclimbing sets. Normalized per-formance scores are plotted on the y-axis. For comparison,the graphs include the performance achieved by picking thesingle best model (MODSEL).

Unsurprisingly, the performance achieved with both en-semble selection and model selection using only 100 pointsfor hillclimbing is quite bad. As data increases, both meth-ods do better as they overfit less. Interestingly, ensemble

selection is hurt less by a small hillclimbing set than modelselection, suggesting that it is less prone to overfitting thanmodel selection. Because of this, the benefit of ensembleselection over the best models appears to be strongest whentraining data is scarce, a regime [7] did not examine. (Theyused 5k training data with 1k points held aside for ensemblestepwise selection.) As the size of the hillclimbing sets goesfrom 1k to 10k, ensemble selection maintains its edge overmodel selection.

With small hillclimb sets, using bagging with ensembleselection is crucial to getting good performance; withoutit, mean performance using a 100 point hillclimb set dropsfrom 0.888 to 0.817. In contrast, bagging provides verylittle if any benefit when a very large hillclimb set is used(more than 5000 points with our data sets).

6 Cross-Validated Ensemble Selection

It is clear from the results in Section 5 that the larger thehillclimb set, the better ensemble selection’s performancewill be. To maximize the amount of available data, we applycross-validation to ensemble selection. Simply wrappingcross-validation around ensemble selection, however, willnot help because the algorithm will still have just a fractionof the training data available for hillclimbing. Instead, weembed cross-validation within ensemble selection so that allof the training data can be used for the critical ensemblehillclimbing step. Conceptually, the procedure makes cross-validated models, then runs ensemble selection the usualway on a library of cross-validated base-level models.

A cross-validated model is created by training a modelfor each fold with the same model parameters. If there are5 folds, there will be 5 individual models (each trained on4000 points) that are ‘siblings’; these siblings should onlydiffer based on variance due to their different training sam-ples. To make a prediction for a test point, a cross-validatedmodel simply averages the predictions made by each of thesibling models. The prediction for a training point (that sub-sequently will be used for ensemble hillclimbing), however,only comes from the individual model that did not see thepoint during training. In essence, the cross-validated modeldelegates the prediction responsibility for a point that willbe used for hillclimbing to the one sibling model that is notbiased for that point.

Selecting a cross-validated model, whether during modelselection or ensemble selection, means choosing all of thesibling models as a unit. If 5-fold cross-validation is used,selection chooses groups containing 5 sibling models at atime. In this case, when selection adds a cross-validatedmodel to a growing ensemble, it really adds 5 different mod-els of the same model type to the ensemble, each of whichreceives the same weight in the ensemble average.

We ran ensemble selection with 5-fold cross-validation;

4

0.5 0.55 0.6

0.65 0.7

0.75 0.8

0.85 0.9

0.95 1

100 1000 10000

nor

mal

ized

per

form

ance

hillclimb data size

ACC

baggingno bagging

modsel 0.55 0.6

0.65 0.7

0.75 0.8

0.85 0.9

0.95 1

100 1000 10000

nor

mal

ized

per

form

ance

hillclimb data size

FSC

baggingno bagging

modsel 0.84 0.86 0.88 0.9

0.92 0.94 0.96 0.98

1

100 1000 10000

nor

mal

ized

per

form

ance

hillclimb data size

LFT

baggingno bagging

modsel

0.84 0.86 0.88 0.9

0.92 0.94 0.96 0.98

1

100 1000 10000

nor

mal

ized

per

form

ance

hillclimb data size

ROC

baggingno bagging

modsel 0.82 0.84 0.86 0.88 0.9

0.92 0.94 0.96 0.98

1

100 1000 10000

nor

mal

ized

per

form

ance

hillclimb data size

APR

baggingno bagging

modsel 0.82 0.84 0.86 0.88 0.9

0.92 0.94 0.96 0.98

100 1000 10000

nor

mal

ized

per

form

ance

hillclimb data size

BEP

baggingno bagging

modsel

0.55 0.6

0.65 0.7

0.75 0.8

0.85 0.9

0.95 1

100 1000 10000

nor

mal

ized

per

form

ance

hillclimb data size

RMS

baggingno bagging

modsel 0.4

0.5

0.6

0.7

0.8

0.9

1

100 1000 10000

nor

mal

ized

per

form

ance

hillclimb data size

MXE

baggingno bagging

modsel 0.7

0.75

0.8

0.85

0.9

0.95

1

100 1000 10000 n

orm

aliz

ed p

erfo

rman

ce

hillclimb data size

MEAN

baggingno bagging

modsel

Figure 1. Learning curves for ensemble selection with and without bagging, and for picking the bestsingle model (modsel).

Table 2. Performance with and without cross-validation for ensemble selection and model selection.ACC FSC LFT ROC APR BEP RMS MXE MEAN

ES-BOTH-CV 0.935 0.926 0.982 0.996 0.992 0.977 0.984 0.989 0.973MODSEL-BOTH-CV 0.907 0.923 0.971 0.985 0.968 0.963 0.945 0.961 0.953ES-BOTH 0.920 0.888 0.967 0.982 0.972 0.964 0.932 0.944 0.946MODSEL-BOTH 0.871 0.861 0.939 0.973 0.948 0.938 0.901 0.916 0.918

this is analogous to normal ensemble selection with a 5000point hillclimb set. Table 2 shows the results averaged overall the problems. Not only does cross-validation greatly im-prove ensemble selection performance, it also provides thesame benefit to model selection. Five-fold cross-validatedmodel selection actually outperforms non-cross-validatedensemble selection by a small but noticeable amount. How-ever, ensemble selection with embedded cross-validationcontinues to outperform model selection.

Table 3 provides a different way to look at the results.The numbers in the table (except for the last row) are thepercent reduction in loss of cross-validated ensemble se-lection, relative to non-cross-validated model selection, thebaseline used in Caruana et al. [7]. For example, if model

selection achieves a raw accuracy score of 90%, and cross-validated ensemble selection achieves 95% accuracy, thenthe percent reduction in loss is 50%—the loss has been re-duced by half. The MEAN row is the average improvementfor each metric, across datasets. For comparison, the PREVrow is the performance of the original non-cross-validatedensemble selection method (i.e. no cross-validation andonly SVMs are calibrated).

Embedding cross-validation within ensemble selectiondoubles its benefit over simple model selection (from 6.90%to 12.77%). This is somewhat of an unfair comparison; if across-validated model library is available, it is just as easyto do cross-validated model selection as it is to do cross-validated ensemble selection. The last row in Table 3 shows

5

Table 3. Percent loss reduction by dataset.

ACC FSC LFT ROC APR BEP RMS MXE MEANADULT 2.77 5.89 8.72 7.45 6.70 7.58 2.26 4.08 5.68BACT 2.08 3.83 16.42 4.13 5.49 1.76 1.42 4.15 4.91CALHOUS 7.95 9.49 48.00 8.69 8.81 6.15 7.17 12.74 13.63COD 5.73 7.46 14.33 9.14 10.52 7.11 2.39 3.79 7.56COVTYPE 6.68 7.26 12.35 11.34 14.99 7.64 7.80 12.92 10.12HS 13.66 16.36 12.32 37.53 37.78 16.77 12.65 27.43 21.81LETTER.p2 15.21 14.50 100.00 32.84 33.05 15.85 17.13 29.47 32.26LETTER.p1 21.55 25.66 0.29 69.10 45.29 19.25 19.59 34.58 29.41MEDIS 2.77 -0.05 2.08 6.33 7.28 4.62 1.40 2.70 3.39MG 4.45 1.98 4.25 11.84 12.65 6.04 2.57 6.10 6.23SLAC 2.49 3.27 13.65 6.92 9.62 2.73 1.66 3.33 5.46MEAN 7.76 8.70 21.13 18.67 17.47 8.68 6.91 12.84 12.77PREV 4.96 4.56 16.22 8.43 6.24 5.15 3.27 6.39 6.90MEANcv 2.89 3.07 10.82 9.97 9.37 2.84 2.54 4.22 5.71

the percent loss reduction of cross-validated ensemble se-lection compared to cross-validated model selection. Com-paring PREV and MEANcv, we see that after embeddingcross-validation, ensemble selection provides slightly lessbenefit over model selection than un-cross-validated ensem-ble selection did over un-cross validated model selection.

While training five times as many models is computa-tionally expensive, it may be useful for domains where thebest possible performance is needed. Potentially more in-teresting, in domains where labeled data is scarce, cross-validated ensemble selection is attractive because a) it doesnot require sacrificing part of the training data for hillclimb-ing, b) it maximizes the size of the hillclimbing set (whichFigure 1 shows is critical when hillclimb data is small), andc) training the cross-validated models is much more feasiblewith smaller training data.

7 Direct Metric Optimization

One interesting feature of ensemble selection is its abil-ity to build an ensemble optimized to an arbitrary metric. Totest how much benefit this capability actually provides, wecompare ensemble selection that optimizes the target met-ric with ensemble selection that optimizes a predeterminedmetric regardless of the target metric. For each of the 8metrics, we train an ensemble that optimizes it and evaluatethe performance on all metrics. Optimizing RMS or MXEyields the best results.

Table 4 lists the performance of ensemble selection for a)always optimizing to RMS, b) always optimizing to MXE,and c) optimizing the true target metric (OPTMETRIC).When cross-validation is not used, there is modest benefitto optimizing to the target metric. With cross-validation,however, the benefit from optimizing to the target metric is

Table 4. Performance of ensemble selectionwhen forced to optimize to one set metric.

RMS MXE OPTMETRICES-BOTH-CV 0.969 0.968 0.973ES-BOTH 0.935 0.936 0.946

significantly smaller.The scatter plots in Figure 2 plot the performance of op-

timizing to RMS against the performance of optimizing toOPTMETRIC, with one graph per target metric. Again, wecan see that ensemble selection performs somewhat betterwith OPTMETRIC. Always optimizing RMS is frequentlyvery competitive, especially when performance gets closeto a normalized score of 1. This is why the benefit of directmetric optimization is so small for cross-validated ensem-ble selection. These results suggest that optimizing RMS(or MXE) may be a good alternative if the target metric istoo expensive to use for hillclimbing.

8 Model Library Pruning

Including a large number of base level models, with awide variety of parameter settings, in the model libraryhelps ensure that at least some of the models will havegood performance regardless of the metric optimized. Atthe same time, increasing the number of available modelsalso increases the risk of overfitting the hillclimb set. More-over, some of the models have such poor performance thatthey are unlikely to be useful for any metric one would wantto optimize. Eliminating these models should not hurt per-formance, and might help.

In this section we investigate ensemble selection’s per-

6

1

1 0.57

ACC 1

1 0.24

FSC 1

1 0.89

LFT 1

1 0.92

ROC

1

1 0.86

APR 1

1 0.89

BEP 1

1 0.82

RMS 1

1 0.84

MXE

1

1 0.75

MEAN

Figure 2. Scatter plots of ensemble selection performance when RMS is optimized (x-axis) vs whenthe target metric is optimized (y-axis). Points above the line indicate better performance by optimizingto the target metric (e.g. accuracy) than when optimizing RMS. Each point represents a different dataset; circles are averages for a problem over 5 folds, and X’s are performances using cross-validation.Each metric (and the mean across metrics) is plotted separately for clarity.

formance when employing varying levels of library prun-ing. The pruning works as follows: the models are sortedby their performance on the target metric (with respect tothe hillclimb set), and only the top X% of the models areused for ensemble selection. Note that this pruning is dif-ferent from work on ensemble pruning [12, 22, 23, 26, 13].This is a pre-processing method, while ensemble pruningpost-processes an existing ensemble.

Figure 3 shows the effect of pruning for each perfor-mance metric, averaged across the 11 data sets and 5folds using non-cross-validated ensemble selection withand without bagging. For comparison, flat lines illustratethe performance achieved by model selection (modsel) andnon-pruned ensemble selection (es-both). The legend isshown in the ACC graph.

The figure clearly shows that pruning usually does nothurt ensemble selection performance, and often improvesit. For ACC, LFT, and BEP pruned ensemble selection(the line with boxes) seems to yield the same performanceas non-pruned ensemble selection . For the other metrics,pruning yields superior performance. Indeed, when usingmore than 50% of the models performance decreases. In-terestingly, library pruning reduces the need for bagging,presumably by reducing the potential for overfitting.3

3The bagging line at 100% does not always match the es-both line,even though these should be equivalent configurations. This is particularlyevident for FSC, the highest variance metric. The sorting performed beforepruning alters ensemble selection’s model sampling, resulting in additional

The graphs in Figure 3 show the average behavior acrossour 11 data sets. Ensemble selection’s behavior under prun-ing may in fact vary when each data set is considered in-dividually. Averaging across problems could hide differentpeak points. Figure 4 shows RMS performance for each ofthe problems.

Although performance starts to decline at different prun-ing levels for the different problems, it is clear that largermodel libraries increase the risk of overfitting the hillclimbset. Using 100% of the models is never worthwhile. Atbest, using the full library can match the performance of us-ing only a small subset. In the worst case, ensemble selec-tion overfits. This is particularly evident for the COD dataset where model selection outperforms ensemble selectionunless pruning is employed.

While further work is needed to develop good heuristicsfor automatically choosing an appropriate pruning level fora data set, simply using the top 10–20% models seems to bea good rule of thumb. An open problem is finding a betterpruning method. For example, taking into account modeldiversity (see for example [11, 17]) might find better prunedsets.

variance.

7

0.87 0.875 0.88

0.885 0.89

0.895 0.9

0.905 0.91

0.915 0.92

0.925

0 20 40 60 80 100

norm

aliz

ed p

erfo

rman

ce

% models

ACC

baggingno bagging

es-bothmodsel-both

0.86 0.865 0.87

0.875 0.88

0.885 0.89

0.895 0.9

0.905

0 20 40 60 80 100

norm

aliz

ed p

erfo

rman

ce

% models

FSC

0.935 0.94

0.945 0.95

0.955 0.96

0.965 0.97

0 20 40 60 80 100

norm

aliz

ed p

erfo

rman

ce

% models

LFT

0.972 0.974 0.976 0.978 0.98

0.982 0.984 0.986 0.988 0.99

0 20 40 60 80 100

norm

aliz

ed p

erfo

rman

ce

% models

ROC

0.945 0.95

0.955 0.96

0.965 0.97

0.975 0.98

0 20 40 60 80 100

norm

aliz

ed p

erfo

rman

ce

% models

APR

0.935 0.94

0.945 0.95

0.955 0.96

0.965 0.97

0 20 40 60 80 100

norm

aliz

ed p

erfo

rman

ce

% models

BEP

0.9

0.91

0.92

0.93

0.94

0.95

0.96

0 20 40 60 80 100

norm

aliz

ed p

erfo

rman

ce

% models

RMS

0.915 0.92

0.925 0.93

0.935 0.94

0.945 0.95

0.955 0.96

0.965

0 20 40 60 80 100

norm

aliz

ed p

erfo

rman

ce

% models

MXE

0.915 0.92

0.925 0.93

0.935 0.94

0.945 0.95

0.955

0 20 40 60 80 100no

rmal

ized

per

form

ance

% models

MEAN

Figure 3. Pruned ensemble selection performance.

9 Discussion

In this section we further analyze the benefit of embed-ding cross validation within ensemble selection and alsobriefly describe other work we are doing to make ensem-ble selection models smaller and faster.

9.1 Benefits of Cross-Validation

The results in Section 6 show that embedding cross-validation within ensemble selection significantly increasesthe performance of ensemble selection. There are two fac-tors that could explain this increase in performance. First,the bigger hillclimbing set could make selecting models toadd to the ensemble more reliable and thus make overfit-ting harder. Second, averaging the predictions of the siblingmodels could provide a bagging-like effect that improvesthe performance of the base-level models. To tease apartthe benefit due to each of these factors we perform two ad-ditional experiments.

In one experiment, we use the same hillclimbing set ascross-validated ensemble selection, but instead of averag-

ing the predictions of the sibling models, we use only thepredictions of one of the siblings. Using this procedure weconstruct five ensemble models, one for each fold, and re-port their mean performance. This provides a measure ofthe benefit due to the increase in the size of the hillclimb set(from cross-validation) while eliminating the bagging-likeeffect due to sibling model averaging.

In the other experiment, we use the smaller hillclimb setsused by un-cross-validated ensemble selection, but we doaverage the predictions of the sibling models. We againconstruct five ensemble models, one for each fold, and re-port their mean performance. This allows us to identify theperformance increase due to the bagging-like effect of aver-aging the predictions of the sibling models.

Table 5 shows the results of these experiments. Entries inthe table show the improvement provided by using a largerhillclimb set (ES-HILL) and by averaging the sibling mod-els (ES-AVG) as a percentage of the total benefit of cross-validated ensemble selection. For example, looking at theACC column, increasing the size of the hillclimb set from1k to 5k yields a benefit equal to 32.9% of the total benefitprovided by cross-validated ensemble selection, and aver-

8

0.94 0.945 0.95

0.955 0.96

0.965 0.97

0.975 0.98

0 20 40 60 80 100

norm

aliz

ed p

erfo

rman

ce

% models

ADULT: RMS

0.87 0.88 0.89 0.9

0.91 0.92 0.93 0.94 0.95 0.96 0.97

0 20 40 60 80 100

norm

aliz

ed p

erfo

rman

ce

% models

BACT: RMS

0.93 0.935 0.94

0.945 0.95

0.955 0.96

0.965 0.97

0.975 0.98

0.985

0 20 40 60 80 100

norm

aliz

ed p

erfo

rman

ce

% models

CALHOUS: RMS

baggingno bagging

es-bothmodsel-both

0.87 0.88 0.89 0.9

0.91 0.92 0.93 0.94

0 20 40 60 80 100

norm

aliz

ed p

erfo

rman

ce

% models

COD: RMS

0.905 0.91

0.915 0.92

0.925 0.93

0.935 0.94

0.945

0 20 40 60 80 100

norm

aliz

ed p

erfo

rman

ce

% models

COVTYPE: RMS

0.88 0.89 0.9

0.91 0.92 0.93 0.94 0.95 0.96 0.97 0.98

0 20 40 60 80 100

norm

aliz

ed p

erfo

rman

ce

% models

HS: RMS

0.88 0.89 0.9

0.91 0.92 0.93 0.94 0.95 0.96

0 20 40 60 80 100

norm

aliz

ed p

erfo

rman

ce

% models

LETTER.p1: RMS

0.945

0.95

0.955

0.96

0.965

0.97

0 20 40 60 80 100

norm

aliz

ed p

erfo

rman

ce

% models

LETTER.p2: RMS

0.76 0.78 0.8

0.82 0.84 0.86 0.88 0.9

0.92 0.94

0 20 40 60 80 100no

rmal

ized

per

form

ance

% models

MEDIS: RMS

0.86

0.87

0.88

0.89

0.9

0.91

0.92

0 20 40 60 80 100

norm

aliz

ed p

erfo

rman

ce

% models

MG: RMS

0.9 0.91 0.92 0.93 0.94 0.95 0.96 0.97

0 20 40 60 80 100

norm

aliz

ed p

erfo

rman

ce

% models

SLAC: RMS

Figure 4. RMS performance for pruned ensemble selection.

aging the sibling models yields a benefit equal to 80.5%.The third row in the table is the sum of the first two rows.

If the sum is lower than 100% the effects from ES-HILLand ES-AVG are super-additive, i.e. combining the two ef-fects provides more benefit than the sum of the individualimprovements. If the sum is higher than 100% then the twoeffects are sub-additive. For ACC, the sum is 113.4%, indi-cating that the effects of these two factors are sub-additive:the total performance is slightly less than would be expectedif the factors were independent. Except for the high vari-ance metrics, FSC and ACC, the sums are close to 100%,indicating that the two effects are nearly independent.

The learning curves in Figure 1 suggest that increas-

ing the size of the hillclimb set from 1k to 5k would ex-plain almost all of the benefit of cross-validation. Theseresults, however, show that on average across the eight met-rics the benefit from ES-HILL and ES-AVG are roughlyequal. About half of the benefit from embedding cross-validation within ensemble selection appears to result fromthe increase in the size of the hillclimb set, and the otherhalf appears to result from averaging the sibling models.Increasing the size of the hillclimb set via cross-validation(as opposed to having more data available for hillclimbing)provides less benefit in practice because there is a mismatchbetween the base-level models used to make predictions onthe hillclimbing set and the sibling-averaged models that

9

Table 5. Breakdown of improvement from cross-validation.

ACC FSC LFT ROC APR BEP RMS MXE MEANES-HILL 32.9% 37.2% 48.0% 38.8% 40.8% 19.4% 55.1% 56.7% 41.1%ES-AVG 80.5% 13.6% 54.0% 59.0% 55.7% 77.4% 46.8% 51.8% 54.9%SUM 113.4% 50.8% 102.0% 97.8% 96.5% 96.8% 101.9% 108.5% 96.0%

will be used in the ensemble. In other words ensemble se-lection is hillclimbing using slightly different models thanthe ones it actually adds to the ensemble.

9.2 Model Compression

While very accurate, the ensembles built by ensemble se-lection are exceptionally complex. On average, storing thelearned ensemble requires 550 MB, and classifying a singletest case takes about 0.5 seconds. This prohibits their usein applications where storage space is at a premium (e.g.PDAs), where test sets are large (e.g. Google), or wherecomputational power is limited (e.g. hearing aids). In a sep-arate paper we address these issues by using a model com-pression [6] method to obtain models that perform as wellas the ensembles built by ensemble selection, but which arefaster and more compact.

The main idea behind model compression is to train afast and compact model to approximate the function learnedby a slow, large, but high performing model. Unlike the truefunction that is unknown, the function learned by the highperforming model is available and can be used to label largeamounts of synthetic data. A fast, compact and expressivemodel trained on enough synthetic data will not overfit andwill closely approximate the function learned by the orig-inal model. This allows a slow, complex model such as amassive ensemble to be compressed into a fast, compactmodel with little loss in performance.

In the model compression paper, we use neural networksto compress ensembles produced by ensemble selection. Onaverage the compressed models retain more than 90% of theimprovement provided by ensemble selection (over modelselection), while being more than 1000 times smaller and1000 times faster.

10 Conclusions

Embedding cross-validation inside ensemble selectiongreatly increases its performance. Half of this benefit is dueto having more data for hillclimbing; the other half is due toa bagging effect that results from the way cross-validationis embedded within ensemble selection. Unsurprisingly, re-ducing the amount of hillclimbing data hurts performancebecause ensemble selection can overfit this data more easily.

In comparison to model selection, however, ensemble selec-tion seems much more resistant to overfitting when data isscarce. Further experiments varying the amount of trainingdata provided to the base-level models are needed to see ifensemble selection is truly able to outperform model selec-tion by such a significant amount on small data sets.

Counter to our and others’ intuition [9], calibrating mod-els to put all predictions on the same scale before averagingthem did not improve ensemble selection’s effectiveness.Most of calibration’s improvement comes from the superiorbase-level models.

Our experiments show that directly optimizing to a tar-get metric is better than always optimizing to some prede-termined metric. That said, always optimizing to RMS orMXE was surprisingly competitive. These metrics may begood optimization proxies if the target metric is too expen-sive to compute repeatedly during hillclimbing.

Finally, pruning the number of available models reducesthe risk of overfitting during hillclimbing while also yield-ing faster ensemble building. In our experiments pruningrarely hurt performance and frequently improved it.

Acknowledgments

We thank Lars Backstrom for help with exploring alter-native model calibration methods and the anonymous re-viewers for helpful comments on paper drafts. This workwas supported by NSF Award 0412930.

References

[1] S. D. Bay. Combining nearest neighbor classifiers throughmultiple feature subsets. In ICML, pages 37–45, 1998.

[2] L. Breiman. Heuristics of instability in model selection.Technical report, Statistics Department, University of Cal-ifornia at Berkeley, 1994.

[3] L. Breiman. Bagging predictors. Machine Learning,24(2):123–140, 1996.

[4] L. Breiman. Random forests. Machine Learning, 45(1):5–32, 2001.

[5] R. K. Bryll, R. Gutierrez-Osuna, and F. K. H. Quek. At-tribute bagging: Improving accuracy of classifier ensem-bles by using random feature subsets. Pattern Recognition,36(6):1291–1302, 2003.

10

[6] C. Bucila, R. Caruana, and A. Niculescu-Mizil. Model com-pression: Making big, slow models practical. In Proc. of the12th International Conf. on Knowledge Discovery and DataMining (KDD’06), 2006.

[7] R. Caruana, A. Niculescu-Mizil, G. Crew, and A. Ksikes.Ensemble selection from libraries of models. In ICML,2004.

[8] P. Domingos. Bayesian averaging of classifiers and the over-fitting problem. In ICML, pages 223–230. Morgan Kauf-mann, San Francisco, CA, 2000.

[9] R. P. W. Duin. The combining classifier: To train or not totrain? In ICPR (2), pages 765–770, 2002.

[10] P. Giudici. Applied Data Mining. John Wiley and Sons, NewYork, 2003.

[11] L. I. Kuncheva and C. J. Whitaker. Measures of diversity inclassifier ensembles and their relationship with the ensembleaccuracy. Machine Learning, 51(2):181–207, 2003.

[12] D. D. Margineantu and T. G. Dietterich. Pruning adaptiveboosting. In ICML, pages 211–218. Morgan Kaufmann,1997.

[13] G. Martınez-Munoz and A. Suarez. Pruning in ordered bag-ging ensembles. In ICML, pages 609–616, New York, NY,USA, 2006. ACM Press.

[14] A. Munson, C. Cardie, and R. Caruana. Optimizing toarbitrary NLP metrics using ensemble selection. In HLT-EMNLP, pages 539–546, 2005.

[15] A. Niculescu-Mizil and R. Caruana. Predicting good proba-bilities with supervised learning. In ICML’05, 2005.

[16] C. Perlich, F. Provost, and J. S. Simonoff. Tree inductionvs. logistic regression: A learning-curve analysis. J. Mach.Learn. Res., 4:211–255, 2003.

[17] A. H. Peterson and T. R. Martinez. Estimating the poten-tial for combining learning models. In Proc. of the ICMLWorkshop on Meta-Learning, pages 68–75, 2005.

[18] J. Platt. Probabilistic outputs for support vector machinesand comparison to regularized likelihood methods. In Adv.in Large Margin Classifiers, 1999.

[19] F. J. Provost and T. Fawcett. Analysis and visualization ofclassifier performance: Comparison under imprecise classand cost distributions. In Knowledge Discovery and DataMining, pages 43–48, 1997.

[20] F. Roli, G. Giacinto, and G. Vernazza. Methods for design-ing multiple classifier systems. In Multiple Classifier Sys-tems, pages 78–87, 2001.

[21] R. Schapire. The boosting approach to machine learning: Anoverview. In In MSRI Workshop on Nonlinear Estimationand Classification, 2001.

[22] W. N. Street and Y.-H. Kim. A streaming ensemble algo-rithm (SEA) for large-scale classification. In KDD, pages377–382, 2001.

[23] G. Tsoumakas, L. Angelis, and I. Vlahavas. Selective fu-sion of heterogeneous classifiers. Intelligent Data Analysis,9(6):511–525, 2005.

[24] I. H. Witten and E. Frank. Data Mining: Practical MachineLearning Tools and Techniques. Morgan Kaufmann, SanFrancisco, second edition, 2005.

[25] D. H. Wolpert. Stacked generalization. Neural Networks,5:241–259, 1992.

[26] Y. Zhang, S. Burer, and W. N. Street. Ensemble pruning viasemi-definite programming. Journal of Machine LearningResearch, 7:1315–1338, 2006.

A Learning Methods

In addition to the learning methods used by Caruanaet al. [7] (decision trees, bagged trees, boosted trees andstumps, KNN, neural nets, and SVMs), we use three moremodel types: logistic regression, naıve bayes, and randomforests. These are trained as follows:Logistic Regression (LOGREG): we train both unregular-ized and regularized models, varying the ridge parameterby factors of 10 from 10−8 to 104. Attributes are scaled tomean 0 and standard deviation 1.Random Forests (RF): we use the Weka implementa-tion [24]. The forests have 1024 trees, and the size of thefeature set to consider at each split is 1, 2, 4, 6, 8, 12, 16 or20.Naıve Bayes (NB): we use the Weka implementation andtry all three of the Weka options for handling continuous at-tributes: modeling them as a single normal, modeling themwith kernel estimation, or discretizing them using super-vised discretization.

In total, around 2,500 models are trained for each dataset. When calibrated models are included for ensemble se-lection the number doubles to 5,000.

B Data Sets

We experiment with 11 binary classification prob-lems. ADULT, COV TYPE, HS, LETTER.P1, LET-TER.P2, MEDIS, and SLAC were used by Caruana etal. [7]. The four new data sets we use are BACT, COD,CALHOUS, and MG. COD, BACT, and CALHOUS arethree of the datasets used in Perlich et al. [16]. MG is amedical data set. See Table 6 for characteristics of the 11problems.

Table 6. Description of problemsPROBLEM #ATTR TRAIN TEST %POZ

ADULT 14/104 4000 35222 25%BACT 11/170 4000 34262 69%COD 15/60 4000 14000 50%CALHOUS 9 4000 14640 52%COV TYPE 54 4000 25000 36%HS 200 4000 4366 24%LETTER.P1 16 4000 14000 3%LETTER.P2 16 4000 14000 53%MEDIS 63 4000 8199 11%MG 124 4000 12807 17%SLAC 59 4000 25000 50%

11

Table 7. Scales used to compute normalized scores. Each entry shows bottom / top for the scale.ACC FSC LFT ROC APR BEP RMS MXE

ADULT 0.752 / 0.859 0.398 / 0.705 1.000 / 2.842 0.500 / 0.915 0.248 / 0.808 0.248 / 0.708 0.432 / 0.312 0.808 / 0.442BACT 0.692 / 0.780 0.818 / 0.855 1.000 / 1.345 0.500 / 0.794 0.692 / 0.891 0.692 / 0.824 0.462 / 0.398 0.891 / 0.697CALHOUS 0.517 / 0.889 0.681 / 0.893 1.000 / 1.941 0.500 / 0.959 0.517 / 0.964 0.517 / 0.895 0.500 / 0.283 0.999 / 0.380COD 0.501 / 0.784 0.666 / 0.796 1.000 / 1.808 0.500 / 0.866 0.499 / 0.864 0.499 / 0.782 0.500 / 0.387 1.000 / 0.663COVTYPE 0.639 / 0.859 0.531 / 0.804 1.000 / 2.487 0.500 / 0.926 0.362 / 0.879 0.361 / 0.805 0.480 / 0.320 0.944 / 0.478HS 0.759 / 0.949 0.389 / 0.894 1.000 / 3.656 0.500 / 0.985 0.243 / 0.962 0.241 / 0.898 0.428 / 0.198 0.797 / 0.195LETTER.p1 0.965 / 0.994 0.067 / 0.917 1.000 / 4.001 0.500 / 0.999 0.036 / 0.975 0.035 / 0.917 0.184 / 0.067 0.219 / 0.025LETTER.p2 0.533 / 0.968 0.696 / 0.970 1.000 / 1.887 0.500 / 0.996 0.534 / 0.997 0.533 / 0.970 0.499 / 0.157 0.997 / 0.125MEDIS 0.893 / 0.905 0.193 / 0.447 1.000 / 2.917 0.500 / 0.853 0.108 / 0.462 0.107 / 0.469 0.309 / 0.272 0.491 / 0.365MG 0.831 / 0.900 0.290 / 0.663 1.000 / 3.210 0.500 / 0.911 0.170 / 0.740 0.169 / 0.686 0.375 / 0.278 0.656 / 0.373SLAC 0.501 / 0.726 0.667 / 0.751 1.000 / 1.727 0.500 / 0.813 0.501 / 0.816 0.501 / 0.727 0.500 / 0.420 1.000 / 0.755

C Performance Scales

Table 7 lists the performance numbers that determine thenormalized scores. Each entry contains the baseline per-formance (bottom of the scale) and the best performanceachieved by any model or ensemble (top of the scale).

12


Recommended