Randomized Continuous Testing: Solr & Lucene Use Case

Post on 14-Jun-2015

580 views 0 download

Tags:

description

See conference video - http://www.lucidimagination.com/devzone/events/conferences/ApacheLuceneEurocon2011 We have been taught that unit tests should be repeatable and most people (including the author) for a long time considered this an equivalent to "static", single-path execution. Solr and Lucene employ an interesting JUnit runner strategy where tests are randomized -- run with various data, various implementation of allowed interfaces, various configurations. The number of combinations makes running them all impossible, but execution randomization proves very successful at pinpointing implementation and regression bugs. This talk will provide an overview of this approach and practical considerations on when and how to port them to your own projects. Everything that stems from Lucene/Solr is not directly connected to search/ document retrieval and can still be useful and reused in other projects. This session is probably best suited to developers/ CTOs.

transcript

!

!!!!!

6WULQJ >@ ZRUGV ^�%DUFHORQD�� �EHDFK�� � �VXQ�� � �RUDFOH� `�SXEOLF YRLG UXQ0HWKRG�5DQGRP U� ^

IRU �LQW L �� L � ���� L��� ^LQW LQGH[

0DWK�DEV�U�QH[W,QW��� � ZRUGV�OHQJWK�6\VWHP�RXW�SULQWOQ�ZRUGV>LQGH[@��

``

#7HVWSXEOLF YRLG WHVW%DUFHORQD:RUGV�� ^UXQ0HWKRG�QHZ 5DQGRP����

`

#7HVW#6HHG�����D��E�� �� JHW XQOXFN\���SXEOLF YRLG WHVW%DUFHORQD��� ^

UXQ0HWKRG�5DQGRPL]HG&RQWH[W�FXUUHQW��

�JHW5DQGRP����`

!!!!!!!!!!!!

� )URP� /XFHQHV B7HVW8WLO� 6WDUW DQG HQG DUH %27+ LQFOXVLYH� �

LQW QH[W,QW�5DQGRP U� LQW VWDUW� LQW HQG� ^UHWXUQ VWDUW � U�QH[W,QW�HQG � VWDUW � ���

`

QH[W,QW�U� �� ,QWHJHU�0$;B9$/8(�

!!!!!!!!!!!!

� )URP� /XFHQHV B7HVW8WLO� 6WDUW DQG HQG DUH %27+ LQFOXVLYH� �

LQW QH[W,QW�5DQGRP U� LQW VWDUW� LQW HQG� ^UHWXUQ VWDUW � U�QH[W,QW�HQG � VWDUW � ���

`

QH[W,QW�U� �� ,QWHJHU�0$;B9$/8(�

!!!!!!!!

!

ILQDO LQW LWHUV DW/HDVW������IRU �LQW LWHU �� LWHU � LWHUV� LWHU��� ^�� ���

`

LI �UDQGRP�QH[W%RROHDQ��� ^WHUP B7HVW8WLO�UDQGRP5HDOLVWLF8QLFRGH6WULQJ�UDQGRP��

` HOVH ^�� ZH ZDQW WR PL[ LQ OLPLWHG�DOSKDEHW V\PEROV VR�� ZH JHW PRUH VKDULQJ RI WKH QRGHV JLYHQ KRZ IHZ�� WHUPV ZH DUH WHVWLQJ���WHUP VLPSOH5DQGRP6WULQJ�UDQGRP��

`

SXEOLF VWDWLF 6WULQJ UDQGRP'LUHFWRU\�5DQGRP UDQGRP� ^LI �UDUHO\�UDQGRP�� ^UHWXUQ &25(B',5(&725,(6>UDQGRP�QH[W,QW�&25(B',5(&725,(6�OHQJWK�@�

` HOVH ^UHWXUQ �5$0'LUHFWRU\��

``

ORFDOH 7(67B/2&$/(�HTXDOV��UDQGRP�� " UDQGRP/RFDOH�UDQGRP� � ���/RFDOH�VHW'HIDXOW�ORFDOH��WLPH=RQH 7(67B7,0(=21(�HTXDOV��UDQGRP�� " UDQGRP7LPH=RQH�UDQGRP� � ���7LPH=RQH�VHW'HIDXOW�WLPH=RQH��

ORQJ VHHG 6\VWHP�FXUUHQW7LPH0LOOLV���6\VWHP�RXW�SULQWOQ�VHHG��5DQGRP UQG QHZ 5DQGRP�VHHG��SDVV7R7HVWV�UQG��

�� #5XQ:LWK�/XFHQH7HVW&DVH5XQQHU�FODVV`SXEOLF FODVV 0\5DQGRPL]HG7HVW

H[WHQGV /XFHQH7HVW&DVH

�� #5XQ:LWK�5DQGRPL]HG5XQQHU�FODVV`SXEOLF FODVV 0\5DQGRPL]HG7HVW

H[WHQGV 5DQGRPL]HG7HVW

ORQJ VHHG 6\VWHP�FXUUHQW7LPH0LOOLV���6\VWHP�RXW�SULQWOQ�VHHG��5DQGRP UQG QHZ 5DQGRP�VHHG��SDVV7R7HVWV�UQG��

�� #5XQ:LWK�/XFHQH7HVW&DVH5XQQHU�FODVV`SXEOLF FODVV 0\5DQGRPL]HG7HVW

H[WHQGV /XFHQH7HVW&DVH

�� #5XQ:LWK�5DQGRPL]HG5XQQHU�FODVV`SXEOLF FODVV 0\5DQGRPL]HG7HVW

H[WHQGV 5DQGRPL]HG7HVW

ORQJ VHHG 6\VWHP�FXUUHQW7LPH0LOOLV���6\VWHP�RXW�SULQWOQ�VHHG��5DQGRP UQG QHZ 5DQGRP�VHHG��SDVV7R7HVWV�UQG��

�� #5XQ:LWK�/XFHQH7HVW&DVH5XQQHU�FODVV`SXEOLF FODVV 0\5DQGRPL]HG7HVW

H[WHQGV /XFHQH7HVW&DVH

�� #5XQ:LWK�5DQGRPL]HG5XQQHU�FODVV`SXEOLF FODVV 0\5DQGRPL]HG7HVW

H[WHQGV 5DQGRPL]HG7HVW

assign master seed-Dtests.seed=xxx:...

@Seed(class)mhash(timeMillis)

suite validation

test candidates (+iters)assign method seeds

-Dtests.seed=xxx:yyy@Seed(method)

derive(master, name)

shuffle execution order

@BeforeClass @Before @Test @After @AfterClass

shuffle execution order

SXEOLF FODVV 7HVW&ODVV� H[WHQGV 5DQGRPL]HG7HVW ^#%HIRUH&ODVV SULYDWH VWDWLF YRLG EHIRUH&ODVV�� ^ SULQWOQ��FODVV �� EHIRUH&ODVV��� `#%HIRUH SULYDWH YRLG EHIRUH�� ^ SULQWOQ�� FODVV �� EHIRUH��� `#7HVW SXEOLF YRLG WHVW�B��� ^ SULQWOQ�� FODVV �� WHVW���� `#7HVW SXEOLF YRLG WHVW�B��� ^ SULQWOQ�� FODVV �� WHVW���� `#7HVW SXEOLF YRLG WHVW�B��� ^ SULQWOQ�� FODVV �� WHVW���� `#$IWHU SULYDWH YRLG DIWHU�� ^ SULQWOQ�� FODVV �� DIWHU��� `#$IWHU&ODVV SULYDWH VWDWLF YRLG DIWHU&ODVV�� ^ SULQWOQ��FODVV �� DIWHU&ODVV��� `

`

SXEOLF FODVV 7HVW&ODVV� H[WHQGV 7HVW&ODVV� ^#%HIRUH&ODVV SULYDWH VWDWLF YRLG EHIRUH&ODVV�� ^ SULQWOQ��FODVV �� EHIRUH&ODVV��� `#%HIRUH SULYDWH YRLG EHIRUH�� ^ SULQWOQ�� FODVV �� EHIRUH��� `#7HVW SXEOLF YRLG WHVW�B��� ^ SULQWOQ�� FODVV �� WHVW���� `#7HVW SXEOLF YRLG WHVW�B��� ^ SULQWOQ�� FODVV �� WHVW���� `#7HVW SXEOLF YRLG WHVW�B��� ^ SULQWOQ�� FODVV �� WHVW���� `#$IWHU SULYDWH YRLG DIWHU�� ^ SULQWOQ�� FODVV �� DIWHU��� `#$IWHU&ODVV SULYDWH VWDWLF YRLG DIWHU&ODVV�� ^ SULQWOQ��FODVV �� DIWHU&ODVV��� `

`

FODVV �� EHIRUH&ODVVFODVV �� EHIRUH&ODVVFODVV �� EHIRUHFODVV �� EHIRUHFODVV �� WHVW�

FODVV �� DIWHUFODVV �� DIWHUFODVV �� EHIRUHFODVV �� EHIRUHFODVV �� WHVW�

FODVV �� DIWHUFODVV �� DIWHUFODVV �� EHIRUHFODVV �� EHIRUHFODVV �� WHVW�

FODVV �� DIWHUFODVV �� DIWHUFODVV �� EHIRUHFODVV �� EHIRUHFODVV �� WHVW�

FODVV �� DIWHUFODVV �� DIWHUFODVV �� EHIRUHFODVV �� EHIRUHFODVV �� WHVW�

FODVV �� DIWHUFODVV �� DIWHUFODVV �� EHIRUHFODVV �� EHIRUHFODVV �� WHVW�

FODVV �� DIWHUFODVV �� DIWHU

FODVV �� DIWHU&ODVVFODVV �� DIWHU&ODVV

FODVV �� EHIRUH&ODVVFODVV �� EHIRUH&ODVVFODVV �� EHIRUHFODVV �� EHIRUHFODVV �� WHVW�

FODVV �� DIWHUFODVV �� DIWHUFODVV �� EHIRUHFODVV �� EHIRUHFODVV �� WHVW�

FODVV �� DIWHUFODVV �� DIWHUFODVV �� EHIRUHFODVV �� EHIRUHFODVV �� WHVW�

FODVV �� DIWHUFODVV �� DIWHUFODVV �� EHIRUHFODVV �� EHIRUHFODVV �� WHVW�

FODVV �� DIWHUFODVV �� DIWHUFODVV �� EHIRUHFODVV �� EHIRUHFODVV �� WHVW�

FODVV �� DIWHUFODVV �� DIWHUFODVV �� EHIRUHFODVV �� EHIRUHFODVV �� WHVW�

FODVV �� DIWHUFODVV �� DIWHU

FODVV �� DIWHU&ODVVFODVV �� DIWHU&ODVV

FODVV �� EHIRUH&ODVVFODVV �� EHIRUH&ODVV

FODVV �� EHIRUHFODVV �� EHIRUHFODVV �� WHVW�

FODVV �� DIWHUFODVV �� DIWHUFODVV �� EHIRUHFODVV �� EHIRUHFODVV �� WHVW�

FODVV �� DIWHUFODVV �� DIWHUFODVV �� EHIRUHFODVV �� EHIRUHFODVV �� WHVW�

FODVV �� DIWHUFODVV �� DIWHUFODVV �� EHIRUHFODVV �� EHIRUHFODVV �� WHVW�

FODVV �� DIWHUFODVV �� DIWHUFODVV �� EHIRUHFODVV �� EHIRUHFODVV �� WHVW�

FODVV �� DIWHUFODVV �� DIWHUFODVV �� EHIRUHFODVV �� EHIRUHFODVV �� WHVW�

FODVV �� DIWHUFODVV �� DIWHU

FODVV �� DIWHU&ODVVFODVV �� DIWHU&ODVV

5DQGRPL]HG&RQWH[W FW[ 5DQGRPL]HG&RQWH[W�FXUUHQW���

FW[�JHW7DUJHW&ODVV��FW[�JHW5DQGRP�� �� ! QR VHW6HHG���FW[�JHW5DQGRPQHVV���JHW6HHG��FW[�LV1LJKWO\�����

#1LJKWO\ #7HVW #5HSHDW�LWHUDWLRQV ��SXEOLF YRLG WHVW1LJKWO\�� ^� VQRR]H �`

#7HVW #5HSHDW�LWHUDWLRQV ��SXEOLF YRLG WHVW$OZD\V�� ^

DVVHUW7UXH�UDQGRP%RROHDQ����`

5DQGRPL]HG&RQWH[W FW[ 5DQGRPL]HG&RQWH[W�FXUUHQW���

FW[�JHW7DUJHW&ODVV��FW[�JHW5DQGRP�� �� ! QR VHW6HHG���FW[�JHW5DQGRPQHVV���JHW6HHG��FW[�LV1LJKWO\�����

#1LJKWO\ #7HVW #5HSHDW�LWHUDWLRQV ��SXEOLF YRLG WHVW1LJKWO\�� ^� VQRR]H �`

#7HVW #5HSHDW�LWHUDWLRQV ��SXEOLF YRLG WHVW$OZD\V�� ^

DVVHUW7UXH�UDQGRP%RROHDQ����`

5DQGRPL]HG&RQWH[W FW[ 5DQGRPL]HG&RQWH[W�FXUUHQW���

FW[�JHW7DUJHW&ODVV��FW[�JHW5DQGRP�� �� ! QR VHW6HHG���FW[�JHW5DQGRPQHVV���JHW6HHG��FW[�LV1LJKWO\�����

#1LJKWO\ #7HVW #5HSHDW�LWHUDWLRQV ��SXEOLF YRLG WHVW1LJKWO\�� ^� VQRR]H �`

#7HVW #5HSHDW�LWHUDWLRQV ��SXEOLF YRLG WHVW$OZD\V�� ^

DVVHUW7UXH�UDQGRP%RROHDQ����`

#6HHG��GHDGEHHI��#7HVW #5HSHDW�LWHUDWLRQV ��� XVH&RQVWDQW6HHG WUXH�SXEOLF YRLG WHVW$OZD\V�� ^DVVHUW7UXH�UDQGRP%RROHDQ����

`

�'WHVWV�VHHG ��GHDGEHHI �� VHHG IRUPDW� PDVWHU>�PHWKRG@�'WHVWV�LWHU ���'WHVWV�PHWKRG WHVW$OZD\V

#6HHG��GHDGEHHI��#7HVW #5HSHDW�LWHUDWLRQV ��� XVH&RQVWDQW6HHG WUXH�SXEOLF YRLG WHVW$OZD\V�� ^DVVHUW7UXH�UDQGRP%RROHDQ����

`

�'WHVWV�VHHG ��GHDGEHHI �� VHHG IRUPDW� PDVWHU>�PHWKRG@�'WHVWV�LWHU ���'WHVWV�PHWKRG WHVW$OZD\V

#7HVW #7LPHRXW�PLOOLV �����SXEOLF YRLG LQWHUUXSWDEOH�� WKURZV ([FHSWLRQ ^

7KUHDG�VOHHS���������`

2FW ��� ���� ������� 30 UDQGRPL]HGWHVWLQJ�5DQGRPL]HG5XQQHU WU\7R7HUPLQDWH:$51,1*� $WWHPSWLQJ WR VWRS WKUHDG� 7KUHDG���������������� FXUUHQWO\ DW�

MDYD�ODQJ�7KUHDG�VOHHS�1DWLYH 0HWKRG�UDQGRPL]HGWHVWLQJ�H[DPSOHV�7HVW([DPSOH��LQWHUUXSWDEOH�7HVW([DPSOH��MDYD����VXQ�UHIOHFW�1DWLYH0HWKRG$FFHVVRU,PSO�LQYRNH��1DWLYH 0HWKRG�VXQ�UHIOHFW�1DWLYH0HWKRG$FFHVVRU,PSO�LQYRNH�1DWLYH0HWKRG$FFHVVRU,PSO�MDYD����VXQ�UHIOHFW�'HOHJDWLQJ0HWKRG$FFHVVRU,PSO�LQYRNH�'HOHJDWLQJ0HWKRG$FFHVVRU,PSO�MDYD����MDYD�ODQJ�UHIOHFW�0HWKRG�LQYRNH�0HWKRG�MDYD�����UDQGRPL]HGWHVWLQJ�5DQGRPL]HG5XQQHU�LQYRNH�5DQGRPL]HG5XQQHU�MDYD�����UDQGRPL]HGWHVWLQJ�5DQGRPL]HG5XQQHU�DFFHVV���5DQGRPL]HG5XQQHU�MDYD�����UDQGRPL]HGWHVWLQJ�5DQGRPL]HG5XQQHU���HYDOXDWH�5DQGRPL]HG5XQQHU�MDYD�����UDQGRPL]HGWHVWLQJ�5DQGRPL]HG5XQQHU�UXQ:LWK5XOHV�5DQGRPL]HG5XQQHU�MDYD�����UDQGRPL]HGWHVWLQJ�5DQGRPL]HG5XQQHU�UXQ6LQJOH7HVW�5DQGRPL]HG5XQQHU�MDYD�����UDQGRPL]HGWHVWLQJ�5DQGRPL]HG5XQQHU�DFFHVV���5DQGRPL]HG5XQQHU�MDYD�����UDQGRPL]HGWHVWLQJ�5DQGRPL]HG5XQQHU���UXQ�5DQGRPL]HG5XQQHU�MDYD�����MDYD�ODQJ�7KUHDG�UXQ�7KUHDG�MDYD�����

2FW ��� ���� ������� 30 UDQGRPL]HGWHVWLQJ�5DQGRPL]HG5XQQHU WU\7R7HUPLQDWH:$51,1*� ,QWHUUXSWHG D UXQDZD\ WKUHDG� 7KUHDG���������������

#7HVW #7LPHRXW�PLOOLV �����SXEOLF YRLG XQLQWHUUXSWDEOH�� WKURZV ([FHSWLRQ ^

ZKLOH �WUXH� � VSLQ ��`

2FW ��� ���� ������� 30 UDQGRPL]HGWHVWLQJ�5DQGRPL]HG5XQQHU WU\7R7HUPLQDWH:$51,1*� $WWHPSWLQJ WR VWRS WKUHDG� 7KUHDG���������������� FXUUHQWO\ DW�

UDQGRPL]HGWHVWLQJ�H[DPSOHV�7HVW([DPSOH��XQLQWHUUXSWDEOH�7HVW([DPSOH��MDYD����VXQ�UHIOHFW�1DWLYH0HWKRG$FFHVVRU,PSO�LQYRNH��1DWLYH 0HWKRG�VXQ�UHIOHFW�1DWLYH0HWKRG$FFHVVRU,PSO�LQYRNH�1DWLYH0HWKRG$FFHVVRU,PSO�MDYD����VXQ�UHIOHFW�'HOHJDWLQJ0HWKRG$FFHVVRU,PSO�LQYRNH�'HOHJDWLQJ0HWKRG$FFHVVRU,PSO�MDYD����MDYD�ODQJ�UHIOHFW�0HWKRG�LQYRNH�0HWKRG�MDYD�����UDQGRPL]HGWHVWLQJ�5DQGRPL]HG5XQQHU�LQYRNH�5DQGRPL]HG5XQQHU�MDYD�����UDQGRPL]HGWHVWLQJ�5DQGRPL]HG5XQQHU�DFFHVV���5DQGRPL]HG5XQQHU�MDYD�����UDQGRPL]HGWHVWLQJ�5DQGRPL]HG5XQQHU���HYDOXDWH�5DQGRPL]HG5XQQHU�MDYD�����UDQGRPL]HGWHVWLQJ�5DQGRPL]HG5XQQHU�UXQ:LWK5XOHV�5DQGRPL]HG5XQQHU�MDYD�����UDQGRPL]HGWHVWLQJ�5DQGRPL]HG5XQQHU�UXQ6LQJOH7HVW�5DQGRPL]HG5XQQHU�MDYD�����UDQGRPL]HGWHVWLQJ�5DQGRPL]HG5XQQHU�DFFHVV���5DQGRPL]HG5XQQHU�MDYD�����UDQGRPL]HGWHVWLQJ�5DQGRPL]HG5XQQHU���UXQ�5DQGRPL]HG5XQQHU�MDYD�����MDYD�ODQJ�7KUHDG�UXQ�7KUHDG�MDYD�����

2FW ��� ���� ������� 30 UDQGRPL]HGWHVWLQJ�5DQGRPL]HG5XQQHU WU\7R7HUPLQDWH:$51,1*� 'RHV QRW UHVSRQG WR LQWHUUXSW��� WU\LQJ WR VWRS��� 7KUHDG���������������2FW ��� ���� ������� 30 UDQGRPL]HGWHVWLQJ�5DQGRPL]HG5XQQHU WU\7R7HUPLQDWH:$51,1*� 6WRSSHG D UXQDZD\ WKUHDG� 7KUHDG���������������

#7HVW #7LPHRXW�PLOOLV �����SXEOLF YRLG FKULVWRSKHU/DPEHUW�� WKURZV ([FHSWLRQ ^

ZKLOH �WUXH� ^WU\ ^ 7KUHDG�VOHHS������� ` FDWFK �7KURZDEOH W� ^`

``

2FW ��� ���� ������� 30 UDQGRPL]HGWHVWLQJ�5DQGRPL]HG5XQQHU WU\7R7HUPLQDWH:$51,1*� $WWHPSWLQJ WR VWRS WKUHDG� 7KUHDG���������������� FXUUHQWO\ DW�

MDYD�ODQJ�7KUHDG�VOHHS�1DWLYH 0HWKRG�UDQGRPL]HGWHVWLQJ�H[DPSOHV�7HVW([DPSOH��FKULVWRSKHU/DPEHUW�7HVW([DPSOH��MDYD����VXQ�UHIOHFW�1DWLYH0HWKRG$FFHVVRU,PSO�LQYRNH��1DWLYH 0HWKRG�VXQ�UHIOHFW�1DWLYH0HWKRG$FFHVVRU,PSO�LQYRNH�1DWLYH0HWKRG$FFHVVRU,PSO�MDYD����VXQ�UHIOHFW�'HOHJDWLQJ0HWKRG$FFHVVRU,PSO�LQYRNH�'HOHJDWLQJ0HWKRG$FFHVVRU,PSO�MDYD����MDYD�ODQJ�UHIOHFW�0HWKRG�LQYRNH�0HWKRG�MDYD�����UDQGRPL]HGWHVWLQJ�5DQGRPL]HG5XQQHU�LQYRNH�5DQGRPL]HG5XQQHU�MDYD�����UDQGRPL]HGWHVWLQJ�5DQGRPL]HG5XQQHU�DFFHVV���5DQGRPL]HG5XQQHU�MDYD�����UDQGRPL]HGWHVWLQJ�5DQGRPL]HG5XQQHU���HYDOXDWH�5DQGRPL]HG5XQQHU�MDYD�����UDQGRPL]HGWHVWLQJ�5DQGRPL]HG5XQQHU�UXQ:LWK5XOHV�5DQGRPL]HG5XQQHU�MDYD�����UDQGRPL]HGWHVWLQJ�5DQGRPL]HG5XQQHU�UXQ6LQJOH7HVW�5DQGRPL]HG5XQQHU�MDYD�����UDQGRPL]HGWHVWLQJ�5DQGRPL]HG5XQQHU�DFFHVV���5DQGRPL]HG5XQQHU�MDYD�����UDQGRPL]HGWHVWLQJ�5DQGRPL]HG5XQQHU���UXQ�5DQGRPL]HG5XQQHU�MDYD�����MDYD�ODQJ�7KUHDG�UXQ�7KUHDG�MDYD�����

2FW ��� ���� ������� 30 UDQGRPL]HGWHVWLQJ�5DQGRPL]HG5XQQHU WU\7R7HUPLQDWH:$51,1*� 'RHV QRW UHVSRQG WR LQWHUUXSW��� WU\LQJ WR VWRS��� 7KUHDG���������������2FW ��� ���� ������� 30 UDQGRPL]HGWHVWLQJ�5DQGRPL]HG5XQQHU WU\7R7HUPLQDWH6(9(5(� &RXOG QRW LQWHUUXSW RU VWRS WKUHDG� 7KUHDG���������������

6WULQJ UHDG)LOH�)LOH ILOH� &KDUVHW FKDUVHW� ^E\WH>@ E\WHV QHZ E\WH>�LQW� ILOH�OHQJWK��@�'DWD,QSXW6WUHDP LV

QHZ 'DWD,QSXW6WUHDP�QHZ )LOH,QSXW6WUHDP�ILOH���WU\ ^

LV�UHDG)XOO\�E\WHV��` ILQDOO\ ^ LV�FORVH��� `UHWXUQ QHZ 6WULQJ�E\WHV� FKDUVHW��

`

#7HVW #5HSHDW�LWHUDWLRQV ���SXEOLF YRLG WHVW5HDG)LOH�� WKURZV ,2([FHSWLRQ ^)LOH WHPS)LOH QHZ7HPS)LOH���)LOH2XWSXW6WUHDP IRV QHZ )LOH2XWSXW6WUHDP�WHPS)LOH��6WULQJ FRQWHQWV UDQGRP8QLFRGH6WULQJ���IRV�ZULWH�FRQWHQWV�JHW%\WHV�87)����IRV�FORVH���

6WULQJ UHDG%DFN ,28WLOV�UHDG)LOH�WHPS)LOH� 87)���$VVHUW�DVVHUW(TXDOV�FRQWHQWV� UHDG%DFN��

`

� )LOH RSHQLQJ SRLQWFXWV �VLPSOLVWLF�� �SRLQWFXW RSHQ)LOH���

FDOO�MDYD�LR�)LOH,QSXW6WUHDP�QHZ����� __FDOO�MDYD�LR�5DQGRP$FFHVV)LOH�QHZ������

� $Q\ UHDGLQJ IURP DQ\ VWUHDP� �SRLQWFXW UHDG6WUHDP���

FDOO� MDYD�LR�,QSXW6WUHDP��UHDG ������

� $OO WHVW PHWKRGV� �SRLQWFXW WHVW0HWKRGV��� H[HFXWLRQ�#7HVW ����

� ([HFXWLRQ XQGHU D WHVW PHWKRG� �SRLQWFXW WHVWLQJ7KUHDG��� FIORZEHORZ�WHVW0HWKRGV����

� )LOH RSHQLQJ SRLQWFXWV �VLPSOLVWLF�� �SRLQWFXW RSHQ)LOH���

FDOO�MDYD�LR�)LOH,QSXW6WUHDP�QHZ����� __FDOO�MDYD�LR�5DQGRP$FFHVV)LOH�QHZ������

� $Q\ UHDGLQJ IURP DQ\ VWUHDP� �SRLQWFXW UHDG6WUHDP���

FDOO� MDYD�LR�,QSXW6WUHDP��UHDG ������

� $OO WHVW PHWKRGV� �SRLQWFXW WHVW0HWKRGV��� H[HFXWLRQ�#7HVW ����

� ([HFXWLRQ XQGHU D WHVW PHWKRG� �SRLQWFXW WHVWLQJ7KUHDG��� FIORZEHORZ�WHVW0HWKRGV����

� )LOH RSHQLQJ SRLQWFXWV �VLPSOLVWLF�� �SRLQWFXW RSHQ)LOH���

FDOO�MDYD�LR�)LOH,QSXW6WUHDP�QHZ����� __FDOO�MDYD�LR�5DQGRP$FFHVV)LOH�QHZ������

� $Q\ UHDGLQJ IURP DQ\ VWUHDP� �SRLQWFXW UHDG6WUHDP���

FDOO� MDYD�LR�,QSXW6WUHDP��UHDG ������

� $OO WHVW PHWKRGV� �SRLQWFXW WHVW0HWKRGV��� H[HFXWLRQ�#7HVW ����

� ([HFXWLRQ XQGHU D WHVW PHWKRG� �SRLQWFXW WHVWLQJ7KUHDG��� FIORZEHORZ�WHVW0HWKRGV����

� 6LPXODWH D UDQGRP ,�2 VORZGRZQ� �EHIRUH�� � UHDG6WUHDP�� WHVWLQJ7KUHDG�� ^5DQGRPL]HG&RQWH[W FW[ 5DQGRPL]HG&RQWH[W�FXUUHQW���5DQGRP U FW[�JHW5DQGRP����� ��� ,�2 HUURUV UDWH�LI �U�QH[W,QW����� � ��� ^WU\ ^7KUHDG�VOHHS�UDQGRP,QW%HWZHHQ�U� ���� �������

` FDWFK �,QWHUUXSWHG([FHSWLRQ H� ^� LJQRUH �``

`

� )LOH RSHQLQJ SRLQWFXWV �VLPSOLVWLF�� �SRLQWFXW RSHQ)LOH���

FDOO�MDYD�LR�)LOH,QSXW6WUHDP�QHZ����� __FDOO�MDYD�LR�5DQGRP$FFHVV)LOH�QHZ������

� $Q\ UHDGLQJ IURP DQ\ VWUHDP� �SRLQWFXW UHDG6WUHDP���

FDOO� MDYD�LR�,QSXW6WUHDP��UHDG ������

� $OO WHVW PHWKRGV� �SRLQWFXW WHVW0HWKRGV��� H[HFXWLRQ�#7HVW ����

� ([HFXWLRQ XQGHU D WHVW PHWKRG� �SRLQWFXW WHVWLQJ7KUHDG��� FIORZEHORZ�WHVW0HWKRGV����

� 6LPXODWH D UDQGRP ,�2 VORZGRZQ� �EHIRUH�� � UHDG6WUHDP�� WHVWLQJ7KUHDG�� ^5DQGRPL]HG&RQWH[W FW[ 5DQGRPL]HG&RQWH[W�FXUUHQW���5DQGRP U FW[�JHW5DQGRP����� ��� ,�2 HUURUV UDWH�LI �U�QH[W,QW����� � ��� ^WU\ ^7KUHDG�VOHHS�UDQGRP,QW%HWZHHQ�U� ���� �������

` FDWFK �,QWHUUXSWHG([FHSWLRQ H� ^� LJQRUH �``

`

� )LOH RSHQLQJ SRLQWFXWV �VLPSOLVWLF�� �SRLQWFXW RSHQ)LOH���

FDOO�MDYD�LR�)LOH,QSXW6WUHDP�QHZ����� __FDOO�MDYD�LR�5DQGRP$FFHVV)LOH�QHZ������

� $Q\ UHDGLQJ IURP DQ\ VWUHDP� �SRLQWFXW UHDG6WUHDP���

FDOO� MDYD�LR�,QSXW6WUHDP��UHDG ������

� $OO WHVW PHWKRGV� �SRLQWFXW WHVW0HWKRGV��� H[HFXWLRQ�#7HVW ����

� ([HFXWLRQ XQGHU D WHVW PHWKRG� �SRLQWFXW WHVWLQJ7KUHDG��� FIORZEHORZ�WHVW0HWKRGV����

� 6LPXODWH D UDUH ,�2 SUREOHP ZKHQ RSHQLQJ ILOHV� �2EMHFW DURXQG�� � RSHQ)LOH�� WHVWLQJ7KUHDG�� ^5DQGRPL]HG&RQWH[W FW[ 5DQGRPL]HG&RQWH[W�FXUUHQW����� ��� ,�2 HUURUV UDWH�LI �FW[�JHW5DQGRP���QH[W,QW����� � ��� ^5HWKURZ�UHWKURZ�QHZ ,2([FHSWLRQ�

�6LPXODWHG ,�2 H[FHSWLRQ RQ ILOH RSHQLQJ�����`UHWXUQ SURFHHG���

`

� )LOH RSHQLQJ SRLQWFXWV �VLPSOLVWLF�� �SRLQWFXW RSHQ)LOH���

FDOO�MDYD�LR�)LOH,QSXW6WUHDP�QHZ����� __FDOO�MDYD�LR�5DQGRP$FFHVV)LOH�QHZ������

� $Q\ UHDGLQJ IURP DQ\ VWUHDP� �SRLQWFXW UHDG6WUHDP���

FDOO� MDYD�LR�,QSXW6WUHDP��UHDG ������

� $OO WHVW PHWKRGV� �SRLQWFXW WHVW0HWKRGV��� H[HFXWLRQ�#7HVW ����

� ([HFXWLRQ XQGHU D WHVW PHWKRG� �SRLQWFXW WHVWLQJ7KUHDG��� FIORZEHORZ�WHVW0HWKRGV����

� 6LPXODWH D UDUH ,�2 SUREOHP ZKHQ RSHQLQJ ILOHV� �2EMHFW DURXQG�� � RSHQ)LOH�� WHVWLQJ7KUHDG�� ^5DQGRPL]HG&RQWH[W FW[ 5DQGRPL]HG&RQWH[W�FXUUHQW����� ��� ,�2 HUURUV UDWH�LI �FW[�JHW5DQGRP���QH[W,QW����� � ��� ^5HWKURZ�UHWKURZ�QHZ ,2([FHSWLRQ�

�6LPXODWHG ,�2 H[FHSWLRQ RQ ILOH RSHQLQJ�����`UHWXUQ SURFHHG���

`

SRLQWFXW QHZ7HPS'LU���FDOO�VWDWLF )LOH 5DQGRPL]HG7HVW�QHZ7HPS'LU����

� 5HPHPEHU ZKHQ D WHPSRUDU\ IROGHU LV UHTXHVWHG� �)LOH DURXQG�� � QHZ7HPS'LU�� WHVWLQJ7KUHDG�� ^)LOH GLU SURFHHG����� 72'2� 5HPHPEHU WKH FXUUHQW WKUHDGV VWDFN�� DQG WKH IDFW LW DOORFDWHG GLUUHWXUQ GLU�

`

� $IWHU D WHVW PHWKRG H[LWV FKHFN DVVHUWLRQV� �DIWHU�� � WHVW0HWKRGV�� ^

�� 72'2� IRU WKH H[LWLQJ WKUHDG� HQVXUH DOO�� DOORFDWHG GLUV FDQ EH UHPRYHG �DUH QRW ORFNHG���� ,I WKH\ DUH ORFNHG� IDLO WKH WHVW FDVH DQG UHSRUW�� WKH GLUV DOORFDWLRQ VWDFN�`

`

SRLQWFXW QHZ7HPS'LU���FDOO�VWDWLF )LOH 5DQGRPL]HG7HVW�QHZ7HPS'LU����

� 5HPHPEHU ZKHQ D WHPSRUDU\ IROGHU LV UHTXHVWHG� �)LOH DURXQG�� � QHZ7HPS'LU�� WHVWLQJ7KUHDG�� ^)LOH GLU SURFHHG����� 72'2� 5HPHPEHU WKH FXUUHQW WKUHDGV VWDFN�� DQG WKH IDFW LW DOORFDWHG GLUUHWXUQ GLU�

`

� $IWHU D WHVW PHWKRG H[LWV FKHFN DVVHUWLRQV� �DIWHU�� � WHVW0HWKRGV�� ^

�� 72'2� IRU WKH H[LWLQJ WKUHDG� HQVXUH DOO�� DOORFDWHG GLUV FDQ EH UHPRYHG �DUH QRW ORFNHG���� ,I WKH\ DUH ORFNHG� IDLO WKH WHVW FDVH DQG UHSRUW�� WKH GLUV DOORFDWLRQ VWDFN�`

`

GDZLG�ZHLVV#FDUURWVHDUFK�FRP