Ha Confirmation, Dis Confirmation, And Information in Hypothesis Testing

8/6/2019 Ha Confirmation, Dis Confirmation, And Information in Hypothesis Testing

1/18

l~ychological Review Copyright 1987 by the American PsychologicalAssociation, Inc.1987, Vol. 94, No. 2, 211-228 0033-295X/87/$00.75

Con firm ation, Disconfirm ation, and Inform ation in H ypothesis TestingJoshua K layman and Young-Won H aC e n t e r f o r D e c i s i o n R e s e a r c h , G r a d u a t e S c h o o l o f B u s in e s s, U n i v e r s it y o f C h i c a g o

Strategies for hypothesis testing in scientific investigation an d everyday reasoning ha ve interestedboth psychologists and philosophers. A numb er o f these scholars stress the importance of disconfir.marion in reasoning and suggest hat peop le are instead p rone to a general deleterious "confirmationbias " I n part icula~ it is suggested that people tend t o test those cases that have the best c hance o fverifying curr ent beliefs rather than those tha t have the best ch ance o f falsifying them. We show,howeve~ that m any phenomena labeled "confirmation bias" are better understood in terms of ageneral pos i t i ve t e s t s t ra t e~ . With this strategy, there is a tendency to test cases that are e xpected (o rknown) to have the property o f interest rather tha n those expec ted (or known) to lack that property.This strategy is not equivalent to confirmation b ias in the first sense; we show that the positive teststrategy can b e a very good heurist ic for determining the tru th or falsi ty of a hypothesis underrealistic conditions~ It can, howeve~ ead to sy stematic errors o r inefficiencies. Th e approp riatenessof hum an h ypotheses-testing strategies and p rescriptions ab out optimal strategies mu st he under-stood in terms of he interaction betw een the strategy and the task at hand.

A subs t a n t i a l p rop o r t i on o f t he p sycho log i ca l l i t e r at u r e o nhyp o t hes i s t e s t i ng ha s dea l t w i t h i s sues o f con f i rma t i o n a nd d i s -con f i rma t i on . I n t e r e s t i n t h i s top i c w a s sp u r r ed by t he r e sea r chf i nd ings o f W a son (e .g ., 1960 , 1968) a nd by w r i t ings i n t he p h i -lo sop hy o f sc i ence ( e.g. , L a k a t o s , 1970 ; P l a tt , 1964 ; Pop p e r ,1959 , 1972) , w h i ch r e l a t ed hyp o t hes i s te s t i ng t o t he p u r su i t o fsc i ent i fi c i nqu i ry . M uch o f the w o rk i n t h i s ar ea , bo t h emp i r i ca la nd t heo r e t i cal , s t r e sses the i mp or t a nc e o f d i scon f i rm a t i on i nl ea rn i ng a nd r ea son i ng . I n con t r a s t , huma n r ea son i ng i s o f t ensa i d t o be p ro ne t o a " con f i rma t i on b i a s" t ha t h i nde r s e f f ec ti vel ea rn ing . H ow eve~ con f i rma t i on b i a s ha s me a n t d i f f e r en t t h i ngst o d i f fe r en t inves t i gat o r s , a s F i schb o f f a nd B ey t h - M a rom p o i n tou t i n a r ecen t r ev i ew (1983) . Fo r exa mp le , r e sea r che r s s t udy i ngt h e p e r c e p t i o n o f c o r r e la t i on s h a v e p r o p o s e d t h a t p e o p l e a r eo v e r ly in f l u e n ce d b y t h e c o - o c c u r r e n c e o f t w o e v e n t s a n d i n -su i ti c i ent ly in f luenced by i n s t a nces i n w h i c h one e ven t occ u r sw i t hou t t he o t he r (e .g ., A rkes & H a rkness , 1983 ; C rocke r ; 1 981 ;Jenk i n s & W a rd , 1965 ; N i sbe t t & R oss , 1980 ; Schu s t a ck &St e rnbe rg , 1981 ; Sha ldee & Mi r es , 1982 ; Sme ds lund , 1963 ;W a rd & Jenk i ns , 1965) . O t he r r e sea r che r s ha ve sugges ted t ha tp e o p l e t e n d t o d i s c r e d it o r r e i n t e r p r e t i n f o r m a t i o n c o u n t e r t o ahyp o t hes i s t hey ho ld ( e.g. , L o rd , R oss , & L ep p e~ 1979 ; N i sbe t t& R o s s , 1 9 8 0 ; R o s s & L e p p e ~ 1 9 8 0 ) o r t h e y m a y c o n d u c t b i a se dt e s t s tha t p ose l i tt le r i sk o f p roduc i ng d i scon f i rmi ng r e su l t s

This work was supported by Gran t SES-8309586 from the Decisionand Management Sciences program of the N ational Science Founda-tion. We thank Hillel Ein hom , Ward Edwards, Jackie Gnep p, WilliamGoldstein, Steven Hoch, Robin Hogarth, George Loewenstein, NancyPennington, Jay Russo, Paul Schoemaker, William Swann, Tom Tra-basso, Ryan Tweney, and three anonym ous reviewers for invaluablecomm ents o n earlier drafts.Correspondence concerning *~his article should be addressed t oJoshua Klayman, Graduate School of Business, University of Chicago,1101 East 58th Street, Chicago, Illinois 60637.211

(e .g. , Snyde r , 1981 ; Snyde r & C a mp be l l , 1980 ; Snyde r &Sw a rm, 1978) .

T he i nves t i gat i on o f hyp o t hes i s t e s ti ng ha s been conc e rnedw i t h bo t h desc r i p t ive a nd p r e sc r i p t i ve i ssues. O n t he one ha nd ,r e sea r che r s ha ve been i n t e r e s t ed i n unde r s t a nd i ng t he p rocessesby w h i ch p eo p le f o rm, t e s t , a nd r ev i se hyp o t hese s i n soc i a l ud g -men t , l og ica l rea son i ng , sc i en t i fi c i nves t iga t ion , a n d o t he r do -ma i ns . O n t he o t he r ha nd , t he r e ha s a l so been a s t r ong i mp l i ca -t i on t ha t p eop le a r e d o i ng t h ings t he w ro ng w a y a nd t ha t e f fo r t ss h o u l d b e m a d e t o c o r r e c t o r c o m p e n s a t e f o r t h e f a il in g s o f h u -m a n hy i x~ thes is t e s ti ng . T h i s con ce rn ha s b een exp r essed w i t hr ega rd t o ev e ryda y rea son i ng ( e .g ., see B rune r , 1951 ; N i sbe t t& Ross, 1980) as wel l as p rofessiona l sc ient i f ic endeavor (e .g . ,M a honey , 1979 ; P l a n , 1964) .

I n t h i s a r ti c l e , w e focus on hyp o t heses a bou t t he f a c t o r s t h a tp r e d i c t , e x p l ai n , o r d e s c r i b e t h e o c c u r r e n c e o f s o m e e v e n t o rp rop e r t y o f in t e r e s t. W e mea n t h i s b roa d ly , t o i nc lude hyp o t he -ses a bo u t ca usa t i on ( "C loud seed i ng i nc r ea ses r a i n f a ll " ) , ca t e -go r i za t i on ( " John i s a n ex t rove r t " ) , p r ed i c t i on ( "T he m a j o r r iskf a ct o rs f o r s c h iz o p h r en i a a r e . . . " ) , a n d d i ag n o si s ( " T h e m o s td i a g n o s ti c s i gn s o f m a l i g n a n c y a r e . . : ' ) . W e c o n s id e r b o t h d e -sc r i p t i ve a nd p r e sc r i p t i ve i s sues conce rn i ng i n fo rma t i on ga t he r -i ng i n hyp o t hes i s - t e s t i ng t a sks . W e i nc lude unde r t h i s r ub r i ct a sks t ha t r equ i r e t he a cqu i s i t i on o f ev i dence t o de t e rm i new h e t h e r o r n o t a h y p o t h e s is i s c o r r e c t. T h e t a s k m a y r e q u i r e t h esub j ec t t o de t e rm i ne t he t r u t h va lue o f a g i ven hyp o t hes i s (e .g .,J e n k i n s & W a r d , 1 9 6 5 ; S n y d e r & C a m p b e l l , 1 9 8 0 ; W a s o n ,1966) , o r t o f i nd t he one t r ue hy p o t hes i s a mo ng a se t o r un i ve r seo fp oss i b i l i t ie s ( e .g ., B rune r , G oodnow , & A us t i n , 1956 ; My na t t ,Do her ty , & Tweney, 1977, 197 8; Waso n, 19 60, 1968) .

T he t a sk kn ow n a s r u l e d i scove ry (W a son , 1960) se rves a s t heba s i s f o r t he dev e lop m en t o f ou r a na ly ses , w h i c h w e l a t e r ex t endt o o t he r k i nds o f hyp o t hes i s t e s ti ng . W e f ir s t exa m i ne w ha t"co n f i rm a t i on" mea ns i n hyp o t hes i s te s t ing . D i f f e r en t senses o fcon f i rma t i on ha ve been p oo r ly d i s t i ngu i shed i n t he l i t e r a t u r e ,c o n t r i b u t i n g t o m i s i n te r p r e t a ti o n s o f b o t h e m p i r i c a l f i n d in g s


2/18

2 1 2 J OS H U A K L A Y M A N A N D Y O U N G - W O N H Aa n d t h e o r e ti c a l p re s c r ip t io n s . W e p r o p o s e t h a t m a n y p h e n o m -e n a o f h u m a n h y p o t h e s i s t e s ti n g c a n b e u n d e r s t o o d i n t e r m s o fa general positive te st strategy. A ccor d i ng t o t h i s s t r a t egy , yout e st a h y p o t h e s i s b y e x a m i n i n g i n s ta n c e s i n w h i c h t h e p r o p e r t yo r e v e n t i s e x p e ct e d t o o c c u r ( t o s e e i f it d o e s o c c u r ) , o r b y e x a m -i n in g i n s t a n ce s i n w h i c h i t is k n o w n t o h a v e o c c u r r e d ( t o s ee i ft he hypo t hes i zed cond i t i ons p r eva i l ) . T h i s bas i c s t r a t egy sub-s u m e s a n u m b e r o f s tr a te g i es o r t e n d e n c ie s t h a t h a v e b e e n s u g -ges t ed f o r pa r t i cu l a r t a sks , such a s co nf i r m a t i on s t r a t egy , ver i -f i ca ti on s t ra t egy , ma t ch i ng b i a s , and i ll i ci t conve r s i on . A s so meof the se nam es i mpl y , t h i s appr o ach i s no t t heo r e t i ca l l y p rope r .We show, however , tha t the pos i t ive tes t s t ra tegy i s ac tual ly agoo d a l l -pu r pose heur i s t i c ac r os s a r ange o f hypo t hes i s - te s t i ngs i tua t i ons , i nc l ud i ng s i t ua t i ons in w h i ch r u l e s an d f eedba ck a r ep r obab i l i st i c . U nde r com mo nl y occu r r i ng cond i t i ons , th i s s tr a t -e g y c a n b e w e l l s u it e d t o t h e b a s i c g o a l o f d e t e r m i n i n g w h e t h e ro r no t a hyp o t hes i s i s co r r ec t .

N ex t , w e show how t h e pos i t i ve te s t s t r a t egy p r ov i des an i n t e -g r a ti v e f r a m e f o r u n d e r s t a n d i n g b e h a v i o r i n a v a r i e ty o f s e em -i ng l y d i spa r a t e d oma i ns , i nc l ud i ng conc ep t i den t i f ica t i on , l og i -ca l r ea son i ne , i n t u i t i ve pe r sona l i t y t e s t i ng , l e a r n i ng f r om ou t -c o m e f e e d b a c k , a n d j u d g m e n t o f c o n t i n g e n c y o r c o rr e l a ti o n .O ur t he s i s i s t ha t w hen conc r e t e , t a sk - spec i f i c i n f o r ma t i on i sl ack ine~ o r cogn i t i ve dem and s a r e h i gh , peop l e r e l y on t he pos i -t ive tes t s t ra tegy as a general defa ul t heur i s t i c . L ike an y a l l -pur -p o s e s t ra t eg y , th i s m a y l e a d t o a v a r i e t y o f p r o b l e m s w h e n a p -p l i ed t o pa r t i cu l a r s i t ua ti ons , and m any o f the b i a se s and e r r o r sdesc r i bed i n t he l i t e r a t u r e can be unde r s t ood i n t h i s l i gh t . O nt he o t he r hand , t h i s gene r a l heu r i s t i c i s o f t en qu i t e adequa t e ,a n d p e o p l e d o s e e m t o b e c a p a b l e o f m o r e s o p h i s ti c a te d s t ra t e -g i e s w hen t a sk cond i t i ons a r e f avor ab l e .

F i na ll y , w e d is cus s som e w ays in w h i ch ou r t a sk ana l ys i s canb e e x t e n d e d t o a w i d e r ra n g e o f s i tu a t io n s a n d h o w i t c a n c o n -t r i bu t e t o f u r t he r i nves t i ga t i on o f hypo t hes i s - t e s t ing p r oces se s .

C o n f i r m a t i o n a n d D i s c o n f i r m a t i o n i n R u l e D i s c o v e r yT h e R u l e D i s c ov e r y T a s k

B r ie fl y, t he r u l e d i s cov e r y ta sk can be desc r i bed a s f o l low s :T h e r e i s a c l as s o f o b je c t s w i th w h i c h y o u a r e c o n c e r n e d ; s o m eof t he ob j ec t s have a pa r t i cu l a r p r op e r t y o f i n t e r e s t and o t he r sd o n o t . T h e t a s k o f r u l e d is c o v e r y i s t o d e t e r m i n e t h e s e t o fcha r ac t e r i s t ic s t ha t d i f f e r en ti a t e t hose w i t h t h i s t a r ge t p r ope r t yf r o m t h o s e w i t h o u t i t . T h e c o n c e p t i d e n t if i c at io n p a r a d i g m i nl ea r n i ng s t ud i e s is a f ami l i a r examp l e o f a l ab or a t o r y r u l e - d i s -cove r y ta sk ( e .g . B r une r , G oo dno w , & A us t i n , 1956 ; L ev i ne ,1966 ; T r abas so & B ow er , 1968). H e r e , t he ob j ec t s m ay be , f o rexampl e , v i sua l s t i mu l i i n d i f f e r en t shapes , co l o r s , and l oca -t ions . So me cho i ces o f s ti mu l i a r e r e i n f o r ced , o t he r s a r e no t .T he l ea r ne r ' s goa l is t o d i s cove r t he r u l e o r " co nce p t " ( e.g. , r edc i rc l e s) t ha t de t e r m i nes r e i n f o r cem en t .

W ason ( 1960) w as t he f i rs t t o use t h i s t ype o f t a sk t o s t ud yp e o p l e ' s u n d e r s ta n d i n g o f th e l o g ic o f c o n f i r m a t i o n a n d d i s c o n -f i r ma t i on . H e s aw t he r u l e - d i s cove r y ta sk a s r ep r e sen t a t i ve o fan i m po r t an t a spec t o f sc i en ti f ic r ea son i ng ( s ee a lso Mah oney ,1976, 1979 ; M yn at t e t a l . , 1977, 1 978; Sim on, 1973). To i l lus-t r a t e t he pa r a l le l be t w een r u l e d i s cove r y and s c i en t if i c inves t i ga -t i on , cons i de r t he f o l l ow i ng hypo t he t i ca l ca se . Y ou a re an a s t r o -

p h y s ic i st , a n d y o u h a v e a h y p o t h e s i s a b o u t w h a t k i n d s o f s t a rsd e v e l o p p la n e t a r y s y s te m s . T h i s h y p o t h e s i s m i g h t b e d e r i v e df r o m a l a r g er th e o r y o f a s tr o p h y s i c s o r m a y h a v e b e e n i n d u c e df r o m p a s t o b s e r v a t i o n . T h e h y p o t h e s i s c a n b e e x p r e s s e d a s ar u l e , such t h a t t hose s t a r s t ha t have t he f ea t u r e s spec i f i ed i n t her u l e a r e h y p o t h e s i z e d t o h a v e p l a n et s a n d t h o s e n o t f i r i n g t h er u l e a r e h y p o t h e s i z ed t o h a v e n o p l a n e ts . W e w i ll u s e t h e s y m b o lR H f o r t he hy po t hes i zed r u l e , H f o r the s e t o f i n s t ances t ha t f i tt ha t hypo t hes i s , and H f o r t he s e t t ha t do no t f i t i t. T he r e i s ad o m a i n o r " u n i v e r s e " t o w h i c h t h e r u l e i s m e a n t t o a p p l y (e .g .,a l l s t a r s in o u r ga l axy) , and i n t ha t d om a i n t he r e i s a t a r ge t s et( t hose s t a rs t ha t r ea l l y do have p l ane t s ). Y ou w ou l d l i ke t o f i ndt h e r u l e t h a t e x a c t ly s p e ci fi e s w h i c h m e m b e r s o f t h e d o m a i n a r ei n t he t a r ge t s e t ( t he r u l e t ha t de sc r i bes exac t l y w ha t t ype o fs t a r s have p l ane ts ) . W e w i ll u se T f o r the t a r ge t s e t, an d R x f o rt he " co r r ec t " r u l e , w h i ch spec if i es t he t a r ge t s et exac tl y. L e t u sa s sum e f o r now t ha t such a pe r f ec t r u l e ex is ts . ( A l t e r na t e ve r -s i o n s o f t h e r u l e m i g h t e x i st , b u t f o r o u r p u r p o s e s , r u l e s c a n b econs i de r e d i den t i ca l i f t hey spec i f y exac t l y t he s am e se t T .) T hec o r r e c t r u l e m a y b e e x t r e m e l y c o m p l e x , i n c l u d i n g c o n j u n c -t i ons , d i s j unc t i ons , and t r ade - o f f s am on g f ea t u r e s . Y our goa l a sa s c i en ti s t, t hou gh , i s t o b r i ng t he hyp o t hes i zed r u l e R H i n l i new i t h t h e c o r r e c t r u l e R x a n d t h u s t o h a v e t h e h y p o t h e s i z e d s e tH m a t c h t h e t a r g e t s e t T. Y o u c o u l d t h e n p r e d i c t e x a c tl y w h i c hs t a r s do a nd d o n o t have p l ane t s . S i mi l a r l y , a p sycho l og i s t m i g h tw i sh t o d i f f e r en t i a t e t hose w ho a r e a t r i sk f o r s ch i zophr en i af r o m t h o s e w h o a r e n o t , o r a n e p i d e m i o l o g i s t m i g h t w i s h t ou n d e r s t a n d w h o d o e s a n d d o e s n o t c o n t r a c t A I D S . T h e s a m es t r u c t u r e c a n a l s o b e a p p l i e d i n a d i a g n o s t ic c o n t e x t. F o r e x a m -p l e , a d i a g n o s t i c i a n m i g h t s e e k t o k n o w t h e c o m b i n a t i o n o fs i gns t ha t d i f f e r en t ia t e s ben i gn f r om m a l i gnan t t umo r s .

I n e a c h c a s e , a n i m p o r t a n t c o m p o n e n t o f t h e i nv e s ti g at iv ep r oces s i s t he t e s t ing o f hypo t heses . T ha t i s , the i nves t i ga t o rw an t s t o kno w i f t he hy po t hes i ze d r u l e R H i s i den t i ca l t o t hecor r ec t r u l e R z and i f no t , how t he y d if fe r. T h i s i s accom pl i shedt h r o u g h t h e c o l le c t io n o f e v i d e n ce , t h a t i s , th e e x a m i n a t i o n o fi n s ta n c e s . F o r e x a m p l e , y o u m i g h t c h o o s e a s t a r h y p o t h e s i z e dt o have p l an e t s and t r a i n y ou r t e le scope on i t t o s ee i f i t doesi n d e e d h a v e p l a n e ts , o r y o u m i g h t e x a m i n e t u m o r s e x p e c t e d t ob e b e n i g n , t o s e e i f a n y a r e i n f a c t m a l i g n a n t .

W ason ( 1960 , 1968) deve l oped a l abor a t o r y ve r s i on o f r u l ed i s cove r y t o s t ud y peop l e ' s hypo t hes i s - t e s t ing s t r a t eg ie s ( i n pa r -t i cu l a r , t he i r u se o f conf i r m a t i on an d d i s co nf i r ma t i on ) , i n a t a skt ha t " s i mul a t e s a m i n i a t u r e s c i en t i f i c p r ob l em" ( 1960 , p . 139) .I n W ason ' s t a sk , t he un i ve r se w as ma de up o f a ll pos s ib l e s e tso f th r ee nu mb er s ( " t r i p l e s " ) . Som e o f t he se t r ip l e s f it t he r u l e ,i n o t h e r w o r d s , c o n f o r m e d t o a r u l e t h e e x p e r i m e n t e r h a d i nmi n d . I n ou r te r ms , f i tt i ng t he expe r i m en t e r ' s r u l e i s t he t a r ge tp r op e r t y t ha t sub j ec t s mus t l e a r n t o p r ed i c t . T h e t r ip l e s t ha t f itt he r u l e , t hen , cons t i t u t e t he t a r ge t s e t , T . Sub j ec t s w e r e p r o -v i ded w i t h on e t a r ge t t ri p l e ( 2 , 4 , 6 ), and cou l d a sk t he expe r i -m e n t e r a b o u t a n y o t h e r s t h e y c a r e d t o. F o r e a c h t ri p l e t h e s u b -j e c t p r o p o s e d , t h e e x p e r i m e n t e r r e s p o n d e d yes ( f i ts the rule) o rn o ( does no t f it ). A l t ho ugh sub j ec t s m i gh t s t a r t w i t h o n l y av a g u e g u e ss , t h e y q u i c k l y f o r m e d a n i n it ia l h y p o t h e s i s a b o u t t h er u l e ( R H ) . For exampl e , t hey mi gh t gues s t ha t t he r u l e w as" t h r e e c o n s e c u t i v e e v e n n u m b e r s . " T h e y c o u l d t h e n p e r f o r mo n e o f t w o t y p e s o f h y p o t h e s i s t es t s (H t e s ts ) : t h e y c o u l d p r o p o s ea t r ip le they expected to be a t arget (e .g . , 6 , 8 , 10) , or a t r ip le


3/18

CONFIRMATION, DISCONFIRMATION, AND INFORMATION 21 3

Figure 1. Representation of a situation in which the hyp othesized rule is embedded within the correct rule,as in W ason's (1960) "2, 4, 6 " task. (U = the universe of possible instances [e.g., all triples o f numbers];T = the set of instances that have the target p r o l ~ [e.g., they fit the experimenter's rule: increasing];H = the set o f instances that fit the hypothesized rule [e.g., increasing by 2].)

they expe cted n ot t o be (e.g., 2, 4, 7). In this paper, we will referto these as a positive hypothesis test (+Htest) and a negativehypo thesis test (-H tes t), respectively.

W a s o n f o u n d t h a t p e o p le m a d e m u c h m o r e u s e o f + H t e s t sthan -Htes t s . The subjec t whose hypothes i s was "consecu-t ive evens," for example , would t ry man y exam ples o f consec-ut ive-even t r iples and relat ively few others . Subjects ofte n be-came qui t e conf ident of the i r hypotheses a f t e r a s e ri es of +Ht -es ts only. In Wason's (1960) task this confidence was usual lyunfo und ed, for reasons we discuss la ter . Wason described thehypothes i s t e s t e rs as " seeking conf i rmat ion" because theylooked pred omina nt ly a t cases tha t f i t t he i r hypothes ized ru lefor targets (e .g. , di fferent sets of consecut ive even numb ers) .We th ink i t more appropr ia te to v iew th i s "conf i rmat ionbias " as a mani fes ta t ion of the genera l hypothes i s - t est ings trategy we cal l the posi t ive tes t (+ tes t ) s t rategy. In rule dis-covery , the + tes t s t ra tegy l eads to the predo min ant use of+Htes t s , in o the r words , a t endency to t e s t cases you th in kwil l have the target p roperty.

The general tendency toward +tes t ing has been widely repl i-cated. In a variety o f different rule-discovery tasks (Klaym an& Ha, 1985; Maho ney, 1976, 1979; Myn att et al., 1977, 1978;Taplin, 1975; Tweney et al., 1980; Wason & Johnson-Laird,1972) people look pred omina nt ly at cases they expect wil l havethe target property, rather than cases they expect will not. Aswith nearly all strategies, people do not seem to adhere strictlyto +tes t ing, however. For ins tance, given an adequate num ber of

tes t opportuni t ies and a lack of pressure for a quick evaluat ion,people seem wil l ing to tes t more widely (Gorman & Gorman,1984; Klay man & Ha, 1985). Of part icular interes t is one m a-nipulat ion that great ly impro ved success at Wason's 2, 4, 6 task.Tweney et al. (1980) used a task structura lly identical to Wa-son's but modified the presentat ion of feedback. Triples wereclass if ied as ei ther DAX or M ED, rather than y e s (fits the rule)o r n o (does not f i t ) . The rule for DAX was Wason's originalascending-order rule , an d al l other t r iples were MED . Subjectsin the DAX /M ED version used even fewer -Htes t s than usua l.Howeve~ they t reated the DAX rule and the MED rule as twoseparate hypotheses , and tes ted each with +Htes ts , thereby fa-cilitating a solution.

The th rus t of this work has been m ore tha n jus t descript ive,however. T here has been a s t rong emphasis on the not ion that a+test strategy (or something like it) will lead to serious errorsor inefficiencies in the testing of hypotheses. W e begin by takin ga closer look at this assumption. We examine what phi losophersof science such as Popper and Plat t have been arguing, and howthat t rans lates to prescript ions for information gathering indifferent hypothesis-testing situations. We then examine thetask characteris t ics that control the extent to which a +tes ts t rategy deviates from those prescript ions . We begin with rulediscovery as described above, and then consider what happensif addi t ional in forma tion is available (examples o f known tar-gets and nontargets) , and i f an element o f probabil is t ic erro r isintroduced. The bas ic ques t ion is , i f you are t rying to d etermine

Figure 2. Representation of a situation in which the hypothesized rule overlaps the correct rule.


4/18

21 4 JOSHUA KLAYMAN AND YOUNG-WO N HA

Figure 3. Representation of a situation in which the hypothesized rul e surro und s the correct rule.

the truth or falsity of a hypothesis, when is a + test strateg y un-wise and whe n is it not?The Logic o f Amb iguous Versus Conclusive Events

As a class, laborato ry rule-discovery tasks share three simpli-fying assumptio ns. First, feedb ack is deterministically accurate.The experimenter provides the hypothes is tes ter with error-freefeedback in accordance with an underlying rule. Second, thegoal is to determine the one correct rule (RT). All other rulesare class if ied as incorrect , wi thout regard to how wrong Rx maybe, although the tester may be concerned with where i t is wrongin order to form a new hypothes is . Third, correctness requiresbot h sufficiency and necessity: A rule is incorrect if it predictsan instance will be in the target set when it is not (false positive),or predicts it will not be in the target set when it is (false nega-tive). We discuss later the extent to which e ach o f these assu mp-tiom restricts generalization to other tasks.

Consider again Wason's original task. Given the triple (2, 4,6) , the hypotheses that occur to m ost people are "consecut iveeven numbers ; ' "increasing by 2" and the l ike. The correctrule, however, is much broade~. "increasing numbers : ' Con-s ider subjects whose hypothes ized rule is " increas ing by 2"Those wh o use only +Htes ts ( t r iples that increase by 2, such as6, 8, 10) can n ever discover that their rule is incorrect, bec auseal l examples of " increas ing by 2 " also f i t the rule of " increas-ing? ' Thus , i t i s crucial to t ry -Htes ts ( t r iples that do not in-crease by 2, such as 2, 4, 7). This situation is depicted in Figure1. Here, U represents th e universe o f instances, all possible tri-ples of num bers. T rep resents the target set, triples that fit theexperimenter ' s rule (" increas ing"). H represents the hypothe-sized set, triples that fit the tester s h ypothesiz ed ru le (say, "in-creasing by 2"). Ther e are in prin ciple fo ur classes of instances,al though they do not a l l exis t in this part icular example:

1. H n T: instances correctly hypothesized to be in the target set(positive hits).2. H n T: instances incorrectly hypothesized to be in the target set(false positives).3.

4.

H n T: instances m e e t l y hypothesized to be outside the targetset (negative hits).n T: instances incorrec tly hypothesized to be outside the tar-get set (false negatives).

Instance s of the types H n T and H n T falsify the hypothesis.Th at is, the occu rrenc e o f either shows conclusively that H 4= T,thus Rx ~ Rx; the hypothes ized rule is not the correct one.Instance s of the types H n T and H n T verify the hypothesis,in the sense of providing favorable evidence. However, these in-s tances are ambiguous: The hypothes is may be correct , butthese ins tances can occur even i f the hypothes is is not correct .Note that th ere are o nly conclusive falsifications, no conclusiveverifications. This logical condition is the back bon e o f philoso-pities of science that urge investigators to seek falsificationrather th an verification of their hypo theses (e.g., Popper, 1959).Put somewhat simplistically, a lifetime of verifications can becoun tered by a single conclusive falsification, so it makes sensefor scientists to make the discovery of falsifications their pri-mary goal .

Suppose, then, tha t you are the tes ter in Wason's task, withthe hypothes is of" in creas in g by 2 7 If you t ry a +H tes t (e .g. , 6,8, 10) you will get either a yes response, w hich is an a mbi guo usverification of the type H n T, or a no, which is a conclusivefalsification of the type H n T. The falsification H n T wouldshow that meet ing the condi t ions o f your rule is not sufficientto guarantee me mbers hip in T. Thus , +Htes ts can be said to betests of the ru le's sufficiency. However, unkno wn to the subjectsin the 2, 4, 6, task (Figure 1) there are no ins tances of H N T,because the hypoth esized rule is sufficient: Any instance follow-ing RH ("increasing by 2") will in fact be in the target set T("increasing "). Thu s, +H test s will never pro duc e falsification.If you ins tead t ry a -H te s t (e .g. , 2, 4, 7) you wil l get e i ther a noanswer which is an ambiguous veri f icat ion (H n T) or a yesanswer which is a conclusive falsification (H n T). Th e falsifica-tion A n T shows that your conditions are not necessary formembership in T. Thus , -Htes ts tes t a rule 's necess i ty. In the2, 4, 6 task, -Htests can result in conclusive falsification be-cause RH is sufficient but not necessary (i.e., there are sometarget triples that do not increase by 2).

In the above situation, the Popperian exhortation to seek fal-s i fication can be fulfi l led only by -Ht es t in & and those who relyon + Htes ts are l ikely to be mis led by the abundan t veri ficationthey receive. Indeed, Wason deliberately designed his task sothat this would be the case, in order to show the pi t fal ls of "con-flrmation bias" (Wason, 1962). The hypothesis-testcr's situa-tion is not always like this, however. Consider the situation inwhich the hypothesized set merely overlaps the target set, asshown in Figure 2, ra ther than being embedded within it , as


5/18

CONFIRMATION, DISCONFIRMATION, AND INFORMATION 21 5

Figure 4. Representation of a situation in which the hypothesized rule a nd the correct rule are disjoint.

shown in Figure 1. This would be the case if , for example, thecorrect rule were " three even numbers ." The re would be somemem bers o f H N T, ins tances that were " increas ing by 2" b utnot " three evens" (e.g. , l , 3, 5) , and some mem bers o fH N T," three evens" but not " increas ing by 2" (e .g. , 4, 6, 2) . Thus ,conclus ive fals i ficat ion could oc cur with ei ther +Hte s ts o r - H -tests. Indeed, it is possible to be in a situation jus t the oppositeof Wason's , shown in Figure 3. Here, the hypothes is is too br oadand "surro und s" the target set. Th is would be the case i f thecorrect rule were, say, "consecutive even numbers." Now a tes-ter who did only -Htests could be sorely misled, because thereare no falsifications of the type H rl T; any insta nce tha t violates" increas ing by 2" also violates "consecut ive evens ." Only + H-tests can reveal conclusive falsifications (H A T instances suc has 1, 3,5).

Aside from these three situations, there are two other possiblerelat ionships between H and T. When H an d T are dis joint (Fig-ure 4), any +Htest will produce conclusive falsification, be-cause nothing in H is in T; -H tes ts could pro duce ei ther veri fi -cation or falsification. This is not likely in the 2, 4, 6 task, be-cause you are given one kno wn target ins tance to begin with. Inthe last case (Figure 5), yo u have finally fou nd the co rrect rule,and H coincides with T. Here, every tes t produces am biguo usinfo rmati on; a final pr oo f is possible only if there is a finite uni-verse of instances and every case is searched.

In natural ly o ccurring s i tuat ions , as in Wason's (1960) task,one could f ind oneseif in a ny o f the condi t ions depicted, usual lywith no way of knowing which. Suppose, for example, that y ouare a ma nufacturer t rying to determ ine the bes t way to advert iseyou r l ine of products , an d yo ur curren t hypo thes is is that te levi-

s ion commercials are the m ethod of choice. For you, the uni-verse, U, is the set of possible advertising m ethod s; the targetset, T, is the set of metho ds tha t are effective, and the hyp othe -sized set, H, is television commercials. Suppose that in fact theset of effective advert is ing m ethods for these produ cts is muchbroader: any visual medium (magazine ads , e tc .) wil l work.This is the s i tuat ion depicted in Figure 1. If you t ry +Htes ts(i.e., try instances in y ou r hypothesize d set, television comm er-cials) you will never discover that you r rule is wrong, becausetelevision comm ercial s will be effective. On ly by tryin g thingsyou think will not work (-H tes ts ) can you ob tain fals if icat ion.You migh t then discover an ins tance o f the type R N T: nontele-vision advertising that is effective.

Suppose instead that the correct rule for effectively advertis-ing these produ cts is to use humor. This is the s i tuat ion in Fig-ure 2. You could f ind a (serious) te levision com mercial th at y outhought would work, but does not (H N T), or a (humorous)nontelevis ion ad that you thought would not work, but does(H N T). Thus, conclusive falsification could occur with eithera +Hte s t or a -H tes t . If ins tead the correct rule for these prod-ucts is more restricted, say, "prime-time television only," youwould have an overly broad hypothes is , as shown in Figure 3.In that case, you will never obtain fals i ficat ion i f you use -H -tes ts ( i.e. , f f you ex periment with me thods y ou think will notwork), because anything that is not on televis ion is a lso not o nprime time. Only +Htests can reveal conclusive falsifications,by f inding ins tances of H N T (ins tances of te levis ion com mer-cials that are not effective).

Wh at is cr i tical, then, is not the tes t ing of cases that do notf it your hypothes is , but the tes t ing of cases that are m ost l ikely

Figure 5. Representation o f he situation in which the hypothesized rule coincides with the correct rule.


6/18

2 1 6 J OS H UA K .L Y M A N A N D Y O U N G - W O N H At o p r o v e y o u w r o n g . I n W a s o n ' s t a s k th e s e t w o a c t i o n s a r e i d e n -t i ca l , b u t a s s h o w n i n F i g u r e s 2 t h r o u g h 5 , th i s i s n o t g e n e r a l lys o . T h u s , i t i s v e r y i m p o r t a n t t o d i s t i n g u i sh b e t w e e n t w o d i ff e r-e n t s e n s es o f " s e e k i n g d i s c o n f i r m a t i o n : ' O n e s e n se i s t o e x a m -i n e i n s t a n c e s t h a t y o u p r e d i c t w i l l n o t h a v e t h e t a r g e t p r o p e r t y .T h e o t h e r s e n se i s t o e x a m i n e i n s t a n c e s y o u m o s t e x p e c t t o f a l -s if y , r a t h e r t h a n v e r if y , y o u r h y p o t h e s i s . T h i s d i s t i n c t i o n h a s n o tb e e n w e l l r e c o g n i z e d i n p a s t a n a l y s e s , a n d c o n f u s i o n b e t w e e nt h e t w o s e n se s o f d i s c o n f i r m a t i o n h a s f i g u r e d i n a t l e a s t tw op u b l i s h e d d e b a t e s , o n e i n v o l v i n g W a s o n ( 1 9 6 0 , 1 9 6 2 ) a n dW e t h e r i c k ( 19 6 2 ), t h e o t h e r i n v o l v i n g M a h o n e y ( 1 9 7 9 , 1 9 80 ),H a r d i n ( 1 9 8 0 ) , a n d T w e n e y, D o h e r ty , a n d M y n a t t ( 1 98 2 ) . T h ep r e s c r i p t i o n s o f P o p p e r a n d P l a t t e m p h a s i z e t h e i m p o r t a n c e o ff a l s if i c a t io n o f t h e h y p o t h e s i s , w h e r e a s e m p i r i c a l i n v e s t i g a t i o n sh a v e f o c u s e d m o r e o n t h e t e s ti n g o f in s t a n c e s o u t s i d e th e h y -p o t h e s i z e d s e t.

C o n f i r m a t i o n a n d D i s c o n f l r m a t i o n :W h e r e ' s t h e I n f o r m a t i o n ?

T h e d i s t i n c t io n b e t w e e n - t e s t i n g a n d s e e k i n g fa l s if i c at i onl e a d s to a n i m p o r t a n t q u e s t i o n f o r h y p o t h e s i s t es t er s : G i v e n t h ec h o i c e b e t w e e n + t e s t s a n d - t e s t s , w h i c h i s m o r e l i k e ly t o y i e l dc r i t i c a l fa l s i f ic a t i o n ? A s i s i l l u s t r a t e d i n F i g u r e s I t h r o u g h 5 , t h ea n s w e r d e p e n d s o n t h e r e l a t i o n b e t w e e n y o u r h y p o t h e s i z e d s e ta n d t h e t a r g e t s e t . T h i s , o f c o u r s e , is i m p o s s i b l e t o k n o w w i t h o u tf ir s t k n o w i n g w h a t t h e t a r g e t s e t i s. E v e n w i t h o u t p r e s c i e n c e o ft h e t r u t h , h o w e v e r , i t is p o s s i b l e f o r a t e s t e r t o m a k e a r e a s o n e dj u d g m e n t a b o u t w h i c h k i n d o f t e s t t o p e r f o r m . P r e s c r i p t i o n sc a n b e b a s e d o n ( a t l ea s t ) t w o c o n s i d e r a ti o n s : ( a ) W h a t t y p e o fe r r o r s a r e o f m o s t c o n c e r n , a n d ( b ) W h i c h t e s t c o u l d b e e x -p e c t e d , p r o b a b i l i s t i c a l l y , t o y i e l d c o n c l u s i v e f a l s i f i c a t i o n m o r eo f te n . T h e f i r st p o i n t h i n g e s o n t h e f a c t t h a t + H t e s t s a n d - H -t e s t s r e v e a l d i f f e r e n t k i n d s o f e r r o r s ( f a l s e p o s i ti v e s a n d f a l s en e g a t i v e s , r e s p e c t iv e l y ) . A t e s t e r m i g h t c a r e m o r e a b o u t o n et h a n t h e o t h e r a n d m i g h t b e a d v i s e d t o t e s t a c c o r d in g l y . A l -t h o u g h t h e r e i s a l m o s t a l w a y s s o m e c o s t t o e i t h e r ty p e o f e rr o r ,o n e c o s t m a y b e m u c h h i g h e r t h a n t h e o t h e r . F o r e x a m p l e , ap e r so n n e l d ir e c t o r m a y b e m u c h m o r e c o n c e r n e d a b o u t h i r i n ga n i n c o m p e t e n t p e r s o n ( H N T ) t h a n a b o u t p a s s i n g o v e r s o m ep o t e n t i a l l y c o m p e t e n t o n e s ( H f~ T ) . S o m e o n e i n t h i s p o s i t i o ns h o u l d fa v o r + H t e s t s ( e x a m i n i n g a p p l i c a n t s u d g e d c o m p e t e n t ,t o f i n d a n y f a i l u r e s ) b e c a u s e t h e y r e v e a l p o t e n t i a l f a l s e p o s i t i v e s .O n t h e o t h e r h a n d , s o m e s i t u a t io n s r e q u i r e g r e a t e r c o n c e r n w i t hf a l s e n e g a t i v e s t h a n f a l s e p o s i t i v e s . F o r e x a m p l e , w h e n d e a l i n gw i t h a m a j o r c o m m u n i c a b l e d i s ea s e , i t i s m o r e s e r i o u s t o a l l o wa t r u e c a s e t o g o u n d i a g n o s e d a n d u n t r e a t e d ( f i N T ) t h a n i t i st o m i s t a k e n l y t r e a t s o m e o n e ( H N T ) . H e r e t h e e m p h a s i s s h o u l db e o n - H t e s t s ( e x a m i n i n g p e o p l e w h o t e s t n e g a t iv e , t o f i n d a n ym i s s e d c a s e s ) , b e c a u s e t h e y r e v e a l p o t e n t i a l f a l s e n e g a t i v e s .

I t c o u l d b e , t h e n , t h a t a p r e f e r e n c e f o r + H t e s t s m e r e l y r e f l e c t sa g r e a t e r c o n c e r n w i t h s u f f i c i e n c y t h a n n e c e s s i ty . T h a t i s , t h et e s te r m a y s i m p l y b e m o r e c o n c e r n e d t h a t a l l c h o s e n c a s e s a r et r u e t h a n t h a t a l l tr u e c a s e s a r e c h o s e n . F o r e x a m p l e , e x p e r i -m e n t s b y V o g e l a n d A n n a u ( 1 9 7 3) , T s c h i rg i ( 1 9 8 0 ), a n dS c h w a r t z ( 1 9 8 t , 1 9 8 2) s u g g es t t h a t a n e m p h a s i s o n t h e s u f fi -c i e n c y o f o n e ' s a c t i o n s i s e n h a n c e d w h e n o n e i s r e w a r d e d f o re a c h i n d i v i d u a l s u c c e ss r a t h e r t h a n o n l y f o r t h e f i n a l r u l e d i s c o v -e ry . C e r ta i n ly , in m a n y r e a l s i t u a t i o n s ( c h o o s i n g a n e m p l o y e e ,

a j o b , a s p o u s e , o r a c a r ) p e o p l e m u s t s i m i l a r l y l i ve w i t h t h e i rm i s t a k e s . T h u s , p e o p l e m a y b e n a t u r a l l y i n c l i n e d t o f o c u s m o r eo n f a l s e p o s i t i v e s t h a n o n f a l s e n e g a t i v e s i n m a n y s i t u a t i o n s . At e n d e n c y t o w a r d + H t e s t i n g w o u l d b e e n t i r e l y c o n s i s t e n t w i t hs u c h a n e m p h a s i s . H o w e v e r , i t i s st i ll p o s s i b le t h a t p e o p l e r e t a i na n e m p h a s i s o n s u f fi c ie n c y w h e n i t i s i n a p p r o p r i a t e ( a s i n W a -s o n ' s t a s k ) .

S u p p o s e t h a t y o u a r e a t e s t e r w h o c a r e s a b o u t b o t hs u f f ic i e n c y a n d n e c e s s it y : y o u r g o a l i s s i m p l y t o d e t e r m i n ew h e t h e r o r n o t y o u h a v e f o u n d t h e c o r r e c t r u l e . I t i s s t i ll p o s s i b l et o a n a l y z e t h e s i t u a t i o n o n t h e b a s i s o f r e a s o n a b l e e x p e c t a t i o n sa b o u t t h e w o r l d . I f y o u a c c e p t t h e r e a s o n i n g o f P o p p e r a n d P l a t t ,t h e g o a l o f y o u r t e s t i n g s h o u l d b e t o u n c o v e r c o n c l u s i v e fa l s if i -c a t i o n s. W h i c h k i n d o f t e s t, t h e n , s h o u l d y o u e x p e c t t o b e m o r el ik e l y t o d o s o ? A s s u m e t h a t y o u d o n o t k n o w i n a d v a n c ew h e t h e r y o u r h y p o t h e s i z e d s et i s e m b e d d e d i n , o v e r la p s , o r s u r -r o u n d s t h e t a r g e t. T h e g e n e r a l c a s e c a n b e c h a r a c t e r i z e d b y f o u rq u a n t i t i e s l :

p( t ) The overall base-ra te proba bi l i ty that a mem ber ofthe do main i s in the target se t. This would b e , for ex-amp le , the p ropor t ion of s tars in the galaxy that haveplanets.

p (h ) The ove ra ll p robab i l i ty h a t a m em ber o f t he dom a inis in the hypo thes ized se t. This w ould be the pro por-t ion o f s tars that f i t you r hypothes ized cr i ter ia for hav-ing planets.z + = p(~[h) The overal l probab i l i ty h at a pos i t ive predic t ion wi l lprove false, for exam ple, that a star hypothesized tohave planets wi l l turn out not t o .z- = p( t lh) The overal l probab i l i ty hat a negat ive predic t ion wi l lprove fa lse , for examp le , that a s tar hypothesized notto have planets wi l l turn out in fac t to have them.

T h e q u a n t i t i e s z + a n d z - a r e i n d e x e s o f t h e e r r o r s m a d e b y t h eh y p o t h e s is . T h e y c o r r e s p o n d t o t h e f a l se - p o s i ti v e r a t e a n d f a ls e -n e g a t i v e r a t e f o r t h e h y p o t h e s i z e d r u l e R H (c f . E i n h o r n & H o -g a r t h , 1 9 7 8 ). I n o u r a n a l y s e s , a l l f o u r o f th e a b o v e p r o b a b i l i t i e sa r e a s s u m e d t o b e g r e a t e r t h a n z e r o b u t l e s s t h a n o n e ? T h i sc o r r e s p o n d s t o t h e c a s e o f o v e r l a p p i n g t a r g e t a n d h y p o t h e s i ss e t s , a s s h o w n i n F i g u r e 2 . H o w e v e r , o t h e r s i t u a t i o n s c a n b e r e -g a r d e d a s b o u n d a r y c o n d i t i o n s to t h i s g e n e r a l c a s e. F o r e x a m -p l e , t h e e m b e d d e d , s u r r o u n d i n g , a n d c o i n c i d e n t s i t u a t i o n s( F i g u r e s 1 , 3 , a n d 5 ) a r e c a s e s i n w h i c h z + = p ( t ] h ) = 0 , z - =p ( t l h ) = 0 , o r b o t h , r e s p e c ti v e l y , a n d i n t h e d i s j o i n t s i t u a t i o n(F ig u re 4 ) , z + = 1 .

R e c a l l t h a t t h e r e a r e t w o s e t s o f c o n c l u s i v e f a l s i f ic a t i o n s : H NT ( y o u r h y p o t h e s i s p r e x i ic t s p l a n e t s , b u t t h e r e a r e n o n e ) , a n df3 T ( y o u r h y p o t h e s i s p r e d i c t s n o p l a n e t s, b u t t h e r e a r e s o m e ) .I f y o u p e r f o r m a + H t e s t , t h e p r o b a b i l i t y o f a c o n c l u s i v e fa l si fi -c a t i o n , p ( F n J + H t e s t ) , i s e q u a l t o t h e f a l s e p o s i t iv e r a t e , z + .I f y o u p e r f o r m a - H t e s t , t h e c h a n c e o f f a ls i fi c a ti o n ,

We use a lowercase letter to designate an instanc e of a given type: ti s an ins tance in se t T, Tis an ins tance in T , and so on.2 Ou r analyses t rea t the se ts U , T, and H as f in ite , but a lso apply toinfinite sets, as long as T and H designate finite, nonze ro fractions o f U.In Wason ' s task (1960) , for example , i fU = a l l se ts of hree num bers andH = a l l se ts of three even numbers , then we can say that H des ignates t / sof a l l t he m em ber s o f U , i n o the r w ords , ~ h ) = l/ s.


7/18

CONFIRMATION, DISCONFIRMATION, AN D INFORMATION 21 7p ( F n I - H t e s t ) , i s e q u a l t o t h e f a ls e n eg a t iv e r a te , z - . A P o p p e r -i a n h y p o t h e s is - te s te r m i g h t w i s h t o p e r f o r m t h e t y p e o f t es t w i t ht he h i ghe r expec t ed c han ce o f f a ls i fi ca ti on . O f cour se , you can -n o t h a v e a n y d i r e c t e v i d e n c e o n z + a n d z - w i t h o u t o b t a i n i n gs o m e f a l si fi c at io n , a t w h i c h p o i n t y o u w o u l d p r e s u m a b l y f o r ma d i f f e r en t hypo t hes i s . H ow eve r , t he cho i ce be t w een t e s t s doesn o t d e p e n d o n t h e v a l u e s o f z + a n d z - p e r s e, b u t o n t h e r e l a ti o n -sh i p be t w een t hem , and t ha t i s a f unc t i on o f tw o quan t i f i e sa b o u t w h i c h a n i n v e s ti g a to r m i g h t w e l l h a v e s o m e i n f o r m a t i o n :p ( t ) a n d p ( h ) . W h a t i s r e q u i r e d i s a n e s t i m a t e o f t h e b a s e r a t eo f th e p h e n o m e n o n y o u a r e t r y i n g t o p r e d i c t (e .g ., w h a t p r o p o r -t i o n o f s t ar s h a v e p l a ne t s , w h a t p r o p o r t i o n o f t h e p o p u l a t i o nf all s v i c ti m t o s c h i z o p h r e n i a o r A I D S , w h a t p r o p o r t i o n o f tu -m o r s a r e m a l i g n a n t ) a n d a n e s t i m a t e o f th e p r o p o r t i o n y o u rh y p o t h e s i s w o u l d p r e d ic t . T h e n

z + = p(~lh) = 1 - p( t th )= 1 - p ( t f 3 h ) / p ( h )= 1 - [ p ( t ) - p ( t f 3 h ) ] / p ( h )

p ( t ) p ( t l h ) . p ( I S )= 1 - ~ + p ( h )

z + = _ p ( h ) . ( 1 p ( t)z < l )A c c o r d i n g to E q u a t i o n 1 , e v e n if y o u h a v e n o i n f o r m a t i o n

a b o u t z + a n d z - , y o u c a n e s t im a t e t h e i r r e l a ti o n s h i p f r o m e s ti -m a t e s o f t h e t a rg e t a n d h y p o t h e s i s b a s e r a t e s, p ( t ) a n d p ( h ) . I ti s no t neces sa r i l y t he ca se t ha t t he t e s t e r know s t hese quan t i t i e sexac tl y . H ow eve r , the r e i s u sua l l y som e ev i denc e ava i l ab le f o rf o r m i n g e s t i m a t es o n w h i c h t o b a s e a j u d g m e n t . I n a n y c a s e , i ti s u sua l l y ea s i e r t o e s t i ma t e , s ay, how m an y peop l e su f f e r f r oms c h i z o p h r e n i a th a n i t is t o d e t e r m i n e t h e c o n d i t i o n s t h a t p r o -duce i t .

I t s e e m s r e a s o n a b le t o a s s u m e t h a t i n m a n y c a s es t h e t e st e r 'shypo t he s i s is a t l e a s t abo u t t he r i gh t s ize . Peop l e a r e no t l i ke lyt o p u t m u c h s t o c k i n a h y p o t h e s i s t h a t t h e y b e li e ve g r e a tl y o v e r-p r e d i c t s o r u n d e r p r e d i c t s t h e t a r g e t p h e n o m e n o n . L e t u s a s -s u m e , t h e n , t h a t y o u b e li e v e t h a t p ( h ) ~ p ( t ). U n d e r t h e s e c i r -c u m s t a n c e s , E q u a t i o n 1 c a n b e a p p r o x i m a t e d a s

z + = P t ) . z - ( 2 )p( t )T h u s , i f p ( t ) < . 5, t h e n z + > z - , w h i c h m e a n s t h a tp ( F n l + H t e s t ) > p ( F n l - H t e s t ) . I n o t h e r w o rd s , if y o u a r e a t -t e m p t i n g t o p r e d i c t a m i n o r i t y p h e n o m e n o n , y o u a r e m o r el i ke l y t o r ece i ve f a l s i f i ca t i on us i ng + H t es t s t han - H t e s t s . Wew ou l d a r gue t ha t , i n f ac t , r ea l - w or l d hypo t hes i s t e s ti ng mo s t o f -t e n c o n c e r n s m i n o r i t y p h e n o m e n a . F o r e x a m p l e , a r e c e n t e s ti -ma t e f o r t he p r op or t i on o f st a r s w i t h p l ane t s i s I/3 ( Sagan , 1980 ,p . 300) , f o r t he p r eva l ence o f s ch i zophr en i a , l e s s t han 1%( A m er i can Psych i a t r i c A ssoc i a t i on , 1980) , and f o r t he i nc i -d e n c e o f A I D S i n t h e U n i t e d S t a t es , s o m e t h i n g b e t w e e n 1 0 4and 10 5 ( C en t e r s f o r D i sease C on t r o l , 1986) . E ven i n W ason ' so r i g in a l t a s k ( 1 96 0 ) , t h e r u l e t h a t s e e m e d s o b r o a d ( a n y i n c r e a s -i ng ) has a p ( t ) o f on l y 1 /6 , a s sum i ng on e cho oses f r om a l a r ger a n g e o f n u m b e r s . I n d e e d , i f p ( t ) w e r e g r e a te r t h a n .5 , t h e p e r -cep t i on o f t a r ge t and n on t a r ge t w ou l d l i ke l y r eve rse . I f 80% o f

T ab l e 1Condit ions Favor ing + Htes t s or -H tes t s as Means o fOb tainin g Conclusive FalsificationTarget and hypothesis Comparison of probabili ty of falsificationbase rates (In ) for +Htests and -H tests"

p(t) < .5p( t ) > p(h)p( t ) = p(h)P(0 < p(h) ~ .5p( t ) < .5 .5p(t) ~ .5 > p(h)p(t) > p(h) > .5p( t ) = p(h)p(O < p(h)

Depends on specific values ofz + and z-p(Fn]+Htest) > p(F n]-H test)p(Fn]+Htes t ) > p(Fn[-Htes t )Depends o n specific values of z + and z -Depends on specific values of z + and z -p(Fnl+Htest ) < p(Fn I-Htes t )p(Fn]+Htest) < p(F n]-H test)Depends o n specific values of z + and z -

9 See Equation l f or derivation.

t h e p o p u l a t i o n h a d s o m e d i s ea s e , i m m u n i t y w o u l d b e t h e t ar g e tp r o p e r ty , a n d p ( t ) w o u l d t h e n b e . 2 ( c f. B o u r n e & G u y , 1 9 6 8 ;E i n h o r n & H o g a r t h , 1 9 86 ).

T h u s , u n d e r s o m e v e r y c o m m o n c o n d i t i o n s , t h e p r o b a b i li t yo f r ece iv i ng fa l s if i ca ti on w i t h + H t es t s cou l d be m uch g r ea t e rt h a n w i t h - H t e s t s . I n tu i ti v e ly , th i s m a k e s s e n se . W h e n y o u a r ei nves t i ga t ing a r e l a ti ve l y r a r e p hen om eno n , p ( t ) i s low and t hese t H i s la r ge . F i nd i ng a t i n H ( ob t a i n i n g f a l s i fi ca ti on w i t h - H -t e s ts ) can be l i kened t o t he p r ove r b i a l s ea r ch f o r a need l e i n ah a y s t a c k . I m a g i n e , f o r e x a m p l e , l o o k i n g f o r A I D S v i c t i m sa m o n g p e o p l e b e li e v e d n o t a t r i sk f o r A I D S . O n t h e o t h e r h a n d ,t h e se s a m e c o n d i t i o n s a ls o m e a n t h a t p ( t ) i s h i g h , a n d s e t H i ss m a l l. T h u s , f i n d i n g a t i n H ( w i t h + H t e s t s ) i s l ik e ly t o b e m u c he as ie r. H e r e , y o u w o u l d b e e x a m i n i n g p e o p l e w i t h t h e h y p o t h e -s i zed r i sk f ac t o r s . I f you ha ve a f a i r ly goo d hyp o t hes i s , p( ~[ h) i sappr ec i ab l y l ow er than p ( t ) , bu t you a r e s t il l l ike l y t o f i ndhea l t hy peop l e i n t he hypo t hes i zed r i sk g r oup , and t hese ca se sa r e i n f o r m a t i ve . ( Y ou mi gh t a l so f o ll ow a s tr a t egy based o n ex -a m i n i n g k n o w n v i c t i ms ; w e d i s cus s t h i s k i nd o f t e s t ing l a te r. )

T h e c o n d i ti o n s w e a ss u m e a b o v e ( a m i n o r i t y p h e n o m e n o n ,a n d a h y p o t h e s i s o f a b o u t t h e r i g h t s iz e ) s e e m t o a p p l y t o m a n yna t u r a l l y occ ur r i n g s i t ua ti ons . H ow eve r , t he se a s sump t i onsm a y n o t a l w a y s h o l d . T h e r e m a y b e c a s e s i n w h i c h a m a j o r i t yph eno me no n i s t he t a r ge t (e .g ., becau se i t w as unexpec t ed ) ; t henp ( t ) > . 5. T h e r e m a y a l so b e s i t u a ti o n s i n w h i c h a h y p o t h e s i s ist e s t ed even t hou gh i t i s no t be l i eved t o be t he r i gh t s i ze , so t ha tp ( h ) # p ( t ) . F o r e x a m p l e , y o u m a y n o t b e c o n f i d e n t o f y o u re s t i ma t e f o r e i t he r p ( t ) o r p ( h ) , so you a r e n o t w i l l ing t o r e j ec ta t heo r e t i ca l ly appea l i ng hypo t hes i s on t he bas i s o f t hose e s t i-m a t e s. O r y o u m a y s i m p ly n o t k n o w w h a t t o a d d t o o r s u b t r a ctf r om yo ur hypo t hes i s , so t ha t a s ea r ch f o r f a ls i f ica t i on i s neces -s a r y t o s u g g e s t w h e r e t o m a k e t h e n e c e s s a r y c h a n g e . I n a n y c a s e ,a t e s te r w i t h s o m e s e n s e o f t h e b a s e r a t e o f t h e p h e n o m e n o n c a nm a k e a r e a s o n e d g u e s s a s t o w h i c h k i n d o f e s t is m o r e p o w e r f u l ,i n t he s ense o f be i ng mo r e l i ke ly t o f i nd c r i t i c a l f a l s i fi ca ti on .T h e c o n d i t io n s u n d e r w h i c h + H t e s t s o r - H t e s t s a r e f a v or e d a r es u m m a r i z e d i n T a b le 1 .

T h e r e a r e t w o m a i n c o n c l u s i o n s t o b e d r a w n f r o m t h i s a n a ly -s i s . F i r s t , i t i s i mpor t an t t o d i s t i ngu i sh be t w een t w o pos s i b l esenses o f " s eek i ng d i s co nf i r m a t i on" : ( a ) t e s t ing ca se s your hy -


8/18

2 1 8 J OS H U A K L A Y M A N A N D Y O U N G- W O N H AT abl e 2Condi tions Favoring + Ttests or - Ttests as Mean s ofOb taining Conclusive Falsif icationTarget and hypothesis Com parison of probability of falsificationbase rates (Fn) for +Ttests and -Ttes ts"

p(t) < .5p( t ) > p(h)p(t) = p(h)p(t) < p(h)p(t) > .5p( t ) > p(h)p(t) = p(h)p( t ) < p(h)

p(Fnl+Ttest) > p(F nl-T test)p(Fnl+Ttest) > p(F nl-T test)Depends on specific values of x + and x -Depends on specific alues of x + a n d x -p(Fnl+Ttest) < p(F ni-T test)p(FnI+Ttest) < p(F ni-T test)

9 See Equation 3 for derivation.

po t hes i s p r ed i c t s t o be n on t a r ge t s , an d C o) t e s t i ng ca se s t ha t a r emo s t l i ke ly t o f a ls i fy t he hypo t hes i s . I t i s t he l a t t e r t ha t i s gene r -a l l y p r e sc r i bed a s op t i ma l . Second , t he r e l a t i on be t w een t heset w o a c ti o n s d e p e n d s o n t h e s t r u c t u r e o f h e e n v i r o n m e n t . U n d e rs o m e s e e m i n g l y c o m m o n c o n d i t i o n s , t h e t w o a c t i o n s c a n , i nf ac t, conf l ic t . T he ups ho t i s t ha t , de sp i t e i t s sho r t comi ng s , t he+ t e s t s t r a te g y m a y b e a r e a s o n a b l e w a y t o t e s t a h y p o t h e s i s inm a n y s i t u at io n s . T h i s i s n o t t o s a y t h a t h u m a n h y p o t h e s i s t e s t-e r s a re a c t u a l l y a w a r e o f th e t a s k c o n d i t i o n s t h a t f a v o r o r d i s fa -v o r t h e u s e o f a + t e s t s t ra t eg y . I n d e e d , p e o p l e m a y n o t b e a w a r eof t he se f ac t o r s p rec i s e l y because t he gene r a l heu r i s t i c t hey useo f t en w or ks w e l l.I n f o r m a t i o n i n T a r g e t T e s t s

T h e 2 , 4 , 6 t a s k i n v o lv e s o n l y o n e - h a l f o f t h e p r o p o s e d + t e s ts t ra t egy , t ha t i s, t he t e s t i ng o f case s hypo t hes i zed t o have t het a r ge t p r ope r t y ( + H t es t i ng ) . I n som e t a sks , how ever , t he t e s t e rm a y a l s o h a v e a n o p p o r t u n i t y t o e x a m i n e c a s e s i n w h i c h t h et a rg e t p r o p e r t y i s k n o w n t o b e p r e s e n t ( o r a b s e n t ) a n d t o r e c e i v ef eedback ab ou t w he t he r t he i n s t ance f i t s t he hypo t hes i s . Fo r ex -a m p l e , s u p p o s e t h a t y o u h y p o t h e s i z e t h a t a c e r t a i n c o m b i n a -t i o n o f h o m e e n v i r o n m e n t , g e n e t ic c o n d i ti o n s , a n d p h y s i c a lh e a l t h ~ n m j d s ~ s c h i z o p h r e n i c i n d i v id u a l s f r o m o t h e rs . I tw o u l d b e n a t u r a l t o s e l e c t s o m e o n e d i a g n o s e d a s s c h i z o p h r e n i ca n d c h e c k w h e t h e r t h e h y p o t h e s i z e d c o n d i t i o n s w e r e p r e s e n t.We w i l l c a l l th i s a pos i t i ve t a r ge t te s t ( + T t e s t ) , becau se yo u s e -l ec t an i n s t ance know n t o be i n t he t a r ge t s e t . S i mi l a r l y , youc o u l d e x a m i n e t h e h i s t o ry o f s o m e o n e j u d g e d n o t t o b e s c h i z o-p h r e n i c t o s e e i f t h e h y p o t h e s i z e d c o n d i t i o n s w e r e p r e s e n t. W eca l l t h i s a nega t i ve t a r ge t t e s t ( - T t e s t ) . G ene r a l l y , T t e s t s may bem o r e n a t u r a l i n c a s e s in v o l v i n g d ia g n o s t i c o r e p i d e m i o l o g i c a lq u e s ti o n s , w h e n o n e i s fa c e d w i t h k n o w n e f f e ct s f o r w h i c h t h ec a u s es a n d c o r r e la t e s m u s t b e d e t e r a fi n e d .

T t e s t s b e h a v e i n a m a n n e r q u i t e p a r a l l e l t o t h e H t e s t s d e -s c r i bed above . A + T t es t r e su l t s i n ve r i f ica t i on ( T n H ) i f t hekno w n t a r ge t t u r ns ou t t o f it t he hyp o t hes i zed r u l e ( e .g ., s om e-o n e d i a g n o s e d a s s c h i z o p h r e n i c t u r n s o u t t o h a v e t h e h i s t o ryhypo t hes i zed t o be d is t i nc ti ve t o s ch i zophr en i a ) . A + T t es t r e -su l t s in f a l si f ica t ion i f a kno w n t a r ge t f a i ls t o hav e t he f ea t u r e shypo t hes i zed t o d i s t i ngu i sh t a r ge t s ( T O H ) . T he p r obab i l i t yof fa l s i f ica t ion wi th a +T est , de s igna ted x + , i s p(hl t ) . Th is i s

e q u i v a le n t t o t h e m i s s r a t e o f si g n a l d e te c t i o n t h e o r y ( G r e e n& Sw e t s , 1966). T he f a l s if y i ng i n s t ances r evea l ed by + T t e s t s( m i s s ed t ar g e ts , T n H ) a r e t h e s a m e k i n d r e v e a l e d b y - H t e s t s( f a l s e nega t i ves , H n T ) . N o t e , t hough , t ha t t he mi s s r a t e o f+ T t e s t s i s c a l cu l a t ed d i f f e r en t ly t han t he f a l s e nega t ive r a t e o f- H t e s t s [ x + = p ( h l t) ; z - = p ( t l h )] . B o t h + T t e s t s a n d - H t e s t sa s se s s w he t he r t he cond i t i ons i n R H a r e necessary f o r s ch i zo -p h r e n i a .

W i t h - T t e s t s , v e r if i ca t io n s a r e o f t h e t y p e T n H ( n o n s c h i z o -p h r e n i c s w h o d o n o t h a v e t h e h i s t o r y h y p o t h e s i z e d f o r s c h i z o -phr en i c s ) , and f a l s i fi ca ti ons a r e o f t he t ype T n H ( non sch i zo -phr en i c s w h o do have t ha t h i s t o r y ) . T he p r oba b i l i t y o f f al si fi ca -t i o n w i t h - T t e s t s , d e s i g n a t e d x - , i s p ( h ~ ) . T h i s i s e q u i v a le n tt o t h e f a ls e a l a r m r a t e i n s i g n a l d e t e c t i o n t h e o ry . - T t e s t s a n d+ H t es t s r evea l the s am e k i nds o f f a l s i fy i ng i n s t ances ( f a ls ea l a r m s o r f a l s e posi ti ve s ). T h e r a t e o f f a ls i f ica t i on w i t h - T t e s t si s x - = p ( h~ - ) com par ed t o z + = p( t- ]h ) f o r + H t es t s . B o t h - T -t e s ts a n d + H t e s t s a s s es s w h e t h e r t h e c o n d i t i o n s i n R x a r e s u ~ -cient.

W e c a n c o m p a r e t h e t w o t y p e s o f T t e s ts i n a m a n n e r p a r a ll e lt o t h a t u s e d t o c o m p a r e H t e s ts . T h e v a l u e s x + a n d x - ( t h e m i ssr a t e an d f a l s e a l a r m r a t e , r e spec t ive l y ) can b e r e l a t ed f o l l ow i ngt h e s a m e l o g i c u s e d i n E q u a t i o n 1 :

PO) + (1 p(h)x + = x - 7 ( 6 - 7 ( 6 / " < 3)I f w e a g a i n a s s u m e t h a t p ( t ) < . 5 a n d p ( h ) = p ( t ) , t h e n x + > x - .T h i s m ean s t ha t + T t e s t s a r e m or e l i ke l y t o r e su l t i n f a l s if i cat i ont h a n a r e - T t e s t s . T h e f u ll s e t o f c o n d i t io n s f a v o r i n g o n e t y p eo f T te s t o v e r t h e o t h e r a r e s h o w n i n T a b l e 2 . U n d e r c o m m o nc i r c u m s t a n c e s , i t c a n b e n o r m a t i v e l y a p p r o p r i a t e t o h a v e a s e c -o n d k i n d o f " c o n f i r m a t i o n b i a s , " n a m e l y , a te n d e n c y t o t e s tc a s e s k n o w n t o b e t a rg e t s r a t h e r t h a n t h o s e k n o w n t o b e n o n t a r -gets .

I t i s a l so i n t e r e s t ing t o cons i de r t he r e l a t i ons be t w een T t e s t sa n d H t e s ts . I n s o m e s it u a ti o n s , i t m a y b e m o r e n a t u r a l t o t h i n kab ou t on e o r t he o t he r . I n an ep i dem i o l _ogical s t udy , f o r exam -p l e , c a se s o f t en come p r e so r t ed a s T o r T ( e .g . , d i agnosed v i c -t i m s o f d i s e as e v s . n o r m a l i n d iv i d u a ls ) . I n a n e x p e r i m e n t a ls t udy , on t he o t h e r hand , t he i nves t i ga t o r u sua l l y de t e r mi ne s t hep r e s e n c e o r a b s e n c e o f h y p o t h e s iz e d f a c t o rs a n d t h u s m e m b e r -sh i p i n H o r H ( e .g . , t r ea t men t vs . con t r o l g r oup) . Suppose ,t h o u g h , t h a t y o u a r e i n a s i t u a t i o n w h e r e a l l fo u r t y p e s o f te s ta r e f ea s i b le . T he r e a r e t hen t w o t e s t s t ha t r evea l f a ls i fi ca ti ons o ft he t ype H n T ( f a l se pos i t i ve s o r f a ls e a l a r ms ) , n am e l y + H t es t sand - T t e s t s . T hese f a l s if i ca ti ons i nd i ca t e t ha t t he hypo t hes i zedc o n d i t i o n s a r e n o t suJ~cient f o r t h e t a r g e t p h e n o m e n o n . F o r e x -a m p l e , s u p p o s e a t e a m o f m e t e o r o lo g i s ts w a n t s t o t e s t w h e t h e rc e r t a i n w e a t h e r c o n d i t i o n s a r e s u f f ic i e n t t o p r o d u c e t o r n a d o e s .T h e t e a m c a n l o o k f o r t o r n a d o e s w h e r e t h e h y p o t h e s i z e d c o n d i -t i o n s e x is t ( + H t e s t s ) o r t h e y c a n t e s t f o r t h e c o n d i t i o n s w h e r et o r n a d o e s h a v e n o t o c c u r r e d ( - T t e s t s ) . T h e p r o b a b i l it y o f d i s-cov er in g faLqi ficat ion wi th e ach kin d o f t es t i s as fol lows:

p ( F n l + H t es t ) = z + = p ( t [ h ) = . p (h n t -)p ( h )p ( F n I - T t e s t ) = x - = p (h ~ ) = p ( h A t )p6)


9/18

CONFIRMATION, DISCONFIRMATION, AN D INFORMATION 21 9

z - p 6 )X . p - - ~ . ( 4 )

T h u s , i f w e a ss u m e , as b e f o re , t h a t p ( t ) < . 5 , a n d p ( h ) = p ( t ) ,t hen z + > x - : t he p r obab i l i t y o f fi nd i ng a f a l s if y i ng i n s t ance( h A t ) is hi g h e r w i t h + H t e s t s t h a n w i t h - T t e s t s .

T he r e a r e a l so t w o t e s t s t ha t r evea l f a ls i f ica t ions o f t he t yp en T ( fa ls e nega t ives o r m i sse s ): + T t e s t s an d - H t e s t s . T hesef a l s i f i ca t i ons i nd i ca t e t ha t t he hypo t hes i zed cond i t i ons a r e no tnecessary f o r t h e t a r g et p h e n o m e n o n . T h e m e t e o r o l o g is t s c a nt e st w h e t h e r t h e h y p o t h e s i z e d w e a t h e r c o n d i t i o n s a r e n e c e s s a r yf o r t o r n a d o e s b y l o o k i n g a t c o n d i t i o n s w h e r e t o r n a d o e s a r es i g h te d ( + T t e s t s ) o r b y l o o k i n g f o r t o r n a d o e s w h e r e t h e h y p o t h -e s i zed cond i t i ons a r e lack i ng ( - H t e s t s ) . T h e p r oba b i l i t y o f f a ls i -f i ca t ion w i t h t he se t w o t e s ts can be com par e d , pa r a l l e l t o E qu a -t i on 4 , above :

x = z - . P ( h ) ( 5 )p ( t ) "T h u s , t h e p r o b a b i l i t y o f f in d i n g H n T f a l s i fi c a t i o n s i s h i g h e rw i t h + T t e s t s t h a n w i t h - H t e s t s .

T hese r e l a t i onsh i ps r e i n f o r ce t he i dea t ha t i t may w e l l be ad -v a n t a g e o u s i n m a n y s i t u a ti o n s t o h a v e tw o k i n d s o f " c o n f i r m a -t i o n b i a s " i n c h o o s i n g t es ts : a t e n d e n c y t o e x a m i n e c a s e s h y -p o t h e s i z e d t o b e ta r g e ts ( + H t e s t s ) a n d a t e n d e n c y t o e x a m i n ecases know n t o be t a r ge t s ( + T t e s t s ) . T aken t oge t he~ t hese t w ot endenc i e s com pos e t he gene r a l + t e s t s t ra t egy . U n de r t he usua la s s u m p t i o n s [ p ( t ) < . 5 a n d p ( t) ~ p ( h ) ] , + H t e s t s a r e f a v o re do v e r - H t e s t s , a n d + T t e s t s o v e r - T t e s t s , a s m o r e l i ke l y t o f i n df a l si f ica t ions . M or eove r , i f you w i sh t o t e s t yo ur r u l e ' s su ff i-c i en c y , + H t e s t s a r e b e t t e r t h a n - T t e s t s ; i f y o u w i s h t o t e s t th er u l e ' s n ec e ss it y , + T t e s t s a r e b e t te r t h a n - H t e s t s . T h u s , i t m a ybe advan t ageous f o r t he me t eo r o l og i s t s t o f ocus t he i r f i e l d r e -s e a r c h o n a r e a s w i t h h y p o t h e s i z e d to r n a d o c o n d i t i o n s a n d a r e a so f a c t u a l t o r n a d o s i g h ti n g (w h i c h , i n f a c t , t h e y s e e m t o d o ; s e eL uc as & Whi t t em or e , 1985) . L i ke m an y o t he r cogn i ti ve heur i s -t i cs , how ever , t h i s + t e s t heu r i s t i c m ay p r ove m a l adap t i ve i n pa r -t i t u l a r s i t u a ti o n s, a n d p e o p l e m a y c o n t i n u e t o u s e t h e s t r a te g yi n t hose s i t ua t ions no ne t he l e s s ( ef . H oga r t h , 1981 ; T ve r s ky &K a h n e m a n , 1 9 7 4 ) .H y p o t h e s i s T e s t i n g i n P r o b a b il i st i c E n v i r o n m e n t s

L a b o r a t o r y v e r s io n s o f r u l e d i s c o v e r y u s u a ll y t a k e p l a c e i n ad e t e r m i n is t ic e n v i r o n m e n t : T h e r e i s a c o r r e c t r u le t h a t m a k e sa b s o l u te l y n o e r r o r s , a n d f e e d b a c k a b o u t p r e d i c t io n s i s c o m -p l e t e ly e rr o r - f ree ( see K er n , 1983 , and G or m an , 1986 , f o r i n t e r -e s t i ng excep t i ons ). I n r ea l i nqu i r y , how eve~ one does n o t expe c tt o f i nd a r u l e t ha t p r ed i c t s eve r y s ch i zophr en i c i nd i v i dua l o rp l a n e t a r y s y s t e m w i t h o u t e r ro ~ a n d o n e r e c og n i T e s t h a t t h eab i l i ty t o d e t ec t p sycho l og i ca l d i so r de r s o r ce l e s t ia l phe no me nai s i mper f ec t . W ha t , t hen , i s the no r ma t i ve s t a t us o f the + t e s theur i s t i c i n a p r obab i l i s t i c s e t t ing?Irreducible error I n a p r obab ' fl i st i c env i r onm en t , i t is some-w h a t o f a m i s n o m e r t o c a l l a n y h y p o t h e s i s co r r e c t , b e c a u s e e v e nt he bes t pos s i b l e hypo t hes i s w i l l make som e f a l s e -pos i ti ve andf a l se - nega ti ve p r ed i c t i ons . T hese i r r edu c i b l e e r r o r s m i gh t ac t u -a l ly b e d u e t o i m p e r f e c t f e e d b a c k , b u t f r o m t h e t e s te r 's p o i n t o fv i ew t hey l ook l i ke f a ls e pos i t ive s o r f a ls e nega t ives . A l t e r na -

t i v e l y , t h e w o r l d m a y h a v e a t r u l y r a n d o m c o m p o n e n t , o r t h ep r o b l e m m a y b e s o c o m p l e x t h a t i n p r a c t i c e p e r fe c t p r e d ic t i o nw o u l d b e b e y o n d h u m a n r e a c h . I n a n y c a s e , t h e s e t T c a n b ede f i ned a s t he s e t o f i n s t ances t ha t t he f eedb ack i nd i ca t e s a r et a r get s . A bes t pos s ib l e r u l e, R s , can be pos t u l a t ed t ha t de f i nest he s e t B . B ma t ch es T a s c l ose ly a s pos s i b l e , bu t n o t exac t ly .B ecause o f p r obab i l i s t i c e r r o r , even t he bes t r u l e m akes f al se -pos i t ive and f a l se - nega t ive p r ed i c t i on e r r o r s ( i. e. , p ( t lb ) > 0 andp( t [ b ) > 0 ) . T he p r obab i l i t i e s o f t he se e r r o r s , de s i gna t ed r and~ -, re p r e s e n t t h e o r e t ic a l o r p r a c t ic a l m i n i m u m e r r o r r a t e s )

Q ua l i t a t i ve l y , t he mos t i mpor t an t d i f f e r ence be t w een de t e r -mi n i s t i c and p r obab i l i s t i c env i r onmen t s i s t ha t bo t h ve r i f i ca -t i on an d f a l s i f ica t ion a r e o f fi n i te va l ue a nd sub j ec t t o som e de -g r ee o f p r obab i l i s t i c e r ro r . T hu s , f a l s if i ca ti ons a r e no t conc l u -s ive bu t mer e l y cons t i t u t e s ome ev i den ce aga i ns t the hypo t hes i s ,and ve r i f i ca t i ons mus t a l so be cons i de r ed i n f o r ma t i ve , de sp i t et he i r l og ica l ambi gu i t y . U l t i ma t e l y , i t c an neve r be kn ow n w i t hce r t a i n t y t h a t a ny g i ven hypo t hes i s i s o r i s no t t he bes t pos s i b l e .O n e c a n o n l y f o r m a b e l i e f a b o u t t h e p r o b a b i l i t y t h a t a g i v e nhypo t hes i s i s co r r ec t , i n l i gh t o f t he co l l ec t ed ev i dence .

D e s p i t e t h e s e n e w c o n s i d e r a t io n s , i t c a n b e s h o w n t h a t t h ebas i c f i nd i ngs o f ou r ea r l ie r ana l yses s t il l app l y . A l t houg h t her e l a t i onsh i p i s mo r e com pl i ca t ed , t he r e l a t ive va l ue o f + t e s t sand - t e s t s i s s ti ll a f unc t i on o f e s t i mab l e t a sk cha r ac t e r i s t i c s. I ngene r a l , i t i s s t i l l t he ca se t ha t + t e s t s a r e f avor ed w hen p ( t ) i ssma l l and p( h ) ~ p ( t ) , a s sugges t ed ea rl ie r. A l t houg h w e d i scus son l y H t e s t s he r e , a pa r a l l e l ana l ys i s can b e pe r f o r me d f o r T t e s t sas w ell.Revision of beliefs. A s s u m e t h a t y o u r g o a l i s t o o b t a i n t h em o s t e v i d en c e y o u c a n a b o u t w h e t he r o r n o t y o u r c u r r e n t h y -po t hes i s i s the bes t pos s i b le . W hi ch t ype o f t e s t w il l, o n ave r age ,b e m o r e i n f o r m a t i v e ? T h i s k i n d o f p r o b l e m c a ll s fo r a n a n a l y si so f t he expec t ed va l ue o f i n f o r m a t i on ( e.g. , s ee E dw ar ds , 1965 ;R a i f f a , 1968). S uch ana l yses a r e based o n B ay es ' s equa t i on ,w h i ch p r ov i des a no r m a t i ve s t a t i st i c a l me t h od f o r a s se ss i ng t heex t en t t o w h i ch a sub j ec t i ve degr ee o f be l i e f shou l d b e r ev i s edi n l i gh t o f new da t a . T o pe r f o r m a f u l l -f l edged B ayes i an ana l ys iso f v a l u e o f i n f o r m a t i o n , i t w o u l d b e n e c e s s a r y t o r e p r e se n t t h ec o m p l e t e r e w a r d s t ru c t u r e o f t h e p a r t i c u l a r ta s k a n d c o m p u t et he t e s t e r ' s sub j ect i ve expec t ed u t i l it y o f each pos s i b l e ac t i on .S u c h a n a n a l y s i s w o u l d b e v e r y c o m p l e x o r w o u l d r e q u i r e ag r ea t m an y s i mpl i f y i ng a s sumpt i ons . I t is pos s i b le , t hough , t ou s e a s im p l e , g e n e ra l m e a s u r e o f " i m p a c t , " s u c h a s t h e e x p e c t e dchang e i n be l i e f (E A P) .

S u p p o s e y o u t h i n k t h a t t h e r e i s s o m e c h a n c e y o u r h y p o t h e s isi s t h e b e s t p o s s ib l e , p ( R H = R s ) . T h e n , y o u p e r f o r m a + H t e s t ,and r ece i ve a ve r i f ica t i on (V n) . Y ou w ou l d n ow have a som ew h a th i g h e r es t i m a t e o f th e c h a n c e t h a t y o u r h y p o t h e s i s is t h e b e s tone p ( R H = R s [ V n , + H ) . C a l l the i m pac t o f t h i s t e s t A Pv~ ,+ n ,t h e a b s o l u t e m a g n i t u d e o f c h a n g e i n d e g r e e o f be li ef . O f c o u rs e ,you mi gh t have r ece i ved a f a l s i f i ca t i on ( Fn) i n s t ead , i n w h i chc a s e y o u r b e l i e f t h a t R H = RB w o u l d b e r e d u c e d b y s o m eam oun t , A PF~.+H . T he expec t ed chang e i n be l i e f f o r a + H t e s t ,

3 Fo r simplicity, we ignore the p ossibility that a rule might p roduce,say, fewer false positives bu t m ore false negatives than the be st rule. Weassume that the m inimu m ~+ and ~- can b oth be achieved at the sametime. Th e m ore general case could be analyzed by defining a joint func-t ion o f~ + and ~- which is to be minimized.


10/18

2 2 0 J O S H U A K L A Y M A N A N D Y O U N G - W O N H Ag i v en t h a t y o u d o n o t k n o w i n a d v a n c e w h e t h e r y o u w i ll r ec e i v ea ve r i f i ca t ion o r a f a l s if i cat i on , w ou l d t hus heE A P+ H = p ( F n l + H t e s t ) . A P v , . + n + p ( V n [ + H t e s t ) . A P v , . + m ( 6 )I n t h e a p p e n d i x , w e s h o w t h a t

rA PFn ,+ H = 1 2 "+ ,

1 - - ~ +APvn.+H = 1 - z +

a n d

m - - l ,

p( F n l + H t es t ) . A Pr n,+H ffi ( z + - ~ + ) 'P ( R n = R a ) ,

p(V nl+ Ht es t ) . APv, .+H = (z + - E+)-P(RH = RB) .

(7 )(8 )(9 )

(10)(11)T h u s E A P+ H = 2 ( z + - ,+ ) . p ( R H = R e) .Simi lar ly ,

E A P - H = 2 ( z - - C ) . p ( R H = R e ) . ( 1 2 )T h i s p r ob ab i l i s t ic ana l ys i s l ooks d i f f e ren t f r om i t s de t e r m i n -

i s t i c coun t e r pa r t i n one r e spec t . B e f o r e , t he emphas i s w ass t r i c t l y on f a l s i f i ca t i on . H e r e , ve r i f i ca t i on can some t i mes bemo r e i n f o r ma t i ve t han f a l si f ica t ion . U s i ng + H t e s t s t o i l l u s t ra t e ,E qua t i ons 7 and 8 i mpl y t ha t i f z + > . 5 , t hen A Pv . ,+ n > A Pr . ,+ H .A hy po t hes i s w i t h z + > . 5 i s a w e ak hy po t hes i s ; you be l i eve t hema j o r i t y o f p r ed i c t ed t a r ge t s w i ll p r ove w r ong . P e r hap s t h i s i san o l d hypo t hes i s t ha t i s now o u t o f favor , o r a new sho t - i n - t he -d a r k g u e s s. T h e A P m e a s u r e c a p t u r e s t h e i n t u i t io n t h a t s u r p r i sev e r if i ca t io n o f a l o n g s h o t h y p o t h e s is h a s m o r e i m p a c t t h a n t h eant ic ipated fa l s i f ica t ion.

I n c o n s i d e r in g t h e e x p e c t e d i m p a c t o f a t e st , y o u m u s t b a l -a n c e t h e g r e a t e r i m p a c t o f u n e x p e c t e d r e s u lt s a g a i n s t th e f a c tt h a t y o u d o n o t t h i n k s u c h r e s u lt s a r e li k el y t o h a p p e n . W i t ht he E A P mea sur e , t he n e t r e su l t i s t ha t ve r i f i ca t i ons and f a l si fi -c a t i o n s a r e e x p e c t e d t o m a k e e q u a l c o n t r i b u t i o n s t o c h a n g e s i nbel ief , overa l l (as shown in Equat ions 9 and 10) . Ver i f ica t ionsand f a l s i f i ca t i ons have equa l expec t ed i mpac t even i n a de t e r -m i n i s t ic e n v i r o n m e n t , a c c o r d i n g t o t h i s d e f in i t io n o f i m p a c t .T h e d e t e r m i n i s t i c e n v i r o n m e n t i s m e r e l y a s p e c i a l c a s e i nwh ich ~+ = C = 0.

G i ven t h i s p r obab i l i s ti c v i ew o f t he va l ue o f ve r i f i ca ti on andf a ls i fi c at io n , w h e r e s h o u l d o n e l o o k f o r i n f o r m a t i o n ? T h e a n -s w e r t o t h i s q u e st i o n , b a s e d o n t h e c o m p a r i s o n b e t w e e n + H t e s t sand - H t es t s , chang es ve r y li t tl e f r om t he de t e r mi n i s t i c ea se . Itw o u l d b e a r a t i o n a l p o l i c y f o r a t e s te r t o c h o o s e t h e t y p e o f H t e s ta s soc i a t ed w i t h t he g r ea t e s t expec t ed change i n be l i e f . I n t ha tc a s e, a c c o r d i n g t o E q u a t i o n s 1 1 a n d 1 2, y o u w a n t t o c h o o s e t h et e s t f o r w h i ch z - ~ i s g r ea t e s t : + H t e s t s i f ( z + - ~+) > (z - -C ) . I n o t h e r w o r d s , c h o o s e t h e t e s t f o r w h i c h y o u b e li e v e th ep r oba b i l i t y o f fa l s if i ca t i on ( z ) is mo s t abo ve t he l eve l o f i r r educ -i b le e r r o r ( 0 . T h i s p r e s c r i p t io n i s o b v i o u s l y v e r y s i m i l a r t o t h econd i t i ons spec i f ied f o r t he de t e r mi n i s t i c env i r onm en t . I ndee d ,i f t he t w o ~ s a r e eq ua l ( even i f no nze r o ) t he r u l e i s i den t i ca l :C ho ose t he t e st w i t h t he h i ghe r z . T hu s , t he p r e sc r i p t i ons sh ow ni n T ab l e 1 ho l d i n a p r obab i l i s t i c env i r o nm en t , a s l ong a s i r r e -d u c i b l e e r r o r i s a l s o t a k e n i n t o a c c o u n t . I n t h e A p p e n d i x w ea l so p r e sen t an a l t e r na t i ve measu r e o f i n f o r ma t i venes s ( a mea -s u r e o f " d i a g n o s t i c i t y " o f t e n u s e d i n B a y e s ia n a n a ly s e s) ; th e b a -s ic p r e m i s e s o f o u r c o m p a r i s o n r e m a i n i n t a c t . Q u a l i t a ti v e l y

s i mi l a r r e su l t s ob t a i n even w he n us i ng a non- B ay es i an ana l ys i s ,ba sed on s t a t i st i c al i n f o r m a t i on t heo r y (s ee K l ay ma n , 1986) .Information in H ypothesis Testing: Conclusions

T h e f o u n d a t i o n o f o u r a n a l y s i s is t h e s e p a r a t io n o f d i s co n -f i r ma t i on a s a goa l f r om d i s con f i r ma t i on a s a s ea r ch s t ra t egy . I ti s a w i de l y accep t ed p r e sc r i p t i on t ha t an i nves t i ga t o r shou l dseek f a l s if i ca ti on o f hypo t heses . O ur an a l yses show , t hough , t ha tt h e r e i s n o c o r r e s p o n d i n g l y s i m p l e p r e s c r i p t io n f o r t h e s e a r c hs t r a t egy bes t su it ed t o t ha t goa l . T he op t i m a l s t r a t egy is a f unc -t i o n o f a v a r i e t y o f t a sk v a r ia b l e s s u c h a s t h e b a s e r a t e s o f t h et a r g e t p h e n o m e n o n a n d t h e h y p o t h e s i z e d c o n d i t i o n s . I n d e e d ,even a t t em pt i ng f a l s if i ca ti on is no t neces sa r i l y t he pa t h t o max i -m u m i n f o r m a t i o n ( s e e a l s o K l a y m a n , 1 9 86 ).

W e d o n o t a s s u m e t h a t p e o p l e a r e a w a r e o f t h e t a sk v a r i ab l e st ha t de t e r m i ne t he bes t t e s t s t ra t eg ie s . R a t he r , w e suggest t ha tpeop l e use a gene r a l , a l l - pu r pose heur i s t i c , t he pos i t i ve t e s ts t ra t egy , w h i ch i s app l i ed ac r os s a b r o ad r ange o f hypo t hes i s -t e s t i ng t a sks . L i ke any a l l - pu r pose heur i s t i c , t h i s + t e s t s t r a t egyi s no t a l w ays op t i ma l and can l ead t o s e r i ous d i f f icu l t ie s i n ce r -t a i n s i t ua t i ons ( a s i n W ason ' s 2 , 4 , 6 t a sk ) . H ow eve r , ou r ana l ysess h o w t h a t + t e s t i n g is n o t a b a d a p p r o a c h i n g e n er a l. U n d e r c o m -m o n l y o c c u r r i n g c o n d i t io n s , t h e + t e s t s t r a t e g y le a d s p e o p l e t op e r f o r m t e s ts o f b o t h s u f fi c ie n c y a n d n e c e s s it y ( + H t e s t s a n d+ T t es t s ) , u s i ng t he t ypes o f t e s t s mos t l ike l y t o d i s cove r v i o -la t ions o f e i ther.

B e y o n d R u l e D i s c o v e r y : T h e P o s it iv e T e stS t r a te g y i n O t h e r C o n t e x t s

T h e m a i n p o i n t o f o u r a n a l y si s i s n o t t h a t p e o p l e a r e b e t t e rh y p o t h e s i s t e st e rs t h a n p r e v i o u s l y t h o u g h t ( a l t h o u g h t h a t m a ybe so ) . R a t he r ; the + t e s t s t r a t egy ca n p r ov i de a bas i s f o r unde r -s t a n d i n g t h e s u c c e s s es a n d f a il u r e s o f h u m a n h y p o t h e s i s t e s ti n gi n a va r i e t y o f s i t ua ti ons . I n t h i s s ec ti on , w e app l y ou r app r oac ht o s eve r a l d i f f e r en t hypo t hes i s - t e s t ing s i t ua t ions . E ac h o f t het a sks w e d i s cus s has an ex t ens i ve re sea r ch l i t e r a t u r e o f i t s ow n .H ow eve r , t he r e h as bee n l i t tl e c r os s - t a sk gene r a l i ty bey on d t heu s e o f t h e c o m m o n " c o n f i r m a t i o n b i a s " l a be l. W e s h o w h o wt hese d i ve r se t a sks can he g i ven an i n t eg r a t i ve i n t e r p r e t a t i onb a s e d o n t h e g e n e r a l + t e s t s t r a t e g y . E a c h t a s k h a s i t s u n i q u er equ i r emen t s , and i dea l l y , peop l e shou l d adap t t he i r s t r a t eg i e st o t he cha r ac t e r i s t i c s o f t he spec i f ic t a sk a t h and . Peop l e mayi n d e e d r e s p o n d a p p r o p r i a t e l y t o s o m e o f th e s e c h a r a c t e ri s ti c sund e r f avor ab l e cond i t i on s ( w hen t he r e i s conc r e t e t a sk - spec if i ci n f o r m a t i o n , f i g h t m e m o r y l o a d , a d e q u a t e t i m e , e x t e n si v e ex p e -r i ence , e t c . ) . We p r opose t ha t , unde r l e s s f r i end l y cond i t i ons ,hypo t hes i s t e s t e r s r e l y on a gene r a l l y app l i cab l e de f au l t ap -p r o a c h b a s e d o n t h e + t e s t s t ra t eg y .Concep t Identification

A t t h e b e g i n n i n g o f h i s p a p e r, w e d e s c r ib e d t h e c o n c e p t - i d e n -t if i c at i o n t a s k ( B r u n e r e t a l. , 1 9 5 6 ) a s a f o r e r u n n e r o f W a s o n ' sr u l e - d i s cove r y t a sk ( Wason , 1960) . I n bo t h t a sks , t he sub j ec t ' sg o a l is t o i d e n ti f y t h e r u l e o r c o n c e p t t h a t d e t e r m i n e s w h i c h o fa subse t o f s t imu l i a r e des i g na t ed a s co r r ec t . I n co nce p t i den t i -f i ca ti on , how eve r ; t he s e t o f pos s i b l e i n s t ances a nd pos s i b l e r u l e s


11/18

C O N F I R M A T I O N , D I S C O N F I R M A T I O N , A N D I N F O R M A T I O N 2 2 1i s h i g h l y r e s tr i c te d . F o r e x a m p l e , t h e s t i m u l i m a y c o n s i s t o f a l lc o m b i n a t i o n s o f f o u r b i n a r y c u e s ( l e t t e r X o r T , l a r g e o r s m a l l ,b l a c k o r w h i t e , o n t h e r i g h t o r l e ft ), w i t h i n s t r u c t i o n s t o c o n s i d e ro n l y s i m p l e ( o n e - f e a t u r e ) r u l e s ( e .g . , L e v i n e , 1 9 6 6 ). T h e h y -p o t h e s i s s e t , t h e n , i s r e s t r i c t e d t o o n l y e i g h t p o s s i b i l i t i e s . E v e nw h e n c o n j u n c t i o n s o r d i s j u n c t i o n s o f f e a t u r e s a r e a l lo w e d ( e .g . ,B o u r n e , 1 97 4 ; B r u n e r e t al . , 1 9 5 6 ), t h e h y p o t h e s i s s e t r e m a i n sc i r c u m s c r i b e d .

A n u m b e r o f s t u d ie s o f c o n c e p t i d e n t if ic a t io n h a v e d o c u -m e n t e d a b a s i c w i n - . q a y , l o s e - s h i f t s t r a t e g y ( e . g . , s e e L e v i n e ,1 9 6 6 , 1 9 7 0 ; T r a b a s s o & B o w e r ; 1 9 6 8 ) . T h a t i s , t h e l e a r n e r f o r m sa n i n i t i a l h y p o t h e s i s a b o u t w h i c h s t i m u l i a r e r e i n f o r c e d ( e . g . ," X s o n t h e l e f t " ) a n d r e s p o n d s i n a c c o r d a n c e w i t h t h a t h y p o t h e -s i s a s lo n g a s c o r r e c t c h o i c e s a r e p r o d u c e d . I f a n i n c o r r e c tc h o i c e o c c u r s , t h e l e a r n e r s h i f ts t o a n e w h y p o t h e s i s a n d r e -s p o n d s i n a c c o r d a n c e w i t h t h a t , a n d s o o n . I n o u r t e r m s , t h i s i s+ H t e s t i n g . I t is w h a t w e w o u l d e x p e c t t o s e e , e s p e c i a l l y s i n c et o t a l s u c c e s s r e q u i r e s a r u l e t h a t i s s u f f ic i e n t f o r re w a r d , o n l y .I n t h e c o n c e p t - i d e n t if i c a t io n t a s k + H t e s t i n g a l o n e c o u l d l e a d t oa s u c c e s s f u l s o l u t io n . H o w e v e r , b e c a u s e t h e r e a r e o n l y a f i n i ten u m b e r o f i n s ta n c e s ( c u e c o m b i n a t i o n s ) , a n d a f i ni te n u m b e ro f h y p o t h e s e s , + t e s t i n g i s n o t t h e m o s t e f f e c ti v e s t ra t e g y . A m o r ee f f ic i e n t s t r a t e g y i s t

Date post:	07-Apr-2018
Category:	Documents
Upload:	sophia-robinson-garriques
View:	217 times
Download:	0 times

Ha Confirmation, Dis Confirmation, And Information in Hypothesis Testing

Documents