+ All Categories
Home > Documents > Shaw Icse2003

Shaw Icse2003

Date post: 03-Apr-2018
Category:
Upload: vigneshmahamuni
View: 233 times
Download: 0 times
Share this document with a friend

of 11

Transcript
  • 7/28/2019 Shaw Icse2003

    1/11

    W r i ti n g G o o d S o f tw a r e E n g i n e e r in g R e s e a r ch P a p e r sM i n i t u t o r i a lMary Shaw

    Carnegie Mellon Universitymary . shaw@cs .cmu .edu

    A b s t r a c tSo f tware eng ineer ing researchers so l ve p rob l ems o fsevera l d i fferent kinds. To do so , they prod uce seve raldi f ferent kinds o f result s, and they sho uld developappropriate evidence to validate these results. They oftenreport their research in conference papers. I analyze d theabstracts o f research pap ers sub mi t ted to ICSE 2002 inorder to ide nt i~ the types o f research rep orted in thesubmi t t ed and accep t ed papers , and I observe d theprogra m commi t tee d i scuss ions abou t wh i ch papers t oaccept . This report prese nts the research parad igms oft he papers , common concerns o f t he p rogram commit tee ,an d statistics on succes s rates. T his information sho uldhelp researchers design be t ter research projects a nd wri tepap ers that prese nt their resul t s to be st advantage.Keywords : re sea rch des ign , re sea rch pa rad igm s ,va l ida t ion , so f tware p rofes s ion , t echn ica l wr i t ingI . I n t r o d u c t i o n

    In sof tware eng inee r ing , re sea rch pape rs a re cus tom aryveh ic le s fo r repor t ing re su l t s to the re sea rch com m uni ty .In a re sea rch pape r , the au thor exp la ins to an in te re s tedreade r wha t he o r she accom pl i shed , and how the au thoraccom pl i shed i t , and w hy the reade r shou ld ca re . A go odresea rch pape r should answer a nu m ber o f ques t ions :

    Wh a t , p rec i s e ly , was your con t r ibu t ion? W ha t ques t ion d id you answer? W hy should the reade r ca re? W ha t l a rge r ques t ion does th i s addres s?

    W ha t is your new resu l t ? W h a t n e w k n o w l e d g e h a v e y o u c o n t ri b u te d t h a t

    the reade r can use e l s ewhere? W ha t p rev ious work (yours o r som eone e l s e ' s )

    d o y o u b u i l d o n ? W h a t d o y o u p r o v i d e a s u p e r io ral ternat ive to? Ho w is you r re su l t d i f fe ren t f rom and be t t e r than

    th i s p r io r work? Wha t , p rec i s e ly and in de ta i l, i s you r new resu l t ?

    W hy should the reade r be l i eve you r re su l t? Wh a t s t anda rd should be used to eva lua te you r

    c la im ?

    W ha t conc re te ev idence shows tha t you r re su l ts a t i sf i e s your c la im ?

    If you answer these ques t ions c lea r ly , you ' l l p robab lycom m unica te your re su l t we l l . I f in add i t ion your re su l trepresen t s an in te re s t ing , sound , and s ign i f i can t con tr ibu-t i o n t o o u r k n o w l e d g e o f s o f t w a r e e n g i n e e r i n g , y o u ' l lhave a g ood ch ance o f ge t t ing it accep ted fo r pub l ica t ionin a con fe rence o r journa l .

    Othe r f i e lds o f s c ience and eng inee r ing have we l l -e s tab l i shed re sea rch pa rad igm s . F or exam ple , thee x p e r im e n t a l m o d e l o f p h y s ic s a n d t h e d o u b l e - b l i n ds tud ies o f m edic ines a re unders tood , a t l ea s t in b roado u t l in e , n o t o n l y b y t h e r e s e a r c h c o m m u n i t y b u t a l s o b ythe pub l ic a t l a rge . In add i t ion to p rov id ing gu idance fo rthe des ign of re sea rch in a d i s c ip l ine , these pa rad igm ses tab l i sh the s cope of s c ien t i f i c d i s c ip l ines th rough asoc ia l and po l i t i ca l p roces s o f "boun dary s e t t ing" [5 ].

    S of tware eng inee r ing , however , has no t ye t deve lopedth i s so r t o f we l l -unders tood gu idance . I p rev ious ly [19 ,20] d i s cus sed ea r ly s t eps toward such unders tand ing ,i n c l u d in g a m o d e l o f th e w a y s o f t w a r e e n g i n e e ri n gtechn iques m a ture [17, 18] and c r i tiques o f the l ack o fr igor in expe r im enta l so f tware eng inee r ing [1 , 22 , 23 , 24 ,25] . Those d i s cus s ions c r i t ique sof tware eng inee r ingresea rch repor t s aga ins t the s t anda rds o f c la s s ica lpa rad igm s . The d i s cus s ion he re d i f fe rs f rom those in tha tth i s d i s cus s ion repor t s on the types o f pape rs tha t a reaccep ted in p rac t i ces a s good re sea rch repor t s . Anothe rcur ren t ac t iv i ty , the Im pac t P ro jec t [7 ] s eeks to t race thein f luence o f so f tware eng inee r ing re sea rch on prac t i ce .The d i s cus s ion he re focuses on the pa rad igm s ra the r thanthe con ten t o f the re sea rch

    This repor t exam ines how sof tware eng inee rs answerthe ques t ions above , w i th em phas i s on the des ign of theresea rch pro jec t and the o rgan iza t ion o f the repor t. O the rsources (e . g ., [4 ]) dea l wi th spec i f i c i s sues o f t echn ica lwr i t ing . Very conc re te ly , the exam ples he re com e f romthe pape rs subm i t t ed to ICS E 2002 and the p rogramcom m it tee rev iew o f those pape rs . These exam ples repor tre sea rch re su l t s in sof tware eng inee r ing . Confe rencesof ten inc lude o the r k ind s o f pape rs , inc lud ing expe r iencerepor t s , m a te r ia l s on sof tware eng inee r ing educa t ion , andopin ion e s says .

    0-7695-1877-X /03 $17.00 2003 IEEE 72 6

  • 7/28/2019 Shaw Icse2003

    2/11

    2 . W h a t , p r e c i se l y , w a s y o u r c o n t r i b u t i o n ?Before repor t ing wha t you d id , exp la in wha t p rob lem

    you se t ou t to so lve o r w ha t ques t ion you se t ou t to answer- - a n d w h y t h i s is i m p o rt a n t.2 .1 W h a t k i n d s o f q u e s t i o n s d o s o f tw a r ee n g i n e e r s i n v e s t i g a te ?Genera l ly speak ing , so f tware eng inee r ing re sea rche rsseek be t t e r ways to deve lop and eva lua te so t tware . Deve l -opment includes a l l the synthet ic act ivi t ies that involvec rea t ing and m odi fy ing the sof tware , inc lud ing the code ,des ign docum ents , docum enta t ion , e tc . Eva lua t ion

    inc ludes a l l the ana ly t i c ac t iv i ti e s a s soc ia ted w i th p red ic t -ing , de te rm in ing , and e s t im a t ing prope r t i e s o f the sof tw aresys tem s , inc lud ing bo th fu nc t iona l i ty and ex t ra - func t iona lprope r t ie s such a s pe r form ance or re l i ab i l i ty .

    S of tware eng inee r ing re sea rch answers q ues t ions aboutm e thods o f deve lopm ent o r ana lys i s , about de ta i l s o fdes ign ing or eva lua t ing a pa r t i cu la r ins tance , about gene r -a l i za t ions ove r w hole c la s ses o f sys tem s or t echn iques , o rabout exp lora to ry i s sues conce rn ing ex i s tence o r feas ib il -i ty . Tab le 1 l is t s the types o f re sea rch q ues t ions tha t a reasked by sof tware eng inee r ing re sea rch pape rs andprov ides spec i f ic ques t ion t em pla te s .

    Table 1. Types of software eng ineerin g research q uestionsType of question ExamplesM e t h o d o r m e a n s o f H o w c a n w e d o / c r e at e / m o d if y / e v o lv e ( o r a u t o m a t e d o i n g ) X ?

    deve lopm ent W ha t i s a be t t e r way to do /c rea te /m odi fy /evo lve X?Meth od fo r ana lys i s Ho w can I eva lua te the qua l i ty /cor rec tnes s o f X?

    o r e v a l u at i o n H o w d o I c h o o s e b e t w e e n X a n d Y ?Des ign , eva lua t ion , o r Ho w good i s Y? Wh a t i s p rope r ty X of a r t i fac t /m e thod Y?ana lys i s o f a W ha t i s a (be t te r ) des ign , im plem enta t ion , m a in tenance , o r adap ta t ion fo r app l ica t ion X?par t i cu la r ins tance How does X com pare to Y?

    W hat i s the cur ren t s ta te o f X / p rac t i ce o f Y?Genera l i za t ion o r Given X, wha t wi l l Y (neces sa ri ly ) be?cha rac te r iza t ion Wh a t , exac t ly , do we m ean by X? W ha t a re i t s im por tan t cha rac te r i s ti c s?

    W hat i s a good fo rm a l /em pi r ica l m ode l fo r X?Wh at a re the va r ie t i e s o f X, how a re they re la ted?F eas ib i l i ty s tudy or Does X even ex i s t, and i f so wha t i s i t l ike?explora t ion I s i t pos s ib le to accom pl i sh X a t a l l ?

    The f i r s t two types o f re sea rch produce m e thod s o fdeve lopm ent o r o f ana lys i s tha t the au thors inves t iga ted inone s e t t ing , bu t tha t can p resum ably be app l ied in o the rse t tings . The th i rd type o f re sea rch dea l s exp l ic i t ly wi thsom e pa r t i cu la r sys tem , p rac t ice , des ign or o the r ins tanceof a sys tem or m e thod; these m ay range f rom na r ra t ivesabout indus t r i a l p rac t i ce to ana ly t i c com par i sons o fa l t e rna t ive des igns . F o r th i s type o f re sea rch the ins tancei t s e lf s h o u l d h a v e s o m e b r o a d a p p e a l - - a n e v a l u a ti o n o fJava i s m ore l ike ly to be accep ted than a s im ple eva lua t ionof the toy l anguage you deve loped l a s t sum m er .Genera l i za t ions o r cha rac te ri za t ions exp l ic i tly r i s e abo vethe exam ples p resen ted in the pape r . F ina l ly , pape rs tha tdea l wi th an i s sue in a com ple te ly new w ay a re som et im est rea ted d i f fe ren t ly f rom pape rs tha t im prove on p r io r a r t ,so " feas ib i l i ty" i s a s epa ra te ca tego ry ( though no suchpapers were subm i t t ed to ICS E 2002) .

    Newm an ' s c r i t i ca l com par i son of HCI and t rad i t iona leng inee r ing pape rs [12] found tha t the eng inee r ing pape rswere m os t ly inc rem enta l ( im proved m ode l , im provedtechnique ) , whereas m an y of the HCI pape rs b roke ne wground (obse rva t ions p re l im ina ry to a m ode l , b rand new

    technique ) . One reasonab le in te rpre ta t ion i s tha t thet rad i tiona l eng inee r ing d i sc ip l ines a re m uc h m o re m a turethan HCI , and so the cha rac te r o f the re sea rch m igh treasonab ly d i f fe r [17 , 18] . Also , i t appea rs tha t d i f fe ren td i s c ip l ines have d i f fe ren t expec ta t ions about the " s ize" o fa re sea rch re su l t - - th e ex ten t to which i t bu i lds on ex i s t ingk n o w l e d g e o r o p e n s n e w q u e s t io n s . I n t h e e a s e o f I C S E ,the k inds o f ques tions tha t a re o f in te re s t and the m in im uminte res t ing inc rem ent m ay d i f fe r f rom on e a rea to ano the r .2 .2 W h i c h o f t h es e ar e m o s t c o m m o n ?

    T h e m o s t c o m m o n k i n d o f I C S E p a p e r r e p o r ts a ni m p r o v e d m e t h o d o r m e a n s o f d e v e lo p i n g so t t w a r e t h a ti s , o f des ign ing , im plem ent ing , evo lv ing , m a in ta in ing , o ro the rwise ope ra t ing on the sof tware sy s tem i t se l f . P ape rsaddres s ing these ques t ions dom ina te bo th the subm i t t edand the accep ted pape rs . Also fa i r ly com m o n a re pape rsabout m e thods fo r reason ing about so f tware sys tem s ,pr inc ipa l ly ana lys i s o f cor rec tnes s ( t e s t ing andver i f i ca tion) . Ana lys i s pape rs have a m od es t accep tanceedge in th i s ve ry s e lec tive confe rence .

    727

  • 7/28/2019 Shaw Icse2003

    3/11

    Table 2 g ives the d i s t r ibu t ion o f subm is s ions to I C S E2002 , based on read ing the abs t rac t s (no t the fu l l pape rs -- -bu t rem em ber tha t the abs t rac t t e l l s a reade r wha t to ex-pec t f rom the pape r ) . F or each type o f re sea rch ques t ion ,

    the t ab le g ives the num be r o f pape rs subm i t t ed and ac -cep ted , the pe rcen tage o f the to ta l pape r s e t o f each k ind ,and the accep tance ra t io wi th in each type o f ques t ion .F igures 1 and 2 sho w these counts and d i s t r ibu t ions .

    Table 2 . Types of research quest ions represented in ICSE 2002 subm issions and acceptancesType o f ques ti on Submi tted Accepted Rat io Acc/SubM e t h o d o r m e a n s o f d e v e l o p m e n t 1 4 2 (4 8 % ) 1 8 ( 4 2 % ) ( 1 3 % )Me thod fo r ana lys i s o r eva lua t ion 95 (32%) 19 (44%) (20%)Des ign , eva lua t ion , o r ana lys i s o f a pa r t i cu la r ins tance 43 (14%) 5 (12%) (12%)Genera l i za t ion o r cha rac te r i za t ion 18 (6%) 1 (2%) (6%)F eas ib i l i ty s tudy or exp lora t ion 0 (0%) 0 (0 %) (0%)

    T O T A L 2 9 8 ( 1 0 0 .0 % ) 4 3 ( 1 0 0 .0 % ) ( 1 4 % )

    Q u e s t i o n3 0 0 - - ~2 5 0200 ~:~t1 5 0

    o o ~%~;~~ o ~

    oD e v el A n a ly E v a l G e n er F e a s T o t a l

    Accepted [ ] Rejected IIFigure 1 . Counts of acceptances and reject ionsby type o f research quest ion

    2 .3 W h a t d o p r o g r a m c o m m i t te e s l o o k f or ?Act ing on beh a l f o f p rospec t ive reade rs , the p rogramcom m it tee looks fo r a c lea r s t a tem ent o f the spec i f i cp r o b l e m y o u s o l v e d - - t h e q u e s t io n a b o u t s o f t w a r e d e v el -o p m e n t y o u a n s w e r e d - - a n d a n e x p l a n a ti o n o f h o w t h eanswer wi l l he lp so lve an im por tan t so t tware eng inee r ingprob lem . You ' l l devo te m os t o f your pape r to desc r ib ingyour re su l t , bu t you should beg in by exp la in ing wha tques t ion you ' re answer ing and w hy the answ er m a t te rs .

    I f the p rogram com m i t tee has t roub le f igur ing ou tw h e t h e r y o u d e v e l o p e d a n e w e v a l u a t i o n t e c h n i q u e a n ddem on s t ra ted i t on an exam ple , o r app l ied a t echn ique yourepor ted l a s t yea r to a new rea l -wor ld exam ple , o reva lua ted the use o f a we l l -e s tab l i shed eva lua t iontechn ique , you have n o t been c lea r .3 . W h a t i s y o u r n e w r e s u lt ?

    Expla in p rec i s e ly wha t you have con t r ibu ted to thes to re o f sot~ware eng inee r ing knowled ge and how th i s i su s e fu l b e y o n d y o u r o w n p r o j e c t.

    O O i o - i ,80*/0

    6 0 %4 0 %2 0 % -I I i i

    Dewl

    Q u e s t i o n

    i:Gi:~ii i ,A n a ly E v a l G e n er F e a s T o t a l

    Accepted [ ] RejectedFigure 2 . Distribution of accepta nces an d reject ionsby type o f research ques t i on

    3 .1 W h a t k i n d s o f r e s u l ts d o s o f t w a r e e n g i n e e r sp r o d u c e ?The tang ib le con t r ibu t ions o f so f tware eng inee r ingresea rch m ay be p rocedures o r t echn iques fo r deve lop-

    m ent o r ana lys i s ; they m ay be m od e l s tha t gene ra l i ze f romspec i f i c exam ples , o r they m a y be spec i f i c too l s , so lu t ions ,o r re su l t s about pa r t i cu la r sys tem s . Tab le 3 l i st s the typesof re sea rch re su l t s tha t a re repo r ted in sof tware eng inee r -ing re sea rch pape rs and prov ides spec i f i c exam ples .3 .2 W h i c h o f th e s e ar e m o s t c o m m o n ?

    B y f a r th e m o s t c o m m o n k i n d o f I C S E p a p e r r e p o r ts an e w p r o c e d u r e o r t e c h n i q u e f o r d e v e l o p m e n t o r a n a ly s i s.M o d e l s o f v a r i ou s d e g r e e s o f p r e c i si o n a n d f o r m a l i t y w e r ea l so com m on, wi th be t t e r succes s ra te s fo r quan t i t a t ivethan fo r qua l i t a t ive m ode l s . Tools an d no ta t ions were we l lrepresen ted , usua l ly a s aux i l i a ry re su l t s in com bina t ionwi th a p rocedure o r t echn ique . Tab le 4 g ives the d i st r ibu-t ion o f subm is s ions to ICS E 2002 , based on read ing theabs t rac t s (bu t no t the pape rs ) , fo l low ed by graphs o f thecounts and d i s t r ibu t ions in F igures 3 and 4 .

    728

  • 7/28/2019 Shaw Icse2003

    4/11

    T ab le 3 . T yp es o f so f t ware en g i n eer i n g research resu l t sT y p e o f r e s u ltProcedure o rtechnique

    Qual i tat ive o rdescr ip t ive model

    Empir ical modelAnal y ti c m odelTool o r notat ionSpec ific solution,prototype, answer,o r judgme n tRepor t

    E x a m p l e sNe w or bet ter way to do som e task , such as design, implementat ion, maintenance,measurement, evaluation, selection from alternatives; include s tech niqu es for

    implementation, representation, managem ent, an d analysis; a tech niqu e should beopera t ional - -not advice or guidel ines, b ut a procedu reSt ructure or taxonom y for a problem area; archi tectural s ty le, f rame work, or design pat tern;non-formal domain analysis, well-grounded checklists, well-argued informalgeneralizations, guida nce for integrating o ther results, w ell-org anize d interest ingobservat ionsEmpir ical predict ive mod el based on obs erved dataStructural model that permits formal analysis or automatic manipulationImplemented tool that embodies a technique; formal language to su ppor t a technique or mo del(should have a calculus, semantics, or o ther basis for comput ing o r doing inference)Solut ion to applicat ion problem that show s application of SE p r inciples - m ay be design,

    prototy pe, or full implementation; careful analysis o f a syste m or i ts develop m ent, result ofa specific analysis, evaluation, or comp arisonInterest ing observat ions, ru les of thn m b, but not suff icient ly general o r system at ic to r i se to thelevel of a descriptive mo del.

    T ab le 4 . T yp es o f re search resu l t s rep resen t ed i n I C S E 2002 su b m i ss i on s an d accep t an cesT y p e o f r e s u lt S u b m i t t e d A c c e p t e dProced ure or technique 152(44%) 28 (51% )Qual i tat ive or descr ip t ive mo del 50 (14%) 4 (7% )Empir ical mod el 4 (1%) 1 (2% )Anal y ti c model 48 (14%) 7 (13%)Too l o r no t a ti on 49 (14%) 10 (18%)Speci f ic solut ion, prototype, answer, or judgm ent 34 (10%) 5 (9% )Repo r t 11 (3%) 0 (0% )

    TO TA L 348(100 .0%) 55 (100 .0%)

    R a t i o A c c / S u b18 %

    8%2 5%15 %2 0%15 %

    0%16 %

    3 5 03 0 02 5 02 0 01 5 0 1 0 0

    5 0 '0

    R e s u l t

    ]iiiiiiiiii

    i i t )/ . to , j . o . ,l " A c e e p , e d [ ] R e , c t e d j

    1 0 0 % ,8 0 %

    6 0 % -

    R e s u l t

    . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . :'E'?.i i l i )~ I I; i i ! ! i i i i i i i i i i i i i i2 0 0 , 0 I ' i , L

    A c c e p t e d [ 3 R e j e c t e d

    Fi gu re 3 . C ou n t s o f accep t an ces an d re jec t i on sb y t yp e o f re su l t F i gu re 4 . D i s t r i b u t ion o f accep t an ces an d re jec t i on sb y t y p e o f r e s u lt

    729

  • 7/28/2019 Shaw Icse2003

    5/11

    T h e n u m b e r o f re s u lt s i s l a rg e r t h a n t h e n u m b e r o fpape r s because 50 pape r s i nc l uded a suppor t i ng r e su l t ,usua l l y a t oo l o r a qua l i ta t i ve mode l .

    R e s e a r c h p r o j e c t s c o m m o n l y p r o d u c e r e s u lt s o f s e v e r a lk i nds . However , conf e r ences , i nc l ud i ng I CS E , usua l l yi mpose s t r ic t page l i mi t s . I n mos t cases , t h i s p r ov i des t ool it tl e s p a c e t o a l l o w f u ll d e v e l o p m e n t o f m o r e t h a n o n ei dea , pe r haps w i t h one o r t wo suppor t i ng i deas . M anyau t hor s p r esen t t he i nd i v i dua l i deas i n co nf e r en ce pape r s ,and t hen syn t hes i ze t hem i n a j our na l a r t i c l e t ha t a l l owss p a c e t o d e v e l o p m o r e c o m p l e x r e l a ti o n s a m o n g r e su l ts .3 .3 W h a t d o p r o g r a m c o m m i t t e e s l o o k f o r ?

    T h e p r o g r a m c o m m i t t e e l o o k s f o r i n t e r e s t i n g , n o v e l ,exc i t i ng r e su l t s t ha t s i gn i f i can t l y enhance our ab i l i t y t od e v e l o p a n d m a i n t a i n s o f tw a r e , t o k n o w t h e q u a l i t y o f t h esof t war e we deve l op , t o r ecogn i ze gene r a l p r i nc i p l e sabou t so f t war e , o r t o ana l y ze p r ope r t i e s o f so f t war e .

    Y o u s h o u l d e x p l a i n y o u r r e s u lt i n s u c h a w a y t h a ts o m e o n e e l s e c o u l d u s e y o u r i d e a s. B e s u r e t o e x p l a inwha t ' s nov e l o r o r i g i na l - i s it the i dea , t he app l i ca t i on o ft he i dea , t he i mpl em ent a t i on , t he ana l ys i s , o r wha t ?

    Def i ne c r i t i ca l t e r ms p r ec i se l y . Use t hem cons i s t en t l y .T h e m o r e f o r m a l o r a n a l y t i c th e p a p e r , t h e m o r e i m p o r t a n tthis is.

    H e r e a r e s o m e q u e s t i o n s t h a t t h e p r o g r a m c o m m i t t e em a y a s k a b o u t y o u r p a p e r :W h a t , p rec i se l y , d o yo u c la im to co n t r ib u te?

    Does your r e su l t f i J l l y s a t i s f y your c l a i ms? Ar e t hede f i n i t i ons p r ec i se , and a r e t e r m s used co ns i s t en t l y?

    A u t h o r s t e n d t o h a v e t r o u b l e i n s o m e s p e c i f i cs i t ua t i ons . Her e a r e some exampl es , w i t h adv i ce f o rs t ay i ng ou t o f tr oub l e : "

    I f y o u r r e s u l t o u g h t t o w o r k o n la r g e s y s t e m s , exp l a i nw h y y o u b e l i e v e i t s c al e s.

    I f y o u c l a i m y o u r m e t h o d i s " a u t o m a t ic " , us i ng i tshou l d n o t r equ i r e hum an i n t e r ven t i on . I f i t' sau t omat i c when i t ' s ope r a t i ng bu t r equ i r e s manua las s i s t ance t o conf i gur e , s ay so . I f i t ' s au t omat i ce x c e p t f o r c e r t a i n e a s e s , s a y s o , a n d s a y h o w o f t e nt he excep t i on s occur .

    I f y o u c l a i m y o u r r e s u l t i s " d i s t ri b u te d " , i t p r o b a b l yshou l d no t ha ve a s i ng l e cen t r a l con t r o l l e r o r s e r ve r .I f i t does , ex p l a i n wha t pa r t o f it i s d i s t ri bu t ed andwha t pa r t i s no t .

    I f y o u ' r e p r o p o s i n g a n e w n o t a t i o n f o r a n o l dp r o b l e m , e x p l a i n w h y y o u r n o t a t i o n i s c l e a r l ysupe r i o r t o t he o l d one .

    I f y o u r p a p e r i s a n " e x p e r i e n c e re p o r t" , r e l a t i ng t heu s e o f a p r e v i o u s l y - r e p o r t e d t o o l o r t e c h n i q u e i n apr ac t i ca l so f t war e p r o j ec t , be su r e t ha t you exp l a i nw h a t i d e a t h e r e a d e r c a n t a k e a w a y f r o m t h e p a p e r t o

    use i n o t he r s e t t ings . I f t ha t i dea i s i nc r ease dc o n f i d e n c e i n t h e to o l o r t e c h n i q u e , s h o w h o w y o u re x p e r i e n c e s h o u l d i n c r e a s e t h e r e a d e r ' s c o n f i d e n c ef o r a p p l ic a t io n s b e y o n d t h e e x a m p l e o f th e p a p e r .

    W h a t ~ n e w h e r e ?T h e p r o g r a m c o m m i t t e e w a n t s t o k n o w w h a t i s n o v e l

    o r exc i t i ng , and why . W ha t , spec i f i ca l l y , i s t hec o n t r ib u t i o n ? W h a t i s t h e i n c r e m e n t o v e r e a r l ie r w o r k b yt he s ame au t hor s? by o t he r au t hor s? I s t h i s a su f f i c i en ti n c r e m e n t , g i v e n th e u s u a l s t a n d a r d s o f s u b d i s c i p l i n e ?

    A b o v e a l l , t h e p r o g r a m c o m m i t t e e a l s o w a n t s t o k n o ww h a t y o u a c t u a l ly c o n t r i b u t e d t o o u r s t o r e o f k n o w l e d g ea b o u t s o f t w a r e e n g i n e e ri n g . S u r e , y o u w r o t e t h i s t o o l a n dt r i e d i t o ut . B u t w a s y o u r c o n t r i b u t i o n t h e t e c h n i q u e t h a t i se m b e d d e d i n th e t o o l , o r w a s i t m a k i n g a t o o l th a t ' s m o r ee f f e c t iv e th a n o t h e r t o o l s t h a t i m p l e m e n t t h e t e c h n i q u e , o rw a s i t s h o w i n g t h a t t h e t o o l y o u d e s c r i b e d i n a p r e v i o u sp a p e r a c t u a l l y w o r k e d o n a p r a c t i c a l l a r g e - s c a le p r o b l e m ?I t ' s b e t t e r f o r y o u a s t h e a u t h o r t o e x p l a i n t h a n f o r t h ep r o g r a m c o m m i t t e e t o g u e ss . B e c l e a r a b o u t y o u r c la i m . . .

    A w f u l I c o m p l e t e l y a n d g e n e r a l l y s o l v e d . . .I ( u n le s s y o u a c t u a l l y d i d ! )

    B a d I w o r k e d o n g a l u m p h i n g .( o r s t ud i ed , i nves t i ga t ed , sough t ,e x p l o r e d )

    P o o r I w o r k e d o n i m p r o v i n g g a l u m p h i n g .( o r con t r i bu t ed t o , pa r t i c i pa t ed i n ,h e l p e d w i t h )

    G o o d I s h o w e d t h e f e a s i b i li t y o f c o m p o s i n gb l i t z i ng w i t h f l i t z i ng .

    i I s i g n i fi c a n t ly i m p r o v e d t h e a c c u r a c y o ft he s t anda r d de t ec t o r .( o r p r o v e d , d e m o n s t r a t e d , c r e a t e d ,e s t a b li s h e d , f o u n d , d e v e l o p e d )

    B e t t e r I a u t o m a t e d t h e p r o d u c t i o n o f r i t zt ab l e s f r om spec i f i ca t i ons .

    W i t h a n o v e l a p p l i c a t io n o f th e b l i v e tt r a n s f o rm , I a c h i e v e d a 1 0 % i n c r e a s ei n s p e e d a n d a 1 5 % i m p r o v e m e n t inc o v e r a g e o v e r t h e s t a n d a r d m e t h o d .

    U s e v e r b s th a t s h o w r e s u l ts a n d a c h i e v e m e n t , n o t j u s te f f o r t and ac t i v i ty .[ " T 0 ' n o t . D o , o r d o n o t . T h e r e i s n o t O , . " -- Y o d a .[W h a t h a s b e e n d o n e b e f o r e ? H o w i s y o u r w o r k d i f f e r e n to r b e t t e r?

    W h a t e x i s t in g t e c h n o l o g y d o e s y o u r r e s e a r c h b u i l d o n ?W h a t e x i s t in g t e c h n o l o g y o r p r i o r r e s e a r c h d o e s y o u rr e s e a r c h p r o v i d e a s u p e r i o r a l t e r n a t i v e t o ? W h a t ' s n e wh e r e c o m p a r e d t o y o u r o w n p r e v i o u s w o r k ? W h a ta l t e r na t i ves have o t he r resea rch ers p u r s u e d , a n d h o w i sy o u r w o r k d i f f e re n t o r b e t t e r ?

    7 3 0

  • 7/28/2019 Shaw Icse2003

    6/11

    As i n o t he r a reas o f s c i ence and eng i nee r i ng , so f t war eeng i nee r i ng knowl edge g r ows i nc r ement a l l y . P r ogr amcommi t t ees a r e ve r y i n t e r e s t ed i n your i n t e r p r e t a t i on o fp r i o r w o r k i n t h e a re a . T h e y w a n t t o k n o w h o w y o u r w o r ki s r e la t ed t o t he p r i o r w or k , e i t he r by bu i l d i ng on i t o r bypr ov i d i n g an a l t e r na t i ve . I f you do n ' t ex p l a i n th i s , i t 'sh a r d f o r t h e p r o g r a m c o m m i t t e e to u n d e r s ta n d h o w y o u ' v ea d d e d t o o u r s t o r e o f k n o w l e d g e . Y o u m a y a ls o d a m a g ey o u r c r e d i b i l i t y i f t h e p r o g r a m c o m m i t t e e c a n ' t t e l lw h e t h e r y o u k n o w a b o u t r e l a t e d w o r k .

    E xpl a i n t he r e l a t i on t o o t he r wor k c l ea r l y . . .A w f u l T h e g a l u m p h i n g p r o b l e m h a s a t tr a c te d

    much a t t en t i on [ 3 ,8 ,10 ,18 ,26 ,32 ,37]Bad S mi t h [ 36] and Jones [ 27] wor k ed on

    g a l u m p h i n g .P oo r S mi t h [ 36] addr es sed ga l umphi ng by

    b l i t z ing , wher eas Jones [ 27] took af l i t z ing appr oach .

    G o o d S m i t h ' s b l i tz i n g a p p r o a c h t o g a l u m p h i n g[ 3 6 ] a c h i e v e d 6 0 % c o v e r a g e [ 3 9 ].Jones [ 27] ach i eved 80% b y f li t z ing ,bu t on l y f o r po i n t e r - f r ee cases [ 16].

    Be t t e r S m i t h ' s b l i t z i ng appr oa ch t o ga l umphi ng[ 3 6 ] a c h i e v e d 6 0 % c o v e r a g e [ 3 9 ].Jones [ 27] ach i eved 80% b y f l it z i ng ,bu t o n l y f o r po i n t e r - f r ee cases [ 16].W e m o d i f i e d th e b l i tz i n g a p p r o a c h t ouse t he k e r ne l r epr esen t a t i on o f f l i t z inga n d a c h i e v e d 9 0 % c o v e r a g e w h i ler e l ax i ng t he r e s t r i c t i on so t ha t on l ycyc l i c da t a s t r uc t u res a r e p r oh i b i t ed .What , precisely , i s the resul t?

    E x p l a i n w h a t y o u r r e s u l t i s a n d h o w i t w o r k s . B econcr e t e and spec i f i c . Use exampl es .

    l f y o u i n t ro d u c e a n e w m o d e l , be c l ea r abou t i t s pow er .How gene r a l i s i t ? I s i t based on empi r i ca l da t a , on af o r ma l s emant i c s , on ma t hemat i ca l p r i nc i p l e s? Howf or mal i s i t - - a qua l i t a t i ve mode l t ha t p r ov i des des i gng u i d a n c e m a y b e a s v a l u a b l e a s a m a t h e m a t ic a l m o d e l o fsome aspec t o f co r r ec t nes s , bu t t hey w i l l have t o s a t i s f yd i f f e r en t s t anda r ds o f p r oof . W i l l t he mode l s ca l e up t op r o b l e m s o f s i ze a p p r o p r i a t e t o i ts d o m a i n ?

    l f y o u i n t ro d u c e a n e w m e t r i c , def i ne i t p r ec i se ly . Doe si t measur e wha t i t pur por t s t o measur e and do so be t t e rt han t he a l t e r na t i ves? W hy?

    I f yo u i n t roduce a new arch i t ec t ura l s t yl e , des i gnpa t t e rn , or s i m i l ar des i gn e l ement , t r ea t i t a s i f it wer e an e w g e n e r a l iz a t i o n o r m o d e l . H o w d o e s i t d i f f e r f r o m t h ea l t e r na t i ves? I n wha t way i s i t be t t e r ? W ha t r ea l p r ob l emdoes i t so l ve? Does i t s ca l e?

    I f y o u r c o n t r ib u t io n i s p r i n c i p a l l y t h e s y n th e s i s o ri n t egra t i on o f o t her re su l t s or comp onen t s , b e c l e a r a b o u twh y t he syn t hes i s i s i t s e l f a con t r i bu t i on . W ha t i s nove l ,e x c i t i n g , o r n o n o b v i o u s a b o u t t h e i n t e g r a t i o n ? D i d y o ugener a l i ze p r i o r r e su l t s ? D i d you f i nd a be t t e rr e p r e s e n ta t i o n ? D i d y o u r r e s e a r c h i m p r o v e t h e i n d i v i d u alr e su l t s o r componen t s a s we l l a s i n t egr a t i ng t hem? Ap a p e r t h a t s i m p l y r e p o r t s o n u s i n g n u m e r o u s e l e m e n t st oge t he r i s no t enough , eve n i f it 's we l l - eng i nee r ed . T he r em u s t b e a n i d e a o r l e s s o n o r m o d e l t h a t th e r e a d e r c a n t a k ef r o m t h e p a p e r a n d a p p l y t o s o m e o t h e r s i tu a t io n .

    I f y o u r p a p e r i s c h i e fl y a r e p o r t o n e x p e r i e n c eappl y i ng research resu l t s t o a pra c t i ca l prob l em, s a y w h a tt h e r e a d e r c a n l e a r n f r o m t h e e x p e r i e n c e . A r e y o u rc o n c l u s i o n s s t r o n g a n d w e l l - s u p p o r t e d ? D o y o u s h o wcompar a t i ve da t a and / o r s t a t i s t i c s? An anecdo t a l r epor t ona s i ng l e p r o j ec t i s usua l l y no t eno ugh . A l so , i f your r epor tmi xes add i t i ona l i nnova t i on w i t h va l i da t i on t h r oughe x p e r i e n c e , a v o i d c o n f u s i n g y o u r d i s c u s s io n o f t h ei n n o v a t i o n w i t h y o u r r e p o r t o n e x p e r i e n c e . A f t e r a l l , i fy o u c h a n g e d t h e r e s u l t b e f o r e y o u a p p l i e d i t , y o u ' r ee v a l u a ti n g t h e c h a n g e d r e s u l t. A n d i f y o u c h a n g e d t h er e s u l t w h i l e y o u w e r e a p p l y i n g i t , y o u m a y h a v ec o n f o u n d e d t h e e x p e r i e n c e s w i th t h e t w o v e r s io n s .

    I f a t o o l p l a y s a f e a t u r e d r o l e i n y o u r p a p e r , what i st h e r o l e o f th e t o o l ? D o e s i t s i m p l y s u p p o r t t h e m a i ncon t r i bu t i on , o r i s t he t oo l i t s e l f a p r i nc i pa l con t r i bu t i on ,o r i s s o m e a s p e c t o f t h e t o o l ' s u s e o r i m p l e m e n t a t i o n th em a i n p o i n t ? C a n a r e a d e r a p p l y th e i d e a w i t h o u t t h e t o o l ?I f t he t oo l i s a cen t r a l pa r t o f r e su l t, wha t i s t he t echn i ca li n n o v a t i o n e m b e d d e d i n t h e t o o l o r i t s i m p l e m e n t a ti o n ?

    I f a s y s t e m i m p l e m e n t a t i o n p l a y s a f e a t u r e d r o l e iny o u r p a p e r , w h a t i s t h e r o l e o f t h e i m p l e m e n t a t i o n ? I s t h es y s t e m s o u n d ? D o e s i t d o w h a t y o u c l a i m i t d o e s ? W h a ti d e a s d o e s t h e s y s t e m d e m o n s t r a t e ?

    I f t he impl em ent a t i on i l l us t ra t e s an arch i t ec t ure ordes i gn s t ra t egy , w h a t d o e s i t r e v e a l a b o u t t h ea r c h i t e c t u r e ? W h a t w a s t h e d e s i g n r a t i o n a l e ? W h a tw e r e t h e d e s i g n t ra d e o f f s ? W h a t c a n t h e r e a d e r a p p l yt o a d i f f e r en t i mpl em ent a t i on?

    I f t h e i m p l e m e n t a t io n d e m o n s t r a t es a ni mpl ement a t i on t echn i que , h o w d o e s i t h e l p t h er e a d e r u s e t h e t e c h n i q u e i n a n o t h e r s e t t in g ?

    I f the i mpl ement a t i on demon s t ra t e s a capab i l i ty orp e r f o r m a n c e i m p r o v e m e n t , w h a t c o n c r e t e e v i d e n c ed o e s i t o f f e r t o s u p p o r t t h e c l a i m ?

    I f t he sy s tem i s i t s e l f t he re su l t , i n wha t way i s i t ac o n t r i b u t i o n t o k n o w l e d g e ? D o e s i t , f o r e x a m p l e ,s h o w y o u c a n d o s o m e t h i n g t h a t n o o n e h a s d o n eb e f o r e ( e s p e c i a ll y i f p e o p l e d o u b t e d t h a t t h is c o u l db e d o n e ) ?

    731

  • 7/28/2019 Shaw Icse2003

    7/11

    4 . W h y s h o u l d t h e r e a d e r b e l i e v e y o u r r e s u l t?S h o w e v i d e n c e t h a t y o u r r e su l t i s v a l i d - - t h a t i t a c t u a l l y

    h e l p s t o s o l v e t h e p r o b l e m y o u s e t o u t t o s o l v e .4 . 1 . W h a t k i n d s o f v a li d a t io n d o s o f t w a r ee n g i n e e r s d o ?

    S o f t w a r e e n g i n e e r s o f f e r s e v e r a l k i n d s o f e v i d e n c e i nsu p p o r t o f t h e i r r e sea rch r e su l t s . I t is e s sen t i a l t o s e l ec t af o r m o f v a l i d a t io n t h a t i s a p p r o p r i a t e f o r t he t y p e o f

    r e s e a r c h r e s u l t a n d t h e m e t h o d u s e d t o o b t a i n t h e r e s u l t .A s a n o b v i o u s e x a m p l e , a f o r m a l m o d e l s h o u l d b es u p p o r t e d b y r i g o r o u s d e r i v a t i o n a n d p r o o f , n o t b y o n e o rt w o s i m p l e e x a m p l e s . O n t h e o t h e r h a n d , a s i m p l ee x a m p l e d e r i v e d f r o m a p r a c ti c a l s y s t e m m a y p l a y a m a j o rr o l e i n v a l i d a t i n g a n e w t y p e o f d e v e l o p m e n t m e t h o d .T a b l e 5 l i s ts t h e t y p e s o f r e s e a r c h v a l i d a t i o n th a t a r e u s e di n s o f t w a r e e n g i n e e r i n g r e s e a r c h p a p e r s a n d p r o v i d e ss p e c i f i c e x a m p l e s . I n t h i s t a b le , t h e e x a m p l e s a r e k e y e d t ot h e t y p e o f r e s u lt t h e y a p p l y t o .

    Ta ble 5. Types of software engineering research val idat ionT y p e of val idation E x a m p l e sA n a l y s i s

    E v a l u a t i o n

    E x p e r i e n c e

    E x a m p l e

    P e r s u a s i o n

    I h a v e a n a l y z e d m y r e s u l t a n d f i n d i t s a t i s f a c t o r y t h r o u g h r i g o r o u s a n a l y s i s , e . g . . .F o r a f o r m a l m o d e l . . . r i g o r o u s d e r i v a t i o n a n d p r o o fF o r a n e m p i r i c a l m o d e l . . . d a t a o n u s e i n c o n t r o l l e d s i t u a t io nF o r a c o n t r o l l e d e x p e r i m e n t . . . c a r e f u l l y d e s i g n e d e x p e r i m e n t w i t h s t a t i st i c a l ly s i g n i f ic a n t

    r e su l t sG iv en th e s t a t ed c r i t e r i a , m y r e su l t .. .F o r a d e s c r i p t i v e m o d e l . . . a d e q u a t e l y d e s c r i b e s p h e n o m e n a o f i n t e re s t . . .

    F o r a q u a l i t a ti v e m o d e l . .. a c c o u n t s f o r t h e p h e n o m e n a o f i n t e r e s t . . .F o r a n e m p i r i c a l m o d e l . .. i s a b l e t o p r e d i c t . . . b e c a u s e . . . , o r

    . . . g en e ra t e s r e su l t s t h a t f i t ac tu a l d a t a . . .I n c lu d es f e as ib i l i t y s tu d i e s , p i l o t p ro j ec t sM y r e s u lt h a s b e e n u s e d o n r e a l e x a m p l e s b y s o m e o n e o t h e r t h a n m e , a n d t h e e v i d e n c e o f it s

    c o r r e c t n e s s / u s e f u ln e s s / e f f e c t i v e n e s s i s . . .F o r a q u a l i t a t i v e m o d e l . . . n a r r a t i v eF o r a n e m p i r i c a l m o d e l o r t o o l . . . d a ta , u s u a l l y s t a ti s t ic a l , o n p r a c t i c eF o r a n o t a t i o n o r t e c h n i q u e . . . c o m p a r i s o n o f s y s t e m s i n a c t u a l u s e

    H e r e ' s a n e x a m p l e o f h o w i t w o r k s onF o r a t e c h n i q u e o r p r o c e d u r e . . . a " s l ic e o f l i f e " e x a m p l e b a s e d o n a r e a l s y s t e m . . .F o r a t e c h n i q u e o r p r o c e d u r e . . . a s y s t e m t h a t I h a v e b e e n d e v e l o p i n g . . .F o r a t e c h n i q u e o r p r o c e d u r e . . . a t o y e x a m p l e , p e r h a p s m o t i v a t e d b y r e a l i t yT h e " s l i c e o f l i fe " e x a m p l e i s m o s t l i k e l y t o b e c o n v i n c i n g , e s p e c i a l l y i f a c c o m p a n i e d b y a ne x p l a n a t i o n o f w h y t h e s i m p l i f i e d e x a m p l e r e t a in s t h e e s s e n c e o f t h e p r o b l e m b e i n g s o l v e d .T o y o r t e x t b o o k e x a m p l e s o f t e n f a i l t o p r o v i d e p e r s u a s i v e v a l i d a t i o n , ( e x c e p t f o r s t a n d a r de x a m p l e s u s e d a s m o d e l p r o b l e m s b y t he f i e l d ).

    I t h o u g h t h a r d a b o u t t h i s , a n d I b e l i e v e p a s s i o n a t e l y t h a t . ..F o r a t e c h n i q u e . .. i f y o u d o i t t h e f o l l o w i n g w a y , t h e n . ..F o r a s y s t e m . . . a s y s t e m c o n s t r u c t e d l i k e t h i s w o u l d . . .F o r a m o d e l . .. t h is e x a m p l e s h o w s h o w m y i d e a w o r k s

    V a l i d a t i o n p u r e l y b y p e r s u a s i o n i s r a r e l y s u ff i c i e n t f o r a r e s e a r c h p a p e r . N o t e , t h o u g h , t h a t i f t h eo r i g in a l q u e s t i o n w a s a b o u t f e a s i b i li t y , a w o r k i n g s y s t e m , e v e n w i t h o u t a n a l y s i s , c a n s u f f i c e

    B l a t a n t a s s e r t i o n N o s e r i o u s a t t e m p t t o e v a l u a t e r e s u lt . T h i s is h i g h l y u n l i k e l y to b e a c c e p t a b l e4 .2 W h i c h o f t h es e a re m o s t c o m m o n ?

    A l a s , w e l l o v e r a q u a r t e r o f t h e I C S E 2 0 0 2 a b s t r a c t sg i v e n o i n d i c a ti o n o f h o w t h e p a p e r ' s r e s u l t s a r e v a l i d a t e d ,i f a t a ll . E v e n w h e n t h e a b s t r a c t m e n t i o n s t h a t t h e r e s u l tw a s a p p l i e d t o a n e x a m p l e , i t w a s n o t a l w a y s c l e a rw h e t h e r t h e e x a m p l e w a s a t e x t b o o k e x a m p l e , o r a r e p o r to n u se i n t h e f i e ld, o r so m e th in g i n b e tw een .

    T h e m o s t s u c c e s s f ul k i n d s o f v a l i d a ti o n w e r e b a s e d o na n a l y s i s a n d r e a l - w o r l d e x p e r i e n c e . W e l l - c h o s e n e x a m p l e sw e r e a l s o s u c c e s s f u l . P e r s u a s i o n w a s n o t p e r s u a s i v e , a n dn a r r a t i v e e v a l u a t i o n w a s o n l y s l i g h t l y m o r e s u c c e s s f u l .T a b l e 6 g i v e s t h e d i s t r ib u t i o n o f s u b m i s s i o n s t o I C S E2 0 0 2 , b a s e d o n r e a d i n g t h e a b s t r a c t s ( b u t n o t t h e p a p e r s ) ,f o l l o w e d b y g r a p h s o f t h e c o u n t s a n d d i s tr i b u ti o n s .F i g u r e s 5 a n d 6 s h o w t h e s e c o u n t s a n d d i s t r ib u t i on s .

    7 3 2

  • 7/28/2019 Shaw Icse2003

    8/11

    T a b l e 6 . T y p e s o f r e s e a r c h v a l i d a t i o n r e p r e s en t e d i n I C S E 2 0 0 2 s u b m i s s i o n s a n d a c c e p t a n c e sT y p e o f v a l i d a t io n S u b m i t t ed A c c e p t e d R a t i o A c c / S u bAnalys i s 48 (16%) 11 (26%) 23%Eva lua t ion 21 (7%) 1 (2%) 5%Exper ience 34 (11%) 8 (19%) 24%Exam p le 82 (27%) 16 (37%) 20%S om e exam ple , can ' t t e l l whe the r i t 's toy o r ac tua l use 6 (2%) 1 (2%) 17%P ersuas ion 25 (8%) 0 (0 . 0%) 0%No m ent ion of va l ida t ion in abs t rac t 84(28% ) 6 (14%) 7%

    14 %O T A LVal idation

    30 0 -250200 i10050

    o

    - " ) , A c c e p t . t

    300(100 . 0%) 43 (100 . 0%)Val ida t ion

    100%-80% . . . . . . . . . . . . . . . . ~ : i ~ ~

    2 0 % . ~0% - ~~!!i f l l .~ , . + ~ ~

  • 7/28/2019 Shaw Icse2003

    9/11

    I f you per form ed a con tro l led experiment , exp l a i n t heexper i m ent a l des i gn . W ha t i s t he hypo t hes i s? W ha t i st h e t r e a tm e n t ? W h a t i s b e in g c o n t r o l l e d ? W h a t d a t ad i d y o u c o l l e c t, a n d h o w d i d y o u a n a l y z e i t? A r e t h er esu l t s s i gn i f i can t ? W ha t a r e t he po t en t i a l l yc o n f o u n d i n g f a c to r s , a n d h o w a r e th e y h a n d l e d ? D ot h e c o n c l u s i o n s f o l l o w r i g o r o u s l y f r o m t h ee x p e r i m e n t a l d a t a ?

    I f you per form ed an empir ica l study , e x p l a i n w h a ty o u m e a s u r e d , h o w y o u a n a l y z e d i t , a n d w h a t y o uc o n c l u d e d . W h a t d a t a d i d y o u c o l le c t , a n d h o w ? H o wi s the a na l ys i s r e l a t ed to t he goa l o f suppo r t i ng yourc l a i m abou t t he r e su l t ? Do no t conf use cor r e l a t i onwi t h causa l i t y .

    l f yo u use a smal l example for exp la in ing the r esu lt ,p r o v i d e a d d i t io n a l e v i d e n c e o f i ts p r a c t ic a l u s e a n dscalabi l i ty .

    5 . H o w d o y o u c o m b i n e th e e l e m e n t s i n to ar e s e a r c h s t r a t e g y ?

    I t i s c l ea r t ha t no t a l l comb i na t i ons o f a r e sea r chques t i on , a r e su l t , and a va l i da t i on s t r a t egy l ead t o goodr e s e a r c h . S o f t w a r e e n g i n e e r i n g h a s n o t d e v e l o p e d g o o dgener a l gu i dan ce on t h i s ques ti on .

    T ab l es 1 , 3 , and 5 de f i ne a 3 - d i mens i ona l space . S om ep o r t i o n s o f t h a t s p a c e a r e d e n s e l y p o p u la t e d : O n ec o m m o n p a r a d i g m i s t o f in d a b e t t e r w a y t o p e r f o r m s o m esof t war e deve l opment o r ma i n t enance t a sk , r ea l i ze t h i s i na c o n c r e t e p r o c e d u r e s u p p o r t e d b y a t o o l , a n d e v a l u a t e t h ee f f e c t i v e n e s s o f t h i s p r o c e d u r e a n d t o o l b y d e t e r m i n i n ghow i t s use a f f ec t s some measur e ( e .g . , e r r o r r a t e s ) o fq u a l it y . A n o t h e r c o m m o n p a r a d i g m i s t o f r e d a b e t te r w a yt o eva l ua t e a f o r r na l i zab l e p r ope r t y o f a so f t war e sys t em,d e v e l o p a f o r m a l m o d e l t h a t s u p p o r t s i n f e r e n c e , a n d t os h o w t h a t t h e n e w m o d e l a l l o w s f o r m a l an a l y si s o r p r o o fof t he p r op e r t i e s o f i n te r e s t.

    C l e a r l y , t h e r e s e a r c h e r d o e s n o t h a v e f r e e c h o i c e t om i x a n d m a t c h t h e t e c h n i q u e s - - v a l i d a t i n g t h e c o r r e c tn e s so f a f o r m a l m o d e l t h r o u g h f i e l d s tu d y i s a s i n a p p r o p r i a tea s a t t e m p t i n g f o r m a l v e r i fi c a t io n o f a m e t h o d b a s e d o ng o o d o r g a n i z a t io n o f r u le s o f t h u m b .

    S e l ec t i ng a t ype o f r e su l t t ha t w i ll answe r a g i venq u e s t i o n u s u a l l y d o e s n o t s e e m t o p r e s e n t m u c h d i f fi c u l ty ,a t l ea s t f o r r e sea r che r s wh o t h i nk ca r e f u l l y abou t t hec h o i c e . B l i n d l y a d o p t i n g t h e r e s e a r c h p a r a d i g m s o m e o n eu s e d l a st y e a r f o r a c o m p l e t e l y d i f fe r e n t p r o b l e m i s ad i f f e r en t case , o f cour se , and i t can l ead t o s e r i ous mi s f i t s.

    C h o o s i n g a g o o d f o r m o f v a l id a t i o n is m u c h h a r d e r ,and t h i s i s o f t en a sou r ce o f d i f f i cu l t y i n com pl e t i ng as u c c e s s f u l p a p e r . T a b l e 6 s h o w s s o m e c o m m o n g o o dm a t c h e s . T h i s d o e s n o t , u n f o r t u n a te l y , p r o v i d e c o m p l e t eg u i d a n c e .

    W h e n I a d v i s e P h D s t u d e n t s o n t h e v a l i d a t i o n s e c t i o nof t he i r t heses , I o f f e r t he f o l l ow i ng heur i s t i c : L oo kc a r e f u l ly a t th e s h o r t s ta t e m e n t o f t h e r e s u l t - - t h e p r i n c i p a lc l a i m o f t he t hes is . T h i s o f t en h as t wo or t h r ee c l auses( e .g . , I fo u n d a n e f f i c i e n t a n d c o m p l e t e m e t h o d . . . " ) ; i f s o ,e a c h p r e s e n ts a s e p a r a t e v a l id a t i o n p r o b l e m . A s k o f e a c hc l ause whe t he r i t i s a g l oba l s t a t ement ( "a l ways" , " f u l l y" ) ,a q u a l i f i e d s t a t e m e n t ( " a 2 5 % i m p r o v e m e n t " , " f o rnon cyc l i c s t ruc t u r es . . . " ) , o r an ex i s t en t ia l s t a t ement {"wef o u n d a n i n s t a n c e o f ' ) . G l o b a l s t a t e m e n t s o f t e n r e q u i r eana l y t i c va l i da t i on , qua l i f i ed s t a t ement s ca n o f t e n bev a l i d a t e d b y e v a l u a t i o n o r c a r e f u l e x a m i n a t i o n o fe x p e r i e n c e , a n d e x i s t e n t i a l s t a t e m e n t s c a n s o m e t i m e s b ev a l i d a te d b y a s i n g le p o s i t i v e e x a m p l e . A f r e q u e n t r e s u lto f t h i s d i s cus s i on i s tha t s t uden t s r e s t a t e t he t hes i s c l a i mst o r e f l e c t m o r e p r e c i s e l y w h a t t h e t h e s e s a c t u a l l y a c h i e v e .I f w e h a v e t h i s d i s c u s s io n e a r l y e n o u g h i n t h e t h e s isp r oces s , s t uden t s t h i nk abou t p l ann i ng t he r e sea r ch w i t hdemons t r ab l e c l a i ms i n mi nd .

    C o n c r e t e l y , T a b l e 7 s h o w s t h e c o m b i n a t i o n s t h a t w e r er e p r e s e n t e d a m o n g t h e a c c e p t e d p a p e r s a t I C S E 2 0 0 2 ,o m i t t i n g th e 7 f o r w h i c h t h e a b s t r ac t s w e r e u n c l e a r a b o u tva l i da t i on :

    T a b l e 7 . P a r a d i g m s o f I C S E 2 0 0 2 a c c e p t a n c e sQ u e s t i o n R e s u l t V a l i d a t i o nDevel method Proced ure AnalysisDevel method Procedure Exper ienceDevel method Proced ure ExampleDevel method Qual model Exper ienceDevel method Analytic model Exp erienceDevel method Notat ion or tool Exper ience

    Analysisnalysis method Proced ureAnalysis method Proced ure EvaluationAnalysis method Proce dure Expe rienceAnalysis method Proce dure ExampleAnalysis methodAnalysis methodAnalysis method

    Analytic modelAnalytic m odelT oolSpecific analysisSpecific analysis

    Eval of instanceEval o f instance

    Exper ienceE xampl eAnalysisAnalysisExample

    #233221512612132

    6 . D o e s t h e a b s t r a c t m a t t e r ?T h e a b s t r a c ts o f p a p e r s s u b m i t te d t o I C S E c o n v e y a

    s e n s e o f t h e k i n d s o f r e s e a r c h s u b m i t t e d t o t h e c o n f e r e n c e .S o m e a b s t ra c t s w e r e e a s i e r t o r e a d a n d ( a p p a r e n t l y ) m o r ei n f o r m a t iv e t h a n o t h e rs . M a n y o f t h e d e a r e s t a b s t ra c t s h a da c o m m o n s t r uc t u re :

    T w o o r th r e e s e n t e n c e s a b o u t t h e c u r r e n t s t a te o f th ea r t , iden t i f y i ng a pa r t i cu l a r p r ob l e m

    O n e o r t w o s e n t e n c e s a b o u t w h a t t h i s p a p e rcon t r i bu t e s t o i mpr ov i ng t he s i t ua t i on

    7 3 4

  • 7/28/2019 Shaw Icse2003

    10/11

    One o r t wo sen t ences abou t t he spec i f i c r e su l t o f t hep a p e r a n d t h e m a i n i d e a b e h in d i t

    A sen t enc e abou t how t he r e su l t i s dem ons t r a t ed o rd e f e n d e dAbs t r ac t s i n r oug h l y th i s f o r ma t o f t en exp l a i ned c l ea r l ywha t r ead e r s cou l d exp ec t in t he pape r .

    A c c e p t a n c e r a t e s w e r e h i g h e s t f o r p a p e r s w h o s eabs t r ac t s i nd i ca t e t ha t ana l ys i s o r expe r i en ce p r ov i dese v i d e n c e i n s u p p o r t o f t h e w o r k . D e c i s i o n s o n p a p e r s w e r em a d e o n t h e b a s is o f th e w h o l e p a p e rs , o f c o u r s e , n o t j u s tt he abs t r ac t s - - bu t i t i s r easonab l e t o a s sume t ha t t heabs t r ac t s r e f l ec t wha t ' s i n t he pa pe r s .

    W h e t h e r y o u l ik e i t o r n o t , p e o p l e j u d g e p a p e r s b y t h e i rabs t r ac t s and r ead t he abs t r ac t i n o r de r t o dec i de whe t he rt o r ead t he whol e pape r . I t ' s i mpor t an t f o r t he abs t r ac t t ot e l l t he s t o r y . Don ' t a s sume , t hough , t ha t s i mpl y add i ng asen t ence abou t ana l ys i s o r expe r i ence t o your abs t r ac t i ssu f f i c i en t ; t he pape r mus t de l i ve r wha t t he abs t r ac tp r o m i s e s7 . Q u e s t io n s y o u m i g h t a s k a b o u t t h is r e p o r t7 .1 . Is th i s a sure- f i re rec ipe?

    No, not a t a l l . F i rs t , i t ' s not a recipe. Second, not a l lso f t war e eng i nee r s sha r e t he s ame v i ews o f i n te r e s t i ng ands i gn i f i can t r e sea r ch . E ven i f your pape r i s c l ea r abou tw h a t y o u ' v e d o n e a n d w h a t y o u c a n c o n c l u d e , m e m b e r s o fa p r o g r a m c o m m i t t e e m a y n o t a g r e e a b o u t h o w t oi n t e r p r e t your r e su l t . T hese a r e usua l l y hones t t echn i ca ld i sagr eemen t s , and comm i t t ee mem ber s w i l l t r y ha r d t ou n d e r s t a n d w h a t y o u h a v e d o n e . Y o u c a n h e l p b ye x p l a i n in g y o u r w o r k c l e ar ly ; t h is r e p o r t s h o u l d h e l p y o udo t ha t .7 .2 I s ICS E d i f f e r e n t f r o m o t h e r c o n f e r e n c e s ?

    I CS E r ecogn i zes s eve r a l d i s t i nc t t ypes o f t echn i ca lpape r s [ 6 ] . F or 2002 , t hey wer e pub l i shed sepa r a t e l y int h e p r o c e e d i n g s

    S e v e r a l o t h e r c o n f e r e n c e s o f f e r " h o w t o w r i t e a p a p e r "adv i ce :I n 1 9 9 3 , se v e r a l O O P S L A p r o g r a m c o m m i t te e v e t e ra n s

    g a v e a p a n e l o n " H o w t o G e t a P a p e r A c c e p t e d a tOO P S L A" [ 9 ] . T h i s upda t ed the 1991 adv i ce f o r t he s ameconf e r ence [ 14]S I G S O F T o f f e rs tw o e s s a y s o n g e t t i n g p a p e r saccep t ed , t hough ne i t he r was ac t ua l l y wr i t t en f o r as o f t w a re e n g i n e e r in g a u d i en c e . T h e y a r e " H o w t o H a v eY o u r A b s t r a c t R e j e c t e d " [ 2 6 ] ( w h i c h f o c u s e s o nt h e o r e ti c a l p ap e r s ) a n d " A d v i c e t o A u t h o r s o f E x t e n d e dAbs t r ac t s " , whi ch was wr i t t en f o r P L D I . [ 16] .

    Ra t h e r o l de r , L ev i n and Redde l l , the 1983 S OS P( o p e r a t in g s y s t e m s ) p r o g r a m c o - c h a i r s o f f e r e d a d v i c e o n

    w r i ti n g a g o o d s y s t e m s p a p e r [ 1 1 ] . U S E N I X n o w p r o v i d e st h i s adv i ce t o i t s au t hor s . A l so i n t he sys t ems ve i n ,P a r tr i d g e o f f e r s a d v i c e o n " H o w t o I n c r e a s e t h e C h a n c e sY o u r P a p e r i s A c c e p t e d at A C M S I G C O M M " [ 1 5] .

    S I G C H I o f f e r s a " G u i d e t o S u c c e s s f u l P a p e r sS ubmi ss i on" t ha t i nc l udes c r i t e r i a f o r eva l ua t i on andd i s cu s s io n o f c o m m o n t y p e s o f C H I r e s u lt s , t o g e t h e r w i thh o w d i f f e re n t e v a l u a t io n c r i te r i a a p p l y f o r d i f f e r e n t t y p e sof r e su lt s [ 13] . A s t ud y [ 8 ] o f r eg i ona l f ac t o r s t ha t a f f ec ta c c e p t a n c e f o u n d r e g i o n a l d i f f e r e n c e s i n p r o b l e m s w i t hnove l t y , s i gn i f i cance , f ocus , and w r i t i ng qua l i t y .

    I n 1 9 9 3 , t h e S I G G R A P H c o n f e r e n c e p r o g r a m c h a i rw r o t e a d i s c u s s io n o f th e s e l e c t i o n p r o c e ss , " H o w t o G e tY o u r S I G G R A P H P a p e r R e j e c t e d " [ 1 0 ] . T h e 2 0 0 3S I GGR.AP H ca l l f o r pape r s [ 21 ] has a d esc r i p t i on o f ther e v i e w p r o c e s s a n d a f r e q u e n t l y - a s k e d q u e s ti o n s s e c t i o nw i t h a n e x t e n s i v e s e t o f q u e s t io n s o n " G e t t in g a P a p e rA c c e p t e d " .7 .3 . W h a t a bo ut t h i s r e p o r t i t s e l f ?

    P e o p l e h a v e a s k e d m e , " w h a t w o u l d h a p p e n i f y o us u b m i t t e d t h i s t o I C S E ? " W i t h o u t v e n t u r i n g t o p r e d i c tw h a t a n y g i v e n I C S E p r o g r a m c o m m i t te e w o u l d d o , I n o t et ha t a s a r e sea r ch r e su l t o r t echn i ca l pape r ( a " f i nd i ng" i nBr oo ks ' s ense [ 3 ] ) i t f a l l s shor t in a num ber o f ways :

    T h e r e i s n o a t te m p t t o s h o w t h a t a n y o n e e l s e c a na p p l y t h e m o d e l . T h a t i s , th e r e i s n o d e m o n s t r a t io n o fi n t e r - r a t e r r e l i ab i l i t y , o r f o r t ha t ma t t e r evenr epea t ab i l i t y by t he s ame r a t e r .

    T h e m o d e l i s n o t j u s t i fi e d b y a n y p r i n c i p le d a n a l ys i s ,t h o u g h f r a g m e n t s , s u c h a s t h e t y p e s o f m o d e l s t h a tcan se r ve a s r e su l t s , a r e p r i nc i p l ed . I n de f ense o f t hem o d e l , B o w k e r a n d S t a r t [ 2 ] s h o w t h a t u s e f u lc l a s s i f i ca t i ons b l end p r i nc i p l e and p r agmat i cd e s c r i p ti v e p o w e r .

    O n l y o n e c o n f e r e n c e a n d o n e p r o g r a m c o m m i t t e e i sr e f l e c t e d h e r e . T h e use o f abs t r ac t s a s p r ox i es f o r fu l l pap e r s i s

    suspec t . T h e r e i s l i tt l e d i s cus s i on o f r e l a t ed wo r k o t he r t han

    t h e e s s a y s a b o u t w r i t i n g p a p e r s f o r o t h e rc o n f e r e n c e s . A l t h o u g h d i s c u s s io n o f r e l a t e d w o r kd o e s a p p e a r i n t w o c o m p l e m e n t a r y p a p e r s [ 1 9 , 2 0 ],t h i s r epo r t does no t s t and a l one .

    O n t h e o t h e r h a n d , I b e l i e v e t h a t th i s r e p o r t d o e s m e e tB r o o k s ' s t a n d a r d f o r " r u l e s o f t h u m b " ( g e n e r a li z a ti o n s ,s i g n e d b y t h e a u t h o r b u t p e r h a p s i n c o m p l e t e l y s u p p o r t e db y d a t a, j u d g e d b y u s e f u l n e s s a n d f r e sh n e s s ) , a n d I o f f e r i tin that sense.8 . A c k n o w l e d g e m e n t s

    T h i s w o r k d e p e n d e d c r i ti c a l ly o n a c c e s s t o t h e e n t i r eb o d y o f su b m i t t e d p a p e r s f o r th e I C S E 2 0 0 2 c o n f e r e n c e ,

    735

  • 7/28/2019 Shaw Icse2003

    11/11

    w h i c h w o u l d n o t h a v e b e e n p o s s i b l e w i t h o u t t h ec o o p e r a t i o n a n d e n c o u r a g e m e n t o f t h e I C S E 2 0 0 2p r o g r a m c o m m i t t e e . T h e d e v e l o p m e n t o f th e s e i d e a s h asa l s o b e n e f i t e d f r o m d i s c u s s i o n w i t h t h e I C S E 2 0 0 2p r o g r a m c o m m i t te e , w i t h c o l l ea g u e s a t C a r n e g i e M e l l o n ,a n d a t o p e n d i s c u s s io n s e s s i on s a t F S E C o n f e r e n c e s . T h ew o r k h a s b e e n s u p p o r t e d b y t h e A . J . P e r l i s C h a i r a tC a r n e g i e M e l l o n U n i v e r s i ty .9 . R e f e r e n c e s1. Victor R. Basi l i . The exper ime ntal paradigm in sof twareengineer ing. In Experimental Software Engineering ~Issues: CriticaIAssessment and Future Directives. P r ocof Dags t uh l - W or kshop , H . D i e t e r Rombach , V i c t o r R .Basi l i , and Richard Se lby (eds), publ i she d as LectureNotes in Computer Science #706, Spr inger-Ver lag 1993.2 . Geo f f r ey Bow ker and S usan L e i gh S ta r: Sorting ThingsOut: Classification and lts Consequences.M I T P r es s,

    19993. Freder ick P . Brooks , J r . Grasping Real i ty Thro ughI l lus ion-- Interact ive Graphics Serving Science. Proc1988 A CM SIGCHI Human Factors in ComputerSystems Conf (CHI '88)pp. 1-11.4 . Rebecca Bumet t . Technical Communication. T h o m s o nHei n l e 2001 .5. Thom as F . Gieryn. Cultural Boundaries of Science:Credibility on the line. Uni v o f Chi cago P r es s , 1999 .6 . I CS E 2002 P r ogr am Comm i t tee . Types oflCSEpapers.http://icse-con erences.org/2002/in o/paperTyp es.html7. Impact Project. "Determ ining the impact of sof twareengineer ing research upon p ract ice . Panel sum mary,

    Proc. 23rd lnternational Conference on SoftwareEngineering (ICSE 2001), 20018. E l len Isaacs and John Tang. Why don't more non-North-American papers get accepted to CHI?http://aem.org/sigchi/bulletin/1996.1/isaacs.htrnl9 . Ra l ph E . Johnson & pane l. How t o Ge t a P ape r Accep t eda t OOP S L A . Proc OOPSLA'93, pp . 429- 436 ,http://aem.org/sigplan/oopsla/oopsla96/how93.html10 . J i m Ka j iya . How t o Ge t Yo ur S I GG RAP H P aperRejec ted. Mirroredat

    http //ww w.ee.gateeh.edu/student.services/phd/phd-advice/kajiya11. Roy L ev i n and Davi d D . Rede l l. Ho w ( and How Not ) t oW r i t e a Goo d S ys t ems P ape r. ACM SIGOPS Operating

    Systems Review, Vol . 17, No . 3 (July, 1983), pages 35-40. http://fip.digital.com/pub/DEC/SRC/other/SOSPadvice.txt12. Wi l l iam Newm an. A prel im inary analys is of the productsof HC I research, us ing pro form a abstracts . Proc 1994

    ACM SIGCHI Human Factors in Computer Systems Conf(CHI '94), pp .278- 284 .13. Wi l l iam New ma n et a l . Guide to Successful PapersSubmission at CH12001. http://aem.org/sigs/sigehi/chi200 /eall/submissions/guide-papers.html14. OOP S L A '91 P r ogr am Com mi t t ee . How t o ge t yo ur pape r

    accep t ed a t OOP S L A . Proc OOPSLA'91, pp .359- 363 .http://acm.org/sigplan/oopsla/oopsla96/how91html15 . C r a i g P a r tr i dge. Ho w t o I nc r ease the Chances you r P ape ri s A c ce p t e d at A C M S I G C O M M .http://www.acm.org/sigcomrn/conference-misc/author-guide.html16 . W i l l iam P ugh and P DL I 1991 P r ogr am Comm i t tee .Advice to Authors o f Extended Abstracts.

    http //acre.org/sigsoft/con erenees/pughadvicehtml17. Samu el Redw ine, e t a l . DoD Related SoftwareTechnology Requirements, Practices, and Prospects fo rthe Future. I DA P aper P - 1788 , June 1984.18 . S . Redwi ne & W . R i dd l e. S o f t war e techno l ogy

    maturat ion. Proceedings of the Eighth InternationalConference on Software Engineering, M ay 1985 , pp .189-200.

    19 . M ar y S haw. T h e com i ng- of - age o f so f t war e a r ch i tec t u r eresearch. Proc. 23rd Int'l Conf on Software Engineering(ICSE 2001), pp. 656-664a.20. M ar y S haw. W ha t makes goo d r e sea rch i n so f t war eeng i nee r ing? P r esen t ed a t E T AP S 02 , appea r ed i n

    Opi n i on Com er depa r tment , Int'l Jour on Software Toolsfor Tech Transfer, vol 4 , DOI 10 .1007 / s10009- 002- 0083-4 , June 2002 .21. S igGraph 2003 Cal l for Papers .http://www.siggraph.org/s2003/c p/papers/index.html22. W. F . T ichy, P . Lukow icz, L . Prechel t , & E. A. Hein z.

    "E xper i ment a l eva l ua ti on i n comp ut e r s c i ence : Aquant i ta t ive s tudy." Journal of Systems Software, Vol .28, No . I , 1995, pp. 9-18.23 . W al t e r F . T i chy . "S houl d compu t e r s c ien t is t s expe r i men tmor e? 16 r easons t o avo i d expe r i ment a ti on . " 1EEEComputer, Vol . 31 , No . 5 , M ay 199824 . M ar v i n V . Z e l kowi t z and D e l or es W al lace . E xper i ment a l

    val idat ion in sof tware engineer ing. Information andSoftware Technology, Vol 39, no 11, 1997, pp. 735-744.25 . M ar v i n V . Z e l kow i t z and De l or es W al l ace . E xper i ment a l

    mode l s f o r va l i da ti ng t echno l ogy . IEEE Computer, Vol .31, No. 5, 1998, pp.23-31.26. M ar y- C l a ir e van L eunen and R i cha r d L i p t on . How to

    have your abstract rejected.http://acm'rg/sigsfdcn erences/vanLeunenLiptn'html


Recommended