Download - Sample Size Planning for Appropriate Stascal Power...Samantha F. Anderson Sample Size Planning for Appropriate Stascal Power Samantha F. Anderson Arizona State University 2/9/19 Sample

SamanthaF.Anderson

SampleSizePlanningforAppropriateSta2s2calPower

SamanthaF.AndersonArizonaStateUniversity

SampleSize 12/9/19

SPSPPortland,Oregon

2/9/2019

SamanthaF.Anderson

h@p://www.gpower.hhu.de

  BUCSSisavailableasaseriesofwebapps:h@ps://designingexperiments.com/shiny-r-web-apps/

DownloadingG*Power&BUCSS

2/9/19 SampleSize 2

SamanthaF.Anderson

  OverviewofstaOsOcalpower  IngredientsofstaOsOcalpower  Minimumclinicaldifferenceapproach

  SoSwareopOons  G*Powertutorial

  SamplesizeplanningviapriorinformaOon  Adjustmentsforuncertaintyand/orpublicaOonbias  BUCSStutorial

  Reallife“issues”insamplesizeplanning  AlternaOveapproaches

Outline

SampleSize 32/9/19

SamanthaF.Anderson

  StaOsOcalpower:Theprobabilityofdetec%nganeffectofinterest,undertheassumpOonthatthenullhypothesisisfalse

  Detect:tobeabletodeclarestaOsOcallysignificant(p<α)  H0false:relatestothattablefromourintrostatscourse

•  Power=1-β

StaOsOcalPower

SampleSize 42/9/19

H0true H0false

StaOsOcallysignificant TypeIerror Correct

Nonsignificant Correct TypeIIerror(β)

SamanthaF.Anderson

  Distantpast:.48formediumeffects(Cohen,1962)  Recent(?)past:noimprovement(Sedlmeier&Gigerenzer,1986)

  Present  Generalbetween-subjectsesOmate:.35(Bakkeretal2012)  Neuroscience:.18(Bu@onetal2013)  Healthpsychology:.34-.36forsmalleffects(Maddock&Rossi2001)

  Future?  80%J

PowerinPsychology

SampleSize 52/9/19

SamanthaF.Anderson

  PublicaOonbias  ManyjournalsaresOlllesslikelytopublishnullresults  Thistendencyisnotnecessarilyillogical  Andjournaleditorsmightnotbetheonlyonesmakingthisdecision

  Ifyoureffectisreal:Morepoweràmorelikelytoreachsignificance

  àMorelikelytopublish

ImportanceofpowerforYOU

SampleSize 62/9/19

SamanthaF.Anderson

  Power-PublicaOonParadox?  LowpowerbutresearcherssOllareabletopublish

•  1.Lowpowerdoesn’tmeanyouhavea0%chanceofdetecOngyoureffect

•  2.MulOpletesOng– MorecommontoadjusttheTypeIerrorrateformulOpletesOnginANOVAthaninregression

•  3.Researcherdegreesoffreedom/quesOonableresearchpracOces/p-hacking

  Reducingp-hackingàPowerwillbemoreimportantfortheindividualresearcher

  PreregistraOon/transparencymovement  Samplesizeplanningisanhonestwayto“p-hack”

ImportanceofpowerforYOU

SampleSize 72/9/19

SamanthaF.Anderson

OneofthefactsoflifeasapsychologististhatyouareexpectedtoreadalotofresearcharOcles.Let’ssaythatoverthenextfewweeks/months/decades,youreadexactly100journalarOclespublishedinpsychologyjournals.Forsimplicity,let’ssupposethateveryoneofthesearOclesreportsastaOsOcallysignificantresultatthe.05level,andthateacharOclecontainsonlyonestaOsOcaltest.Atfirstglance,wemightbelievetheresultsreportedinall100studiesbecauseineachandeverycase,theauthorsobtaineddatathatwouldbeveryunlikelyifthenullhypothesisweretrue.Shouldyoubelievethatthenullhypothesishasbeencorrectlyrejectedineveryoneofthesestudies?Whatisyourbestguessaboutthepercentageofthesestudiesthatreportanincorrectresult?  a)50%  b)5%  c)0%  d)Itdepends

ImportanceofpowerforTHEFIELD

SampleSize 82/9/19

SamanthaF.Anderson

  Lowerfalsediscoveryrate  “Mostpublishedresearchfindingsarefalse”(Ioannidis2005)

  FDR=

  Thelargerthepower,thecloserFDRistotheTypeIerrorrate

  Improvedreplicability  StaOsOcalpowerofthereplicaOonstudy&originalstudyaffectsreplicaOonsuccess

ImportanceofpowerforTHEFIELD

SampleSize 92/9/19

α × P(H0true)α × P(H0true) + power × P(H0 false)

SamanthaF.Anderson

  α-level  Popula%onmagnitudeofeffectsize

  Moretechnically,powerdeterminedbythenoncentralityparameter(combinaOonofsamplesize&effectsize)

  Samplesize**Frameworksforsamplesizeplanningdifferinhowtheydetermineanappropriatevaluefortheeffectsize

IngredientsofStaOsOcalPower

SampleSize 102/9/19

SamanthaF.Anderson

  Rulesofthumbtypicallyassumeaconstanteffectsizeforalleffects

  Greatdiversityinpsychologytopics,manipula2ons,doses,designs

  àastandardeffectsizeisunlikely

  Dependingontrueeffectsize,ruleofthumbcouldleadtounderpoweredoroverpoweredstudy

  Todaywe’llputthe“planning”insamplesizeplanning.J

RulesofThumb

2/9/19 SampleSize 11

SamanthaF.Anderson

  EffectsizespecifiedisatheoreOcalpopulaOonvalue  Doesnota@emptto“guess”thetruesizeofthehypothesizedeffect

  ImplicitlyassumeseffectssmallerthanthetheoreOcalvaluearenotworthwhiletodetect

  SensibleifthereisaclearclinicalorpracOcallyrelevantthresholdofaneffectsizeworthdetecOng

MinimallyImportantDifferenceApproach


SamanthaF.Anderson

  G*Powerh@p://www.gpower.hhu.de  User-friendly,butsomewhatlimitedinscope

  GLIMMPSEh@p://glimmpse.samplesizeshop.org/#/  HasmorefuncOonalityforenteringrawmetrics(e.g.,meansandSDs)

WebPowerh@p://webpower.psychstat.org/wiki/  SupportslongitudinaldesignsandSEM

  PASS(PowerAndSampleSize)h@p://powerandsamplesize.comWidevarietyofcomplexdesigns

  Notfreelyavailable

SoSwareOpOons


SamanthaF.Anderson

Independentttest

  Testfamily:  ttest

  StaOsOcaltest:  Means(differencebetweentwoindependentmeans(twogroups)

  Typeofpoweranalysis:  Apriori:Computerequiredsamplesize–givenα,powerandeffectsize

G*PowerTutorial


SamanthaF.Anderson

Independentttest

  Tails  Effectsized

  CanenterdirectlyOR  Canclick“Determine”andenterrawmeansandSDs

  αerrorprob  Power(1-βerrorprob)  AllocaOonraOo(N2/N1)

G*PowerTutorial


SamanthaF.Anderson

Independentttest

  Tails  Two

  Effectsized  0.5

  αerrorprob  .05

  Power(1-βerrorprob)  0.80

  AllocaOonraOo(N2/N1)  1

G*PowerTutorial


SamanthaF.Anderson

Independentttest

  Tails  Two

  Effectsized  0.5



  AllocaOonraOo(N2/N1)  1

G*PowerTutorial


SamanthaF.Anderson

  Change  AllocaOonraOo:1à2  Power:.8à.95  d:.5(medium)à.2(small)  αerrorprob:.05à.005(BenjaminetalNaturearOcle)

G*PowerTutorial


SamanthaF.Anderson

B.S.FactorialANOVA:Interac%on

  Testfamily:  Ftest

  StaOsOcaltest:  ANOVA:fixedeffects,special,maineffects,andinteracOons


G*PowerTutorial


SamanthaF.Anderson

B.S.FactorialANOVA:Interac%on  Effectsizef(S=.1,M=.25,L=.4)

  SDofgroupmeans/errorSD(commonSDofgroupscores)

  CanenterdirectlyOR  CancalculatefromvarianceexplainedbyeffectanderrorvarianceOR

  Cancalculatefromη2

  αerrorprob  Power(1-βerrorprob)  Numeratordf  Numberofgroups

  #oftotalgroupsinthedesign

G*PowerTutorial


f =

σmσ e

η2 =

σ m2

σ total2

f = η

2

1−η2

SamanthaF.Anderson


  Effectsizef•  .1



  Numeratordf  ThreewayinteracOon  (a-1)*(b-1)*(c-1)=2

  Numberofgroups•  3x2x2design=12groups

G*PowerTutorial


SamanthaF.Anderson

MixedANOVA:W.S.MainEffect

  Testfamily:  Ftest

  StaOsOcaltest:  ANOVA:repeatedmeasures,withinfactors


G*PowerTutorial


SamanthaF.Anderson


  Effectsizef  αerrorprob  Power(1-βerrorprob)  Numberofgroups

  #ofbetween-subjectsgroups

  Numberofmeasurements  #withinsubjectscondiOons/Omes

  CorrelaOonamongrepeatedmeasuresNonsphericitycorrecOon(ε)

  1/n-1<ε<1

G*PowerTutorial


•  Can only accommodate 1 W.S. factor and 1 B. S. factor

SamanthaF.Anderson


  Effectsizef:.3  αerrorprob:.05  Power(1-βerrorprob):.95  Numberofgroups:2  Numberofmeasurements:4  CorrelaOonamongrepeatedmeasures:.5NonsphericitycorrecOon(ε):1

G*PowerTutorial


SamanthaF.Anderson

  EffectsizespecifiedisasampleesOmateofapopulaOonvalue

  Ideaisto“guess”thelikelyeffectsizeinadvance  UseinformaOonabouttheexpectedeffectfrompriorresearch

  Setupyourstudytohave80%powertodetecttheeffectsizeyoubelievetheeffectis

  Themoreaccurateyourguessis,themorelikelythesamplesizesuggestedbythepoweranalysiswillachieveyourdesiredlevelofpower

  Typically–takesampleeffectsizefrompriorstudyatfacevalue

SampleSizePlanningviaPriorInformaOon


SamanthaF.Anderson

  Goal:90%power  Previousstudy

  d=0.7  n=20/group

  àEnterintoG*Power  Tails:Two  Effectsized:0.7  αerrorprob:.05  Power(1-βerrorprob):0.90  AllocaOonraOo(N2/N1):1

MoOvaOngExample


SamanthaF.Anderson

MoOvaOngExample


Suggested n = 44/group

SamanthaF.Anderson

MoOvaOngExample


Suggested n = 44/group

ACTUAL POWER = ~ 15%

SamanthaF.Anderson

  PublicaOonbias  JournalpreferenceforsignificantfindingsandselecOvereporOngofmulOpletests

  ShiSsthecenteroftheconfidenceintervalsurroundingthesampleeffectsizeupward

  TruncatesthedistribuOonofpossiblesampleeffectsizes,“censoring”valuesbelowacertainthreshold,regardlessofthetrueeffectsize

  Uncertainty  SampleeffectsizeisanesOmateofthepopulaOoneffectsize

  Thereisalwaysgoingtobesamplingerror

PublicaOonBias&Uncertainty

SampleSize 312/9/19

SamanthaF.Anderson

  ScienOficjournalsusuallydonotpublishresultsunlesstheyarestaOsOcallysignificantatp<.05

  “…publicaOonbiasinpsychologyprimarilyinvolvesthesuppressionofnonsignificantresults.”[1]

  “…prevalenceofp-valuesjustbelowthearbitrarycriterionforsignificancewasobservedinallthreejournals”[2]

PublicaOonBias


1.Simonsohn,Nelson,&Simmons.(2014).Perspec2vesonPsychologicalScience.2.Masicampo&Lalande.(2012).QuarterlyJournalofExperimentalPsychology.

SamanthaF.Anderson

  HypotheOcalExample  Independentt-test  d=0.6  n=25δ=?

PublicaOonBias

SampleSize 332/9/19

SamanthaF.Anderson

  HypotheOcalExample  Independentt-test  d=0.6  n=25δ=?

  =0.16

PublicaOonBias

SampleSize 342/9/19

0

1

2

3

4

5

0.50 0.75 1.00 1.25 1.50Value of d

Prob

abilit

y D

ensi

ty

δ = 0.2

δ = 0.5

δ = 0.8

δ̂ ML

SamanthaF.Anderson

  HypotheOcalExample  Independentt-test  d=0.6  n=25  =0.16

PublicaOonBiasandUncertainty

SampleSize 352/9/19

δ̂ ML

0.000

0.025

0.050

0.075

0.100

0.00 0.25 0.50 0.75 1.00Value of δ

Like

lihoo

d

n = 25

SamanthaF.Anderson

  HypotheOcalExample  Independentt-test  d=0.6  n=25  =0.16

  Independentt-test  d=0.6  n=150  =0.6

PublicaOonBiasandUncertainty

SampleSize 362/9/19

δ̂ ML

δ̂ ML

0.000

0.025

0.050

0.075

0.100

0.00 0.25 0.50 0.75 1.00Value of δ

Like

lihoo

d

n = 25 n = 150

SamanthaF.Anderson

Perugini,Gallucci,&ConstanOni(2014)

  Recommendedthelowerboundofthe90%confidenceintervalforδinpowercalculaOons

  Adjustsforuncertainty

SafeguardPower


SamanthaF.Anderson

  AdjustsforpublicaOonbias  DistribuOonofdundertheinfluenceofpublicaOonbias(smallvaluesofdcensored)

  MaximumlikelihoodesOmateofd:findthevalueofδthatmaximizesthethelog-likelihoodoftheaboveequaOon

  NosoSwaretodothiseasilyL

Hedges’(1984)Method


h*(t|δ ,n) = h(t|δ ,n)

A(δ ,n,α )= PDF(t|δ ,n)Power(δ ,n,α )

SamanthaF.Anderson

  AdjustsforuncertaintyandpublicaOonbias

  FindvalueofδassociatedwithspecificCDFprobabiliOes:

  Selectδcorrespondingto50thpercenOleàadjustforpublicaOonbiasonly

  SelectmoreconservaOvevalueofδàadjustforuncertaintytoo

  MuchlesscomplicatedtoperformthanitlooksJ

BUCSSApproach


LT[FO ;νnum,νden ,δ ] =

LF [FO |νnum,νden ,δ ]1 - CDFF [Fcrit (1 -αO)|νnum,νden ,δ ]

SamanthaF.Anderson

  HowoSenamethodachievesitsintendedpowerinthelongrun

δ=0.5dorig=0.72

replicaOonn=32actualpower=.50

Assurance


0.0

0.2

0.4

0.6

0.8

1.0

0 25 50 75 100Study

Powe

r of P

lann

ed S

tudy

SamanthaF.Anderson

  HowoSenamethodachievesitsintendedpowerinthelongrun

  BUCSSPercenOle=1–desiredassurance

Assurance


0.0

0.2

0.4

0.6

0.8

1.0

0 25 50 75 100Study

Powe

r of P

lann

ed S

tudy

Prior Study d < .5 Prior Study d > .5

SamanthaF.Anderson

  BUCSSconsistentlyachieveshigherpowerwhencomparedtousingthesampleeffectsizedirectly(“atfacevalue”)insamplesizeplanning

  CansOllbeusedevenifthepriorstudydidnotreportaneffectsizemeasure

  Onlyneedstoknowthe“t”or“F”staOsOc

  Letsusersspecify  1.Desiredpower  2.Desiredassurance/adjustmentforuncertainty  3.AmountofadjustmentforpublicaOonbias

BenefitsofBUCSSApproach


SamanthaF.Anderson

EffectSizescanBeComplicated


  Design:One-waywithinsubjects,2groups,N=25,ρ=.3,f2=.16(large)  1.G-Power("asinCohen1988”)

ncp=4.00,power=.48  2.G-Power:("asinGpower3.0”)

ncp=11.428,power=.90  3.Rpwr.f2.test:

  pwr.f2.test(u=1,v=24,f2=.16,sig.level=.05,power=NULL)  Power=.49

  4.Powercalcula%ons,ncpfromformulancp=f2*(df.effect+df.error+1)=.16*(1+24+1)=4.16

  1-pf(Fcrit,df1=1,df2=24,ncp=(.16*(1+24+1)))  Power=.49

  5.Powercalcula%ons,usingncpfrom#2  1-pf(FcritES,df1=1,df2=24,ncp=11.428)  Power=.90

  6.G-Powerdependentttest,withdz=2*f=.8,totalN=25,2tailsncp=4,power=.97

f 2 =

σm2

σ e2

SamanthaF.Anderson

  Adjustedδ=zero  Originalstudyhashighuncertaintyandbias

  EffectsizeesOmateisuntrustworthyatthedesiredlevelofassurance

  HighsensiOvityfordetecOngTypeIerror

  SoluOon:Lowerassuranceand/orpublicaOonbiasadjustment

NoncentralityParameter=Zero


n=20/group

SamanthaF.Anderson

  RPackage  BUCSS  AvailableonCRAN

  WebApplicaOonsDesigningExperiments.com

SoSware


SamanthaF.Anderson

  PreviousStudy  N=30(n=15)  t(28)=2.40,p=.01  d=0.88

BUCSSExample


Vohs&Schooler.(2008)PsychologicalScience.

SamanthaF.Anderson

  PreviousStudy  N=30(n=15)  t(28)=2.40,p=.01  d=0.88

  PlannedStudy  90%intendedpower  N=58(n=29)  t(56)=-0.77,p=.44

BUCSSExample


SamanthaF.Anderson

•  BUCSSWebApp:Independentttest

BUCSSExample


SamanthaF.Anderson

BUCSSExample


nrep=977

SamanthaF.Anderson

  RevisedPreviousStudy  N=100(n=50)  t(98)=4.40,p<.0001  d=0.88

BUCSSExample


nrep = 29

SamanthaF.Anderson

2x2B.S.ANOVA:Interac%on  ObservedF-valuefromthepreviousstudyTotalsamplesizeofthepreviousstudy

  NumberoflevelsofFactorA  NumberoflevelsofFactorB  Effectofinterest(A,BorinteracOon)  alpha-levelofpreviousstudy

  ThisisthepublicaOonbiasadjustment  .05meansthepriorstudylikelywouldhavebeenrejectedifp>.05

  alpha-levelofplannedstudy  Assurance

  Between.5(nouncertaintyadjustment)and.99(biguncertaintyadjustment)  .80assurancemeansthat80%oftheOme,youwillachieveyourdesiredpower

  StaOsOcalpower

BUCSSTutorial


SamanthaF.Anderson

2x2B.S.ANOVA:Interac%on  ObservedF-value:7Totalsamplesize:120

  NumberoflevelsofFactorA:2  NumberoflevelsofFactorB:2  Effectofinterest:InteracOon  alpha-levelofpreviousstudy:1  alpha-levelofplannedstudy:.05  Assurance:0.80  StaOsOcalpower:0.80

BUCSSTutorial


BUCSSTutorial


*The output will always be “per-group” sample size for designs with between-subjects factors*

SamanthaF.Anderson

3x3W.S.ANOVA:Interac%on  ObservedF-valuefromthepreviousstudyTotalsamplesizeofthepreviousstudy

  NumberoflevelsofFactorA  NumberoflevelsofFactorB  Effectofinterest(A,BorinteracOon)  alpha-levelofpreviousstudy




  StaOsOcalpower

BUCSSTutorial


SamanthaF.Anderson

3x3W.S.ANOVA:Interac%on  ObservedF-value:7Totalsamplesize:60

  NumberoflevelsofFactorA:3  NumberoflevelsofFactorB:3  Effectofinterest:InteracOon  alpha-levelofpreviousstudy:.05  alpha-levelofplannedstudy:.05  Assurance:0.80  StaOsOcalpower:0.80

BUCSSTutorial


BUCSSTutorial


* Because all factors are within-subjects, suggested sample size is total N *

SamanthaF.Anderson

Mul%pleLinearRegressionDetecteffectofasinglepredictor,controllingforotherpredictors

  Observedt-valuefromthepreviousstudyTotalsamplesizeofthepreviousstudy

  Numberofpredictors  alpha-levelofpreviousstudy




  StaOsOcalpower

BUCSSTutorial


SamanthaF.Anderson

Mul%pleLinearRegression  Observedt-value:3Totalsamplesize:80

  Numberofpredictors:4  alpha-levelofpreviousstudy:.05  alpha-levelofplannedstudy:.05  Assurance:0.80  StaOsOcalpower:0.80

BUCSSTutorial


BUCSSTutorial


* For regression, suggested sample size is total N *

SamanthaF.Anderson

  “General”appsallowformorecomplicatedeffects/designsinANOVA

  Plannedcomparisons•  E.g.thepairwisedifferencebetweengroups1and3ina3-groupdesign

  >2-factordesigns•  E.g.,3x3x3design

  Threeappsrelatedtolinearregression

  TesOngsinglepredictor,setofpredictors,modelR2

BUCSSTutorial


Citation: Anderson, Kelley, & Maxwell. (2017).

Psychological Science.

SamanthaF.Anderson

  IfyourstudyhasmulOpleeffects/hypotheses,besttoplansamplesizefor

  Yourfocaleffect  Thesmallesteffect

  PlanaheadforpotenOalmissingdata  PlanaheadwhencorrecOngformulOpletesOng

  IfyouplantodoaBonferroniadjustment,yourplannedstudyalpha-levelwon’tbe.05

IssuesinSampleSizePlanning


SamanthaF.Anderson

  Samplesizeplanningforprecision  AccuracyinParameterEsOmaOon(AIPE)  Planinsteadtohaveanarrowconfidenceinterval

•  MBESSpackageinR  SequenOalanalysis

  Samplesizenotfixed  Dataevaluatedastheycomeinatpre-specified“stoppingpoints”  Approachesfromtodaycanbeusedforthe“full”N

  Don’tforgetaboutdesign!  Within-subjectsdesignsJ  AdjusOngforcovariates(ANCOVA)J  Stronger/moreintensemanipulaOonJ

AlternaOveApproaches


SamanthaF.Anderson

  GeneralOverviewsLenth.(2001).AmericanSta2s2cian.

  Maxwell,Kelley,&Rausch.(2008).AnnualReviewofPsychology.Perugini,Gallucci,&ConstanOni.(2018).Interna2onalReviewofSocialPsychology.

  GeneralUncertaintyAdjustmentMcShane&Bockenholt.(2014).Perspec2vesonPsychologicalScience.Perugini,Gallucci,&ConstanOni.(2014).Perspec2vesonPsychologicalScience.

  BUCSSApproach  Anderson,Kelley,&Maxwell.(2017).PsychologicalScience.  Anderson&Maxwell.(2017).Mul2variateBehavioralResearch.

  Psychologists’intuiOonsonpower  Bakkeretal.(2016).PsychologicalScience.

FurtherReading


SamanthaF.Anderson

  [email protected]

  AnyquesOons?

Thankyou!


SamanthaF.Anderson

Anderson,S.F.,Kelley,K.,&Maxwell,S.E.(2017).Samplesizeplanningformoreaccuratesta%s%calpower:Amethodadjus%ngeffectsizesforpublica%onbiasanduncertainty.PsychologicalScience.

Anderson,S.F.,&Kelley,K.(2017).Bias-UncertaintyCorrectedSampleSize(BUCSS).[RPackage].Anderson,S.F.,&Maxwell,S.E.(2016).There’smorethanonewaytoconductareplica%onstudy:Beyondsta%s%calsignificance.PsychologicalMethods.Anderson,S.F.&Maxwell,S.E.(2017).Addressingthereplica%oncrisis:Usingoriginalstudiestodesignreplica%onstudieswithappropriatesta%s%calpower.MBR.

Bakker,etal.(2016).Researchers’intui%onsaboutpoweranalysisinpsychologicalresearch.PsychologicalScience.Brand,A.etal.(2008).Accuracyofeffectsizees%matesfrompublishedpsychologicalresearch.PerceptualandMotorSkills.Cohen,J.(1988).Sta;s;calpoweranalysisforthebehavioralsciences(2nded.).Gelman,A.,&Loken,E.(2013)Thegardenofforkingpaths:Whymul%plecomparisonscanbeaproblemevenwhenthereisno“fishingexpedi%on”or“phacking”andtheresearchhypothesiswaspositedaheadof%me.AmericanScien;st.

Ioannidis,J.P.A.(2005).Whymostpublishedresearchfindingsarefalse.PLoSMedicine.Lane,D.M.,&Dunlap,W.P.(1978).Es%ma%ngeffectsize:Biasresul%ngfromthesignificancecriterionineditorialdecisions.Bri;shJournalofMathema;calandSta;s;calPsychology.

Masicampo,E.J.,&Lalande,D.R.(2012).Apeculiarprevalenceofpvaluesjustbelow.05.QuarterlyJournalofExperimentalPsychology.Maxwell.,S.E.etal.(2008).Samplesizeplanningforsta%s%calpowerandaccuracyinparameteres%ma%on.AnnualReviewofPsychology.Maxwell,S.E.etal.(2015).Ispsychologysufferingfromareplica%oncrisis?Whatdoesfailuretoreplicatereallymean?AmericanPsychologist.McShane,B.B.,&Bockenholt,U.(2014).Youcannotstepintothesamerivertwice:Whenpoweranalysesareop%mis%c.Perspec;vesonPsychologicalScience.Muller,K.&Feqerman,B.(2002).RegressionandANOVA:AnIntegratedApproach.Olejnik,S.,&Algina,A.(2000).Measuresofeffectsizeforcompara%vestudies:Applica%ons,interpreta%ons,andlimita%ons.ContemporaryEduca;onalPsychology.

Pren%ce,D.A.,&Miller,D.T.(1992).Whensmalleffectsareimpressive.PsychologicalBulle;n.Sedlmeier,P.,&Gigerenzer,G.(1989).Dostudiesofsta%s%calpowerhaveaneffectonthepowerofstudies?PsychologicalBulle;n.Simmons,J.P.etal.(2011).False-posi%vepsychology:Undisclosedflexibilityindata-collec%onandanalysisallowspresen%nganythingassignificant.PsychologicalScience.

Simonsohn,U.,Nelson,L.D.,&Simmons,J.P.(2014).Pcurveandeffectsize:correc%ngforpublica%onbiasusingonlysignificantresults.Perspec;vesonPsychologicalScience.

Taylor,D.J.,&Muller,K.E.(1996).Biasinlinearmodelpowerandsamplesizecalcula%onduetoes%ma%ngnoncentrality.Communica;onsinSta;s;cs:TheoryandMethods.

Yuan,K.H..&Maxwell,S.E.(2005).Onthepost-hocpowerintes%ngmeandifferences.JournalofEduca;onalandBehavioralSta;s;cs.

SelectedReferences


SamanthaF.Anderson

  Hedges(1984)  Recentsurveyshadshown94–97%publishedresultsp<.05

  JournaleditorscitedstaOsOcalsignificanceascriteriaforpublishing

  FormedHedges’modelofpublicaOonbias

  PublicaOonbiasismissingdataproblem

PublicaOonBias:ClassicWork


SamanthaF.Anderson

δ~Unif(0.1,1)  n=20

•  Differentperformanceforalldvalues

–  δ<.3:LBunsuccessfulwhileTM=0

–  δ>.3:TMmoreoSensuccessfulthanLB

  n=250•  d>.3:TM=LB•  d<.3:methodsdiffer

–  δ=.1–.4:LBunsuccessfulwhileTM=0

–  δ>.4:TM=LB

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

0.639

0.8

1.0

1.2

1.4

0.25 0.50 0.75 1.00δ

Sam

ple

d

● UnsuccessfulSuccessful

a

●

●

●●

●

0.639

0.8

1.0

1.2

1.4

0.25 0.50 0.75 1.00δ

Sam

ple

d

● UnsuccessfulSuccessfulTM Estimate = 0

c

●

●

●

●

●

●

●

●●

●

●

●

●

●

●●

0.175

0.4

0.6

0.8

1.0

1.2

0.25 0.50 0.75 1.00δ

Sam

ple

d

● UnsuccessfulSuccessful

b

●

●

●

●

●●

●

●

●●

●

●

●

●

●

●●

0.175

0.4

0.6

0.8

1.0

1.2

0.25 0.50 0.75 1.00δ

Sam

ple

d

● UnsuccessfulSuccessfulTM Estimate = 0

d

BUCSSvsSafeguardPower


SamanthaF.Anderson

PopulaOonEffectSizeFormulas


dz =µdiffσ diff

=µdiff

σ 12 +σ 2

2 − 2ρσ 1σ 2

d =µdiffσ

f 2 =(αβ )2∑ / abσε2

f 2 =(αβ )2∑ / (a −1)(b−1) +1

σε2

SamanthaF.Anderson

NoncentralityParameter


λ =(θ -θ0 ′) M

-1(θ -θ0)σ 2

M= C(GN)IG ′C = C ′Cn

M-1 =n(C ′C )-1

λ =(θ -θ0)'n(C ′C )

-1(θ -θ0)σ 2

λ =nΣ

λ:noncentralityparameterθ:vectorofmeansθ0:hypothesizedvalueC:contrastmatrixG:numberofgroupsN:totalsamplesizen:pergroupsamplesizeΣ:populaOoneffectsize

Muller&Fe@erman.(2002).SASIns2tute

SamanthaF.Anderson

  Designs  Dependentt:  BetweensubjectsANOVA

  InteracOoninsplitplotdesign

NoncentralityParameterExamples


λ =

n × (µ1 - µ2)2× (1 - ρ)σ

λ = f2 × (dfeffect +dferror +1)

λ =

n (αβ )2∑σ 2(1 - ρ)

SamanthaF.Anderson

UsingRelaOonshipBetweenλ&n


if (TM.5[i] > 0) { nrep