Post on 03-Feb-2021
transcript
SamanthaF.Anderson
SampleSizePlanningforAppropriateSta2s2calPower
SamanthaF.AndersonArizonaStateUniversity
SampleSize 12/9/19
SPSPPortland,Oregon
2/9/2019
SamanthaF.Anderson
h@p://www.gpower.hhu.de
BUCSSisavailableasaseriesofwebapps:h@ps://designingexperiments.com/shiny-r-web-apps/
DownloadingG*Power&BUCSS
2/9/19 SampleSize 2
SamanthaF.Anderson
OverviewofstaOsOcalpower IngredientsofstaOsOcalpower Minimumclinicaldifferenceapproach
SoSwareopOons G*Powertutorial
SamplesizeplanningviapriorinformaOon Adjustmentsforuncertaintyand/orpublicaOonbias BUCSStutorial
Reallife“issues”insamplesizeplanning AlternaOveapproaches
Outline
SampleSize 32/9/19
SamanthaF.Anderson
StaOsOcalpower:Theprobabilityofdetec%nganeffectofinterest,undertheassumpOonthatthenullhypothesisisfalse
Detect:tobeabletodeclarestaOsOcallysignificant(p<α) H0false:relatestothattablefromourintrostatscourse
• Power=1-β
StaOsOcalPower
SampleSize 42/9/19
H0true H0false
StaOsOcallysignificant TypeIerror Correct
Nonsignificant Correct TypeIIerror(β)
SamanthaF.Anderson
Distantpast:.48formediumeffects(Cohen,1962) Recent(?)past:noimprovement(Sedlmeier&Gigerenzer,1986)
Present Generalbetween-subjectsesOmate:.35(Bakkeretal2012) Neuroscience:.18(Bu@onetal2013) Healthpsychology:.34-.36forsmalleffects(Maddock&Rossi2001)
Future? 80%J
PowerinPsychology
SampleSize 52/9/19
SamanthaF.Anderson
PublicaOonbias ManyjournalsaresOlllesslikelytopublishnullresults Thistendencyisnotnecessarilyillogical Andjournaleditorsmightnotbetheonlyonesmakingthisdecision
Ifyoureffectisreal:Morepoweràmorelikelytoreachsignificance
àMorelikelytopublish
ImportanceofpowerforYOU
SampleSize 62/9/19
SamanthaF.Anderson
Power-PublicaOonParadox? LowpowerbutresearcherssOllareabletopublish
• 1.Lowpowerdoesn’tmeanyouhavea0%chanceofdetecOngyoureffect
• 2.MulOpletesOng– MorecommontoadjusttheTypeIerrorrateformulOpletesOnginANOVAthaninregression
• 3.Researcherdegreesoffreedom/quesOonableresearchpracOces/p-hacking
Reducingp-hackingàPowerwillbemoreimportantfortheindividualresearcher
PreregistraOon/transparencymovement Samplesizeplanningisanhonestwayto“p-hack”
ImportanceofpowerforYOU
SampleSize 72/9/19
SamanthaF.Anderson
OneofthefactsoflifeasapsychologististhatyouareexpectedtoreadalotofresearcharOcles.Let’ssaythatoverthenextfewweeks/months/decades,youreadexactly100journalarOclespublishedinpsychologyjournals.Forsimplicity,let’ssupposethateveryoneofthesearOclesreportsastaOsOcallysignificantresultatthe.05level,andthateacharOclecontainsonlyonestaOsOcaltest.Atfirstglance,wemightbelievetheresultsreportedinall100studiesbecauseineachandeverycase,theauthorsobtaineddatathatwouldbeveryunlikelyifthenullhypothesisweretrue.Shouldyoubelievethatthenullhypothesishasbeencorrectlyrejectedineveryoneofthesestudies?Whatisyourbestguessaboutthepercentageofthesestudiesthatreportanincorrectresult? a)50% b)5% c)0% d)Itdepends
ImportanceofpowerforTHEFIELD
SampleSize 82/9/19
SamanthaF.Anderson
Lowerfalsediscoveryrate “Mostpublishedresearchfindingsarefalse”(Ioannidis2005)
FDR=
Thelargerthepower,thecloserFDRistotheTypeIerrorrate
Improvedreplicability StaOsOcalpowerofthereplicaOonstudy&originalstudyaffectsreplicaOonsuccess
ImportanceofpowerforTHEFIELD
SampleSize 92/9/19
α × P(H0true)α × P(H0true) + power × P(H0 false)
SamanthaF.Anderson
α-level Popula%onmagnitudeofeffectsize
Moretechnically,powerdeterminedbythenoncentralityparameter(combinaOonofsamplesize&effectsize)
Samplesize**Frameworksforsamplesizeplanningdifferinhowtheydetermineanappropriatevaluefortheeffectsize
IngredientsofStaOsOcalPower
SampleSize 102/9/19
SamanthaF.Anderson
Rulesofthumbtypicallyassumeaconstanteffectsizeforalleffects
Greatdiversityinpsychologytopics,manipula2ons,doses,designs
àastandardeffectsizeisunlikely
Dependingontrueeffectsize,ruleofthumbcouldleadtounderpoweredoroverpoweredstudy
Todaywe’llputthe“planning”insamplesizeplanning.J
RulesofThumb
2/9/19 SampleSize 11
SamanthaF.Anderson
EffectsizespecifiedisatheoreOcalpopulaOonvalue Doesnota@emptto“guess”thetruesizeofthehypothesizedeffect
ImplicitlyassumeseffectssmallerthanthetheoreOcalvaluearenotworthwhiletodetect
SensibleifthereisaclearclinicalorpracOcallyrelevantthresholdofaneffectsizeworthdetecOng
MinimallyImportantDifferenceApproach
2/9/19 SampleSize 12
SamanthaF.Anderson
G*Powerh@p://www.gpower.hhu.de User-friendly,butsomewhatlimitedinscope
GLIMMPSEh@p://glimmpse.samplesizeshop.org/#/ HasmorefuncOonalityforenteringrawmetrics(e.g.,meansandSDs)
WebPowerh@p://webpower.psychstat.org/wiki/ SupportslongitudinaldesignsandSEM
PASS(PowerAndSampleSize)h@p://powerandsamplesize.comWidevarietyofcomplexdesigns
Notfreelyavailable
SoSwareOpOons
2/9/19 SampleSize 13
SamanthaF.Anderson
Independentttest
Testfamily: ttest
StaOsOcaltest: Means(differencebetweentwoindependentmeans(twogroups)
Typeofpoweranalysis: Apriori:Computerequiredsamplesize–givenα,powerandeffectsize
G*PowerTutorial
2/9/19 SampleSize 14
SamanthaF.Anderson
Independentttest
Tails Effectsized
CanenterdirectlyOR Canclick“Determine”andenterrawmeansandSDs
αerrorprob Power(1-βerrorprob) AllocaOonraOo(N2/N1)
G*PowerTutorial
2/9/19 SampleSize 15
SamanthaF.Anderson
Independentttest
Tails Two
Effectsized 0.5
αerrorprob .05
Power(1-βerrorprob) 0.80
AllocaOonraOo(N2/N1) 1
G*PowerTutorial
2/9/19 SampleSize 16
SamanthaF.Anderson
Independentttest
Tails Two
Effectsized 0.5
αerrorprob .05
Power(1-βerrorprob) 0.80
AllocaOonraOo(N2/N1) 1
G*PowerTutorial
2/9/19 SampleSize 17
SamanthaF.Anderson
Change AllocaOonraOo:1à2 Power:.8à.95 d:.5(medium)à.2(small) αerrorprob:.05à.005(BenjaminetalNaturearOcle)
G*PowerTutorial
2/9/19 SampleSize 18
SamanthaF.Anderson
B.S.FactorialANOVA:Interac%on
Testfamily: Ftest
StaOsOcaltest: ANOVA:fixedeffects,special,maineffects,andinteracOons
Typeofpoweranalysis: Apriori:Computerequiredsamplesize–givenα,powerandeffectsize
G*PowerTutorial
2/9/19 SampleSize 19
SamanthaF.Anderson
B.S.FactorialANOVA:Interac%on Effectsizef(S=.1,M=.25,L=.4)
SDofgroupmeans/errorSD(commonSDofgroupscores)
CanenterdirectlyOR CancalculatefromvarianceexplainedbyeffectanderrorvarianceOR
Cancalculatefromη2
αerrorprob Power(1-βerrorprob) Numeratordf Numberofgroups
#oftotalgroupsinthedesign
G*PowerTutorial
2/9/19 SampleSize 20
f =
σmσ e
η2 =
σ m2
σ total2
f = η
2
1−η2
SamanthaF.Anderson
B.S.FactorialANOVA:Interac%on
Effectsizef• .1
αerrorprob .05
Power(1-βerrorprob) 0.80
Numeratordf ThreewayinteracOon (a-1)*(b-1)*(c-1)=2
Numberofgroups• 3x2x2design=12groups
G*PowerTutorial
2/9/19 SampleSize 21
SamanthaF.Anderson
B.S.FactorialANOVA:Interac%on
Effectsizef• .1
αerrorprob .05
Power(1-βerrorprob) 0.80
Numeratordf ThreewayinteracOon (a-1)*(b-1)*(c-1)=2
Numberofgroups• 3x2x2design=12groups
G*PowerTutorial
2/9/19 SampleSize 22
SamanthaF.Anderson
MixedANOVA:W.S.MainEffect
Testfamily: Ftest
StaOsOcaltest: ANOVA:repeatedmeasures,withinfactors
Typeofpoweranalysis: Apriori:Computerequiredsamplesize–givenα,powerandeffectsize
G*PowerTutorial
2/9/19 SampleSize 23
SamanthaF.Anderson
MixedANOVA:W.S.MainEffect
Effectsizef αerrorprob Power(1-βerrorprob) Numberofgroups
#ofbetween-subjectsgroups
Numberofmeasurements #withinsubjectscondiOons/Omes
CorrelaOonamongrepeatedmeasuresNonsphericitycorrecOon(ε)
1/n-1<ε<1
G*PowerTutorial
2/9/19 SampleSize 24
• Can only accommodate 1 W.S. factor and 1 B. S. factor
SamanthaF.Anderson
MixedANOVA:W.S.MainEffect
Effectsizef:.3 αerrorprob:.05 Power(1-βerrorprob):.95 Numberofgroups:2 Numberofmeasurements:4 CorrelaOonamongrepeatedmeasures:.5NonsphericitycorrecOon(ε):1
G*PowerTutorial
2/9/19 SampleSize 25
SamanthaF.Anderson
MixedANOVA:W.S.MainEffect
Effectsizef:.3 αerrorprob:.05 Power(1-βerrorprob):.95 Numberofgroups:2 Numberofmeasurements:4 CorrelaOonamongrepeatedmeasures:.5NonsphericitycorrecOon(ε):1
G*PowerTutorial
2/9/19 SampleSize 26
SamanthaF.Anderson
EffectsizespecifiedisasampleesOmateofapopulaOonvalue
Ideaisto“guess”thelikelyeffectsizeinadvance UseinformaOonabouttheexpectedeffectfrompriorresearch
Setupyourstudytohave80%powertodetecttheeffectsizeyoubelievetheeffectis
Themoreaccurateyourguessis,themorelikelythesamplesizesuggestedbythepoweranalysiswillachieveyourdesiredlevelofpower
Typically–takesampleeffectsizefrompriorstudyatfacevalue
SampleSizePlanningviaPriorInformaOon
2/9/19 SampleSize 27
SamanthaF.Anderson
Goal:90%power Previousstudy
d=0.7 n=20/group
àEnterintoG*Power Tails:Two Effectsized:0.7 αerrorprob:.05 Power(1-βerrorprob):0.90 AllocaOonraOo(N2/N1):1
MoOvaOngExample
2/9/19 SampleSize 28
SamanthaF.Anderson
MoOvaOngExample
2/9/19 SampleSize 29
Suggested n = 44/group
SamanthaF.Anderson
MoOvaOngExample
2/9/19 SampleSize 30
Suggested n = 44/group
ACTUAL POWER = ~ 15%
SamanthaF.Anderson
PublicaOonbias JournalpreferenceforsignificantfindingsandselecOvereporOngofmulOpletests
ShiSsthecenteroftheconfidenceintervalsurroundingthesampleeffectsizeupward
TruncatesthedistribuOonofpossiblesampleeffectsizes,“censoring”valuesbelowacertainthreshold,regardlessofthetrueeffectsize
Uncertainty SampleeffectsizeisanesOmateofthepopulaOoneffectsize
Thereisalwaysgoingtobesamplingerror
PublicaOonBias&Uncertainty
SampleSize 312/9/19
SamanthaF.Anderson
ScienOficjournalsusuallydonotpublishresultsunlesstheyarestaOsOcallysignificantatp<.05
“…publicaOonbiasinpsychologyprimarilyinvolvesthesuppressionofnonsignificantresults.”[1]
“…prevalenceofp-valuesjustbelowthearbitrarycriterionforsignificancewasobservedinallthreejournals”[2]
PublicaOonBias
2/9/19 SampleSize 32
1.Simonsohn,Nelson,&Simmons.(2014).Perspec2vesonPsychologicalScience.2.Masicampo&Lalande.(2012).QuarterlyJournalofExperimentalPsychology.
SamanthaF.Anderson
HypotheOcalExample Independentt-test d=0.6 n=25δ=?
PublicaOonBias
SampleSize 332/9/19
SamanthaF.Anderson
HypotheOcalExample Independentt-test d=0.6 n=25δ=?
=0.16
PublicaOonBias
SampleSize 342/9/19
0
1
2
3
4
5
0.50 0.75 1.00 1.25 1.50Value of d
Prob
abilit
y D
ensi
ty
δ = 0.2
δ = 0.5
δ = 0.8
δ̂ ML
SamanthaF.Anderson
HypotheOcalExample Independentt-test d=0.6 n=25 =0.16
PublicaOonBiasandUncertainty
SampleSize 352/9/19
δ̂ ML
0.000
0.025
0.050
0.075
0.100
0.00 0.25 0.50 0.75 1.00Value of δ
Like
lihoo
d
n = 25
SamanthaF.Anderson
HypotheOcalExample Independentt-test d=0.6 n=25 =0.16
Independentt-test d=0.6 n=150 =0.6
PublicaOonBiasandUncertainty
SampleSize 362/9/19
δ̂ ML
δ̂ ML
0.000
0.025
0.050
0.075
0.100
0.00 0.25 0.50 0.75 1.00Value of δ
Like
lihoo
d
n = 25 n = 150
SamanthaF.Anderson
Perugini,Gallucci,&ConstanOni(2014)
Recommendedthelowerboundofthe90%confidenceintervalforδinpowercalculaOons
Adjustsforuncertainty
SafeguardPower
2/9/19 SampleSize 37
SamanthaF.Anderson
AdjustsforpublicaOonbias DistribuOonofdundertheinfluenceofpublicaOonbias(smallvaluesofdcensored)
MaximumlikelihoodesOmateofd:findthevalueofδthatmaximizesthethelog-likelihoodoftheaboveequaOon
NosoSwaretodothiseasilyL
Hedges’(1984)Method
2/9/19 SampleSize 38
h*(t|δ ,n) = h(t|δ ,n)
A(δ ,n,α )= PDF(t|δ ,n)Power(δ ,n,α )
SamanthaF.Anderson
AdjustsforuncertaintyandpublicaOonbias
FindvalueofδassociatedwithspecificCDFprobabiliOes:
Selectδcorrespondingto50thpercenOleàadjustforpublicaOonbiasonly
SelectmoreconservaOvevalueofδàadjustforuncertaintytoo
MuchlesscomplicatedtoperformthanitlooksJ
BUCSSApproach
2/9/19 SampleSize 39
LT[FO ;νnum,νden ,δ ] =
LF [FO |νnum,νden ,δ ]1 - CDFF [Fcrit (1 -αO)|νnum,νden ,δ ]
SamanthaF.Anderson
HowoSenamethodachievesitsintendedpowerinthelongrun
δ=0.5dorig=0.72
replicaOonn=32actualpower=.50
Assurance
2/9/19 SampleSize 40
0.0
0.2
0.4
0.6
0.8
1.0
0 25 50 75 100Study
Powe
r of P
lann
ed S
tudy
SamanthaF.Anderson
HowoSenamethodachievesitsintendedpowerinthelongrun
BUCSSPercenOle=1–desiredassurance
Assurance
2/9/19 SampleSize 41
0.0
0.2
0.4
0.6
0.8
1.0
0 25 50 75 100Study
Powe
r of P
lann
ed S
tudy
Prior Study d < .5 Prior Study d > .5
SamanthaF.Anderson
BUCSSconsistentlyachieveshigherpowerwhencomparedtousingthesampleeffectsizedirectly(“atfacevalue”)insamplesizeplanning
CansOllbeusedevenifthepriorstudydidnotreportaneffectsizemeasure
Onlyneedstoknowthe“t”or“F”staOsOc
Letsusersspecify 1.Desiredpower 2.Desiredassurance/adjustmentforuncertainty 3.AmountofadjustmentforpublicaOonbias
BenefitsofBUCSSApproach
2/9/19 SampleSize 42
SamanthaF.Anderson
EffectSizescanBeComplicated
2/9/19 SampleSize 43
Design:One-waywithinsubjects,2groups,N=25,ρ=.3,f2=.16(large) 1.G-Power("asinCohen1988”)
ncp=4.00,power=.48 2.G-Power:("asinGpower3.0”)
ncp=11.428,power=.90 3.Rpwr.f2.test:
pwr.f2.test(u=1,v=24,f2=.16,sig.level=.05,power=NULL) Power=.49
4.Powercalcula%ons,ncpfromformulancp=f2*(df.effect+df.error+1)=.16*(1+24+1)=4.16
1-pf(Fcrit,df1=1,df2=24,ncp=(.16*(1+24+1))) Power=.49
5.Powercalcula%ons,usingncpfrom#2 1-pf(FcritES,df1=1,df2=24,ncp=11.428) Power=.90
6.G-Powerdependentttest,withdz=2*f=.8,totalN=25,2tailsncp=4,power=.97
f 2 =
σm2
σ e2
SamanthaF.Anderson
Adjustedδ=zero Originalstudyhashighuncertaintyandbias
EffectsizeesOmateisuntrustworthyatthedesiredlevelofassurance
HighsensiOvityfordetecOngTypeIerror
SoluOon:Lowerassuranceand/orpublicaOonbiasadjustment
NoncentralityParameter=Zero
2/9/19 SampleSize 44
n=20/group
SamanthaF.Anderson
RPackage BUCSS AvailableonCRAN
WebApplicaOonsDesigningExperiments.com
SoSware
2/9/19 SampleSize 45
SamanthaF.Anderson
PreviousStudy N=30(n=15) t(28)=2.40,p=.01 d=0.88
BUCSSExample
2/9/19 SampleSize 46
Vohs&Schooler.(2008)PsychologicalScience.
SamanthaF.Anderson
PreviousStudy N=30(n=15) t(28)=2.40,p=.01 d=0.88
PlannedStudy 90%intendedpower N=58(n=29) t(56)=-0.77,p=.44
BUCSSExample
2/9/19 SampleSize 47
SamanthaF.Anderson
• BUCSSWebApp:Independentttest
BUCSSExample
2/9/19 SampleSize 48
SamanthaF.Anderson
BUCSSExample
2/9/19 SampleSize 49
nrep=977
SamanthaF.Anderson
RevisedPreviousStudy N=100(n=50) t(98)=4.40,p<.0001 d=0.88
BUCSSExample
2/9/19 SampleSize 50
nrep = 29
SamanthaF.Anderson
2x2B.S.ANOVA:Interac%on ObservedF-valuefromthepreviousstudyTotalsamplesizeofthepreviousstudy
NumberoflevelsofFactorA NumberoflevelsofFactorB Effectofinterest(A,BorinteracOon) alpha-levelofpreviousstudy
ThisisthepublicaOonbiasadjustment .05meansthepriorstudylikelywouldhavebeenrejectedifp>.05
alpha-levelofplannedstudy Assurance
Between.5(nouncertaintyadjustment)and.99(biguncertaintyadjustment) .80assurancemeansthat80%oftheOme,youwillachieveyourdesiredpower
StaOsOcalpower
BUCSSTutorial
2/9/19 SampleSize 51
SamanthaF.Anderson
2x2B.S.ANOVA:Interac%on ObservedF-value:7Totalsamplesize:120
NumberoflevelsofFactorA:2 NumberoflevelsofFactorB:2 Effectofinterest:InteracOon alpha-levelofpreviousstudy:1 alpha-levelofplannedstudy:.05 Assurance:0.80 StaOsOcalpower:0.80
BUCSSTutorial
2/9/19 SampleSize 52
BUCSSTutorial
2/9/19 SampleSize 53
*The output will always be “per-group” sample size for designs with between-subjects factors*
SamanthaF.Anderson
3x3W.S.ANOVA:Interac%on ObservedF-valuefromthepreviousstudyTotalsamplesizeofthepreviousstudy
NumberoflevelsofFactorA NumberoflevelsofFactorB Effectofinterest(A,BorinteracOon) alpha-levelofpreviousstudy
ThisisthepublicaOonbiasadjustment .05meansthepriorstudylikelywouldhavebeenrejectedifp>.05
alpha-levelofplannedstudy Assurance
Between.5(nouncertaintyadjustment)and.99(biguncertaintyadjustment) .80assurancemeansthat80%oftheOme,youwillachieveyourdesiredpower
StaOsOcalpower
BUCSSTutorial
2/9/19 SampleSize 54
SamanthaF.Anderson
3x3W.S.ANOVA:Interac%on ObservedF-value:7Totalsamplesize:60
NumberoflevelsofFactorA:3 NumberoflevelsofFactorB:3 Effectofinterest:InteracOon alpha-levelofpreviousstudy:.05 alpha-levelofplannedstudy:.05 Assurance:0.80 StaOsOcalpower:0.80
BUCSSTutorial
2/9/19 SampleSize 55
BUCSSTutorial
2/9/19 SampleSize 56
* Because all factors are within-subjects, suggested sample size is total N *
SamanthaF.Anderson
Mul%pleLinearRegressionDetecteffectofasinglepredictor,controllingforotherpredictors
Observedt-valuefromthepreviousstudyTotalsamplesizeofthepreviousstudy
Numberofpredictors alpha-levelofpreviousstudy
ThisisthepublicaOonbiasadjustment .05meansthepriorstudylikelywouldhavebeenrejectedifp>.05
alpha-levelofplannedstudy Assurance
Between.5(nouncertaintyadjustment)and.99(biguncertaintyadjustment) .80assurancemeansthat80%oftheOme,youwillachieveyourdesiredpower
StaOsOcalpower
BUCSSTutorial
2/9/19 SampleSize 57
SamanthaF.Anderson
Mul%pleLinearRegression Observedt-value:3Totalsamplesize:80
Numberofpredictors:4 alpha-levelofpreviousstudy:.05 alpha-levelofplannedstudy:.05 Assurance:0.80 StaOsOcalpower:0.80
BUCSSTutorial
2/9/19 SampleSize 58
BUCSSTutorial
2/9/19 SampleSize 59
* For regression, suggested sample size is total N *
SamanthaF.Anderson
“General”appsallowformorecomplicatedeffects/designsinANOVA
Plannedcomparisons• E.g.thepairwisedifferencebetweengroups1and3ina3-groupdesign
>2-factordesigns• E.g.,3x3x3design
Threeappsrelatedtolinearregression
TesOngsinglepredictor,setofpredictors,modelR2
BUCSSTutorial
2/9/19 SampleSize 60
Citation: Anderson, Kelley, & Maxwell. (2017).
Psychological Science.
SamanthaF.Anderson
IfyourstudyhasmulOpleeffects/hypotheses,besttoplansamplesizefor
Yourfocaleffect Thesmallesteffect
PlanaheadforpotenOalmissingdata PlanaheadwhencorrecOngformulOpletesOng
IfyouplantodoaBonferroniadjustment,yourplannedstudyalpha-levelwon’tbe.05
IssuesinSampleSizePlanning
2/9/19 SampleSize 61
SamanthaF.Anderson
Samplesizeplanningforprecision AccuracyinParameterEsOmaOon(AIPE) Planinsteadtohaveanarrowconfidenceinterval
• MBESSpackageinR SequenOalanalysis
Samplesizenotfixed Dataevaluatedastheycomeinatpre-specified“stoppingpoints” Approachesfromtodaycanbeusedforthe“full”N
Don’tforgetaboutdesign! Within-subjectsdesignsJ AdjusOngforcovariates(ANCOVA)J Stronger/moreintensemanipulaOonJ
AlternaOveApproaches
2/9/19 SampleSize 62
SamanthaF.Anderson
GeneralOverviewsLenth.(2001).AmericanSta2s2cian.
Maxwell,Kelley,&Rausch.(2008).AnnualReviewofPsychology.Perugini,Gallucci,&ConstanOni.(2018).Interna2onalReviewofSocialPsychology.
GeneralUncertaintyAdjustmentMcShane&Bockenholt.(2014).Perspec2vesonPsychologicalScience.Perugini,Gallucci,&ConstanOni.(2014).Perspec2vesonPsychologicalScience.
BUCSSApproach Anderson,Kelley,&Maxwell.(2017).PsychologicalScience. Anderson&Maxwell.(2017).Mul2variateBehavioralResearch.
Psychologists’intuiOonsonpower Bakkeretal.(2016).PsychologicalScience.
FurtherReading
2/9/19 SampleSize 63
SamanthaF.Anderson
Feelfreetocontactmeatsamantha.f.anderson@asu.edu
AnyquesOons?
Thankyou!
2/9/19 SampleSize 64
SamanthaF.Anderson
Anderson,S.F.,Kelley,K.,&Maxwell,S.E.(2017).Samplesizeplanningformoreaccuratesta%s%calpower:Amethodadjus%ngeffectsizesforpublica%onbiasanduncertainty.PsychologicalScience.
Anderson,S.F.,&Kelley,K.(2017).Bias-UncertaintyCorrectedSampleSize(BUCSS).[RPackage].Anderson,S.F.,&Maxwell,S.E.(2016).There’smorethanonewaytoconductareplica%onstudy:Beyondsta%s%calsignificance.PsychologicalMethods.Anderson,S.F.&Maxwell,S.E.(2017).Addressingthereplica%oncrisis:Usingoriginalstudiestodesignreplica%onstudieswithappropriatesta%s%calpower.MBR.
Bakker,etal.(2016).Researchers’intui%onsaboutpoweranalysisinpsychologicalresearch.PsychologicalScience.Brand,A.etal.(2008).Accuracyofeffectsizees%matesfrompublishedpsychologicalresearch.PerceptualandMotorSkills.Cohen,J.(1988).Sta;s;calpoweranalysisforthebehavioralsciences(2nded.).Gelman,A.,&Loken,E.(2013)Thegardenofforkingpaths:Whymul%plecomparisonscanbeaproblemevenwhenthereisno“fishingexpedi%on”or“phacking”andtheresearchhypothesiswaspositedaheadof%me.AmericanScien;st.
Ioannidis,J.P.A.(2005).Whymostpublishedresearchfindingsarefalse.PLoSMedicine.Lane,D.M.,&Dunlap,W.P.(1978).Es%ma%ngeffectsize:Biasresul%ngfromthesignificancecriterionineditorialdecisions.Bri;shJournalofMathema;calandSta;s;calPsychology.
Masicampo,E.J.,&Lalande,D.R.(2012).Apeculiarprevalenceofpvaluesjustbelow.05.QuarterlyJournalofExperimentalPsychology.Maxwell.,S.E.etal.(2008).Samplesizeplanningforsta%s%calpowerandaccuracyinparameteres%ma%on.AnnualReviewofPsychology.Maxwell,S.E.etal.(2015).Ispsychologysufferingfromareplica%oncrisis?Whatdoesfailuretoreplicatereallymean?AmericanPsychologist.McShane,B.B.,&Bockenholt,U.(2014).Youcannotstepintothesamerivertwice:Whenpoweranalysesareop%mis%c.Perspec;vesonPsychologicalScience.Muller,K.&Feqerman,B.(2002).RegressionandANOVA:AnIntegratedApproach.Olejnik,S.,&Algina,A.(2000).Measuresofeffectsizeforcompara%vestudies:Applica%ons,interpreta%ons,andlimita%ons.ContemporaryEduca;onalPsychology.
Pren%ce,D.A.,&Miller,D.T.(1992).Whensmalleffectsareimpressive.PsychologicalBulle;n.Sedlmeier,P.,&Gigerenzer,G.(1989).Dostudiesofsta%s%calpowerhaveaneffectonthepowerofstudies?PsychologicalBulle;n.Simmons,J.P.etal.(2011).False-posi%vepsychology:Undisclosedflexibilityindata-collec%onandanalysisallowspresen%nganythingassignificant.PsychologicalScience.
Simonsohn,U.,Nelson,L.D.,&Simmons,J.P.(2014).Pcurveandeffectsize:correc%ngforpublica%onbiasusingonlysignificantresults.Perspec;vesonPsychologicalScience.
Taylor,D.J.,&Muller,K.E.(1996).Biasinlinearmodelpowerandsamplesizecalcula%onduetoes%ma%ngnoncentrality.Communica;onsinSta;s;cs:TheoryandMethods.
Yuan,K.H..&Maxwell,S.E.(2005).Onthepost-hocpowerintes%ngmeandifferences.JournalofEduca;onalandBehavioralSta;s;cs.
SelectedReferences
2/9/19 SampleSize 65
SamanthaF.Anderson
Hedges(1984) Recentsurveyshadshown94–97%publishedresultsp<.05
JournaleditorscitedstaOsOcalsignificanceascriteriaforpublishing
FormedHedges’modelofpublicaOonbias
PublicaOonbiasismissingdataproblem
PublicaOonBias:ClassicWork
2/9/19 SampleSize 66
SamanthaF.Anderson
δ~Unif(0.1,1) n=20
• Differentperformanceforalldvalues
– δ<.3:LBunsuccessfulwhileTM=0
– δ>.3:TMmoreoSensuccessfulthanLB
n=250• d>.3:TM=LB• d<.3:methodsdiffer
– δ=.1–.4:LBunsuccessfulwhileTM=0
– δ>.4:TM=LB
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
0.639
0.8
1.0
1.2
1.4
0.25 0.50 0.75 1.00δ
Sam
ple
d
● UnsuccessfulSuccessful
a
●
●
●●
●
0.639
0.8
1.0
1.2
1.4
0.25 0.50 0.75 1.00δ
Sam
ple
d
● UnsuccessfulSuccessfulTM Estimate = 0
c
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
0.175
0.4
0.6
0.8
1.0
1.2
0.25 0.50 0.75 1.00δ
Sam
ple
d
● UnsuccessfulSuccessful
b
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●●
0.175
0.4
0.6
0.8
1.0
1.2
0.25 0.50 0.75 1.00δ
Sam
ple
d
● UnsuccessfulSuccessfulTM Estimate = 0
d
BUCSSvsSafeguardPower
2/9/19 SampleSize 67
SamanthaF.Anderson
PopulaOonEffectSizeFormulas
2/9/19 SampleSize 68
dz =µdiffσ diff
=µdiff
σ 12 +σ 2
2 − 2ρσ 1σ 2
d =µdiffσ
f 2 =(αβ )2∑ / abσε2
f 2 =(αβ )2∑ / (a −1)(b−1) +1
σε2
SamanthaF.Anderson
NoncentralityParameter
2/9/19 SampleSize 69
λ =(θ -θ0 ′) M
-1(θ -θ0)σ 2
M= C(GN)IG ′C = C ′Cn
M-1 =n(C ′C )-1
λ =(θ -θ0)'n(C ′C )
-1(θ -θ0)σ 2
λ =nΣ
λ:noncentralityparameterθ:vectorofmeansθ0:hypothesizedvalueC:contrastmatrixG:numberofgroupsN:totalsamplesizen:pergroupsamplesizeΣ:populaOoneffectsize
Muller&Fe@erman.(2002).SASIns2tute
SamanthaF.Anderson
Designs Dependentt: BetweensubjectsANOVA
InteracOoninsplitplotdesign
NoncentralityParameterExamples
2/9/19 SampleSize 70
λ =
n × (µ1 - µ2)2× (1 - ρ)σ
λ = f2 × (dfeffect +dferror +1)
λ =
n (αβ )2∑σ 2(1 - ρ)
SamanthaF.Anderson
UsingRelaOonshipBetweenλ&n
2/9/19 SampleSize 71
if (TM.5[i] > 0) { nrep