+ All Categories
Home > Science > Selective Correlations- Not Voodoo

Selective Correlations- Not Voodoo

Date post: 23-Aug-2014
Category:
Upload: jonathan-rosenblatt
View: 25 times
Download: 5 times
Share this document with a friend
Description:
On a solution to selective estimation using post selection confidence intervals.
Popular Tags:
29
Selective Correlations- Not Voodoo Malach Lab 8.5.2014 Jonathan Rosenblatt- WIS
Transcript
Page 1: Selective Correlations- Not Voodoo

Selective Correlations- Not Voodoo

Malach Lab 8.5.2014

Jonathan Rosenblatt- WIS

Page 2: Selective Correlations- Not Voodoo

Acknowledgments

• Part of PhD dissertation under guidance of Prof. Yoav Benjamini

• Prof. Russ Poldrack

• Ms. Neomi Singer, Mr. Omri Perez, Prof. Talma Hendler

Page 3: Selective Correlations- Not Voodoo

Outline

• Tom et al., “The Neural Basis of Loss Aversion in Decision-Making Under Risk.”Science 315 (January 26, 2007)

• Selection Bias- Problem & Remedy.

• Revisiting Tom et al.

• Discussion.

Page 4: Selective Correlations- Not Voodoo

Tom et al. (2007)

Page 5: Selective Correlations- Not Voodoo

Reported Correlations

Page 6: Selective Correlations- Not Voodoo

Enter Vul

• Vul, Edward, Christine Harris, Piotr Winkielman, and Harold Pashler. “Puzzlingly High Correlations in fMRI Studies of Emotion, Personality, and Social Cognition.” Perspectives on Psychological Science 4, no. 3 (May 1, 2009): 274–290.

Page 7: Selective Correlations- Not Voodoo

And The People Rejoice• Diener, Ed. “Editor’s Introduction to Vul et Al. (2009) and Comments.” Perspectives on Psychological Science 4, no. 3 (May 1, 2009): 272–273.

• Fiedler, Klaus. “Voodoo Correlations Are Everywhere—Not Only in Neuroscience.” Perspectives on Psychological Science 6, no. 2 (March 1, 2011)

• Jabbi et al. “Response to ‘Voodoo Correlations in Social Neuroscience’ by Vul et Al.–summary Information for the Press.” Accessed July 30, 2013.

• Kriegeskorte et al. “Everything You Never Wanted to Know about Circular Analysis, but Were Afraid to Ask.” Journal of Cerebral Blood Flow & Metabolism 30, no. 9 (September 2010): 1551–1557.

• Lazar, Nicole A. “Discussion of ‘Puzzlingly High Correlations in fMRI Studies of Emotion, Personality, and Social Cognition’ by Vul et Al. (2009).” Perspectives on Psychological Science 4, no. 3 (May 1, 2009): 308–309.

• Lieberman, Matthew D., Elliot T. Berkman, and Tor D. Wager. “Correlations in Social Neuroscience Aren’t Voodoo: Commentary on Vul et Al. (2009).” Perspectives on Psychological Science 4, no. 3 (May 1, 2009): 299–307. doi:10.1111/j.1745-6924.2009.01128.x.

• Lindquist, Martin A., and Andrew Gelman. “Correlations and Multiple Comparisons in Functional Imaging: A Statistical Perspective (Commentary on Vul et Al., 2009).” Perspectives on Psychological Science 4, no. 3 (May 1, 2009): 310–313. doi:10.1111/j.1745-6924.2009.01130.x.

• Nichols, Thomas E., and Jean-Baptist Poline. “Commentary on Vul et Al.’s (2009) ‘Puzzlingly High Correlations in fMRI Studies of Emotion, Personality, and Social Cognition.’” Perspectives on Psychological Science 4, no. 3 (May 1, 2009): 291–293. doi:10.1111/j.1745-6924.2009.01126.x.

• Poldrack, Russell A., and Jeanette A. Mumford. “Independence in ROI Analysis: Where Is the Voodoo?” Social Cognitive and Affective Neuroscience 4, no. 2 (June 1, 2009): 208–213. doi:10.1093/scan/nsp011.

• Vul, Edward, Christine Harris, Piotr Winkielman, and Harold Pashler. “Puzzlingly High Correlations in fMRI Studies of Emotion, Personality, and Social Cognition.” Perspectives on Psychological Science 4, no. 3 (May 1, 2009): 274–290. doi:10.1111/j.1745-6924.2009.01125.x.

• “Reply to Comments on ‘Puzzlingly High Correlations in fMRI Studies of Emotion, Personality, and Social Cognition.’” Perspectives on Psychological Science 4, no. 3 (May 1, 2009): 319–324. doi:10.1111/j.1745-6924.2009.01132.x.

• Vul, Edward, and Nancy Kanwisher. “Begging the Question: The Non-Independence Error in fMRI Data Analysis.” Foundational Issues for Human Brain Mapping (2010): 71–91.

• Yarkoni, Tal. “Big Correlations in Little Studies: Inflated fMRI Correlations Reflect Low Statistical Power—Commentary on Vul et Al. (2009).” Perspectives on Psychological Science 4, no. 3 (May 1, 2009): 294–298. doi:10.1111/j.1745-6924.2009.01127.x.

Page 8: Selective Correlations- Not Voodoo

The Usual Suspects

• Multiplicity control

• Small samples => underpowered

• Reporting standards

Page 9: Selective Correlations- Not Voodoo

n.subjects: 12 n.subjects: 29 n.subjects: 47 n.subjects: 64 n.subjects: 82 n.subjects: 100

●●

● ● ● ● ●

● ● ● ● ● ● ●

● ● ● ● ● ● ●

● ● ● ●●

● ● ● ● ●●

●● ● ● ● ●

● ●● ●

● ● ●●

● ● ● ● ●●

● ●●

● ●● ●

● ● ● ●●

●● ●

● ●●

● ● ● ●●

●●

●●

●●

●●

● ● ●●

0.4

0.6

0.8

1.0

0.4

0.6

0.8

1.0

0.4

0.6

0.8

1.0

resels: 5000resels: 10000

resels: 3e+05

0.000.250.500.751.000.000.250.500.751.000.000.250.500.751.000.000.250.500.751.000.000.250.500.751.000.000.250.500.751.00True Correlation

Mea

n of

Sel

ecte

d C

orre

latio

ns

Selection Bias

Page 10: Selective Correlations- Not Voodoo

Cureton, Edward (1950)

• When a validity coefficient is computed from the same data used in making an item analysis, this coefficient cannot be interpreted uncritically. And, contrary to many statements in the literature, it cannot be interpreted “with caution” either. There is one clear interpretation for all such validity coefficients. This interpretation is– “Baloney”

Page 11: Selective Correlations- Not Voodoo

Selective Estimation

• A.k.a. “circular inference”, “double dipping”, “voodoo correlations”,...

• Estimation with quality guarantees following a parameter selection stage.

• Relation to selective testing.

Page 12: Selective Correlations- Not Voodoo

Ingredients of Estimation

• Point/interval?

• Error criterion?

• Algorithms with uniform error bounds?

Page 13: Selective Correlations- Not Voodoo

False Coverage statement Rate

• An error criterion for selective interval estimation.

• R= selected parameters

• V= false coverage statements

• FCP= V/R and 0 if none are selected.

• FCR=E(FCP)

Page 14: Selective Correlations- Not Voodoo

FCR of Nominal CIs

• Parameters selected: R=3 (big dots)

• Parameters not covered: V=2 (grey bars)

• FCP=V/R= 0.66

• Desired FCR= 1-confidence level= 0.1

• Nominal CIs do not control the FCR.

● ●

●● ● ●

● ● ●●

● ● ● ●

●●

● ●

1 2 3 4 5 6 7 8 9 10 12 14 16 18 20

0

● ● ●● ● ●

Page 15: Selective Correlations- Not Voodoo

FCR Adjusted CIs

• Benjamini, Yekutieli (2005)

• Motivation: conservative nominal CIs.

• In practice:

Page 16: Selective Correlations- Not Voodoo

Conditional CIs

• Weinstein, Fithian, Benjamini (2013)

• Motivation: invert acceptance region of conditional distribution.

• In practice: selectiveCI R package.

Page 17: Selective Correlations- Not Voodoo

(non)Uniqueness of Acceptance Region

• Conditional Shortest Length (CSR): Short interval, but indifferent to sign ambiguity.

• Conditional Modified Pratt (CMP): Best sign determination, while no larger than r times the CSR.

• Conditional Quasi Conventional (CQC): Shortest interval with penalty for sign flip

Page 18: Selective Correlations- Not Voodoo

Properties

Page 19: Selective Correlations- Not Voodoo

Remarks

• “Simple”: varying the value of a selected i’th estimator in its selectable range, does not change R.

• FCR adjusted CIs are B-H selection duals. Duality does not hold in general.

• Some selection rules are very hard to condition on.

Page 20: Selective Correlations- Not Voodoo

Applying FCR Adjusted CIs

“Confidence Calibration Plot”

Page 21: Selective Correlations- Not Voodoo

Applying CQC CIs

“Confidence Calibration Plot”

Page 22: Selective Correlations- Not Voodoo

Conditional or FCR Adjusted?

Page 23: Selective Correlations- Not Voodoo

“If the functional contrast is demonstrably independent of the

effects to be estimated for the selected data, then the same data may be used for effect estimation. Otherwise, independent data are

required to render the effect estimate independent.” Kriegeskorte (2013)

Page 24: Selective Correlations- Not Voodoo

●●●●●●●●

● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●0.00

0.25

0.50

0.75

1.00

0.00 0.25 0.50 0.75 1.00FullAndSplit

SplitAndFull2

Split.count●

010000200003000040000

Spatial Agreement of a Half Split

Page 25: Selective Correlations- Not Voodoo

CI Agreement with Half Split

Page 26: Selective Correlations- Not Voodoo

CI Agreement with Half Split

• In less than 0.1% of voxels, splitting will provide strong sign determination and CQC will not.

• In more than 50% of possible splits, 1/3 of jointly selected voxels will have opposing strong sign determination.

• Conclusion: CQC has higher probability of catching the right effect sign.

Page 27: Selective Correlations- Not Voodoo

Summary

• Selective estimation in social-neuroscience: Acknowledged but untreated.

• FCR controlling CIs as a general remedy.

• Better than splitting small samples.

Page 28: Selective Correlations- Not Voodoo

Open Problems

• Conditioning

• Inference on aggregates

• Other Error measures


Recommended