Towards Sustainable Insights

Tim Kraska <tim_kraska@brown.edu>

http://www.huffingtonpost.co.uk/2016/01/08/a-glass-of-red-wine-is-the-equivalent-to-an-hour-at-the-gym-says-new-study_n_7317240.html

ANewStudyshows:AGlassOfRedWineIsTheEquivalentToAnHourAtTheGym[FoxNews02/15andothers]

Anewstudyshows:Secrettowinninganobel prize?EatMoreChocolate[Time 10/12]

Anewstudyshows:Secrettowinninganobelprize?EatMoreChocolate[Time 10/12]

Scientistsfindthesecretoflongerlifeformen(Thebadnews:castrationisthekey) [DailyMailUK,09/12]

http://www.dailymail.co.uk/sciencetech/article-2207981/Scientists-secret-living-life-men-bad-news-Castration-key.html

There has been an explosion of (data-driven) discoveries, many of which being questionable.

Reasons are manifold, but…thedatabasecommunity

… andmanyothers

workshardontobenotleftout(again)

AnoteforReviewer2:Weactuallylikedyourcommentsandithelpedustosharpenourpoints.Ifyoufeelinanywayoffendedbythistalk,thiswasnotmyintentionandIammorethanhappytomakeituptoyouwithalotofwhisky.Justcometomeafterthetalkandsayweneedtodrink.Knowingthiscrowd,enoughpeoplewilldoitandIwillevenneverfindoutyouridentityifyoudonotwishso.

Let me introduce (virtual) Reviewer 2:

Thepaper'sshortcomingsareinitsmotivation,solution,andpresentation.

ThepartofthepaperthatIdidlikewastheexamplesgiveninSec

2.2.2.

OutlinePart I: The problem with:

A. Interactive Data Exploration

B. Visualization Recommendation Systems

C. Hypothesis Generator

A. Part II: Solutions

A) Interactive Data Exploration Tools (Vizdom as an Example)

Why Visualizations contribute to the problemIf a visualization provides any insight, it is an hypothesis test (just one where you not necessarily know if it is statistical significant)

Otherwise, visualizations have just to be taken as pretty pictures about (potentially) random facts

gender

coun

t

Male Female Other

A

salary over 50k

coun

t

True Falsegender

coun

t

Male Female Other

gender

coun

tMale Other

salary over 50k

coun

t

True False

gender

coun

t

Male Female Other

B C

education

coun

t

HS Bachelor Master PhD

marital status

coun

t

Married NeverMarried

NotMarried

Widowed

Female

salary over 50k

coun

t

True False

education

coun

t

HS Bachelor Master PhD

marital status

coun

t

Married NeverMarried

NotMarried

Widowed

age

coun

t

10 20 30 50 60 7040 9080

ageco

unt

10 20 30 50 60 7040 9080

0.011p

t-test

D

E F

salary over 50k

coun

t

True False

education

coun

t

HS Bachelor Master PhD

marital status

coun

t

Married NeverMarried

NotMarried

Widowed

gender

coun

t

Male Female Other

A

salary over 50k

coun

t

True Falsegender

coun

t

Male Female Other

gender

coun

t

Male Other

salary over 50k

coun

t

True False

gender

coun

t

Male Female Other

B C

education

coun

t

HS Bachelor Master PhD

marital status

coun

t

Married NeverMarried

NotMarried

Widowed

Female

salary over 50k

coun

t

True False

education

coun

t

HS Bachelor Master PhD

marital status

coun

t

Married NeverMarried

NotMarried

Widowed

age

coun

t

10 20 30 50 60 7040 9080

age

coun

t

10 20 30 50 60 7040 9080

0.011p

t-test

D

E F

salary over 50k

coun

t

True False

education

coun

t

HS Bachelor Master PhD

marital status

coun

t

Married NeverMarried

NotMarried

Widowed

If visualizations are used to find something interesting, the user is doing multiple hypothesis testing

Running Example: Survey on Amazon Mechanical Turk

Our goal: To find good indicators (correlations) that somebody knows who Mike Stonebraker is.

And after searching for a bit, one of my favorites

Pearsoncorrelationsignificance-levelp<0.05

But Why Does the DB community make the situation worse?

So What Did Reviewer 2 say?Blamingthemultiple-comparisonproblemonfastvisualization-generationislikeblamingfastcarsforchilddrivercasualties

duetocaraccidents…

But…

2) Visual Recommendation Systems (SeeDB as an Example)

0

0.2

0.4

0.6

0.8

1

V1 V2

Normalize

dAg

gr(CollumnA)

CollumnB(filteredColumnC=V?)

Target

0

0.2

0.4

0.6

0.8

1

V1 V2

Normalize

dAg

gr(CollumnA)

CollumnB(filteredColumnC=V?)

Reference

0

0.2

0.4

0.6

0.8

1

V1 V2

Normalize

dAg

gr(CollumnA)

CollumnB(filteredColumnD=V?)

Target

Uninteresting

Interesting

What is different

The system automatically generates thousands of visualizations and ranks them somehow (e.g., based effect size)

SeeDB on Our Survey Data

Startup CorporationFilter: All

0

0.2

0.4

0.6

0.8

% C

hedd

ar &

Sou

r Cre

am Potato Chips vs Workspace Preference

Startup CorporationFilter: Belief in Alien Existence

0

0.5

1

% C

hedd

ar &

Sou

r Cre

am Potato Chips vs Workspace Preference

Startup CorporationFilter: Disbelief in Alien Existence

0

0.2

0.4

0.6

0.8

% C

hedd

ar &

Sou

r Cre

am Potato Chips vs Workspace Preference

Startup CorporationFilter: Prefer Blow Hair Drying

0

0.1

0.2

0.3

0.4

% C

hedd

ar &

Sou

r Cre

am Potato Chips vs Workspace Preference

…Ididlike[…]theexample…

What is the Problem?

The user is in the dark what the system did. The system might have “tested” thousands of potential visualization, just to find something interesting.

What did Reviewer 2 say?

Thesesystemsarenotdesignedforanaveragepersontorunandgetinsightsthattheycanpublishmedicalarticleson!The

endusersarestillanalysts.TheonlydifferenceisthattheyautomatehypothesesgenerationandNOThypotheses

testing,…

WARNINGAfter using the tool,throwaway thedata.

It is not safe!1

My suggestions, papers should include in the future a a warning like

1To be more precise: you do not have to throw it all away, but you can not use the same data anymore for significance testing

3) Real Hypothesis Generators(Data Polygamy as an Example)

(Data) Polygamy is bad, especially if you do not know what is going on.

OutlinePart I: The problem with:

A. Interactive Data Exploration

B. Visualization Recommendation Systems

C. Hypothesis Generator

A. Part II: Solutions

Should we stop working on IDE, Recommenders, etc?

• Actively inform the user about the risk factors

• Try your techniques over random data with different data sizes

• If possible, split data into a exploration and a validation set. • Be aware, significantly lowers the power • Everything on the validation data set has to be carefully handled (i.e., use

multi-hypothesis control)

• If possible, use additional experiments (e.g., A/B testing)• Requires a small number of hypothesis and careful design• Might not always be possible or is very expensive

Better: control the multi-hypothesis problem from the start

NO

QUD

EQuantifyingtheUn

certaintyin

DataExploratio

n

Python

BigDAWG

IDEAInteractiveDataExploration

Accelerator

LegacySystems

Mlbase2

With

hypothesis

control

OurInteractiveDataExplorationStack(BIDES)

Many Interesting Open Problems

• Transparent hypothesis testing how to automatically derive what the hypothesis is the user is testing

• How to convey the meaning to the user(e.g., FDR vs family-wise error)

• Safe recommender techniques(we are currently exploring new techniques based VC-dimensions to control the error)

• Incremental multiple-hypothesis control techniques (for example, see ”Controlling False Discoveries During Interactive Data Exploration” CoRR abs/1612.01040 how we use new alpha-investing policies to do that)

• Dependencies between hypothesis (this can safe ”hypothesis budget”)

• …

Wearejustatthebeginning

A Final Note from Reviewer 2 onIs the Situation really so Bad?

..,thesystemsthatarecriticizedbythispaperareessentiallythree

tools[4,6,28]… Sotheproblemisnotreallyasseriousasitmightseemasnoneofthesesystemsareusedbyanyoneinpractice

Tim Kraska <tim_kraska@brown.edu>

Specialthanksto:

AlastnotetoReviewer2:1st Isincerelyhopeyouarenotoneofmyletterwritersformytenurecase:)2nd Yourcommentsactuallyhelpedustoimprovethepaperandhelpedwiththetalk.Sothankyou!3rd Iamhappytopayforyourdrinkstonighttomakeituptoyou.