Research design

IMD09120: Collaborative Media

Brian Davison 2010/11

• Recap• Variables• Types of design• Hypotheses• Sampling• Your designs

Research design

Recap: tests you have done (1)

• Can the number of social networking accounts predict the number of contacts a person has?

• Do female students have significantly more social networking contacts than male students?

• Does face-to-face differ significantly from WebCT chat as a channel of communication in terms of satisfaction?

• Do students from different schools differ significantly in the number of social networking contacts they have?

Recap: tests you have done (2)

• Do students differ significantly by age in the number of social networking contacts they have?

• Does the quality of video and/or group size have a significant effect on satisfaction in mediated group communication?

• Is the preference for Facebook statistically significant?

• Is preferred social networking site related to gender?

Recap: tests you have used

• Correlation• Independent t-test• Paired t-test• ANOVA

• 2

Variables

• Things you can measure– Height, weight, age– Number of car journeys– Gender, nationality

– Attitude, belief, satisfaction

Types of variable• Continuous

– Infinitely divisible

• Discrete– Values come in defined

ordered steps

• Categorical– Values are categories with no

ordering

• Examples– Height– Number of car journeys– Gender– Nationality– Observed symptoms of disease– Attitude– Age– Belief– Weight– Satisfaction

Dependence• Independent variable

– Manipulated by the investigator

• Dependent variable– Measured for effect

• Examples– Does face-to-face differ

significantly from WebCT chat as a channel of communication in terms of satisfaction?

– Is preferred social networking site related to gender?

Confounding variable

• A factor that affects the dependent variable and which is not included in the design

• What confounding variables might there be in the F2F-chat experiment?

Research designs

• Correlational – Looking at how one variable changes in relation to another– eg smoking and incidence of cancer

• Experimental and quasi-experimental– Looking for effects of one variable on another– IV and DV

Correlation designs

• Collect data about both variables• Measure the extent to which their distributions match• Causation cannot be inferred from a correlation

• Which is correlational:

1. Can the number of social networking accounts predict the number of contacts a person has?

2. Do female students have significantly more social networking contacts than male students?

3. Does face-to-face differ significantly from WebCT chat as a channel of communication in terms of satisfaction?

Experimental designs

• Researcher manipulates IV• Subjects are randomly assigned to test conditions

• Which is experimental:

1. Can the number of social networking accounts predict the number of contacts a person has?

2. Do female students have significantly more social networking contacts than male students?

3. Does face-to-face differ significantly from WebCT chat as a channel of communication in terms of satisfaction?

Quasi-experimental designs

• Some variables cannot be manipulated– eg gender, occupation– Therefore random allocation to conditions is impossible

• Which is quasi-experimental:

1. Do students differ significantly by age in the number of social networking contacts they have?

2. Does the quality of video and/or group size have a significant effect on satisfaction in mediated group communication?

3. Is the preference for Facebook statistically significant?

Between or within subjects

• Between-subjects design– Different subjects randomly assigned to different conditions

• Within-subjects design– Same subjects used in all conditions

Within-subjects

• Also called related or repeated measures

• Pro:– Using same subjects helps to eliminate confounding variables– Fewer participants needed

• Con:– Subjects may perform differently on second test – ordering effects– Subjects more likely to guess the purpose of the study – demand effects– Not possible for quasi-experimental studies

• Ordering effects can be eliminated by counterbalancing

Between-subjects

• Also called independent or unrelated• May be significant variation within each group

• Pro:– No ordering effects– Demand effects are less likely

• Con:– More participants required– Less control over confounding variables

What kind of design?

• Do female students have significantly more social networking contacts than male students?

• Does the quality of video and/or group size have a significant effect on satisfaction in mediated group communication?

• Does face-to-face differ significantly from WebCT chat as a channel of communication in terms of satisfaction?

Population

Pop. mean

Sampling error

Pop. mean

SamplePopulation

Spl. mean

What rule connects this series of numbers?

10, 20, 30

• Confirmation bias

Testing hypotheses

• Null hypothesis– There is no relationship / difference between two variables– ie. All observed differences are the result of sampling error– Probability values in tests = probability of the observed result if the null

hypothesis is true

• Alternative hypothesis– There is a relationship / difference between two variables– Specific to the study you are doing

Logic of hypothesis testing

1. Formulate a hypothesis

2. Measure the variables and examine their relationship (using samples)

3. Calculate the probability of the result if the null hypothesis is true

4. If the calculated probability is small enough, reject the null hypothesis

Two types of test

Is x significantly larger / smaller than expected?

Is x significantly different from expected?

How many tails?

• Do female students have significantly more social networking contacts than male students?

• Does face-to-face differ significantly from webCT chat as a channel of communication in terms of satisfaction?

• Do students from different schools differ significantly in the number of social networking contacts they have?

Short break

Effect size

Small effect Large effect

• A large effect size is easier to detect than a small one• The power of a test is a measure of its ability to detect an effect• More participants are required to detect a small effect size

Cohen’s power primer

• Tabulates required sample size according to– Confidence interval– Statistical test– Effect size

• Values for 95% confidence

• Cohen (1992)

Test Small Medium Large

Pearson’s r 783 85 28

T-test 393 64 26

ANOVA sample sizes for 95% confidence

Groups Small Medium Large

2 393 64 26

3 322 52 21

4 274 45 18

5 240 39 16

6 215 35 14

7 195 32 13

2 sample sizes for 95% confidence

df Small Medium Large

1 785 87 26

2 964 107 39

3 1,090 121 44

4 1,194 133 48

5 1,293 143 51

6 1,362 151 54

• More precise figures can be calculated using G*Power

How many variables do you have?

Are you looking for differences between conditions, or

relationships among variables?

Are you looking for differences between conditions, or

relationships among variables?

Do you have a between-participants or a within-

participants design?

Do you want a regression equation,

or simply the strength of a relationship?

Are you interested in a regression equation, or exploring clusters

of correlations?

Do you have more than one DV?

Do you have more than one IV?

Do you want to look for differences between conditions while controlling for the effects

of another variable?Independent

t-testRelated

t-testPearson’s

Product Moment Correlation Coefficient

Linear regression

Multiple regression

Factor analysis

One-way ANOVA ANOVA for multiple IVs

MANOVA

Analysis of covariance

Two More than two

Differences Relationships DifferencesRelationships

Between Within Strength Regression Regression Correlation

Yes

Yes

Yes

No

No

NoDancey & Reidy (2007) Statistics without maths for psychologists. Prentice Hall (p. 157)

How many variables do you have?

Are you looking for differences between conditions, or

relationships among variables?

Are you looking for differences between conditions, or

relationships among variables?

Do you have a between-participants or a within-

participants design?

Do you want a regression equation,

or simply the strength of a relationship?

Are you interested in a regression equation, or exploring clusters

of correlations?

Do you have more than one DV?

Do you have more than one IV?

Do you want to look for differences between conditions while controlling for the effects

of another variable?Independent

t-testRelated

t-testPearson’s

Product Moment Correlation Coefficient

Linear regression

Multiple regression

Factor analysis

One-way ANOVA ANOVA for multiple IVs

MANOVA

Analysis of covariance

Two More than two

Differences Relationships DifferencesRelationships

Between Within Strength Regression Regression Correlation

Yes

Yes

Yes

No

No

NoDancey & Reidy (2007) Statistics without maths for psychologists. Prentice Hall (p. 157)

Your turn

• Design a study to test whether caffeine affects level of anxiety.

Your turn again

• Design a study to test whether mathematical ability is related to musical ability

One more time

• Design a study to test whether there is a relationship between doing sport and intention to vote

Coursework: variables and hypotheses

• What are you trying to improve?– eg group identity, social presence, group effectiveness– This is your DV

• What are you changing?– eg additional features, different layout, modified presentation– Your IV is WebCT version: may be old/new, or +/- combined factors

• What do you expect?– Eg adding feature X leads to reduced social loafing– This is your hypothesis– Null hypothesis is “there is no difference”

Coursework: research design

• Levels of IV

• How to measure DV

• Experimental groups– Within subjects or between subjects

• Eliminating confounding variables and unwanted effects– Quality of system – create an “as-is” Powerpoint– Ordering effects– Demand effects– Researcher bias

Statistical test

• Method 1– What question are you asking?– Which practical example matches?

• Method 2– Follow the flowchart

• Use one method to check the other