8/18/2019 Experiments(1)

http://slidepdf.com/reader/full/experiments1 1/22

Nicole Ronald

Experimentation

CRICOS Provider: 00111D | TOID: 3059

8/18/2019 Experiments(1)

http://slidepdf.com/reader/full/experiments1 2/22

Swinburne

SCIENCE | TECHNOLOGY | INNOVATION | BUSINESS | DESIGN

Experimentation

- Understanding of experimental research design inIT/IS/CS settings

- A brief refresher of research questions

- Different types of human experiments

- An overview of CS experiments

2

Today’s aim

8/18/2019 Experiments(1)

http://slidepdf.com/reader/full/experiments1 3/22

Swinburne

SCIENCE | TECHNOLOGY | INNOVATION | BUSINESS | DESIGN

Experimentation

- Surveys are useful when you need to understand what iscurrently happening (or could happen)

- Experiments change something and then measure

(Gray distinguishes between descriptive and analyticalsurveys)

3

How do experiments differ to surveys?

8/18/2019 Experiments(1)

http://slidepdf.com/reader/full/experiments1 4/22

Swinburne

SCIENCE | TECHNOLOGY | INNOVATION | BUSINESS | DESIGN

Experimentation

4

Process

Identify issue

Reviewliterature/theories

Develop

hypotheses

Identifyin/dependent

values

Conduct study

Analyse Accept or rejecthypotheses

Report

8/18/2019 Experiments(1)

http://slidepdf.com/reader/full/experiments1 5/22

Swinburne

SCIENCE | TECHNOLOGY | INNOVATION | BUSINESS | DESIGN

Experimentation

• “What is happening”

Descriptive

• “What is happening compared to what should happen”

Normative

• “What is the relationship/strength between X and Y”

Correlative

• “What impact does a change in X have on Y”

Impact

5

Question types

8/18/2019 Experiments(1)

http://slidepdf.com/reader/full/experiments1 6/22

Swinburne

SCIENCE | TECHNOLOGY | INNOVATION | BUSINESS | DESIGN

Experimentation

- Testing algorithms, e.g., a new searching algorithm

- Testing ideas in a simulated world, e.g., giving someentities different information

- Testing parameters, e.g., for an algorithm, in a simulation

- A lab situation is usually an experiment

6

When would experimentation be used?

8/18/2019 Experiments(1)

http://slidepdf.com/reader/full/experiments1 7/22

Swinburne

SCIENCE | TECHNOLOGY | INNOVATION | BUSINESS | DESIGN

Experimentation

- Dependent variable- Outcome

- Independent variable- Treatment

7

Variables

8/18/2019 Experiments(1)

http://slidepdf.com/reader/full/experiments1 8/22

Swinburne

SCIENCE | TECHNOLOGY | INNOVATION | BUSINESS | DESIGN

Experimentation

- Experimental- Randomise allocation to groups- Variables can be manipulated/control group

- Quasi-experimental- Group membership cannot be randomised, or is pre-

existing- Variables can be manipulated/control group

- Non-experimental- Group membership cannot be randomised, or is pre-

existing

- Variables cannot be manipulated 8

Experimental design

8/18/2019 Experiments(1)

http://slidepdf.com/reader/full/experiments1 9/22

Swinburne

SCIENCE | TECHNOLOGY | INNOVATION | BUSINESS | DESIGN

Experimentation

- Two groups, randomised:- One receives treatment, the other not (control)

- Both are evaluated before and after treatment

- Example: time-to-completion of IT troubleshooting tickets- Need to check that times in both groups are similar; also

control for experience, age etc.

- Experimental group receives training, control does not9

Experimental group with control

Pre-test (t1) Treatment (t2) Post-test (t3)Experimental Yes Yes Yes

Control Yes No Yes

8/18/2019 Experiments(1)

http://slidepdf.com/reader/full/experiments1 10/22

Swinburne

SCIENCE | TECHNOLOGY | INNOVATION | BUSINESS | DESIGN

Experimentation

- Two groups, randomised:- One receives treatment, the other not (control)- Only one group measured beforehand

- Avoid influencing outcome by pretesting

10

Four-group design

Pre-test (t1) Treatment (t2) Post -test (t3)

Experimental Yes Yes Yes

Experimental No Yes Yes

Control Yes No YesControl No No Yes

8/18/2019 Experiments(1)

http://slidepdf.com/reader/full/experiments1 11/22

Swinburne

SCIENCE | TECHNOLOGY | INNOVATION | BUSINESS | DESIGN

Experimentation

- Two groups:- Could be pre-determined, by class/department- One receives treatment, the other not (control)

- Both groups measured beforehand, hopefully equal

11

Quasi-experimental with control

Pre-test (t1) Treatment (t2) Post-test (t3)

Experimental Yes Yes Yes

Control Yes No Yes

8/18/2019 Experiments(1)

http://slidepdf.com/reader/full/experiments1 12/22

Swinburne

SCIENCE | TECHNOLOGY | INNOVATION | BUSINESS | DESIGN

Experimentation

- One group, observe afterwards

- Either unable to observe beforehand (e.g., unexpecteddisaster) or just didn’t

- Example: student evaluations

- Can also use a control group

12

Non-experimental

Treatment (t1) Post-test (t2)Experimental Yes Yes

8/18/2019 Experiments(1)

http://slidepdf.com/reader/full/experiments1 13/22

Swinburne

SCIENCE | TECHNOLOGY | INNOVATION | BUSINESS | DESIGN

Experimentation

- Studies over time

- Cross-sectional: same measurement over time

- Longitudinal: same subjects and measurements over time

13

Longitudinal and cross-sectional studies

8/18/2019 Experiments(1)

http://slidepdf.com/reader/full/experiments1 14/22

Swinburne

SCIENCE | TECHNOLOGY | INNOVATION | BUSINESS | DESIGN

Experimentation

- Internal- Assume only independent variable influences

dependent variable

- Threats: selection, external maturation/events, dropout,pretesting, sharing info

- External- Effectiveness of generalising

- Threats: people, places, time

14

Validity

8/18/2019 Experiments(1)

http://slidepdf.com/reader/full/experiments1 15/22

Swinburne

SCIENCE | TECHNOLOGY | INNOVATION | BUSINESS | DESIGN

Experimentation

- Similar principles:- Control and experimental algorithms- Different treatments or scenarios

- Fewer ethical issues (if not accessing personal data)

15

Algorithmic experiments

8/18/2019 Experiments(1)

http://slidepdf.com/reader/full/experiments1 16/22

Swinburne

SCIENCE | TECHNOLOGY | INNOVATION | BUSINESS | DESIGN

Experimentation

- Representation of the world

- Can simulate many objects

- Useful when real-world experiments impractical

16

Simulation

8/18/2019 Experiments(1)

http://slidepdf.com/reader/full/experiments1 17/22

Swinburne

SCIENCE | TECHNOLOGY | INNOVATION | BUSINESS | DESIGN

Experimentation

- Base case (if using simulation then verify against realworld)

- Altered case

17

Experimental design in IT/IS/CS

8/18/2019 Experiments(1)

http://slidepdf.com/reader/full/experiments1 18/22

Swinburne

SCIENCE | TECHNOLOGY | INNOVATION | BUSINESS | DESIGN

Experimentation

- Repeatable- Get same results with same code/data

- Reproducible- Get same results with method

- Stochastic vs. deterministic

18

Replication

8/18/2019 Experiments(1)

http://slidepdf.com/reader/full/experiments1 19/22

Swinburne

SCIENCE | TECHNOLOGY | INNOVATION | BUSINESS | DESIGN

Experimentation

1. Experiments require careful design

2. Form research questions carefully

3. Think about validity4. Think about how your simulations/experiments relate to

real world

19

Takeaways

8/18/2019 Experiments(1)

http://slidepdf.com/reader/full/experiments1 20/22

Experimentation

Nicole Ronaldnronald@swin.edu.au

8/18/2019 Experiments(1)

http://slidepdf.com/reader/full/experiments1 21/22

Swinburne

SCIENCE | TECHNOLOGY | INNOVATION | BUSINESS | DESIGN

Experimentation

- Discuss experimental paper

- Design an experiment with human subjects

- Design an experiment using computers/equipment

21

Today’s tasks

8/18/2019 Experiments(1)

http://slidepdf.com/reader/full/experiments1 22/22

Experimentation

Nicole Ronaldnronald@swin.edu.au