How could it be fun?

Selling methods and statistics Methods are good for you!

Research methods and statistics develop transferable skills. Will you ever need to write an essay ….. But report?:

The ‘ah but’ mentality

Always query:

advertisement claims

politicians’ claims

anyone who can’t produce evidence

the nature of any evidence – look at ‘Bad Science’

The nature of evidenceThe nature of evidence

1. ABBA

Did A cause B or could B have caused A?

(or did C cause A and B..... etc.)

Doesn’t work with iced-cream sales and temperature!

THE LONELINESS OF VIRTUAL LIVING 169 participants followed up over three month period after purchasing internet capable computer. Researchers* measured internet use and depression. Participants used predominantly social features (e-mail. chat) BUT

(according to Guardian article):

“Researchers found that one hour a week on the Internet led to an average increase of 1% on the depression scale, a loss of 2.7 members of the subject’s social circle, and an increase of 0.4% on the so-called loneliness scale.”

* Kraut, R., Patterson, M., Lundmark, V., Kiesler, S., Mukophadhyay, T. & Scherlis, W. (1998) Internet paradox: A social technology that reduces social involvement and psychological well-being? American Psychologist, 53(9), Sep, 1017-1031.

THE NATURE OF EVIDENCE

2. Big numbers are serious

Great news! Twenty-five per cent of maths teachers to go!

A recent article writing in rather panicky terms about the shortage of maths teachers in the country made the claim that 25% of maths teachers were due to retire in the next ten years. Is this an exceptional worry?

HEALTH SERVICE HEADING FOR £1BILLION DEFICIT! In 2006 the NHS was heading for a £1 billion

deficit This eventually came down to around £800 million

which was about 1% of the total budget.

Average predicted deficit for whole of government spending was 2%.

NHS was doing twice as well as other departments.

Most businesses would think spending hitting target within 1% would be sheer magic.

Source: Blastland and Dilnot (2007) The tiger that isn’t (London: Profile Books)

The nature of evidenceThe nature of evidence

3. Mis-use of statistics

Shock finding! Scientists find millions of Shock finding! Scientists find millions of English people have above average English people have above average number of legs!number of legs!

Sadly a few have only one or none and even fewer have three (?) so mean is 1.999…..

What about the median?

Should be the mode!

The nature of evidenceThe nature of evidence

4. Problems with significance

A newspaper once published the results of a field experiment (quasi) in which 10 wallets containing money had been dropped in each of several towns or cities. The wallets contained identification so they could be retuned and the numbers that did come back are shown below.

Returned Kept Returned Kept

Glasgow 8 2 Pontefract 7 3

Leamington Spa/ Warwick

8 2 Liverpool 6 4

Basildon 7 3 Exeter 5 5

London 7 3 Cardiff 4 6

Adapted from The Guardian, 17 June 1996The journalist said, “Residents of Glasgow and Leamington Spa/Warwick emerged as Britain’s most honest citizens, each handing in eight out of 10 wallets.” The author of the original article (Jack Crossley,1996) claims: “Medium size towns were slightly more honest than cities”, and shows that the respective totals returned were 27 and 25. What do you think?

• Sex determination clinic – ‘So far we’ve had 6 couples through and four of these have left satisfied’

SVELTE GEL BY CHRISTIAN DIOR

550 women given free sample Asked to use it for a month.

“52% of women reported losing up to one inch from their hips”

“56% reported losing up to one inch from their thighs” “56% reported losing up to one inch from waist”

How measured? Perhaps take a measure now and another after one month.

What would happen if Svelte was useless (the null hypothesis) We would expect about half to ‘lose’ and half to ‘gain’ Also, how many other parts of the body were measured? And how many samples were used?

5. Questioning the method – professional ethics

Richard Lynn’s estimate of ‘Black African IQ’ at 75 points

Majority of the baseline data South Africa, during white rule or Apartheid. Lynn’s (1991) data - 2/3rds of participants from South Africa

Rest from 5 other countries.

Akin to assessing the overall mean European Catholic IQ from 5 Northern Ireland samples, 2 from Portugal, 1 from Spain and 1 from Norway.

Mix of ages, test types and rather a range of periods (1929 to 1991).

The nature of evidence

SOME TRICKSFold paper three times. Note it gets slightly thicker. Suggest 50 more folds if physically possible (Blue Peter did it simply by stacking paper sheets). How thick would the paper be? Tall as a house?

Folding paper Challenges simple ‘common sense’

Telepathy Think of a number under 50

Must have two digits

Both must be odd and different

11 would not count but 15 would

Can use to discuss testing hypotheses

Also demonstrates empirical and logical probability

The rope trick

Birthdays How many people needed in a room to have 50% chance of two having same birthday?

Anti –common sense and probability values

A piece of rope circles the earth. We want to raise it off the surface by I foot. How much more rope would we need?

ANS: 23

ANS: about 6.3 feet Challenges simple ‘common sense’

Circumference = 2r

New circumference = 2 (r+1) = 2r + 2

new circ – old circ = 2r + 2 - 2r = 2

MORE FUN…..

If you do it yourself – use what you already know from experience

…. The heat/aggression hypothesis

If it relates to everyday events and normal thinking

….. Most statistics and especially significance (later) (e.g. dispersion and darts)

If it is integrated into theory delivery

Practicals, written up, need to be extremely simple (even ‘artificial’) in order to demonstrate very clearly the points being made – e.g. design, analysis

e.g. Pichert and Anderson (1977). What’s the difference?

Practicals I’ve found fun:Clock this!Barnum effectCaffeine and dropped ruler as RT measureIQ estimatesStroopWiggly wireAttribution: self-serving bias and football manager explanationsRound things and long thingsDriver behaviour and gender/type of car/van or carRating cartoons whilst holding pen in mouth sideways or forwardsNicolson 1981 – faster people talk the better STM they have (quicker articulatory loop)Buss: mate choice and adsSelf-concept gap and anxiety (Rogers)Santa’s sack before and after Christmas (schema)Audience effects (e.g. Bopper game)Jokes from celebrities or others

You have a need for other people to like and admire you, and yet you tend to be critical of yourself. While you have some personality weaknesses you are generally able to compensate for them. You have considerable unused capacity that you have not turned to your advantage. Disciplined and self-controlled on the outside, you tend to be worrisome and insecure on the inside. At times you have serious doubts as to whether you have made the right decision or done the right thing. You prefer a certain amount of change and variety and become dissatisfied when hemmed in by restrictions and limitations. You also pride yourself as an independent thinker; and do not accept others' statements without satisfactory proof. But you have found it unwise to be too frank in revealing yourself to others. At times you are extroverted, affable, and sociable, while at other times you are introverted, wary, and reserved. Some of your aspirations tend to be rather unrealistic.

AQA Spec B Specimen papers for new 2009 syllabus:

“Write an appropriate hypothesis for this study”.

One-tailed: The frequency of friendly behaviours is higher in same sex pairs of children (AO3,1) than in boy-girl pairs (AO3,1)

Two-tailed: There will be a difference between the frequency of friendly behaviours of same-sex pairs (AO3,1) and boy-girl pairs (AO3,1).

“Null: There will be no difference between the frequency of friendly behaviours of same-sex pairs (AO3,1) and boy-girl pairs (AO3,1).”

OCR Specimen syllabus 2009

“Suggest an appropriate null hypothesis for this experiment.”

“There will be no difference between the number of words recalled when participants learn and recall in the same room and when participants learn and recall in different rooms.”

“If, having carried out your investigation and an inferential statistical test, your experimental hypothesis was found to be significant for p < 0.05, what would ‘p < 0.05’ mean?”

“The probability that there is a significant difference between the two conditions is 0.05

or

“a 1 in 20 (95%) likelihood that the result occurred by chance”

Because we assume Ho that: N reds = N blacks

AND that cards were selected at random

Nine reds from a pack of cards – AMAZING! -- WHY?

Significance and the null hypothesis

That’s amazing if the pack was shuffled

That’s a low p if the null hypothesis is true

The null hypothesis is always about populations from which samples were drawn

No researcher ever states their null hypothesis

A null hypothesis can only be stated once we know exactly which statistical test is to be applied to which specific data

A null hypothesis is therefore out of place at the end of an introduction

What is stated at the end of an introduction is usually the research prediction (s)

A typical null hypothesis:0 : imagery rehearsalH

Population mean for imagery condition scores is equal to the population mean for rehearsal condition scores

We test the alternative hypothesisWe test the alternative hypothesis H1: Female reading scores are higher

than male reading scores

By making a specific research prediction:

e.g. The female sample will have a higher mean reading score than the male sample

This is not a hypothesis it is a test of one.

A concrete example A concrete example

Does the drawer contain 50% right and left hand gloves?

You pull out five right hand gloves in succession

The null hypothesis H0: fright = fleftSignificance?

Probability of five right hand gloves is about 0.03

(ignore non-replacement)

In every significance test we:

• Assume a null hypothesis of no effect

• Calculate the probability that our difference would occur if the null hypothesis is true ( ‘under H0’ )

• Declare a significant difference if p is less than 0.05

back

REMEMBER THIS MANTRA:

What’s the odds of that happening if there’s nothing going on?

“If they’re not seeing each other then how come he knew her number?”

“If she’s not mean than how come she’s never first to get a round in?”

“Five times out of seven the tripe was underweight and twice it was over. How can that happen if he ain’t a crook?

Evidence that we use significance reasoning in everyday life reasoning

NOW LET’S TRY SCREWS

Normal screws Suspect screws

Random sample N

Random sample S

Null hypothesis: normal = suspect

BACK TO PSYCHOLOGY EXPERIMENTS

Decaff condition

Random sample C

Random sample D

Caffeine condition

H0: caffeine= decaff

Psychology chips drawn

from

Sociology chips drawn

from

Alternative hypothesis

Psychology chips drawn

from

Sociology chips drawn

from

Null hypothesis

Trouble at t’chip shop

H0 null hypothesis

H1 alternative hypothesis

p is the probability that the sample difference would occur under the null hypothesis

(NOT ‘by chance’)

Substitute ‘by chance’ with under H0

We do NOT have

‘The probability that the null hypothesis is true’

‘The probability that the results occurred by chance’

From now on never say:

“The null hypothesis is that there will be no (significant) difference between the rehearsal and imagery scores”

The null hypothesis is a statistical claim about the underlying population or populations from which samples have been drawn:

“The means of the rehearsal and imagery population scores are equal.”

This is a prediction from the null hypothesis so it cannot be the null hypothesis

Population 1 (untrained/control)

Population 2 (trained/experimental)

Hypothesis is about these:

Sample 1 Sample 2 research study test is about these FINDINGS

CONCLUSIONS

Validity is about how you might have drawn the wrong conclusion from the samples you tested

The link with validity

Did manipulation ofIV really affect DV?

Internal and external validity

Internal validity

External validity

Will the effect generalise to:

Populations?Locations

(‘ecological validity’)Times and contexts?

1. The original technical meaning Brunswik (e.g. 1947) introduced the term ‘ecological validity’ to psychology

as an aspect of his work in perception ‘to indicate the degree of correlation between a proximal (e.g., retinal) cue and the distal (e.g., object) variable to which it is related’ (Hammond, 1998, para 18). This is a very technical use. The proximal stimulus is the information received directly by the senses – for instance two lines of differing lengths on our retinae. The distal stimulus is the nature of that actual object in the environment that we are receiving information from. If we know that the two lines are from two telegraph poles at different distances from us we might interpret the two poles as the same size but one further away than the other. The two lines have ecological validity in so far as we know how to usefully interpret them in an environment that we have learned to interpret in terms of perspective cues. The two lines do not appear to us as having different lengths because we interpret them in the context of other cues that tell us how far away the two poles are. In that context their ecological validity is high in predicting that we are seeing telegraph poles.

2. The external validity meaning

Bracht and Glass (1968) defined ecological validity as an aspect of external validity and referred to the degree of generalisation that is possible from results in one specific study setting to other different settings. This has usually had an undertone of comparing the paucity of the experimental environment with the greater complexity of a ‘real’ setting outside the laboratory. In other words people asked how far will the results of this (valid) laboratory experiment generalise to life outside it? Cook and Campbell (1979) also supported this interpretation though they have more recently, and because of the controversy, replaced it with the term ‘external validity with regard to settings’. On this view effects can be said to have demonstrated ecological validity the more they generalise to different settings and this can be established quantitatively by replicating studies in different research contexts.

Cook and Campbell, 1979, Quasi-experimentation

3. The ‘pop’ version The pop version is the definition very often taught on basic psychology courses. It takes the view that a study has (high) ecological validity so long as the setting in which it is conducted is ‘realistic’, or the materials used are ‘realistic’, or indeed if the study itself is naturalistic or in a ‘natural’ setting. The idea is that we are likely to find out more about ‘real life’ if the study is in some way close to ‘real life’, begging the question of whether the laboratory is not ‘real life’.

Knee-jerk mantra – the more realistic the more ecological validity.

There is however no way to gauge the extent of this validity. Teaching students that ecological validity refers to the realism of studies or their materials simply adds a new ‘floating term’ to the psychological glossary that is completely unnecessary since we already have the terminology. The word to use is ‘realism’. As it is, students taught the pop version simply have to learn to substitute ‘realism’ when they see ‘ecological validity’ in an examination question.

Milgram or Hofling – which is more ecologically valid?

Milgram’s results replicated in many different ways and in many differing environments, not least, in basement of private company rather than in prestigious university.

Hofling’s result never properly replicated. Rank and Jacobson (1977) showed how ‘artificial’ Hofling’s study was – when nurses allowed to talk with peers and familiar with drug no obedience effect.

Kvavilasvilli and Ellis, 2004, Ecological validity and real-life/laboratory controversy in memory research: A critical and historical review, History of Philosophy and Psychology, 6, 59-80.

They argue that a highly artificial and unrealistic experiment can still demonstrate an ecologically valid effect. They cite as an example Ebbinghaus’s memory tasks with nonsense syllables.

Hammond (1998) – pop version is unnecessary since Brunswick also talked about representative design.

WHEN IS AN EXPERIMENT NOT WHEN IS AN EXPERIMENT NOT AN EXPERIMENT?AN EXPERIMENT?

1. When participants not allocated to conditions at random

2. When experimenter is not in control of variables/does not manipulate the independent variable

Cook and Campbell (1979) – QUASI-EXPERIMENTS

Why worry? - Greater threats to validity

Isolate phenomena in controlled environment; obtain reliable effect between phenomena

Observe several phenomena in everyday life

Form theory about relationship between phenomena

Support theory and extend/apply theory in real life settings

Artificiality of the laboratory experiment? Relationship between theory, controlled research environment and real life settings

Sampling boxes

Arrange numbers 1 to 11 and 4 to 14 in the following proportions in two boxes:

1 10 45 120 210 252 210 120 45 10 1

- show you can make fair estimate of population mean from sample of 10- show that samples of 5 give more variance around true mean of 6 than do samples of 10- show that differences between two means of random samples tend to centre around zero as they should under the null hypothesis. If you get as far as discussing the variation of these mean differences (you have them all written on the board) you can talk of their standard deviation or standard error and you are an ace away from the entire rationale of the t test – how many SEs is our difference from zero = t

Easy Mann-Whitney – the original Select group with higher scores

For each score in that group:

Give 1 point each time beaten by score in the other group

Give ½ point each time there is a tie

U is the addition of all points for that group

Ua + Ub = Na x Nb Na x Nb because each score in a group is compared with each score in the other group. Just like playing darts matches league style with members of two teams.

Resources • British Psychological Society The Psychologist • APA

• http://www.psychology.heacademy.ac.uk/ • http://davidmlane.com/hyperstat/ - HYPERSTAT• Research Methods Knowledge Base• http://davidmlane.com/hyperstat/ Rice virtual lab in stats• http://www.stattucino.com/berrie/dsl/regression/regression.html

play with correlation scatterplot • http://psych.unn.ac.uk/prac/y1pr.html Psychology practicals on the web*• http://psychology.wikicities.com/wiki/Main_Page • Sandy McRae’s disc from British Psychological Society • Murphy’s Law video – BBC Education, Wood Lane, London• Psychology Review• McIlveen BPS Manual of practicals• http://www.wma.ac.uk/vle/Online%20courses/Psychology%20memory

%20experiments/htmlFiles/frame.htm• http://psychcentral.com/blog/archives/2008/08/18/top-ten-online-

psychology-experiments/

Qualitative sources

• Braun & Clarke (2006): A definitive article describing thematic analysis and giving a step by step methodological guide. Qualitative Research in Psychology, 3, 77-101

• Flick (2008): This is the new practical hands-on volume from Flick, a leading authority on qualitative research, and takes the student through all the design, data collection and analysis stages of a project.

• Hahn (2008)A very useful text that shows how qualitative coding can be carried out using Microsoft Word, Excel and Access

• Willig (2008):A comprehensive text taking you through the way to carry out qualitative research under several approaches (IPA, GT, DA — discursive and Foucaldian), memory work, plus a discussion of quality and many research examples.

• Smith (2008):This text is also hands-on, covering much the same ground as Willig (2008), but includes narrative analysis, CA, focus groups and cooperative enquiry. Perhaps more practical and less theoretical than Willig.

Qualitative sources

• Online QDA – Online information about many different qualitative analysis approaches. Don’t worry about the title in the address this is all methods (27 of them in all) with links to further reading.

• http://onlineqda.hud.ac.uk/methodologies.php#Interpretive_Phenomenological_Analysis

• The content analysis Guidebook – is what the title suggests. A comprehensive guide to content Analysis with many links.

• http://academic.csuohio.edu/kneuendorf/content

The Psychology Network at the Higher Education Academy whose web address is http://www.heacademy.ac.uk/ - select the Psychology network under ‘Subject Centres’ or use this link if it still works: http://www.psychology.heacademy.ac.uk/ .