“How many are enough? Justifying quantitative and qualitative research sample sizes” Mark NK...

1

“How many are enough? Justifying quantitative and qualitative research sample

sizes”

www.surrey.ac.uk

Mark NK SaundersSurrey Business School

The Surrey Business Schoolwww.surrey.ac.uk/sbs

Learning outcomes … you will

• be aware of the relationship between research question, population, target population, the sample selected and the implications of sample size.

• understand the implications of using different probability and non-probability sampling techniques and the need to combine techniques within a research project.

• be able to explain sample selection in terms of technique and sample size precisely in your methodand justify this for your own research project

• have the opportunity to ask questions.


Aldi’s “Like Aldi… like the price” campaign

What is the population?

a) Supermarket customers

b) Aldi customersc) Kellogs likers?d) Special K

likers who shop at Aldi

e) Other?Source:http://www.likealdi.co.uk/home/


Population, target population, sample and individual cases

But what characteristics do the sample of 179 “Kellogs Special K likers” have? What are they representative of?How was the target population from which the sample of “Kellogs Special K likers” was selected actually defined?


Questions (about the Aldi sample)

1. How has the target population from which the sample is selected been defined?

2. Why has the target population been defined in this way?

3. What are the target population’s characteristics?

4. How does the target population relate to the research question being addressed?

5. What technique(s) were used to select the sample?

6. Why were these technique(s) chosen?

7. What are the implications of the technique(s) for the characteristics of the sample?


Yves St Laurent’s skin cream

“A small test of 50 women found

almost nine out of 10 of them said their skin looked more luminous after using it, while 72 per cent said fine lines appeared less visible.”


Yet the total population of women in UK is 31.4 million (Source: 2011 Census)

Probability sampling -and sample size

Sample size for 95% confidence level with 5% margin of error

0

50

100

150

200

250

300

350

0 500 1000 1500 2000 2500 3000 3500 4000 4500 5000

Total Population

Sa

mp

le s

ize

The problem of making statistical inferences from a small probability sample

population 5% 2%1,000,000 384 239510,000,000 384 240034,100,000 384 2401


(Additional) questions about the YSL sample

1. Was a probability sample (or a quota sample) used? (need to use such sampling techniques to make statistical estimates)

2. Which precise sampling technique was used to select the sample, and what were the implications for the nature of the sample?

3. Assuming a sample of 50, a confidence level of 95% and a population of 34.1 million this gives a 14% margin of error: somewhere between 58% and 86% said ‘fine lines appeared less visible’…

4. What are the implications of sample size for utility of findings?


Non probability sampling techniques

Sampling

Probability Non-probability

Volunteer Haphazard

Snowball Selfselection Convenience

Heterogeneouspurposive

Homogeneouspurposive

Critical casepurposive

Typicalcase

purposiveTheoretical

Extremecase

purposive

Purposive

Quota

Quota

Different non-probability techniques

have differing implications


Probability sampling techniques

Systematic Stratified ClusterSimplerandom

Multi-stage

Sampling

Probability


Sampling matters!

• Target population (and its characteristics) must be defined and justified in relation to the research question to be answered

• Relationship between target population and sample selection technique (probability or non-probability) impact on research question that can be answered-issue of representativeness (to what extent does the sample represent the population from which it is drawn?)-issue of generalisability (to what extent can the findings be generalised to other settings?)

• Choosing appropriate probability and/or non probability sample selection techniques depends on the research question being answered, but which should be used when?

• Sample size is important, but how many?


Non-probability sample size -recommendations

Nature of study Size Source(s)

Qualitative >=15 Bertaux (1991)

Interview 12-60 Adler and Adler (2012)

1 may be enough Becker (2012)

5-25 Brinkmann and Kvale (2015)

Homogenous population 6-8 Kuzel (1992)

Heterogeneous population 12-20 Kuzel (1992)

Ethnographic Usually c. 36 Bernard (2000)

1 cultural group Cresswell (2007)

30-50 Morse (1994)

Grounded Theory 15-20 Cresswell (2002)

20-35 Cresswell (2007)

c. 35 Morse (1994)

Case study 3-5 per case Cresswell (2007)

Source: Developed from Saunders (2012), Saunders and Townsend (2015)


Non-probability sample size -example precedence studies

Author Focus Findings Recomendations

Collins et al. (2007)

121 MM studies in unspecified Health and Social science journals

98.4% reported sample size, for qual. and quant. phases, but not discussed further

None re sample size

Marshall et al. (2013)

83 qual. interview studies in 5 Information Systems journals

“many” invoked idea of data saturation but only few provided evidence

20-30 interviews for Grounded Theory and 15-30 for case studies. Can establish precedence

Safman and Sobel (2004)

93 qual. studies (57 interviews) in a Health Sci journal

60% reported sample size; varied 2-720, median 50

Sample size should be reported. None re size

Saunders & Townsend (2015)

244 qual. Interview studies in Org. & Workplace journals

81% reported sample size, 50% justified size by purpose, 4.2% by saturation, 2.6% by reccs.

15-60 interviews (will be exceptions). Sample should be reported and justified


Non-probability sample size -data saturation

Author Focus Findings Recommendations

Francis et al. (2010)

14 general medical practitioners, 17 patients’ relatives

Saturation after 13 (practitioners); 14 (relatives)

Specify criteria for saturation in advance and report in findings; evidence needed to establish convention

Guest et al. (2006)

30 Nigerian and 30 Ghanaian females -high risk HIV

Saturation after 12 interviews; 6 sufficient for meaningful themes

6-12 enough for saturation in homogenous population, but be cautious

Marshall (1996)

10 medical leaders, 24 medical practitioners

Saturation after 15 interviews, acceptable interpretive framework after 24 interviews

None


Non probability sampling –in summary

REMEMBER –making generalisations to theory not about a population (except with quota samples)

Sample size depends upon research question and objectives… research purpose what will be useful what is authentic and credible what can be done within time and resources available

Sample size should be stated and also justified by one or more of: Recommendations of cited experts Precedence from cited previous cognate studies Data saturation (with explanation of how know reached)


Probability sample size calculation

0

50

100

150

200

250

300

350

0 500 1000 1500 2000 2500 3000 3500 4000 4500 5000

Total Population

Sa

mp

le s

ize Sample size for 95%

certainty with 5% margin of error

population 5% 2%

10,000 370 1936

1,000,000 384 2395

10,000,000 384 2400


Probability sampling –in summary

REMEMBER –making statistical generalisations about the target population where are cases/elements are known (sampling frame)

Sample size depends upon… size of target population (sampling frame)

margin of error and confidence level likely active response rate (taking into account partial response)

Sample size, response rate and number of respondents/participants should be stated and also justified in relation to: size of target population margin of error and confidence level precedence from cited previous cognate studies

WARNING: Beware of effect size


Sampling –what do you want to ask?

Spend a few minutes noting down at least two questions

regarding sampling and your doctoral research you

would like answered by

the end of this session

You can ask about techniques, sample size,

relation to research questions, reporting, further

reading… anything!


What are

What are the sampling issues?


More detail

In February 2011 the UK’s Outdoor Media Centre launched its ‘Hall of Fame’ competition to identify the 100 best advertising posters of all time. Working with the History of Advertising Trust, they generated a list of 500 posters. This was reduced to a shortlist of 228 posters by a committee of media and creative experts together with the editor of weekly magazine - Campaign. These were displayed on a dedicated website www.outdoorhalloffame.co.uk. Creative agencies, media planners, advertisers, media owners and the general public were invited in an article in Campaign to go to the web site, view the advertising campaigns and cast their votes for what they considered to be the best outdoor posters. Each person was able to cast a total of ten votes, the best advertisements being “chosen after more than 10,000 reader votes”.

http://www.outdoorhalloffame.co.uk/


Sample selection mattered!

12

3

4

5

67

8


Sample selection … realities in management research

REALITIES Don’t always have sampling frame Sampling frame often restricted Difficult to show sample truly random

CONSEQUENCES Population assumed itself to be a random sample from larger population or Inferences (including statistical) confined to actual population from which sample drawn –can’t generalise statistically May not be able to answer certain questions… rephrase the question!

Ideally generalise to the theoretical population of managers in UK & Eire

Can gain access to target population of managers taking PG courses at a University

Using the sampling frame of their email addresses

Sample comprises those who respond


Sampling matters…. remember: • Statistical representativeness requires probability sampling

• Probability sampling requires a sampling frame

• Relatively large sample sizes are needed for statistical representativeness, but beware of effect size

• Generalisability requirements necessitate careful thought at the sample selection stage

• Selection of sampling technique always has implications for the Research Question that can be answered and the data collection technique(s) used

• The choice of sampling technique(s) needs to be explained clearly and justified in relation to the research question and with reference to the literature (methods and past research)

• Non probability sample size should be stated explicitly and justified in terms of purpose, expert recommendations, precedence, and/or data saturation

• Probability sample size should be stated explicitly and justified in terms of purpose, confidence level, margin of error and precedence


Selected references (1)

American Association for Public Opinion Research (2008) Standard Definitions: Final

Dispositions of Case Codes and Outcome Rates for Surveys (5th edition) Lenexa, KA:

AAPOR Baker, S.E. and Edwards, R. (2012). ‘How many qualitative interviews are enough?’

National Centre for Research methods Review Paper. Available at

http://eprints.ncrm.ac.uk/2273/ [Accessed 20 Jan. 2015]. Baruch Y and Holtom BC (2008) Survey response rate levels and trends in organizational

research Human Relations 6.8 1139-60. Ellis PD (2010) The essential guide to effect sizes Cambridge: Cambridge University Press Francis, J.J., Johnston, M., Robertson, C., Glidewell, L., Entwistle, V., Eccles, M., and

Grimshaw, M. (2010). ‘What is an adequate sample size? Operationalizing data saturation

for theory-based studies.’ Psychology and Health. 25.10, 1229-1245. Groves, R.M., Dillman, D.A., Eltinge, J.L. and Little, R.J.A. (eds.) (2001) Survey Non-

response. New York: John Wiley. Guest, G., Bunce, A. and Johnson, L. (2006). How man y interviews are enough? An

experiment with data saturation and validity. Field Methods. Vol. 18, No. 1, pp. 59-82 Highhouse S and Gillespie JZ (2009) ‘Do samples really matter that much?’ In CE Lance

and RJ Vandenberg (Eds) Statistical and Methodological Myths and Urban Legends New

York: Routledge 247-66 Huck S.W. (2009) Statistical Misconceptions New York: Routledge, Chapter 6

http://eprints.ncrm.ac.uk/2273/


Selected references (2)

Neuman, W.L. (2000) Social Research Methods. 2nd edn. London: Allyn and Bacon.

Patton, M.Q. (2002) Qualitative Research and Evaluation Methods. 3rd edn. Thousand Oaks, CA: Sage, Ch.5

Marshall, B., Cardon, P., Poddar, A. and Fontenot, R. (2013). Does sample size matter in qualitative research: a review of qualitative interviews in IS research. Journal of Computer Information Systems 54.1, 11-22

O’Reilly, M. and Parker, N. (2013). ‘’Unsatisfactory saturation:’ a critical exploration of the notion of the notion of saturated sample sizes

Robinson, O.C. (2014) Sampling in interview based qualitative research: a theoretical and practical guide. Qualitative Research in Psychology. 11.1 25-41

Rogelberg SG and Stanton JM (2007) Introduction: Understanding and dealing with organizational survey non-response Organizational Research Methods 10.2 195-209

Saunders, M.N.K., Lewis, P. and Thornhill, A. (2016) Research Methods for Business Students. 7th edn. Harlow: Pearson, Ch.7.

Saunders M.N.K. (2012) ‘Choosing research participants’ In G Symon and C Cassell (Eds) The Practice of Qualitative Organizational Research: Core Methods and Current Challenges.London: Sage 37-55

Saunders M.N.K. and Townsend K. (2015) ‘An analysis of the number of interview participants in organisation and workplace journal articles’ Full paper presented at the British Academy of Management Annual Conference, Portsmouth.

Date post:	02-Jan-2016
Category:	Documents
Upload:	lionel-chase
View:	218 times
Download:	3 times

“How many are enough? Justifying quantitative and qualitative research sample sizes” Mark NK...

Documents