Chapter 16Chapter 16

Measuring in ResearchMeasuring in Research

Measurement Challenges in Research Measurement Challenges in Research in PE, Sport, & Exercise Sciencein PE, Sport, & Exercise Science

• For physical education:For physical education:– Involvement of many different variables Involvement of many different variables – Influence of state and local education standardsInfluence of state and local education standards– Difficulty in qualifying and quantifying learningDifficulty in qualifying and quantifying learning

• Difficulty in replicating activities or injuries in Difficulty in replicating activities or injuries in laboratory conditionslaboratory conditions

• Unknown elements and complex Unknown elements and complex measurements in exercise physiologymeasurements in exercise physiology

Field vs. Laboratory StudiesField vs. Laboratory Studies

• Field studies: Field studies: – Conducted in the actual context of the Conducted in the actual context of the

measurement rather than in an artificial settingmeasurement rather than in an artificial setting– Advantages: Realistic; high ecological validityAdvantages: Realistic; high ecological validity– Disadvantage: Loss of control over variables Disadvantage: Loss of control over variables

affecting the measurementsaffecting the measurements

• Laboratory studies: Laboratory studies: – Simulations in a setting where the environment Simulations in a setting where the environment

is controlledis controlled

Quantitative and Qualitative Quantitative and Qualitative ResearchResearch

• Qualitative research: Qualitative research: – Seeks to understand human behavior Seeks to understand human behavior – Uses small but intensely studied samplesUses small but intensely studied samples

• Quantitative research:Quantitative research:– Uses experimental measures to test Uses experimental measures to test

hypotheses; extrapolates results from a hypotheses; extrapolates results from a sample to a larger populationsample to a larger population

– Uses random samples; unbiased Uses random samples; unbiased representation of the populationrepresentation of the population

Measurement Challenges in Measurement Challenges in Quantitative ResearchQuantitative Research

• Getting an overview of the data—the Getting an overview of the data—the “big picture”“big picture”

• Using non-normally distributed dataUsing non-normally distributed data• Confusing practical vs. statistical Confusing practical vs. statistical

differencesdifferences• Choosing appropriate research subjectsChoosing appropriate research subjects• Measuring the right variablesMeasuring the right variables

(continued)(continued)

More ChallengesMore Challenges

• Recognizing ceiling effects, floor effects, Recognizing ceiling effects, floor effects, and regression to the meanand regression to the mean

• Relying too much on group meansRelying too much on group means

• Using sample sizes that are too smallUsing sample sizes that are too small

• Making mistakes in data interpretationMaking mistakes in data interpretation

• Having equipment issuesHaving equipment issues

Ceiling Effects, Floor Effects, and Ceiling Effects, Floor Effects, and Regression to the MeanRegression to the Mean

• Ceiling effect: Ceiling effect: – Research subjects are near the maximal score and Research subjects are near the maximal score and

cannot be expected to exceed the “ceiling” value. cannot be expected to exceed the “ceiling” value.

• Floor effect: Floor effect: – Research subjects are near the lowest possible score Research subjects are near the lowest possible score

and cannot be expected to go below the “floor” value. and cannot be expected to go below the “floor” value.

• Regression to the mean: Regression to the mean: – The tendency for extreme scores to move toward the The tendency for extreme scores to move toward the

average (mean) the second time a test is taken. average (mean) the second time a test is taken.

Power Analysis of a Statistical TestPower Analysis of a Statistical Test

• The probability that the test will find a The probability that the test will find a statistically significant difference between statistically significant difference between measures.measures.

• Five factors:Five factors:– Effect sizeEffect size– Sample sizeSample size– Alpha (p) levelAlpha (p) level– Variability of the dataVariability of the data– Tails (one- or two-sided t-tests)Tails (one- or two-sided t-tests)

• Online: Online: www.cs.uiowa.edu/~rlenth/Power/www.cs.uiowa.edu/~rlenth/Power/

Power ErrorsPower Errors

1.1. Making a Type II error: Failing to find a Making a Type II error: Failing to find a difference between two groups/ difference between two groups/ treatments that really does exist.treatments that really does exist.

2.2. Making a research decision based on a Making a research decision based on a Type II error.Type II error.

Your ViewpointYour Viewpoint

• Can you think of an example of wrong Can you think of an example of wrong conclusions being drawn, or of data being conclusions being drawn, or of data being overinterpreted, in an experiment or overinterpreted, in an experiment or research hypothesis? research hypothesis?

• What happened? What was the result?What happened? What was the result?

Equipment IssuesEquipment Issues

• Equipment validity: How accurate is it?Equipment validity: How accurate is it?– Calibration: References the equipment to a Calibration: References the equipment to a

known, valid measurement. known, valid measurement.

• Equipment reliability: Is it dependable?Equipment reliability: Is it dependable?

• Equipment objectivity: Equipment objectivity: – Is it being used in the same way by every Is it being used in the same way by every

person?person?– Are instructions clear? Are they being properly Are instructions clear? Are they being properly

followed?followed?

Evaluating Equipment for PurchaseEvaluating Equipment for Purchase

• Establish specifications in terms of Establish specifications in terms of validity, reliability, and objectivity:validity, reliability, and objectivity:– How much accuracy do you need?How much accuracy do you need?– How reliable is it?How reliable is it?– How difficult is it to use?How difficult is it to use?

• Review the cost:Review the cost:– Initial costInitial cost– Cost of upkeep or operation costsCost of upkeep or operation costs

Challenges in Clinical and Challenges in Clinical and Epidemiological Quantitative ResearchEpidemiological Quantitative Research

• Sensitivity: Probability of a positive test Sensitivity: Probability of a positive test among patients with a disease.among patients with a disease.

• Specificity: Probability of a negative test Specificity: Probability of a negative test among people without a disease.among people without a disease.

• Predictive value: Predictive value: – Proportion of patients correctly diagnosed Proportion of patients correctly diagnosed

who displayed positive test results.who displayed positive test results.– Must be greater than the proportion of the Must be greater than the proportion of the

disease in the population.disease in the population.

Measurement Challenges in Measurement Challenges in Qualitative ResearchQualitative Research

• Establishing validity:Establishing validity:– Honesty and completeness of data collectionHonesty and completeness of data collection– What did investigators do, how much, and how often?What did investigators do, how much, and how often?

• Establishing reliability:Establishing reliability:– Internal-consistency reliability; confirmabilityInternal-consistency reliability; confirmability

• Establishing objectivity:Establishing objectivity:– Biases of investigators and methods of data collectionBiases of investigators and methods of data collection– No research or measurement is ever free of No research or measurement is ever free of

subjectivitysubjectivity

Increasing Objectivity Increasing Objectivity in Qualitative Researchin Qualitative Research

• Have multiple investigators analyze the Have multiple investigators analyze the datadata

• Have computer programs analyze dataHave computer programs analyze data

• Use triangulation:Use triangulation:– Find agreement among three different Find agreement among three different

perspectives or sources of dataperspectives or sources of data

Personal Disclosure StatementPersonal Disclosure Statement for a Qualitative Study for a Qualitative Study

Your ViewpointYour Viewpoint

• Think of the magazines and journals Think of the magazines and journals you read regularly. Do you think the you read regularly. Do you think the authors’ backgrounds may have authors’ backgrounds may have influenced their writings and opinions?influenced their writings and opinions?

• If so, how will this alter the way you If so, how will this alter the way you read their articles?read their articles?

Reading Research PapersReading Research Papers

• Find relevant research papers.Find relevant research papers.

• Read quantitative journal articles.Read quantitative journal articles.

• Understand the peer-review Understand the peer-review evaluation process.evaluation process.

Elements of Quantitative Elements of Quantitative Journal ArticlesJournal Articles

Some Issues Considered Some Issues Considered in the Peer-Review Processin the Peer-Review Process

• How relevant/original is How relevant/original is the topic?the topic?

• Are the measurement Are the measurement techniques techniques appropriate?appropriate?

• Are the study methods Are the study methods presented in enough presented in enough detail?detail?

• Is the analysis free Is the analysis free from major error?from major error?

• Are any conclusions not Are any conclusions not supported by the data?supported by the data?

• Does anything else Does anything else explain the results?explain the results?

• Did the authors overlook Did the authors overlook anything?anything?

• Does this manuscript Does this manuscript contribute to the relevant contribute to the relevant body of knowledge?body of knowledge?