NEUROPSYCHOLOGICAL EVALUATION OF SPORTS CONCUSSION/TBI:
HISTORICAL AND PSYCHOMETRIC CONSIDERATIONS
Ruben J. Echemendia
FDA Workshop Assuring the Safety and Effectiveness of Seizure, Cognitive
Function, and TBI/Sports Concussion Diagnostic Devices
Washington, DC: June, 2011
Disclosure
Dr. Echemendia is a paid consultant to: The National Hockey League Major League Soccer US Soccer Federation
Historical Background
Neuropsychological assessment has been a key component in evaluating/managing TBI for decades.
Neuropsychological evaluations have become a key tool in the evaluation of MTBI since mid 70’s.
Focus on functional vs. structural assessment.
NP assessment of sports concussions began in mid 80’s.
Background
Sports Laboratory Assessment Model Pre-post injury evaluations Paper & Pencil Assessment Batteries Computerized Batteries
Psychometric Considerations Validity vs. Reliability Validity
Does the test measure what it is supposed to measure? Construct Validity –Operationalization of the construct,
measurement of the construct Cognitive tests DO NOT measure concussion! Assess cognitive domains that are believed to be affected by
concussion Convergent/Divergent Validity
Does the test correlate with other known measures of the construct?
Does the test not correlate with domains that it is not supposed to measure Face Validity
Face Validity Does the test appear to be measuring what it is supposed to
measure
Psychometric Considerations
Sensitivity vs. SpecificitySensitivity: Is the measure “sensitive” in
detecting the construct being measured?○ Do concussed individuals score lower than
non-concussed individuals?○ What proportion of concussed individuals are
identified by the test? Positive Predictive Power
○ What proportion of Non-concussed individuals are identified by the test? Negative Predictive Power
HYPOTHETICAL 2X2 VALIDITY MATRIX
Abnormal NP Scores Normal NP Scores
0
0 150150
300
Concussed
Normal
150150
HYPOTHETICAL 2X2 VALIDITY MATRIX
Abnormal NP Scores Normal NP Scores
30/ 20%
150
300
Concussed
Normal
150
120/ 80%
90/ 60%60/ 40%
True Positive False Negative
False Positive True Negative
Psychometric Considerations
Sensitivity vs. SpecificitySpecificity: Is the measure “specifically”
measuring the identified construct and not others?
Concussion vs ADHD, Depression, Anxiety, Fatigue, etc.
Ideal: 100% Sensitivity – 100% SpecificityReal: NOT!Sensitivity vs Specificity trade-offTypically as sensitivity increases, specificity
decreasesThe user must decide on what side to err.
ROC Curve
Psychometric Considerations
Reliability – Estimating ErrorCan we expect to get the same results when
the test is given….○ To the same person an infinite number of
times? (Confidence Intervals)○ To the same person under different
circumstances? (Sideline vs. Office)○ To the same person at different points in time?○ To the same person by different test
administrators?○ In different/alternate forms?
Tests Used in Sports TBI….
Paper & Pencil Tests Selected from tests already in use and validated
for assessing MTBI Test Requirements:
ShortEasily administered by non-NPsEasily scoredSensitive to changes in specified domainAcceptable psychometricsAlternate forms.
Assess several cognitive domains. Typical Tests: HVLT. BVMT. Trails, Digit Span,
SDMT, Verbal Fluency, LNS, Cancellation tests.
Paper & Pencil Tests Cont. Advantages:
Direct observation of player behaviorDirect monitoring of effort/motivation & task
performanceSuperior assessment of memoryBroad normative db with multiple samples.
Disadvantages:Labor intensive (Expensive)Variability in standardized adminLess reliable assessment of info proc speed.Tests must be hand scored
Computerized Batteries Two approaches:
Adapt existing P&P measures into computer platform
Measure multiple cognitive functions (memory, proc speed, reaction time, etc.)
Develop tests de novo for computers Limited assessment of domains (largely
RT, Info Proc Speed) New Instruments are being developed rapidly Existing Instruments: ImPACT, ANAM Sports,
HeadMinders CRI, CNS Vital Signs, CogSport.
Computerized Tests Cont. Advantages:
Can be used in groups (Less costly)Standardized admininistrationImmediate scoring/data captureFiner assessment of RT/Proc Speed
(altho other problems)Multiple languagesTheoretically infinite alternate
forms
Computerized Tests Cont. Disadvantages:
Less complete assessment of memoryLoss of observational dataLess control of effort/motivationDiminished ability to assess
understanding/track task performanceComputer-specific measurement issuesData loss
Hybrid Approaches
Combine strengths of both P&P and Computerized tests
Once adequate normative data have been established….
Computer for baseline testingComputer plus P&P post-injury
Sports MTBI specific issues
Phase of Evaluation
Baseline
Acute – On Field Management
Sub acute – Office/Training Room
Return to Play/Work/School
Long Term Follow-up
Sports MTBI specific issues Baseline Testing – Is it necessary/helpful? Theoretically designed to minimize error due
to intra-individual comparison Does it do this? Does it increase sensitivity/specificity? Does it introduce additional sources of error? What about multiple post-injury tests? How often should it be repeated? Age dependent?
Sports MTBI specific issues Test – Retest Reliability Significant variability across studies What test-retest intervals should be
used? Reliability estimates for multiple tests
Sports MTBI specific issues Practice Effects Content Practice vs. Procedural Use of Alternate Forms Reliability/Validity of Alternate Forms Are practice effects bad? How to control for / appropriately use
practice effects?RCI, regression based estimates
Sports MTBI specific issues Motivation/Effort Unique aspect of sports was the
assumption that players are motivated to perform well.
Different motivation as different phases of assessment
Baseline vs. Post-injury Individual vs. Group Identification of poor effort
Test specific indicesOutlier identification
User Qualifications
Although not the responsibility of the FDA, clear delineation of user qualifications for administering and interpreting tests is critical.
Failure to do so may lead to scope of practice challenges, medico-legal quandaries and poor practice.
Thank you!