MEQMEQAnalysisAnalysis

OutlineOutline ValidityValidity ReliabilityReliability Difficulty IndexDifficulty Index Power of DiscriminationPower of Discrimination

ValidityValidity““evidence present to support or evidence present to support or refute the meaning assigned to refute the meaning assigned to

assessment results”assessment results”

face validityface validitycontent validitycontent validitycriterion-related validitycriterion-related validityconstruct validityconstruct validity

Face ValidityFace Validity high face validityhigh face validity

seemed valid (only)seemed valid (only) ~ person~ person

knee jerk & nervous systemknee jerk & nervous system for doctor => high face validityfor doctor => high face validity for lay people => low face for lay people => low face

validityvalidity MEQ => high face validityMEQ => high face validity

Content ValidityContent Validity ~ sample & population~ sample & population MCQ => high content MCQ => high content

validityvalidity MEQ => low content MEQ => low content

validityvalidity

ValidityValidity““evidence present to support or evidence present to support or refute the meaning assigned to refute the meaning assigned to

assessment results”assessment results”

face validityface validitycontent validitycontent validitycriterion-related validitycriterion-related validityconstruct validityconstruct validity

- don’t need the score- before using the test

- score needed

- after using the test

Criterion-Related Criterion-Related ValidityValidity

Predictive validityPredictive validity MEQ score MEQ score && close observation close observation

score of real practice at ERscore of real practice at ER Concurrent ValidityConcurrent Validity

MEQ score MEQ score && VIVA score in the VIVA score in the same topicsame topic

Statistic = correlation Statistic = correlation coefficientcoefficient

Construct ValidityConstruct Validity based on theorybased on theory

communication skill ~ leadershipcommunication skill ~ leadershipcorrelation OSCE : communication correlation OSCE : communication VS VS questionnaire : leadershipquestionnaire : leadership

good ethics ~ beloved doctorgood ethics ~ beloved doctorMEQ : medical ethics MEQ : medical ethics VSVS questionnaire : beloved doctor?questionnaire : beloved doctor?

ReliabilityReliability stabilitystability internal consistencyinternal consistency equivalentequivalent

StabilityStability - test retest reliability- test retest reliability parallel form reliability parallel form reliability - intra rater reliability- intra rater reliability

~ scoring key ~ scoring key statistics : correlation co statistics : correlation co

eff.eff. -01-01

Internal Internal ConsistencyConsistency

[homogeneity][homogeneity] item - item correlationitem - item correlation item - total correlationitem - total correlation split half correlationsplit half correlation

Item-Item Item-Item CorrelationCorrelation

each itemeach item DichotomousDichotomous

Phi CorrelationPhi Correlation IntervalInterval

Pearson’s Product Moment Pearson’s Product Moment CorrelationCorrelation

whole testwhole test Mean of ...Mean of ...

Item-Total Item-Total CorrelationCorrelation

each item each item DichotomousDichotomous

Point Biserial Correlation Point Biserial Correlation IntervalInterval

Pearson’s Product Moment Correl Pearson’s Product Moment Correlationation

whole test whole test Mean of ... Mean of ...

Spilt Half Spilt Half ReliabilityReliability

DichotomousDichotomous - 20 2Kuder Richardson (KR- 20 2Kuder Richardson (KR00

IntervalInterval - 21 2Kuder Richardson (KR- 21 2Kuder Richardson (KR1)1)

aaaaa aaaaaaaaaaa’ aaaaa aaaaaaaaaaa’

EquivalentEquivalent parallel item on parallel item on

alternate form reliabilityalternate form reliability inter-rater reliabilityinter-rater reliability

agreementagreement kappakappa

Difficulty IndexDifficulty Index [p][p] (mean H + mean L)(mean H + mean L)//2(full score)2(full score) p = 1 => very easyp = 1 => very easy p = 0 => very difficultp = 0 => very difficult expecting p = 0.2-0.8expecting p = 0.2-0.8

must => 0.7must => 0.7 should => 0.5should => 0.5

Power of Power of DiscriminationDiscrimination

[r][r] (mean H - mean L)(mean H - mean L)//(full score)(full score) > 0.40 => very good item> 0.40 => very good item 0.30 - 0.39 => good item0.30 - 0.39 => good item 0.20 - 0.29 => borderline0.20 - 0.29 => borderline < 0.20 => poor item< 0.20 => poor item

ConclusionConclusion ValidityValidity ReliabilityReliability Difficulty IndexDifficulty Index Power of DiscriminationPower of Discrimination