1 EPSY 546: LECTURE 1 INTRODUCTION TO MEASUREMENT THEORY George Karabatsos.

EPSY 546: LECTURE 1

INTRODUCTION TO MEASUREMENT THEORY

George Karabatsos

What is test theory?

WHAT IS A TEST?Test: A procedure for obtaining a sample of person behavior from a specified domain of items.

WHAT IS A TEST?Test: A procedure for obtaining a sample of person behavior from a specified domain of items.

General: Exam, questionnaire, survey, judge-observed task, etc.

ITEM RESPONSE SCORINGTest item responses are scored. Some Examples:

Dichotomous : 1 = Correct, 0 = Incorrect

(Scored from possibly a multiple choice test item)


Rating Scale:1 = Strongly Disagree 2 = Disagree3 = Agree4 = Strongly Agree


Partial Credit:1 = Completely incorrect 2 = Partially correct3 = Completely correct

WHAT TESTS DOTests are designed to measure latent traits that manifest in the responses to the test items.

LATENT VARIABLESSome substantive examples of latent traits:Exam: Ability on long division.Attitude Questionnaire: Agreement towards capital punishment.Survey: Frequency of drug use.Survey: Quality of life.

LATENT VARIABLESLatent trait = latent variable = psychological trait/variable/attribute = unidimensional variable = construct

LATENT VARIABLESFor measurement, latent variables are often numerically represented either:

by total test score (person or item),

or by parameters of person ability or item difficulty.

Some Challenges of latent trait measurement (5)1. No single approach to the measurement of a latent trait is universally accepted.

Some Challenges of latent trait measurement (5)1. No single approach to the measurement of a latent trait is universally accepted.

** Two theorists may possibly selectdifferent items to measure a particularlatent trait (e.g., math ability).

Some Challenges of latent trait measurement (5)2. Psychological measurements are usually based on limited samples of behavior.


** Practically impossible to confront respondents with all possible items that represent the latent trait (e.g., all long division items)


** N = 1, for each person on an item.

Some Challenges of latent trait measurement (5)3. Latent trait measurement obtained is always subject to error.


Random: sampling error of respondents, and of items; inherent unreliability of respondents (e.g., boredom, lucky guess, carelessness).


Systematic:Cheating on exam; Response bias; item does not measure latent trait;misscoring; test form out of order.

Some Challenges of latent trait measurement (5)4. Establishing measurement scales for the latent trait.


Stevens (1946):the assignment of numerals or events according to rules. (NOT!)


Michell: Measurement requires tests of the hypothesis that the variable is quantitative. (Echoing Luce, Krantz, Suppes, Tversky, in three FM volumes)

Some Challenges of latent trait measurement (5)5. Latent traits must also demonstrate relationships to other important traits or observable phenomena.

Some Challenges of latent trait measurement (5)5. Latent traits must also demonstrate relationships to other important traits or observable phenomena.

**Measurements of latent traits have value when they can be related to other traits or events in the real world.

WHAT IS TEST THEORY?The study of the 5 pervasive measurement problems just described, and developing/applying methods for their resolution.

TEST THEORY COURSEBecome aware of the logic and mathematical models that underlie practices in test use and construction.

TEST THEORY COURSEAwareness of these models, including their assumptions and limitations, should lead to an improved practice in test construction and more intelligent use of test information in decision making.

TEST THEORY COURSETest theory provides general framework for viewing the process of instrument development.

Test theory distinguishes from the more applied subject of educational and psychological assessment (focuses on administration and interpretation of specific tests).

Process of Test Construction

TEST CONSTRUCTION10 steps can be followed to construct an test for the measurement of persons (and items).

(C&A, Chapter 4)

TEST CONSTRUCTION1. Identify the primary purpose(s) for which the test measurements will be used.

TEST CONSTRUCTION1. Identify the primary purpose(s) for which the test measurements will be used.

2. Hypothesize items that define the latent trait of interest.

TEST CONSTRUCTION3. Prepare a set of test specifications, delineating the proportion of items that should focus on each type of behavior identified in Step 2.

TEST CONSTRUCTION3. Prepare a set of test specifications, delineating the proportion of items that should focus on each type of behavior identified in Step 2.

4. Construct an initial pool of items.

TEST CONSTRUCTION5. Have items reviewed and revised.


6. Hold preliminary item tryouts (and revise).


6. Hold preliminary item tryouts (and revise).

7. Field test the items on a large sample representative of the examinee population for whom the test is intended. (PILOT STUDY)

TEST CONSTRUCTION8. Determine statistical properties of the items, and when appropriate, eliminate items that do not meet pre-established criteria.

TEST CONSTRUCTION8. Determine statistical properties of the items, and when appropriate, eliminate items that do not meet pre-established criteria.

9. Design and conduct reliability and validity studies for the final form of the test.

TEST CONSTRUCTION10. Develop guidelines for administration, scoring, and interpretation of the test scores. (e.g., prepare norm tables, suggest recommended cutting scores or standards for performance, etc.)

Statistical Concepts for Test Theory

BASIC STATISTICS (C&A2)Frequency tables and graphs

DistributionNormal distribution (p.d.f., c.d.f.)Central tendency: Mode, median, mean.Variability: Variance, standard deviation.Z - scoresFor infinite populations.

BASIC STATISTICS (C&A2)Relationship between two variablesScatterplot.Pearsons correlation coefficient.Ordinary linear regression.Standard error of Y predictions, for a given regression equation.

BASIC STATISTICS (C&A5)Statistics: Test ItemsMean and total score for an item, over respondents (item difficulty).

Variance of responses on a test item

Inter-item correlation (Pearsons product moment correlation or phi-correlation)

VARIANCE OF TEST SCORES AND TEST ITEMSSince tests are usually scored by the sum of the item scores, it follows that there should be some relationship between individual item variances and the variance of the total test scores.

VARIANCE OF TEST SCORES AND TEST ITEMSIn fact, since the measurement of individual differences is a central goal of testing, one goal of test construction should be to maximize the variance of the total test scores. The reliability and validity of a test depends on this variance.

VARIANCE OF TEST SCORES AND TEST ITEMSCovariance between items i and j :

N = Number of respondentsJ = number of items = population mean

VARIANCE OF TEST SCORES AND TEST ITEMS

Variance-Covariance Matrix


Total Test Score Variance =

Sum of item variances + sum of item covariances


Implications of Equation (first term)

Total test score variance increases as the number of items (J) is increased.

(except when the added items have a nonpositive correlation with the other items).


Implications of Equation (second term)

Test score variance increases when items are added that have positive covariances with the other test items.


Implications of EquationTest score variance is maximized when:

items are equal in difficulty (this increases item covariances),

and of medium difficulty (this increases item variances).

Introduction To Scaling

4 SCALES OF MEASUREMENT1. Nominal Scale:

Used for classification.

Assigns the same numbers to objects that are equivalent, and a different number to objects that are not.

4 SCALES OF MEASUREMENT1. Nominal Scale:

Class of admissible transformations: class of one-to-one transformations.

i.e., ni(x) = ni(y) iff nj(x) = nj(y) for all scales i, j, and objects x, y.

4 SCALES OF MEASUREMENT2. Ordinal Scale:

With respect to some attribute, this scale orders objects in magnitude, but does not measure distances between the objects.

Example: Ranking

4 SCALES OF MEASUREMENT2. Ordinal Scale:

Class of admissible transformations: class of increasing monotonic transformations.

i.e., ni(x) > ni(y) iff njj(x) > nj(y) for all scales i, j, and objects x, y.

4 SCALES OF MEASUREMENT3. Interval Scale:Involves the numerical representation of relation upon the differences between entities with respect to some attribute. (no absolute zero point)

Example: temperature measurement. (Fahrenheit, Celsius)

4 SCALES OF MEASUREMENT3. Interval Scale:Class of admissible transformations: class of positive linear transformations. nj(x) = a[ni(x)] + b for a > 0, 0 0e.g., C = (5/9)F (160/9)

4 SCALES OF MEASUREMENT4. Ratio Scale:Has properties of order, equal distance between units, and an absolute zero point.Non-zero measurements on this scale may be expressed as ratios of one another.Examples: Length, weight, etc.

4 SCALES OF MEASUREMENT4. Ratio Scale:Class of admissible transformations: class of multiplicative transformations

ni(x) = [ nj(x) ] c, for c > 0

MEASUREMENTAs mentioned earlier, establishing a measurement scale for a given variable requires hypothesis tests.

The measurement of directly observable, physical phenomena is easily obtainable and verifiable.

MEASUREMENTHowever, this is not the case for the measurement of latent psychological phenomena (e.g., ability, intelligence, attitudes, beliefs, etc.), which are not directly observable.

CONJOINT MEASUREMENTThe axioms of conjoint measurement can be tested to determine whether latent traits are measurable on an ordinal or interval scale.

INDEPENDENCE AXIOM (row)

ITEMS

Hard ((((((( Easy

j = 1

2

3

Test

Score

Group

3

(i = 1)

P11

P12

P13

4

(i = 2)

P21

P22

P23

5

(i = 3)

P31

P32

P33

W1 Premise W1 Implication

Monotone Homogeneity (MH)

Chart15

0.020.210.14

0.060.260.15

0.410.260.15

0.420.50.16

0.440.50.16

0.630.50.23

0.860.520.42

0.870.530.63

0.870.580.74

0.870.950.74

0.880.950.76

-2

0

1

Theta

Pj(Theta)

Sheet1

110.7d

00.20c

10.51.5a

-201delta

-5.05.26.00

-4.12.30.00

-3.27.35.00

-2.50.42.01

-1.73.50.03

0.88.60.13

1.95.70.35

2.98.78.57

3.99.85.67

41.00.90.69

51.00.94.70

MONOTONE HOMOGENIETY

-201

-5.02.21.14

-4.06.26.15

-3.41.26.15

-2.42.50.16

-1.44.50.16

0.63.50.23

1.86.52.42

2.87.53.63

3.87.58.74

4.87.95.74

5.88.95.76

DOUBLE MONOTONICYY

-201

-5.11.26.01

-4.33.26.03

-3.33.26.03

-2.61.26.03

-1.62.27.11

0.63.50.23

1.86.52.42

2.87.53.63

3.87.64.74

4.87.95.74

5.88.95.76

Sheet1

-2

0

1

Theta

Pj(Theta)

Sheet2

-2

0

1

Theta

Pj(Theta)

Sheet3

0.020.210.14

0.330.260.15

0.330.260.15

0.350.50.16

0.620.50.16

0.630.50.23

0.860.520.42

0.870.530.63

0.870.640.74

0.870.950.74

0.880.950.76

-2

0

1

Theta

Pj(Theta)

2PL:

Chart8

0.04742587320.075858180.0001233946

0.1192029220.1192029220.0005527786

0.26894142140.18242552380.0024726232

0.50.26894142140.0109869426

0.73105857860.37754066880.0474258732

0.8807970780.50.1824255238

0.95257412680.62245933120.5

0.982013790.73105857860.8175744762

0.99330714910.81757447620.9525741268

0.99752737680.8807970780.9890130574

0.99908894880.924141820.9975273768

-2

0

1

Theta

Pj(Theta)

Sheet1

-201

-5.05.01.00

-4.12.02.01

-3.27.05.02

-2.50.12.05

-1.73.27.12

0.88.50.27

1.95.73.50

2.98.88.73

3.99.95.88

41.00.98.95

51.00.99.98

61.001.00.99

71.001.001.00

81.001.001.00

91.001.001.00

101.001.001.00

10.51.5

-201

-5.05.08.00

-4.12.12.00

-3.27.18.00

-2.50.27.01

-1.73.38.05

0.88.50.18

1.95.62.50

2.98.73.82

3.99.82.95

41.00.88.99

51.00.921.00

Sheet1

-2

0

1

Theta

Pj(Theta)

Sheet2

-2

0

1

Theta

Pj(Theta)

Sheet3

3PL:

Chart9

0.04742587320.2606865440.0001233946

0.1192029220.29536233760.0005527786

0.26894142140.3459404190.0024726232

0.50.41515313710.0109869426

0.73105857860.5020325350.0474258732

0.8807970780.60.1824255238

0.95257412680.6979674650.5

0.982013790.78484686290.8175744762

0.99330714910.8540595810.9525741268

0.99752737680.90463766240.9890130574

0.99908894880.9393134560.9975273768

-2

0

1

Theta

Pj(Theta)

Sheet1

111d

00.20c

10.51.5a

-201delta

-5.05.26.00

-4.12.30.00

-3.27.35.00

-2.50.42.01

-1.73.50.05

0.88.60.18

1.95.70.50

2.98.78.82

3.99.85.95

41.00.90.99

51.00.941.00

Sheet1

-2

0

1

Theta

Pj(Theta)

Sheet2

Sheet3

4PL:

Chart10

0.04742587320.2606865440.0000863762

0.1192029220.29536233760.000386945

0.26894142140.3459404190.0017308362

0.50.41515313710.0076908598

0.73105857860.5020325350.0331981112

0.8807970780.60.1276978667

0.95257412680.6979674650.35

0.982013790.78484686290.5723021333

0.99330714910.8540595810.6668018888

0.99752737680.90463766240.6923091402

0.99908894880.9393134560.6982691638

-2

0

1

Theta

Pj(Theta)

Sheet1

110.7d

00.20c

10.51.5a

-201delta

-5.05.26.00

-4.12.30.00

-3.27.35.00

-2.50.42.01

-1.73.50.03

0.88.60.13

1.95.70.35

2.98.78.57

3.99.85.67

41.00.90.69

51.00.94.70

Sheet1

-2

0

1

Theta

Pj(Theta)

Sheet2

Sheet3

INDEPENDENCE AXIOM (column)

ITEMS

Hard ((((((( Easy

j = 1

2

3

Test

Score

Group

3

(i = 1)

P11

P12

P13

4

(i = 2)

P21

P22

P23

5

(i = 3)

P31

P32

P33



ISOP (Scheiblechner 1995)

Chart3

0.20.220.0136695196

0.210.240.15

0.220.30.15

0.240.310.15

0.240.310.16

0.310.310.17

0.310.310.19

0.420.50.22

0.430.550.38

0.430.70.43

0.470.80.47

0.670.920.61

0.70.930.64

0.950.960.7

-1

1

3

Theta

Pr[Correct Response]

Sheet1

delta = -1delta = 1delta = 3

-113

-50.020.000.00

-40.050.010.00

-30.120.020.00

-20.270.050.01

-10.500.120.02

00.730.270.05

10.880.500.12

20.950.730.27

30.980.880.50

40.990.950.73

51.000.980.88

61.000.990.95

71.001.000.98

81.001.000.99


-113

-50.120.000.02

-40.180.000.03

-30.270.000.05

-20.380.010.08

-10.500.030.12

00.620.150.18

10.730.500.27

20.820.850.38

30.880.970.50

40.920.990.62

50.951.000.73

60.971.000.82

70.981.000.88

80.991.000.92

0.76

0.25


-113

-50.340.000.01

-40.390.000.02

-30.450.000.04

-20.530.010.06

-10.630.030.09

00.720.150.14

10.800.500.20

20.860.850.29

30.910.970.38

40.940.990.47

50.961.000.56

60.981.000.62

70.991.000.67

80.991.000.70


-113

-50.200.220.01

-40.210.240.15

-30.220.300.15

-20.240.310.15

-10.240.310.16

00.310.310.17

10.310.310.19

20.420.500.22

30.430.550.38

40.430.700.43

50.470.800.47

60.670.920.61

70.700.930.64

80.950.960.70

Sheet1

000

000

000

000

000

000

000

000

000

000

000

000

000

000

-1

1

3

Theta


Sheet2

-1

1

3

Theta


Sheet3

0.25 delta = -1 -1

0.25 delta = 1 1

0.25 delta = 3 3

Theta


-1

1

3

Theta


RASCH-1PL:

Chart7

0.04742587320.00669285090.0024726232

0.1192029220.017986210.0066928509

0.26894142140.04742587320.01798621

0.50.1192029220.0474258732

0.73105857860.26894142140.119202922

0.8807970780.50.2689414214

0.95257412680.73105857860.5

0.982013790.8807970780.7310585786

0.99330714910.95257412680.880797078

0.99752737680.982013790.9525741268

0.99908894880.99330714910.98201379

-2

0

1

Theta

Pj(Theta)

Sheet1

-201

-5.05.01.00

-4.12.02.01

-3.27.05.02

-2.50.12.05

-1.73.27.12

0.88.50.27

1.95.73.50

2.98.88.73

3.99.95.88

41.00.98.95

51.00.99.98

61.001.00.99

71.001.001.00

81.001.001.00

91.001.001.00

101.001.001.00

Sheet1

-2

0

1

Theta

Pj(Theta)

Sheet2

Sheet3

Thomsen condition(e.g.,double cancellation)

ITEMS

Hard ((((((( Easy

j = 1

2

3

Test

Score

Group

3

(i = 1)

P11

P12

P13

4

(i = 2)

P21

P22

P23

5

(i = 3)

P31

P32

P33

ITEM RESPONSE FUNCTION

PARAMETERS

Model

Mono.

Slope

Intersect

Set (

Ability (i

1

Rasch

SI

Const

N

(, (

2

DM

ND

Var

N

3

2PL

I

Var

Y

(, (, a

4

3PL

I

Var

Y

(, (, a, c

No

5

4PL

I

Var

Y

(, (, a, c , d

No

6

MH

ND

Var

Y

_1077002409.unknown

_1077002448.unknown

_1077002624.unknown

_1077002352.unknown

MH analysisICC Crossings

Chart3

0.0060.0060.0070.0070.0070.008000000

0.0340.0330.1070.120.3380.3770.0310.0310.1060.1180.3380.376

0.1060.1160.1720.3130.6120.6830.1050.1150.1710.3130.6130.683

0.1920.2470.4080.6030.7250.8230.1930.2470.4090.6020.7260.824

0.3410.4580.6320.8260.8270.9140.340.4570.6330.8270.8280.915

0.6990.6810.8360.9130.90.9610.6980.6820.8380.9160.9030.964

0.9910.9910.9910.9910.9910.992111111

1

2

3

4

5

6

1

2

3

4

5

6

Test Score Group


NAEP

===============================================================

**************** NON-PARAMETRIC IRT ESTIMATION *****************

Markov Chain Monte-Carlo: Metropolis-Hastings/Gibbs sampler

===============================================================

PWW data analysis

===============================================================

ITERATION & ALGORITHM REPORT

MH sample file: NAEP-MH

ISOP sample file: NAEP-ISOP

Iterations interpreted for posterior distribution= 1000 to 11000 Total iterations =10001

Burn in Iterations = 1 to 999

Starting values for Thetas

0.020.170.230.430.620.76

0.020.170.250.440.650.78

0.080.180.260.450.660.81

0.090.180.350.460.670.85

0.10.20.360.470.680.92

0.150.210.410.50.710.93

0.160.220.410.50.760.93

===============================================================

================================================

GLOBAL MODEL FIT ANALYSIS

================================================

MODELMean DevPenaltyDICPred p-val

MH53.83829.8983.7280.461

ISOP53.59425.60179.1950.499

================================================

========================================================================================================================================================

POSTERIOR PREDICTIVE P-VALUES (CHI-SQUARE) : MH & ISOP

MHISOP

========================================================================================================================================================

123456Row Fit123456Row Fit

10.6620.6770.6630.6440.6350.5680.35810.9850.9220.790.6170.430.2480.385

20.5540.540.5250.5040.4970.5210.50320.580.5040.5570.550.5340.5210.537

30.5440.5180.540.5120.5070.5150.52630.5560.5280.5140.5180.5410.4990.543

40.530.5150.4990.4980.5150.5160.49940.5180.5160.5020.5170.5110.5220.509

50.5270.5080.5030.5210.5020.5130.50750.4980.5170.5060.5140.5220.5180.487

60.5060.4870.5270.5340.5360.5510.47460.4790.4720.5190.4440.4850.5420.416

70.6520.6440.6520.6530.6420.6790.21170.2040.4010.5870.7620.910.9810.191

Col Fit0.4950.4830.4740.4440.430.3980.461Col Fit0.5530.5380.5470.4830.3870.2410.499

========================================================================================================================================================

========================================================================================================================================================

================================================================================================================================================================================================================================================================================================

MH POSTERIOR DISTRIBUTION: MEAN 1% 2.5% 97.5% 99%

================================================================================================================================================================================================================================================================================================

MEAN1%5%97.50%99%

================================================================================================================================================================================================================================================================================================

123456123456123456123456123456

10.0060.0060.0070.0070.0070.0081000000100000010.0230.0220.0240.0260.0270.02610.0270.0260.030.0330.0340.03

20.0340.0330.1070.120.3380.37720.0170.0170.0750.0850.2860.32620.0190.0180.080.0890.2930.33420.0520.0520.140.1510.3840.42520.0570.0570.1450.1590.3920.432

30.1060.1160.1720.3130.6120.68330.0810.0910.1420.2770.570.64130.0840.0950.1470.2810.5740.64730.1290.140.2020.3470.6490.71930.1330.1440.2070.3530.6570.726

40.1920.2470.4080.6030.7250.82340.160.2110.3640.5550.6850.78740.1650.2160.370.5660.6930.79240.2220.2780.4430.6430.7580.8540.2290.2860.4530.6510.7640.854

50.3410.4580.6320.8260.8270.91450.30.4120.5850.7890.7890.88550.3050.4180.5930.7950.7960.8950.3780.4960.6710.8550.8560.93550.3870.5030.6780.8590.860.939

60.6990.6810.8360.9130.90.96160.6340.6130.7820.8720.860.93360.6470.6250.7910.8780.8660.93860.7510.7350.8750.9410.930.97960.7560.7420.8840.9450.9350.982

70.9910.9910.9910.9910.9910.99270.9590.9620.960.9570.9590.96770.9650.9670.9670.9680.9670.97271111117111111

================================================================================================================================================================================================================================================================================================

================================================================================================================================================================================================================================================================================================

================================================================================================================================================================================================================================================================================================

ISOP POSTERIOR DISTRIBUTION: MEAN 1% 2.5% 97.5% 99%

================================================================================================================================================================================================================================================================================================

MEAN1%5%97.50%99%

================================================================================================================================================================================================================================================================================================

123456123456123456123456123456

10.0010.0020.0040.0060.0090.0151000.0010.0010.0020.0041000.0010.0020.0030.00510.0040.0070.010.0140.0210.03410.0050.0080.0110.0160.0230.04

20.0280.0380.1020.1240.3360.38220.0150.0220.0730.0950.2870.33520.0160.0240.0770.0990.2920.34220.0420.0560.1270.1550.3770.42720.0450.0610.1310.1630.3820.435

30.1030.120.1720.3130.6130.68230.080.0980.1410.2750.570.6430.0830.1010.1460.2810.5790.64830.1220.1420.2020.350.6480.71630.1250.1470.2060.3580.6530.722

40.1930.2480.4090.6020.7240.82340.1610.2120.3690.5590.6860.78840.1660.2180.3750.5670.6920.79540.2220.280.4440.6380.7560.8540.2270.2860.450.6460.7630.854

50.340.4580.6320.8170.8350.91450.2940.4080.5850.7840.8050.88550.3020.4170.5920.790.810.88950.3790.4990.6710.8410.860.93550.390.5060.6770.8450.8650.939

60.6770.7010.8350.8990.9140.96160.6310.6570.7810.8640.8840.93260.6360.6620.7930.8690.8890.93860.7120.7420.8740.9240.9370.9860.720.7540.880.9270.940.983

70.9760.9860.9910.9940.9970.99870.9410.9660.9760.9840.9890.99370.9510.970.980.9860.9910.99470.9910.9960.9980.9991170.9920.9970.9980.99911

================================================================================================================================================================================================================================================================================================

================================================================================================================================================================================================================================================================================================

========================================================================================================================================================================================

DATA: Proportions n (successes) N (trials)

========================================================================================================================================================================================

ProportionsItemsSuccessesTrials

123456123456123456

100000010000001145145145145145145

20.0310.0310.1060.1180.3380.3762131344491411572417417417417417417

30.1050.1150.1710.3130.6130.683375821222234374873713713713713713713

40.1930.2470.4090.6020.7260.82441391782954355245954722722722722722722

50.340.4570.6330.8270.8280.91552002693724864875385588588588588588588

60.6980.6820.8380.9160.9030.96462152102582822782976308308308308308308

711111171071071071071071077107107107107107107

========================================================================================================================================================================================

========================================================================================================================================================================================

========================================================================================================================

TESTING LOCAL ITEM INDEPENDENCE

========================================================================================================================

123456123456

10.5050.50.5010.4990.50610.4990.5040.4980.5230.519

20.5050.5020.5050.50320.5040.5110.5020.513

30.510.5060.50630.4950.5090.509

40.5030.50840.5060.521

50.49850.497

66

========================================================================================================================

========================================================================================================================

PROPORTIONS

================================================================

Items

123456

0000000

10.0310.0310.1060.1180.3380.376

Test20.1050.1150.1710.3130.6130.683

Score30.1930.2470.4090.6020.7260.824

40.340.4570.6330.8270.8280.915

50.6980.6820.8380.9160.9030.964

6111111

================================================================

================================================================

Items

123456123456

00.0060.0060.0070.0070.0070.008000000

10.0340.0330.1070.120.3380.3770.0310.0310.1060.1180.3380.376

20.1060.1160.1720.3130.6120.6830.1050.1150.1710.3130.6130.683

30.1920.2470.4080.6030.7250.8230.1930.2470.4090.6020.7260.824

40.3410.4580.6320.8260.8270.9140.340.4570.6330.8270.8280.915

50.6990.6810.8360.9130.90.9610.6980.6820.8380.9160.9030.964

60.9910.9910.9910.9910.9910.992111111

NAEP

000000000000

000000000000

000000000000

000000000000

000000000000

000000000000

000000000000

1

2

3

4

5

6

1

2

3

4

5

6

Test Score Group


DM analysis

Chart4

0.0010.0020.0040.0060.0090.015000000

0.0280.0380.1020.1240.3360.3820.0310.0310.1060.1180.3380.376

0.1030.120.1720.3130.6130.6820.1050.1150.1710.3130.6130.683

0.1930.2480.4090.6020.7240.8230.1930.2470.4090.6020.7260.824

0.340.4580.6320.8170.8350.9140.340.4570.6330.8270.8280.915

0.6770.7010.8350.8990.9140.9610.6980.6820.8380.9160.9030.964

0.9760.9860.9910.9940.9970.998111111

1

2

3

4

5

6

1

2

3

4

5

6

Test Score Group


NAEP

===============================================================



===============================================================

PWW data analysis

===============================================================







0.020.170.230.430.620.76

0.020.170.250.440.650.78

0.080.180.260.450.660.81

0.090.180.350.460.670.85

0.10.20.360.470.680.92

0.150.210.410.50.710.93

0.160.220.410.50.760.93

===============================================================

================================================


================================================

MODELMean DevPenaltyDICPred p-val

MH53.83829.8983.7280.461

ISOP53.59425.60179.1950.499

================================================

========================================================================================================================================================

POSTERIOR PREDICTIVE P-VALUES (CHI-SQUARE) : MH & ISOP

MHISOP

========================================================================================================================================================

123456Row Fit123456Row Fit

10.6620.6770.6630.6440.6350.5680.35810.9850.9220.790.6170.430.2480.385

20.5540.540.5250.5040.4970.5210.50320.580.5040.5570.550.5340.5210.537

30.5440.5180.540.5120.5070.5150.52630.5560.5280.5140.5180.5410.4990.543

40.530.5150.4990.4980.5150.5160.49940.5180.5160.5020.5170.5110.5220.509

50.5270.5080.5030.5210.5020.5130.50750.4980.5170.5060.5140.5220.5180.487

60.5060.4870.5270.5340.5360.5510.47460.4790.4720.5190.4440.4850.5420.416

70.6520.6440.6520.6530.6420.6790.21170.2040.4010.5870.7620.910.9810.191

Col Fit0.4950.4830.4740.4440.430.3980.461Col Fit0.5530.5380.5470.4830.3870.2410.499

========================================================================================================================================================

========================================================================================================================================================

================================================================================================================================================================================================================================================================================================


================================================================================================================================================================================================================================================================================================

MEAN1%5%97.50%99%

================================================================================================================================================================================================================================================================================================

123456123456123456123456123456

10.0060.0060.0070.0070.0070.0081000000100000010.0230.0220.0240.0260.0270.02610.0270.0260.030.0330.0340.03

20.0340.0330.1070.120.3380.37720.0170.0170.0750.0850.2860.32620.0190.0180.080.0890.2930.33420.0520.0520.140.1510.3840.42520.0570.0570.1450.1590.3920.432

30.1060.1160.1720.3130.6120.68330.0810.0910.1420.2770.570.64130.0840.0950.1470.2810.5740.64730.1290.140.2020.3470.6490.71930.1330.1440.2070.3530.6570.726

40.1920.2470.4080.6030.7250.82340.160.2110.3640.5550.6850.78740.1650.2160.370.5660.6930.79240.2220.2780.4430.6430.7580.8540.2290.2860.4530.6510.7640.854

50.3410.4580.6320.8260.8270.91450.30.4120.5850.7890.7890.88550.3050.4180.5930.7950.7960.8950.3780.4960.6710.8550.8560.93550.3870.5030.6780.8590.860.939

60.6990.6810.8360.9130.90.96160.6340.6130.7820.8720.860.93360.6470.6250.7910.8780.8660.93860.7510.7350.8750.9410.930.97960.7560.7420.8840.9450.9350.982

70.9910.9910.9910.9910.9910.99270.9590.9620.960.9570.9590.96770.9650.9670.9670.9680.9670.97271111117111111

================================================================================================================================================================================================================================================================================================

================================================================================================================================================================================================================================================================================================

================================================================================================================================================================================================================================================================================================


================================================================================================================================================================================================================================================================================================

MEAN1%5%97.50%99%

================================================================================================================================================================================================================================================================================================

123456123456123456123456123456

10.0010.0020.0040.0060.0090.0151000.0010.0010.0020.0041000.0010.0020.0030.00510.0040.0070.010.0140.0210.03410.0050.0080.0110.0160.0230.04

20.0280.0380.1020.1240.3360.38220.0150.0220.0730.0950.2870.33520.0160.0240.0770.0990.2920.34220.0420.0560.1270.1550.3770.42720.0450.0610.1310.1630.3820.435

30.1030.120.1720.3130.6130.68230.080.0980.1410.2750.570.6430.0830.1010.1460.2810.5790.64830.1220.1420.2020.350.6480.71630.1250.1470.2060.3580.6530.722

40.1930.2480.4090.6020.7240.82340.1610.2120.3690.5590.6860.78840.1660.2180.3750.5670.6920.79540.2220.280.4440.6380.7560.8540.2270.2860.450.6460.7630.854

50.340.4580.6320.8170.8350.91450.2940.4080.5850.7840.8050.88550.3020.4170.5920.790.810.88950.3790.4990.6710.8410.860.93550.390.5060.6770.8450.8650.939

60.6770.7010.8350.8990.9140.96160.6310.6570.7810.8640.8840.93260.6360.6620.7930.8690.8890.93860.7120.7420.8740.9240.9370.9860.720.7540.880.9270.940.983

70.9760.9860.9910.9940.9970.99870.9410.9660.9760.9840.9890.99370.9510.970.980.9860.9910.99470.9910.9960.9980.9991170.9920.9970.9980.99911

================================================================================================================================================================================================================================================================================================

================================================================================================================================================================================================================================================================================================

========================================================================================================================================================================================


========================================================================================================================================================================================


123456123456123456

100000010000001145145145145145145

20.0310.0310.1060.1180.3380.3762131344491411572417417417417417417

30.1050.1150.1710.3130.6130.683375821222234374873713713713713713713

40.1930.2470.4090.6020.7260.82441391782954355245954722722722722722722

50.340.4570.6330.8270.8280.91552002693724864875385588588588588588588

60.6980.6820.8380.9160.9030.96462152102582822782976308308308308308308

711111171071071071071071077107107107107107107

========================================================================================================================================================================================

========================================================================================================================================================================================

========================================================================================================================

TESTING LOCAL ITEM INDEPENDENCE

========================================================================================================================

123456123456

10.5050.50.5010.4990.50610.4990.5040.4980.5230.519

20.5050.5020.5050.50320.5040.5110.5020.513

30.510.5060.50630.4950.5090.509

40.5030.50840.5060.521

50.49850.497

66

========================================================================================================================

========================================================================================================================

PROPORTIONS

================================================================

Items

123456

0000000

10.0310.0310.1060.1180.3380.376

Test20.1050.1150.1710.3130.6130.683

Score30.1930.2470.4090.6020.7260.824

40.340.4570.6330.8270.8280.915

50.6980.6820.8380.9160.9030.964

6111111

================================================================

================================================================

Items

123456123456

00.0010.0020.0040.0060.0090.015000000

10.0280.0380.1020.1240.3360.3820.0310.0310.1060.1180.3380.376

20.1030.120.1720.3130.6130.6820.1050.1150.1710.3130.6130.683

30.1930.2480.4090.6020.7240.8230.1930.2470.4090.6020.7260.824

40.340.4580.6320.8170.8350.9140.340.4570.6330.8270.8280.915

50.6770.7010.8350.8990.9140.9610.6980.6820.8380.9160.9030.964

60.9760.9860.9910.9940.9970.998111111

NAEP

000000000000

000000000000

000000000000

000000000000

000000000000

000000000000

000000000000

1

2

3

4

5

6

1

2

3

4

5

6

Test Score Group


Model Selection & Evaluation


===============================================

MODEL

Penalty

DIC

Pred p-value

DM

28.0

53.6

25.6

79.2

.50

MH

23.9

53.8

29.9

83.7

.46

===============================================

_1077018375.unknown

_1077018376.unknown

Model Assessment: Detailed

NAEP

===============================================================



===============================================================

PWW data analysis

===============================================================







0.020.170.230.430.620.76

0.020.170.250.440.650.78

0.080.180.260.450.660.81

0.090.180.350.460.670.85

0.10.20.360.470.680.92

0.150.210.410.50.710.93

0.160.220.410.50.760.93

===============================================================

================================================


================================================

MODELMean DPenaltyDICPred p-value

MH53.829.983.70.461

ISOP53.625.679.20.499

================================================

========================================================================================================================================================

POSTERIOR PREDICTIVE P-VALUES (CHI-SQUARE) : MH & ISOP=================================================================================================================================================================================================================

MHPOSTERIOR PREDICTIVE P-VALUES: DM MODEL

=========================================================================================================================================================================================================================================================================================

Item

123456Row Fit123456Group Fit

10.6620.6770.6630.6440.6350.5680.3580.99.92.79.62.43.25.39

20.5540.540.5250.5040.4970.5210.5031.58.50.56.55.53.52.54

30.5440.5180.540.5120.5070.5150.526Score2.56.53.51.52.54.50.54

40.530.5150.4990.4980.5150.5160.499Group3.52.52.50.52.51.52.51

50.5270.5080.5030.5210.5020.5130.5074.50.52.51.51.52.52.49

60.5060.4870.5270.5340.5360.5510.4745.48.47.52.44.49.54.42

70.6520.6440.6520.6530.6420.6790.2116.20.40.59.76.91.98.19

Col Fit0.4950.4830.4740.4440.430.3980.461Item Fit.55.54.55.48.39.24.50

====================================================================================================================================================================

====================================================================================================================================================================

================================================================================================================================================================================================================================================================================================


================================================================================================================================================================================================================================================================================================

MEAN1%5%97.50%99%

================================================================================================================================================================================================================================================================================================

123456123456123456123456123456

10.0060.0060.0070.0070.0070.0081000000100000010.0230.0220.0240.0260.0270.02610.0270.0260.030.0330.0340.03

20.0340.0330.1070.120.3380.37720.0170.0170.0750.0850.2860.32620.0190.0180.080.0890.2930.33420.0520.0520.140.1510.3840.42520.0570.0570.1450.1590.3920.432

30.1060.1160.1720.3130.6120.68330.0810.0910.1420.2770.570.64130.0840.0950.1470.2810.5740.64730.1290.140.2020.3470.6490.71930.1330.1440.2070.3530.6570.726

40.1920.2470.4080.6030.7250.82340.160.2110.3640.5550.6850.78740.1650.2160.370.5660.6930.79240.2220.2780.4430.6430.7580.8540.2290.2860.4530.6510.7640.854

50.3410.4580.6320.8260.8270.91450.30.4120.5850.7890.7890.88550.3050.4180.5930.7950.7960.8950.3780.4960.6710.8550.8560.93550.3870.5030.6780.8590.860.939

60.6990.6810.8360.9130.90.96160.6340.6130.7820.8720.860.93360.6470.6250.7910.8780.8660.93860.7510.7350.8750.9410.930.97960.7560.7420.8840.9450.9350.982

70.9910.9910.9910.9910.9910.99270.9590.9620.960.9570.9590.96770.9650.9670.9670.9680.9670.97271111117111111

================================================================================================================================================================================================================================================================================================

================================================================================================================================================================================================================================================================================================

================================================================================================================================================================================================================================================================================================


================================================================================================================================================================================================================================================================================================

MEAN1%5%97.50%99%

================================================================================================================================================================================================================================================================================================

123456123456123456123456123456

10.0010.0020.0040.0060.0090.0151000.0010.0010.0020.0041000.0010.0020.0030.00510.0040.0070.010.0140.0210.03410.0050.0080.0110.0160.0230.04

20.0280.0380.1020.1240.3360.38220.0150.0220.0730.0950.2870.33520.0160.0240.0770.0990.2920.34220.0420.0560.1270.1550.3770.42720.0450.0610.1310.1630.3820.435

30.1030.120.1720.3130.6130.68230.080.0980.1410.2750.570.6430.0830.1010.1460.2810.5790.64830.1220.1420.2020.350.6480.71630.1250.1470.2060.3580.6530.722

40.1930.2480.4090.6020.7240.82340.1610.2120.3690.5590.6860.78840.1660.2180.3750.5670.6920.79540.2220.280.4440.6380.7560.8540.2270.2860.450.6460.7630.854

50.340.4580.6320.8170.8350.91450.2940.4080.5850.7840.8050.88550.3020.4170.5920.790.810.88950.3790.4990.6710.8410.860.93550.390.5060.6770.8450.8650.939

60.6770.7010.8350.8990.9140.96160.6310.6570.7810.8640.8840.93260.6360.6620.7930.8690.8890.93860.7120.7420.8740.9240.9370.9860.720.7540.880.9270.940.983

70.9760.9860.9910.9940.9970.99870.9410.9660.9760.9840.9890.99370.9510.970.980.9860.9910.99470.9910.9960.9980.9991170.9920.9970.9980.99911

================================================================================================================================================================================================================================================================================================

================================================================================================================================================================================================================================================================================================

========================================================================================================================================================================================


========================================================================================================================================================================================


123456123456123456

100000010000001145145145145145145

20.0310.0310.1060.1180.3380.3762131344491411572417417417417417417

30.1050.1150.1710.3130.6130.683375821222234374873713713713713713713

40.1930.2470.4090.6020.7260.82441391782954355245954722722722722722722

50.340.4570.6330.8270.8280.91552002693724864875385588588588588588588

60.6980.6820.8380.9160.9030.96462152102582822782976308308308308308308

711111171071071071071071077107107107107107107

========================================================================================================================================================================================

========================================================================================================================================================================================

========================================================================================================================

TESTING LOCAL ITEM INDEPENDENCETESTING LOCAL ITEM INDEPENDENCE

========================================================================================================================

MH MODELISOP MODEL

ItemItem

123456123456

10.5050.50.5010.4990.5061.50.50.50.52.52

20.5050.5020.5050.5032.50.51.50.51

30.510.5060.5063.50.51.51

40.5030.5084.51.52

50.4985.50

66

========================================================================================================================

========================================================================================================================

PROPORTIONS

================================================================

Items

123456

0000000

10.0310.0310.1060.1180.3380.376

Test20.1050.1150.1710.3130.6130.683

Score30.1930.2470.4090.6020.7260.824

40.340.4570.6330.8270.8280.915

50.6980.6820.8380.9160.9030.964

6111111

================================================================

================================================================

Items

123456123456

00.0010.0020.0040.0060.0090.015000000

10.0280.0380.1020.1240.3360.3820.0310.0310.1060.1180.3380.376

20.1030.120.1720.3130.6130.6820.1050.1150.1710.3130.6130.683

30.1930.2480.4090.6020.7240.8230.1930.2470.4090.6020.7260.824

40.340.4580.6320.8170.8350.9140.340.4570.6330.8270.8280.915

50.6770.7010.8350.8990.9140.9610.6980.6820.8380.9160.9030.964

60.9760.9860.9910.9940.9970.998111111

NAEP

0.0010.0020.0040.0060.0090.015000000

0.0280.0380.1020.1240.3360.3820.0310.0310.1060.1180.3380.376

0.1030.120.1720.3130.6130.6820.1050.1150.1710.3130.6130.683

0.1930.2480.4090.6020.7240.8230.1930.2470.4090.6020.7260.824

0.340.4580.6320.8170.8350.9140.340.4570.6330.8270.8280.915

0.6770.7010.8350.8990.9140.9610.6980.6820.8380.9160.9030.964

0.9760.9860.9910.9940.9970.998111111

1

2

3

4

5

6

1

2

3

4

5

6

Test Score Group


Model Assessment: DetailedPerson Fit Posterior Item PredictiveExaminee Responses P-value 2154 110100 .67 279 101001 .12 987 000011 .00

Date post:	31-Dec-2015
Category:	Documents
Upload:	louisa-pope
View:	220 times
Download:	2 times

1 EPSY 546: LECTURE 1 INTRODUCTION TO MEASUREMENT THEORY George Karabatsos.

Documents